Summary
In the first two articles of this series, we saw why high-quality data models are the foundation for business-friendly mapping, and how we reverse engineer physical source models into logical ones.
But having data models is not enough.
We also need to control them.
And the best way to do that is to treat metadata just like any other data source:
loaded, versioned, historized, validated, and monitored.
This mindset is what transforms a data platform into a truly metadata-driven architecture.
What We Mean by Metadata
Metadata is often defined as “data about data”. But in practice, for a data platform, it includes:
- Data models (logical and physical, entities, attributes, relationships).
- Mappings between logical and physical models.
- Transformations (functions, filters, derivations).
- Business rules and reference codes.
- Lineage (where data comes from, how it changes, who consumes it).
This metadata is the blueprint for all ETL/ELT pipelines, dashboards, and data services.
If it’s wrong, everything downstream is at risk.
Metadata as a Data Source
To control metadata, we apply the same principles we use for source data:
- Load it
- Import metadata from modeling tools (Erwin, PowerDesigner, SqlDbm, Ellie.ai, CrossModel).
- Accept delivery formats like JSON, YAML, CSV from providers.
- Reverse engineer DDL or schema files if no model exists.
Map it
- Align source metadata with our internal generic metadata model.
- This model defines how all data models are represented internally, independent of tool or format.
Version it
- Every metadata load creates a new version.
- Changes are tracked over time, just like historical data.
Historize it
- Using the same dual SCD2 methodology we apply to business data.
- This allows us to see exactly how a model evolved over time.
Validate it
- Apply data quality checks to metadata.
- Example: does every column have a datatype? Is every foreign key resolvable?
- Validate provider metadata against expectations when new versions are delivered.
Dashboards for Metadata
Just as business users need dashboards for KPIs, data engineers need dashboards for metadata health.
These show:
- Which models changed since last load.
- Which attributes were added, renamed, or deprecated.
- Whether source metadata deliveries passed validation.
- The impact of upcoming changes on downstream consumers.
This transparency makes metadata governance actionable instead of theoretical.
Why Metadata Quality Matters
In a metadata-driven platform, ETL/ELT pipelines are generated from metadata and mappings.
That means:
- Bad metadata = broken code.
- Uncontrolled changes = production failures.
- Missing lineage = blind impact analysis.
When metadata is treated as first-class data:
- We always know which pipelines are affected by a change.
- We can automatically notify consumers of impacted columns or tables.
- We can simulate changes before applying them.
This is what makes a data platform resilient and scalable.
Example: Source Metadata Delivery
Imagine a provider delivers a new JSON schema for customer data.
Our process would:
- Load the schema into the generic metadata model.
- Compare with previous version → identify changes.
- Historize the metadata (dual SCD2).
- Validate rules (e.g. no dropped keys without notice).
- Update metadata dashboards.
- Trigger impact analysis to see which mappings and consumers are affected.
All automated. All controlled.
Closing Thoughts
Metadata is not an afterthought. It is the lifeblood of a metadata-driven data platform.
By treating metadata as data, we gain the ability to:
- Load, map, version, historize, and validate it.
- See changes over time.
- Anticipate impacts.
- Keep automation safe and reliable.
This discipline ensures that our business-friendly mappings are always grounded in trusted, high-quality metadata.
In the next article, we’ll take this a step further: using logical source models to generate Source Access Views (SAVs) — the bridge between raw historized data and business-friendly integration.