Traditional data diagrams show what data looks like and how it moves. Model-Driven Data Engineering unites both — turning metadata into an executable design that powers automation, governance, and understanding.
Summary
Entity Relationship Diagrams (ERDs) visualize structure.
Data Lineage visualizes movement.
Model-Driven Data Engineering combines both — using metadata as code to define, generate, and validate everything from models to mappings, SQL, and documentation.Instead of drawing what already exists, we now design once and generate many.
Press enter or click to view image in full size
1. The Limits of Static Diagrams in a Dynamic World
For decades, data teams have drawn ERDs to represent how tables relate and lineage diagrams to show how data flows.
Both are useful lenses — but they describe reality after it’s built.
In modern cloud ecosystems, where schemas evolve daily and transformations live in hundreds of SQL files, manually maintained diagrams quickly lose relevance.
What was once a “map of truth” becomes a static artifact of yesterday’s data.
We don’t just need visualizations. We need living metadata that keeps the system aligned as it changes.
2. Two Views, One Problem
- ERDs describe the static structure — entities, attributes, relationships, and keys.
- Lineage describes the dynamic flow — how data moves and transforms through pipelines.
But both live in isolation.
The ERD doesn’t know that “Customer” in your data warehouse comes from “Counterparty” in the source.
The lineage doesn’t know what “Customer” means.
This disconnect between structure and flow fragments understanding.
It’s why new data engineers spend weeks deciphering transformations, and why business stakeholders often distrust the numbers they see.
3. The Bridge: Executable Metadata
The answer is not another diagram. It’s a shift in where truth lives.
In Model-Driven Data Engineering, metadata is the system design.
It defines entities, attributes, and relationships — just like an ERD — but also the mappings, rules, and transformations that describe how data flows — just like lineage.
This metadata is stored as code (often in YAML or JSON) and becomes the single source from which we generate:
- ERDs and logical models
- SQL transformations and dbt models
- Lineage and impact graphs
- Documentation and data dictionaries
- Unit tests and plausibility checks
Everything downstream — including code — is derived, not drawn.
4. Business-Oriented Data Modeling
In this approach, models are not purely technical.
They are business-oriented: built around concepts that domain experts understand — Customer, Contract, Product, Event.
Each entity connects to business definitions in a shared vocabulary, allowing both engineers and analysts to reason in the same language.
This alignment bridges the classic gap between business semantics and technical implementation.
Rather than a static ERD hidden in a tool, the model becomes a living artifact: validated, versioned, and shared across teams.
5. Business-Friendly Data Mapping
Lineage diagrams tell you where data moves, but not why.
They rarely explain what transformations occur or how business meaning changes.
Model-Driven Data Engineering introduces business-friendly mappings — readable YAML structures that describe how source attributes map to target entities and how transformations apply.
For example:
source: AnaCredit.Counterparty_Identifier
target: Counterparty.counterparty_id
rule: trim(left(value, 20))
meaning: Unique identifier of the counterparty
From this single definition, the framework can generate:
- Transformation SQL
- Mapping diagrams
- Column-level lineage
- Validation tests
These mappings are both human-readable and machine-executable, making them the missing semantic layer between ERD and lineage.
6. Universal Data Models: Anchors of Consistency
In traditional modeling, each system has its own ERD — isolated, inconsistent, and named differently.
A Universal Data Model (UDM) provides the shared backbone across domains.
It defines canonical entities and attributes — the “universal nouns” of the enterprise.
Each new data model or mapping references these entities, ensuring consistent naming, relationships, and governance across all platforms.
This way, lineage across domains becomes semantic, not just technical.
A “Customer” remains a Customer — whether in the CRM, warehouse, or analytical mart.
7. From Visualization to Generation
In a model-driven architecture:
- The model defines both structure and flow
- The mappings define semantics and transformation logic
- The platform generates SQL, lineage, diagrams, and documentation automatically
This reverses the old order:
Instead of documenting what we’ve built, we build what we’ve documented.
The ERD, lineage graph, and data catalog are no longer separate deliverables — they’re views of the same metadata.
8. Why It Matters
- For engineers: less duplication, faster iteration, automated testing, and reliable refactoring.
- For analysts: clear semantics and traceability from business terms to SQL.
- For governance: lineage, ownership, and data quality are built in, not bolted on.
- For the enterprise: a shared, living model that drives every artifact — from documentation to deployment.
This is the foundation of Business-Oriented, Model-Driven Data Engineering — where diagrams, lineage, and transformations all originate from one executable source of truth.
Key Takeaways
- ERDs show structure. Lineage shows movement.
Model-Driven Data Engineering unifies both into executable metadata. - Metadata as code enables generation of SQL, lineage, and documentation directly from models.
- Business-oriented models ensure shared understanding between business and engineering.
- Universal Data Models provide semantic consistency across domains.
- Governance becomes emergent — lineage, ownership, and validation are outcomes, not manual tasks.
💡 This article is part of the ongoing series on metadata-driven data architecture — exploring how business-oriented data modeling, business-friendly mappings, and universal data models converge into a unified model-first methodology.