Follow-up to “From ERDs and Lineage to Executable Metadata”
When I published From ERDs and Lineage to Executable Metadata, a reader asked:
“Do you advocate generating config and code from diagrams_, or generating diagrams_ from metadata_?
And if it’s the latter, what does that metadata look like — and why not use standards like UML, BPMN, or XMI to capture meaning graphically?”_
A great question — and one that gets to the heart of the shift from descriptive diagrams to executable metadata.
1 The Direction of Generation
Traditional approaches start with visual models (ERDs, BPMN, UML) and then try to synchronize them with code.
In Model-Driven Data Engineering (MDDE), we invert that logic:
- We define the system declaratively as structured metadata — YAML or JSON.
- We generate diagrams, SQL, lineage, and documentation from that metadata.
The diagrams become views of the metadata, not sources of truth.
This inversion ensures every visualization, script, and catalog entry remains in sync — because all are derived from one living definition.
2 What Executable Metadata Looks Like
Executable metadata combines structural and behavioral information in a machine-readable, version-controlled format:
entity: Counterparty
attributes:
– name: counterparty_id
type: string
business_definition: Unique identifier of the counterparty
– name: country_code
type: string
domain: ISO3166
mapping:
source: AnaCredit.Counterparty_Identifier
target: Counterparty.counterparty_id
rule: trim(left(value, 20))
meaning: Aligns AnaCredit identifiers to enterprise counterparty model
From this single file, the framework can generate:
- SQL transformation code
- ERD and lineage diagrams
- Data-dictionary entries
- Validation and plausibility tests
Every artifact you see or run is a projection of the same underlying model.
3 YAML vs UML: A Quick Comparison
Press enter or click to view image in full size
In practice, UML excels at communicating models, while YAML excels at executing them.
MDDE keeps both: the YAML is the source, the UML-style diagram is generated output.
4 Side-by-Side Example
UML-Style Class Diagram (Conceptual)
+——————-+
| Counterparty |
+——————-+
| counterparty_id |
| country_code |
+——————-+
^
|
+——————-+
| AnaCreditRecord |
+——————-+
| Counterparty_ID |
| CountryCode |
+——————-+
This diagram conveys relationships, but it lacks transformation logic, validation, or runtime behavior.
YAML Executable Metadata (Same Concept)
mappings:
– source_entity: AnaCreditRecord
target_entity: Counterparty
rules:
– target: counterparty_id
expression: trim(left(source.Counterparty_ID, 20))
tests: [not_null, unique]
– target: country_code
expression: map_country(source.CountryCode)
domain: ISO3166
This version defines not only structure and mapping but also validation and domain semantics — all interpretable by code generators.
5 Why Not Just Use XMI or BPMN?
XMI and BPMN attempted to make diagrams machine-readable, but they carry three practical limitations in modern data-engineering workflows:
- Tool Lock-In — most XMI exports are proprietary extensions of UML tools.
- Verbosity — hundreds of lines of XML for a simple model make diffing and merging painful.
- Limited Runtime Semantics — they describe shape and sequence, not executable transformations or data rules.
YAML/JSON offers a simpler contract: human-readable, Git-friendly, composable, and easy to parse in any programming language.
It’s not “just YAML magic” — it’s metadata that’s structured, validated, and interpreted by generators that produce concrete artifacts.
6 A Reference Structure for Executable Metadata
A technology-independent schema typically includes:
- Model Layer — entities, attributes, relationships, keys
- Mapping Layer — source-target links, transformation expressions, lineage anchors
- Governance Layer — ownership, business definitions, validation rules
- Semantic Layer — domains, vocabularies, and universal data-model references
This can live in a hierarchical folder structure:
/metadata
/models
customer.yaml
counterparty.yaml
/mappings
anacredit_to_counterparty.yaml
/domains
ISO3166.yaml
Each layer can be validated against a JSON Schema, versioned in Git, and consumed by generators or visualization services.
7 From Drawing to Declaring
In the MDDE paradigm, we no longer “draw and deploy.”
We declare and derive.
Visual models, lineage graphs, and documentation are views of a single metadata source.
When the metadata changes, everything else — code, diagrams, and governance — updates automatically.
That is the moment metadata stops being documentation and becomes part of the operational system.
8 Closing Thought
Model-Driven Data Engineering doesn’t reject UML or BPMN — it extends their spirit into a world of automation.
Executable metadata gives us what those visual standards always aspired to: a shareable, technology-independent, living specification that both humans and machines can trust.
💡 This article is part of the ongoing series on metadata-driven data architecture — exploring how business-oriented modeling, business-friendly mappings, and universal data models converge into a unified, executable methodology.