Bridging the gap between schemas and meaning to unlock model-driven data engineering.
We’ve automated pipelines, scanned every schema, and still the business asks: “But what does this field mean?”
The day the “complete” data map failed
I was once in a meeting where the data engineering team proudly unveiled a comprehensive data map. Every column from every system was listed. Lineage was traced through dozens of pipelines. Profiling stats were perfect.
Then a marketing manager raised her hand.
“Which column tells me if a customer is active_? We want to track retention.”_
The room went silent. We had cust_status_code, active_flag, and status_date — but no one could explain what active meant to the business. The map was technically complete, but functionally useless.
That moment taught me: metadata without meaning isn’t enough.
Press enter or click to view image in full size
🛠 Technical metadata — the plumbing
Most modern data stacks excel at technical metadata:
- Tables, columns, data types
- File formats, partitions, indexes
- ETL jobs, schedules, pipeline lineage
- Profiling: row counts, null %, min/max values
It’s essential. It keeps pipelines running and helps engineers automate work.
But technical metadata stops at structure. It doesn’t explain whether revenue_amt is gross or net. It doesn’t say when a “customer” becomes a “member.” That’s where teams stumble.
🧭 Functional metadata — the meaning
Functional (business) metadata describes data in terms humans can use:
- Business definitions (“Customer: someone with at least one paid order”)
- KPI rules (how to calculate active users or churn)
- Data quality expectations (“Invoice date can’t be in the future”)
- Data ownership & stewardship
- Business process context (when and why data is created)
This is the semantic layer analysts and business users crave.
Without it, a data catalog is just a list of cryptic fields.
🚧 Why the split kills data initiatives
- Mappings lose intent. Engineers link columns but don’t know the business rule behind them.
- Trust evaporates. Business users can’t verify that dashboards match their reality.
- Change gets risky. A definition shift (“customer” → “account”) breaks pipelines no one knows about.
- Self-service stalls. Analysts spend hours decoding fields instead of answering questions.
🔗 Model-driven data engineering depends on both
Modern model-driven data engineering (MDE) tries to break this cycle by combining both metadata types.
- Start with a semantic model
Define business entities and KPIs first — not just tables. Capture their definitions and rules. - Attach technical context
Use schema scanners and lineage tools to map the semantic model to real sources automatically. - Generate, don’t hand-code
Build transformations, tables, and documentation from the combined metadata. - Keep it round-trip
When a definition changes, the model updates the pipelines.
When pipelines change, lineage updates the model.
The result: a platform that’s understandable to the business but still automatable for engineering.
⚡️ Practical ways to close the gap
- Business glossary first. Build and maintain a single place for definitions and KPI rules.
- Enrich your catalog. Add plain-language descriptions, stewards, and usage tips to each field.
- Version control your model. Let both engineers and business sign off on changes (like code reviews).
- Generate code from metadata. Use your unified model to produce DDL, pipelines, and docs automatically.
- Expose both views. A technical lineage view for engineers and a business semantics view for analysts.
🚀 The takeaway
Technical metadata keeps data moving.
Functional metadata makes data meaningful.
Model-driven engineering only works when you unite the two.
If your goal is business-friendly data mapping and modeling, don’t just crawl schemas — capture meaning.
That’s how data engineering finally becomes a bridge between technology and the business it serves.