How simple control tables evolve into full model-driven data platforms
The Starting Point: Metadata as Control
In a recent article by Franco Patano, Databricks showed how easy it is to create a metadata-driven ingestion framework using Workflows and dynamic SQL.
Instead of hardcoding pipelines, you define metadata in a control table — describing what data to load, how, and where — and let a simple loop dynamically generate and execute the ingestion SQL.
It’s a clean demonstration of what many of us in data engineering have been striving for:
Replace hand-coded pipelines with metadata-driven automation.
That concept alone already transforms the way we think about scale. But it’s only the beginning.
Beyond Ingestion: The Metadata-Driven Mindset
The real breakthrough happens when we apply the same pattern beyond ingestion — across all layers of the data platform.
In our metadata-driven architecture, we treat metadata not just as configuration, but as a living model that drives and validates the entire data lifecycle.
Once metadata becomes the single source of truth, every part of the data platform — from ingestion to delivery — can be generated, validated, and documented automatically.
The Logical Progression
Patano’s example shows the seed of metadata-driven execution. Your architecture framework is what happens when that seed grows into a full ecosystem.
Metadata Capture
Functional metadata is stored declaratively — for example, in YAML, tables, or DuckDB models.
It defines mappings, transformations, and lineage.
Metadata Interpretation
Scripts or generators interpret this metadata to produce SQL, dbt models, or Databricks workflows.
Execution & Orchestration
Declarative pipelines (like those now supported in Databricks) execute this logic dynamically — using the metadata as runtime context.
Validation & Observability
Test cases, lineage views, and DQ rules are generated from the same metadata, ensuring transparency and trust.
Publishing & Governance
Documentation, definitions, and lineage are automatically published to business-facing portals (e.g., Confluence or Notion).
The result is an end-to-end metadata-driven ecosystem, not just a control-table hack.
Why This Evolution Matters
Most data teams still operate in the “control table” phase: parameterized ingestion, but hand-built logic everywhere else. This is a bottleneck — technical debt in disguise.
A truly metadata-driven architecture:
- Reduces human-coded logic to a minimum.
- Brings functional consistency across all pipelines.
- Enables safe migration and version control through declarative models.
- Bridges the gap between logical design and physical execution.
It’s the natural next step for organizations embracing data contracts, declarative pipelines, and metadata-as-code principles.
Declarative Pipelines: The Execution Layer
Databricks’ new Declarative Pipeline features now allow this entire philosophy to reach the execution layer itself. We can finally connect the dots:
Logical metadata → Generated transformations → Declarative execution.
This closes the loop between design-time and runtime, letting your metadata describe not only what to build but also how it runs.
In Practice: A Framework for Metadata-Driven Delivery
In our model-driven framework, every pipeline is defined by a metadata record — describing:
- Source and target datasets
- Mapping rules
- Join logic
- DQ checks
- Validity periods
- Versioning and lineage links
The engine then generates all physical SQL, tests, and documentation automatically. Adding a new source means adding metadata — not code.
Over time, the metadata repository itself becomes a knowledge graph of the data platform, describing how everything fits together.
Why the Future Is Declarative
Metadata-driven ingestion was the first step.
Metadata-driven architecture is the destination.
It brings together:
- Declarative design
- Dynamic execution
- Automated documentation
- Embedded quality and lineage
And it does so in a way that’s maintainable, auditable, and explainable — exactly what modern data platforms need.