Bridging Declarative Execution and Metadata-Driven Coordination
When Databricks introduced Declarative Pipelines (Lakeflow), it sparked a subtle but important question:
If pipelines are declarative, do we still need orchestration?
A reader asked me exactly that after my recent article on declarative execution. It’s a great question — because it highlights a common misconception:
Declarative pipelines simplify execution, but they don’t eliminate orchestration. They change what orchestration means, and more importantly, where it lives.
1. The Old World: Imperative Orchestration
In the traditional world, orchestration meant control. We designed elaborate DAGs — in Airflow, ADF, or Step Functions — that encoded exactly how data should flow:
- Run job A, then B, then C.
- Retry on failure.
- Wait for dependency X to complete.
- Notify Y on success.
This worked, but it came at a cost: Every dependency, every branch, every SLA was hand-wired into code. Orchestration became the hardest part to maintain — tightly coupled to operational details, brittle to change, and hard to reason about at scale.
2. The New World: Declarative Execution
Declarative pipelines turn that model inside out. Instead of defining how to run data transformations, we describe what the system should produce — and let the platform infer the rest.
In Databricks Lakeflow (and similar systems like dbt Cloud or Dagster assets), each pipeline declares:
- Its inputs and outputs
- Its logic, expressed declaratively (Select, Derive, Filter, Join, etc.)
- Its constraints (via EXPECT or test clauses)
From that description, the platform automatically builds the DAG, manages retries, and handles lineage. Orchestration within the pipeline becomes implicit. Dependencies are resolved from metadata, not manually wired.
We call this intra-pipeline orchestration, and it’s now declarative.
3. When You Still Need Orchestration
But in any real-world architecture, not everything lives in one pipeline or one domain. You still need to coordinate:
- Multiple data domains or data products
- External ingestion processes or APIs
- Streaming and batch flows that interact
- ML pipelines consuming curated data
Declarative execution doesn’t remove this need — it relocates orchestration to a higher level, where it becomes metadata-driven rather than code-driven.
We use a lightweight central orchestrator (e.g., Airflow, Lakeflow control plane, or even an event bus) — not to define step sequences, but to react to metadata.
4. Metadata-Driven Choreography
Here’s the key shift: Instead of scripting control flow, we let metadata define choreography.
Each pipeline publishes its contracts:
- Output schema and freshness
- Data quality status
- Contract version or semantic model
The orchestrator — or an event-driven trigger — simply listens for these signals. When an upstream dataset is updated and passes its quality constraints, dependent pipelines activate automatically.
No DAG rewiring, no hardcoded dependencies — just metadata-based coordination.
That’s what we mean by data choreography: Each domain knows its part of the dance, and coordination happens through shared metadata, not central control.
5. Architecture in Practice
In our metadata-driven architecture, we organize execution into three layers:
Declarative Pipelines → Execute logic within a data domain
Metadata Contracts → Describe dependencies and quality
Orchestrator / Event Hub → Coordinate across domains
Within each layer:
- Declarative Pipelines (e.g., Databricks Lakeflow) manage transformation and dependency logic automatically.
- The Metadata Layer (our “Common Integration Layer”) acts as the single source of truth for schema, lineage, and freshness.
- The Orchestration Layer becomes event-driven — triggering only when metadata contracts are satisfied.
In short: the orchestration brain has moved into metadata.
6. Why This Matters
This architecture changes both governance and agility:
✅ Less boilerplate.
No manual DAG definitions — dependencies are inferred.
✅ Built-in quality.
Pipelines declare their EXPECT clauses, and orchestration only continues when contracts are met.
✅ Unified batch and streaming.
The same declarative model runs in both modes without rewriting logic.
✅ Simpler change management.
New dependencies appear automatically in metadata; no code changes required in orchestration scripts.
In large enterprises, where hundreds of pipelines span multiple teams and domains, this shift reduces friction dramatically. Orchestration becomes a metadata event network — transparent, adaptive, and resilient.
7. The Big Shift: From Imperative Control to Semantic Flow
Declarative Pipelines didn’t kill orchestration — they demoted it.
The “intelligence” of the data platform now lives in metadata contracts and declarative intent, not in procedural scripts.
That’s the essence of our Continuous Metadata Integration philosophy:
- Spec-Driven Development (SDD): metadata defines intent
- Semantic DevOps: deployment and validation through metadata contracts
- Declarative Execution: the platform handles operational details
Together, these principles turn orchestration into choreography — a coordinated, self-describing flow of data between domains.
Closing Thoughts
In a declarative world, orchestration doesn’t disappear; it becomes context-aware. The system no longer asks, “what should I run next?” — it knows, because the metadata already says so.
That’s the quiet revolution behind declarative pipelines: Not less orchestration — smarter orchestration.
Related Reading
Part of the Model-Driven Data Architecture series.