Part 2: Code patterns, Databricks integration, and production recipes Part 2: Code patterns, Databricks integration, and production recipes In Part 1, we covered three strategies for multi-tenant DAG generation: Contract-Based Generation — Generate only relevant DAGs per entity Hierarchical Structure — Coordinator + entity-specific execution DAGs Dynamic Task Mapping — Single DAG with runtime-expanded tasks This article provides implementation patterns for Databricks-based pipelines. Your setup: Airflow orchestrates Python code that executes SQL in Databricks. The...

Practical patterns for metadata-driven orchestration in multi-entity data platforms Practical patterns for metadata-driven orchestration in multi-entity data platforms Your architecture is elegant: SQL dependencies derived from transformation code, Airflow calling Python loaders that check metadata conditions, a generic data model serving multiple entities. But you’re hitting a wall. For each entity, you generate all Airflow DAGs. Then the Python loader checks at runtime if the SQL statement should execute. This works, but it doesn’t scale:...

When should your pipelines run? Not on a schedule — when there’s new data to process. When should your pipelines run? Not on a schedule — when there’s new data to process. Hugo Lu’s article on State-Aware Orchestration identifies a fundamental issue: scheduled pipelines waste resources when: Data doesn’t land on time No new data has arrived since last run Models are forgotten but keep running Full refreshes happen when incremental would suffice His framing...

How enterprise teams are treating semantic models like production code How enterprise teams are treating semantic models like production code The days of Power BI as a “drag-and-drop reporting tool” are over. At Power BI Gebruikersdagen 2026, session after session drove home the same message: modern Power BI development is software development. PBIP (Power BI Project) format, TMDL, Git integration, and quality gates have transformed how enterprise teams build and deploy semantic models. This article...

The Problem Most data platforms have architectural principles. No SELECT * Deterministic joins Naming conventions Explicit keys Reusable logic No implicit cartesian joins Stable projection ordering They live in slides. Or wiki pages. Or Confluence. But they do not live in the system. And when architecture is not executable, it becomes advisory. Advisory architecture erodes under delivery pressure. Metadata-Driven Data Engineering treats architecture as queryable structure. Once SQL is parsed into metadata: Entities exist Attributes...

Every Model Workspace Generates Automation Scripts Most frameworks fail not because they lack features. They fail because they add friction. If engineers must remember five CLI commands, configure paths manually, or guess the correct sequence of steps — adoption slows down. MDDE was built under migration pressure. Which means: Workflows had to be repeatable. Deterministic. Low-friction. And safe. That is why every model workspace generates automation scripts. In a typical metadata-driven workflow, you need to:...

Architecture usually fades after delivery Most data architectures are strongest at the start of a project. Decisions are discussed. Diagrams are drawn. Guidelines are agreed upon. Then delivery accelerates — and architecture slowly fades into the background. New queries appear. Exceptions accumulate. Rules become optional. The problem is not intent. It’s the lack of continuous enforcement. MDDE addresses this by treating architecture as something that runs — not something that is reviewed once. This article...

Bridging Declarative Execution and Metadata-Driven Coordination When Databricks introduced Declarative Pipelines (Lakeflow), it sparked a subtle but important question: If pipelines are declarative, do we still need orchestration? A reader asked me exactly that after my recent article on declarative execution. It’s a great question — because it highlights a common misconception: Declarative pipelines simplify execution, but they don’t eliminate orchestration. They change what orchestration means, and more importantly, where it lives. In the traditional world, orchestration meant control. We...

Bridging Metadata and Execution — How Declarative Pipelines Turn Data Design into Automated Delivery When Databricks announced Declarative Data Pipelines — now even available on the Free Edition — it didn’t just simplify pipeline creation; it represented a convergence between how we design data transformations and how they’re executed. For years, our data architecture has been moving toward metadata-driven automation — where functional logic is described declaratively, and code is generated automatically. Declarative pipelines now allow that philosophy to extend all the way into...

How we built a rule engine that reads YAML, routes AI prompts, validates transformations, and writes explainable SQL. 💡 Summary At this point in the series, we had defined: - Structured metadata for every SQL query - A library of YAML rules representing transformation logic - A framework of AI prompts to apply those rules The missing piece was orchestration — a way to connect all of it. So we built a Python rule engine that loads the YAML rules, detects which apply, and...