Part 2: Code patterns, Databricks integration, and production recipes Part 2: Code patterns, Databricks integration, and production recipes In Part 1, we covered three strategies for multi-tenant DAG generation: Contract-Based Generation — Generate only relevant DAGs per entity Hierarchical Structure — Coordinator + entity-specific execution DAGs Dynamic Task Mapping — Single DAG with runtime-expanded tasks This article provides implementation patterns for Databricks-based pipelines. Your setup: Airflow orchestrates Python code that executes SQL in Databricks. The...

Why attaching question-SQL pairs to your semantic model pushes accuracy past the baseline Why attaching question-SQL pairs to your semantic model pushes accuracy past the baseline Text-to-SQL systems have made remarkable progress. With a well-designed semantic model, modern LLMs can translate natural language questions into SQL with 70–80% accuracy. But that last 20–30% is stubborn. The system might generate syntactically correct SQL that doesn’t quite answer the question. Or it might miss subtle business logic....

Rule-based measure creation that scales Rule-based measure creation that scales Every Power BI semantic model needs measures. Total Sales. Average Order Value. Year-over-Year Growth. Count of Customers. Most teams write these by hand. Every. Single. Time. But measures follow patterns. If you can describe the pattern, you can generate the measure. This article shows how model-driven data engineering transforms entity definitions into complete DAX measures — automatically. Consider a simple entity: Sales with one numeric...

How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies In modern data platforms, metadata lives everywhere: ETL tools store pipeline schedules and dependencies Data catalogs store descriptions and classifications Governance platforms store PII tags and ownership Quality tools store validation rules BI tools store metric definitions The result? Metadata sprawl. The same column gets...

Structural normalization and determinism SQL optimizers inside databases already reorder joins. So why would a metadata-driven framework care? Because there are two very different concerns: Runtime execution optimization (handled by the database engine) Structural normalization and determinism (handled by MDDE) These are not the same problem. MDDE does not attempt to replace the database optimizer. It ensures that the structural representation of joins is: Explicit Deterministic Canonical Traceable Before SQL ever reaches execution. In real...

Why Derived Queries Must Be Modeled — Not Rendered Most SQL complexity does not live in base tables. It lives in: CTEs Subqueries UNION chains Nested derived queries In many data platforms, these constructs are treated as textual blocks. Parsed. Rendered. Forgotten. MDDE treats them differently. Derived queries are modeled as entities. Not as strings. Consider: WITH filtered_orders AS ( SELECT * FROM orders WHERE order_date >= '2024-01-01' ), aggregated_orders AS ( SELECT customer_id, COUNT(*)...

Why Your Queries May Return Different Results Every Time — And Why It Matters Recently, a data team spent several days investigating what appeared to be a serious defect. Their validation tests kept failing. The same query produced different results in different runs. After careful analysis, they uncovered the real root cause: SELECT customer_id, region, ROW_NUMBER() OVER (PARTITION BY region) AS rn FROM customers The migration was correct. The SQL was broken. Without an ORDER...

MDDE is not just a metadata repository Most articles talk about modeling. Few talk about engines. MDDE is not just a metadata repository. It is an engine composed of: Parser layer Metadata writer Optimizer pipeline Proposal system SQL generator Diagnostics framework Let’s look inside. Each optimizer pass follows the same contract: def run(model_id: str, db: Connection) -> OptimizationReport: ... Each pass: Scans metadata Detects patterns Emits diagnostics Optionally produces proposals Never mutates blindly Diagnostics are...

Metadata-First DDL Generation. Most data platforms are not single-dialect anymore. In modern environments, you often see: Snowflake for analytics Databricks for processing BigQuery for BI DuckDB for local experimentation PostgreSQL for operational workloads And sometimes all of them in the same organization. The question becomes: How do you keep structural consistency across platforms without rewriting everything? MDDE approaches this through metadata-first DDL generation. Consider a simple table: CREATE TABLE customer ( customer_id VARCHAR(50) NOT NULL,...

A continuous feedback cycle In the previous article, we looked at the architecture absorption problem: As systems grow, structure disappears into SQL. MDDE does not solve this by replacing SQL. It solves it by creating a loop. Not a one-time migration step. Not a modeling exercise. A continuous feedback cycle. This is the MDDE Loop: Parse → Observe → Optimize → Generate And then back again. The loop always begins with existing SQL. Not conceptual...