Part 2: Code patterns, Databricks integration, and production recipes Part 2: Code patterns, Databricks integration, and production recipes In Part 1, we covered three strategies for multi-tenant DAG generation: Contract-Based Generation — Generate only relevant DAGs per entity Hierarchical Structure — Coordinator + entity-specific execution DAGs Dynamic Task Mapping — Single DAG with runtime-expanded tasks This article provides implementation patterns for Databricks-based pipelines. Your setup: Airflow orchestrates Python code that executes SQL in Databricks. The...

Practical patterns for metadata-driven orchestration in multi-entity data platforms Practical patterns for metadata-driven orchestration in multi-entity data platforms Your architecture is elegant: SQL dependencies derived from transformation code, Airflow calling Python loaders that check metadata conditions, a generic data model serving multiple entities. But you’re hitting a wall. For each entity, you generate all Airflow DAGs. Then the Python loader checks at runtime if the SQL statement should execute. This works, but it doesn’t scale:...

Why attaching question-SQL pairs to your semantic model pushes accuracy past the baseline Why attaching question-SQL pairs to your semantic model pushes accuracy past the baseline Text-to-SQL systems have made remarkable progress. With a well-designed semantic model, modern LLMs can translate natural language questions into SQL with 70–80% accuracy. But that last 20–30% is stubborn. The system might generate syntactically correct SQL that doesn’t quite answer the question. Or it might miss subtle business logic....

When should your pipelines run? Not on a schedule — when there’s new data to process. When should your pipelines run? Not on a schedule — when there’s new data to process. Hugo Lu’s article on State-Aware Orchestration identifies a fundamental issue: scheduled pipelines waste resources when: Data doesn’t land on time No new data has arrived since last run Models are forgotten but keep running Full refreshes happen when incremental would suffice His framing...

How AI certification maps to data modeling challenges — and why the timing is perfect How AI certification maps to data modeling challenges — and why the timing is perfect When Anthropic launched the Claude Certified Architect (CCA) Foundations credential in March 2026, I didn’t expect it to feel so familiar. After 15 years in data architecture — designing Data Vaults, building dimensional models, and wrestling with metadata management — the exam domains read like...

How to collect, organize, document, and deploy specialized knowledge for Claude and other AI coding agents How to collect, organize, document, and deploy specialized knowledge for Claude and other AI coding agents AI coding agents are powerful, but they’re generalists. They’ve been trained on vast amounts of code, but that breadth comes at a cost — they often produce “plausible but wrong” output when working in specialized domains. Enter Agent Skills : structured knowledge bundles...

How enterprise teams are treating semantic models like production code How enterprise teams are treating semantic models like production code The days of Power BI as a “drag-and-drop reporting tool” are over. At Power BI Gebruikersdagen 2026, session after session drove home the same message: modern Power BI development is software development. PBIP (Power BI Project) format, TMDL, Git integration, and quality gates have transformed how enterprise teams build and deploy semantic models. This article...

Make your semantic model AI-ready with systematic checks Make your semantic model AI-ready with systematic checks Microsoft Copilot in Power BI is only as good as the semantic model it works with. Feed it a poorly documented model with cryptic names and implicit measures, and you’ll get hallucinated DAX and frustrated users. This article provides a comprehensive checklist of 15 requirements for Copilot-optimized semantic models — and shows how to validate them automatically. When Copilot...

Rule-based measure creation that scales Rule-based measure creation that scales Every Power BI semantic model needs measures. Total Sales. Average Order Value. Year-over-Year Growth. Count of Customers. Most teams write these by hand. Every. Single. Time. But measures follow patterns. If you can describe the pattern, you can generate the measure. This article shows how model-driven data engineering transforms entity definitions into complete DAX measures — automatically. Consider a simple entity: Sales with one numeric...

How to configure VSCode and AI assistants for productive MDDE development How to configure VSCode and AI assistants for productive MDDE development Data engineers juggle multiple tools daily: SQL editors, YAML validators, Python IDEs, Git clients, documentation systems. Each tool works in isolation, forcing context switches that kill productivity. When working with Metadata-Driven Data Engineering (MDDE), this fragmentation becomes acute. You’re constantly moving between: Model definitions (YAML/MDDE files) Generated SQL (DDL, DML, quality checks) Metadata...