The Problem Most data platforms have architectural principles. No SELECT * Deterministic joins Naming conventions Explicit keys Reusable logic No implicit cartesian joins Stable projection ordering They live in slides. Or wiki pages. Or Confluence. But they do not live in the system. And when architecture is not executable, it becomes advisory. Advisory architecture erodes under delivery pressure. Metadata-Driven Data Engineering treats architecture as queryable structure. Once SQL is parsed into metadata: Entities exist Attributes...

Strong architectures need rules. But real systems need exceptions. Strong architectures need rules. But real systems need exceptions. If an architectural approach cannot deal with exceptions explicitly, they will still happen — just invisibly, inconsistently, and without learning. This article explains how MDDE treats exceptions as first-class architectural signals , not failures of discipline. Many architectures fail in one of two ways: Rules are enforced too weakly → everything becomes “best effort” Rules are enforced...

Why guidelines alone are not enough Most data teams have guidelines. They live in: Wikis Confluence pages Readme files Slide decks They are usually well-intended, carefully written, and widely ignored. Not because people disagree with them — but because guidelines that rely on memory and discipline do not scale , especially under delivery pressure. MDDE made a deliberate shift: If a guideline matters, it must be executable. This article explains what that means in practice....

Bridging Teradata’s NORMALIZE and the Islands & Gaps Problem — for Bi-Temporal SCD2 Architectures in Modern SQL Platforms When working with historized data — especially in a bi-temporal SCD2 architecture — we often encounter redundant timelines, also known in SQL as the Islands and Gaps Problem. The issue becomes visible when a single source table contains multiple business aspects (like customer master data) that evolve independently over time. Let’s take a simple example. Imagine a source...

A deterministic, auditable pattern to find, flag, and quarantine duplicates — while row-level DQ “breadcrumbs” flow all the way to your data products. ROW_NUMBER() must have an ORDER BY. Without it your “first row” is non-deterministic. Don’t hide evidence with DISTINCT/UNION/GROUP BY. Instead: Detect → Flag → Route. Add row-level DQ status/labels and reasons; carry them downstream. Publish two outputs: a golden dataset (winners) and a quarantine (losers for analysis). Track metrics and alert on spikes — keep pipelines running while making issues visible. – ❌ Non-deterministic...

How rules and metadata turned SQL migration into a governed, explainable process. 💡 Summary As our AI-assisted SQL migration matured, something unexpected happened: it started to behave like a governance system. By combining metadata, YAML rules, and validation feedback, we didn’t just automate SQL rewriting — we made every decision explainable, every transformation traceable, and every piece of logic reusable. This is the moment where technical automation became data governance in action. Press enter or click to view...

How rule categories helped us turn complex SQL into clean, modular, and governed transformations. 💡 Summary After extracting metadata and defining our governance framework, we faced the next challenge: how to translate hundreds of SQL quirks into consistent transformation patterns. The solution came from organizing our YAML rules into categories — each representing a specific type of logic. By applying these rule sets systematically, we transformed legacy SQL into modular building blocks that were cleaner, faster, and auditable....

How to turn messy legacy pipelines into a scalable, transparent, and future‑ready data foundation. If your organization has been around for more than a decade, chances are your ETL landscape is packed with SQL code that nobody wants to touch. Layers of historization logic, business rules, and data quality fixes often live in the same queries—making them fragile, hard to debug, and expensive to maintain. But there’s good news: migrating to a modern dual‑SCD2 platform...

How to write safer, more maintainable queries. 📖 This article is part of my series on SQL Best Practices for Clear & Concise Data Transformation. Each post tackles a common anti-pattern and shows a cleaner way to write queries that are easier to test, reuse, and maintain. Ever opened a SQL query and seen a giant inline UNION ALL crammed inside a FROM clause — complete with casts, filters, and calculations sprinkled in? It might work, but it’s messy, hard to...