How 7,450+ AI-generated tests actually work — and what it means for test managers How 7,450+ AI-generated tests actually work — and what it means for test managers Marc Teunissen, a former colleague, asked me a pointed question after seeing that our metadata framework had over 7,000 unit tests: “How does AI generate 7,000 tests? What are they based on? And how do you reproduce bugs when the ‘developer’ is an AI?” These are exactly...

Why Your Queries May Return Different Results Every Time — And Why It Matters Recently, a data team spent several days investigating what appeared to be a serious defect. Their validation tests kept failing. The same query produced different results in different runs. After careful analysis, they uncovered the real root cause: SELECT customer_id, region, ROW_NUMBER() OVER (PARTITION BY region) AS rn FROM customers The migration was correct. The SQL was broken. Without an ORDER...

MDDE is not just a metadata repository Most articles talk about modeling. Few talk about engines. MDDE is not just a metadata repository. It is an engine composed of: Parser layer Metadata writer Optimizer pipeline Proposal system SQL generator Diagnostics framework Let’s look inside. Each optimizer pass follows the same contract: def run(model_id: str, db: Connection) -> OptimizationReport: ... Each pass: Scans metadata Detects patterns Emits diagnostics Optionally produces proposals Never mutates blindly Diagnostics are...

Why generating SQL from intent proved safer than maintaining it by hand. Why generating SQL from intent proved safer than maintaining it by hand. Once SQL is treated as a structured artifact rather than text, a new question inevitably arises: If intent can be parsed, analyzed, and validated — why is SQL still written manually? For a long time, hand-written SQL feels like the most direct and flexible option. It gives engineers control, expressiveness, and...

How systematic comparison reshaped the way I think about data architecture. How systematic comparison reshaped the way I think about data architecture. Regression testing is usually discussed in operational terms. It’s something teams do to reduce risk, verify outcomes, or gain confidence during change. Important, but typically downstream from architecture and design. During recent migration work, that perspective shifted. Regression testing stopped being a validation step and became something more fundamental: a way to make...

How to unify reproducible historization, governed deduplication, and data quality observability into one metadata-driven framework. 🧩 Part of the Model-Driven Data Quality series — exploring reproducible, metadata-driven patterns for historization, deduplication, and bi-temporal data management. In previous articles, we looked at two critical components of reproducible data pipelines: Historizing Source Tables Without a Primary Key — how to build bi-temporal SCD2 tables based on deterministic hash identities. Correct, Governed Deduplication in SQL (Without Breaking Your Pipelines) — how to detect, flag,...

How to automatically validate migrated SQL with generated test datasets When modernizing data platforms, SQL logic is often rewritten or migrated — for example, from Azure Databricks to Snowflake, or between different engines within the same architecture. The challenge: how do we prove that these migrated or optimized queries still produce the same results? In this article, we show how to use synthetic and golden datasets to automatically validate query equivalence across modern data platforms — without exposing production...

How metadata, business-friendly mappings, and AI-assisted specifications redefine data engineering. In data architecture, clarity is power. Data platforms thrive when every transformation has a single source of truth — when intent, code, and validation all trace back to a clear and verifiable definition. That’s the promise of Spec-Driven Development (SDD): building and evolving systems where specifications — not raw code — drive development. It’s an idea emerging fast in the software world, and it fits perfectly into...

A practical approach to consistent, testable data pipelines. Use source subtypes as translation blocks to integrate data into a clean, business-aligned model without duplication. Press enter or click to view image in full size One query pulls from raw source tables. Another joins a derived table directly. Yet another mixes historized and current records inconsistently. The result? Fragile, irreproducible transformations. We solve this with Access Views (AVs). Every downstream transformation reads only from AVs, never directly from...