How auxiliary tables extend your data model without polluting your core schema ** How auxiliary tables extend your data model without polluting your core schema** In data engineering, we often need to track information about our data that doesn’t belong in the data itself. History of changes. Quality scores. Audit trails. Load statistics. Version information. The naive approach is to add columns directly to the main table: created_at, updated_at, created_by quality_score, last_checked_at version_number, previous_value But...

How documentation-as-code transforms AI from assistant to collaborator How documentation-as-code transforms AI from assistant to collaborator Series: Building a 6,600+ Test Framework with Claude Code Part 1: The Claude-First Architecture (this article) Part 2: MCP Integration — Tools, Not Just Chat Part 3: Practical Tips for AI-Assisted Development When people ask me about AI-assisted development, I don’t show them a chatbot writing a todo app. I show them MDDE — a metadata-driven data engineering framework...

Tools, not just chat: How Model Context Protocol transforms AI assistance Tools, not just chat: How Model Context Protocol transforms AI assistance Series: Building a 6,600+ Test Framework with Claude Code Part 1: The Claude-First Architecture Part 2: MCP Integration — Tools, Not Just Chat (this article) Part 3: Practical Tips for AI-Assisted Development In Part 1, we built the documentation foundation: CLAUDE.md, ADRs, and self-documenting schemas. That’s table stakes for effective AI collaboration. Now...

How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies In modern data platforms, metadata lives everywhere: ETL tools store pipeline schedules and dependencies Data catalogs store descriptions and classifications Governance platforms store PII tags and ownership Quality tools store validation rules BI tools store metric definitions The result? Metadata sprawl. The same column gets...

How sidecar schemas and quality attestations enable data consumers to make informed trust decisions How sidecar schemas and quality attestations enable data consumers to make informed trust decisions Every data pipeline has a dirty secret: quality checks happen, but the results disappear into logs. When data arrives at a downstream system, consumers face a fundamental question: “Can I trust this data?” Today, most data platforms answer this question with silence. The source system ran quality...

Moving beyond Confluence sprawl to a unified, AI-consumable documentation strategy Moving beyond Confluence sprawl to a unified, AI-consumable documentation strategy Every data team I’ve worked with has the same problem: documentation is everywhere and nowhere. Confluence : Data dictionaries, architecture decisions, meeting notes SharePoint : Business glossaries, compliance documents, data contracts Local drives : ERDs, design documents, “that one Excel with all the mapping rules” Wiki : Runbooks, troubleshooting guides, onboarding docs Code repos :...

BIRD Bench research reveals that well-structured data models achieve 95% SQL generation accuracy — without semantic layer tools BIRD Bench research reveals that well-structured data models achieve 95% SQL generation accuracy — without semantic layer tools MotherDuck’s analysis of the BIRD Benchmark — a rigorous SQL generation test — reveals a counterintuitive truth: “AI-ready data simply means a clean data model. Well-named tables, straightforward joins, and clear column names are enough for frontier LLMs to...

How 7,450+ AI-generated tests actually work — and what it means for test managers How 7,450+ AI-generated tests actually work — and what it means for test managers Marc Teunissen, a former colleague, asked me a pointed question after seeing that our metadata framework had over 7,000 unit tests: “How does AI generate 7,000 tests? What are they based on? And how do you reproduce bugs when the ‘developer’ is an AI?” These are exactly...

The Problem Every few months, someone claims: “AI will generate your entire data platform.” It won’t. Large language models do not understand your architecture. They do not understand lineage. They do not understand semantic stability. They generate text. Text is not structure. And SQL is not just text. AI becomes interesting only when it operates on structured metadata. Once SQL has been parsed into: entity attribute relationship attribute_mapping function_def filter_def entity_window optimizer_diagnostics The system is...

The Problem Most data platforms have architectural principles. No SELECT * Deterministic joins Naming conventions Explicit keys Reusable logic No implicit cartesian joins Stable projection ordering They live in slides. Or wiki pages. Or Confluence. But they do not live in the system. And when architecture is not executable, it becomes advisory. Advisory architecture erodes under delivery pressure. Metadata-Driven Data Engineering treats architecture as queryable structure. Once SQL is parsed into metadata: Entities exist Attributes...