How to configure VSCode and AI assistants for productive MDDE development How to configure VSCode and AI assistants for productive MDDE development Data engineers juggle multiple tools daily: SQL editors, YAML validators, Python IDEs, Git clients, documentation systems. Each tool works in isolation, forcing context switches that kill productivity. When working with Metadata-Driven Data Engineering (MDDE), this fragmentation becomes acute. You’re constantly moving between: Model definitions (YAML/MDDE files) Generated SQL (DDL, DML, quality checks) Metadata...

How auxiliary tables extend your data model without polluting your core schema ** How auxiliary tables extend your data model without polluting your core schema** In data engineering, we often need to track information about our data that doesn’t belong in the data itself. History of changes. Quality scores. Audit trails. Load statistics. Version information. The naive approach is to add columns directly to the main table: created_at, updated_at, created_by quality_score, last_checked_at version_number, previous_value But...

Tools, not just chat: How Model Context Protocol transforms AI assistance Tools, not just chat: How Model Context Protocol transforms AI assistance Series: Building a 6,600+ Test Framework with Claude Code Part 1: The Claude-First Architecture Part 2: MCP Integration — Tools, Not Just Chat (this article) Part 3: Practical Tips for AI-Assisted Development In Part 1, we built the documentation foundation: CLAUDE.md, ADRs, and self-documenting schemas. That’s table stakes for effective AI collaboration. Now...

How documentation-as-code transforms AI from assistant to collaborator How documentation-as-code transforms AI from assistant to collaborator Series: Building a 6,600+ Test Framework with Claude Code Part 1: The Claude-First Architecture (this article) Part 2: MCP Integration — Tools, Not Just Chat Part 3: Practical Tips for AI-Assisted Development When people ask me about AI-assisted development, I don’t show them a chatbot writing a todo app. I show them MDDE — a metadata-driven data engineering framework...

How sidecar schemas and quality attestations enable data consumers to make informed trust decisions How sidecar schemas and quality attestations enable data consumers to make informed trust decisions Every data pipeline has a dirty secret: quality checks happen, but the results disappear into logs. When data arrives at a downstream system, consumers face a fundamental question: “Can I trust this data?” Today, most data platforms answer this question with silence. The source system ran quality...

How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies In modern data platforms, metadata lives everywhere: ETL tools store pipeline schedules and dependencies Data catalogs store descriptions and classifications Governance platforms store PII tags and ownership Quality tools store validation rules BI tools store metric definitions The result? Metadata sprawl. The same column gets...

BIRD Bench research reveals that well-structured data models achieve 95% SQL generation accuracy — without semantic layer tools BIRD Bench research reveals that well-structured data models achieve 95% SQL generation accuracy — without semantic layer tools MotherDuck’s analysis of the BIRD Benchmark — a rigorous SQL generation test — reveals a counterintuitive truth: “AI-ready data simply means a clean data model. Well-named tables, straightforward joins, and clear column names are enough for frontier LLMs to...

Moving beyond Confluence sprawl to a unified, AI-consumable documentation strategy Moving beyond Confluence sprawl to a unified, AI-consumable documentation strategy Every data team I’ve worked with has the same problem: documentation is everywhere and nowhere. Confluence : Data dictionaries, architecture decisions, meeting notes SharePoint : Business glossaries, compliance documents, data contracts Local drives : ERDs, design documents, “that one Excel with all the mapping rules” Wiki : Runbooks, troubleshooting guides, onboarding docs Code repos :...

How 7,450+ AI-generated tests actually work — and what it means for test managers How 7,450+ AI-generated tests actually work — and what it means for test managers Marc Teunissen, a former colleague, asked me a pointed question after seeing that our metadata framework had over 7,000 unit tests: “How does AI generate 7,000 tests? What are they based on? And how do you reproduce bugs when the ‘developer’ is an AI?” These are exactly...

Series Context Merging Multiple SCD2 Timelines Without Explosion: ASOF Sampling, Gaps-and-Islands, and PIT Tables In Part 1, we standardized heterogeneous sources into controlled bi-temporal SCD2. Now comes the hard part. Once everything is SCD2… Joining them becomes dangerous. Each SCD2 table has its own change points. Consider a simple 360-degree customer view requiring three SCD2 sources: Combined timeline: 9 change points from just 3 sources with 3 changes each. Each change fragments the timeline. Add...