When should your pipelines run? Not on a schedule — when there’s new data to process. When should your pipelines run? Not on a schedule — when there’s new data to process. Hugo Lu’s article on State-Aware Orchestration identifies a fundamental issue: scheduled pipelines waste resources when: Data doesn’t land on time No new data has arrived since last run Models are forgotten but keep running Full refreshes happen when incremental would suffice His framing...

How auxiliary tables extend your data model without polluting your core schema ** How auxiliary tables extend your data model without polluting your core schema** In data engineering, we often need to track information about our data that doesn’t belong in the data itself. History of changes. Quality scores. Audit trails. Load statistics. Version information. The naive approach is to add columns directly to the main table: created_at, updated_at, created_by quality_score, last_checked_at version_number, previous_value But...

How sidecar schemas and quality attestations enable data consumers to make informed trust decisions How sidecar schemas and quality attestations enable data consumers to make informed trust decisions Every data pipeline has a dirty secret: quality checks happen, but the results disappear into logs. When data arrives at a downstream system, consumers face a fundamental question: “Can I trust this data?” Today, most data platforms answer this question with silence. The source system ran quality...

How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies How to build data pipelines that are portable, auditable, and deployment-ready without external metadata dependencies In modern data platforms, metadata lives everywhere: ETL tools store pipeline schedules and dependencies Data catalogs store descriptions and classifications Governance platforms store PII tags and ownership Quality tools store validation rules BI tools store metric definitions The result? Metadata sprawl. The same column gets...

BIRD Bench research reveals that well-structured data models achieve 95% SQL generation accuracy — without semantic layer tools BIRD Bench research reveals that well-structured data models achieve 95% SQL generation accuracy — without semantic layer tools MotherDuck’s analysis of the BIRD Benchmark — a rigorous SQL generation test — reveals a counterintuitive truth: “AI-ready data simply means a clean data model. Well-named tables, straightforward joins, and clear column names are enough for frontier LLMs to...

Moving beyond Confluence sprawl to a unified, AI-consumable documentation strategy Moving beyond Confluence sprawl to a unified, AI-consumable documentation strategy Every data team I’ve worked with has the same problem: documentation is everywhere and nowhere. Confluence : Data dictionaries, architecture decisions, meeting notes SharePoint : Business glossaries, compliance documents, data contracts Local drives : ERDs, design documents, “that one Excel with all the mapping rules” Wiki : Runbooks, troubleshooting guides, onboarding docs Code repos :...

The Problem Every few months, someone claims: “AI will generate your entire data platform.” It won’t. Large language models do not understand your architecture. They do not understand lineage. They do not understand semantic stability. They generate text. Text is not structure. And SQL is not just text. AI becomes interesting only when it operates on structured metadata. Once SQL has been parsed into: entity attribute relationship attribute_mapping function_def filter_def entity_window optimizer_diagnostics The system is...

Structural normalization and determinism SQL optimizers inside databases already reorder joins. So why would a metadata-driven framework care? Because there are two very different concerns: Runtime execution optimization (handled by the database engine) Structural normalization and determinism (handled by MDDE) These are not the same problem. MDDE does not attempt to replace the database optimizer. It ensures that the structural representation of joins is: Explicit Deterministic Canonical Traceable Before SQL ever reaches execution. In real...

Why Derived Queries Must Be Modeled — Not Rendered Most SQL complexity does not live in base tables. It lives in: CTEs Subqueries UNION chains Nested derived queries In many data platforms, these constructs are treated as textual blocks. Parsed. Rendered. Forgotten. MDDE treats them differently. Derived queries are modeled as entities. Not as strings. Consider: WITH filtered_orders AS ( SELECT * FROM orders WHERE order_date >= '2024-01-01' ), aggregated_orders AS ( SELECT customer_id, COUNT(*)...

The Architecture Itself Becomes Queryable Most data platforms are built on SQL. MDDE is built on metadata. Not documentation metadata. Not optional annotations. Structured, relational metadata that describes: Entities Attributes Relationships Transformations Lineage Quality rules Optimizer findings Deployment behavior In other words: The architecture itself becomes queryable. This article explains the schema that makes that possible. When SQL is parsed into MDDE, it is not stored as text blobs. It is decomposed into: Entities Attributes...