Engineering Time in Data Platforms — A 3-Part Series From Snapshots to Deterministic Timelines: Engineering Time in Data Platforms Most data platforms believe they are historized. They are not. They store history — but they do not engineer time. In this three-part MDDE deep dive, we will move from: Basic SCD2 storage to bi-temporal standardization (this article) Timeline compression and deterministic integration (Part 2) Valuation reconstruction with impact-driven refresh (Part 3) This first article lays...

The Problem Most data platforms have architectural principles. No SELECT * Deterministic joins Naming conventions Explicit keys Reusable logic No implicit cartesian joins Stable projection ordering They live in slides. Or wiki pages. Or Confluence. But they do not live in the system. And when architecture is not executable, it becomes advisory. Advisory architecture erodes under delivery pressure. Metadata-Driven Data Engineering treats architecture as queryable structure. Once SQL is parsed into metadata: Entities exist Attributes...

The Problem Every few months, someone claims: “AI will generate your entire data platform.” It won’t. Large language models do not understand your architecture. They do not understand lineage. They do not understand semantic stability. They generate text. Text is not structure. And SQL is not just text. AI becomes interesting only when it operates on structured metadata. Once SQL has been parsed into: entity attribute relationship attribute_mapping function_def filter_def entity_window optimizer_diagnostics The system is...

Structural normalization and determinism SQL optimizers inside databases already reorder joins. So why would a metadata-driven framework care? Because there are two very different concerns: Runtime execution optimization (handled by the database engine) Structural normalization and determinism (handled by MDDE) These are not the same problem. MDDE does not attempt to replace the database optimizer. It ensures that the structural representation of joins is: Explicit Deterministic Canonical Traceable Before SQL ever reaches execution. In real...

Why Derived Queries Must Be Modeled — Not Rendered Most SQL complexity does not live in base tables. It lives in: CTEs Subqueries UNION chains Nested derived queries In many data platforms, these constructs are treated as textual blocks. Parsed. Rendered. Forgotten. MDDE treats them differently. Derived queries are modeled as entities. Not as strings. Consider: WITH filtered_orders AS ( SELECT * FROM orders WHERE order_date >= '2024-01-01' ), aggregated_orders AS ( SELECT customer_id, COUNT(*)...

The Architecture Itself Becomes Queryable Most data platforms are built on SQL. MDDE is built on metadata. Not documentation metadata. Not optional annotations. Structured, relational metadata that describes: Entities Attributes Relationships Transformations Lineage Quality rules Optimizer findings Deployment behavior In other words: The architecture itself becomes queryable. This article explains the schema that makes that possible. When SQL is parsed into MDDE, it is not stored as text blobs. It is decomposed into: Entities Attributes...

Why Your Queries May Return Different Results Every Time — And Why It Matters Recently, a data team spent several days investigating what appeared to be a serious defect. Their validation tests kept failing. The same query produced different results in different runs. After careful analysis, they uncovered the real root cause: SELECT customer_id, region, ROW_NUMBER() OVER (PARTITION BY region) AS rn FROM customers The migration was correct. The SQL was broken. Without an ORDER...

MDDE is not just a metadata repository Most articles talk about modeling. Few talk about engines. MDDE is not just a metadata repository. It is an engine composed of: Parser layer Metadata writer Optimizer pipeline Proposal system SQL generator Diagnostics framework Let’s look inside. Each optimizer pass follows the same contract: def run(model_id: str, db: Connection) -> OptimizationReport: ... Each pass: Scans metadata Detects patterns Emits diagnostics Optionally produces proposals Never mutates blindly Diagnostics are...

Every Model Workspace Generates Automation Scripts Most frameworks fail not because they lack features. They fail because they add friction. If engineers must remember five CLI commands, configure paths manually, or guess the correct sequence of steps — adoption slows down. MDDE was built under migration pressure. Which means: Workflows had to be repeatable. Deterministic. Low-friction. And safe. That is why every model workspace generates automation scripts. In a typical metadata-driven workflow, you need to:...

Metadata-First DDL Generation. Most data platforms are not single-dialect anymore. In modern environments, you often see: Snowflake for analytics Databricks for processing BigQuery for BI DuckDB for local experimentation PostgreSQL for operational workloads And sometimes all of them in the same organization. The question becomes: How do you keep structural consistency across platforms without rewriting everything? MDDE approaches this through metadata-first DDL generation. Consider a simple table: CREATE TABLE customer ( customer_id VARCHAR(50) NOT NULL,...