Engineering Time in Data Platforms — A 3-Part Series From Snapshots to Deterministic Timelines: Engineering Time in Data Platforms Most data platforms believe they are historized. They are not. They store history — but they do not engineer time. In this three-part MDDE deep dive, we will move from: Basic SCD2 storage to bi-temporal standardization (this article) Timeline compression and deterministic integration (Part 2) Valuation reconstruction with impact-driven refresh (Part 3) This first article lays...

Series Context Merging Multiple SCD2 Timelines Without Explosion: ASOF Sampling, Gaps-and-Islands, and PIT Tables In Part 1, we standardized heterogeneous sources into controlled bi-temporal SCD2. Now comes the hard part. Once everything is SCD2… Joining them becomes dangerous. Each SCD2 table has its own change points. Consider a simple 360-degree customer view requiring three SCD2 sources: Combined timeline: 9 change points from just 3 sources with 3 changes each. Each change fragments the timeline. Add...

How metadata-as-code principles extend beyond model generation to manage safe, automated, and traceable schema evolution in production How metadata-as-code principles extend beyond model generation to manage safe, automated, and traceable schema evolution in production In my previous article, we explored how conceptual and logical metadata can automatically generate physical data models — turning business meaning into structure and SQL. After publication, Kristian Poulsen asked a question that perfectly captures the next step in this journey:...

How to keep bi-temporal historization reproducible when the source’s primary key is conditional, incomplete, or context-dependent. 🧩 Part of the Model-Driven Data Quality series — exploring reproducible, metadata-driven patterns for historization, deduplication, and bi-temporal data management. In my previous article, Historizing Source Tables Without a Primary Key, I showed how we can build a reproducible, bi-temporal SCD2 pattern using technical hash keys instead of relying on source-defined primary keys. But real-world data rarely fits neatly into theory....

Reproducible Bi-Temporal SCD2 Using _hash(business_columns)_ + _row_number()_ per Hash 🧩 Part of the Model-Driven Data Quality series — exploring reproducible, metadata-driven patterns for historization, deduplication, and bi-temporal data management. In a metadata-driven data platform, historization is the foundation of reproducibility and auditability. Our bi-temporal SCD2 design records both valid-time (business reality) and system-time (warehouse load time), allowing us to reconstruct any dataset as it was known at any point in time. Most implementations assume that the source defines a primary key — but in reality, many source systems don’t. They simply deliver...

We use SCD2 to historize data based on primary keys. Some source datasets contain duplicate PKs that cannot be fixed at source in the short term. We must: Keep all records for full reproducibility, Ensure downstream pipelines only process one valid record per PK. Databricks does not enforce PK constraints at the table level. Ingest all records – no data is dropped. Flag duplicates – add dq_status (OK / PK_DUP) and metadata (dq_rank, survivor_flag) to...

Structure versioning deals with the shape of the data: tables, columns, relationships, and schemas. Content versioning deals with the meaning of the data: business rules, filters, derivations, and how values are interpreted. Both are equally important: Structure changes affect how systems read/write the data. Content changes affect how the same data values should be understood. The platform’s physical core (historization + integration layers) is a superset of all structures ever modeled. Columns and tables are...

A practical approach to consistent, testable data pipelines. Use source subtypes as translation blocks to integrate data into a clean, business-aligned model without duplication. Press enter or click to view image in full size One query pulls from raw source tables. Another joins a derived table directly. Yet another mixes historized and current records inconsistently. The result? Fragile, irreproducible transformations. We solve this with Access Views (AVs). Every downstream transformation reads only from AVs, never directly from...