Summary
In previous articles, we explored the foundation of business-friendly mapping, the role of logical source models, the treatment of metadata as data, and how Source Access Views (SAVs) bridge physical and logical worlds.
Now we turn to a critical aspect of scalable data platforms: historization.
We know that dual SCD2 historization (bitemporal) gives us a true picture of changes over time. But historizing everything 1:1 as delivered by the source can be expensive, inefficient, and hard to work with.
Logical source models give us the key to optimizing historization.
The Problem with Raw Historization
When we historize source data directly in the form we receive it, we often end up with:
- Huge tables that store every attribute, even when most rarely change.
- Storage inefficiency: small updates (like a balance amount) trigger new rows with all attributes repeated.
- Complex queries: consumers must filter out technical metadata to find relevant changes.
This is correct, but far from optimal.
How Logical Source Models Help
Logical source models allow us to:
- Separate frequently and slowly changing attributes.
- Group attributes into stable entities (core), semi-stable sets (characteristics), and volatile attributes.
- Normalize denormalized deliveries into manageable structures.
- Identify supertypes and subtypes, which influence historization strategies.
This isn’t just modeling for clarity — it directly impacts storage efficiency and query performance.
Attribute Grouping: An Example
Imagine we receive an Account dataset:
Physical delivery (denormalized OBT)
- Account_ID
- Customer_Name
- Customer_Address
- Customer_Employment_Status
- Account_Balance
- Credit_Limit
- Last_Update_Date
- Ingest_TS
If we historize this as-is, every small balance change re-creates the whole row.
Logical source model separation:
- Account (core entity): Account_ID, Credit_Limit
- Customer (stable attributes): Name, Address
- Employment Status (semi-stable)
- Balance (volatile)
Now, historization happens per entity:
- Address changes → historize only the Address entity.
- Balance changes → historize only the Balance entity.
- Credit Limit changes → historize only the Account entity.
This massively reduces data churn and storage.
Historization Strategies Guided by Models
From logical source models, we can drive:
- Optimized table structures: splitting large OBT tables into normalized historized tables.
- Subtype historization: historizing Persons and Organizations separately, even if delivered together.
- Reference-driven historization: linking changing codes to reference data models for consistency.
- Automated evolution: generating SQL that adjusts historization structures when the logical model changes.
The result is historization that is still complete (dual SCD2), but much more efficient.
The Role of Metadata
Because logical source models are exported as metadata:
- We can automatically derive historization rules.
- We can detect when new attributes require historization structures.
- We can validate that historized tables align with logical expectations.
- We can historize not only data, but metadata itself — keeping track of how models evolve over time.
This is metadata-driven historization in practice.
Closing Thoughts
Dual SCD2 historization is the foundation for reliable history in data platforms. But applying it blindly to raw physical deliveries leads to inefficiency.
Logical source models give us the intelligence to:
- Split attributes into meaningful groups.
- Normalize where necessary.
- Optimize storage and queries.
- Automate historization structures based on metadata.
The combination of logical modeling + dual SCD2 delivers historization that is both correct and efficient.
In the next article, we’ll bring everything together: showing how business-friendly mapping builds step by step from logical source models, SAVs, and historization into the integration layer.