How Metadata and Bi-Temporal Storage Bridge Pragmatism and Possibility
A metadata-driven perspective on why we historize everything but only model what’s needed — creating a pragmatic bridge between raw data capture and governed integration.
1. Rethinking “All the Data, All the Time”
Remco Broekmans recently wrote a thoughtful piece challenging the mantra “all the data, all the time.” It’s a phrase that’s often repeated with enthusiasm — but less often questioned for its implications.
In practice, the challenge isn’t whether we should store everything, but how to do so meaningfully. What if we could capture all source data without modeling it all upfront, yet still make it available — structured, traceable, and ready for activation when needed?
That’s the balance we aim for in our metadata-driven, bi-temporal data architecture.
2. Historizing Everything — But with Purpose
In our platform, we historize all raw source data using Slowly Changing Dimension Type 2 (SCD2), or what Remco might call an HSA (Historical Staging Area).
This gives us:
- a complete temporal trail of every attribute,
- the ability to reconstruct any point in time, and
- a stable audit baseline for future use cases.
We do this because we can’t always predict what data will matter later.
Regulatory changes, new data products, or evolving business logic often surface data needs that weren’t visible at ingestion time.
So, we store everything bi-temporally — not because all data has equal value, but because we want to preserve future optionality.
But there’s a key distinction we always make:
👉 Historizing is not modeling.
3. Historizing vs. Modeling — A Clear Separation of Concerns
Historization is a technical process: capturing all source data changes accurately and consistently.
Modeling, on the other hand, is semantic and demand-driven: we model data only when there’s a concrete business or analytical requirement.
This separation allows us to move at two different speeds:
- Ingestion speed — capture everything bi-temporally, ensuring no data is lost.
- Modeling speed — integrate only what is needed now, based on use cases and demand.
Between those two worlds lies an elegant bridge: Source Access Views.
4. Source Access Views — Making Raw History Usable
Even though we historize all raw source data, we don’t expose it directly.
Instead, we create Source Access Views (SAVs) — thin, logical layers that make historized data query-friendly and consistent.
This concept is explained in detail in these related articles:
- Source Access Views — Consistent Access to Historized Source Data
- Source Access Views — Bridging Physical and Logical Worlds
SAVs do a few critical things:
- Rename columns to logical business names (e.g.
CustomerIdinstead ofCUST_NO), - Cast data types to consistent formats,
- Hide technical ingestion metadata like
_filenameor_batch_id, - Add business keys that link to entities in the common integration layer, if such logical concepts already exist.
This makes SAVs a small but crucial modeling step — guided by source metadata, not business semantics.
They ensure that historized data remains accessible, comprehensible, and reusable — without compromising the purity of the raw ingestion layer.
Think of them as the “front door” of historized data — stable, clean, and ready for use.
5. Demand-Based Modeling — Only What’s Needed, When It’s Needed
Above the SAVs, we maintain the common integration layer — the governed foundation for reporting and cross-domain data sharing.
Here, we model only what’s required for enterprise truth and formal reporting.
Take the collateral and asset example in a financial institution:
- The integration layer models the common semantics of collateral — identifiers, valuations, links to customers and exposures.
- But source systems may contain dozens of extra attributes for specific collateral types, such as:
- engine type or manufacturer for aircraft,
- energy label or square meters for real estate,
- IMO registration for ships.
These attributes are historized, but not modeled in the common layer.
They are accessible through their Source Access Views and can be linked to the integrated data via shared business keys.
When a data consumer needs those attributes — for instance, an aircraft portfolio valuation — we model them on demand, based on the same metadata-driven principles.
This approach:
- Keeps the integration layer lean and stable,
- Avoids semantic bloat, and
- Preserves the ability to expand when new use cases arise.
6. The Hybrid Exploration Layer
Between raw and integrated data, we maintain what we call the hybrid exploration layer.
It combines the flexibility of the historized data with the structure of the integrated model:
- SAVs expose the historized data logically,
- Business keys link them to integrated entities,
- Analysts can join both worlds dynamically to explore or prototype new datasets.
This means:
- You can extend integrated datasets with not-yet-modeled attributes,
- You can validate business rules before formalizing them,
- You can innovate faster without breaking governance.
And when those experiments mature into governed requirements, they’re simply promoted — metadata already describes how to generate the integration model.
7. Right-Time, Not Real-Time
While our ingestion can operate in near real-time, our principle is right-time: capture everything as soon as it’s available, but model and integrate only when it becomes meaningful.
Freshness alone isn’t value — context is.
8. The Pragmatic Middle Ground
“All the data, all the time” sounds absolute. But in reality, our goal is “All the data, available any time — when it matters.”
We historize everything bi-temporally to preserve optionality.
We model only on demand to preserve clarity.
We use Source Access Views to bridge the two — making historized data logical and business-linked without over-modeling.
That’s how we balance freedom with control — and how metadata-driven architecture turns the ambition of “all the data” into an actionable, maintainable reality.
Related Articles
💡 This article builds upon reflections from Remco Broekmans’ post “All the data all the time…” and connects them with our ongoing series on metadata-driven automation and declarative data architecture.
Explore related pieces: