In this series, we’ve explored how data models, metadata governance, reverse engineering, source access views, and optimized historization set the stage for business-friendly mapping.
Summary
Now it’s time to show how it all comes together: the mapping process itself.
This is where messy source data is transformed, step by step, into a clean, business-aligned integration layer.
The Goal of Business-Friendly Mapping
The goal is simple:
- Allow business users and data stewards to define mappings in business terms.
- Store these mappings in metadata.
- Let automation generate the SQL code for ETL/ELT pipelines.
This ensures mappings are transparent, reusable, and governed — instead of hidden in code.
Step 1: Start with Logical Source Models
The mapping journey begins with logical source models:
- Cleaned-up versions of source deliveries.
- Only relevant attributes modeled.
- Business-friendly names applied.
- Subtypes and supertypes made explicit.
These logical models serve as the contract for how source data is understood in our platform.
Step 2: Source Access Views
From logical source models, we generate Source Access Views (SAVs):
- Business-friendly views over historized source data.
- Technical metadata stripped away.
- Derived attributes and filters applied.
- Reference codes mapped where available.
SAVs expose the source in a way that is ready for mapping.
Step 3: Metadata-Driven Mapping
Mappings are defined in metadata, not in hand-written SQL.
This metadata includes:
- Source attribute → business attribute mappings.
- Transformation functions (from a function library).
- Filters (logical rules for subsets).
- Reference lookups (code translations).
For example:
All stored as metadata, versioned, historized, validated.
Step 4: Function & Filter Libraries
To make mappings reusable, we maintain libraries:
- Functions: logical descriptions of transformations (e.g. Calculate Age, Normalize Phone Number).
- Filters: logical rules (e.g. Only Active Customers).
- Source-specific vs shared:
- Shared functions apply across sources.
- Source-specific functions are tied to one domain.
Data engineers implement these functions as SQL macros (e.g. in DBT). Automation then applies them consistently.
Step 5: Integration Layer
The integration layer is based on the business logical data model.
Mappings define how source attributes flow into business entities:
- Party, Person, Organization, Account, Product, etc.
- Relationships tracked via historization.
- Reference codes unified.
Because everything is metadata-driven, the integration layer is transparent, consistent, and maintainable.
Why This Works
This approach works because each step has its own focus:
- Physical source models → ingestion and historization.
- Logical source models → business-friendly source understanding.
- Source Access Views → clean presentation of historized data.
- Mappings → metadata that aligns source to business.
- Integration layer → stable business data model serving all consumers.
Separation of concerns keeps the architecture scalable, and metadata governance keeps it controlled.
Closing Thoughts
Business-friendly mapping isn’t just a buzzword. It’s a practical methodology that:
- Starts with high-quality logical models.
- Uses metadata as the engine of automation.
- Bridges raw source data and business integration.
- Ensures changes are transparent and impact can be managed.
With this approach, data engineers stop hardcoding transformations, and business users gain clarity on how data flows.
The result is a platform that is both technically efficient and business-aligned.