CrossModel Meets Model-Driven Data Engineering- A Modern Fit for Business-Friendly Mapping
How CrossModel can enhance metadata-as-data, historization, and business-friendly mapping by integrating with DuckDB, Python, and Notion.
Summary
CrossModel is a collaborative, technology-agnostic data modeling tool that supports conceptual, logical, and physical layers, lineage, and mappings.
It aligns with the principles of model-driven data engineering and business-friendly mapping you’ve been advocating.
By integrating CrossModel with DuckDB (execution metadata), Python (SQL generation), and Notion (business-facing annotations), you can combine strong modeling capabilities with executable, governed pipelines.
Strengths: collaboration, lineage, version control, multi-layer modeling.
Gaps: runtime execution, historization (Dual SCD2), temporal support.
Recommended fit: CrossModel for modeling and lineage, DuckDB/Python for execution and historization, Notion for business reviews.
Introduction
In modern data engineering, we need more than just good models — we need collaboration, governance, temporal support, and integration with execution pipelines. Our methodology emphasizes model-driven engineering, metadata-as-data, historization, and business-friendly mapping.
Enter CrossModel — a new generation data modeling platform. Let’s explore how it fits with your proposed architecture (DuckDB + Python + Notion) and how it enhances your methodology.
What is CrossModel?
CrossModel offers:
- Conceptual, logical, and physical modeling in one tool.
- Data lineage and mapping across model layers.
- Real-time collaboration and Git-based version control.
- Custom translations from model-to-model, supporting schema/code generation.
- Browser + desktop access for broad adoption.
In short: a modeling workbench that bridges business and technical perspectives.
How CrossModel Fits into the Proposed Architecture
1. Business-Oriented Data Modeling
Our architecture starts with business-oriented logical models that flow into source access views, integration models, and exposure views. CrossModel provides a structured way to model conceptual → logical → physical layers and keep them aligned.
2. Business-Friendly Mapping
Mappings are at the core of our methodology. CrossModel supports attribute-level mappings and lineage, which reduces the need for scattered spreadsheets and makes mapping logic transparent. This complements our idea of business-friendly mapping stored as metadata.
3. Metadata-as-Data
While CrossModel is strong for modeling and lineage, DuckDB becomes the metadata execution layer. Exports from CrossModel (models, mappings) can be ingested into DuckDB tables, making them queryable, governable, and ready for integration with Python scripts.
4. Historization & Dual SCD2
CrossModel does not natively provide historization logic. That’s where our DuckDB + Python components come in: the metadata exported from CrossModel serves as input for generating historized queries (Dual SCD2, PIT tables, exposure views). CrossModel provides the modeling and mapping context, DuckDB provides the execution and historization.
5. Execution Integration
CrossModel documents and translates; Python + DuckDB execute. Python scripts can transform CrossModel’s exported mappings into dbt models, Databricks notebooks, or SQL scripts, operationalizing our architecture.
6. Governance & Collaboration
CrossModel brings real-time collaboration and Git-based version control. Combined with Notion for business-facing annotations, approvals, and governance workflows, the result is a closed loop: business definitions ↔ technical metadata ↔ executable pipelines.
Benefits vs. Gaps
Example Flow: CrossModel in Action
- Model conceptual + logical data structures in CrossModel.
- Define mappings from source systems to integration models.
- Export models + mappings from CrossModel.
- Ingest exports into DuckDB metadata tables (
meta.dataset,meta.column,meta.mapping). - Generate SQL with Python for historization (Dual SCD2), PIT tables, and exposure views.
- Review mappings in Notion for business validation and approval.
- Execute pipelines in Databricks/dbt using the generated SQL.