How simple control tables evolve into full model-driven data platforms In a recent article by Franco Patano, Databricks showed how easy it is to create a metadata-driven ingestion framework using Workflows and dynamic SQL. Instead of hardcoding pipelines, you define metadata in a control table — describing what data to load, how, and where — and let a simple loop dynamically generate and execute the ingestion SQL. It’s a clean demonstration of what many of us in data engineering...

Bridging Declarative Execution and Metadata-Driven Coordination When Databricks introduced Declarative Pipelines (Lakeflow), it sparked a subtle but important question: If pipelines are declarative, do we still need orchestration? A reader asked me exactly that after my recent article on declarative execution. It’s a great question — because it highlights a common misconception: Declarative pipelines simplify execution, but they don’t eliminate orchestration. They change what orchestration means, and more importantly, where it lives. In the traditional world, orchestration meant control. We...

How Metadata and Bi-Temporal Storage Bridge Pragmatism and Possibility A metadata-driven perspective on why we historize everything but only model what’s needed — creating a pragmatic bridge between raw data capture and governed integration. Remco Broekmans recently wrote a thoughtful piece challenging the mantra “all the data, all the time.” It’s a phrase that’s often repeated with enthusiasm — but less often questioned for its implications. In practice, the challenge isn’t whether we should store everything, but how to do so...

From proprietary semantics to open metadata — understanding Ontology as a paradigm, not a product. When I published Palantir’s Ontology, Kimball’s Star Schema, and Model-Driven Data Engineering, several readers reached out with smart questions: “Is Palantir’s Ontology a product or a modeling paradigm?” “Can you use the Ontology concept outside of Foundry?” “How does your model-driven approach compare to the Ontology itself, not just to Palantir?” Let’s unpack these — because understanding what Ontology really means in Palantir’s...

Bridging Metadata and Execution — How Declarative Pipelines Turn Data Design into Automated Delivery When Databricks announced Declarative Data Pipelines — now even available on the Free Edition — it didn’t just simplify pipeline creation; it represented a convergence between how we design data transformations and how they’re executed. For years, our data architecture has been moving toward metadata-driven automation — where functional logic is described declaratively, and code is generated automatically. Declarative pipelines now allow that philosophy to extend all the way into...

What we’re solving, and exactly how to do it with dual SCD2, PIT/exposure views, and dbt on Databricks When we debug odd results, we don’t just need dataset-level lineage — we need a row-level story: which source snapshots fed the row and which logical steps shaped it (select, derive, filter, aggregate, union, pivot). Our integration layer is already structured around these transformation stages, as outlined in Modeling Data Transformations in Logical Steps. Data Breadcrumbs make that story portable at the row level —...

Enabling business-driven data corrections with Power BI write-back and bi-temporal historization Even in the most automated data platforms, manual corrections and adjustments are inevitable. Source systems may contain errors, late changes, or missing data — and business users often know best what should be corrected. Yet, giving users the ability to correct data directly can easily compromise governance, lineage, and auditability — unless carefully designed. This article outlines a proposed architecture for a governed correction and adjustment module within...

How to Build a Metadata-Driven Translation Layer Between Legacy and Modern Systems The Anti-Corruption Layer (ACL) isn’t just a theory from Domain-Driven Design — it’s a practical architectural component. It’s the firewall of meaning between legacy systems and your modern data platform. If the Common Integration Layer (CIL) defines your organization’s shared vocabulary, then the ACL is the translator that keeps that language uncorrupted during every integration, migration, and synchronization. This article walks through what an ACL...

Clarifying the relationship between reusable domain products and tailored data views As data platforms evolve toward domain-oriented, product-driven architectures, the distinction between Data Products and Consumer Datasets becomes critical. This article explains their relationship, roles, and how metadata-driven management ensures consistency, traceability, and scalability across teams and domains. Modern data platforms are no longer monolithic pipelines feeding endless dashboards. They are ecosystems of reusable, governed Data Products — and fit-for-purpose Consumer Datasets built on top of them. Yet, these two concepts are often...

Over the past months, I’ve created a growing collection of Medium publications that explore the world of data modeling, metadata, architecture, and automation from two complementary perspectives — the technical and the human. This article explains how these publications connect, why the naming evolved, and how they now form a consistent ecosystem for readers and practitioners. Data architecture has many audiences. Some want to understand how models, metadata, and platforms work. Others need to see why these structures matter to business outcomes. When...