Jaco van der Laan

From Technical Lineage to Functional Understanding: Why Code-Based Lineage Isn’t Enough

31/10/2025  ·  Data Engineering  ·  Metadata

databricks governance lineage notebook sql
‹ ›

How to move beyond technical extraction and build business-friendly lineage that truly explains data flow

💡 Summary

Modern data catalogs like Databricks Unity Catalog and Microsoft Purview promise automated lineage across your entire data platform. They scan SQL code, notebooks, and ETL pipelines to show how data moves between tables and systems.

But there’s a catch: this lineage is purely technical — extracted from code syntax, not from business intent. It tells you what technically happens, but not why it happens.

To understand real data flow — especially for impact analysis and governance — we need a functional layer: a business-friendly mapping that abstracts away SQL complexity and captures the logic in a readable, declarative form.

That’s where Business-Friendly Mapping in YAML comes in.

🔍 1. What Technical Lineage Really Shows

Tools like Databricks Catalog, Purview, and OpenLineage automatically detect lineage by parsing:

  • SQL queries (SELECT, JOIN, CTE, INSERT INTO)
  • Data pipeline definitions (Airflow, Unity, dbt)
  • Notebook metadata and job runs

The result: a graph of dataset dependencies, typically visualized as arrows between tables.

Useful? Yes — but limited.

Because what you get is:

  • Every CTE, temporary view, and intermediate step
  • Low-level joins and filters
  • System-level references and internal job names

This makes lineage accurate but noisy — like reading the source code of a movie script instead of its story.

⚙️ 2. The Problem: Technical ≠ Functional

Here’s the issue: A business stakeholder doesn’t care that a dataset was joined using LEFT JOIN on key CUSTOMER_ID. They care why it was joined — e.g. “To enrich transactions with customer demographics.”

Technical lineage often looks like this:

CTE1 -> TEMP_VIEW_2 -> ANALYTICS.SALES_ENRICHED

While the functional lineage would read:

Raw Transactions + Customer Profiles → Sales Enriched

The difference? One shows syntax, the other shows semantics.

And only the second form is useful for:

  • Impact analysis (“What happens if we change the definition of a customer?”)
  • Change control (“Which KPIs depend on this attribute?”)
  • Governance (“Who owns the logic that produces this measure?”)

🧩 3. From Code to Function: Business-Friendly Mapping in YAML

Instead of extracting lineage after code is written, we can design it declaratively in YAML — in a way that is both human-readable and machine-executable.

Example:

mapping:
source: customer
target: sales_enriched
rules:
– description: “Add customer demographics to sales”
join: “sales.customer_id = customer.id”
output_fields:
– total_sales
– age_group
– region

From this functional mapping, the SQL can be generated automatically — and lineage can be derived at both:

  • the technical level (SQL dependencies)
  • the functional level (business intent and flow)

This approach is reversible: you can trace both directions — from source tables to business concepts, and back.

🧠 4. Why Functional Lineage Matters for Impact Analysis

Functional lineage helps you answer questions like:

  • “Which KPIs depend on customer_age?”
  • “If we redefine active_customer, which reports are affected?”
  • “Where in the process is data filtered or aggregated?”

Technical lineage can’t easily answer that, because it lacks context. That’s why we need Context-Aware Lineage (as discussed in our previous articles) — lineage that adapts based on transformation logic and data domains.

And even further: Conditional Lineage — lineage that changes depending on conditions, filters, or runtime parameters.

Together, these concepts form the foundation for metadata-driven understanding, rather than just code-level tracing.

🧭 5. The Way Forward

Organizations expecting technical lineage tools to deliver business understanding will be disappointed. They’ll see dependency graphs — but not meaning.

To bridge the gap:

  1. Separate functional and technical layers.
  2. Capture mappings in YAML or metadata-as-code format.
  3. Generate both code and lineage from the same declarative source.
  4. Augment with conditional and context-aware lineage for deeper insight.

This approach transforms lineage from documentation into design — a living artifact that connects business and engineering.

🧾 Key Takeaways

  • Technical lineage tools show what happens, not why it happens
  • Business-friendly mapping in YAML abstracts away SQL noise
  • Functional lineage is essential for impact analysis and governance
  • Context-aware and conditional lineage connect code, logic, and meaning
  • The future of lineage is model-driven, not extracted post-hoc

Related Readings