SQL Ate Your Architecture

The Architecture Absorption Problem

How MDDE Makes Structure Visible Again

Your data platform works.
Queries run. Dashboards refresh. Pipelines complete.

But ask a simple question:

What does this table represent conceptually?
Why is this calculation defined this way?
What breaks if I change this join?

And the answer is often:

“Read the SQL.”

That’s not architecture.
That’s archaeology.

The Architecture Absorption Problem

Early in a project, architecture is visible.

You have:

Source systems
Transformations
Clear data flows
Documented intent

It looks something like this:

Source → Transform → Target

Then scale happens.

More features.
More joins.
More edge cases.
More business rules.
More compliance requirements.

Gradually, structure collapses into a single layer:

SELECT ...  
FROM ...  
JOIN ...  
WHERE ...  
GROUP BY ...

A 500-line SQL file that contains:

Business logic
Structural modeling
Temporal semantics
Data quality rules
Migration workarounds
Performance optimizations

All mixed together.
Architecture didn’t disappear.

It was absorbed into SQL.

Why This Becomes Dangerous

SQL captures execution.
It does not capture intent.

Over time, this leads to:

Invisible semantics
Fear of refactoring
Inconsistent patterns
Duplicated logic
Hard-to-explain lineage

Teams don’t avoid change because they lack skill.
They avoid change because they lack visibility.

After working on sustained migration programs — with regression testing, large-scale SQL analysis, and compliance-driven validation — one thing became obvious:

The real risk is not writing new SQL.
The real risk is not understanding the existing structure.

The MDDE Premise

MDDE (Metadata-Driven Data Engineering) does not replace SQL.
It treats SQL as a runtime artifact — and extracts its structural meaning.

The core idea:

Parse SQL into structured metadata
Surface implicit logical entities
Make relationships and mappings explicit
Enable observation and optimization
Regenerate canonical SQL if desired

SQL remains central.
But architecture becomes visible again.

Take this transformation:

SELECT  
    o.order_id,  
    c.customer_name,  
    SUM(oi.quantity * oi.unit_price) AS total_amount,  
    MAX(oi.line_date) AS last_line_date  
FROM orders o  
JOIN customers c ON o.customer_id = c.customer_id  
JOIN order_items oi ON o.order_id = oi.order_id  
GROUP BY o.order_id, c.customer_name;

This query implicitly defines:

A derived logical entity
Attributes (both direct and computed)
Relationships between source entities
Aggregation semantics
Column-level lineage

But none of that is explicit beyond the SQL text.
MDDE parses this into structured metadata.

Logical Entity

entity:  
  name: order_summary  
  entity_type: query  
  source_entities:  
    - orders  
    - customers  
    - order_items

Attributes

attributes:  
  - name: order_id  
    origin: orders.order_id  
- name: customer_name  
    origin: customers.customer_name  
  - name: total_amount  
    expression: SUM(quantity * unit_price)  
  - name: last_line_date  
    expression: MAX(line_date)

Relationships

relationships:  
  - from: orders.customer_id  
    to: customers.customer_id  
    type: inner_join  
- from: orders.order_id  
    to: order_items.order_id  
    type: inner_join

Mappings (Lineage)

mappings:  
  - target: total_amount  
    sources:  
      - order_items.quantity  
      - order_items.unit_price  
- target: last_line_date  
    sources:  
      - order_items.line_date

Now:

The entity boundary is explicit
Attribute lineage is visible
Relationships are defined
Expressions can be extracted as reusable functions

Architecture was always there.
Now it is inspectable.

Classification Is Optional

Once logical structure is extracted, MDDE can optionally classify entities using stereotypes:

entity:  
  name: order_summary  
  stereotype: fact

Or:

entity:  
  name: customer  
  stereotype: dimension

But this is a second step.

MDDE does not assume dimensional modeling.
It extracts logical structure first — and lets modeling approaches layer on top.

In Practice (VSCode + Claude Code)

The MDDE framework is currently being built in VSCode, with Claude Code assisting in:

Parser refinement
Optimizer pass generation
Metadata validation
Canonical SQL generation

A typical workflow:

python -m mdde.importer.sql_importer path/to/sql --name sales_model  
python -m mdde.scripts.analyze_model mdl_sales --verbose  
python -m mdde.scripts.optimize_model mdl_sales --verbose  
python -m mdde.scripts.export_model mdl_sales \  
    --output workspace/applications/sql/sales_model \  
    --html-documentation

The result is not abstract modeling.

It is structured visibility:

Logical entities
Attributes
Relationships
Functions
Filters
Source-to-target mappings
Lineage
Optimization proposals

SQL remains the execution surface.
Metadata becomes the architectural surface.

Visibility Without Obligation

MDDE separates:

Observation from enforcement
Detection from rewriting
Insight from obligation

You can:

Parse without generating
Analyze without refactoring
Detect patterns without enforcing them

Understanding comes first.
Change remains human-controlled.

This Is Not Model-Driven Perfection

MDDE does not assume:

Systems can be perfectly modeled upfront
Metadata replaces engineering judgment
SQL disappears

It works the other way around:
Implementation → Extraction → Observation → Refinement

The model follows reality.
Not the other way around.

Why This Matters Now

Modern data platforms are:

Multi-team
Multi-dialect
Compliance-sensitive
Migration-heavy
Increasingly AI-assisted

Hidden structure does not scale in this environment.

Explicit structure does.

What Comes Next

In the next article:

The MDDE Loop — Parse, Observe, Optimize, Generate

In the next article The MDDE Loop — Parse, Observe, Optimize, Generate we’ll walk through:

The internal metadata schema
How optimizer passes work
How observations become proposals
How canonical SQL is regenerated
And how the loop remains deterministic and human-controlled

This article is part of the MDDE Framework Deep Dive (2026) series.

🔧 Technical Deep Dive — Inspecting Parsed Structure

Once SQL is imported, structure becomes queryable metadata.

Inspect Extracted Entities

SELECT  
    entity_id,  
    entity_name,  
    entity_type,  
    inheritance_kind  
FROM metadata.entity  
WHERE model_id = 'mdl_sales'  
ORDER BY entity_name;

This shows:

Physical tables
Derived queries
CTE entities
Subqueries

Architecture is no longer hidden inside text.

Inspect Attribute Lineage

SELECT  
    tgt.entity_name   AS target_entity,  
    tgt_attr.attribute_name AS target_attribute,  
    src.entity_name   AS source_entity,  
    src_attr.attribute_name AS source_attribute  
FROM metadata.attribute_mapping m  
JOIN metadata.attribute tgt_attr  
  ON m.target_attribute_id = tgt_attr.attribute_id  
JOIN metadata.entity tgt  
  ON tgt_attr.entity_id = tgt.entity_id  
JOIN metadata.attribute src_attr  
  ON m.source_attribute_id = src_attr.attribute_id  
JOIN metadata.entity src  
  ON src_attr.entity_id = src.entity_id  
WHERE tgt.entity_name = 'order_summary';

This is column-level lineage, materialized.

Not inferred.
Not heuristic.