How to quantify model quality, completeness, and evolution using metadata-as-data
After nine articles on metadata-driven modeling, we’ve built a full framework:
- Conceptual models capture business understanding.
- Logical models define structure and lineage.
- Physical models generate executable code.
- DuckDB catalogs, lineage graphs, and data quality rules keep everything observable.
Now comes the final question: how do we measure whether our data models — and our modeling process — are improving?
This is where metadata metrics come in.
1. Why Measure Metadata Quality?
Every data platform measures data quality.
But how often do we measure model quality?
Your data models themselves are living assets.
They evolve, grow, and — if unmanaged — degrade over time.
By monitoring metadata metrics, you can answer:
- “Are our models becoming more complete?”
- “Is lineage coverage improving?”
- “Which domains are growing most rapidly?”
- “Are quality rules expanding alongside entities?”
Metadata metrics turn data modeling into a measurable practice — one that improves continuously.
2. Categories of Metadata Metrics
We can group metadata metrics into five key categories:
These metrics are all derivable directly from the YAML-based metadata we’ve built.
3. Storing Metrics in DuckDB
Let’s define a table to store metrics snapshots:
CREATE TABLE IF NOT EXISTS metadata.metrics (
measured_at TIMESTAMP,
completeness REAL,
consistency REAL,
lineage_coverage REAL,
quality_coverage REAL,
total_entities INTEGER,
new_entities INTEGER
);
You can then calculate and insert metrics daily, weekly, or per CI run.
4. Calculating Metrics with SQL
Here’s a simple DuckDB script that generates the core metrics.
– Completeness: attributes with datatype and derived_from
WITH completeness AS (
SELECT COUNT() FILTER (WHERE datatype IS NOT NULL AND derived_from != ‘’) * 1.0 / COUNT() AS ratio
FROM metadata.attributes
),
– Consistency: logical entities that link to conceptual
consistency AS (
SELECT COUNT() FILTER (WHERE derived_from LIKE ‘conceptual.%’) * 1.0 /
COUNT() FILTER (WHERE layer = ‘logical’) AS ratio
FROM metadata.entities
),
– Lineage coverage: entities with lineage links
lineage AS (
SELECT COUNT() FILTER (WHERE derived_from != ‘’) * 1.0 / COUNT() AS ratio
FROM metadata.entities
),
– Quality coverage: attributes with rules
quality AS (
SELECT COUNT(DISTINCT a.name) FILTER (WHERE r.name IS NOT NULL) * 1.0 /
COUNT(DISTINCT a.name) AS ratio
FROM metadata.attributes a
LEFT JOIN metadata.quality_rules r ON a.entity = r.entity AND a.name = r.attribute
)
INSERT INTO metadata.metrics
SELECT
CURRENT_TIMESTAMP,
completeness.ratio,
consistency.ratio,
lineage.ratio,
quality.ratio,
(SELECT COUNT(*) FROM metadata.entities),
0 – new entities (calculated via diffs later)
FROM completeness, consistency, lineage, quality;
Run this automatically after every model update — and you’ll have a continuous record of model maturity.
5. Visualizing Model Health
You can now visualize trends like:
- Completeness Over Time
- Lineage Coverage Growth
- Model Expansion (Entity Count)
- Quality Rule Adoption
Example DuckDB query for a time series:
SELECT
measured_at,
completeness * 100 AS completeness_pct,
lineage_coverage * 100 AS lineage_pct,
quality_coverage * 100 AS quality_pct
FROM metadata.metrics
ORDER BY measured_at;
These metrics can feed dashboards in:
📊 Metabase
📈 Grafana
🧭 Notion / Confluence embeds
You’re no longer just visualizing data lineage — you’re visualizing metadata evolution.
6. Building Maturity Scores
You can combine the metrics into a Model Health Index (MHI):
SELECT
measured_at,
ROUND((
completeness +
consistency +
lineage_coverage +
quality_coverage
) / 4 * 100, 2) AS model_health_index
FROM metadata.metrics
ORDER BY measured_at;
A rising MHI indicates increasing model clarity, coverage, and quality — while sudden drops may signal incomplete migrations or missing lineage links.
7. Metadata KPIs for Governance Dashboards
Here are practical KPIs you can include in governance dashboards:
These KPIs quantify modeling discipline — not just activity.
8. Closing the Feedback Loop
When metrics drop, you can trace issues directly to their source:
- Missing lineage → incomplete conceptual mapping
- Low completeness → attributes missing definitions
- Poor quality coverage → unvalidated attributes
This enables continuous metadata improvement — where every commit contributes not only new models, but better ones.
9. From Metadata to Continuous Improvement
The same way DevOps introduced CI/CD for software, metadata metrics enable continuous improvement for modeling.
📈 Track how model health evolves across domains
🧩 Identify where modeling standards need reinforcement
🔁 Close the loop between business modeling and technical implementation
This transforms metadata from passive documentation into an active performance system for your data architecture.
10. In Summary
We’ve now built the complete metadata-driven data modeling lifecycle:
Your data models now form a living system — designed, measured, and improved through metadata.
11. What’s Next
“Scaling Metadata-Driven Modeling Across Teams.”
We’ll explore how to share and federate metadata governance — allowing each domain to evolve independently while staying aligned through shared conventions, schemas, and metrics.