Data Quality as Metadata — Checks, Profiles, and Contracts Derived From Structure

Treat quality rules as metadata

Data Quality as Metadata — Checks, Profiles, and Contracts Derived From Structure

Treat quality rules as metadata

Most data quality implementations begin with queries.

SELECT COUNT(*) FROM customer WHERE customer_id IS NULL;

Or:

SELECT customer_id, COUNT(*)  
FROM customer  
GROUP BY customer_id  
HAVING COUNT(*) > 1;

Useful, but fragile.

Why?

Because quality logic is:

Buried in SQL files
Scattered across projects
Not centrally visible
Difficult to propagate
Often forgotten during migration

MDDE approaches data quality differently.

It treats quality rules as metadata.
Not as ad-hoc SQL.

Quality Begins in Structure

When metadata defines:

Primary keys
Mandatory attributes
Relationships
Data types
Stereotypes

Quality expectations are already implicit.
MDDE makes them explicit.

Example logical definition:

entity:  
  name: customer  
  attributes:  
    - name: customer_id  
      mandatory: true  
      is_primary_key: true  
    - name: email  
      mandatory: false

From this, quality checks can be generated automatically.

Auto-Generated Null Checks

For mandatory attributes:

SELECT  
    'customer_id' AS column_name,  
    COUNT(*) AS null_count  
FROM customer  
WHERE customer_id IS NULL;

No manual rule definition required.
Change the metadata → regenerate checks.

Duplicate Detection

Primary keys imply uniqueness.

Generated check:

SELECT  
    customer_id,  
    COUNT(*) AS duplicate_count  
FROM customer  
GROUP BY customer_id  
HAVING COUNT(*) > 1;

Again, derived from metadata.
Not handwritten.

Referential Integrity Checks

If relationships are defined:

relationship:  
  parent: customer  
  child: orders  
  parent_key: customer_id  
  child_key: customer_id

Generated check:

SELECT *  
FROM orders o  
LEFT JOIN customer c  
    ON o.customer_id = c.customer_id  
WHERE c.customer_id IS NULL;

Lineage and relationships drive validation.

Range and Domain Checks

If metadata defines:

attribute:  
  name: discount_percentage  
  min_value: 0  
  max_value: 100

Generated rule:

SELECT *  
FROM sales  
WHERE discount_percentage < 0  
   OR discount_percentage > 100;

Structure → quality.

Profiling Queries

Beyond binary validation, MDDE can generate profiling queries:

SELECT  
    COUNT(*) AS row_count,  
    COUNT(DISTINCT customer_id) AS distinct_ids,  
    MIN(customer_id),  
    MAX(customer_id)  
FROM customer;

Profiling becomes standardized.
Not reinvented per project.

Quality as a First-Class Layer

In MDDE, quality rules live next to entities:

entities/customer/code/dq/  
    duplicates.sql  
    nulls.sql  
    referential.sql  
    ranges.sql

And are also reflected in metadata.
This means:

Rules are regenerable
Rules are versioned
Rules are auditable
Rules propagate with structure

Quality is not a separate discipline.
It is derived from modeling.

Data Contracts

Because metadata is structured, contracts can be exported.

For example:

Required attributes
Expected types
Allowed value ranges
Relationship constraints

These can be transformed into:

Open Data Contract Standard (ODCS) exports
JSON schema definitions
API validation layers

The same metadata drives:

DDL
Documentation
Lineage
Quality
Contracts

One structural core.
Multiple projections.

CI Integration

Quality checks can integrate into CI:

python -m mdde.scripts.validate_quality mdl_sales

Or through generated SQL executed in pipeline.

Fail thresholds can be:

Blocking
Warning-based
Scored

Quality becomes:

Measurable
Repeatable
Visible

Not tribal knowledge.

Migration Context

In migration programs, quality metadata is critical.

It enables:

Before/after validation
Drift detection
Regression comparisons
Semantic parity checks

Without re-writing validation logic.
Quality travels with the model.

Why This Matters

In many organizations:

Modeling is separate
Quality is separate
Governance is separate
Documentation is separate

MDDE unifies them.
Because they derive from the same structure.

What This Is Not

This is not a replacement for:

Great Expectations
dbt tests
Enterprise DQ platforms

It complements them.
By generating consistent, structural baselines.

Closing

Quality should not depend on memory.
It should depend on structure.

When metadata defines intent, validation becomes mechanical.
When validation becomes mechanical, confidence increases.

And confident systems evolve faster.

Next in the Series

In the next article:

Automation Scripts — One-Click Pipelines for Model Management

We’ll explore:

Generated reimport/analyze/optimize/export scripts
Cross-platform execution
Deterministic pipelines
Reducing operational friction

Structure modeled.
Structure validated.

Next: structure automated.

🔧 Technical Deep Dive — Generated DQ Execution

Generated checks are registered:

SELECT  
    entity_id,  
    check_type,  
    generated_sql  
FROM metadata.data_quality_check  
WHERE model_id = 'mdl_sales';

CI integration example:

python -m mdde.scripts.run_quality_checks mdl_sales \  
    --fail-on-error

Quality becomes executable metadata.