Jaco van der Laan

Data Quality as Metadata — Checks, Profiles, and Contracts Derived From Structure

16/02/2026  ·  Data Engineering  ·  Metadata

metadata migration quality sql
‹ ›

Treat quality rules as metadata

Data Quality as Metadata — Checks, Profiles, and Contracts Derived From Structure

Treat quality rules as metadata

Most data quality implementations begin with queries.

SELECT COUNT(*) FROM customer WHERE customer_id IS NULL;

Or:

SELECT customer_id, COUNT(*)  
FROM customer  
GROUP BY customer_id  
HAVING COUNT(*) > 1;

Useful, but fragile.

Why?

Because quality logic is:

  • Buried in SQL files
  • Scattered across projects
  • Not centrally visible
  • Difficult to propagate
  • Often forgotten during migration

MDDE approaches data quality differently.

It treats quality rules as metadata.
Not as ad-hoc SQL.

Quality Begins in Structure

When metadata defines:

  • Primary keys
  • Mandatory attributes
  • Relationships
  • Data types
  • Stereotypes

Quality expectations are already implicit.
MDDE makes them explicit.

Example logical definition:

entity:  
  name: customer  
  attributes:  
    - name: customer_id  
      mandatory: true  
      is_primary_key: true  
    - name: email  
      mandatory: false

From this, quality checks can be generated automatically.

Auto-Generated Null Checks

For mandatory attributes:

SELECT  
    'customer_id' AS column_name,  
    COUNT(*) AS null_count  
FROM customer  
WHERE customer_id IS NULL;

No manual rule definition required.
Change the metadata → regenerate checks.

Duplicate Detection

Primary keys imply uniqueness.

Generated check:

SELECT  
    customer_id,  
    COUNT(*) AS duplicate_count  
FROM customer  
GROUP BY customer_id  
HAVING COUNT(*) > 1;

Again, derived from metadata.
Not handwritten.

Referential Integrity Checks

If relationships are defined:

relationship:  
  parent: customer  
  child: orders  
  parent_key: customer_id  
  child_key: customer_id

Generated check:

SELECT *  
FROM orders o  
LEFT JOIN customer c  
    ON o.customer_id = c.customer_id  
WHERE c.customer_id IS NULL;

Lineage and relationships drive validation.

Range and Domain Checks

If metadata defines:

attribute:  
  name: discount_percentage  
  min_value: 0  
  max_value: 100

Generated rule:

SELECT *  
FROM sales  
WHERE discount_percentage < 0  
   OR discount_percentage > 100;

Structure → quality.

Profiling Queries

Beyond binary validation, MDDE can generate profiling queries:

SELECT  
    COUNT(*) AS row_count,  
    COUNT(DISTINCT customer_id) AS distinct_ids,  
    MIN(customer_id),  
    MAX(customer_id)  
FROM customer;

Profiling becomes standardized.
Not reinvented per project.

Quality as a First-Class Layer

In MDDE, quality rules live next to entities:

entities/customer/code/dq/  
    duplicates.sql  
    nulls.sql  
    referential.sql  
    ranges.sql

And are also reflected in metadata.
This means:

  • Rules are regenerable
  • Rules are versioned
  • Rules are auditable
  • Rules propagate with structure

Quality is not a separate discipline.
It is derived from modeling.

Data Contracts

Because metadata is structured, contracts can be exported.

For example:

  • Required attributes
  • Expected types
  • Allowed value ranges
  • Relationship constraints

These can be transformed into:

  • Open Data Contract Standard (ODCS) exports
  • JSON schema definitions
  • API validation layers

The same metadata drives:

  • DDL
  • Documentation
  • Lineage
  • Quality
  • Contracts

One structural core.
Multiple projections.

CI Integration

Quality checks can integrate into CI:

python -m mdde.scripts.validate_quality mdl_sales

Or through generated SQL executed in pipeline.

Fail thresholds can be:

  • Blocking
  • Warning-based
  • Scored

Quality becomes:

  • Measurable
  • Repeatable
  • Visible

Not tribal knowledge.

Migration Context

In migration programs, quality metadata is critical.

It enables:

  • Before/after validation
  • Drift detection
  • Regression comparisons
  • Semantic parity checks

Without re-writing validation logic.
Quality travels with the model.

Why This Matters

In many organizations:

  • Modeling is separate
  • Quality is separate
  • Governance is separate
  • Documentation is separate

MDDE unifies them.
Because they derive from the same structure.

What This Is Not

This is not a replacement for:

  • Great Expectations
  • dbt tests
  • Enterprise DQ platforms

It complements them.
By generating consistent, structural baselines.

Closing

Quality should not depend on memory.
It should depend on structure.

When metadata defines intent, validation becomes mechanical.
When validation becomes mechanical, confidence increases.

And confident systems evolve faster.

Next in the Series

In the next article:

Automation Scripts — One-Click Pipelines for Model Management

We’ll explore:

  • Generated reimport/analyze/optimize/export scripts
  • Cross-platform execution
  • Deterministic pipelines
  • Reducing operational friction

Structure modeled.
Structure validated.

Next: structure automated.

🔧 Technical Deep Dive — Generated DQ Execution

Generated checks are registered:

SELECT  
    entity_id,  
    check_type,  
    generated_sql  
FROM metadata.data_quality_check  
WHERE model_id = 'mdl_sales';

CI integration example:

python -m mdde.scripts.run_quality_checks mdl_sales \  
    --fail-on-error

Quality becomes executable metadata.