Metadata-Driven Data Quality — Validating Data and Metadata with the Same Logic

Bringing semantic consistency to data quality: how historized metadata and Great Expectations can turn your metadata model into an executable DQ framework.

Summary

By embedding data quality and plausibility logic directly into your metadata, you can generate and execute validation queries for both source and integrated datasets — ensuring the same semantic rules are applied across the data lifecycle. With historized metadata and tools like Great Expectations, metadata becomes more than documentation: it becomes the engine of data trust.

The Problem: Divergent Worlds of Data and Quality

In many organizations, data quality (DQ) runs on a parallel track to data modeling. Modelers define entities, mappings, and transformations — while DQ teams define checks, thresholds, and plausibility rules in isolation.

This separation leads to:

• Inconsistent rules between source and integrated layers.
• Duplicated effort — one for ETL validation, one for business DQ monitoring.
• Static DQ frameworks detached from change management and versioning.

Our metadata-driven methodology changes that by treating data quality rules as first-class metadata — derived, versioned, and historized alongside models and mappings.

Historized Metadata as the Foundation

When all mappings, transformations, and attribute semantics are historized in the metadata layer (e.g. in DuckDB or Notion as your control plane), we can derive DQ queries directly from it.

For example:

• Domain constraints (country code ∈ ISO list)
• Completeness rules (non-null for business key attributes)
• Consistency checks (mapping alignment between Counterparty.country and AnaCredit.Residence_Country_Code)
• Plausibility rules (loan_amount > 0, maturity_date > start_date)

These can be expressed declaratively in the metadata — and translated into executable SQL or Great Expectations suites.

From Metadata to DQ Queries

Using our metadata model (with attributes like business_meaning, data_type, nullable, domain_reference, mapping_rule), we can automatically generate:

• SQL queries for completeness, conformity, consistency, and uniqueness
• Cross-layer plausibility checks (source vs. integrated data)
• Temporal plausibility using historized metadata (e.g., attribute definitions valid at runtime date)

This is metadata-driven DQ — not just documentation, but executable semantics.

Integrating Great Expectations

Great Expectations (GX) can serve as the execution engine for this framework:

• Metadata defines expectations (in YAML or generated dynamically)
• GX runs the validations and stores results as validation metadata
• Results feed back into your semantic feedback loop and Metadata Quality Scorecard

For example, a country attribute in metadata can generate:

expect_column_values_to_be_in_set:
column: country
value_set: [“NL”, “DE”, “FR”, “BE”, “ES”]

Or more dynamically, a rule like:

Expect column _maturity_date_ to be after _start_date_ can be derived directly from the mapping metadata or business rule annotations.

Plausibility Across Source and Integrated Data

Once mappings are defined between source and integrated models, the same logic can generate cross-dataset plausibility checks, such as:

• sum(source.amount) ≈ sum(integrated.amount) within tolerance
• record count consistency per business key
• alignment of status or reference values

These checks can be automatically regenerated whenever the metadata changes — making data quality version-controlled, reproducible, and continuously validated.

Bringing It Together: Semantic DQ as Code

This approach enables:

1. Unified rule definition in metadata.
2. Automated generation of DQ queries and Great Expectations suites.
3. Version-controlled quality logic aligned with modeling changes.
4. Historized validation results, traceable over time.
5. Integration into Continuous Metadata Integration pipelines.

Key Takeaways

• Treat DQ and plausibility as metadata-driven logic, not separate processes.
• Use historized metadata to execute rules valid for a given semantic version.
• Tools like Great Expectations can run metadata-derived rules automatically.
• Integrate DQ feedback into your Metadata Intelligence layer to learn over time.
• The goal: trust in data through alignment of models, mappings, and quality.

Part of the “Continuous Metadata Integration” series — exploring how semantic DevOps principles bring agility, consistency, and intelligence to modern data architectures.