Data Quality, Tests, and Data Contracts with GenAI

Summary

After building a Generic LDM and mapping AnaCredit, RRE, and BIRD into it, the next challenge is trust.

Can we rely on the mappings? Are relationships consistent? Do sources conform to the expected contracts?

With dbt and GenAI, we can scaffold tests and data contracts that validate business logic and enforce reliability.

Key Takeaways

• Business-friendly mappings must be testable.
• dbt tests check uniqueness, nullability, referential integrity.
• Data contracts define what sources must deliver (shape, quality).
• GenAI helps generate tests and contracts from metadata.
• Quality controls turn models into trustworthy data products.

Why Quality Matters

I’ve learned that no matter how elegant the LDM or mapping sheets are, trust is everything. If a regulator or business user finds inconsistencies, the entire data platform loses credibility.

That’s why I stressed in Data Contracts: Anchoring Reliability in Model-Driven Data Engineering that models and mappings must be backed by enforceable rules.

How Can We Use dbt for Quality?

dbt makes testing lightweight and integrated:

version: 2
models:
  - name: ldm__counterparty
    description: "Business-shaped Counterparty entity"
    columns:
      - name: counterparty_id
        description: "Unique identifier"
        tests:
          - not_null
          - unique
      - name: country
        description: "Country code"
        tests:
          - relationships:
              to: ref('ref__country')
              field: country_code

Here we test:

• Every counterparty must have an ID.
• IDs must be unique.
• Countries must exist in the reference table.

These are data contracts in action: explicit, enforceable, and automated.

Adding Business Rules

We can go beyond technical tests. For example:

• Every mortgage in RRE must have a property valuation within the last 12 months.
• Every counterparty in AnaCredit must have either an LEI or a national ID.

GenAI can help us phrase these as SQL + dbt tests. Example:

-- test__rre_valuation_recent.sql
select *
from {{ ref('ldm__valuation') }}
where valuation_date < current_date - interval '12 months'

If this returns rows, the test fails.

Contracts from Metadata

With AnaCredit, RRE, and BIRD PDFs, GenAI can extract:

• Expected domains (e.g. code lists).
• Mandatory attributes.
• Valid relationships.

From this, we scaffold contracts as YAML + SQL tests.

This closes the gap I described in Metadata as Data: turning documentation into active enforcement.

Why This Fits Model-Driven Data Engineering

• Models define structure (LDM).
• Mappings define meaning (business-friendly sheets).
• Contracts define trust (tests + rules).

Together, they form the foundation of model-driven data engineering.

What’s Next

In the next article, I’ll show how we can publish these models, mappings, and contracts into business-facing knowledge bases like Confluence or Notion — making data models not just technical assets but also collaborative business tools.