How sidecar schemas and quality attestations enable data consumers to make informed trust decisions
Quality Metadata Exchange: Trust Through Transparency
How sidecar schemas and quality attestations enable data consumers to make informed trust decisions
The Trust Gap in Data Pipelines
Every data pipeline has a dirty secret: quality checks happen, but the results disappear into logs. When data arrives at a downstream system, consumers face a fundamental question: “Can I trust this data?”
Today, most data platforms answer this question with silence. The source system ran quality checks — maybe. They passed — probably. The data is accurate — hopefully.
This trust gap creates real problems:
- Redundant validation : Consumers re-run the same quality checks the producer already performed
- Blind consumption : Teams use data without knowing its quality status
- Broken accountability : When quality issues surface, no one knows where the problem originated
- Version mismatches : Producers and consumers operate on different quality expectations
What if quality metadata traveled with the data itself?
Introducing Quality Metadata Exchange
Quality Metadata Exchange (QMEF) is a pattern for sharing quality check results between data producers and consumers. Instead of discarding quality results after pipeline execution, QMEF captures them in a standardized format that:
- Travels with data via sidecar tables
- Proves quality through formal attestations
- Enables trust decisions based on evidence, not assumptions
- Reduces duplication by sharing upstream check results
The Core Concept: Sidecar Schema
A sidecar schema is a companion table that stores quality metadata alongside your data table. Rather than polluting your data with quality columns or losing quality context entirely, the sidecar provides a clean separation:
The sidecar links to data via batch_id, enabling queries like:
Quality Attestations: Formal Proof of Quality
A Quality Attestation is a formal statement from a data producer certifying that quality checks were performed on a data batch. Think of it as a “certificate of quality” that accompanies your data.
Anatomy of an Attestation
{
"qualityAttestation": {
"version": "1.0",
"attestationId": "QA_20260228_143022_abc123",
"entityId": "fct_transactions",
"batchId": "BATCH_2026022814",
"producer": {
"system": "finance-data-platform",
"version": "3.2.1",
"environment": "PROD"
},
"scope": {
"type": "incremental",
"rowCount": 145230
},
"summary": {
"status": "PASSED",
"overallScore": 0.9847,
"dimensions": {
"completeness": 0.9912,
"uniqueness": 1.0000,
"validity": 0.9756
},
"checkCounts": {
"total": 12,
"passed": 11,
"warning": 1,
"failed": 0
}
},
"reconciliation": {
"status": "PASSED",
"sourceRowCount": 145230,
"targetRowCount": 145230,
"withinTolerance": true
}
}
}
What Attestations Contain
The Eight Quality Dimensions
Quality attestations score data across ISO 8000-aligned dimensions:
Each dimension receives a score (0.0–1.0), enabling nuanced trust decisions:
# Consumer can define their own thresholds
if attestation.dimension_scores["completeness"] < 0.99:
logger.warning("Low completeness - some records may have missing values")
if attestation.dimension_scores["validity"] < 0.95:
raise QualityException("Validity too low for production use")
Reconciliation: Proving Data Matches
When data flows between systems, how do you prove the source and target match? Reconciliation attestations provide that proof:
{
"reconciliation": {
"reconciliationId": "RECON_20260228_143030",
"sourceSystem": "core-banking",
"targetSystem": "data-warehouse",
"status": "PASSED",
"sourceRowCount": 145230,
"targetRowCount": 145230,
"withinTolerance": true,
"valueChecks": [
{
"metric": "total_amount",
"aggregation": "SUM",
"sourceValue": 45678901.23,
"targetValue": 45678901.23,
"differencePercent": 0.0,
"status": "passed"
}
]
}
}
Common Reconciliation Checks
Contract-Based Quality Exchange
Quality attestations become truly powerful when integrated with data contracts. A contract can specify what attestations a producer must provide:
contract:
entity: fct_transactions
version: "2.1.0"
owner: finance-data-team
quality_exchange:
# What attestations producer must provide
required_attestations:
- dimension: completeness
scope: full
min_score: 0.99
- dimension: validity
scope: full
min_score: 0.98
# Reconciliation requirements
reconciliation:
required: true
checks:
- type: row_count_match
tolerance_percent: 0.1
- type: sum_match
column: amount
tolerance_percent: 0.01
# How quality metadata is delivered
delivery:
format: sidecar
sidecar_table: "{entity}_quality_sidecar"
consumers:
- team: risk-analytics
criticality: critical
quality_requirements:
min_overall_score: 0.95
required_dimensions: [completeness, validity]
Validating Attestations Against Contracts
from mdde.quality.exchange import QualityExchangeManager
manager = QualityExchangeManager(conn)
# Validate that attestation meets contract requirements
validation = manager.validate_attestation(attestation, contract)
if not validation.valid:
print(f"Missing checks: {validation.missing_checks}")
print(f"Insufficient scores: {validation.insufficient_scores}")
for issue in validation.issues:
print(f"- {issue.message}")
Implementation Patterns
Pattern 1: Producer Creates Attestation
After running quality checks, the producer creates an attestation:
from mdde.quality.exchange import QualityExchangeManager, AttestationScope
manager = QualityExchangeManager(conn)
# Create attestation from quality report
attestation = manager.create_attestation(
entity_id="fct_transactions",
batch_id="BATCH_20260228",
quality_report=quality_report,
reconciliation=reconciliation_result,
producer_version="2.1.0",
environment="PROD",
scope=AttestationScope.INCREMENTAL,
row_count=145230,
)
# Write to sidecar table
manager.write_sidecar(attestation, schema="warehouse")
# Export for external consumers
json_export = manager.export_attestation(attestation, format="json")
Pattern 2: Consumer Reads Attestation
Before using data, the consumer checks the attestation:
# Read latest attestation
attestations = manager.read_sidecar(
entity_id="fct_transactions",
schema="warehouse",
since=datetime(2026, 2, 1),
)
latest = attestations[0]
# Make trust decision
if latest.overall_status == AttestationStatus.FAILED:
raise DataQualityError("Cannot proceed - quality checks failed")
if latest.overall_score < 0.95:
logger.warning(f"Quality score {latest.overall_score:.2%} below threshold")
Pattern 3: Quality-Filtered Queries
Build quality awareness into your SQL:
-- View: Only high-quality transactions
CREATE VIEW v_trusted_transactions AS
SELECT t.*
FROM fct_transactions t
INNER JOIN fct_transactions_quality_sidecar q
ON t.batch_id = q.batch_id
WHERE q.overall_status = 'PASSED'
AND q.completeness_score >= 0.99
AND q.reconciled_with_source = TRUE;
Exchange Formats
Quality attestations can be exchanged in multiple formats:
JSON (Machine-Readable)
Best for API integrations and programmatic access:
{
"qualityAttestation": {
"attestationId": "QA_20260228_143022_abc123",
"summary": {
"status": "PASSED",
"overallScore": 0.9847
}
}
}
YAML (Human-Readable)
Best for configuration and documentation:
qualityAttestation:
attestationId: QA_20260228_143022_abc123
summary:
status: PASSED
overallScore: 0.9847
Sidecar Tables (Database-Native)
Best for SQL-based consumers and dashboard integration:
SELECT
batch_id,
overall_status,
overall_score,
completeness_score,
validity_score
FROM fct_transactions_quality_sidecar
WHERE load_timestamp >= CURRENT_DATE - INTERVAL '7 days'
ORDER BY load_timestamp DESC;
Benefits of Quality Metadata Exchange
For Producers
- Accountability : Prove that quality checks were performed
- Transparency : Share quality status with consumers
- Reduced support : Consumers can self-diagnose quality issues
For Consumers
- Informed decisions : Trust data based on evidence
- Reduced duplication : Skip redundant quality checks
- Quality filtering : Only use data that meets your standards
For the Organization
- Data lineage : Track quality across the pipeline
- Audit trail : Demonstrate compliance and governance
- Cost savings : Eliminate redundant validation work
Getting Started
Step 1: Generate Sidecar Schema
from mdde.quality.exchange import QualityExchangeManager
manager = QualityExchangeManager(conn)
# Generate DDL for your entity
ddl = manager.generate_sidecar_ddl(
schema="warehouse",
entity_name="fct_transactions",
dialect="snowflake",
)
# Execute the DDL
conn.execute(ddl)
Step 2: Create Attestations in Your Pipeline
# After your quality checks complete
attestation = manager.create_attestation(
entity_id="fct_transactions",
batch_id=current_batch_id,
quality_report=quality_engine.execute("fct_transactions"),
)
manager.write_sidecar(attestation, schema="warehouse")
Step 3: Validate Against Contracts
# Ensure attestations meet contract requirements
validation = manager.validate_attestation(attestation, contract)
if not validation.valid:
raise ContractViolation(validation.issues)
Conclusion
Quality Metadata Exchange transforms quality from an invisible pipeline step into a shareable, verifiable asset. By adopting sidecar schemas and quality attestations, organizations can:
- Build trust between data producers and consumers
- Eliminate redundant quality checks
- Enable evidence-based data consumption decisions
- Create accountability across the data supply chain
The question is no longer “Can I trust this data?” but rather “What does the attestation tell me about this data?”
Quality Metadata Exchange — Because trust should be proven, not assumed.