Introduction
In every data platform, the quality of incoming data determines the trustworthiness of the insights that follow. Too often, data quality (DQ) issues are only discovered after data lands in the common platform, forcing back-and-forth discussions with providers and delaying delivery timelines.
What if data providers could validate their datasets before sending them, using the exact same rules the platform applies? This is the idea behind Data Quality as a Service (DQaaS) — a metadata-driven framework that empowers providers to pre-check their data, while ensuring full alignment with central governance.
The Core Idea
Our DQ framework is built on two pillars:
Metadata-driven checks:
- Table and column definitions
- Primary/foreign key relationships
- Mandatory fields
Custom business rules:
- Plausibility checks
- Reference data validations
- Temporal comparisons (e.g., between reporting periods)
Using these ingredients, we generate portable validation packages — notebooks or SQL scripts — that providers can run directly in their own environments. The results are written into a standardized log structure, easily visualized with a Power BI dashboard template.
This approach guarantees that data is validated in the same way on both sides: at the source and in the platform.
How It Works
1. Central Metadata & Rules Management
- The platform team maintains the data model metadata (tables, columns, relationships).
- A central DQ rule library contains standard checks (PK uniqueness, FK consistency) and custom business checks.
- Whenever metadata or rules change, new validation packages are generated.
2. Portable Validation Package
Each provider receives:
- Notebook/SQL scripts with DQ queries.
- Metadata/config file with model details.
- Settings file to adapt schema and database names locally.
The scripts mirror the ingestion layer structure of the platform, ensuring consistency.
3. Execution at Provider Side
- Provider runs the package on their dataset.
- Results are logged into a DQ results table with a standardized schema.
- A Power BI dashboard template allows providers to analyze results immediately.
4. Validation at Platform Side
Once delivered, data is ingested into the common platform. Here, the framework re-runs checks, including:
- Reference data validations using central master data.
- Temporal plausibility checks comparing with historical deliveries.
Results are published in the central Power BI DQ dashboard.
Types of Checks
Structural checks
- PK uniqueness
- Mandatory field completeness
- FK/PK relationship validation
Semantic checks
- Code list validation
- Reference data matching
- Cross-table consistency
Temporal checks
- Volume spikes and drops
- Continuity of entities across reporting periods
- Sudden changes in key attributes
Benefits
- Shift-left quality: Issues are caught before data reaches the platform.
- Consistency: One rule set governs both provider and platform validation.
- Transparency: Providers see the same results and dashboards as platform teams.
- Efficiency: Reduces rework, accelerates onboarding of new data sources.
- Governance: Central control with federated execution.
Future Outlook
This approach opens the door for:
- CI/CD integration — data providers embed validation into their delivery pipelines.
- Automated feedback loops — providers notified of failed checks in real time.
- Advanced checks — anomaly detection and ML-driven plausibility validation.
- Data contracts — formalizing expectations on data quality and structure.
Conclusion
Data Quality as a Service is more than a framework — it’s a shift in mindset. By giving data providers the ability to pre-validate their data using centrally defined rules, organizations can reduce friction, increase trust, and accelerate the journey from raw data to reliable insights.
In a world where data is increasingly federated, portable and metadata-driven DQ frameworks will be the backbone of collaborative data ecosystems._