Jaco van der Laan

Scaling Metadata-Driven Modeling Across Teams

24/10/2025  ·  Data Modeling  ·  Metadata

‹ ›

How to balance autonomy and consistency in a federated data architecture

Summary

Over the previous ten articles, we’ve built a complete metadata-driven modeling framework — from conceptual models to physical automation, complete with lineage, quality, and maturity metrics.
But in most real organizations, data modeling isn’t the work of a single architect. It’s distributed across teams, domains, and departments.

The challenge now is:

How can we scale metadata-driven modeling without losing coherence?

This article explores practical patterns for federating metadata governance — enabling teams to move independently while staying aligned through shared schemas, conventions, and metrics.

1. The Need for Federated Metadata Governance

Centralized modeling doesn’t scale. When every change must flow through one architecture team, bottlenecks form, and local context is lost.
Yet full decentralization leads to chaos: duplicated entities, diverging naming conventions, inconsistent lineage.

The solution is federation — allowing teams to own their local models while adhering to shared metadata standards.

In other words:

  • Local ownership of conceptual and logical models.
  • Central coordination of modeling templates, YAML schemas, and validation.
  • Automated alignment through shared tools and metrics.

2. The Federated Metadata Pattern

A scalable setup typically includes three layers:

global/
├── schema/ # Shared metadata schema definitions
├── validation/ # Global validation rules
├── templates/ # Standard YAML patterns for entities, attributes
domains/
├── customer/
│ ├── conceptual/
│ ├── logical/
│ └── physical/
├── product/
├── finance/
└── operations/

Each domain models its business concepts independently but uses the same metadata schema and naming conventions. Global scripts validate, aggregate, and publish the models — ensuring everyone stays consistent.

3. Shared YAML Schema for Consistency

All teams share a single schema definition for models, for example:

global/schema/model_schema.yaml

required:
– layer
– entity
properties:
layer:
enum: [conceptual, logical, physical]
entity:
pattern: “^[A-Z][A-Za-z0-9_]*$”
attributes:
items:
properties:
name: { type: string }
datatype: { type: string }
derived_from: { type: string }

Every domain validates its YAMLs against this schema using a lightweight Python CLI or CI workflow.

4. Domain Autonomy Through Local Metadata Repos

Each team manages its own metadata folder or Git repository. Their workflow remains independent:

  • Define conceptual models in YAML.
  • Generate logical and physical models automatically.
  • Publish to Notion or Confluence.
  • Sync metadata to DuckDB for analysis.

A nightly or weekly pipeline aggregates all domains into a central metadata catalog, combining lineage, quality rules, and metrics.

5. Automated Global Validation

Global scripts enforce consistency across domains:

import yaml, jsonschema, pathlib

SCHEMA = yaml.safe_load(open(“global/schema/model_schema.yaml”))
def validate_all_domains():
for domain in pathlib.Path(“domains”).iterdir():
for file in domain.rglob(“*.yaml”):
data = yaml.safe_load(open(file))
jsonschema.validate(instance=data, schema=SCHEMA)
print(“✅ All domains validated successfully”)
if name == “main”:
validate_all_domains()

This ensures uniformity without requiring central approval for every model change.

6. Aggregating Metadata into a Central Catalog

Once validated, metadata from all domains is loaded into a shared DuckDB database:

CREATE TABLE IF NOT EXISTS metadata.all_entities AS
SELECT domain, layer, entity, derived_from
FROM read_parquet(‘domains/*/export/entities.parquet’);

This enables cross-domain analysis and lineage queries:

  • Which conceptual entities are reused across domains?
  • How many logical models link to shared reference data?
  • Where do business rules overlap or diverge?

You can even compute reuse metrics to measure collaboration effectiveness.

7. Shared Metrics and Maturity Dashboards

Each domain contributes its own metrics snapshot:

Aggregated dashboards highlight both local excellence and global consistency — making model health a visible, measurable outcome across the organization.

8. Enabling Collaboration Through Shared Metadata Services

At scale, metadata becomes a shared service layer:

  • Central DuckDB or Postgres for metadata cataloging
  • APIs exposing entities, lineage, and metrics
  • Notion or Confluence dashboards for shared visibility
  • CLI tools for model generation and validation

This creates a self-service metadata ecosystem: each team contributes models, and the system maintains coherence automatically.

9. Lessons Learned from Large-Scale Deployments

When metadata-driven modeling scales across teams:

  • Governance becomes embedded in the process, not imposed afterward.
  • Domains evolve independently but stay interoperable.
  • Metrics and lineage replace manual oversight.

The architecture becomes model-driven, measurable, and federated — not centralized or chaotic, but balanced.

10. In Summary

Scaling metadata-driven modeling means shifting from control to coordination. It’s about giving teams autonomy within alignment — shared semantics, shared metrics, and a shared understanding of what “good” modeling looks like.

This is the true maturity stage of metadata-driven data architecture:

  • Local ownership
  • Global visibility
  • Continuous measurement

11. Next in the Series

“Metadata Mesh: Designing the Federated Architecture for Enterprise-Wide Modeling.”
We’ll explore how to design a metadata mesh where domains exchange models through APIs — a true data mesh foundation powered by metadata.