Jaco van der Laan

Metadata Mesh: The Federated Architecture for Enterprise-Wide Data Modeling

28/10/2025  ·  Data Architecture  ·  Metadata

governance lineage metadata
‹ ›

Metadata Mesh The Federated Architecture for Enterprise-Wide Data Modeling

How to scale metadata governance and collaboration using domain nodes and shared APIs

Summary

As organizations adopt [[metadata-driven data modeling]], a new challenge emerges: How do we connect multiple autonomous domains — each modeling its own business — into a coherent, enterprise-wide view without creating a central bottleneck?

The answer is [[Metadata Mesh]] — a [[distributed metadata architecture]] inspired by the same principles as [[Data Mesh]]: [[domain ownership]], [[interoperability]], and [[product thinking]].

1. From Metadata Catalog to Metadata Mesh

A metadata catalog collects models and lineage into one searchable place. A metadata mesh, on the other hand, distributes that catalog across domains — allowing each team to publish and consume metadata as a service.

Think of it like this:

Data Mesh → shares data products
Metadata Mesh → shares model definitions, lineage, and rules

Each domain owns and exposes its metadata. A central coordination layer ensures discoverability and alignment — not control.

2. The Core Principles of a Metadata Mesh

Metadata Mesh builds on four principles, adapted from Data Mesh thinking:

Together, these principles make metadata operational — not just documentation.

3. The Metadata Mesh Architecture

A typical Metadata Mesh architecture looks like this:

                ┌───────────────────────┐  
                │  Metadata Platform    │  
                │  (Shared Services)    │  
                │  - Schema registry    │  
                │  - Validation API     │  
                │  - Lineage graph      │  
                └─────────┬─────────────┘  
                          │  
 ┌──────────────┬─────────┴──────────────┬──────────────┐  
 │              │                        │              │

┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│Customer │ │Product │ │Finance │ │Ops/HR │
│Domain │ │Domain │ │Domain │ │Domain │
└─────────┘ └─────────┘ └─────────┘ └─────────┘
│ │ │ │
└────── publish/consume metadata via APIs ─────────────┘

Each domain is both a producer and a consumer of metadata.

4. Defining Metadata APIs per Domain

Each domain exposes its metadata through a lightweight API or endpoint — serving JSON or YAML.

Example: /metadata/entities

[
{
“entity”: “Customer”,
“layer”: “conceptual”,
“attributes”: [
{ “name”: “CustomerID”, “datatype”: “integer” },
{ “name”: “Email”, “datatype”: “string” }
],
“derived_from”: null
}
]

The central metadata platform periodically fetches this data, validates it, and adds it to the global catalog — keeping the mesh synchronized.

5. Synchronizing the Mesh

A daily sync job orchestrates metadata updates from all domains:

import requests, duckdb, yaml

DOMAINS = [“customer”, “product”, “finance”, “ops”]
conn = duckdb.connect(“data/metadata_mesh.duckdb”)
for d in DOMAINS:
url = f”https://{d}.metadata/api/entities”
response = requests.get(url)
entities = response.json()
conn.execute(“INSERT INTO metadata.entities SELECT * FROM read_json_auto(?)”, [entities])
print(“✅ Metadata Mesh synchronized”)

This creates a distributed-yet-consistent metadata ecosystem.

6. Federated Schema and Validation

The metadata platform includes a shared schema registry that domains use for validation:

curl -X POST https://metadata.platform/api/validate \
-H “Content-Type: application/yaml” \
-d @logical_model.yaml

Each model is validated against global standards, such as:

  • naming conventions
  • required attributes
  • lineage links
  • completeness metrics

Domains remain autonomous, but their outputs are structurally compatible.

7. Global Discovery and Lineage Integration

Once ingested, the central platform aggregates metadata into:

  • A global lineage graph (conceptual → logical → physical → domain interlinks)
  • A federated entity catalog (searchable by name or concept)
  • Quality dashboards (coverage, rule adherence, and maturity per domain)

Example query:

SELECT domain, COUNT(*) AS entities
FROM metadata.entities
GROUP BY domain
ORDER BY entities DESC;

Or visualize the entire mesh using pyvis:

from pyvis.network import Network
net = Network(directed=True)
for (src, tgt, dom) in conn.execute(“SELECT entity, derived_from, domain FROM metadata.entities”).fetchall():
net.add_node(src, group=dom)
if tgt: net.add_edge(src, tgt)
net.show(“metadata_mesh.html”)

8. Metadata as an API Layer

At maturity, the mesh evolves into a metadata service fabric:

  • Each domain publishes metadata via REST or GraphQL APIs
  • The platform provides APIs for lineage queries, schema validation, and quality metrics
  • Tools like VSCode, dbt, and Notion integrate directly via these APIs

Example API endpoint:

GET /metadata/lineage?entity=Customer
→ returns upstream/downstream entities across domains

This turns metadata into a connective tissue for the entire enterprise.

9. Benefits of the Metadata Mesh

The mesh becomes both a modeling ecosystem and a living knowledge graph.

10. In Summary

The Metadata Mesh represents the final evolution of metadata-driven data architecture:

By adopting a Metadata Mesh, organizations turn metadata into infrastructure — a system of shared understanding, continuously enriched by every domain.

11. Next in the Series

“The Semantic Layer of Metadata: Linking Concepts, Rules, and Policies.”
We’ll explore how to extend metadata models into semantics — connecting concepts, rules, and ontology-level meaning for true enterprise understanding.