💡 We have open formats, automation, and AI — but not yet one coherent ecosystem.
1️⃣ The Reality Check: Open, But Fragmented
In the last few years, the data modeling world has made incredible progress. We’ve moved from proprietary repositories to open metadata, from ==static diagrams== ==to== ==version-controlled YAM==L, and from manual governance to automated validation.
But as one reader recently pointed out in response to my article Do We Still Need Traditional Data Modeling Tools?:
“There’s no integrated tool to gather these open data formats into a coherent single experience.”
And that’s exactly the truth.
We now have:
- ==YAML and JSON to describe metadata==
- Git for version control and collaboration
- dbt for data transformation logic
- CI/CD pipelines for validation
- AI assistants for conceptual modeling
- And visual tools for diagrams and semantic layers
Yet we still lack one integrated experience that ties all of these together — open, modular, and seamless. We’ve built the parts; we haven’t built the orchestra.
2️⃣ ==The Two Layers of Modeling:== Fuzzy vs Deterministic
Another important point raised in that same comment was about AI’s role.
AI is excellent at fuzzy interpretation — translating workshops, documents, or conversations into conceptual structures. It helps us turn unstructured input into early models.
But when it comes to deterministic modeling — generating consistent DDL, dbt code, or governed metadata — AI quickly hits its limits. That’s where rules, code, and automation pipelines still outperform AI every time.
So the balance is clear:
Use AI for context, and code for consistency.
3️⃣ What We Have Today: The Patchwork Approach
Right now, the most successful teams use a composable stack that looks something like this:
Modeling / Visualization → Hackolade, SqlDBM, Ellie.ai, Draw.io
Metadata as Code → YAML / JSON files in Git
Validation / Governance → JSON Schema, Pydantic, Great Expectations
Code Generation → Jinja2 templates for DDL, dbt, docs
Automation / Delivery → GitHub Actions / GitLab CI / Airflow
AI Assistance → Concept extraction, naming, documentation
This works. It’s modern, open, and powerful.
But it’s also disconnected — each piece does its job in isolation.
We still spend time passing YAML between systems, converting diagrams into code, or reconciling metadata versions between catalogs, dbt, and notebooks.
In short, we’ve gone from monoliths to microtools — but we haven’t achieved flow.
4️⃣ What’s Still Missing: The Metadata Operating System
What’s needed now isn’t another modeling tool. It’s a metadata operating system — an environment that connects the pieces we already have.
Such a platform would:
- Synchronize all metadata artifacts across tools
- Manage relationships and lineage across YAML, dbt, and catalogs
- Provide APIs for visualization, validation, and generation
- Let AI reason over metadata, not replace it
- Be vendor-neutral and file-based, yet user-friendly for business stakeholders
It wouldn’t reinvent the wheel — it would connect the wheels we already built.
5️⃣ A Blueprint for Integration
We can already imagine this architecture:
┌───────────────────────────────┐
│ Business Layer │
│ Browser / Diagram tools │
│ (Hackolade, Ellie.ai, etc.) │
└──────────────┬────────────────┘
│
┌──────────────┴────────────────┐
│ Metadata Core (Git) │
│ – YAML/JSON as source of truth│
│ – Schema validation (CI/CD) │
│ – Change lineage tracking │
└──────────────┬────────────────┘
│
┌──────────────┴────────────────┐
│ Automation Layer │
│ – dbt, Jinja, SQL generation │
│ – Testing and CI pipelines │
│ – AI-assisted enrichment │
└──────────────┬────────────────┘
│
┌──────────────┴────────────────┐
│ Data Platform Layer │
│ (Snowflake, Databricks, etc.)│
└───────────────────────────────┘
🔹Every layer is open.
🔹Every artifact lives in Git.
🔹Every process is automated.
And AI helps bridge the gaps between human context and machine structure.
6️⃣ Conclusion: The Next Frontier
🔹We’ve proven that metadata-as-code works.
🔹We’ve proven that AI can assist in modeling.
🔹Now we need to make it coherent.
The next breakthrough won’t come from a new modeling notation or another diagramming tool — it will come from integration. A unified environment where metadata, AI, and automation coexist naturally.
Until then, we’ll keep building the orchestra — one instrument at a time.