How rule-based AI, metadata, and modeling standards are redefining how we build and govern data platforms.
💡 Summary
Over the past months, we’ve built something far beyond a migration tool.
What started as an experiment to automate SQL refactoring evolved into a metadata-driven collaboration framework — one that unites AI, business modelers, and engineers around a single governed workflow.This final article looks at where this journey leads next: toward a metadata-first, AI-assisted data development culture, where collaboration, documentation, and governance are all part of the same continuous process.
Press enter or click to view image in full size
From Code Conversion to Knowledge Engineering
When we first used AI to help migrate SQL, our goal was simple:
translate existing logic from one data platform to another.
But each step we took revealed a deeper opportunity.
- Parsing SQL gave us structure.
- Defining YAML rules gave us governance.
- Automating validation gave us control.
- Integrating Copilot gave us collaboration.
By combining these elements, we turned AI from a text generator into a knowledge partner — one that understands both the business intent and the technical context of every transformation.
The Metadata Core
At the heart of everything is metadata.
It’s what allows AI to reason, validate, and explain.
Our DuckDB metadata layer now acts as a central knowledge graph containing:
- Table and column structures
- Relationships and lineage
- SCD2 validity rules
- Rule compliance reports
- Validation results
- Documentation summaries
Each SQL query, YAML rule, and AI prompt becomes a record in that graph — traceable, explainable, and reusable.
Instead of static SQL files, we’re building a living metadata ecosystem.
The Evolution of the Rule Framework
When we began defining rules, they were focused on technical hygiene:
no SELECT *, isolate GROUP BY, remove unused CTEs.
Now, those same rules have become governance policies:
- Rules for deduplication are tied to DQ ownership.
- Rules for SCD2 filtering enforce business timelines.
- Rules for naming conventions align with data model design.
- Validation invariants ensure auditability.
What used to be “best practices” are now machine-executable contracts between teams. They define not just how we code — but how we collaborate.
Copilot and the Human–AI Partnership
As we integrated Copilot into VS Code, something changed in the way teams worked.
Instead of handing off Excel mappings or static diagrams, business developers and engineers began co-creating.
Copilot now assists both sides:
- Modelers get help writing mappings, ERDs, and documentation.
- Engineers get optimized SQL, validation hints, and rule enforcement.
- Governance gets transparency — every AI decision is logged, validated, and explainable.
AI stopped being an external “assistant” and became part of the team structure — a governed collaborator working within clear boundaries.
Governance as a Natural Part of Development
In most organizations, governance is still treated as a separate function — something that happens after code is written. Our approach flips that paradigm.
By embedding governance into:
- Rule definitions (YAML)
- Validation scripts (Python)
- Documentation generation (AI)
we made it invisible — part of the development cycle, not an audit step.
Developers aren’t filling governance forms anymore. They’re producing governed artifacts by design.
Toward Open Standards and Semantic Interoperability
We’re not alone in this direction.
The industry is converging toward semantic interoperability — the idea that models, rules, and transformations should share a common language across tools.
Initiatives like the Open Semantic Interchange (OSI) between Snowflake, Salesforce, dbt Labs, and others show where this is heading: data models and semantics defined once, applied everywhere.
Our framework aligns naturally with that future:
- Metadata stored in open formats (DuckDB, YAML, Markdown).
- Diagrams generated as Draw.io files — not proprietary.
- Rules expressed as plain YAML — readable by both humans and machines.
- AI integrated through prompts, not locked-in APIs.
That’s interoperable governance — and it’s the foundation for long-term sustainability.
What We’ve Learned
Looking back, this project taught us more than we expected about the role of AI in data engineering.
Here’s what stands out:
- AI thrives in structure.
The more context we gave — metadata, rules, examples — the more consistent and useful the results became. - Governance can be automated without being bureaucratic.
When expressed as rules and validations, it becomes part of development — not a blocker. - Documentation should be a side effect, not a deliverable.
The best documentation is generated from metadata and rules — not written afterward. - Metadata is the real product.
SQL, YAML, diagrams — all are just representations of the same underlying semantics. - The future of data development is collaborative.
Engineers, modelers, and AI working in one governed environment will replace disconnected tools and handovers.
Where We Go from Here
We’re now extending this approach across the Finance & Risk Data Domain — scaling it from a migration experiment into a full metadata-driven development ecosystem.
Next steps include:
- Integrating Copilot’s rule awareness into more use cases (data quality, lineage validation).
- Exposing metadata through APIs for analytics, lineage dashboards, and impact analysis.
- Linking governance metadata to business glossaries and KPI definitions.
- Contributing learnings to open metadata and OSI communities.
Because in the end, the mission isn’t just to migrate SQL — it’s to elevate the way we design, document, and reason about data.
Key Takeaways
- The combination of AI, metadata, and governance as code redefines how data platforms evolve.
- The rule framework enables explainable, reproducible, and auditable transformations.
- Copilot turns modeling and documentation into a collaborative, interactive process.
- Metadata becomes the single source of truth — powering automation and understanding alike.
- The future of data engineering is not just technical — it’s semantic, collaborative, and explainable.
💬 If you’ve followed this series from the start — from SQL parsing to rule-based AI and Copilot-assisted modeling — you’ve seen how we’re reimagining data development as a metadata-driven, governed, and human–AI collaborative process.
🧭 About This Series
This article is part of the series “AI-Assisted SQL Migration” — an exploration of how we can combine metadata, Python, and Generative AI to transform the way we design, migrate, and govern data platforms.
Series Overview:
1️⃣ Why AI-Assisted SQL Migration Matters
2️⃣ Extracting the Hidden Metadata Inside SQL
3️⃣ From Raw SQL to Logical Building Blocks
4️⃣ Teaching AI to Apply Rules: Prompts, Validation, and Feedback Loops
5️⃣ Governance by Design: Making SQL Transparent, Modular, and Auditable
6️⃣ The Power of YAML Rules: Deduplication, Unions, and Optimization Patterns
7️⃣ The Python Rule Engine: Automating Transformation, Validation, and Documentation
8️⃣ Beyond Migration: Copilot-Assisted Data Modeling in VS Code
9️⃣ AI and Metadata-Driven Collaboration: The Future of Data Development
💬 Related Essay:
Do We Still Need Traditional Data Modeling Tools? — A reflection on whether AI-assisted, VS Code-based modeling makes legacy tools like PowerDesigner or ERwin obsolete.
🤝 Join the Conversation
I’m exploring how AI, metadata, and model-driven thinking can make data platforms more transparent, explainable, and collaborative.
If you’ve experimented with similar ideas — or have thoughts on applying AI to data modeling, migration, or governance — I’d love to hear from you in the comments or connect on LinkedIn.