A modern metadata-driven development workflow where data modelers define logic and semantics declaratively, while data engineers operationalize these definitions in SQL and Jinja macros. Powered by VS Code, JSON validation, and language-server intelligence, this approach unites modeling and engineering into a single semantic DevOps loop.

Summary

Treat models as code.
Data modelers express business meaning, structure, and mappings in YAML; data engineers build reusable macros and SQL templates that translate these definitions into executable code.
Both collaborate in VS Code with language-server support, automated diagram generation (ERD, Mapping, Data Flow), and Copilot assistance. Logical and physical models coexist in one repository, validated by JSON Schemas and connected through shared metadata.

Press enter or click to view image in full size

1. A New Division of Labor

Modern data teams thrive when modelers and engineers work in parallel, not in sequence.

Press enter or click to view image in full size

The modeler defines the intent — what data means and how it should relate.
The engineer defines the mechanism — how those semantics become performant, maintainable SQL.

2. Repository Layout — Shared but Layered

repo/
├─ models/
│ ├─ ldm/ # Owned by Data Modelers
│ │ ├─ entities/
│ │ │ └─ counterparty/entity.yaml
│ │ ├─ mappings/anacredit_to_ldm.yaml
│ │ └─ subject_areas/risk.yaml
│ ├─ pdm/ # Owned by Data Engineers
│ │ ├─ warehouses/snowflake/
│ │ │ └─ tables/counterparty/table.yaml
│ │ └─ sql_macros/
│ └─ datalineage/ # Shared – modelers define, engineers automate
├─ common/
│ ├─ domains/ # Shared data type definitions
│ ├─ macros/ # Jinja/SQL templates (engineers)
│ └─ functions/ # Parameterized logic
├─ schemas/ # JSON Schemas for validation
├─ diagrams/ # Auto-generated (no manual edits)
└─ tools/
├─ lsp/ # Language Server implementation
├─ codegen/ # Diagram + SQL generation
└─ validation/

This structure allows both roles to operate independently while maintaining a single source of truth.

3. Modeling: YAML as the Semantic Layer

Modeler’s Input

models/ldm/entities/counterparty/entity.yaml

entity: Counterparty
subject_areas: [retail_banking, risk]
business_keys: [counterparty_id]
attributes:
– name: counterparty_id
type: domain:identifiers.counterparty_id
nullable: false
– name: country
type: domain:identifiers.iso_country
nullable: true
transformations:
– name: derive_country_from_address
macro: geo_lookup_country
parameters:
address_field: “customer_address”

The modeler specifies meaning and rules — not code.
They refer to reusable macros (defined by engineers) via simple parameters.

4. Engineering: Macros as the Execution Layer

Engineer’s Implementation

– common/macros/geo_lookup_country.sql
{% macro geo_lookup_country(address_field) %}
UPPER(SUBSTR({{ address_field }}, -2))

Or more complex domain-aware logic:

– pdm/sql_macros/conform_code.sql
{% macro conform_code(column, domain) %}
CASE
{% for code in load_domain(domain).codes %}
WHEN {{ column }} = ‘{{ code.value }}’ THEN ‘{{ code.standardized }}’
{% endfor %}
ELSE NULL
END

Engineers evolve these templates as reusable building blocks — connecting semantics to SQL execution.

5. Automated Code Generation

The code generator interprets YAML metadata, injects macros, and renders executable SQL:

model sql generate –entity counterparty –platform snowflake

Resulting in:

CREATE OR REPLACE TABLE CORE.COUNTERPARTY AS
SELECT
{{ geo_lookup_country(‘customer_address’) }} AS COUNTRY,
CUSTOMER_ID AS COUNTERPARTY_ID
FROM STG_CUSTOMER;

Exactly like dbt — but metadata-driven and multi-layer aware.

6. The Language Server as Collaboration Layer

The VS Code Language Server empowers both modelers and engineers:

For Modelers:

• Autocomplete for domains, macros, subject areas
• YAML structure validation via JSON Schema
• Inline preview of generated SQL
• Hover tooltips with glossary and domain info

For Engineers:

• Macro parameter suggestions
• Cross-reference validation (macro usage ↔ definition)
• Diagnostics for unused or broken macros

Both share a semantic context — the LSP mediates the conversation between meaning and implementation.

7. Logical–Physical Synchronization

Physical models (PDM) link back to their logical origins:

models/pdm/warehouses/snowflake/tables/counterparty/table.yaml

table: CORE.COUNTERPARTY
source_entity: ldm.entities.counterparty
columns:
– name: COUNTERPARTY_ID
source_attribute: counterparty_id
type: VARCHAR(64)
– name: COUNTRY
source_attribute: country
type: VARCHAR(2)
sql_templates:
– profiling
– ddl
– explore

Modelers stay focused on the abstract model, while engineers manage platform specifics and deployment code.

8. Diagrams for Shared Understanding

All diagrams (ERD, Mapping, Data Flow) are generated automatically from metadata:

model diagrams generate –include pdm –out diagrams/

• ERD: entities and relationships (modeler focus)
• Mapping: source → LDM → PDM (shared view)
• Data Flow: end-to-end lineage (engineer focus)

A single YAML edit can update documentation, lineage, and SQL logic simultaneously.

9. Macro Injection Examples

Press enter or click to view image in full size

Reusable macros codify business logic once — every modeler benefits.

10. Governance and Validation

CI/CD pipelines run all three — producing a semantic health scorecard.

11. Human Workflow

1. Modeler defines or updates an entity and mappings in YAML.
2. Engineer extends macros or SQL templates as needed.
3. Validation runs automatically; diagrams and SQL regenerate.
4. Engineer deploys to platform; modeler reviews lineage.
5. Both teams operate within the same metadata ecosystem.

12. Why It Matters

• Eliminates handover friction between modeling and engineering.
• Codifies business logic as metadata, not prose.
• Enables transparency — you can trace any column to its origin.
• Allows safe automation — every generated SQL is explainable.
• Bridges semantic intent and technical realization.

This is what semantic DevOps for data looks like — a living contract between meaning and execution.

13. The Road Ahead

Next, the language server can evolve into a Semantic Assistant — suggesting macros as modelers type, validating lineage dynamically, and learning from feedback loops to improve metadata quality.
Over time, this architecture becomes a self-documenting, self-validating ecosystem, where data modeling and data engineering converge seamlessly.

Key Takeaways

• Separate semantic ownership (modeler) from technical ownership (engineer).
• Store everything as code: models, domains, macros, diagrams.
• Use JSON Schema + LSP for real-time feedback.
• Automate diagrams and SQL code generation from metadata.
• Empower both roles in the same workspace — VS Code as the bridge.

This article is part of the ongoing series on metadata-driven data architecture and semantic DevOps — exploring how code, meaning, and automation converge to redefine data modeling.