Jaco van der Laan

How dbt and SQLMesh Organize Metadata and Code — and How to Map Them to an Entity-First Model

07/10/2025  ·  Business Friendly Mapping  ·  Data Engineering  ·  Data Modeling

‹ ›

Entity-First Series · Track B (Technical) — Part 1

Series navigation: A1 — Why We Need a Business-Friendly Data Transformation Model • B1 (you are here) • A2 — Entities, Subject Areas & Transformation Steps • B2 — YAML → SQL Generator

Summary

dbt and SQLMesh both center on files that define transformations (SQL + metadata), but they differ in how they structure that metadata and expose lineage.

This article breaks down their folder layouts, concepts, and artifacts — then shows how to map them cleanly to an Entity‑First modeling approach (Entities, Subject Areas, Roles, Step Types).

1) Quick recap of core concepts

dbt (classic analytics engineering)

  • Model: a SQL file compiled by dbt; materialized as view/table/incremental/ephemeral.
  • Source: declared in YAML; references physical upstream objects.
  • Tests: generic (schema) tests in YAML and custom data tests (SQL).
  • Macros: Jinja helpers used inside SQL.
  • Artifacts: manifest.json, run_results.json, docs site.
  • Lineage: constructed via ref() (models) and source() (sources).

SQLMesh (SQL-with-metadata in one file)

  • Model: a SQL file with a MODEL(...) header describing name, kind (VIEW/FULL/INCREMENTAL…), audits, and settings.
  • Audits: validation checks attached to models.
  • Lineage: parsed from SQL + macros; understands dependencies without explicit ref() in many cases.
  • Versioning/Snapshots: internal metadata for change detection and safe deploys.

Key difference: dbt splits a lot of metadata into YAML; SQLMesh embeds more directly in the SQL header. Both can support our Entity‑First structure — we just need a translation layer.

2) Folder layouts, side‑by‑side

Typical dbt project

“`project/
├─ models/
│ ├─ staging/
│ ├─ intermediate/ # optional convention
│ └─ marts/ (delivery)
├─ macros/
├─ snapshots/
├─ tests/
└─ seeds/

**Where the metadata lives:**

- `models/**/*.sql` → transformation logic
- `models/**/schema.yml` → tests, descriptions, sometimes sources/relationships
- `models/**/sources.yml` → source declarations
- `macros/*.sql` → reusable Jinja macros

## Typical SQLMesh project

project/
├─ models/
│ ├─ stg_customer.sql # has MODEL(…) header
│ ├─ order_enriched.sql
│ └─ order_daily_metrics.sql
└─ macros/

**Where the metadata lives:**

- `models/*.sql` → `MODEL(...)` header + SELECT body
- `macros/*.sql` → macros; audits are referenced in headers

> **_Takeaway:_** _dbt favors_ **_YAML + Jinja_** _around SQL files; SQLMesh favors_ **_SQL + header DSL_**_. Either way, files are the unit of work._

## 3) What “model” means — and why naming hurts

In both tools, a **model** is _a transformation node_. That’s pragmatic for engineers, but it blurs the line for modelers:

- A dbt model can be a **view, table, CTE, or incremental table** — the _name_ “model” doesn’t reveal its logical role.
- Logical intent (is this a **Source** entity? **Intermediate** integration? **Delivery** exposure?) is implicit in folders and naming conventions.

**Recommendation:** keep the engines’ terminology internally, but **present and store your source‑of‑truth in Entity‑First terms**:

- **Entity** (name + subject area) with **role** (Source/Intermediate/Delivery)
- **Pattern** (Core/Characteristic/Associative/Reference)
- **Step type** (derive/filter/aggregate/union/translate)
- **Schema** (columns, PK/FKs/constraints) and **relationships**

Then **generate** dbt or SQLMesh files from that metadata. The engines keep “models,” the business sees **entities**.

## 4) Mapping Entity‑First → dbt

> _Goal: Emit dbt‑ready SQL + YAML while preserving business semantics._

**Entity YAML (source of truth)** → generate:

- `models/<layer>/<subject_area>/<entity>.sql`
- Uses `ref()`/`source()` for lineage
- Materialization set via `{{ config(materialized='...') }}` or project config
- `models/<layer>/<subject_area>/schema.yml`
- Column descriptions, PK/UK/NOT NULL tests, relationships
- Exposure definitions where applicable
- Optional `macros/*.sql` for **translate** / **derive** functions

**Layering convention** (map your roles):

source → models/staging/
intermediate → models/integration/
delivery → models/marts/

**Examples** (shortened):

– models/integration/order_enriched.sql
{{ config(materialized=’table’) }}
with o as (select * from {{ ref(‘stg_order’) }}),
c as (select * from {{ ref(‘stg_customer’) }})
select o.order_id, c.country_code, o.amount
from o join c on o.customer_id = c.customer_id


models/integration/schema.yml (excerpt)

version: 2
models:
– name: order_enriched
description: Orders joined with customers
columns:
– name: order_id
tests: [not_null, unique]
– name: customer_id
tests:
– relationships:
to: ref(‘stg_customer’)
field: customer_id

## 5) Mapping Entity‑First → SQLMesh

**Entity YAML** → generate **one file per model** with a `MODEL(...)` header:

MODEL (
name analytics.order_enriched,
kind FULL,
audits (not_null(order_id))
);


WITH o AS (SELECT * FROM ref(‘analytics.stg_order’)),
c AS (SELECT * FROM ref(‘analytics.stg_customer’))
SELECT …

- Use `kind` to express materialization.
- Attach audits for constraints.
- Macros implement reusable transformations (e.g., `translate_code(domain, expr)`).

**Note on lineage:** SQLMesh parses SQL for dependencies; explicit `ref()` keeps things clear and portable.

## 6) Representing step types (derive/filter/aggregate/union/translate)

Your Entity‑First metadata should **constrain** what’s allowed per step and drive generation:

- **derive** → column‑level mappings/expressions, joins via relationships
- **filter** → boolean predicate
- **aggregate** → `group_by`, aggregation functions, post‑derivations, PK for the aggregated grain
- **union** → list of aligned inputs
- **translate** → code/label lookups (reference entities) or macro calls

This yields **predictable SQL** that’s easier to audit and test.

## 7) Tests, DQ, and profiling — auto‑generated

From the entity metadata:

- **PK/UK/NOT NULL/FK** → dbt tests or SQLMesh audits
- **Row‑level quality** → templated queries emitted into `/tests/dq/`
- **Profiling** → simple selects, top‑N, null ratios for sources & intermediates

Keep these generated; hand‑written DQ lives in a separate `tests/` area.

## 8) Naming improvements (what we wish dbt had out‑of‑the‑box)

If dbt started today with a modeling lens, you might see:

entities/
sales/
order/ # entity folder
entity.yml # schema, PK, relationships, step type
dbt.sql # generated SQL (don’t edit)
tests.yml # generated

But we don’t need dbt to change — we can **generate** this into classic dbt structure at build time.

## 9) Migration & adoption checklist

1. **Define the metamodel** (Entities, Roles, Patterns, Steps, Relationships).
2. **Author entity YAMLs** for a small subject area.
3. **Generate dbt/SQLMesh** models + tests; commit to Git.
4. **Run** and compare outputs to existing hand‑written SQL.
5. **Add diagram generation** (draw.io/Mermaid) from entity metadata.
6. **Train the team** on naming & layering rules; enforce with CI validation.
7. **Iterate**: add SCD2, incremental patterns, platform‑specific optimizations via macros.

## 10) Where alternative frameworks fit

- **Dagster (assets)** — asset‑centric DAG aligns with “entity” thinking.
- **Semantic layers (LookML/MetricFlow)** — don’t replace transforms, but inform entity & relationship naming.
- **Orchestrators (Airflow/Prefect)** — schedule & ops; pair with your generator output.
- **Catalog/Lineage (OpenMetadata, DataHub)** — ingest dbt/SQLMesh artifacts and your entity metadata for discovery.

## 11) Tying back to the business‑friendly series

This technical guide sits directly under the business vision introduced in [**A1 — Why We Need a Business‑Friendly Data Transformation Model**](https://medium.com/business-friendly-data-mapping/why-we-need-a-business-friendly-data-transformation-model-63ef3f9579c8). In short:

- **Entities** carry **roles** (Source / Intermediate / Delivery) and **patterns** (Core / Characteristic / Associative / Reference).
- **Step types** (derive, filter, aggregate, union, translate) make intent explicit and **auditable**.
- Everything is modeled in **Git + VSCode**, with diagrams and docs **generated** from metadata and always current.

> _For additional context from my Medium archive:_
> 
> [**Business‑Oriented Data Modeling — Unlock Your Business’s True Potential**](https://medium.com/@jacovanderlaan/unlock-your-businesss-true-potential-the-power-of-business-oriented-data-modeling-651a639c40e5)
> 
> [**Business‑Friendly Mapping Meets dbt — Virtualizing Logical Building Blocks into SQL**](https://medium.com/p/1b7d5775a8f6)

## What’s next (B2 & A2)

- **B2 (Technical):** _Designing an Entity‑First Transformation Framework: YAML → SQL Generator_ — schema, validation, CI checks, and a runnable mini‑project.
- **A2 (Functional):** _From Sources to Delivery: Entities, Subject Areas & Transformation Steps_ — diagrams and plain‑language walkthroughs.

We’ll cross‑link these posts: B2 will reference this article for the conceptual mapping; A2 will reference this one for implementation details.

## Appendix: tiny side‑by‑side snippet

**Entity YAML (conceptual)**

entity: order_enriched
subject_area: sales
role: intermediate
pattern: core
primary_key: order_id
step:
type: derive
inputs:
– entity: sales.order
– entity: sales.customer
relationships:
– left: sales.order.customer_id
right: sales.customer.customer_id
columns:
order_id: sales.order.order_id
country_code: sales.customer.country_code
amount: sales.order.amount

**Emitted dbt model**

{{ config(materialized=’table’) }}
with o as (select * from {{ ref(‘stg_order’) }}),
c as (select * from {{ ref(‘stg_customer’) }})
select o.order_id, c.country_code, o.amount
from o join c on o.customer_id = c.customer_id

**Emitted SQLMesh model**

MODEL (name analytics.order_enriched, kind FULL);
with o as (select * from ref(‘analytics.stg_order’)),
c as (select * from ref(‘analytics.stg_customer’))
select o.order_id, c.country_code, o.amount
from o join c on o.customer_id = c.customer_id;
“`

🔗 Series footer

Previous: A1 — Why We Need a Business‑Friendly Data Transformation Model
This article: B1 — How dbt & SQLMesh Organize Metadata and Code
Next: B2 — YAML → SQL Generator · A2 — Entities, Subject Areas & Transformation Steps