Jaco van der Laan

Spec-Driven Development for Data Platforms: Turning Specs into Reliable Pipelines

16/10/2025  ·  Data Engineering  ·  Reproducibility

‹ ›

How metadata, business-friendly mappings, and AI-assisted specifications redefine data engineering.

In data architecture, clarity is power.
Data platforms thrive when every transformation has a single source of truth — when intent, code, and validation all trace back to a clear and verifiable definition.

That’s the promise of Spec-Driven Development (SDD): building and evolving systems where specifications — not raw code — drive development.
It’s an idea emerging fast in the software world, and it fits perfectly into how we already design, build, and maintain modern data platforms.

Press enter or click to view image in full size

This article explores how SDD principles can be applied to data engineering — how we can use specs to generate, validate, and govern the data pipelines that power analytical ecosystems.

These ideas build on Birgitta Böckeler’s insightful Thoughtworks article, Understanding Spec-Driven Development: Kiro, spec-kit, and Tessl (part of the Exploring Gen AI series).
Here, we extend those concepts into the world of metadata-driven data platforms, where specs become the backbone of reproducible, business-friendly, and AI-assisted data delivery.

Why Spec-Driven Development belongs in a data platform

Data platforms live and die by traceability and repeatability — every column’s purpose, every rule’s rationale, every change’s intent.

Spec-Driven Development (SDD) is a natural fit because it makes the spec — not ad-hoc code — the primary artifact that guides generation, testing, and maintenance.

In our architecture we already favor metadata-driven practices: logical-to-physical mappings, dual SCD2/bi-temporal historization, PIT tables, draw.io-to-metadata flows, dbt/Jinja-style macro generation.
SDD gives this approach a durable, auditable structure:

  • Spec-first: capture intent before touching SQL/Python.
  • Spec-anchored: keep the spec with the feature for later evolution.
  • Spec-as-source: generate code from specs; humans mostly edit specs, not code.

The three SDD levels, mapped to a data stack

1. Spec-first — Document intent before code

Use a lightweight, structured spec before implementing a change.
We already write user stories and acceptance criteria; we’ll formalize them into data-fluent specs that feed our agents and generators.

Examples of use:
Small bug fixes, new columns, optimized queries, or rule adjustments.

Artifacts:

  • Transformation Spec (YAML/MD): input contracts, mapping rules, edge handling, performance SLOs.
  • Acceptance Tests (Gherkin): GIVEN source slices → WHEN rule runs → THEN deterministic outputs.
  • Impact Note: lineage deltas, downstream consumers.

2. Spec-anchored — Evolve the spec with the feature

Keep specs in the repo. Treat them as living documentation — evolving as the code evolves.

Examples of use:
Changing historization logic, re-partitioning large models, updating data product SLAs.

Artifacts:

  • Data Product Charter: business purpose, SLAs, interfaces, retention, and PII policy.
  • Runbook: operational plans, backfills, and rollback strategies.
  • Versioned Contract: schema + semantics with deprecation management.

3. Spec-as-source — Generate and round-trip from specs

Codify the spec so code is generated and round-tripped from it (think “GENERATED FROM SPEC — DO NOT EDIT”).

We already use dbt macros to generate SQL from configuration — this extends the pattern across transformations, tests, and ops.

Examples of use:
Standardized patterns like SCD2, PIT, CDC merges, and DQ validation.

Artifacts:

Low-level component specs that generate:

  • dbt SQL models
  • Databricks/Airflow tasks
  • Automated data quality tests

From Functional Mapping Blocks to Spec-Driven Data Pipelines

Before SDD became a buzzword, we were already applying its principles through our business-friendly data mapping approach — where functional logic, not SQL syntax, defines transformations.

Each transformation step — Select, Derive, Filter, Aggregate (Group), Union, Pivot, etc. — is modeled as a logical building block in metadata.
These blocks describe what to do, not how to code it.
From these specs, our generators create executable SQL (e.g., dbt models or Databricks notebooks) using Jinja macros and metadata templates.

That means our platform already operates at a spec-first level:

  • The mapping metadata is the spec.
  • Generated SQL is just the “last-mile” artifact.
  • The same metadata feeds documentation, lineage, and automated validation.

Under the SDD lens, these mappings form a domain-specific spec language for data pipelines — structured, testable, and business-readable.

Example transformation spec:

block: derive
source: customer_stg
target: customer_enriched
columns:
– name: customer_age_group
expression: CASE
WHEN age < 25 THEN ‘YOUNG’
WHEN age BETWEEN 25 AND 50 THEN ‘ADULT’
ELSE ‘SENIOR’ END
– name: risk_rating
expression: coalesce(risk_score, ‘UNKNOWN’)
tests:
– not_null: customer_id
– values_in_set:
column: risk_rating
values: [LOW, MEDIUM, HIGH, UNKNOWN]

This spec is functional, readable, and transformable:

  • The AI agent or generator can produce dbt SQL from it.
  • Reviewers validate business intent directly in the spec.
  • Tests are automatically generated and aligned with logic.

By formalizing these functional mapping specs within the SDD framework, we unify:

  • Functional definition (business logic)
  • Generated implementation (SQL, dbt, Databricks code)
  • Executable validation (tests derived from the same spec)

This turns the platform into a Spec-as-Source architecture — business logic, code, and validation are synchronized, versioned, and auditable.

Your “Memory Bank” for AI and humans

SDD tools like Kiro and spec-kit introduce the concept of a memory bank — a persistent, shared context that guides all development sessions.
In our data solution architecture, this translates to a project-level knowledge base in Git:

  • ARCHITECTURE.md — layers, naming, historization, PIT policies, time semantics.
  • STANDARDS.md — coding conventions, review checklists, PII rules, SLOs.
  • AGENTS.md — how coding assistants should behave; what to avoid.
  • DOMAINS/ — business glossary and canonical entities.
  • TEMPLATES/ — reusable spec skeletons (Ingest, SCD2, PIT, DQ, Contracts).
  • SAMPLES/ — “golden” examples of well-written specs and generated code.

This mirrors GitHub’s constitution idea in spec-kit — but tailored for data engineers and data product owners.

Tooling patterns inspired by Kiro, spec-kit, and Tessl

We don’t need to adopt these tools literally; their workflows can be implemented using VS Code + Git + dbt + Python.

Pattern A — “Kiro-light” (spec-first)

Use three simple markdown docs: requirements.md, design.md, and tasks.md.
Perfect for small features or one-off changes.

Pattern B — “spec-kit style” (spec-anchored)

Add a repo-level constitution and structured checklists to ensure standards, DQ, lineage, and acceptance are all covered.
Use templates to spawn spec.md, plan.md, tasks.md, dq.md, and api.md.

Pattern C — “Tessl-style” (spec-as-source)

Create YAML-based specs that generate SQL, Python, and tests.
Protect generated files with headers like // GENERATED FROM SPEC – DO NOT EDIT.

Example: Adding a bi-temporal attribute safely

Scenario: Add risk_rating to dim_customer with dual SCD2 historization and a monthly PIT.

requirements.md

  • As a risk analyst, I want risk_rating to appear in dim_customer with full bi-temporal history.
  • Acceptance: GIVEN a source change on 2025–09–01 (as-of) arriving 2025–09–03 (as-at), THEN PIT on 2025–09–30 must reflect correct value.

design.md

  • Lineage: stg_customer.risk_rating → dim_customer.risk_rating → pit_customer_monthly
  • Rules: Default to UNKNOWN; maintain valid_from/valid_to; preserve history.
  • Test plan: Differential tests vs reference snapshot.

dim_customer.spec.yml

  • Defines dual timelines, merge keys, tests, and PIT schedule.

Generated artifacts: dbt models, PIT SQL, test files, and backfill scripts.
All versioned and traceable back to the spec.

Implementation blueprint

  • Repo scaffolding: /specs/, /memory/, /templates/
  • Validation: Pre-commit hooks for linting and reference integrity.
  • Code generation: Python CLI renders dbt SQL and tests from specs.
  • Reverse engineering: Create specs from legacy SQL to onboard brownfield assets.
  • CI/CD: Specs linted, compiled, tested, and verified before merge.
  • Governance: Publish specs and lineage automatically to Confluence or Notion.

Benefits you can measure

  • Traceable intent — every column, every test, every join has a purpose.
  • Faster onboarding — specs read like business logic, not code.
  • Regulatory reproducibility — time travel and historization are spec-governed.
  • Reduced review time — reviewers read specs and diffs, not 500-line SQL.
  • Quality by design — tests and lineage are generated from the same truth.

Pitfalls to watch out for

  • Spec bloat: keep specs concise and structured.
  • Agent non-determinism: constrain LLM outputs; use schema validation.
  • Functional vs technical drift: separate spec.md (intent) from spec.yml (implementation).
  • Rigid workflows: choose the right SDD level (spec-first, anchored, or as-source) per change size.

30–60–90 day rollout roadmap

Days 1–30: Pilot spec-first on small enhancements.
Days 31–60: Introduce anchored specs for shared data products.
Days 61–90: Automate generation and round-trip sync for SCD2/PIT patterns.

Success metrics: shorter PR cycles, fewer production defects, reproducible data states, and self-documenting pipelines.

Final thought

Spec-Driven Development isn’t a tool — it’s a mindset.
Start with spec-first to clarify intent, evolve into spec-anchored for maintainability, and graduate to spec-as-source for automation.

When applied to metadata-driven data platforms, it bridges the gap between business understanding and technical precision — making our systems governed by intent, generated with confidence, and validated by design.

🧭 Acknowledgment

This article was inspired by Birgitta Böckeler’s Understanding Spec-Driven Development: Kiro, spec-kit, and Tessl, part of Thoughtworks’ Exploring Gen AI series.
It extends her exploration of SDD into the world of metadata-driven data architecture and business-friendly data mapping, where functional intent drives the automatic generation of high-quality, testable SQL.