Automation Scripts — One-Click Pipelines for Model Management

Every Model Workspace Generates Automation Scripts

Automation Scripts — One-Click Pipelines for Model Management

Every Model Workspace Generates Automation Scripts

Most frameworks fail not because they lack features.

They fail because they add friction.

If engineers must remember five CLI commands, configure paths manually, or guess the correct sequence of steps — adoption slows down.

MDDE was built under migration pressure.

Which means:

Workflows had to be repeatable.
Deterministic.
Low-friction.
And safe.

That is why every model workspace generates automation scripts.

The Problem: Tooling Friction

In a typical metadata-driven workflow, you need to:

Re-import SQL or source model
Parse and write metadata
Run analysis
Run optimizer passes
Review diagnostics
Export artifacts
Regenerate documentation
Generate DDL
Possibly run quality checks

Even if each step is simple, the sequence matters.

Humans forget sequences.
Automation prevents that.

The MDDE Workspace Pipeline

Each workspace contains:

scripts/  
    reimport.cmd  
    analyze.cmd  
    optimize.cmd  
    export.cmd  
    full_pipeline.cmd

Unix equivalents (.sh) are also generated.

These are not generic scripts.
They are model-aware.

1️⃣ Reimport Script

python -m mdde.importer.sql_importer \  
    --model-id mdl_sales \  
    --db databases/meta.duckdb

Purpose:

Refresh metadata from SQL
Maintain stable entity identifiers
Prevent duplication

Reimport is safe because identity is preserved via entity_id.

2️⃣ Analyze Script

python -m mdde.scripts.analyze_model \  
    mdl_sales \  
    --db databases/meta.duckdb \  
    --verbose

This runs:

Structural validation
Relationship checks
Naming compliance
Quality rule derivation
Observation detection

Output is written to structured diagnostics.
Not console-only warnings.

3️⃣ Optimize Script

python -m mdde.scripts.optimize_model \  
    mdl_sales \  
    --db databases/meta.duckdb \  
    --verbose

Optimizer passes include:

Function extraction
Filter normalization
Seed extraction
Join detection
Window classification
Canonicalization

Each pass emits structured results.
No blind rewriting.

4️⃣ Export Script

python -m mdde.scripts.export_model \  
    mdl_sales \  
    --output workspace/applications/sales \  
    --html-documentation \  
    --ddl \  
    --dialect snowflake \  
    --verbose

Export generates:

Canonical SQL
Multi-dialect DDL
Interactive documentation
Diagrams
Quality SQL
Contracts (optional)

One command.
Multiple artifacts.

5️⃣ Full Pipeline Script

The full pipeline coordinates everything:

@echo off  
echo [1/4] Re-importing model...  
call reimport.cmd


echo [2/4] Analyzing model...  
call analyze.cmd


echo [3/4] Optimizing model...  
call optimize.cmd


echo [4/4] Exporting artifacts...  
call export.cmd


echo Pipeline completed successfully.

Sequence is explicit.
Order is enforced.
Failure stops execution.
Determinism is preserved.

Why This Matters

Automation scripts provide:

Repeatability
Lower onboarding cost
CI integration points
Safe execution ordering
Fewer manual errors

They turn the framework into:
A workflow, not just a library.

CI/CD Integration

The same scripts can be called inside CI:

full_pipeline.sh

You can then:

Fail on compliance score below threshold
Fail on severe optimizer diagnostics
Fail on unresolved lineage
Enforce no SELECT * policy

Architecture becomes enforceable.
Without making it rigid.

VSCode as Operational Surface

Because scripts live in the workspace:

Engineers see the pipeline
Execution is transparent
No hidden automation
No implicit ordering

Everything is inspectable.
Everything is version-controlled.

Determinism and Trust

In migration contexts, trust matters.

When:

SQL is regenerated
DDL is emitted
Documentation is refreshed

Teams must trust:
The pipeline is deterministic.

Automation scripts enforce:

Same order
Same parameters
Same outputs

For identical inputs.

Why This Is More Than Convenience

This is architectural hygiene.

Automation ensures:

Models do not drift
Artifacts remain synchronized
Observations are regularly reviewed
Documentation is never stale
Dialect output remains aligned

Without relying on memory.

🔧 Technical Deep Dive

Inspect Last Pipeline Run

SELECT  
    pass_name,  
    severity,  
    message  
FROM metadata.optimizer_diagnostics  
WHERE model_id = 'mdl_sales'  
ORDER BY created_at DESC;

Execute Full Pipeline

scripts/full_pipeline.cmd

Or:

./scripts/full_pipeline.sh

CI Gate Example

python -m mdde.scripts.analyze_model mdl_sales --fail-on-error

Closing

Architecture is not just modeling.
It is repetition.

Automation is what turns:
Good structure into Sustainable structure.