Lifecycle Traceability
Observations and Proposals — How MDDE Suggests Improvements Without Forcing Change
Lifecycle Traceability
Many frameworks fail at the same point:
They confuse detection with enforcement.
They detect a pattern…
…and immediately try to fix it.
That works in greenfield projects.
It fails in production systems.
MDDE takes a different stance:
Observe first.
Propose second.
Act only when approved.
This separation is deliberate.
Why Automatic Refactoring Is Dangerous
In real systems:
- Queries contain historical edge cases
- Performance optimizations exist for specific reasons
- Workarounds hide migration constraints
- Naming inconsistencies have downstream dependencies
If a tool rewrites SQL blindly, it risks:
- Semantic drift
- Hidden regressions
- Breaking downstream consumers
- Creating distrust
After working in migration-heavy environments, one lesson stands out:
Change must be explainable.
And explainability starts with observation.
Step 1: Observations
After parsing and optimization passes, MDDE produces observations.
An observation is:
- Structured
- Typed
- Scoped
- Non-invasive
Example: naming convention issue
- observation_id: obs_001
category: naming
severity: warning
title: "Entity name does not follow convention"
message: "Entity 'tbl_customers' uses prefix 'tbl_'"
entity_id: ent_abc123
suggested_action: "Rename to 'customer'"
Example: pattern detection
- observation_id: obs_002
category: pattern
severity: info
title: "Deduplication pattern detected"
message: "ROW_NUMBER partition used for deduplication"
entity_id: ent_def456
Example: compliance issue
- observation_id: obs_003
category: compliance
severity: error
title: "SELECT * usage detected"
message: "SELECT * violates methodology rule"
suggested_action: "Specify explicit columns"
Nothing changes automatically.
The system observes.
Observation Categories
MDDE supports multiple observation types:
1. Quality Observations
- Naming conventions
- Data type mismatches
- Nullability inconsistencies
- Primary key violations
2. Pattern Observations
- Aggregation entities
- Deduplication patterns
- Window function usage
- Repeated expressions
3. Structural Observations
- Missing relationships
- Unresolved lineage
- Ambiguous column references
4. Compliance Observations
- Methodology violations
- Policy enforcement gaps
- Required metadata missing
Each observation is data.
Not a log line.
Not a warning buried in output.
Structured metadata.
Step 2: Proposals
An observation can optionally become a proposal.
Proposals are actionable transformations.
But they require human review.
Example:
proposal:
proposal_id: prop_001
observation_id: obs_003
status: proposed
before_example: |
SELECT * FROM customer
after_example: |
SELECT
customer_id,
customer_name,
email
FROM customer
impact_analysis:
attributes_affected: 3
downstream_entities: 2
Notice what’s included:
- Before SQL
- After SQL
- Impact analysis
- Explicit scope
Transparency is mandatory.
Proposal Lifecycle
Every proposal follows a lifecycle:
Detected → Proposed → Accepted → Applied
↘ Rejected
This matters because:
- Decisions are traceable
- Rejections are recorded
- Nothing is implicit
- Audit trails exist
MDDE does not assume correctness.
It assumes review.
Before / After Previews
One of the most powerful features is diff visibility.
Engineers can see:
- Exact SQL changes
- Attribute-level impact
- Lineage adjustments
- Dependency ripple effects
No black-box rewriting.
Batch vs Selective Application
Teams can choose:
- Apply a single proposal
- Apply all proposals of a category
- Apply only “safe” optimizations
- Export without applying anything
Control remains explicit.
Why This Separation Matters
In production systems:
- Some violations are intentional
- Some patterns are temporary
- Some inconsistencies are legacy constraints
By separating observation from enforcement, MDDE:
- Builds trust
- Reduces risk
- Encourages gradual improvement
- Enables incremental adoption
You can start by observing.
That alone creates value.
Observations as Architectural Feedback
Over time, observations become signals:
- Repeated naming violations → update conventions
- Frequent dedup patterns → formalize stereotype
- Many SELECT * violations → tighten CI gate
- Persistent drift → revisit ADR
The observation engine becomes:
A feedback loop for architecture.
Not a policing tool.
CI Integration
Observations can integrate into CI:
- Warnings → informational
- Errors → block merge
- Compliance score thresholds → enforce quality
But enforcement is configurable.
Observation always precedes policy.
VSCode Workflow
In practice:
python -m mdde.scripts.analyze_model mdl_sales --verbose
Engineers see:
- Structured observations
- Proposal previews
- Severity classification
- Compliance scoring
All before any SQL is rewritten.
Why This Is the Real Differentiator
Many tools parse SQL.
Many tools generate SQL.
Few tools create:
- Structured observations
- Reversible proposals
- Explainable optimizations
- Audit-ready metadata transformations
That middle layer is what makes MDDE production-safe.
Closing
Architecture evolves.
But it should evolve visibly.
Observations make structure inspectable.
Proposals make change deliberate.
Separation makes it safe.
Next in the Series
In the next article: Interactive HTML Documentation — Self-Service Data Discovery
We’ll explore:
- Generated entity documentation
- Searchable model catalogs
- Interactive lineage
- D3-based visualization
- Always-current architecture artifacts
Structure is extracted.
Now it becomes explorable.
🔧 Technical Deep Dive — Proposal Lifecycle
Inspect proposals:
SELECT
proposal_id,
observation_id,
status,
created_at
FROM metadata.proposal
WHERE model_id = 'mdl_sales';
Apply proposal programmatically:
python -m mdde.scripts.apply_proposal prop_001
Rejection is tracked:
UPDATE metadata.proposal
SET status = 'rejected',
rejection_reason = 'Intentional denormalization'
WHERE proposal_id = 'prop_003';
Every decision is auditable.