“Shift transformation logic left” basically means: clean, type-cast, standardize, and derive fields upstream (in CTEs/subqueries), so the final SELECT is a thin projection of already-prepared columns. That gives you: One source of truth for each derived field (no copy-pasted formulas). Easier testing (you can SELECT * from any CTE). Better chances for predicate pushdown, column pruning, and join order optimization by the planner. Anti-pattern (transformations at the end) SELECT o.order_id, o.customer_id, -- repeated/complex logic in...

Encapsulate subqueries into Common Table Expressions (CTEs) to improve the readability, organization, and maintainability of SQL queries. Subqueries, especially when nested multiple times, can become hard to debug. By moving subqueries into CTEs, the query becomes modular, easier to follow, and more efficient for debugging and future modifications. If a subquery is long, complicated, or used in multiple places within the query, move it into a CTE. A CTE gives the subquery a name and...

Roelant Vos has posted a brief status update on what’s happening in the open source #datawarehouse #automation ecosystem! The generic interface for Data Warehouse Automation had to be re-based (recreated) in GitHub due to various issues around forking and branching. The repository has been re-created with the same name and is available at https://github.com/RoelantVos/Data_Warehouse_Automation_Metadata_Interface . TEAM is nearing completion for the v1.6 release. The following Kanban board shows what’s happening for v1.6: https://github.com/RoelantVos/TEAM/projects/1 . Virtual Data Warehouse v.1.6.1...

The upcoming Virtual Data Warehouse implementation & automation classroom training will be located in The Netherlands. Check out the details & register at www.virtualdwh.com. The training is delivered in conjunction with Doerffler & Partner and DCYSIVE and will once again focus on the design and implementation of a data platform that is truly flexible, scaleable and resilient. This is a very practical training that will provide the end-to-end overview of how to deliver a hybrid data warehouse solution. Utrecht is centrally...