Data Solution Design Patterns

"For a data warehouse, we do not have enough time."

… sounds familiar to you?

Automation and code generation enables faster, and more flexible, data solution implementation. Learn the revolutionary approach for a fully automated solution from Roelant Vos.

• Implement a Persistent Staging Area
• Apply hybrid modeling techniques, based on Data Vault
• Define robust patterns for data logistics, suitable for code generation
• Define a metadata model for automation, code generation, and virtualization
• Apply DevOps, testing, orchestration, and control frameworks
• Ensure that the delivered data meets the consumers’ expectations

This practical design and implementation training provides you with everything you need to build and maintain an automated data solution, from start to finish.

Your Trainer

Roelant Vos has been active in Data Warehousing (DWH) and Business Intelligence (BI) for more than 20 years, and is well known as an expert in the Data Vault community.

For more than 10 years, he has been sharing his ideas, tips, and thoughts on his blog roelantvos.com.

Having worked as a software developer, consultant, trainer, and decision maker in the corporate world, Roelant has observed data management from various distinctly different points of view.

The common theme has always been a passion for automation, code generation, reusable patterns, and model-driven design – the key to making data solutions manageable and flexible.

His focus is now on providing training, consultancy, and open-source software development to make the delivery of robust data solutions easier. As part of this, he initiated the Data-Solution-Automation-Engine on GitHub.

You want to ...

• Learn what kind of solution architecture supports flexible data delivery that can evolve with the business
• Fully understand the concepts behind the essential data loading patterns, what options can be considered, and how to implement these
• Leverage generation techniques for data logistics (‘ETL’), to be able to spend more time on more value-adding work such as data modelling and improving data delivery
• Work on a Do-It-Yourself (DIY) data solution framework, or have adopted a Data Warehouse Automation (DWA) product and seek deeper understanding on the used patterns and modelling approaches

• Get a full overview of all components that are necessary for a robust and manageable data solution

  This course covers advanced modelling and implementation techniques, and applies to a wide range of data professionals including Data Warehouse professionals, data modelers, architects, and data engineers.

Prerequisites

• Sufficient understanding of English (the course language is English)
• Understanding of data engineering, for example Data Warehousing and ETL development
• Knowledge of SQL (e.g. joining, window functions)
• Some scripting / programming experience
• Familiarity with data modeling techniques for data warehousing (e.g. Dimensional Modeling, Ensemble Logical Modelling techniques including Data Vault)

Is this for me?

By adopting hybrid / Ensemble Logical Model patterns (e.g. Data Vault) on top of a Persistent Staging Area (PSA) – a historised record of all original transactions – an unparalleled level of flexibility in implementing and maintaining a data solution can be achieved. The repetitive aspects of data preparation are reduced, and it becomes easier to adjust the solution to ever-changing business- and technical requirements.

These patterns are seemingly straightforward – almost deceptively so.

But, in fact, every pattern requires far-reaching considerations at a technical and conceptual level to truly match the business expectations.

Data Vault modeling provides elegant features to manage complexities, but success still depends on correct modeling of the data, and correct application of the patterns. Leveraging data logistics (‘ETL’) generation and virtualization techniques allows for a great degree of flexibility, because you can quickly refactor and test different modelling approaches to understand which one fits best for your use-case.

This enables you to spend more time on more value-adding work such as improving the data models and delivery of data.

This advanced training is relevant for anyone seeking to understand how to leverage ‘model-driven-design’ and ‘pattern-based code-generation’ techniques to accelerate development. The content applies to a wide range of data professionals including Data Warehouse specialists, data modellers and architects as well as data engineers and data integration developers.

Flexible design and implementation

The intent of the training is to cover the architecture and concepts for a flexible data solution, with a focus to ‘deep dive’ into the patterns and practical implementation techniques as quickly as possible.

To facilitate this, the training discusses the implementation of the main Data Vault modeling concepts including their various edge-cases and considerations. The mechanisms to deliver information for consumption by business users (i.e. ‘marts’) will also be covered, including details on how to produce the ‘right’ information by implementing business logic and managing multiple timelines for reporting (‘bitemporal’).

The training provides tools and configurations which you can use to start automating your own development – or understand the approaches used in commercial ‘off-the-shelf’ software so that these can be fully utilized.

Training content and schedule

Day 1

• Pattern-based design
• Data solution architecture
• Data staging concepts
• Modeling concepts
• Introducing design metadata
• Code generation

Day 2

• Core Business Concept pattern
• Natural Business Relationships pattern
• Context pattern & historization
• Control framework
• Testing
• Technical considerations
• Orchestration, workflows, and parallelism
• DevOps and versioning

Day 3

• Temporality concepts
• Data delivery for consumption
• Application of business logic
• Completing the solution

Practical content

The training also provides an opportunity to get ‘hands on’ with some of the frameworks that are necessary to deliver a robust, manageable, and flexible solution.

This is done through short exercises as part of the regular content, during training hours.

These exercises use the Microsoft stack (SQL Server, Windows), but the content, approach and templates apply equally to other environments.

By following the exercises, we will go through:

• Setting up a new data automation environment
• Defining source-to-target mappings
• Generate data logistics code, and
• Run & test the solution in various ways

Needed Software

As part of the practical content, we use the following software:

• A Microsoft SQL client, such as SQL Server Management Studio (SSMS), Azure Data Studio, or equivalent, to access the database
• The open-source Virtual Data Warehouse (VDW) tool for creating metadata mappings
• The open-source TEAM tool for code generation

We’ll configure these tools as part of the workshop.

Training available world-wide

This training is offered globally on a regular basis, and can be planned in-house on request to meet specific company objectives. It is also an option to organise on-line coaching, spread across multiple sessions, following the outline of the training or focusing on specific content. Please have a look on the dates or contact us.