vdwh-logo

Data Warehouse Design Patterns

Implementation and Automation

Workshop with Roelant Vos


Register now!


"For a data warehouse, we do not have enough time."

… sounds familiar to you?

DWH-Automation enables faster delivery using agile approaches for DWH implementation. Learn the revolutionary concept of an Automated Enterprise Data Warehouse from Roelant Vos.

  • Implement a Solid Persistent Data Store.
  • Leverage proven hybrid Data Warehouse modelling techniques and patterns based on Data Vault.
  • Build a Proven Meta Data Model for process automation and virtualization.

This practical design and implementation training will discuss the techniques and patterns in great detail. It provides you with everything you need to implement an Automated Data Warehouse Solution from start to finish by choosing the right patterns.


What does Data Warehousing Automation mean?


The idea of an automated virtual Data Warehouse was conceived as a result of working on improvements for generation of Data Warehouse loading processes. It is, in a way, an evolution in ETL generation thinking. Combining Data Vault with a Persistent Historical Data Store provides additional functionality because it allows the designer to refactor parts of the Data Warehouse solution. Hybrid approaches for Data Warehousing are designed to be flexible, to be adaptable to accommodate changes in business use and interpretation. Working with data can be complex, and often the ‘right’ answer for the purpose is the result of a series of iterations where business Subject Matter Experts and data professionals collaborate.

In other words, the Data Warehouse model itself is not always something you always can get right in one go. In fact, it can take a long time for a Data Warehouse model to stabilise, and in the current fast-paced environments this may even never be the case. Choosing the right design patterns for your Data Warehouse helps maintain both the mindset and capability for a data solution to keep evolving with the business, and to reduce technical debt on an ongoing basis. This mindset also enables some truly fascinating opportunities such as the ability to maintain version control of the data model, the metadata and their relationship - to be able to represent the entire Data Warehouse as it was at a certain point in time - or to even allow different Data Models for different business domains.

To allow ideas to grow, creators need an immediate connection to what they are creating. This means that, as a creator, you need to be able to directly see what the effect of your changes are on what you are working on.

This is what the Virtual Data Warehouse as a concept and mindset intends to enable: to enable a direct connection to data to support any kind of exploration and enabling creativity while using it. Thinking of Data Warehousing in terms of virtualisation is in essence about following the guiding principle to establish a direct connection to data. It is about finding ways to seek simplification, to keep working on removing barriers to deliver data and information. It is about enabling ideas to flourish because data can be made available for any kind of discovery or assertion.

Virtual Data Warehousing is the ability to present data for consumption directly from a raw data store by leveraging data warehouse loading patterns, information models and architecture. In many Data Warehouse solutions, it is already considered a best practice to be able to ‘virtualise’ Data Marts in a similar way. The Virtual Data Warehouse takes this approach one step further by allowing the entire Data Warehouse to be refactored based on the raw transactions.

This ability requires a Persistent Historical Data Store, also known as a Persistent Staging Area where the data that is received is stored as it has been received, at the lowest level. If data is retained this way, everything you do with your data can always be repeated at any time – deterministically. In the best implementations, the Virtual Data Warehouse allows you to work at the level of simple metadata mappings, modelling and interpretation "business logic", abstracting away the more technical details.

A Virtual Data Warehouse is not the same as Data virtualisation. These two concepts are fundamentally different. Data virtualisation, by most definitions, is the provision of unified direct access to data across many "disparate" data stores. It is a way to access and combine data without having to physically move the data across environments. Data virtualisation does not however focus on loading patterns and data architecture and modelling.

The Virtual Data Warehouse on the other hand is a flexible and manageable approach towards solving data integration and time variance topics using Data Warehouse concepts, essentially providing a defined schema-on-read.

The Virtual Data Warehouse is enabled by virtue of combining the principles of ETL generation, hybrid data warehouse modelling concepts and a Persistent Historical Data Store. It is a way to create a more direct connection to the data because changes made in the metadata and models can be immediately represented in the information delivery. Persisting of data in a more traditional Data Warehouse sense is always still an option, and may be required to deliver the intended performance. The deterministic nature of a Virtual Data Warehouse allows for dynamic switching between physical and virtual structured, depending on the requirements.

In many cases, this mix of physical and virtual objects in the Data Warehouses changes over time itself, when business focus changes. A good approach is to ‘start virtual’, and persist where required.


Download PDF

Your Trainer

Roelant Vos has been active in Data Warehousing and BI for more than 20 years and is well known as experienced expert in the Data Vault community. Whenever there is some time, he shares his ideas and thoughts on his blog roelantvos.com.

Roelant is General Manager - Enterprise Data Management at Allianz Worldwide Partners in Brisbane, Australia. In a role that is highly focused on analytics, he is working on collecting, integrating, improving and interpreting data to support various business improvement initiatives. Passionate about improving quality and speed of delivery through model-driven design and development automation, he has been at the forefront of contemporary modelling and development techniques for many years.

"The training is great and worth every cent. I can only recommend it to everybody!"
BERNHARD LAUBER, trivadis

"Thanks Roelant, I really enjoyed the course. So much content and lots of interesting discussions."

ANDREAS HAAS

You want to ...

  • Deeply understand the concepts behind data loading patterns and how to implement them.
  • Leverage ETL generation techniques and spend more time on higher value-adding work such as improving the delivery of your data.
  • Work on a Do-It-Yourself (DIY) solution or have adopted any of the available Data Warehouse Automation (DWA) platforms and seek understanding how these use the patterns and modelling approaches.

As advanced modelling and implementation techniques are also covered, this applies to a wide range of data professionals including BI and Data Warehouse professionals, data modelers and architects as well as DBAs and ETL specialists.

Prerequisites

  • Sufficient knowledge of English - Course language is English
  • Understanding of Data Warehouse and ETL development
  • Knowledge of SQL
  • Some scripting / programming experience
    Develop a data solution that can evolve with the business
  • Data Vault Modeling Experience

Optional hands-on sessions only: pre-installed environment with SQL Server 2012, 2014 or 2016, Integration Services and Visual Studio with SQL Server Data Tools.

Is this for me?

By adopting the Data Vault patterns on top of a Persistent Historical Data Store, we can reduce the repetitive aspects of data preparation and maintain consistency in development. These patterns are seemingly straightforward – almost deceptively so. In fact, every pattern needs far-reaching considerations to evaluate both at a technical and conceptual level to truly match the business expectations.

The Data Vault Modelling provides elegant handles to manage complexities, but success depends on correct modelling of the information. Ultimately, leveraging ETL generation and virtualisation techniques allows for a great degree of flexibility because you can quickly refactor and test different modelling approaches to understand which one fits best for your use-case. This enables you to spend more time on higher value-adding work such as improving the data models and delivery of data.

This advanced training is relevant for anyone seeking to understand how to leverage ‘model-driven-design’ and ‘pattern-based code-generation’ techniques to accelerate development. As advanced modelling and implementation techniques are also covered, this applies to a wide range of data professionals including BI and Data Warehouse professionals, data modellers and architects as well as DBAs and ETL specialists.

Flexible design and implementation

The intent of the training is to achieve implementation and advanced techniques as quickly as possible. The training discusses the implementation of the main Data Vault modelling concepts including their various edge-cases and considerations. The mechanisms to deliver information for consumption by business users (i.e. ‘marts’) will also be covered, including details on how to produce the ‘right’ information by implementing business logic and managing multiple timelines for reporting.

The training provides tools and configurations which you can adopt to get started automating your own development – or understand the approaches used in commercial ‘off-the-shelf’ software to be able to fully utilise these.

Training content and schedule

Day 1

  • Model Driven Design overview
  • Solution Design & Architecture
  • Solution pre-requisites and components. What needs to be in place? What concepts should be supported?
  • Data Staging concepts, implementation and approaches
  • Collaborative group modelling workshop
  • Overview of loading patterns and their metadata requirements
  • In-depth Hub pattern considerations and implementation approach (key distribution)

Day 2

  • In-depth Link pattern considerations and implementation approach (relationships)
  • In-depth Satellite & Link-Satellite pattern considerations and implementation approach (handling time-variant data)
  • Technical considerations (indexing, partitioning, joining)
  • Managing scheduling, workflows and parallelism
  • Exception handling
  • Historisation and continuous loading

Day 3

  • Base tables and Derived tables
  • Application of business logic
  • Helper constructs (PIT, Bridge)
  • Dimensions and Facts
  • Flexibility in development (scale-up and scale-out)


Overview PDF

Workshop in Australia

This workshop will be offered by Roelant Vos in Australia. More information will follow soon.

Dates & Prices

Zurich
  • 18th to 20th February 2020
  • Stampfenbachstraße 6
    8001 Zurich (Swiss)
  • 5 min walk from Zurich main station
  • CHF 3.015,60 (incl. VAT)
net cost 2.800 CHF
Inhouse
net cost ???? €

Registration





IMPRINT

×

Dörffler & Partner GmbH
In den Rotwiesen 20
69242 Mühlhausen (Kraichgau)
GERMANY

Tel.: +49 6222 661820
Fax: +49 6222 661822

E-Mail: info@doerffler.com
European VAT-ID: DE111625250
Tax Number: 32492/08960

Company Headquarters: Mühlhausen (Kraichgau), Deutschland
Registration: Amtsgericht Mannheim; HRB Nr. 351759
CEO: Christian Hädrich, Timo Cirkel

GENERAL INFORMATION

Area of Responsibility

This imprint is only valid for content in the domain: www.virtualdwh.com

Responsible for Content and Execution of the Workshop: Roelant Vos

Responsible for Organisation and Accounting: DCYSIVE

Definition of Scope

The website under this domain is part of the WWW and thus linked with other websites, which are subject to change over time. Hence these other websites are not in the area of the responsibility of Dörffler & Partner, and the subsequent information does not apply. All links included on this website have been checked carefully once at the time of insertion whether they violate the law or offend against common decency.

No Liability

This website has been built to the best of our knowledge and its content has been checked carefully. Although no claim is made that the information provided is complete, up-to-date, qualitative and correct. No responsibility can be taken for any damage that is caused by the confidence in the content of this website or its use.

PRIVACY POLICY

Who is responsible for data processing?

Data processing on this website is carried out in compliance with the GDPR by the website operator mentioned above.

Which data is collected, for what and by whom?

Web server

As soon as a user opens the website, our hosting provider Strato collects various technical information, e.g. which browser or operating system is used. In addition, the IP address of the user is recorded. Strato stores these for seven days for its own usage analyses and anonymizes them after this period. We as website operators can only access log files of the web server with anonymized IP addresses. We have concluded a GDPR-compliant contract with Strato for data processing. For more information, please visit the Strato website.

Usage statistics

We use the web analysis service Google Analytics to optimize our website. Google Analytics uses cookies, whose generated information is usually transferred to a Google server in the USA. We have signed a GDPR-compliant contract with Google for data processing.

We use Google Analytics only with IP anonymization enabled. This means that Google shortens the user's IP address and also deletes it after 14 months.

If you do not agree to the collection of data by Google Analytics, you can prevent this function via the following link: Google Analytics Opt-Out. Further information can be found on the Google Analytics website.

Enrollment form

If the user registers in our registration form, we collect further personal data:

  • Name
  • Email address
  • Billing address

The form sends an e-mail with the data, which is then stored within our Microsoft Sharepoint application. We have signed a GDPR-compliant contract with Microsoft for order processing.

We use the data to organise, carry out and settle the Data Warehouse Design Pattern workshop. After completion of the workshop, all data no longer required will be deleted.

What rights does the user have?

The user has the right to receive information about the data collected within one month. He may request a correction, limit the processing or revoke his permission to process the data in full. At the user's request, we are obliged to delete all data about him. If you wish to exercise any of these rights, please contact us by e-mail: info@doerffler.com.

IMAGE SOURCE

All other images belong to Roelant Vos.

If you have any further questions, please contact us:

info@dwhpatterns.com


Copyright: Roelant Vos

Imprint | Privacy Policy | Image Sources