A single version of truth is essential for qualified decision-making. Though especially for larger organisations it is an uneasy goal.
Let’s be open – most organisations we have been to usually approach their reporting in one of those 2 styles:
Quick&Dirty – Reporting directly from your ERP system using some sort of “database connector” and Excel to get unreliable figures quickly. Often there is a vendor who claims that it is sufficient to connect all your systems to their platform to get reports magically quickly, hence with tragically low data quality.
Neverending Story – Paying for consultants to spend a year or two creating a kinda-reliable but costly data lake containing all your sales, marketing and controlling data and then maintaining, redoing and enhancing it with hot-fixes forever.
Neither seems as a good option, right? But how can you get a reliable source of sales, marketing, financial, HR, social, external and other figures combined in a right manner and presented correctly and timely every morning – quickly and cost-efficiently, often starting using them as soon as within 2 or 3 months?
The answer is:
Metadata-driven Data Warehouse
This approach encourages storing incoming data in layers. Each subsequent layer builds on the previous one and allows more and more enriching data transformation, cleaning, validation and business rules application.
An example can be:
L0 – storing data in the same format as in the source system, a 1:1 copy.
L1 – Enabling data consistency checks, cleaning, historization
L2 – Federating Dimensions and Facts for further analysis and reporting, applying specific segmentation and other business rules
L3 – Enriching insights with specific statistical or predictive models
You can argue that building all these layers can take ages – but wait – we have a set of automated procedures that allow to automatically build all layers for every data source and entity like Product, Customer, Sales, P&L, etc. It is enough to know the data source and nature of each column – and you’re set 🙂
You know, we have build tens of data warehouses in the past… And having seen the same issues over and over again – we have built a framework around the data-warehousing that lets us create these complex things much quicker and nicer than the others make them.