Initial Situation
A major German municipal utility company intends to set up a central data platform and establish clearer data governance structures as part of a group-wide programme to improve technical and process-related data processing. The medium-term goal is to standardise and link decentralised data pools and replace Excel-based end-user applications (EUAs). This will create a sound basis for future data analyses, including the use of AI methods.
Even before the final data platform is launched, existing concepts for the data model and data governance are to be trialled and further developed in practice as part of a flagship project for market data in energy trading and risk management.
Project Scope
- Providing a database and application server for prototype implementation
- Connecting external market data sources
- Developing a data model compliant with the programme’s technical specifications (Data Vault 2.0)
- Implementing automated quality assurance routines
- Designing and creating initial market data reports in line with the programme’s technical specifications (Power BI)
Our Contribution
- Implementation of a daily import of EEX prices via Rest API and PFC files
- Design and script-based implementation of a Data Vault 2.0-compliant data model for market data
- Supplementing the data architecture with components for documenting data governance structures, e.g. data ownership
- Automation of existing quality checks and extension by additional check routines
- Transfer of existing market data reports to Power BI and publication via web application
- Conceptual proposal and prototype implementation of automated documentation of data flows, data model and data governance information in Confluence
Customer Benefit
The prototype implementation of the data processing was able to confirm the viability of the concepts developed in the programme and develop them further in a meaningful way. The increased level of automation not only accelerated processes but also made them auditable. Moreover, code-based documentation ensures continues correctness and actuality. Additional improvements in data governance resulted from the clear allocation of data ownership. Finally, the script-based implementation allows the transfer to the final data architecture with minimal effort, without losing the results and technical improvements achieved.
Relevant Skills/Tools
- databricks
- Power BI
- Data Vault 2.0
- Python
- Git
- Confluence
- Azure DevOps
