Integrated Energy Data & MLOps Platform

One platform for a merged energy network: unified data, MLOps and smart grid monitoring

After the merger of two major energy utilities, MADIS designed and built a unified Azure data and MLOps platform – and, in a related project, a smart transformer monitoring platform for proactive network management.

At a glance

8

legacy systems replaced by one platform

10

data sources integrated

20,000

measuring devices planned for smart grid monitoring

Summary: Following the merger of two major energy utilities, decision-support data from both companies had to be brought together. MADIS built a cloud data platform on Azure that replaces eight earlier systems, integrates ten data sources and lays the groundwork for machine learning – complemented by an IoT platform that monitors transformers in real time and raises alerts before critical situations develop.

The client

The client operates electricity distribution networks. After its merger with another major energy utility, the decision-support data of the two distribution system operators (DSOs) needed to live in one place to support strategic decisions, regulatory reporting and the day-to-day management of the network.

The challenge

The merged organisation needed an integrated data platform that connects the data of the DSOs and meets regulatory requirements. Existing reporting relied on local calculations spread across several systems, and confidential information had to be handled according to strict rules.

At the same time, the network itself had to be watched more closely. Stable energy transmission and good network quality depend on spotting deviations, such as voltage limit violations on transformers, before they become critical.

• Fragmented landscape: 8 previously used systems and 10 different data sources, including Oracle, MS SQL, PostgreSQL and SAP.
• Isolated networks: data had to be collected from several separated network segments.
• Regulation and confidentiality: strict requirements for compliance and data protection.
• Real-time network data: quarter-hourly measurements and continuous sensor data streams.

Our approach

The work covered two connected streams: a central data and MLOps platform, and a smart IoT platform for transformer monitoring.

Data lake and governance foundation

We built an Azure Databricks-based data lake for storing the data and integrated it with the Erwin data governance system to provide data quality, compliance and security processes.

Integrating heterogeneous sources

Spark-based pipelines load the data from Oracle, MS SQL, PostgreSQL and SAP-based systems. Azure Data Factory handles data transfer, with integration gateways deployed in each separated network segment. Stream processing handles the quarter-hourly metering data, and error handling covers the entire process.

Analytics, data marts and machine learning

On top of the data lake we built a Synapse-based data platform, with data marts designed around the client’s needs, a permission system and error handling for loading the data marts. We also laid the foundations for machine learning-based reports and analyses, and provided training from data analysis to MLOps practices.

Smart transformer monitoring

IoT devices placed at both the feed-in and the end points of low and medium voltage transformers measure voltage, current, voltage drop and other physical quantities. The platform processes the measurement and alarm data with Databricks and PySpark, stores it in an Azure data lakehouse and displays the values on a map, raising alerts when they leave a predefined range.

The solution

The result is a strong, centralised, cloud-based data platform that forms the backbone of the existing reporting system, replacing local calculations with a central cloud environment. It enables data analysis, reporting, data mining and AI model development in one scalable place.

The smart transformer platform complements it with real-time monitoring of calculated indicators and alarms across the distribution network, designed to manage 20,000 planned measuring devices.

Technologies: Azure Data Lake · Azure Databricks · Azure Synapse Analytics · Azure Data Factory · Azure Machine Learning · Spark · Erwin · Power BI · Azure IoT Hub · Event Hub · Azure Data Explorer

Results

• One unified system: data sources of multiple DSOs connected in one platform, replacing 8 earlier systems.
• Faster, more stable analytics: a more stable and faster environment for analysis and reporting.
• Compliance built in: confidential information handled according to strict regulatory requirements, backed by data governance.
• Acting before failures: continuous monitoring of the network makes it possible to act before a critical situation develops, instead of repairing after the fact.

Facing a similar challenge?

Bringing together data after a merger or modernising your grid analytics? Talk to our data and AI team.