Harnessing the Power of Smart Meter Data: Using Big Data to Make Better Network Decisions
Data is fast becoming the most valuable asset in an EDB's toolkit, with data-driven strategies offering a smarter approach to network resilience than traditional capital investments.
As smart meter data becomes more readily available, Electricity Distribution Businesses (EDBs) are eager to harness its potential. However, transforming that raw data into actionable insights requires smart solutions, as the team at Northpower found during their trial with Hiko Energy Insights.
As more New Zealanders adopt technologies like EVs and solar, creating two-way power flows, utilities and energy providers need to adapt to meet new consumer demands. LV Network visibility has become critical for effective management, with smart meter data and network topology playing pivotal roles.
Partnering with a team experienced in handling big data means your engineers and planners can focus on managing their networks.
The Big Data Challenge
With smart meters collecting data every 5 minutes, EDBs face an unprecedented volume of information. For a utility with 100,000 customers, this translates to billions of data points annually. Processing this data efficiently and securely requires expertise in big data technologies and advanced analytics.
Calvin Varney, Technical Lead at Hiko, is familiar with these challenges. “We utilise AWS big data services to process data as it lands — within a few minutes. When the need arises, we can reprocess 10 billion records in 30 minutes.”
EDBs with existing in-house data capability tools can also benefit from working with external data partners. “Integrating with existing systems caters for many users across different teams, who may have different levels of data capability and needs,” says Varney.
One EDB that fits this scenario is Northpower, who have strong data analytic capability within their team and have invested in the Snowflake data platform. This enables them to ‘self-serve’ using the enhanced LV data that Hiko provided during their trial.
The Northpower team with Hiko Product Manager Natasha Gillison.
The Data Solution for Northpower
The team at Northpower had recently acquired data from Bluecurrent for 40,000 ICPs, about two-thirds of their network.
The sheer volume of this data presented a challenge — just under 30 million records a day. Over the course of the trial, Hiko was able to store and analyse one year of data, or about 10 billion meter read records.
Northpower’s engineers also quickly realised that there were outliers and other data quality issues to solve. “Hiko was able to remove the bad data so that we had a better quality dataset for analysis,” says Andrew Camuso, Network Development Manager at Northpower. “Filtering that out before loading it back onto our platform means our engineers can quickly look at the data and understand the state of our network.”
Varney says the solution to this challenge actually went further than the data. "Guy [Marquet, Hiko Chief Executive] has a wealth of knowledge around metering, and was able to draw on this industry experience to help identify and explain outlier meter readings." The team was then able to cleanse the data before delivering it to Northpower for analysis.
Empowering Better Outcomes for Northpower Customers
Northpower is now able to conduct sophisticated future scenario planning based on granular, real-world data. The accuracy of their load flow modeling has also led to more efficient network operations.
Northpower uses data insights to enhance their distribution management system. The team can see which customers would be affected and use accurate data to even reduce the number of customers affected during planned outages. If faults occur, they can now calculate and choose different outcomes for bringing the maximum number of customers back online without overloading the network.
Northpower also uses the Hiko platform to prevent faults. The platform highlights areas for engineers to investigate, so they can identify if parts of the network are degrading.
“This means we can go out and fix it before a fault happens. That makes our networks safer, and it means fewer outages for customers,” says Camuso.
Informed Planning for Future Growth
"We've been able to use this data to identify some areas for investment. There were certain studies we couldn't do before, without the LV data. It's unlocked more opportunities to improve our network,” says Camuso.
“It’s also had ripple effects across different teams,” says Mick Buckley, Northpower’s Systems Engineer for Future Networks.
“The Customer Connections Team has been able to use it for power quality investigations, and the Network Development team has used it for voltage and load planning.
“Other use cases include outage planning, improved fault dispatch, proactive fault finding and balancing phase loads, as well as congestion management on the High Voltage network,” says Buckley.
Empowering New Zealand EDBs to Focus on Core Operations
For Hiko, innovation in this industry goes beyond data processing. Product Manager Natasha Gillison says: “By leveraging advanced data analytics, EDBs can unlock hidden network capacity, reducing the need for costly infrastructure investments. This not only leads to lower costs for customers but also drives improvements in efficiency, quality, and safety across their Network.”
“Most importantly, they’re empowered to focus on their core business — managing and creating a more resilient network — rather than working through the complexities of big data processing.
There are other benefits to a data partnership. “Because Hiko are New Zealand based, they understand the challenges and regulations specific to New Zealand,” says Northpower’s Buckley. “They've been working with smart meters for a long time, so they have a track record.”
“They were also extremely good at adapting. The team was keen to provide benefits with our data, and they adapted really fast.”
Data Protection and Security
As the energy sector continues to evolve, the ability to process and analyse granular data securely will become increasingly important. Data providers will need to meet the changing needs of the industry, while maintaining high standards of data protection.
“ISO 27001 accreditation was a priority for us — it’s important that we manage the customer’s data appropriately and protect it,” says Varney, Technical Lead at Hiko.
About Hiko Energy Insights
Hiko Energy Insights has been collaborating with New Zealand EDBs to unlock the power of their smart meter data for over 8 years.
Hiko offers scalable data processing capabilities, handling over 250TB of meter reads daily while their ISO 27001 accreditation ensures the highest levels of data security.
There’s no need to invest in IT infrastructure either – Hiko will store and process the data, which EDBs have full access to for as long as they need it. The team uses Amazon Web Services (AWS) for powerful, secure data processing that is scalable and robust.
Every network is unique, from infrastructure to data management practices. Hiko’s team works closely with each partner to provide flexible, tailored solutions that meet specific needs.