Source-linked AI summary
Large-Scale Evaluation of Advanced Imputation Methods for Missing Values in Smart Meter Data
Daniela Stojcheska, Marija Markovska, Dimitar Taskovski, Branislav Gerazov, Boris Nikolov
TL;DR
Missing values from communication outages can distort smart-meter consumption profiles and impair non-technical-loss detection. This paper benchmarks OWA, SoftImpute, and an autoencoder on 17,428 commercial meters across gaps of 1–168 hours, finding OWA most accurate overall.
Problem
Communication outages create missing smart-meter readings, while simple interpolation can distort larger gaps and degrade non-technical-loss detection.
Method
The paper benchmarks three imputation paradigms on 17,428 active commercial smart meters from North Macedonia across simulated gaps lasting 1–168 hours.
Results
OWA is the most accurate global imputation method, maintaining the lowest absolute error even for extended 168-hour gaps.
Takeaways & Limitations
Imputation methods should be selected according to load-curve characteristics, with SoftImpute and shape-based autoencoders serving as fallbacks when temporal context is unavailable.
Takeaways & Limitations
When a meter loses temporal context for an extended period, OWA is insufficient and alternative spatial or shape-based approaches become necessary.
Abstract
from arXiv · showhide
Accurate and reliable collection of electricity consumption data through Advanced Metering Infrastructure (AMI) is of great importance for the operation of smart grids, especially for the detection of non-technical losses (NTL). However, real-world datasets frequently suffer from missing values due to communication failures. This paper presents an empirical evaluation of three advanced algorithms for large-scale data imputation: the Optimally Weighted Average (OWA) method, Low-Rank Matrix Completion via SoftImpute, and a Shape-Modeling Autoencoder. Existing studies on missing value imputation in electricity consumption data often lack validation on larger datasets. Therefore, the goal of this paper is to validate the selected algorithms on a large-scale real-world electricity consumption dataset from North Macedonia that includes 17,428 commercial smart meters over two years. The robustness of each algorithm is evaluated by simulating continuous gaps in the data ranging from 1 to 168 hours. The results indicate that OWA provides the lowest overall reconstruction error across the evaluated gap sizes and strong stability in worst-case scenarios for gaps of up to one week. In contrast, the autoencoder exhibits higher variance, while SoftImpute has stable but inferior accuracy. These findings suggest that imputation methods should be selected based on the characteristics of load curve data and highlight the potential for hybrid algorithmic architectures in future grid management systems.
I. INTRODUCTION
AMI data support smart-grid operations and NTL detection, but communication-related gaps can distort consumption profiles and impair detection. The paper benchmarks three imputation paradigms on a large real-world commercial-meter dataset to address limited scalability and robustness evidence.
- AMI provides high-resolution time series data used for load forecasting, structural grid monitoring, and NTL feature extraction.
- Communication outages range from one-hour breaks to multi-day gaps, and simple interpolation can distort natural consumption profiles.Such distortions can degrade NTL detection, causing false positive theft alarms or masking genuine illegal consumers.
- Existing imputation studies predominantly use localized datasets or synthetic load profiles, leaving large-network scalability and worst-case robustness largely unexplored.
- The paper benchmarks temporal, spatial, and deep-learning imputation paradigms on 17,428 active commercial smart meters from North Macedonia.The evaluation analyzes continuous gap sizes from 1 to 168 hours to establish a practical empirical baseline.
- Robust smart-grid imputation is an active research need because real-world AMI networks are susceptible to hardware malfunctions and communication-related data loss.
A. Raw Data and Aggregation
The study analyzes 25 months of hourly commercial and industrial smart-meter data from North Macedonia after filtering unreliable meters. Missingness is widespread, temporally clustered, and accompanied by substantial load volatility, defining the reconstruction challenge.
- Raw Data and Aggregation: The dataset contains 25 continuous months of nonresidential commercial and industrial readings, aggregated from 15-minute to hourly intervals.It was provided by Elektrodistribucija DOOEL, North Macedonia’s electricity distribution system operator.
- Raw Data and Aggregation: 17,428 active smart meters remained after removing recently installed meters, inactive connections, and meters with outages exceeding one week.
- Missingness Profile and Load Volatility: 0% of meters had a perfect ground-truth record, with individual data losses ranging from 2 to 168 hours.
- Missingness Profile and Load Volatility: 651,652 hours of data were lost across the network over 25 months.
- Missingness Profile and Load Volatility: 10,223 meters had maximum gaps shorter than 6 hours, 5,922 had gaps up to 24 hours, and 1,283 had gaps up to 168 hours.Filtering excluded continuous outages exceeding one week, keeping tasks within the defined recoverable range.
- Missingness Profile and Load Volatility: 56.91% of commercial meters were highly volatile, with CV > 1.0, while missing values clustered at specific hours and occurred most frequently in May and November.Reported hourly spikes occurred at 02:00, 06:00, and 09:00–11:00.
IV. METHODOLOGY OF IMPUTATION ALGORITHMS
The methodology uses a data-driven tuning approach for three imputation algorithms because commercial energy consumption is highly unpredictable.
- The study introduces data-driven parameter tuning rather than relying on standard parameters for the analyzed smart-meter network.
A. Optimally Weighted Average (OWA)
OWA reconstructs missing hourly readings by blending local interpolation with historical patterns, using an exponentially decaying weight to adjust their relative influence by position within the gap.
- OWA blends local sequential context with global historical patterns to reconstruct missing consumption values.
- The historical-average component uses matching time-of-day and day-of-week values within a surrounding ±8-day window.
- The weight w_i follows exponential decay with distance from the nearest valid measurement edge, controlled by positive parameter α.
1) Data-Driven Alpha Optimization:
The study selects OWA’s α separately for each gap length through randomized simulations, then applies those parameters in full-network evaluation.
- Data-Driven Alpha Optimization:: 300 healthy meters received 100 randomly injected continuous outages across 11 gap sizes from 1 to 168 hours.
- Data-Driven Alpha Optimization:: Twelve α candidates from 0.0 to 2.0 were evaluated, including the original study’s α = 0.1081.
- Data-Driven Alpha Optimization:: Millions of evaluations selected the α minimizing Mean Absolute Error (MAE) for each gap size.
- Data-Driven Alpha Optimization:: For micro gaps (G ≤2 hours), the optimal α is 0.00, while all extended outages (G ≥8 hours) stabilize at α = 0.50.
- Data-Driven Alpha Optimization:: The selected α parameters were subsequently used in a full-scale network evaluation.
B. Low-Rank Matrix Completion (SoftImpute)
SoftImpute reconstructs missing smart-meter readings by fitting a low-rank approximation to the network-wide consumption matrix while matching observed data.
- SoftImpute addresses long outages by exploiting spatial correlations across the smart-meter network.
- The network history is represented as matrix X, with meters as rows and hourly timestamps as columns.
- The method matches observed entries while constraining matrix complexity through low rank.
- The objective penalizes nuclear norm and calculates reconstruction error only on healthy data, solved through soft-thresholded SVD iterations.
- The validated dataset was converted into a 17,428 × 18,264 global matrix.
C. Shape Modeling with Autoencoders
The shape-modeling autoencoder separates daily consumption shape from magnitude, then uses calendar features to reconstruct missing 24-hour profiles.
- The approach reconstructs daily profiles by separating normalized shape from total daily consumption.
- Commercial profiles are modeled with calendar-dependent behavior across workdays, Saturdays, Sundays, and national holidays.
- Training used over 13 million healthy 24-hour profiles tagged with Day of the Year and Type of Day.
- The encoder represented each daily shape with two latent numbers, while regressions predicted those variables and total daily energy from calendar features.
- For extended outages, predicted latent variables pass through the decoder, and the reconstructed normalized shape is scaled by predicted daily energy.
- Evaluation used millions of simulated gaps distributed across the 17,428-meter network.
A. Missing Data Simulation
The evaluation simulates continuous missing-data gaps across active meters and measures reconstruction accuracy globally and at the meter level. WAPE is used alongside MAE, RMSE, and R2 to handle near-zero consumption without MAPE’s division-by-zero instability.
- Gap simulation: 11 continuous gap lengths from 1 to 168 hours were artificially injected into healthy periods using five independent Monte Carlo iterations.Each iteration introduced one randomly placed gap of every size to all 17,428 active meters.
- Evaluation metrics: MAE, RMSE, and R2 were calculated globally and for each meter to assess accuracy, consistency, stability, and worst-case reliability.
- Evaluation metrics: WAPE was implemented because near-zero nighttime consumption makes standard MAPE unstable through division by zero.
- Evaluation metrics: WAPE compares hidden actual consumption with imputed values relative to total actual consumption across evaluated data points.
VI. RESULTS AND DISCUSSION
Global results establish OWA as the most accurate method across the tested gap sizes, while SoftImpute provides a stable secondary approach and the Autoencoder has higher overall error but gap-size-insensitive performance.
- Global comparison: OWA is the most accurate global imputation method across the 11 tested gap sizes and the MAE, RMSE, WAPE, and R2 metrics.
- Global comparison: For outages up to 24 hours, OWA achieves exceptionally low error and high R2 by interpolating from immediate neighboring data.
- Global comparison: 168-hour gaps still produce the lowest absolute error for OWA, which uses historical data for extended outages.
- Global comparison: SoftImpute maintains bounded error rates by leveraging spatial correlations from neighboring meters.
- Global comparison: The Autoencoder shows the greatest overall error but a completely flat error curve because it does not rely on adjacent hourly measurements.
B. Meter-Level Stability and Outliers
Meter-level distributions confirm OWA’s reliable performance for most consumers, while SoftImpute has moderate variance and the Autoencoder has higher, more dispersed errors despite being unaffected by gap size.
- Meter-level stability: OWA maintains the lowest median error and tightest interquartile range across all gap sizes at the individual-meter level.The boxplots indicate reliable reconstruction for most consumers.
- Meter-level stability: SoftImpute shows moderate variance across meter-level error distributions.
- Meter-level stability: The Autoencoder has higher median error and wider spread, especially for meters with highly specific behaviors, while remaining unaffected by gap size.
- Data characteristics: Commercial profiles are highly specific and volatile, motivating specialized imputation techniques for large-scale smart-meter data.
- Practical implications: OWA uses past and future anchors to maintain exceptionally low error for outages lasting up to a full week.
- Practical implications: When temporal context is unavailable for extended periods, SoftImpute and Autoencoders can serve as fallback mechanisms.
- Future work: Future work proposes an adaptive hybrid framework that routes gaps to algorithms according to duration and context.