PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-Scale Evaluation of Advanced Imputation Methods for Missing Values in Smart Meter Data

Daniela Stojcheska, Marija Markovska, Dimitar Taskovski, Branislav Gerazov, Boris Nikolov

arXiv:2608.21638v1cs.LG

TL;DR

Missing values from communication outages can distort smart-meter consumption profiles and impair non-technical-loss detection. This paper benchmarks OWA, SoftImpute, and an autoencoder on 17,428 commercial meters across gaps of 1–168 hours, finding OWA most accurate overall.

  • Problem

    Communication outages create missing smart-meter readings, while simple interpolation can distort larger gaps and degrade non-technical-loss detection.

    Source: §I · p. 1

  • Method

    The paper benchmarks three imputation paradigms on 17,428 active commercial smart meters from North Macedonia across simulated gaps lasting 1–168 hours.

    Source: §I · p. 1, §VII · p. 6

  • Results

    OWA is the most accurate global imputation method, maintaining the lowest absolute error even for extended 168-hour gaps.

    Source: A. Global Network Performance · p. 5

  • Takeaways & Limitations

    Imputation methods should be selected according to load-curve characteristics, with SoftImpute and shape-based autoencoders serving as fallbacks when temporal context is unavailable.

    Source: §VII · p. 6

    When a meter loses temporal context for an extended period, OWA is insufficient and alternative spatial or shape-based approaches become necessary.

    Source: §VII · p. 6

Abstract

from arXiv · show

Accurate and reliable collection of electricity consumption data through Advanced Metering Infrastructure (AMI) is of great importance for the operation of smart grids, especially for the detection of non-technical losses (NTL). However, real-world datasets frequently suffer from missing values due to communication failures. This paper presents an empirical evaluation of three advanced algorithms for large-scale data imputation: the Optimally Weighted Average (OWA) method, Low-Rank Matrix Completion via SoftImpute, and a Shape-Modeling Autoencoder. Existing studies on missing value imputation in electricity consumption data often lack validation on larger datasets. Therefore, the goal of this paper is to validate the selected algorithms on a large-scale real-world electricity consumption dataset from North Macedonia that includes 17,428 commercial smart meters over two years. The robustness of each algorithm is evaluated by simulating continuous gaps in the data ranging from 1 to 168 hours. The results indicate that OWA provides the lowest overall reconstruction error across the evaluated gap sizes and strong stability in worst-case scenarios for gaps of up to one week. In contrast, the autoencoder exhibits higher variance, while SoftImpute has stable but inferior accuracy. These findings suggest that imputation methods should be selected based on the characteristics of load curve data and highlight the potential for hybrid algorithmic architectures in future grid management systems.

I. INTRODUCTION

AMI data support smart-grid operations and NTL detection, but communication-related gaps can distort consumption profiles and impair detection. The paper benchmarks three imputation paradigms on a large real-world commercial-meter dataset to address limited scalability and robustness evidence.

  • AMI provides high-resolution time series data used for load forecasting, structural grid monitoring, and NTL feature extraction.Source: §I · p. 1
  • Communication outages range from one-hour breaks to multi-day gaps, and simple interpolation can distort natural consumption profiles.Such distortions can degrade NTL detection, causing false positive theft alarms or masking genuine illegal consumers.Source: §I · p. 1
  • Existing imputation studies predominantly use localized datasets or synthetic load profiles, leaving large-network scalability and worst-case robustness largely unexplored.Source: §I · p. 1
  • The paper benchmarks temporal, spatial, and deep-learning imputation paradigms on 17,428 active commercial smart meters from North Macedonia.The evaluation analyzes continuous gap sizes from 1 to 168 hours to establish a practical empirical baseline.Source: §I · p. 1, §II · p. 2
  • Robust smart-grid imputation is an active research need because real-world AMI networks are susceptible to hardware malfunctions and communication-related data loss.Source: §II · p. 1

A. Raw Data and Aggregation

The study analyzes 25 months of hourly commercial and industrial smart-meter data from North Macedonia after filtering unreliable meters. Missingness is widespread, temporally clustered, and accompanied by substantial load volatility, defining the reconstruction challenge.

Raw Data and Aggregation

  • The dataset contains 25 continuous months of nonresidential commercial and industrial readings, aggregated from 15-minute to hourly intervals.It was provided by Elektrodistribucija DOOEL, North Macedonia’s electricity distribution system operator.Source: A. Raw Data and Aggregation · p. 2
  • 17,428 active smart meters remained after removing recently installed meters, inactive connections, and meters with outages exceeding one week.Source: B. Data Cleansing and Filtering · p. 2

Missingness Profile and Load Volatility

IV. METHODOLOGY OF IMPUTATION ALGORITHMS

The methodology uses a data-driven tuning approach for three imputation algorithms because commercial energy consumption is highly unpredictable.

  • The study introduces data-driven parameter tuning rather than relying on standard parameters for the analyzed smart-meter network.Source: §IV · p. 3

A. Optimally Weighted Average (OWA)

OWA reconstructs missing hourly readings by blending local interpolation with historical patterns, using an exponentially decaying weight to adjust their relative influence by position within the gap.

1) Data-Driven Alpha Optimization:

The study selects OWA’s α separately for each gap length through randomized simulations, then applies those parameters in full-network evaluation.

Data-Driven Alpha Optimization:

B. Low-Rank Matrix Completion (SoftImpute)

SoftImpute reconstructs missing smart-meter readings by fitting a low-rank approximation to the network-wide consumption matrix while matching observed data.

C. Shape Modeling with Autoencoders

The shape-modeling autoencoder separates daily consumption shape from magnitude, then uses calendar features to reconstruct missing 24-hour profiles.

  • The approach reconstructs daily profiles by separating normalized shape from total daily consumption.Source: C. Shape Modeling with Autoencoders · p. 4
  • Commercial profiles are modeled with calendar-dependent behavior across workdays, Saturdays, Sundays, and national holidays.Source: C. Shape Modeling with Autoencoders · p. 4
  • Training used over 13 million healthy 24-hour profiles tagged with Day of the Year and Type of Day.Source: C. Shape Modeling with Autoencoders · p. 4
  • The encoder represented each daily shape with two latent numbers, while regressions predicted those variables and total daily energy from calendar features.Source: §V · p. 4
  • For extended outages, predicted latent variables pass through the decoder, and the reconstructed normalized shape is scaled by predicted daily energy.Source: §V · p. 5
  • Evaluation used millions of simulated gaps distributed across the 17,428-meter network.Source: §V · p. 5

A. Missing Data Simulation

The evaluation simulates continuous missing-data gaps across active meters and measures reconstruction accuracy globally and at the meter level. WAPE is used alongside MAE, RMSE, and R2 to handle near-zero consumption without MAPE’s division-by-zero instability.

Gap simulation

  • 11 continuous gap lengths from 1 to 168 hours were artificially injected into healthy periods using five independent Monte Carlo iterations.Each iteration introduced one randomly placed gap of every size to all 17,428 active meters.Source: A. Missing Data Simulation · p. 5

Evaluation metrics

  • MAE, RMSE, and R2 were calculated globally and for each meter to assess accuracy, consistency, stability, and worst-case reliability.Source: B. Evaluation Metrics · p. 5
  • WAPE was implemented because near-zero nighttime consumption makes standard MAPE unstable through division by zero.Source: B. Evaluation Metrics · p. 5
  • WAPE compares hidden actual consumption with imputed values relative to total actual consumption across evaluated data points.Source: B. Evaluation Metrics · p. 5

VI. RESULTS AND DISCUSSION

Global results establish OWA as the most accurate method across the tested gap sizes, while SoftImpute provides a stable secondary approach and the Autoencoder has higher overall error but gap-size-insensitive performance.

Global comparison

B. Meter-Level Stability and Outliers

Meter-level distributions confirm OWA’s reliable performance for most consumers, while SoftImpute has moderate variance and the Autoencoder has higher, more dispersed errors despite being unaffected by gap size.

Meter-level stability

Data characteristics

  • Commercial profiles are highly specific and volatile, motivating specialized imputation techniques for large-scale smart-meter data.Source: §VII · p. 6

Practical implications

  • OWA uses past and future anchors to maintain exceptionally low error for outages lasting up to a full week.Source: §VII · p. 6
  • When temporal context is unavailable for extended periods, SoftImpute and Autoencoders can serve as fallback mechanisms.Source: §VII · p. 6

Future work

  • Future work proposes an adaptive hybrid framework that routes gaps to algorithms according to duration and context.Source: §VII · p. 6
Loading 2608.21638v1…