PaperlayerView exact revision on arXiv

Source-linked AI summary

Review of Smart Meter Data Analytics: Applications, Methodologies, and Challenges

Yi Wang, Qixin Chen, Tao Hong, Chongqing Kang

arXiv:1802.04117v2cs.CY

TL;DR

The paper addresses how massive smart meter datasets can support more efficient and sustainable power systems amid expanding smart-meter deployment and power-sector deregulation. It conducts an application-oriented review organized by descriptive, predictive, and prescriptive analytics, covering load analysis, forecasting, and management. It synthesizes methodologies and identifies future directions involving big data, machine learning, business models, energy-system transition, and privacy and security.

  • Problem

    The paper examines how massive smart meter datasets can be used to improve power-grid efficiency and sustainability while privacy, data-scale, and data-fusion issues remain open.

  • Method

    The paper conducts an application-oriented review organized around descriptive, predictive, and prescriptive analytics and their smart meter applications.

  • Results

    The review covers load analysis, load forecasting, abnormal detection, consumer segmentation, and demand response, and summarizes methodologies and recent developments.

  • Takeaways & Limitations

    The paper identifies big data, machine learning, business models, energy-system transition, and data privacy and security as future research directions.

Abstract

from arXiv · show

The widespread popularity of smart meters enables an immense amount of fine-grained electricity consumption data to be collected. Meanwhile, the deregulation of the power industry, particularly on the delivery side, has continuously been moving forward worldwide. How to employ massive smart meter data to promote and enhance the efficiency and sustainability of the power grid is a pressing issue. To date, substantial works have been conducted on smart meter data analytics. To provide a comprehensive overview of the current research and to identify challenges for future research, this paper conducts an application-oriented review of smart meter data analytics. Following the three stages of analytics, namely, descriptive, predictive and prescriptive analytics, we identify the key application areas as load analysis, load forecasting, and load management. We also review the techniques and methodologies adopted or developed to address each application. In addition, we also discuss some research trends, such as big data issues, novel machine learning technologies, new business models, the transition of energy systems, and data privacy and security.

I. INTRODUCTION

Smart meter deployment and power-sector deregulation have made smart meter data analytics important for improving demand-side efficiency and sustainability. This review organizes the literature, analyzes publication trends, and identifies its scope and novelty.

  • Smart meters, communication networks, and data-management systems form AMI, which records load profiles and enables bidirectional information flow.
  • Deregulation is increasing participation by retailers, consumers, and aggregators, making smart meter data important for demand-side efficiency and sustainability.
  • Analytics is organized into descriptive, predictive, and prescriptive stages addressing what data show, what will happen, and what decisions can be made.
  • Bibliometric overview: A WoS search identified 200 publications from 2010 to 2017, with publication counts rising rapidly from 2012 and reaching 60 in 2017.
  • Bibliometric overview: IEEE Transactions on Smart Grid published 28 relevant papers since 2012, making it the most popular journal in the review.
  • Review scope: The review emphasizes recent research, presents a broader and more current overview than earlier reviews, and discusses future research directions.

C. Open Load Datasets

Publicly available load datasets support smart meter analytics across household, building, substation, weather, tariff, survey, and photovoltaic-generation settings. The collection spans different temporal granularities, geographic scales, and data modalities.

  • Public datasets are available because privacy and security concerns limit utilities’ willingness to release smart meter data.
  • Residential datasets: Customer Behavior Trials examine how smart metering can shape energy-use behavior across demographics, lifestyles, and home sizes.
  • Residential datasets: Low Carbon London combines smart meter, time-of-use tariff, and survey data from over five thousand households to study low-carbon technologies.
  • High-frequency datasets: PecanStreet provides minute-level whole-home and appliance-circuit consumption data from 500 homes, while UMass Smart contains one-minute whole-home data from 400 homes.
  • Building and distributed-energy datasets: Building Data Genome contains 507 whole-building electrical meters, and Ausgrid Resident combines three years of smart-meter and rooftop-PV data from 300 consumers.
  • Grid-scale and forecasting datasets: Ausgrid Substation provides approximately 180 continuously updated zone-substation load profiles, while GEFCom2012 supplies about four and a half years of hourly load and temperature data from 11 weather stations.

D. Taxonomy

The paper frames demand-side smart meter analytics around five participants and organizes applications through descriptive, predictive, and prescriptive analytics. Its taxonomy emphasizes load analysis, forecasting, management, and associated methodologies, while noting scope limits.

  • Demand-side participants: The demand side includes consumers, retailers, aggregators, distribution system operators, and data service providers.
  • Demand-side businesses: Retailer applications include load forecasting, price design, consumer service through segmentation and characterization, and abnormal detection.
  • Analytics taxonomy: The taxonomy classifies applications according to descriptive, predictive, and prescriptive analytics, including load analysis, load forecasting, and load management.
  • Methodologies: Methods covered include time-series analysis, dimensionality reduction, clustering, classification, outlier detection, deep learning, low-rank matrix methods, compressed sensing, and online learning.
  • Scope boundary: The review excludes NILM studies because high-frequency data are limited and several comprehensive NILM reviews already exist.
  • Contributions: The paper contributes a recent comprehensive review, a taxonomy of applications, and future research questions involving data, machine learning, business models, energy systems, privacy, and security.

II. LOAD ANALYSIS

Load analysis addresses variability in smart meter profiles through anomaly detection, data cleaning, imputation, and energy theft detection. Reviewed methods include time-series, low-rank, time-window, supervised, and unsupervised approaches.

  • A. Bad Data Detection: Smart meter load analysis reviews anomaly detection and load profiling because diverse profiles affect downstream forecasting and clustering.Bad data may arise from missing values, unusual patterns, or failures in data collection, communication, or entry.
  • A. Bad Data Detection: Bad data detection methods are organized into time-series, low-rank matrix, and time-window approaches.Examples include optimally weighted averaging, autoregressive models with exogenous inputs, neural networks, low-rank matrix fitting, and rolling-window clustering.
  • B. Energy Theft Detection: Energy theft detection treats theft as persistent, intentional alteration of smart meter data rather than temporary unintentional bad data.Methods use smart meter data and, in some cases, power-system state data such as node voltages.
  • B. Energy Theft Detection: Supervised theft detection generally combines feature extraction with classification, while unsupervised methods avoid requiring labeled consumers.Unsupervised approaches include Gaussian-distribution clustering, threshold-based anomaly identification, parallel distributed detection, and forecast-based detection.

C. Load Profiling

Load profiling classifies consumers or load curves by electricity-consumption behavior. Research compares direct clustering with indirect approaches that extract features or model variability before clustering.

  • C. Load Profiling: Load profiling classifies load curves or consumers according to electricity-consumption behaviors using direct- or indirect-clustering approaches.Direct methods include K-means, hierarchical clustering, and self-organizing maps.
  • C. Load Profiling: Smart meter data sampled at least every 30 minutes was sufficiently reliable for most clustering purposes in one comparative study.The study compared k-means, hierarchical algorithms, and the Dirichlet process mixture model across frequencies from 1 minute to 2 hours.
  • C. Load Profiling: Indirect clustering reduces dimensionality or extracts features before classification, using methods such as PCA, K-medoids, auto-encoders, and locality-sensitive hashing.These methods target high dimensionality, temporal and spatial patterns, typical profiles, outliers, compression, or efficient comparison.
  • C. Load Profiling: Feature designs can combine local and global load-shape information with calibration, normalization, and profile-error measures to describe usage patterns.The cited approach is described as informative, understandable, and computationally low-complexity.
  • C. Load Profiling: Variability-aware profiling can identify peak-demand periods, generate typical profiles, model residential variability with a second-order Markov chain, and produce pseudo smart meter data.The generated data can support retailers while protecting consumer privacy.
  • D. Remarks: Future profiling research should address streaming data with distributed or incremental clustering and evaluate methods using application-oriented metrics such as forecasting accuracy.The review also identifies meaningful feature extraction and simultaneous representation-learning and clustering as open directions.

III. LOAD FORECASTING

Load forecasting supports utility operations, planning, pricing, procurement, and hedging across different aggregation levels. Smart meter and distribution data introduce volatility, customer behavior, weather, hierarchy, and renewable-integration considerations.

  • A. Forecasting without Smart Meter Data: Load forecasts support distribution-company operations and planning, while retailers use customer-load forecasts for pricing, procurement, and hedging.Normalized weekly profiles differ across houses, factories, feeders, and cities, with smaller-scale loads generally more volatile.
  • A. Forecasting without Smart Meter Data: Forecasting research has expanded from high-voltage aggregate loads toward medium- and low-voltage levels, including distribution feeders and smart meters.The reviewed literature includes both point and probabilistic load forecasting.
  • A. Forecasting without Smart Meter Data: Customer composition, weather responsiveness, work schedules, and customer attrition create distinct sources of volatility across load profiles.A two-stage retail method forecasts individual customer loads and then customer attrition.
  • A. Forecasting without Smart Meter Data: Weather-aware forecasting studies examine temperature lags, moving averages, humidity representations, and local weather-station selection.One station-selection strategy jointly determines how many stations to use and which stations to select through ranking and greedy combination.
  • A. Forecasting without Smart Meter Data: Distribution-edge forecasting is emerging as operational excellence and large-scale renewable integration push forecasting toward the grid edge.One reviewed methodology uses preprocessing, forecasting, and postprocessing across datasets spanning distribution to transmission levels.
  • A. Forecasting without Smart Meter Data: Grid hierarchy can support forecasting by forecasting subtree roots first, then treating child nodes according to similarity and regularity.Regular nodes use forecasts proportional to parent forecasts, whereas irregular nodes are forecast individually with neural networks.

B. Forecasting with Smart Meter Data

Smart meter data enable forecasting for individual buildings and households while offering opportunities to improve aggregate forecasts. Reviewed approaches address volatility, sparsity, aggregation, clustering, ensembles, and evaluation metrics.

  • B. Forecasting with Smart Meter Data: Smart meters support forecasts for individual houses or buildings and may improve aggregate forecast accuracy through high-granularity data.Household and building consumption is more random and volatile than aggregate consumption, so aggregate-level methods may not transfer directly.
  • B. Forecasting with Smart Meter Data: Household-level forecasting research evaluates existing methods, develops new methods, or combines both strategies.Reviewed approaches include neural networks, deep learning, sparse modeling, clustering, and functional time-series methods.
  • B. Forecasting with Smart Meter Data: 10% improvements in forecast accuracy were obtained when sparse coding features were included for data from 5000 households in Chattanooga, Tennessee.The same review also reports that LASSO-based forecasting had low computational complexity and comparable accuracy on Pacific Gas and Electric Company data.
  • B. Forecasting with Smart Meter Data: Clustering can reduce noise, preserve salient features, and assign customer groups to different forecasting models before summing group forecasts.Cluster-based forecasting performance depends on the number of clusters and consumer-group size.
  • B. Forecasting with Smart Meter Data: Forecasting accuracy increases with group size, while MAPE decreases quickly below 100,000 consumers and decreases only slightly above 100,000.These findings were reported for Seasonal-Naive and Holt-Winters algorithms and for Pacific Gas and Electric Company data, respectively.
  • B. Forecasting with Smart Meter Data: Residential ensembles outperformed all individual forecasts from traditional models, motivating frameworks that optimally combine forecasts.The review also describes clustering-based forecast generation and combination.
  • B. Forecasting with Smart Meter Data: MAPE may poorly quantify individual load forecasts because of outliers, violation characteristics, and time shifting.Proposed alternatives include resistant MAPE, MAAPE, and household-level measures designed for intermittent or time-shifted loads.

C. Probabilistic Forecasting

Probabilistic load forecasting extends point forecasting by representing uncertainty in inputs, models, or outputs. The review surveys these approaches and identifies open questions around forecast accuracy, evaluation costs, load transfers, and renewable-heavy systems.

  • Probabilistic forecasts provide more information about future uncertainties than point forecasts.The workflow can be modified through input-scenario generation, probabilistic models, or output post-processing.
  • Temperature scenario generation captures uncertainty in electricity-demand drivers before forecasting.Methods include reusing, shifting, or bootstrapping historical hourly temperatures.
  • Residual simulation and forecast combination convert point forecasts into probabilistic forecasts, but residuals do not always follow a normal distribution.Grouping can increase normality-test passing rates, while simulated normal residuals help deficient models with diminishing improvement.
  • A more accurate point model produces a more skilled probabilistic forecast when the underlying models differ significantly in point-forecast accuracy.
  • Forecast evaluation should account for unequal costs of over- and under-forecasting rather than relying only on measures such as MAPE.This could help bridge forecasting and decision making.
  • Probabilistic forecasting needs further attention for systems with high renewable-energy penetration and large-scale storage.These resources can disrupt traditional load profiles and motivate probabilistic net-load forecasts.

IV. LOAD MANAGEMENT

Load management uses smart meter data to characterize consumers, target demand-response participants, and support program implementation. Reviewed methods connect load profiles with sociodemographic information, estimate response potential, and quantify possible reductions.

  • Load management addresses consumer characterization, demand-response marketing, and price design or baseline estimation.
  • Consumer Characterization: Consumer characterization links load profiles with sociodemographic information to classify users and enable personalized services.
  • Consumer Characterization: Frequency-domain features and classification trees can place consumers into categories from temporal load profiles.Fast Fourier transformation supplies frequency coefficients to a classification and regression tree.
  • Consumer Characterization: Commercial and cartographic data enabled accurate consumer-load-profile prediction with random forests and stochastic boosting.Other analyses identified location, floor area, consumer age, and appliance count as influential factors, while income and home ownership had little relationship with consumption.
  • Consumer Characterization: A two-dimensional CNN automatically extracted consumption features and outperformed traditional methods when linking smart meter data to sociodemographic information.
  • Demand Response Program Marketing: Demand-response potential can be estimated through consumption variability, occupancy states, load shifting, and reduction indices.HMM-based spectral clustering supports targeting suitable consumers across time scales, while probabilistic reduction indices quantify possible reductions.

C. Demand Response Implementation

Demand-response implementation relies on price design for price-based programs and baseline estimation for incentive-based programs. The review covers clustering, optimization, probabilistic baselines, and consumer classification while identifying important modeling limitations.

  • Price design seeks to attract consumers and maximize profit, whereas baseline estimation quantifies performance in incentive-based programs.
  • Price Design: Different clustering algorithms and cluster counts lead to different tariff-design costs.GMM clustering of prices and load profiles was used to develop time-of-use tariffs and quantify their demand-response impact.
  • Baseline Estimation: Five naive baseline methods include High-XofY, MidXofY, LowXofY, exponential moving average, and regression baselines.The reviewed work found bias, rather than accuracy, was the main factor determining which baseline produced the largest profits.
  • Consumer Characterization: Consumer characterization remains a high-dimensional, nonlinear classification problem, with data-driven feature extraction identified as a potential improvement.
  • Demand Response Program Marketing: Marketing methods often use indirect indices because 30-minute-or-lower-frequency data cannot reveal individual-appliance operation states.The review calls for more indices and real-world experimental evidence to test whether these indices reflect demand-response potential.
  • Demand Response Implementation: Price-design results may deviate from optimal results when the assumed consumer acceptance or utility function differs from the real function.The acceptance function is difficult to estimate in the surveyed literature.

A. Connection Verification

Smart meter analytics support distribution connection verification, outage management, and data compression. The reviewed methods infer network connections and outage areas from meter data while addressing the communication and storage burden of massive datasets.

  • Connection Verification: Connection information supports distribution-system operation, but complete topology may be unavailable, especially at low voltage levels.
  • Connection Verification: Voltage and power-consumption correlations can correct connectivity errors, although downstream voltage-decrease assumptions may fail with substantial distributed renewable generation.
  • Connection Verification: Topology identification has been formulated using probabilistic graph models and Lasso-based sparse estimation.
  • Connection Verification: PCA has been used for phase and topology identification while modeling technical-loss, metering, and clock-synchronization errors as Gaussian distributions.Incomplete-data approaches address bad or null meter data.
  • Outage Management: Outage management includes notification, location, and restoration verification, and can identify outage areas through topology analysis followed by smart-meter analysis.Communication impacts and missing-data challenges are also considered.
  • Data Compression: Data compression reduces smart-meter communication and storage burdens, using lossy or lossless methods depending on requirements.Differential coding exploits the small changes between adjacent high-frequency consumption measurements.

D. Data Privacy

Smart meter data analytics faces privacy concerns because fine-grained consumption data can reveal sociodemographic information. Future work must address privacy, security, scalable data processing, machine learning, and data integration challenges.

  • Fine-grained smart meter data can reveal consumers’ sociodemographic information, making privacy a major concern.
  • Data Privacy: Privacy-preserving analytics includes secure aggregation, privacy–utility trade-offs, and methods such as privacy-preserving clustering and PCA.
  • Big Data Issues: Big data research must integrate external information, including economic, meteorological, EV charging, and consumer data, with consumption records.
  • Big Data Issues: Large smart meter datasets require efficient approaches such as distributed, parallel, cloud, GPU, and fog computing.
  • Machine Learning: Deep learning applications must account for dataset size to avoid overfitting, while limited labels motivate transfer learning and other methods.

1) Deep Learning and Transfer Learning:

Smart meter data analytics must adapt to real-time streams, evolving demand-side business models, and changing distribution systems. Emerging applications include transactive energy, prosumer trading, and shared renewable resources.

  • Deep Learning and Transfer Learning: Smart meter data are real-time streams, but online and incremental learning remain rarely used in existing analytics studies.
  • Transactive Energy: Further retail deregulation, distributed renewables, and information-technology progress will accelerate new demand-side business models.
  • Transactive Energy: Transactive energy can support consumer-to-consumer electricity trading through micro-markets involving prosumers with rooftop PV.
  • Sharing Economy: Future distribution systems may use sharing-economy models in which consumers share rooftop PV and storage with neighbors.
  • Energy System Transition: Distributed renewable energy, storage, EVs, and multiple energy systems will change the structure and energy flows of smart homes and distribution systems.

1) High Penetration of Renewable Energy:

High renewable-energy penetration changes electricity consumption behavior and net-load profiles, requiring revised analytics. Privacy and security remain important boundaries as smart meter analytics expands across retail-market applications.

  • High Penetration of Renewable Energy: Behind-the-meter PV can significantly alter electricity consumption behavior and net-load profiles, requiring improved traditional load-profiling methods.
  • High Penetration of Renewable Energy: Combining weather, electricity-price, and net-load data can support estimation of renewable-energy capacity and output.
  • Multiple Energy Systems: Multiple energy systems couple electricity, heat, cooling, and gas consumption, enabling joint forecasting across these energy types.
  • Data Privacy: Data-analytics approaches to privacy include privacy-preserving clustering, PCA, and battery scheduling to mask consumption behavior.
  • Data Security: Cyberattacks targeting electricity-consumption data, including non-technical-loss contexts, require further study beyond established PMU and SCADA security research.
  • Conclusions: The review covers load forecasting, abnormal detection, consumer segmentation, and demand response, while proposing future directions across several emerging challenges.
Loading 1802.04117v2…