Source-linked AI summary
Data Leakage Inflates Generalizability of Power Outage Prediction Models
Yamil Essus, Ranga Raju Vatsavai, Benjamin Rachunok
TL;DR
Power outage prediction models must generalize across changing spatial, temporal, and event conditions, but standard evaluations may not test this adequately. This paper compares evaluation strategies and feature sets, finding that performance degrades substantially under realistic holdouts and often fails to beat a null baseline.
Problem
Power outage prediction models need to generalize across novel spatial, temporal, and event conditions as climate change alters outage-driving weather patterns.
Method
The study evaluates outage models across target definitions, feature sets, and random, spatial, and temporal holdout strategies, including Prithvi WxC embeddings.
Results
Performance degrades substantially on unseen regions and events, with improvements over a null model often small, variable, and not statistically meaningful.
Takeaways & Limitations
Publicly trained outage prediction models offer limited and uncertain operational value under realistic deployment scenarios.
Takeaways & Limitations
Strong spatiotemporal dependence and few independent extreme events constrain generalization from the available data.
Abstract
from arXiv · showhide
Power outage prediction models are increasingly used in assessments of climate-driven infrastructure risk, yet current evaluation practices obscure whether these models generalize to the novel conditions such applications require. We identify three common methodological choices in power outage prediction models that influence their ability to generalize across spatial, temporal, and event-based settings. We compare the predictive performance impacts of different methodological decisions using publicly available data for the U.S. East Coast from 2018 to 2023 and feature sets derived from weather reanalysis and land-cover data, and embeddings from a GeoAI foundation model (Prithvi WxC). Specifically, we assess model performance under multiple test selection strategies, including unfiltered random splits, leave-one-state-out, and leave-one-event-out designs, which increasingly approximate real-world deployment conditions. While random train-test splits yield strong performance, we show that these results are inflated by spatial and temporal autocorrelation. Under spatial and temporal holdout experiments, predictive accuracy degrades substantially, with models often failing to outperform a simple null baseline. Incorporating GeoAI foundation model embeddings yields limited and inconsistent improvements, primarily for spatial generalization, and does not resolve poor event-level transferability. These findings suggest that, given current data availability and evaluation practices, publicly trained outage prediction models offer limited and uncertain operational value. Progress will likely require improved data coverage, more realistic evaluation protocols, and a shift in focus from marginal modeling advances toward addressing structural data constraints.
Introduction
Power outage prediction research seeks relationships that generalize across space and time, but changing environmental conditions and common evaluation choices can obscure true predictive capacity. This paper evaluates how data cleaning and test-set construction affect out-of-sample performance estimates.
- Motivation: Power outages cause substantial quality-of-life and economic losses, motivating machine-learning research using environmental, geographic, and infrastructure data.Existing studies broadly seek spatially and temporally generalizable relationships between covariates and outages.
- Changing modeling context: Climate change is increasing extreme events beyond historical observations, while GeoAI and foundation models enable training on larger, higher-dimensional datasets across regions.These methodological and environmental changes affect the role and reliability of outage prediction models.
- Evaluation problem: Random test splits and manually selected outage events can obscure whether reported performance reflects true predictive capacity across spatial, temporal, and event-based settings.The paper identifies these as common methodological choices influencing generalization.
- Operational consequences: Limited recognition of poor generalization can delay outage responses, prolong outages, and cause cascading disruptions across water, communication, transportation, healthcare, and agriculture.These consequences can extend for months and affect interdependent and essential systems.
- Study objective: The study evaluates state-of-the-art outage models by testing how data cleaning, test-set strategies, and relative versus absolute targets influence out-of-sample performance estimates.Experiments cover the entire U.S. East Coast and compare random, spatial leave-one-one, and temporal leave-one-out test selection.
Related Work
Prior power outage prediction research spans fragility-based methods and statistical models, while differing evaluation and target definitions may limit generalizability. Geospatial foundation models, including Prithvi WxC, offer a potential way to address these limitations through learned weather representations.
- Existing prediction methods: Existing outage prediction methods include fragility-based approaches and statistical models using supervised machine learning techniques such as support vector regression, tree-based models, and artificial neural networks.Common covariates include wind speed and wind gust and precipitation.
- Generalizability limitations: The literature identifies random test splits, absolute outage targets, and heterogeneous study areas and periods as three modeling choices that may affect generalization.Random splits can produce overly optimistic results because weather, land-use, and remote-sensing covariates exhibit high spatio-temporal correlation.
- Generalizability limitations: Evidence remains limited on whether models trained on specific outage events generalize to other events or areas.Arora and Ceferino trained statistical models using data from over 1,900 U.S. cities across three hurricane events and observed signs of overfitting to local conditions.
- Foundation-model approaches: Geospatial Artificial Intelligence foundation models may address generalizability limitations by learning dense, relatively low-dimensional representations from very large datasets using self-supervised training.These models commonly use encoder-decoder architectures.
- Foundation-model approaches: Prithvi WxC is a globally trained weather foundation model whose encoder embeddings can support spatial power outage prediction through transfer learning.The model uses weather features, static features including land cover, and climatological features based on deviations from historical values.
Methods
The study trains XGBoost models while varying input features, outage targets, and train-test split strategies to assess generalizability. It combines county-day weather, land-cover, and outage data with optional Prithvi WxC embeddings under random, spatial, and temporal evaluation designs.
- Experimental design: XGBoost experiments vary input features, output variables, and train-test split strategies.Features include ERA5 variables with optional tree density or Prithvi WxC embeddings; targets are absolute outage counts or percentage of customers affected.
- Feature construction: ERA5 supplies baseline weather information, while tree density from NLCD is included as a static covariate because trees may damage distribution infrastructure.Features are aggregated to the county-day level using spatial joins, with maximum temporal aggregation where specified.
- Data sources and preprocessing: Outage targets are derived from PowerOutage.us county-level records by summing the maximum reported outage across utilities for each county-day.The “Customers Tracked” field provides the reference for calculating the percentage of customers affected.
- Data sources and preprocessing: Data cleaning reduces zero-inflation caused by counties experiencing few significant outages on most days.The paper describes this as a central preprocessing objective before model training.
- Evaluation design: The evaluation compares random 80-20, spatial leave-one-state-out, and temporal leave-one-out test selections across experiments.A random validation sample from the training set is used for early stopping, and the random split is stratified to preserve the proportion of significant outages.
Results
Random 80-20 splits produced strong apparent performance, but spatial and event-based holdouts revealed substantial generalization failures. Prithvi embeddings offered limited, inconsistent benefits, with improvements concentrated in some spatial settings and Hurricane-event prediction.
- Random split: Random 80-20 splits achieved approximately 45% R2 for pCoP and about 33% for CoP, despite systematically similar samples across training and test sets.The split was treated as a best-case baseline and can inflate performance because it ignores spatio-temporal correlations in weather covariates.
- Random split: Feature sets showed no substantial performance variation: Prithvi favored relative outage magnitude, whereas ERA5 plus static covariates moderately favored absolute magnitude.Prithvi’s relative-magnitude advantage came with higher dimensionality and data-processing complexity.
- Spatial generalization: Under leave-state-out testing, average MAE was 0.061 for pCoP and 2,317 for CoP, compared with around 0.06 and 1,800 in the best-case scenario.Absolute-error differences were linked to population size and the absence of historical patterns for the held-out area.
- Spatial generalization: Leave-state-out models often exceeded the null model’s error across states, while Prithvi models were slightly more consistent but remained poor overall.Relative predictions were close on average to the best-case scenario yet varied across states, including consistently poor performance in Delaware and Massachusetts.
- Temporal generalization: In leave-event-out testing, no model meaningfully outperformed the null model for CoP across event types; only Hurricanes showed consistently improved pCoP prediction, with Prithvi performing significantly worse than ERA5.These results indicate stronger temporal generalizability for Hurricane-driven outages than for Winter Storms or Other events.
Discussion and conclusion
Models that perform well under random splits generalize poorly to unseen regions and events, limiting their operational value when trained on publicly available data. The discussion attributes these limitations primarily to spatial-temporal dependence and data constraints rather than insufficient model complexity.
- Generalizability: Predictive performance degrades substantially on unseen regions and events because random-split accuracy is inflated by spatial and temporal autocorrelation.Leave-one-state-out and leave-one-event-out experiments better approximate real-world forecasting conditions than widely used random train-test splits.
- Operational implications: Temporal transfer is especially poor: no evaluated model consistently outperforms a null baseline for absolute outage magnitude when entire events are excluded from training.Only hurricane events show an exception in the supplied passage, whose sentence is truncated before specifying the result.
- GeoAI embeddings: Prithvi WxC embeddings provide modest, inconsistent gains mainly in some spatial generalization settings, especially for relative outage metrics, but not for temporal generalization.Using the embeddings also incurs practical costs from extracting and storing high-dimensional representations.
- Data and future directions: Publicly available weather reanalysis can smooth localized extremes critical to outage prediction, while accurate models often depend on large, region-specific, and proprietary datasets.The paper therefore prioritizes better data coverage, open reporting standards, and evaluation protocols reflecting intended deployment conditions over marginal architectural improvements.
- Model complexity: Training and validation set R2 consistently above 90% and 30% indicates that models fit available data well, making alternative machine-learning architectures unlikely to resolve the observed limitations.The discussion links these limits to the relatively small number of independent extreme events and strong spatiotemporal dependence.
Appendix
The appendix reports the best-performing feature set for each dataset event in leave-event-out experiments, alongside mean absolute error results for relative and absolute outage magnitude.
- Appendix: The appendix reports mean absolute error for both relative and absolute outage magnitude metrics.The errors correspond to the event-specific best feature sets.
- Appendix: Table 5 lists the feature set that produced the best performance for each event in the dataset.These results come from the leave-event-out experiment.