PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning

Christos Petridis, Zoran Obradovic, Mladen Kezunovic

arXiv:2609.04272v1cs.LGcs.CL

TL;DR

Weather-related outage risk prediction matters because occurrence-only approaches do not capture severity. This study compares zero-shot LLMs with supervised classifiers using weather inputs across outage-risk horizons. Supervised models lead on macro-F1 and precision, while newer LLMs are competitive without labeled training data and provide explanations.

  • Problem

    Many outage-risk approaches classify whether an outage occurs but do not capture its severity, limiting the distinction between impacts that may require different responses.

    Source: §1 · p. 2

  • Method

    The study evaluates zero-shot LLM outage-risk prediction against supervised ML using weather inputs and outage records aggregated into 3-, 6-, and 12-hour binary severity labels.

    Source: §1 · p. 2, §5 · p. 4

  • Results

    Supervised models outperform all LLMs on macro-F1 and precision across settings, with the best-model macro-F1 gap ranging from 0.04 to 0.13 points; newer LLMs are comparable without labeled training data.

    Source: §7 · p. 7, §9 · p. 9

  • Takeaways & Limitations

    The authors view ML and LLMs as complementary: ML provides precise risk assessments, while LLMs provide actionable explanations that may aid operator decisions.

    Source: §7 · p. 7, §9 · p. 9

Abstract

from arXiv · show

This study examines the ability of large language models (LLMs) to predict the risk of weather-related forced outages in the distribution grid in a zero-shot framework, without labeled training data. The problem is formulated as a binary severity classification task across three forecast horizons (3h, 6h, 12h), using six years of outage records and high-resolution weather data for a utility service area in central Texas. Four zero-shot LLMs are benchmarked against two supervised classifiers across two input configurations: one using current weather observations and the other using weather forecast data. Results show that supervised models outperform LLMs on macro-F1 and precision, while newer LLM generations achieve competitive scores. Beyond accuracy, LLMs offer complementary strengths in actionable reasoning and geographic scalability, suggesting that combining them with supervised models may be the best practice.

1. Introduction

Weather-related outages threaten grid resilience, while earlier prediction work often requires identified storm events or predicts outage occurrence without capturing severity. The paper introduces zero-shot LLM prediction and compares it with supervised ML, alongside design benefits and costs.

1. Introduction

  • Weather-related phenomena account for approximately 83% of reported major U.S. power outages from 2000 to 2021.The cited phenomena include thunderstorms, strong winds, flash floods, and winter storms.Source: §1 · p. 1
  • Prior outage-risk formulations often classify only zero versus one-or-more outages, so they do not characterize severity for operators’ mitigation decisions.Severity matters because a few outages and many outages can have markedly different operational implications.Source: §1 · pp. 1–2
  • The study presents zero-shot LLM inference for forced-outage risk prediction and evaluates transferability, scalability, interpretability, implementation cost, and performance against supervised ML.The authors describe this as the first application of LLM-based zero-shot inference to the problem.Source: §1 · p. 2

2. Motivation for Large Language Models

Supervised outage-prediction pipelines can demand extensive data preparation, labeled records, and retraining, with transfer to a new utility requiring the process to be repeated. The paper motivates LLMs as a possible alternative and evaluates their design benefits and risk reasoning.

2. Motivation for Large Language Models

  • The supervised pipeline requires extensive data wrangling, feature engineering, labeled outage history, and periodic retraining; deployment in a new territory requires repeating the process.The study’s pipeline aligned feeder-level outage records with a 10×10 weather grid, addressed an approximately 5% positive rate, and used six-year leave-one-year-out validation.Source: §2 · p. 2
  • The study evaluates LLMs on transferability across utilities, scalability across spatiotemporal and geographic conditions, and interpretability for utility staff.The proposed rationale is that prompts can specify time and place, while natural-language explanations can describe risk factors.Source: §2 · p. 2
  • The authors propose that LLMs may identify high-risk conditions from weather inputs alone without historical grid-outage context or prior knowledge of a utility’s operational scale.They frame this as a hypothesis about informed decision-making across utility settings.Source: §2 · pp. 2–3

3. Statistical Analysis

The study matched central Texas outage records with NOAA severe-weather events to examine their relationship, then assessed whether weather activity precedes outages. Weather-event hours had substantially more outages, and both directional tests favored weather preceding outages.

3. Statistical Analysis

  • The analysis paired historical outage records with NOAA storm-event records, expanded events to hourly resolution, and used a ±2-hour matching window.Hours without a nearby weather event served as normal-weather records; the event window accounts for imprecision in NOAA event timing.Source: §3 · p. 3
  • 2,082% more outages occurred on average near weather events than during normal-weather hours (Mann–Whitney U, p < 10−270).Thunderstorm wind, lightning, and tornadoes each showed increases exceeding 3,500% over normal-weather hours; all event types were significant at p < 0.001.Source: §3 · p. 3, Figure 2 · p. 3
  • Weather activity at time t predicted outages at t + 1 h (r = 0.34), versus a maximum reverse correlation of 0.25; Granger testing also favored weather predicting outages.The forward Granger result was F = 2,853 at lag 1 h (p < 0.001), approximately 19 times stronger than the reverse direction.Source: §3 · p. 3

4. Framing Outage Risk Prediction as a Severity Classification Problem

Outage-risk signals can inform warnings, crew positioning, operational decisions, and mitigation, but occurrence-only definitions treat outages with very different impacts as equivalent. The section motivates a risk definition that captures severity.

4. Framing Outage Risk Prediction as a Severity Classification Problem

  • Outage-risk estimates can support early customer warnings, restoration estimates, advance crew positioning, and operational or mitigation decisions.Source: §4 · pp. 3–4
  • Risk defined as the probability of at least one outage does not distinguish a few outages from multiple outages in the same area and interval.The section notes that this occurrence-based definition is used in prior work.Source: §4 · p. 4
  • In the central Texas example, some outages cause minor resilience deterioration while others drive resilience below 0.6 for an extended period.During Winter Storm Uri, the territory experienced a prolonged severe resilience collapse, illustrating why occurrence alone can miss impact differences.Source: §4 · p. 4

5. Data Modeling

Outage records are converted into horizon-specific binary risk labels, while weather inputs are spatially aggregated and tailored to forecast availability.

5. Data Modeling

  • Each window’s target is the outage count reported over the next h hours, as forecast from the current timestep t.Individual feeder outage records are counted in fixed-length, non-overlapping windows.Source: §5 · p. 4
  • High risk is defined as exceeding thresholds of 5, 8, and 10 outages for 3-, 6-, and 12-hour windows, respectively.Each horizon’s threshold was chosen to yield a positive-class rate of approximately 5%.Source: §5 · p. 4
  • Horizon-specific cutoffs reflect that five outages signals elevated stress over 3 hours but not 12 hours, keeping high risk in each horizon’s upper tail.The thresholds account for different accumulation periods across forecast horizons.Source: §5 · pp. 4–5
  • Weather observations summarize 100 grid cells using spatial statistics; forecast inputs aggregate h-hour values by variable, whereas no-forecast inputs use current conditions.The aggregation includes means, minima, maxima, and standard deviations across grid cells.Source: §5 · p. 5

6. Predictive Modeling

The predictive modeling setup compares zero-shot LLMs with supervised classifiers across three forecast horizons and two weather-input configurations. The LLMs produce probability estimates and brief reasons, while the supervised models are evaluated using year-held-out cross-validation.

6. Predictive Modeling

  • High-risk cases make up roughly 5% of instances across the three horizons, with 2021 showing the highest yearly proportion even after February is excluded.Table 1 summarizes high-risk counts relative to total timesteps by year and forecast horizon.Source: §6 · p. 5, Table 1 · p. 6

6.1. Predictive Modeling using LLMs

  • Four LLMs perform zero-shot classification from forecast-window weather or current observations, returning an outage probability and a brief reason.A fixed 0.5 threshold converts predicted probabilities to binary decisions, with no in-context examples or data-based calibration.Source: §6.1 · pp. 5–6

6.2. Predictive Modeling using ML

  • Logistic Regression and LightGBM are the reported supervised models, evaluated with Leave-One-Year-Out cross-validation across held-out calendar years.Both models account for class imbalance; the other evaluated classifiers were not reported because these two performed best on the dataset.Source: §6.2 · p. 6

7. Experimental Results

Across forecast horizons and weather configurations, supervised models lead LLMs on macro-F1 and precision, while LLMs show greater horizon stability and sometimes higher recall.

Effect of forecast horizon

  • LightGBM macro-F1 falls from 0.74 to 0.67 with forecast and from 0.71 to 0.61 without, while GPT-4.1 variants stay around 0.60–0.63 across horizons.Performance generally declines at longer horizons, with a more pronounced drop for ML models.Source: §7 · p. 6

Role of weather information

  • Forecast inputs consistently improve ML performance, while LLMs benefit less overall and some apparent GPT-4o-mini gains reflect a trivially low without-forecast baseline.Across the reported Table 3 gains, forecast information does not degrade any model's performance.Source: §7 · pp. 6–7

Machine Learning and LLM performance

  • Across all settings, ML models outperform LLMs on macro-F1 and precision; the best-ML versus best-LLM macro-F1 gap is 0.04–0.13 points.LightGBM leads at 3h and 12h with forecast, while Logistic Regression leads at 6h.Source: Table 2 · p. 7, §7 · p. 7
  • GPT-4o achieves 0.82 recall at 12h with forecast versus LightGBM's 0.65 and Logistic Regression's 0.55, but its precision is 0.14.GPT-4o-mini also has high recall paired with precision near 0.05–0.07, effectively flagging nearly every window as high risk; ML models have a more balanced operating point.Source: §7 · p. 7

8. Discussion

The discussion finds that ML remains stronger on predictive performance, while LLMs offer transferability, scalability, and natural-language explanations that can support operator decision-making.

8.1. Performance Metrics

  • ML models remain superior on predictive performance, with their advantage most pronounced at shorter horizons where forecast features provide an immediate signal.The discussion attributes this advantage to Logistic Regression and LightGBM being trained directly on the target distribution and learning predictive weather features for the service region.Source: §8.1 · p. 7
  • GPT-4.1 consistently outperforms GPT-4o on macro-F1 and precision, and GPT-4.1-mini reaches 0.61 versus GPT-4o's 0.57 macro-F1 at 3h with forecast.The pattern holds for the mini models across evaluated metrics except recall, and GPT-4.1-mini exceeds GPT-4o in macro-F1 across horizons and configurations.Source: §8.1 · p. 8
  • GPT-4o and GPT-4o-mini over-predict, whereas GPT-4.1 models trade recall for precision; the deployment choice depends on whether catching high-risk events or minimizing unnecessary responses matters more.A uniform 0.5 threshold may not represent the same underlying confidence across models.Source: §8.1 · p. 8
  • LLMs show only modest improvement from full-window forecast inputs, suggesting they rely on salient indicators already present in current-weather snapshots.The discussion notes that LLMs can still provide reasonable estimates from current observations when real-time forecast data is unavailable or costly.Source: §8.1 · p. 8

8.2. Design Criteria

  • GPT-4.1 and GPT-4.1-mini achieve macro-F1 scores of 0.57–0.63 without exposure to this utility's historical outage records, and the same prompt template can transfer across territories without retraining.The discussion contrasts this with supervised models, which must be rebuilt for each new territory.Source: §8.2 · p. 8
  • Across forecast horizons and weather configurations, LLMs maintain stable performance; adapting to a new horizon, region, or input signal requires prompt adjustments rather than retraining or feature engineering.Supervised models require a full retraining cycle when the input feature set changes.Source: §8.2 · p. 8
  • LLM predictions include natural-language explanations of driving factors that supervised models do not provide, supporting a complementary role alongside ML's stronger predictive performance.The discussion says even explanations of moderate quality can support operator decision-making when LLM accuracy falls short of ML.Source: §8.2 · p. 8

9. Conclusion

This initial assessment finds that recent LLMs can approach supervised models without labeled training data, while offering complementary benefits for outage-risk prediction. The conclusion identifies possible improvements and directions for further study.

9. Conclusion

  • Supervised models retain a predictive-performance advantage, while recent LLMs attain comparable performance without labeled training data.The authors describe this as an initial assessment of zero-shot LLMs for weather-related forced outage risk prediction at the distribution grid.Source: §9 · pp. 8–9
  • The paper presents its findings as an initial assessment of zero-shot LLM use for weather-related forced outage risk prediction at the distribution grid.Source: §9 · p. 8
  • The authors frame ML and LLMs as complementary: ML provides precise risk assessments, while LLMs offer actionable explanations for operator decisions.They also report benefits in transferability, scalability, and interpretability across different environments.Source: §9 · p. 9
  • Combining weather observations and forecasts in one prompt, with location, calendar, and historical outage context, may improve LLM accuracy.Source: §9 · p. 9
  • The authors identify open-source models and inference-time information retrieval as future directions, citing API cost, privacy, and accuracy considerations.They suggest open-source models could reduce API costs and let utilities share internal data without privacy concerns; agentic retrieval is described as promising.Source: §9 · p. 9

Appendix

The appendix defines resilience at each time point by the fraction of operational feeders and describes an aggregate index over an event window using trapezoid loss area.

Appendix

  • System resilience level is the fraction of feeders remaining operational at each point in time.The number of operational feeders at time t is Fop(t), and Ftotal is the total number of feeders.Source: Appendix · p. 10
  • The aggregate Resilience Index R is computed over an event window [te, tf] via a trapezoid loss area.Source: Appendix · p. 10
  • For interval i, ϕi denotes resilience level and ∆ti denotes its duration in hours.Source: Appendix · p. 10
Loading 2609.04272v1…