Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events

• Extreme weather requires accurate tail predictions because failures can worsen impacts such as crop loss, wildfires, and floods.Source: §1 · p. 2
• High overall accuracy does not establish reliable performance on extreme events, where extrapolation and sparse observations remain fundamental challenges.Source: §1 · p. 2
• Standard summary scores evaluate performance across variables and lead times but can miss forecast smoothness, physical-balance errors, and event-specific failures.Source: §1 · p. 2
• Extreme-event research has examined temperature thresholds, tropical cyclones, atmospheric rivers, and storms, but many event types remain under-explored at case-study level.Source: §1 · pp. 2–3
• Compound impact metrics and spatial or temporal extent receive limited attention, despite their potential to reveal systematic errors hidden by aggregate scores.Source: §1 · p. 3
• The study compares GraphCast, PanguWeather, and FourCastNet with HRES for the 2021 Pacific Northwest heatwave, 2023 South Asian humid heatwave, and 2021 North American winter storm.Source: §1 · p. 3

2.1 Data

• ERA5 supplies ML initialization and evaluation data, while HRES forecasts are downsampled to the ERA5 grid and compared against HRES-fc0 as ground truth.The analysis uses six-hourly lead times from 0 h to each forecast's maximum available length.Source: §2.1 · p. 3

2.2 Machine Learning Models for Weather Forecasting

• The study evaluates FourCastNet, PanguWeather, and GraphCast, whose key characteristics and predicted variables differ.GraphCast uses two consecutive weather states and substantially more input information than the other ML models, potentially improving predictive accuracy.Source: §2.2 · pp. 3–4, Table 1 · p. 4
• PanguWeather combines 6 h and 24 h model calls through hierarchical temporal aggregation to reduce error accumulation at the evaluated lead times.For a 36 h forecast, the paper uses one 24 h call and two 6 h calls rather than repeated 1 h calls.Source: §2.2 · p. 4
• Additional forecasts are generated from released pre-trained ML models because existing WeatherBench 2 data do not cover all investigated periods.The paper notes that the ai-models GraphCast implementation is a smaller operational version with different training data and variables.Source: §2.2 · p. 4

2.3 Initialization Times

• Initialization comparisons account for unequal ERA5 and HRES assimilation lookahead through mixed initialization methodology in one analysis.Other analyses include all four initialization times, and short HRES forecasts are filled beyond four days using forecasts initialized six hours earlier, which may disadvantage HRES.Source: §2.3 · pp. 4–5

3.1 2021 Pacific Northwest Heatwave

• For the Pacific Northwest heatwave, all models struggled at long lead times, with errors reaching at least twice a typical HRES 10-day error near major cities.In Portland, errors exceeded the typical HRES 10-day error by a factor of four.Source: §3.1 · p. 5
• GraphCast and HRES often had the smallest local errors, FourCastNet the largest, and no model was clearly best overall across lead and validity times.The models showed similar local error magnitudes in several comparisons, but their spatial aggregation differed: FourCastNet under-predicted threshold-exceedance areas while some PanguWeather forecasts over-predicted them.Source: §3.1 · pp. 5, 6, 8
• Beyond one week, all models performed substantially worse during the heatwave than during summer 2022, with ML errors up to three times worse and HRES errors up to two times worse.For lead times up to 6.5 days, HRES errors were smaller than ML errors, although the authors caution that the baseline period was short.Source: §3.1 · p. 6, Figure 3 · p. 8

3.2 2023 South Asian Humid Heatwave

• The humid-heat case required a substitute humidity variable because ML models lacked the surface humidity information needed to calculate surface heat index correctly.The analysis therefore used RH1000hPa; FourCastNet was excluded because its predictions began at 850 hPa.Source: §3.2 · pp. 8–9
• Using RH1000hPa, ML models under-predicted heat index most strongly over Bangladesh, where high heat-index values and danger levels were especially underestimated.The spatial error pattern matched relative-humidity under-prediction, particularly for PanguWeather.Source: §3.2 · p. 9

3.3 2021 North American Winter Storm

• For the North American winter storm, GraphCast and PanguWeather produced substantially lower wind-chill errors than HRES after the storm peak, while FourCastNet errors sometimes exceeded 40 K.All models struggled to predict the minimum wind chill, and all forecasts appeared too warm in the regional comparison.Source: §3.3 · pp. 11, 13
• The winter-storm error patterns differed structurally: ML models tended toward vertical patterns, whereas HRES and FourCastNet showed stronger diagonal structures linked to initial-condition variation.The authors suggest these patterns may reflect differences in how ML and NWP models represent predictable versus hidden atmospheric conditions, while noting that HRES data filling may affect the finding.Source: §3.3 · p. 11
• Wind chill was evaluated outside its formal validity thresholds during the most hazardous days because forecasts could make the index undefined.The study computed the wind-chill expression anyway for visual clarity when temperatures or wind speeds did not meet the definition.Source: §3.3 · pp. 9, 11, 13

Case-study findings

• PanguWeather and GraphCast often match HRES for the Pacific Northwest heatwave, outperform HRES for many winter-storm forecasts, but underestimate the highest danger levels over Bangladesh for the humid heatwave substitute metric.The winter-storm forecasts also show structurally different error buildup across systems.Source: §4 · p. 13

Limitations

• Surface humidity is unavailable from the evaluated ML models, preventing a detailed assessment of health risks during the 2023 South Asian humid heatwave.The authors note that whether the observed effect persists for ML models predicting surface humidity remains unanswered.Source: §4 · p. 13
• ERA5–HRES-fc0 differences complicate estimating true heat-index forecast errors, motivating direct comparison with station observations.Source: §4 · p. 13
• Case-study selection can favor alarmist forecasts through the forecaster’s dilemma, so these results should not alone determine overall forecast-system quality.Source: §4 · p. 13
• The study uses single forecasts rather than probabilistic ensembles, limiting assessment of forecast uncertainty and complicating comparison with ensemble-based NWP.Source: §4 · pp. 13, 15
• Coarse ML time steps may miss short-lived impact peaks, while missing surface humidity, solar radiation, and reliable precipitation limit assessable extremes.Source: §4 · p. 15

Scope and future work

• Case studies can reveal unexpected model deficiencies, but their small number and limited variables and lead times constrain the range of extreme events studied.Source: §4 · p. 15

Future directions

• Impact-focused evaluation, released raw predictions, reusable extreme-event databases, and physically informed or extreme-value methods are identified as promising directions.Source: §4 · pp. 15–16

A.1.1 Computation of the Root Mean Squared Error

• RMSE averages squared forecast errors over initialization times and grid points, with the initialization-time average placed inside the square root.Source: §A.1.1 · p. 17
• Latitude-dependent cosine weights correct for grid-cell area and are normalized to unit mean across included grid cells.Source: §A.1.1 · p. 17
• For a given day and lead time, only initialization times whose forecast validity falls on that day contribute to the average RMSE.Source: §A.1.1 · p. 17
• HRES D5 and D10 contours use long-term forecasts from WeatherBench 2, with HRES-fc0 as ground truth and a 45-day day-of-year window around June 28, 2021.Source: §A.1.1 · p. 17

A.1.2 Additional Figures

• Table A.1 reports long-term average HRES RMSE values for 120-hour and 240-hour lead times.Source: Table A.1 · p. 17
• Figure A.1 tracks T2m RMSE against lead time for three ML models and HRES during summer 2020 in the Pacific Northwest heatwave region.Source: Figure A.1 · p. 18
• Figure A.2 maps average temperature anomalies for June 27–29, 2021 relative to the ERA5 climatology, using forecasts initialized at 00 UTC.Source: Figure A.2 · p. 19
• Figure A.3 assigns each pixel to the model with the lowest RMSE in the argmin panel and the largest RMSE in the argmax panel.Source: Figure A.3 · p. 20

A.2.1 Shape Files

• Country boundaries define the South Asian study regions, while Köppen–Geiger Class A selects tropical grid cells for the India–Bangladesh analysis.Source: §A.2.1 · p. 21

A.2.2 Relative Humidity

• ML models require relative humidity derived from specific humidity because they provide humidity only on pressure levels, not at the surface.FourCastNet is excluded from the humid-heatwave case study because its humidity is available only above 850 hPa.Source: §A.2.2 · p. 21
• Relative humidity is computed from saturation vapor pressure, pressure-dependent saturation mixing ratio, and specific humidity.The resulting relative humidity is expressed in percent, but represents pressure-level humidity rather than surface relative humidity.Source: §A.2.2 · p. 21
• For HRES and ERA5, relative humidity can additionally be computed from 2 m temperature and 2 m dewpoint temperature.This provides the surface-level humidity information unavailable to the ML models.Source: §A.2.2 · p. 21

A.2.3 Heat Index

• The heat index uses 2 m temperature and relative humidity, first computes simplified HIs, and recomputes it with an elaborate formula when (HIs + T2m)/2 exceeds 80 °F.Corrections apply under specified low- and high-humidity temperature ranges, and heat-index values are classified by potential health consequences.Source: §A.2.3 · p. 22

A.2.4 Additional Figures

• Additional figures report humidity and temperature forecast errors, heat-index categories across models and initialization times, and the adapted heat-index categories table.These figures support the analysis of the 2023 South Asian humid heatwave.Source: §A.2.4 · p. 22, Table A.2 · p. 22, Figure A.4 · p. 23

A.3.1 Additional Figures

• Additional figures examine local 2 m temperature, wind-speed, and wind-chill behavior near College Station, Texas.The figures compare forecast errors across validity and lead times with ground-truth time series and station observations.Source: Figure A.7 · p. 26, Figure A.8 · p. 27, Figure A.9 · p. 28

Supplements

• The study uses ERA5, HRES, and TIGGE data, with cloud-optimized ERA5 and HRES versions used for the 2021 case studies.HRES access depends on initialization time and, for some forecasts, requires ECMWF MARS access.Source: Supplements · p. 29
• Forecast-generation code, preprocessed ground-truth data, model forecasts, and analysis code are publicly available under the stated licenses and repository releases.The materials cover GraphCast, PanguWeather, and FourCastNet forecasts and reproduce the analyses and figures.Source: Supplements · p. 29

S.1 Further Details on ML Models

• Supplementary materials identify the surface and pressure-level variables predicted by the ML models.The variable list includes temperature, humidity, wind, pressure, geopotential, and precipitation fields.Source: §S.1 · p. 35, Table S.1 · p. 35

S.2 Further Analysis of the 2021 Pacific Northwest Heatwave

• Unaggregated residuals reveal daily-cycle patterns, negative errors during some heatwave peaks, and positive errors after the peak.Barrier plots provide local diagnostics for grid cells near major affected cities.Source: §S.2 · p. 36, Figure S.1 · p. 37
• The spatial analysis evaluates whether forecasts predict a June 27–29 mean temperature anomaly above 12 K inside the observed contour across initialization times from 7 days to 6 hours before the event.An ideal system predicts the threshold throughout the contour and avoids exceeding it outside the contour.Source: §S.2 · p. 36
• GraphCast and HRES match the observed spatial distribution of the heatwave well, while FourCastNet often underpredicts and PanguWeather notably spills beyond the true 12 K contour.PanguWeather also produces more forecasts reaching large positive anomalies than GraphCast and HRES.Source: §S.2 · p. 36
• Area-weighted anomaly errors within the observed 12 K region show FourCastNet strongly underpredicting early, while the other models capture magnitude sooner.HRES has the smallest underprediction near the event, whereas GraphCast and PanguWeather slightly underpredict until onset; ground-truth differences limit attribution.Source: §S.2 · p. 36
• The supplementary figures also include the temperature-anomaly contour analysis and its area-weighted error panel.These visualizations summarize spatial coverage and magnitude errors within the region exceeding the 12 K anomaly threshold.Source: Figure S.2 · p. 38

S.3.1 Humid Heatwave in the Laos-Thailand region

• The Laos–Thailand region shows conclusions similar to the main India–Bangladesh humid-heatwave analysis for the April 18–21 peak.The supplementary analysis uses the same type of heat-index comparison across forecast methods and ground truths.Source: §S.3.1 · p. 39
• ML models underpredict high heat-index values relative to ERA5 ground truth computed with either pressure-level RH1000hPa or surface RH.HRES follows its HRES-fc0 ground truth more closely.Source: §S.3.1 · p. 39
• Using RH1000hPa for all methods, ML heat-index errors tend toward underprediction, whereas HRES values are slightly larger than HRES-fc0 ground truth.The comparison focuses on the daily time step at which ground-truth heat index peaks.Source: §S.3.1 · p. 39, Figure S.4 · p. 40
• The area distributions compare mean daily maximum heat-index values across danger categories for forecasts initialized six days before the event.They contrast ERA5 and HRES-fc0 ground truths and alternative humidity treatments.Source: Figure S.3 · p. 39