PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI Forecast the Extremes? Evaluating Probabilistic Weather Models on Tropical Cyclones

Nathan Chalumeau, Xavier Renard, Remi Meynadier, Hugo Rakotoarimanga, Marcin Detyniecki, Emmanuel de Bezenac

arXiv:2610.22077v1cs.LGphysics.ao-ph

TL;DR

Whether probabilistic AI weather models can forecast rare extremes remains less understood than their performance on standard benchmarks, particularly for tropical cyclones. This paper benchmarks three such models against IFS-ENS using probabilistic tracking, intensity, calibration, and storm-structure diagnostics. AI models improve cyclone tracking and spatial placement, while intensity and calibration errors remain.

  • Problem

    Earlier evidence on extreme-weather shortcomings focused largely on deterministic models, leaving probabilistic models’ extreme-event forecasting and ensemble uncertainty less understood.

    Source: §1 · p. 1

  • Method

    The paper benchmarks GenCast, FourCastNet 3, and FGN against IFS-ENS across tropical cyclones using probabilistic tracking, intensity, calibration, and storm-structure diagnostics.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    FGN outperforms IFS-ENS on probabilistic track scores for about 90% of the 41 retained storms, while AI models still show intensity biases and calibration errors.

    Source: §4 · p. 6, FGN 90% · p. 7, §7 · p. 9

  • Takeaways & Limitations

    Evaluation beyond aggregate global scores, decomposing location, intensity, uncertainty, and dependence, can identify what needs improvement and what should be preserved.

    Source: §7 · p. 10

    Operational AI models retain weak-wind bias and calibration errors, and the evaluation does not test cyclone-occurrence probability calibration using cases without observed cyclones.

    Source: IFS-ENS · p. 5, FGN 90% · p. 7

Abstract

from arXiv · show

Weather prediction is undergoing an important shift, with data-driven AI models outperforming leading numerical weather prediction systems on average at a fraction of the computational cost. Yet, whether this success extends to extreme events remains contested. Earlier evaluations have exposed important shortcomings, but have largely focused on deterministic models and metrics, leaving the capabilities of the latest probabilistic models less understood. We conduct an extensive benchmark of tropical cyclone forecasting, comparing GenCast, FourCastNet 3, and FGN with ECMWF's operational physics-based ensemble, IFS-ENS. We develop a suite of probabilistic metrics and diagnostic analyses to assess cyclone tracking, intensity biases, uncertainty, and the wind-pressure relationship that characterizes storm structure. We find that, despite remaining limitations in intensity and ensemble spread, leading AI models substantially outperform IFS-ENS in cyclone tracking: FGN achieves better probabilistic location scores for 90% of evaluated storms in 2025. Further analyses reveal model-specific failure modes and suggest a lingering influence of pretraining data, pointing to opportunities for targeted improvement. Nonetheless, these results suggest that AI models are capable of capturing physical relationships that remain predictive well beyond typical weather conditions, motivating further investigation into how these capabilities emerge and how to strengthen them.

1 INTRODUCTION

The success of learned weather models on standard benchmarks leaves open whether probabilistic models can forecast rare extremes and represent their uncertainty. This benchmark evaluates three probabilistic AI models on tropical cyclones and finds stronger tracking performance than IFS-ENS alongside remaining intensity and calibration errors.

1 INTRODUCTION

  • Much earlier evidence on extreme-weather shortcomings focused on deterministic models, motivating renewed assessment of probabilistic models and their ensemble uncertainty.Prior studies raised concerns about record-breaking weather and severe tropical cyclones, including whether performance depends on training-data representation.Source: §1 · pp. 1–2
  • Tropical cyclones are consequential but rare, occupying about 0.5% of grid points in global climate simulations, while preparedness depends on forecasts of location, intensity, and uncertainty.Their rarity and consequences make them a testbed for learned models as they enter operational and cyclone forecasting trials.Source: §1 · p. 2
  • The study compares GenCast, FourCastNet 3, and FGN with IFS-ENS across 178 tropical cyclones, assessing location, intensity, calibration, and wind–pressure dependence.The evaluation uses reanalysis and best-track data to assess forecast skill and storm properties.Source: §1 · p. 2
  • FGN outperforms IFS-ENS on probabilistic track scores for about 90% of the 41 retained storms in 2025, while intensity biases and calibration errors persist.The results distinguish progress in storm location and spatial structure from unresolved errors in intensity and uncertainty.Source: §1 · p. 2

2 EXPERIMENTAL SETUP

The benchmark evaluates three probabilistic AI models in reanalysis and operational settings, using model-specific evaluation windows and a retrospective cyclone forecast protocol. The models differ in training objectives and how they represent uncertainty.

2 EXPERIMENTAL SETUP

  • Evaluation years are chosen according to each model’s training and fine-tuning periods.The two settings use different models, versions, and references, so their comparison does not isolate the effect of fine-tuning.Source: Figure 1 · p. 2, §2 · p. 3
  • GenCast, FCN3, and FGN represent uncertainty differently: conditional diffusion, stochastic perturbations, and ensembles combining independently trained networks with learned perturbations.FCN3 is trained with CRPS in physical and spectral space; FGN optimizes location-wise CRPS.Source: §2 · p. 3
  • The reanalysis setting evaluates ERA5-trained GenCast and FCN3 against ERA5 and IBTrACS, while the operational setting evaluates ERA5-trained, HRES-fc0-fine-tuned GenCast and FGN against HRES-fc0 and IBTrACS.The design examines whether ERA5-related intensity biases remain after operational fine-tuning.Source: §2 · p. 3
  • The retrospective evaluation selects 178 IBTrACS cyclones reaching at least 64 knots and runs 15-day forecasts with 50 reanalysis or 49 operational ensemble members.Forecasts begin 12 hours before the first reported wind observation, with timing adjusted to the preceding forecast cycle when needed.Source: §2 · p. 3
  • For Typhoon Bualoi, forecast dots show individual members, contours show density, and black dots mark observed positions; IFS-ENS members form distinct intermediate-lead clusters.An ensemble mean cannot represent those separate clusters.Source: Figure 2 · p. 3

3 REANALYSIS SETTING: EVALUATION OF CYCLONE FORECASTS ON ERA5-TRAINED MODELS

In the reanalysis setting, probabilistic location scoring favors GenCast over IFS-ENS for most retained storms, while the evaluation also highlights limits of aggregate scores and detected-storm-only assessment. Cyclone-local CRPS and global CRPS provide different views of forecast skill.

3.1 PREDICTING CYCLONE LOCATIONS

  • Ensemble-mean position errors cannot distinguish forecasts with the same mean but different spreads or paths, motivating scores of density at observed storm positions.For Typhoon Bualoi, IFS-ENS members separate into clusters at intermediate leads that the mean cannot represent.Source: §3.1 · p. 4
  • Location NLL scores observed IBTrACS positions under a KDE of predicted positions, averaging first within storms and then across storms with equal storm weight.The evaluation requires at least 10 identified members per model at five or more common lead times, retaining 106 of 125 storms.Source: §3.1 · p. 4
  • GenCast has the lowest mean NLL and outperforms IFS-ENS on 68% of retained storms and FCN3 on 62%.The table reports trajectory NLL and detection rates; its right-hand comparison covers the 106 retained storms.Source: §3.1 · p. 4, Table 1 · p. 5
  • Detection rates show the same model ranking as NLL but no additional skill advantage; occurrence-probability calibration is outside the evaluation’s scope because non-cyclone cases are not assessed.NLL includes only members in which the storm is detected and associated with the observed storm.Source: §3.1 · p. 4

3.2 GLOBAL SCORES, LOCAL SKILL, AND INTENSITY BIAS

  • Global CRPS compares forecast distributions with ERA5 at grid points across seven normalized atmospheric variables, while cyclone-local CRPS focuses on regions enclosing observed tracks.Lower CRPS is better; global averages can conceal differences around cyclones, and local scores can conceal peak-wind weaknesses.Source: §3.2 · p. 4
  • GenCast and FCN3 have very similar global CRPS, consistent with Bonev et al. (2025), despite FCN3’s CRPS-based training objective.Source: §3.2 · p. 4

IFS-ENS

The reanalysis comparison shows that better cyclone-local CRPS does not guarantee lower peak-wind bias. ERA5’s own wind underestimation also limits what agreement with that reference establishes.

IFS-ENS

  • GenCast’s cyclone-region CRPS reduction over FCN3 is 3–5% from 24–220 h, versus about 1% globally, yet GenCast underestimates peak winds at short and intermediate leads.The intensity comparison uses ensemble-mean maximum 10m wind speed normalized by ERA5’s mean maximum wind of 16.6 m/s.Source: Figure 3 · p. 5
  • Global averages conceal cyclone-region differences, and even lower cyclone-local CRPS does not establish better prediction of every storm property or identify the source of the advantage.The evaluation therefore separates spatial agreement from intensity.Source: IFS-ENS · p. 5
  • Because ERA5 itself underestimates cyclone maximum winds, agreement with ERA5 does not establish realistic intensity, nor does IFS-ENS’s positive bias establish storms that are too strong.The subsequent comparison assesses fine-tuned models against HRES-fc0.Source: IFS-ENS · p. 6

4 OPERATIONAL SETTING: TRACKING, INTENSITY, AND SPATIAL AGREEMENT FOR MODELS FINE-TUNED ON OPERATIONAL ANALYSES

In the 2025 operational comparison, FGN and operational GenCast outperform IFS-ENS on probabilistic cyclone tracking, with FGN ahead. Detection-rate comparisons remain setting-dependent, and the cause of IFS-ENS’s lower detection is unresolved.

Cyclone tracking

  • FGN assigns greater probability density to observed locations than IFS-ENS on approximately 90% of 41 retained storms and has the lowest mean NLL.Operational GenCast outperforms IFS-ENS on 78% of storms; FGN ranks first among all three models on 59%, versus 34% for GenCast and 7% for IFS-ENS.Source: §4 · p. 6, Table 2 · p. 6
  • Detection rates follow the NLL ordering, with both AI models ahead of IFS-ENS, but the operational comparison separates them more clearly than 2021–2023.The difference reflects higher detection for operational GenCast and lower detection for IFS-ENS on the 2025 storms.Source: §4 · p. 6
  • Because the numerical baseline changes between evaluation settings, their detection-rate differences cannot be attributed to AI fine-tuning alone, and the cause of IFS-ENS’s lower detection remains unresolved.Source: §4 · p. 6

FGN 90%

FGN leads the operational models in spatial diagnostics and cyclone-region CRPS, while its peak-wind bias remains similar to GenCast’s. The spatial comparison isolates placement and shape from uniform intensity shifts.

Global and cyclone-local CRPS

  • FGN has lower CRPS than GenCast and IFS-ENS globally and around cyclones, and both AI models improve more over IFS-ENS in cyclone regions than globally.Source: FGN 90% · p. 6

Spatial agreement

  • FGN performs best on spatial diagnostics, followed by GenCast and IFS-ENS, at nearly all lead times; storm-footprint metrics show the same ordering for strong winds and low pressures.The diagnostics threshold pooled member values for each model, removing sensitivity to uniform intensity shifts while retaining errors in spatial placement and shape.Source: FGN 90% · pp. 6–7, Figure 4 · p. 7

Intensity and calibration

  • FGN and GenCast have similar negative peak-wind biases at short and intermediate leads, despite FGN’s lower CRPS and better spatial scores.Both tend to underestimate maximum 10m winds relative to HRES-fc0, while IFS-ENS remains closer to that reference.Source: FGN 90% · p. 7
  • Operational fine-tuning has not eliminated the AI models’ weak-wind bias, and later reductions may partly reflect storm dissipation rather than improved forecasts of active storms.Rank histograms also show calibration errors.Source: FGN 90% · p. 7

5 DO THE MODELS REPRODUCE THE WIND–PRESSURE RELATIONSHIP?

The analysis tests whether models preserve the wind–pressure relationship despite biased storm intensities. Across the gridded sources, dependence is similar even though FGN and GenCast predict weaker winds and shallower pressure minima than HRES-fc0.

Wind–pressure relationship

  • Because stronger tropical cyclones generally have higher winds and lower central pressure, the analysis tests whether models reproduce this relationship despite systematic intensity underestimation.This is relevant to FGN’s CRPS objective, which scores variables and locations separately, unlike GenCast’s diffusion approach to learning joint atmospheric fields (Alet et al., 2025; Price et al., 2024).Source: §5 · p. 7

Comparing predicted storms

  • FGN and GenCast produce weaker winds and shallower pressure minima than HRES-fc0, with distributions closer to ERA5 despite operational fine-tuning.The comparison pairs each ensemble member’s strongest 10m wind and lowest sea-level pressure within the cyclone region.Source: §5 · p. 7, Figure 5 · p. 8

Separating dependence from intensity

  • After rank-transforming wind and pressure to identical marginals, the dependence remains consistent: rank correlations are −0.83 for FGN, −0.84 for GenCast, and −0.88 for HRES-fc0.The qualitative result also holds in the earlier evaluation and across separate lead-time ranges.Source: §5 · p. 8

Wind–pressure relationship

  • The coexistence of biased intensity and a realistic wind–pressure association indicates that intensity and dependence errors are distinct, but does not establish an explicit learned physical mechanism or correct wind at a given pressure.Source: §5 · p. 8

6 RELATED WORK

Prior evaluations of learned weather forecasts report mixed results across extremes, and cyclone benchmarks differ in models, settings, and scoring protocols. This paper’s wind–pressure diagnostic complements earlier cyclone-physics studies while not establishing a learned causal mechanism.

AI forecasts of extreme weather

  • Evaluations of extreme-event skill are mixed: Olivetti & Messori (2024) find skill varies by event, region, and lead time, while Zhang et al. (2026) find numerical forecasts superior on record-breaking extremes.For cyclones, DeMaria et al. (2024) report competitive tracking alongside substantial intensity underestimation.Source: §6 · p. 8

Probabilistic cyclone evaluation

  • TCBench evaluates one-to-five-day cyclone forecasts with probabilistic scores and other diagnostics, while GEFS outperforms its evaluated AI ensembles on track CRPS.Different models, seasons, references, and scoring protocols preclude direct comparison with this paper’s rankings.Source: §6 · p. 8

Physical relationships in learned forecasts

  • FGN and GenCast have weaker winds and shallower pressures than HRES-fc0, but the gridded sources show nearly the same wind–pressure dependence.IBTrACS sustained winds differ in definition from the gridded sources’ instantaneous winds.Source: Figure 6 · p. 9
  • The joint wind–pressure analysis adds a distributional diagnostic across storms and ensemble members to prior studies of cyclone patterns and sensitivity to initial conditions (Sahu et al., 2025; Ba˜no-Medina et al., 2025).Agreement with a physical relationship does not establish the underlying learned mechanism or a causal effect of pretraining data.Source: §6 · p. 9

7 CONCLUSION

The benchmark finds that probabilistic AI models can outperform numerical forecasts on cyclone location and surrounding fields, while retaining intensity and calibration errors. The authors recommend more targeted diagnostics and broader evaluation to identify what needs improvement and what spatial skill or wind–pressure structure should be preserved.

7 CONCLUSION

  • AI models can outperform numerical forecasts on cyclone location and surrounding fields while still exhibiting intensity biases and calibration errors.Systematically weak storms alone are insufficient to conclude a general failure in cyclone representation.Source: §7 · p. 9
  • FGN shows wind–pressure dependence comparable to GenCast despite training only on conditional marginals, indicating that meaningful joint structure can emerge without explicit enforcement by the training objective.Source: §7 · p. 9
  • Comparisons with observations and higher-resolution analyses could help distinguish errors inherited from training targets from limitations of the models themselves.Source: §7 · p. 9
  • Potential remedies—including intensity correction, extreme-wind-weighted losses, and improved uncertainty quantification—should be assessed for effects on intensity and for preserving spatial skill and wind–pressure dependence.Source: §7 · p. 9
  • Evaluation across additional seasons, including periods without observed cyclones, would test gain robustness and cyclone-occurrence calibration; the study is retrospective and does not establish operational-warning suitability.Operational use would require evaluation under operational conditions, including cases without cyclones.Source: §7 · p. 10, ETHICS STATEMENT · p. 10

A EVALUATION PROTOCOL AND REPRODUCIBILITY

The study evaluates archived probabilistic forecasts of selected historical cyclones using common verification targets and a suite of location, field, intensity, spatial, and dependence diagnostics. Subgroup comparisons mostly track IFS-ENS variation, while the 2021–2023 models show wind–pressure dependence close to ERA5.

A EVALUATION PROTOCOL AND REPRODUCIBILITY

  • The retrospective benchmark selects 178 cyclones with maximum reported sustained winds of at least 64 knots and runs 15-day forecasts with 50 members in 2021–2023 and 49 in 2025.Selection by lifetime maximum evaluates selected known events, not unconditional cyclone occurrence; equal ensemble size is not an equal-compute comparison.Source: A EVALUATION PROTOCOL AND… · p. 14
  • Field comparisons use 0.25° grids at common 12-hour intervals and sum area-weighted fair-CRPS scores across seven variables; these are marginal, not joint multivariate, scores.The two cohorts use different field references, ERA5 and HRES-fc0, so absolute field scores should not be compared across them.Source: A EVALUATION PROTOCOL AND… · pp. 14, 16
  • Track density scores use matched member trajectories and observed IBTrACS positions, with common eligible lead times and retained storms so all models are scored on the same cases.Storms require at least five lead times with ten or more matched members per model; 106 storms are retained in 2021–2023 and 41 in 2025.Source: A EVALUATION PROTOCOL AND… · pp. 15–16
  • Intensity bias and calibration diagnostics compare member regional maximum winds with the reference and use rank histograms, whose frequencies are averaged across storms.The reported percentage bias is scaled by the cohort’s averaged reference maximum, not averaged storm-specific percentage errors.Source: A EVALUATION PROTOCOL AND… · p. 16
  • Spatial-mask diagnostics threshold references and pooled member values by percentile to remove uniform model intensity bias while retaining within-model member intensity differences.Quantile masks do not separate spatial displacement from shape differences.Source: A EVALUATION PROTOCOL AND… · pp. 16–17
  • The wind–pressure analysis pools paired regional extrema with equal row weights, ranks each source’s variables separately, and estimates Spearman dependence with storm-resampled intervals.Source-specific missingness, temporal weighting, and pressure-bin approximation remain relevant caveats.Source: A EVALUATION PROTOCOL AND… · p. 17

B.1 STABILITY ACROSS STORM SUBGROUPS

  • Across 2021–2023 and 2025, subgroup comparisons show AI models mostly following IFS-ENS patterns; in 2025 IFS-ENS has the highest NLL and CRPS in every subgroup, while FGN is lowest in all but two.GenCast is marginally ahead on NLL in category 3 and CRPS in the South Pacific; some 2025 basin groups contain only one storm.Source: §B.1 · p. 17, Figure 7 · p. 18

B.2 WIND–PRESSURE RELATIONSHIP

  • In the ERA5 setting, GenCast, FCN3, and IFS-ENS have nearly the same wind–pressure dependence as ERA5, with overlapping reported intervals.Their pressures are slightly shallower and winds slightly weaker than ERA5, so the differences concern intensity rather than how wind varies with pressure.Source: §B.2 · p. 18

B.3 RANK HISTOGRAMS

Rank histograms compare reference values with the ranks of ensemble members: flat frequencies indicate calibration, while shape and asymmetry diagnose spread and bias. In both evaluation settings, histogram patterns vary by model and forecast lead, and apparent long-lead uniformity has a storm-dissipation caveat.

B.3 RANK HISTOGRAMS

  • A flat rank histogram is the calibrated-ensemble reference; U shapes indicate insufficient spread, T shapes excessive spread, and asymmetry bias.A high last rank means reference winds exceed every member’s, while a high first rank means the opposite.Source: §B.3 · p. 18
  • Both AI models have U-shaped 2021–2023 footprint histograms, indicating insufficient spread; GenCast’s excess at the last rank indicates footprint winds weaker than ERA5.At short leads both AI models underestimate wind, FCN3 also does so at intermediate leads, and both are close to uniform at long leads.Source: §B.3 · p. 18, Figure 10 · p. 20
  • In 2025, FGN and GenCast have U-shaped footprint and intermediate-lead histograms, while IFS-ENS has a T shape on footprints and at short and intermediate leads.These shapes indicate too little spread for the AI models and too much spread for IFS-ENS; the AI models also show some wind underestimation.Source: §B.3 · p. 20
  • At long leads in 2025, all three models’ histograms are close to uniform, but most storms have faded out by then.Source: §B.3 · p. 20

B.4 CORRELATION BETWEEN TRACK AND INTENSITY ERRORS

Across these diagnostics, better track placement does not ensure accurate intensity, while FGN and GenCast outperform IFS-ENS on spatial agreement measures. Energy spectra broadly match their references at large scales, with model differences emerging at small scales.

B.4 CORRELATION BETWEEN TRACK AND INTENSITY ERRORS

  • Track and intensity errors are moderately linked, but similar track errors span wide intensity errors, so accurate tracks do not guarantee accurate intensity.IFS-ENS has the weakest correlations in every case; the analysis averages each matched member’s track distance and absolute pressure or wind error over forecast times.Source: §B.4 · p. 20

B.5 SPATIAL AGREEMENT

  • IFS-ENS scores worst on every spatial metric for both pressure and wind, while FGN scores best, closely followed by GenCast; gaps are largest before about 180 hours.The comparisons use storm-field masks and footprints, with percentile thresholds intended to reduce sensitivity to uniform intensity shifts while retaining placement and shape errors.Source: §B.5 · pp. 20, 24, Figure 16 · p. 24, Figure 17 · p. 25

B.6 ENERGY SPECTRA

  • At large scales, all models follow their reference spectra; at small scales, FCN3 matches ERA5, while 2025 operational models follow HRES-fc0 and FGN exceeds it.GenCast has slightly less small-scale energy than ERA5 in 2021–2023; in 2025, GenCast is close to HRES-fc0 and IFS-ENS has more wind energy than HRES-fc0.Source: §B.6 · pp. 24–25
Loading 2610.22077v1…