PaperlayerView exact revision on arXiv

Source-linked AI summary

Suitable Measures for the Potential Operational Utility of AI NWP Rainfall Forecasts Over Africa

Shruti Nath, Docko Sow, Koomi Toussaint Amoussouvi, Fenwick Cooper, Josiah Kiarie Kimani, John Bagiliko, Florian Pappenberger, Rendani Mbuvha

arXiv:2609.31775v1stat.APcs.LGphysics.ao-ph

TL;DR

AI weather models may offer lower-cost forecasting for Africa, where rainfall extremes are intensifying and many centres lack infrastructure for extended-lead physical models. This study calibrates and compares AI and physical rainfall forecasts across observational products and regional conditions; results vary by rainfall regime and elevation, with GraphCast near FGN in calibrated skill.

  • Problem

    The operational utility of rapidly developing AI NWP systems for African rainfall remains unclear, especially given infrastructure constraints, observational limitations, and model resolutions that may not capture important rainfall systems.

    Source: §1 · p. 2

  • Method

    The study post-processes GraphCast, GenCast, FGN, and IFS forecasts with IDR and evaluates calibrated CRPS against IMERG, RFEv2, and CHIRPS across seasons, rainfall regimes, elevation bands, and lead times.

    Source: §1 · p. 2, §2.3.2 · p. 6, §4 · p. 16

  • Results

    Across significant-skill areas, GraphCast and FGN show similar median improvements of around 5% across observational products; AI models generally outperform IFS in wet regions, while IFS performs better in dry, high-elevation areas.

    Source: §4 · p. 17

  • Takeaways & Limitations

    Calibrated AI forecasts show potential for computationally efficient rainfall prediction across Africa, while spatial resolution, ensemble design, and regional characteristics remain important.

    Source: §4 · pp. 16–17

    Overall calibrated performance may hide skill differences at extreme rainfall thresholds, for which dedicated threshold-weighted or threshold-specialised scores are warranted.

    Source: §4 · p. 18

Abstract

from arXiv · show

Artificial intelligence (AI)-based weather prediction is approaching the skill of physical numerical weather prediction (NWP) systems at a fraction of the computational cost. This is particularly promising for Africa, where rainfall extremes are intensifying and many forecasting centres lack the infrastructure to run physical models at extended lead times. We present a calibrated comparison of GraphCast, GenCast and the Functional Generative Network (FGN) against the physical NWP model IFS for rainfall prediction across Africa. Deterministic and probabilistic forecasts are postprocessed using Isotonic Distributional Regression and evaluated with the Continuous Ranked Probability Score against IMERG, RFEv2 and CHIRPS across seasons, wet and dry regimes, elevation zones and lead times. All models retain skill beyond climatology across most seasons and at extended lead times. AI models generally outperform IFS in wet regions, whereas IFS performs better in dry, high-elevation areas, where its finer resolution better represents orographic controls on rainfall. Across observational datasets and seasons, AI models achieve a median improvement of approximately 5% over IFS. GraphCast achieves calibrated skill comparable to the ensemble-based FGN, although FGN provides greater significant skill at longer lead times. These results highlight the potential of calibrated AI weather prediction to provide accessible and computationally efficient rainfall forecasts across Africa, while demonstrating the continuing importance of spatial resolution, ensemble design and regional characteristics.

1 Introduction

AI weather models have rapidly approached or exceeded physical NWP skill at far lower computational cost, but their operational value remains difficult to assess. This study evaluates calibrated rainfall forecasts for Africa against multiple observational products and across regional and seasonal conditions.

1 Introduction

  • AI NWP systems have quickly rivaled or exceeded physical NWP skill at a fraction of the computational cost, while rapid development has outpaced operationally relevant evaluation.Operational centres have struggled to absorb the shift from physics- to data-driven modelling.Source: §1 · p. 2
  • For Africa, infrastructure limits and rainfall extremes make AI forecasts promising, but observational dependence and coarse resolution constrain their potential utility.Data and compute are complements: AI systems still rely on observation-constrained initial conditions, and current global models are too coarse to represent many deep convective systems.Source: §1 · p. 2
  • Global evaluations alone cannot establish African rainfall utility, motivating downstream post-processing and regionally relevant assessment.The passage notes that statistical post-processing can improve forecast skill and reliability, particularly over tropical Africa.Source: §1 · p. 2
  • The study calibrates GraphCast, GenCast, FGN, and IFS forecasts with IDR and evaluates CRPS against IMERG, RFEv2, and CHIRPS across seasons, rainfall regimes, and elevation bands.Using multiple observational products helps distinguish model-skill differences from artefacts specific to a single verification dataset.Source: §1 · p. 2

2.1 Numerical Weather Prediction (NWP) models

The study evaluates physical and AI NWP forecasts at observational-product resolution, focusing mainly on years from 2020 onward. It compares IFS with deterministic GraphCast and probabilistic GenCast and FGN configurations.

2.1 Numerical Weather Prediction (NWP) models

  • Evaluation uses observational products’ native resolutions and mainly years after 2020, while allowing some overlap with AI models’ operational fine-tuning period.The study favors a larger evaluation sample despite this data leakage.Source: §2.1 · p. 3, Table 1 · p. 3

2.1.1 Physical NWP benchmark

  • IFS is a 50-member ensemble whose resolution increased from approximately 18 km to 9 km in June 2023, and later cycle changes make it nonstationary during evaluation.The passage advises interpreting small system differences with that benchmark variation in mind.Source: §2.1.1 · p. 3

2.1.2 Deterministic AI NWPs

  • GraphCast is a deterministic autoregressive graph-neural-network model, trained on ERA5 and further fine-tuned on HRES-fc0 for this study.Its encode–processor–decode architecture maps grid states to an icosahedral mesh for learned message passing before decoding them back to the grid.Source: §2.1.2 · p. 3

2.1.3 Probabilistic AI NWPs

  • GenCast samples diffusion-based ensembles conditioned on prior states, whereas FGN generates ensembles through functional parameter perturbations.FGN captures epistemic uncertainty with four independently trained models and aleatoric uncertainty by sampling network parameters.Source: §2.1.3 · pp. 3–4

2.2 Observational Data

Because African precipitation observations are sparse and datasets disagree, the study verifies forecasts independently against three satellite–gauge blended products. Each product has distinct inputs, coverage, and known limitations.

2.2 Observational Data

  • The study treats disagreement among three independently evaluated precipitation products as an indication of observational uncertainty rather than pooling their verification statistics.Sparse and declining rain-gauge coverage means no single dataset is treated as unambiguous truth.Source: §2.2 · p. 4

2.2.1 Integrated Multi-satellitE Retrievals for GPM (IMERG)

  • IMERG Final Run V07 is a 0.1° satellite–gauge product adjusted to a monthly gauge combination, with documented reduced skill over complex terrain and coastal zones.Its inputs include passive-microwave estimates blended with geostationary infrared estimates.Source: §2.2.1 · p. 4

2.2.2 Climate Hazards Group InfraRed Precipitation with Station data (CHIRPS)

  • CHIRPS blends a high-resolution precipitation climatology, satellite cold-cloud-duration estimates, and station observations, but accuracy degrades in station-sparse rural regions.It is a land-only dataset available at 0.05° resolution.Source: §2.2.2 · p. 4

2.2.3 Rainfall Estimate (RFE) version 2

  • RFEv2 provides daily African rainfall estimates at 0.1° by combining satellite imagery, microwave estimates, and rain-gauge reports.Low GTS station reporting rates degrade estimates in some regions.Source: §2.2.3 · p. 5

2.3 Verification

The verification framework compares calibrated probabilistic forecasts with climatology and IFS, using CRPS-based measures to assess skill. It evaluates performance across seasons, rainfall regimes, elevation zones, lead times, and observational products.

2.3.1 Potential Continuous Ranked Probability Score

  • CRPS is the primary verification metric; ΔCRPSpot compares calibrated performance with climatology, while CRPSS compares calibrated performance with IFS.CRPS is lower for better forecasts; ΔCRPSpot > 0 indicates worse performance than climatology, and CRPSS > 0 indicates improved resolution relative to IFS.Source: §2.3.1 · pp. 5–6
  • CRPScal estimates each model’s score after optimal calibration, providing a post-processing proxy for operational probabilistic skill and enabling comparison despite differing calibration properties.IDR is applied to GraphCast, GenCast, FGN, and IFS; under blocked out-of-sample recalibration, small negative miscalibration values may occur.Source: §2.3.1 · p. 5

2.3.2 Isotonic Distributional Regression (IDR) calibration and Blocked Cross-Validation

  • IDR estimates an observation CDF conditional on forecast values, fitted separately by model, lead time, observational product, and climate zone.Blocked evaluation holds out two consecutive years at a time across 2020–2025 to estimate out-of-sample CRPScal; zonal pooling may be strained near sharp climatological gradients.Source: §2.3.2 · p. 6

2.3.3 Decomposition across elevation and areas receiving significant rainfall

  • Verification is stratified by FMAM, JJAS, and ONDJ, and by all, wet, and dry grid cells, to avoid continental averages masking regional skill differences.Wet-grid masks use a monthly climatological precipitation threshold of 1 mm day−1 or more and are computed separately for each observational product.Source: §2.3.3 · pp. 6–7
  • Within significant-rainfall areas, forecasts are also evaluated across four elevation bands at lead times of 1, 3, 5, 7, and 9 days.The bands are < 500 m, 500–1000 m, 1000–2000 m, and > 2000 m.Source: §2.3.3 · p. 7

3 Results

AI models show skill gains over IFS across much of Africa, especially in wet regions, while performance varies by season, elevation and observational product. Their median skill gain is about 5%, but IFS retains advantages in some dry, high-elevation areas.

3.1 Overall performance across seasons and elevations

  • In “Only Wet” areas, AI models improved resolution over IFS by 0.007 for IMERG and roughly 0.0035 for RFEv2 and CHIRPS; dry-area outcomes differed sharply for CHIRPS.For CHIRPS dry areas, all models were worse than climatology, with GraphCast’s degradation around 0.001 versus 0.015–0.02 for other models.Source: §3.1 · p. 7
  • Wet-area skill gaps generally narrowed at higher elevations, while IFS had a consistent dry-area advantage above 1,000 m for CHIRPS and RFEv2.IMERG still showed a notable AI advantage at 1,000–2,000 m in wet areas.Source: §3.1 · p. 9, Figure 5 · p. 12, Figure 6 · p. 12
  • JJAS had the worst wet-area skill across products and models; IFS lost resolution beyond climatology at 5–7 days, while AI models did so at 7 days for RFEv2 and CHIRPS but not IMERG.The authors relate this pattern to challenges forecasting West African monsoon rainfall and organized mesoscale convective systems.Source: §3.1 · p. 8, Figure 4 · p. 11

3.2 Spatial distribution and significance

  • AI models gained skill over most of the continent; Eastern and Southern gains reached 7–10% in FMAM and ONDJ, while Cameroon–Gabon exceeded 15% in JJAS at day 1 before fading.The Cameroon–Gabon gain declined to 2–5% at longer lead times.Source: Figure 7 · p. 13, §3.2 · pp. 13–14, Figure 8 · p. 14
  • Most degraded areas along the Sahel and Ethiopian Highlands were not statistically significant, but the Eastern African coastline retained significant gains across lead times for both AI models.The authors suggest ensemble averaging may dampen FGN’s error growth relative to GraphCast, whose skill degrades faster with lead time in FMAM and ONDJ.Source: Figure 7 · p. 13, Figure 8 · p. 14, §3.2 · p. 14
  • Median CRPSS values were close to 0.05 (about 5% skill gain relative to IFS) across lead times and products, except CHIRPS at 7–9 days in FMAM and JJAS.The boxplots include only grid cells with statistically significant skill changes; CHIRPS also shows more variable performance, with FMAM and ONDJ lower quartiles of at least -0.25 and upper quartiles up to 0.3.Source: §3.2 · pp. 14–15, Figure 10 · p. 16

4 Conclusion and Discussion

The study uses Isotonic Distributional Regression to compare calibrated rainfall forecasts from AI systems and IFS against local observational products. AI models retain skill beyond climatology longer than IFS in JJAS, while performance varies by rainfall regime, elevation, and model design.

4 Conclusion and Discussion

  • Isotonic Distributional Regression post-processes IFS and AI forecasts toward local observations and yields calibrated cumulative distribution functions for probabilistic scoring.This allows calibrated skill estimates for deterministic and probabilistic forecast systems.Source: §4 · p. 16
  • In JJAS, IFS loses resolution beyond climatology at 5–7 days, while AI models do so at 7–9 days.The study reports that all NWPs retain skill beyond climatology across seasons; the lead-time contrast is specific to JJAS.Source: §4 · p. 16
  • Figure 11 displays Diebold–Mariano test p-values for FGN and GraphCast relative to IFS across lead times, seasons, and three observational products.P-values below 0.1 indicate a significant potential CRPS loss differential; whiskers show 90% confidence intervals.Source: Figure 11 · p. 17
  • IFS improves over AI models in dry areas above 2000 m for CHIRPS and RFEv2, while no significant IFS gain appears in high-elevation wet areas.The authors suggest IFS’s 9 km resolution captures dryness-related subsidence, whereas rainfall-driving convection around 4 km remains sub-grid.Source: §4 · p. 17
  • GraphCast nearly matches FGN in calibrated performance, with both showing median improvements around 5%; FGN gains significant skill over more area at longer leads.The authors connect FGN’s longer-lead advantage to ensemble averaging that dampens error growth relative to a single deterministic trajectory.Source: §4 · p. 17
  • Pointwise calibration recovers each grid point’s marginal distribution but not the spatial and temporal coherence of forecast members needed by some downstream applications.Thus, the reported equivalence in calibrated skill is pointwise, not equivalence in all information users may act on.Source: §4 · p. 18
  • The evaluation does not settle skill at extreme rainfall thresholds, and uneven verification periods plus possible training–evaluation overlap constrain the comparisons.For relative skill against IFS, the authors considered 2024–2025, which they describe as largely out of sample and still showing consistent gains.Source: §4 · p. 18

5 Outlook

The outlook calls for closer coordination among forecast providers, model developers, and operational forecasters as AI systems and physical models evolve. It highlights prospective verification, improved observations, and analysis of learned model representations as routes to monitoring and improving forecast quality.

5 Outlook

  • Prospective, real-time verification could become the primary standard for operational readiness as AI model training and update cycles advance.The authors propose embedding verification in operational centres as continuous quality monitoring, including tracking when training periods encroach on evaluation periods.Source: §5 · p. 18
  • The authors call for closer coordination among forecast providers, model developers, and operational forecasters to identify discrepancies between apparent and genuine skill as systems evolve.Changes to IFS cycles can affect analysis used to initialize forecasts and define training distributions; cited evidence reports sensitivity in fine-tuned AI models to a new IFS cycle.Source: §5 · p. 18
  • Improved observational networks could reduce errors in initial conditions that otherwise contribute to rapid forecast error growth.The authors caution that statistical amelioration by AI may not reflect physically consistent corrections.Source: §5 · p. 18
  • Understanding AI models’ internal representations could reveal where and why they outperform physical NWP and guide targeted improvements to physical models.The authors describe a feedback loop in which improved physical models provide better initial conditions and training data for future AI systems.Source: §5 · pp. 18–19

Supplementary Data

The supplementary figures show calibrated CRPS over lead times and spatial CRPS maps for IFS across seasons and observational products. They also compare raw and calibrated forecast outputs, with a separate lead-time view restricted to significantly wet areas.

Supplementary Data

  • Figure S1 compares raw and IDR-based CRPS across lead times for GraphCast, GenCast, FGN, and IFS, averaged over all grid points.The figure includes climatology as a dashed reference line.Source: Figure S1 · p. 20
  • Figure S2 reports CRPS averaged over only significantly wet areas for each month before annual averaging; Figures S3–S5 show calibrated IFS CRPS maps toward IMERG, RFEv2, and CHIRPS across three lead times and seasons.Source: Figure S2 · p. 21, Figure S3 · p. 22, Figure S4 · p. 23
Loading 2609.31775v1…