PaperlayerView exact revision on arXiv

Source-linked AI summary

AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score

Simon Lang, Mihai Alexe, Mariana C. A. Clare, Christopher Roberts, Rilwan Adewoyin, Zied Ben Bouallègue, Matthew Chantry, Jesper Dramsch, Peter D. Dueben, Sara Hahner, Pedro Maciel, Ana Prieto-Nemesio, Cathal O'Brien, Florian Pinault, Jan Polster, Baudouin Raoult, Steffen Tietsche, Martin Leutbecher

arXiv:2412.15832v1physics.ao-ph

TL;DR

Machine-learning weather ensembles need to represent forecast uncertainty, while first-generation MSE-trained ensembles tend to have too little spread. AIFS-CRPS trains a stochastic forecasting model with a proper-score objective. It outperforms the IFS ensemble for most medium-range upper-air fields and surface variables, and shows lower biases and increased MJO skill than the operational subseasonal system at longer ranges.

  • Problem

    First-generation machine-learning ensembles tend to have too little spread, while useful weather forecasts need to represent uncertainty and provide calibrated probabilities.

    Source: §1 · p. 2

  • Method

    AIFS-CRPS is a stochastic AIFS variant trained with a proper-score objective, using the almost fair CRPS and shaping Gaussian noise to represent model uncertainty.

    Source: §1 · p. 2

  • Results

    AIFS-CRPS has higher skill than the 9 km physics-based IFS medium-range ensemble for most upper-air fields and surface variables, and shows lower biases and increased MJO skill than the operational subseasonal system at two to six weeks.

    Source: §7 · pp. 13–14

  • Takeaways & Limitations

    Training with a proper score such as afCRPS can yield a highly skilful ensemble system that preserves realistic variability during long rollouts.

    Source: §7 · p. 13

    AIFS-CRPS is over-dispersive for some variables, such as 500 hPa geopotential in the early medium range, with the authors linking this likely to the IFS initial-condition singular-vector perturbations.

    Source: §6 · p. 13

Abstract

from arXiv · show

Over the last three decades, ensemble forecasts have become an integral part of forecasting the weather. They provide users with more complete information than single forecasts as they permit to estimate the probability of weather events by representing the sources of uncertainties and accounting for the day-to-day variability of error growth in the atmosphere. This paper presents a novel approach to obtain a weather forecast model for ensemble forecasting with machine-learning. AIFS-CRPS is a variant of the Artificial Intelligence Forecasting System (AIFS) developed at ECMWF. Its loss function is based on a proper score, the Continuous Ranked Probability Score (CRPS). For the loss, the almost fair CRPS is introduced because it approximately removes the bias in the score due to finite ensemble size yet avoids a degeneracy of the fair CRPS. The trained model is stochastic and can generate as many exchangeable members as desired and computationally feasible in inference. For medium-range forecasts AIFS-CRPS outperforms the physics-based Integrated Forecasting System (IFS) ensemble for the majority of variables and lead times. For subseasonal forecasts, AIFS-CRPS outperforms the IFS ensemble before calibration and is competitive with the IFS ensemble when forecasts are evaluated as anomalies to remove the influence of model biases.

1 Introduction

Ensemble forecasts represent uncertainty, but ensembles built from first-generation MSE-trained machine-learning models tend to have too little spread. AIFS-CRPS takes a different probabilistic-training approach, using a proper-score objective, and is reported to be competitive with or superior to the physics-based IFS ensemble across forecast ranges.

1 Introduction

  • First-generation machine-learning ensembles tend to have too little spread, consistent with MSE-trained models struggling to represent atmospheric variability.Ensemble forecasts estimate the probability density of the future atmospheric state and aim for calibrated probabilities.Source: §1 · p. 2
  • Probabilistic machine-learning models, including diffusion-based approaches, have achieved competitive ensemble scores and more stable statistics than deterministically trained models.The cited work reports that they do not smooth small-scale structures with forecast lead time.Source: §1 · p. 2
  • AIFS-CRPS uses a probabilistic proper-score objective and learns to represent model uncertainty through shaping Gaussian noise.Its training uses the almost fair CRPS, a modification of the fair CRPS.Source: §1 · p. 2
  • AIFS-CRPS is more skillful than the physics-based IFS ensemble for most upper-air fields and surface variables, and performs competitively or better from days to subseasonal ranges.The introduction characterizes it as a highly skilful machine-learned ensemble forecast model.Source: §1 · p. 2

2 Probabilistic training

AIFS-CRPS trains a stochastic ensemble using an almost fair CRPS objective, with rollout training and a modified state update intended to mitigate accumulated errors. Its training uses staged rollouts and GPU sharding to support larger ensembles, models, and resolutions.

2 Probabilistic training

  • The model follows AIFS’s encoder-processor-decoder design, using transformer-based graph neural networks and a sliding-window transformer processor.The described configurations pair O96 input with O48 processor grids or N320 input with O96 processor grids.Source: §2 · p. 2, Table 1 · p. 3
  • Independent model instances receive Gaussian noise embeddings and are trained on ensemble forecasts scored against a deterministic analysis target.At inference, members use different random seeds and can be generated independently in parallel; the model can produce as many as computationally feasible.Source: §2 · pp. 2–3, Figure 1 · p. 3
  • Training uses autoregressive rollouts, with gradients propagated through the forecast chain during backpropagation.Rollouts are trained to produce forecasts up to 72 hours ahead.Source: §2 · p. 3

2.1 Mitigating error accumulation during rollout

  • During autoregressive rollout, downsampling and then upsampling the reference state before adding the forecast tendency removes small-scale features that could otherwise leave accumulating artifacts.This lets the model focus on forecast tendencies without having to recreate the full state at every step.Source: §2.1 · p. 4

2.2 Loss functions

  • The fair CRPS adjusts for ensemble size but has a degeneracy when all but one member equal the verifying value; almost fair CRPS is introduced to avoid it.The almost fair score blends fair CRPS and CRPS; the stated intention is to use α values close to 1, and its rearranged terms are non-negative for ε ≥ 0.Source: §2.2 · pp. 4–5

2.3 Training

  • Training proceeds through one-step, two-step, expanding multi-step rollouts up to 72 hours, and full-rollout fine-tuning on operational IFS analysis.The first stage uses 300,000 iterations; the second and third use 60,000 and approximately 45,000 iterations, respectively.Source: §2.3 · p. 5

2.4 Parallelism and memory management

  • Data parallelism and model or ensemble sharding distribute training across GPUs, enabling larger ensembles, models, and rollouts as well as higher spatial resolution.Activation checkpointing is also used during the forward pass, including for the afCRPS loss calculation.Source: §2.4 · p. 5

3 Datasets

AIFS-CRPS is trained on ERA5 and operational IFS analyses, takes two atmospheric states as input, and is initialized at inference from operational ECMWF ensemble conditions.

3 Datasets

  • Training uses ERA5 data from 1979–2017 and operational IFS analyses from 2016–2023 for fine-tuning.The model uses states at t−6h and t0 to forecast t+6h.Source: §3 · p. 5
  • At inference, ensemble members start from operational ECMWF ensemble initial conditions constructed using data-assimilation and singular-vector perturbations.Source: §3 · p. 5
  • When needed, ERA5 and operational IFS analyses are interpolated from their native resolutions to the AIFS-CRPS input grid.ERA5 is N320; operational IFS analysis is TCo1279.Source: §3 · p. 5
  • Table 2 lists the input and output variables of AIFS-CRPS.Source: Table 2 · p. 6

4 Experiments

The experiments compare O96 and N320 AIFS-CRPS versions, which differ in input and processor resolution and in two training settings.

4 Experiments

  • The higher-resolution N320 version uses an N320 input grid and O96 processor grid, while O96 uses O96 input and O48 processor grids.Source: §4 · p. 6
  • N320 training uses two ensemble members to mitigate costs and does not use a minimum pressure scaling factor.The paper notes this setup may be revised in the future.Source: §4 · p. 6
  • Table 3 summarizes the training settings for the O96 and N320 experiments.Source: Table 3 · p. 6

5 Evaluation

The evaluation finds that AIFS-CRPS preserves forecast variability across lead times and generally improves medium-range skill, while subseasonal comparisons depend on whether systematic mean-state differences are included.

5.1 Variability

  • AIFS-CRPS members retain realistic meridional-wind variability without visible blurring at 24 and 240 hours, unlike MSE-trained AIFS.Figure 2 compares IFS, MSE AIFS, and AIFS-CRPS members at two lead times and resolutions.Source: §5.1 · pp. 6–7, Figure 2 · p. 7
  • Reference-field truncation avoids the spurious growth of small-scale geopotential variability, while truncated spectra remain stable with lead time.Without truncation, small-scale energy grows and propagates to larger scales; unlike MSE-trained AIFS, AIFS-CRPS does not damp smaller scales with lead time.Source: Figure 3 · p. 8, §5.1 · p. 8, Figure 4 · p. 8

5.2 Medium-Range

  • N320 outperforms IFS for most surface variables; O96 is more skilful for some, but less skilful for others, including total precipitation.N320 precipitation skill is higher for approximately the first 8 forecast days and lower toward the end, likely due to reduced ensemble spread.Source: Figure 5 · p. 9, §5.2 · p. 9
  • AIFS-CRPS spread is often larger in the extra-tropics, but smaller in the tropics after the first forecast days and for most N320 surface variables.For 500-hPa geopotential, AIFS-CRPS is over-dispersive, with spread exceeding ensemble-mean RMSE and poorer correspondence than IFS.Source: §5.2 · pp. 9, 11, Figure 9 · p. 11
  • AIFS-CRPS has higher skill than IFS for most upper-air variables at both tested input grids, though scores can degrade at 100 hPa and above.The reported forecast improvements are in the range of 5-20%; degradation aloft is more pronounced for N320, trained with linear pressure scaling only.Source: §5.2 · p. 9, Figure 6 · p. 10, Figure 7 · p. 10
  • N320 has higher skill than O96 for most variables, especially surface variables, while some 850-hPa temperature and wind comparisons depend on verification data.Those variables appear degraded against analyses but show large improvements against observations; N320 also has increased spread for most variables.Source: §5.2 · p. 9, Figure 8 · p. 11

5.3 Subseasonal

  • Raw subseasonal scores favor AIFS-CRPS for some surface and tropospheric parameters, but anomaly-based comparisons show persistent gains mainly at 2-3 weeks.A large fraction of raw-score differences reflects mean-state differences; at longer leads anomaly-based skill is similar, with degradation in week 6 in the Tropics and in the stratosphere.Source: §5.3 · p. 12
  • AIFS-CRPS exceeds IFS on several MJO skill metrics and has good agreement between ensemble spread and ensemble-mean RMSE across lead times.A single case study suggests better representation of developing zonal-wind anomaly magnitude and eastward propagation, but the authors caution that it may not generalize.Source: §5.3 · pp. 12–13

6 Discussion

AIFS-CRPS shows strong medium-range skill, while its subseasonal results are promising despite a short out-of-sample reforecast period. The discussion also identifies limitations involving initial conditions and reduced stratospheric skill.

6 Discussion

  • At N320, AIFS-CRPS has higher skill than the IFS ensemble for most surface variables, and surface skill improves over O96; upper-air medium-range skill is strong.The surface-resolution improvement is consistent with deterministic forecast performance.Source: §6 · p. 13
  • Training currently uses ERA5 reanalysis and operational IFS analysis, while inference uses perturbed IFS initial conditions; training with perturbed conditions remains for future assessment.The authors note that imperfect uncertainty representations or differences from the unperturbed reference state might affect autoregressive forecasting, while perturbed training could reduce aliasing initial-condition uncertainty into model uncertainty.Source: §6 · p. 13
  • AIFS-CRPS is over-dispersive for some variables, including 500-hPa geopotential in the early medium range, likely because IFS singular-vector perturbations are retained despite lower AIFS-CRPS RMSE.Initial tests with a revised perturbation amplitude improved reliability; operational IFS initial conditions were used as the straightforward choice for experimental real-time use.Source: §6 · p. 13
  • Subseasonal reforecasts are promising, but their assessment is limited by a relatively short out-of-sample period.First longer-range tests indicate good skill can emerge from a system trained only on short-range forecasts.Source: §6 · p. 13
  • If results generalize to real-time operations, AIFS-CRPS subseasonal forecasts are expected to be competitive with or outperform the best physics-based models.The MJO results are notable because IFS MJO forecasts generally compare favorably with other ensemble prediction systems.Source: §6 · p. 13
  • Stratospheric forecast skill is currently reduced, which the authors believe is caused by sensitivity to vertical scaling in the afCRPS objective.Revised loss definitions are planned for future work.Source: §6 · p. 13

7 Conclusions

The conclusions report that proper-score training yields a skilful ensemble with realistic variability and strong medium-range comparisons against IFS. AIFS-CRPS also performs well in subseasonal forecasts, with computational, initialization, and future-work considerations noted.

7 Conclusions

  • AIFS-CRPS has higher forecast skill than the 9 km IFS medium-range ensemble for most upper-air fields and surface variables, without smoothing fields and with realistic variability over long rollouts.The paper concludes that proper-score training such as afCRPS can produce a highly skilful ensemble prediction system.Source: §7 · p. 13
  • Figure 10 score cards compare raw weekly means and anomalies using fCRPSS differences, with positive blue triangles indicating AIFS-CRPS improvement over IFS.Scores aggregate the northern hemisphere and tropics on a regular 2.5° × 2.5° grid.Source: §7 · p. 14, Figure 10 · p. 14
  • For two-to-six-week forecasts, AIFS-CRPS shows lower biases and increased MJO skill than ECMWF’s operational subseasonal forecasting system despite short-range-only training.Source: §7 · p. 14
  • A single-member 15-day forecast takes about one minute at O96 or four minutes at N320 on an NVIDIA A100 40 GB GPU.Source: §7 · p. 14
  • AIFS-CRPS relies on ECMWF physics-based NWP analysis states for both training and forecasting.Source: §7 · p. 14
  • Future work includes higher-resolution forecasts, revised initial-condition perturbations, more forecast parameters, and revised loss scaling for stratospheric skill.Source: §7 · p. 14
  • Figures 11–13 provide MJO diagnostics through surrogate-index correlations and spread, 850-hPa wind evolution, and 46-day phase diagrams.The MJO index approximation excludes outgoing longwave radiation unavailable from AIFS-CRPS.Source: Figure 11 · p. 15, Figure 12 · p. 15, Figure 13 · p. 16
  • The authors expect to begin real-time experimental N320 forecasts at ECMWF and make ensemble forecasts and related products publicly available under the ECMWF open data license.Source: §7 · p. 16

Related papers

Loading 2412.15832v1…