PaperlayerView exact revision on arXiv

Source-linked AI summary

Skillful joint probabilistic weather forecasting from marginals

Ferran Alet, Ilan Price, Andrew El-Kadi, Dominic Masters, Stratis Markou, Tom R. Andersson, Jacklynn Stott, Remi Lam, Matthew Willson, Alvaro Sanchez-Gonzalez, Peter Battaglia

arXiv:2506.10772v1cs.LGphysics.ao-ph

TL;DR

Probabilistic weather forecasting must represent uncertainty arising from nonlinear dynamics and incomplete observations, but skillful joint distributions remain difficult to obtain from marginal training. FGN combines independently trained models with learned functional perturbations and CRPS optimization, achieving state-of-the-art ensemble forecasts while capturing spatial dependence.

  • Problem

    Weather forecasts must represent ranges of probable conditions because nonlinear dynamics and partial observations prevent perfectly precise predictions, while skillful ML-based probabilistic forecasting remains limited.

  • Method

    FGN combines independently trained model ensembles for epistemic uncertainty with learned parameter-space functional perturbations for aleatoric uncertainty, trained using marginal CRPS.

  • Results

    FGN sets a new state-of-the-art in ensemble forecasting, outperforming GenCast and ENS on marginal skill, extreme weather, dependency-structure metrics, and tropical cyclone tracks.

  • Takeaways & Limitations

    FGN captures joint spatial forecast structure despite optimizing only marginal distributions, supporting a general-purpose approach to modeling weather-forecast uncertainty.

  • Takeaways & Limitations

    Forecast states can contain subtle mesh-related honeycomb artifacts, especially in higher-frequency variables with smaller loss weights.

Abstract

from arXiv · show

Machine learning (ML)-based weather models have rapidly risen to prominence due to their greater accuracy and speed than traditional forecasts based on numerical weather prediction (NWP), recently outperforming traditional ensembles in global probabilistic weather forecasting. This paper presents FGN, a simple, scalable and flexible modeling approach which significantly outperforms the current state-of-the-art models. FGN generates ensembles via learned model-perturbations with an ensemble of appropriately constrained models. It is trained directly to minimize the continuous rank probability score (CRPS) of per-location forecasts. It produces state-of-the-art ensemble forecasts as measured by a range of deterministic and probabilistic metrics, makes skillful ensemble tropical cyclone track predictions, and captures joint spatial structure despite being trained only on marginals.

1. Introduction

Probabilistic forecasts are needed because nonlinear dynamics and partial observations make precise weather prediction impossible, while ML-based ensemble forecasting is only recently competitive. FGN addresses this gap with a flexible approach that combines distinct uncertainty mechanisms and outperforms GenCast and ENS across forecast metrics.

  • Motivation: Probabilistic forecasts represent both the most likely weather and less probable scenarios that decision-makers must anticipate.Complex nonlinear dynamics and partial observations prevent even the best models from making perfectly precise forecasts.
  • Motivation: ML-based probabilistic forecasting remains relatively nascent, despite recent progress toward competing with operational systems.Early approaches include generative adversarial networks, flow matching, variational auto-encoders, and hybrid ML-physics models.
  • FGN contribution: FGN combines architectural, training, and inference innovations to provide probabilistic weather forecasts that are faster, more flexible, and higher-performing than GenCast.It is trained on ERA5 reanalysis and HRES initial conditions using CRPS, while separately modeling epistemic and aleatoric uncertainty.
  • FGN contribution: FGN generates ensemble diversity through learned perturbations within models and an ensemble of independently trained models.The first mechanism represents aleatoric uncertainty and the second represents epistemic uncertainty.
  • Results: FGN comprehensively outperforms GenCast and ENS across deterministic and probabilistic benchmark metrics, with generally similar-to-better calibration and extreme-event prediction.The supplied introduction reports this as the paper’s overall evaluation conclusion.

2. Methods

The method samples autoregressive weather trajectories while combining deep ensembles for epistemic uncertainty with learned functional perturbations for aleatoric uncertainty. It trains these ensemble forecasts using a fair CRPS objective on marginal predictions, with architecture choices designed to preserve spatially coherent variability.

  • 2.1. Problem formulation: The forecasting task is to sample T-step weather trajectories from p(X_1:T|X_≤0), conditional on prior weather states.The formulation treats data assimilation of observations before initialization as separate from the forecasting problem.
  • 2.1. Problem formulation: FGN factorizes trajectories with a second-order Markov assumption and samples each future state autoregressively from the previous two states.Repeating the one-step sampling process produces an ensemble of trajectories.
  • 2.2.1. Epistemic uncertainty: Deep ensembles approximate epistemic uncertainty by training multiple independently initialized models and ensembling their predictions.The paper trains J=4 models and uses each model for all timesteps of a trajectory.
  • 2.2.2. Aleatoric uncertainty: Aleatoric uncertainty is modeled by sampling functions, producing independently sampled forecast outputs for each ensemble member and timestep.Parameter-space perturbations can yield structured variation because shared parameters are reused across spatial dimensions.
  • 2.2.3. Reparameterization and CRPS: The reparameterization trick expresses model parameters as θ=θ*+Δ·ϵ, where Δ maps Gaussian noise into parameter perturbations and controls covariance.At inference, the approach composes independently trained models with sampled noise vectors.
  • 2.2.3. Reparameterization and CRPS: FGN trains with the fair CRPS estimator, a strictly proper scoring rule for univariate distributions, averaged across forecast locations, variables, and levels.Training uses N=2 samples and applies loss weights to variable-level-location tuples.
  • 2.3. Model Architecture: Each constituent model uses a grid-to-mesh GNN encoder/decoder and graph-transformer processor, with FGN larger than GenCast and using six-hour forecast timesteps.FGN has approximately 180m parameters per model seed, latent dimension 768, and 24 processor layers.
  • 2.3. Model Architecture: A shared 32-dimensional noise vector enters conditional normalization layers globally, encouraging coherent variability despite marginal-only training.The learned normalization parameters are shared across spatial mesh and grid-node dimensions.

3. Experiments

FGN is evaluated against GenCast in an operational forecasting setting using HRES-fc0-based validation and test data. The experiments emphasize both forecast quality and inference speed.

  • Experimental setup: 2022 and 2023 provide the validation and test years for forecasts generated by models fine-tuned on operational HRES-fc0 data.ERA5 is used for pre-training, while HRES-fc0 supplies real-time operational data for fine-tuning.
  • Experimental setup: GenCast serves as the strong baseline and state-of-the-art model for probabilistic medium-range weather forecasting.The comparison uses the operationally fine-tuned WeatherNext Gen forecasts, referred to as GenCast in this paper.
  • Efficiency: Training each of FGN’s four ensemble models takes approximately 3 wall-clock days using 490 TPUv5p and TPUv6e days of compute.The compute total is reported for each model in the model ensemble.
  • Efficiency: 8 times faster inference is achieved for a 15-day forecast despite FGN being substantially larger and using 6-hour timesteps.FGN generates forecasts with a single forward pass, and ensemble members can be generated in parallel.

4. Evaluation Methodology

The evaluation measures marginal forecast skill, calibration, decision value, spatial and cross-variable dependence, physical plausibility, and tropical-cyclone track prediction. These tests assess both individual-location forecasts and properties of joint forecast distributions.

  • Marginal forecast distributions: Marginal distributions are evaluated using ensemble-mean RMSE, ensemble CRPS, and spread-skill ratios for each variable and pressure level.Spread-skill ratios near 1 indicate well-calibrated forecasts; values below or above 1 indicate under- or over-dispersion under standard assumptions.
  • Marginal forecast distributions: Relative Economic Value measures the benefit of forecast distributions for deciding whether to prepare for threshold-exceeding events.REV is evaluated across C/L ratios, where C is preparation cost and L is avoidable loss without preparation, and is normalized between climatological and perfect forecasts.
  • Joint forecast distributions: Spatially pooled CRPS evaluates forecast dependence across pool sizes from 120 km to 3828 km.Max-pooled and average-pooled CRPS are used to assess spatial correlations and dependencies.
  • Joint forecast distributions: CRPS for 10m wind speed and 300hPa–500hPa geopotential difference evaluates cross-variable dependencies in derived quantities.These quantities depend on correctly modeling the dependencies of their constituent predicted fields.
  • Joint forecast distributions: Spherical harmonic power spectra are compared with HRES-fc0 to diagnose physical plausibility and blurring.Power spectra are described as a standard but imperfect diagnostic of unphysical smoothing in deterministic ML weather models.
  • Tropical cyclone prediction: Cyclone-track forecasts are evaluated using ensemble-mean position error and REV for predicted track probabilities.Track probabilities are represented as 1° heatmaps, with each cell’s probability equal to the fraction of ensemble members placing a cyclone center there.

5. Results

FGN improves marginal forecast skill over GenCast while maintaining calibration, captures spatial and inter-variable dependencies, and delivers strong cyclone-track forecasts.

  • Marginal predictive skill and calibration: 99.9% of targets show better CRPS for FGN than GenCast, with improvements up to 18% and an average improvement of 6.5%.FGN also outperforms GenCast on ensemble-mean RMSE in 100% of variable-level-leadtime combinations, with an average improvement of 5.8%.
  • Marginal predictive skill and calibration: FGN matches GenCast on extreme 2m temperature REV and outperforms it on extreme 10m wind-speed REV.The comparison concerns events exceeding the 99.99th percentile.
  • Structure of the joint distribution: FGN improves average-pooled and max-pooled CRPS by 8.7% and 7.5%, respectively, across spatial scales.It performs better in 99.9% of average-pooled and 99% of max-pooled cases, indicating captured spatial dependencies.
  • Structure of the joint distribution: FGN is on average 4.8% better for derived 10m wind speed and 7.8% better for z300 − z500 forecasts than GenCast.Short-lead improvements reach 10.4% and 15.6%, respectively, while FGN spectra generally match ground truth and avoid high-frequency power loss.
  • Marginal predictive skill and calibration: FGN maintains a spread-skill ratio close to 1 across lead times, indicating well-calibrated ensemble spread.GenCast has good but slightly poorer calibration than FGN over the first 5–7 days.
  • Tropical cyclone tracks: FGN’s ensemble-mean cyclone track is approximately 24 hours more accurate than GenCast’s from 3–5.5-day lead times.A 12-hour-step FGN version still has lower error than GenCast from 2-day lead times onward, while FGN also achieves better track-probability REV through 5 days.

6. Discussion

FGN improves probabilistic weather forecasting while efficiently modeling both uncertainty sources and joint spatial structure, though artifacts and unstable seeds remain weaknesses.

  • FGN captures epistemic uncertainty with independently trained models and aleatoric uncertainty through learned functional variability.Functional variability acts as model-parameter perturbations during forecast generation.
  • FGN outperforms GenCast on marginal forecasts, extreme weather, dependency-sensitive quantities, and tropical cyclone tracks.The paper characterizes these results as setting a new state-of-the-art in ensemble forecasting.
  • Marginals-only training still produces strong joint spatial structure, with a stripped-down FGN outperforming the prior state-of-the-art almost across the board.The authors suggest shared low-dimensional stochasticity and architectural inductive biases help explain this result.
  • Forecasts can contain subtle honeycomb artifacts, spectral kinks, or local correlations, especially in higher-frequency variables and some smoother fields.The authors link these effects to possible over-constraint of the output covariance structure.
  • Poor model seeds can produce unstable forecast rollouts, highlighting the need for careful evaluation before deployment.One unstable seed was removed and retrained after validation on 2022 forecasts.
  • FGN is presented as a general-purpose framework that is easy to work with and extend for weather forecast distributions.The stated scope includes modeling multiple uncertainty sources and joint spatial structure.

A.1. Data

The paper uses ERA5 and HRES-fc0 descriptions adapted from prior work, adding preprocessing details for HRES-fc0 finetuning.

  • The dataset descriptions are largely adapted from Lam et al. (2022) and Price et al. (2023).
  • Additional preprocessing details are included for HRES-fc0.These details support finetuning FGN.
  • The data section covers ERA5 and HRES-fc0 datasets.

A.1.1. ERA5

ERA5 supplies the pre-training data as a long global weather reanalysis archive, restricted here to selected variables, pressure levels, years, and six-hour intervals.

  • ERA5 is a global reanalysis archive spanning 1959 to the present with hourly data for hundreds of weather variables.The archive is based on ECMWF’s HRES model within a 4D-Var data assimilation system.
  • The pre-training dataset uses selected ERA5 variables on 13 pressure levels from 50 to 1000 hPa.These levels correspond to the WeatherBench benchmark.
  • The selected ERA5 period runs from 1979-01-01 to 2018-01-15 and is downsampled to six-hour intervals.The retained times are 00z, 06z, 12z, and 18z each day.
  • Sea-surface-temperature NaNs over land are replaced with the minimum global sea-surface temperature in a preprocessing subset.

A.1.2. HRES-fc0

HRES-fc0 provides initialization-time forecast states similar to ERA5, with preprocessing choices for dataset consistency and variable-specific handling.

  • HRES-fc0 contains the zeroeth step of ECMWF HRES forecasts initialized at 00z, 06z, 12z, and 18z.
  • Unlike ERA5, HRES-fc0 assimilates each forecast time with the latest ECMWF NWP model and observations from only ±3 hours.
  • HRES-fc0 sea-surface-temperature placeholders are imputed with the minimum sea-surface temperature wherever the corresponding ERA5 value is NaN.This ensures consistent representation between the two datasets during training.
  • Pressure is used as the vertical coordinate, with altitude related through the geopotential variable.
  • Table A.1 distinguishes variable type, ECMWF naming, parameter ID, and whether each variable is predicted or used only as input context.A horizontal divider separates predicted variables from input-only variables.
  • Total precipitation is output but not used as an input because its HRES-fc0 value is zero at initialization.

A.2. Model Training

FGN training proceeds through four stages, progressively increasing temporal and spatial resolution before autoregressive fine-tuning on operational HRES-fc0 data.

  • Training stages: FGN uses four training stages spanning 12-hour 1° ERA5 inputs, 6-hour ERA5 inputs at 1° and 0.25°, and 6-hour HRES-fc0 autoregressive training.The stages comprise 400000, 100000, 32000, and 18000 steps respectively.
  • Training stages: The final stage applies 18000 autoregressive steps, including rollouts from 1AR through 8AR.The allocation includes 8000 steps at 1AR, 4000 at 2AR, and 1000 each at 3–8AR.
  • Training setup: Training uses batch size 64 throughout, with detailed stage-specific hyperparameters reported in Table A.2.
  • Data preparation: The 12-hour dataset is formed by stride-two subsampling of 6-hour data, while 1° ERA5 data is produced by stride-four spatial subsampling from 0.25° data.Accumulated variables are aggregated over the longer temporal interval.
  • Resolution changes: The graph-transformer uses a five-times-refined icosahedral mesh at 1° and a sixth refinement at 0.25°, while retaining the same model weights.

A.3. Architecture

FGN closely follows GenCast’s architecture but changes conditioning, capacity, global-feature handling, and grid-to-mesh processing to support single-pass probabilistic forecasting.

  • Conditioning: FGN generates forecasts in a single forward pass and conditions only on the two prior weather states, unlike GenCast’s diffusion process with noisy targets.
  • Noise conditioning: FGN uses conditional layer normalization driven by a 32-element noise vector, directly mapped to 32-dimensional conditioning features.GenCast instead encodes diffusion noise through Fourier features and a two-layer MLP.
  • Capacity: FGN increases core capacity to hidden and output sizes of 768, 6 attention heads, and 24 transformer layers, compared with GenCast’s 512, 4, and 16.Its grid-to-mesh and mesh-to-grid edge-feature MLPs are reduced to size 32 because the graphs contain O(10^5) edges.
  • Global features: FGN broadcasts the sine and cosine of year progress across both the grid and mesh, adding the mesh representation to encoder inputs.
  • Grid-to-mesh encoder: FGN uses graph-based grid-to-mesh translation, with the encoder’s edge model conditioning on sender grid-node and receiver mesh-node features.The passage describes this as the grid-to-mesh mechanism shared with GenCast and GraphCast before FGN’s modifications.

A.4. Ablations

Ablations show that FGN’s CRPS-based setup already matches or exceeds GenCast under matched conditions, while autoregressive training, larger models, and ensembling further improve results across forecast tasks.

  • CRPS ablations: 99% of targets favor autoregressively trained FGN over GenCast on CRPS, with average improvement rising from 2.7% to 4%.These comparisons use matched 12-hour timestep, single-seed, and model-capacity settings.
  • CRPS ablations: More than 90% of targets favor single-step FGN over GenCast, despite matching GenCast’s timestep, seed count, and model capacity.
  • Joint structure: Matched-capacity FGN generally outperforms GenCast while capturing spatial and inter-variable correlations.The comparison controls model capacity, ensemble size, and rollout steps during training.
  • Cyclone tracks: For cyclone-track probability forecasts, single-step FGN performs as well as or better than GenCast, autoregressive training improves REV, and the larger ensemble performs best.
  • Stability limitation: Single-step FGN exhibits rare unstable, non-physical late-lead rollouts at approximately 0.1% prevalence at 15 days; autoregressively trained models show none in this investigation.The authors suggest autoregressive training may improve rollout robustness and stability.
  • Operational comparison: FGN outperforms ENS on more than 99% of targets in comparisons before and after the ENS resolution-cycle change.The data is regridded to 0.25° for the comparison.
  • Evaluation metrics: FGN’s ensemble mean is evaluated against GenCast using RMSE, while precipitation is assessed with RMSE, CRPS, and SEEPS.For SEEPS, the category of the ensemble mean is used because the metric targets deterministic categorical forecasts.
  • Extreme events: FGN achieves significantly higher REV than GenCast for several extreme-weather predictions, with exceptions at some longer lead-time and high-temperature cost-loss combinations.The evaluated variables include extreme wind, high and low temperature, and low mean sea-level pressure.
Loading 2506.10772v1…