PaperlayerView exact revision on arXiv

Source-linked AI summary

FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale

Boris Bonev, Thorsten Kurth, Ankur Mahesh, Mauro Bisson, Jean Kossaifi, Karthik Kashinath, Anima Anandkumar, William D. Collins, Michael S. Pritchard, Alexander Keller

arXiv:2507.12144v2cs.LGphysics.ao-ph

TL;DR

Probabilistic ML weather forecasting needs to represent spatially correlated atmospheric uncertainty while avoiding excessive inference cost, unstable spectra, and unrealistic long-range dynamics. FCN3 addresses this with a spherical, convolutional hidden Markov model trained using spectral and spatial probabilistic objectives. It outperforms traditional ensemble NWP, nearly matches leading diffusion-based ML skill, and maintains realistic spectra and stable forecasts to 60 days at much lower computational cost.

  • Problem

    ML weather models must improve probabilistic and spatially correlated forecasting while addressing physical consistency, long-term stability, and small-scale noise.

  • Method

    FCN3 is a purely convolutional hidden Markov ensemble model using spherical signal processing, spherical group convolutions, and a spectral-and-spatial probabilistic objective.

  • Results

    FCN3 outperforms IFS-ENS, nearly matches GenCast, and retains physically realistic spectra and stable forecasts at lead times up to 60 days.

  • Takeaways & Limitations

    FCN3’s probabilistic skill, spectral fidelity, stability, and low inference cost support large-ensemble medium-range and subseasonal forecasting.

  • Takeaways & Limitations

    CRPS and ensemble-mean RMSE alone do not comprehensively evaluate probabilistic weather forecasts because they can miss spatio-temporal correlations and physical realism.

Abstract

from arXiv · show

FourCastNet 3 advances global weather modeling by implementing a scalable, geometric machine learning (ML) approach to probabilistic ensemble forecasting. The approach is designed to respect spherical geometry and to accurately model the spatially correlated probabilistic nature of the problem, resulting in stable spectra and realistic dynamics across multiple scales. FourCastNet 3 delivers forecasting accuracy that surpasses leading conventional ensemble models and rivals the best diffusion-based methods, while producing forecasts 8 to 60 times faster than these approaches. In contrast to other ML approaches, FourCastNet 3 demonstrates excellent probabilistic calibration and retains realistic spectra, even at extended lead times of up to 60 days. All of these advances are realized using a purely convolutional neural network architecture tailored for spherical geometry. Scalable and efficient large-scale training on 1024 GPUs and more is enabled by a novel training paradigm for combined model- and data-parallelism, inspired by domain decomposition methods in classical numerical models. Additionally, FourCastNet 3 enables rapid inference on a single GPU, producing a 60-day global forecast at 0.25°, 6-hourly resolution in under 4 minutes. Its computational efficiency, medium-range probabilistic skill, spectral fidelity, and rollout stability at subseasonal timescales make it a strong candidate for improving meteorological forecasting and early warning systems through large ensemble predictions.

1 Introduction

FourCastNet 3 addresses the need for fast, large-scale probabilistic weather forecasting while targeting stability, physical consistency, and realistic spatial spectra. It combines a spherical geometric architecture with end-to-end ensemble training to improve forecast skill, efficiency, and extended-range behavior.

  • Motivation: Traditional NWP is computationally intensive, while ML weather models face challenges with physical consistency, out-of-distribution events, long-term stability, and evaluation of dynamical fidelity.Probabilistic ML methods also incur inference costs or develop small-scale noise, leaving stable spectra a key challenge.
  • Approach: FourCastNet 3 is a probabilistic hidden Markov model using spherical signal processing and a spectral-domain probabilistic loss.Its purely convolutional architecture combines local and global spherical convolution kernels to model atmospheric processes across scales while respecting spherical geometry and symmetries.
  • Approach: FCN3 is trained end-to-end as an ensemble model, using hybrid model- and data-parallelism to scale training beyond 1000 GPUs.The training paradigm combines spatial domain decomposition with distributed batch and ensemble parallelism.
  • Results: A single 15-day forecast takes 60 seconds on one NVIDIA H100 GPU, representing an 8x speedup over GenCast and 60x over IFS-ENS.FCN3 outperforms IFS-ENS and nearly matches GenCast’s medium-range forecast skill at double the temporal resolution.
  • Results: FCN3 retains stable predictions and accurate spectra for lead times up to 60 days, mitigating blurring and small-scale noise buildup.These properties support large-ensemble applications spanning medium-range to subseasonal forecasting.

2 Probabilistic forecasts with hidden Markov models

FCN3 represents atmospheric uncertainty with a hidden Markov formulation that conditions each six-hour state transition on stochastic latent variables. Its parameters are trained end-to-end to match observed spatio-temporal distributions using a composite probabilistic objective.

  • Hidden Markov formulation: FCN3 predicts the atmospheric state six hours ahead from the current state, conditioning the transition on a random spherical diffusion-process latent vector.The model computes u_n+1 = F_θ(u_n, t_n, z_n), with stochasticity introduced through diffusion processes having different length and timescales.
  • Training objective: The model parameters are optimized to approximate atmospheric processes and match observed spatio-temporal distributions of physical variables.Training uses an end-to-end ensemble approach with a composite probabilistic loss based on CRPS.
  • Training objective: The composite objective addresses limitations of point-wise CRPS for spatial multivariate forecasts, where aggregated marginal scores may not enforce accurate spatial correlations or spectra.The passage notes that prior CRPS-trained models achieved predictive skill without physically accurate spectra.

3 Spherical neural operator architecture

FCN3 uses a spherical neural operator architecture grounded in signal processing and symmetry considerations. Local and global spherical convolutions provide complementary representations of atmospheric structure across spatial scales while direct state prediction helps limit high-frequency artifacts.

  • Spherical convolutions: FCN3 combines global spectral convolutions based on spherical harmonic transforms with localized discrete-continuous spherical group convolutions.The global filters use the convolution theorem on the sphere, while local filters approximate continuous integrals with quadrature.
  • Spherical convolutions: Localized anisotropic filters are designed to represent atmospheric structures such as vertically tilted isentropes and blocked flow around topography.Their localized formulation resembles finite differencing used in classical NWP models.
  • Architecture: The architecture consists of an encoder, spherical neural operator processor blocks, and a decoder built from convolutional components.The blocks include convolution, GeLU activation, point-wise MLP, and skip connections, while omitting layer normalization because absolute physical magnitudes matter.
  • Architecture: Group convolutions formulate convolution with respect to a symmetry group; on the two-dimensional sphere, that group is SO(3).This provides the symmetry-aware basis for the spherical architecture.
  • State prediction: FCN3 predicts the next atmospheric state directly rather than predicting tendencies, which empirically helps avoid high-frequency artifact buildup.The paper also relates tendency prediction to Euler time-stepping, which may restrict the space of learnable operators.

4 Scalable training through hybrid parallelism

FCN3’s training is limited by the memory and data demands of high-resolution, multivariable atmospheric fields and autoregressive rollouts. A domain-decomposition strategy combines spatial model parallelism with ensemble and batch parallelism to distribute both computation and data.

  • Scaling constraints: Large internal representations and autoregressive rollouts can exceed single-GPU memory, limiting model scale and training capacity.High-resolution weather samples contain 721 × 1440 floating points per variable, with multiple tens of variables typically required.
  • Spatial model parallelism: Spatial model parallelism distributes both the model and atmospheric data across ranks through domain decomposition.The approach requires distributed implementations of spatial algorithms, including spherical harmonic transforms and discrete-continuous convolutions.
  • Data parallelism: Ensemble parallelism distributes independent ensemble members across GPUs, while batch parallelism distributes batch samples.Ensemble members communicate when the ensemble loss is computed.
  • Training system: The hybrid scheme enabled training FCN3 on 1024 NVIDIA H100 GPUs using large-scale ERA5 reanalysis data and curriculum-style training stages.The initial stage used 208,320 optimization steps with batch size 16 and ensemble size 16, taking 78 hours.
  • Training system: Figure 2 shows latitude-based spatial partitioning across four ranks alongside distribution of ensemble members and batch samples.The partitioning reduces per-rank memory and I/O, while the ensemble loss aggregates information across ensemble and spatial parallel ranks.

5 Results

FCN3 matches or surpasses leading probabilistic and conventional weather models while generating forecasts rapidly, and maintains calibration, physical realism, and spectral fidelity through subseasonal lead times.

  • Forecast skill: FCN3 beats IFS-ENS and nearly matches GenCast on CRPS and ensemble-mean RMSE at 6-hourly, 0.25° resolution.Scores are averaged over 12-hourly initial conditions during the out-of-sample year 2020.
  • Computational efficiency: 60 seconds produces one 15-day FCN3 forecast on a single NVIDIA H100 GPU, yielding speedups of ∼8x over GenCast and ∼60x over IFS-ENS.The comparisons use different hardware and resolutions.
  • Probabilistic calibration: The 50-member ensemble is well-calibrated, with spread-skill ratios approaching 1 and rank-histograms supporting interchangeability between observations and ensemble members.Rank-histogram evolution indicates slight over-dispersion at short lead times of up to 2 days.
  • Case study: FCN3 faithfully simulates storm Dennis, reproducing realistic wind intensities, pressure-flow co-variation, and distinct scenarios across ensemble members.The case study evaluates an initialization 48 hours before landfall; the figure caption reports accurate landfall prediction 5 days in advance.
  • Spectral fidelity: At 30 days, FCN3 shows no apparent degradation in angular power spectra, while evaluation-year spectral errors remain bounded from −0.2 to 0.2 at high wavenumbers.The forecasts retain effective resolution and remain sharp at long lead times.
  • Mechanistic interpretation: The geometric architecture and combined CRPS loss are proposed to preserve local and global distributions and encourage correct spatial correlations.The paper contrasts FCN3’s stable spectra with blurring in deterministic or hybrid models and high-frequency buildup in GenCast and AIFS-CRPS.
  • Subseasonal stability: Additional evaluations report probabilistic skill, computational efficiency, unprecedented spectral fidelity, physical realism, and stable forecasts through 60 days.These results support subseasonal forecasting and larger ensembles at those lead times.

6 Conclusions

FourCastNet 3 combines spherical signal processing, hidden-Markov ensembles, and probabilistic training to produce efficient, skillful weather forecasts. Its realistic spectra and stable extended forecasts support large-ensemble subseasonal prediction.

  • FourCastNet 3 uses spherical signal processing and a hidden-Markov ensemble formulation trained end-to-end with probabilistic objectives in spectral and spatial domains.
  • FCN3 produces physically realistic spectra across wavelengths and stable, sharp forecasts at lead times of up to 60 days.The reported spectral fidelity avoids overly smooth or spurious high-frequency artifacts that challenge other machine-learning models.
  • Large-scale ensemble forecasting is made more accessible through parallel training, low inference cost, in-situ diagnostics, and open-source implementation.The approach supports model and ensemble parallelism while reducing storage and I/O bottlenecks during ensemble analysis.
  • FCN3 models atmospheric states as a learned discrete dynamical system mapping the current spherical state to the next state.The learned operator Fθ approximates the unknown true atmospheric evolution operator F∗.
  • Probabilistic training generates ensembles by sampling latent noise, addressing the blurring associated with deterministic objectives under uncertain dynamics.The ensemble samples approximate the conditional distribution of the next atmospheric state.
  • The architecture processes spherical signals using quadrature-based grids and rotation-invariant group convolutions tailored to the sphere.FCN3 considers equiangular and Gaussian grids, with Gaussian quadrature enabling exact integration for suitable polynomial integrands.

B.4 Convolution theorem and spectral convolutions

FourCastNet 3 combines spectral and localized spherical convolutions to represent globally smooth and locally supported, potentially anisotropic filters while respecting spherical geometry. Its localized formulation yields sparse operators with linear asymptotic cost, and spherical interpolation handles grid changes and pole topology.

  • Spectral convolutions: Spectral convolutions represent smooth, globally supported, isotropic kernels through spherical harmonic parametrization.The resulting filters depend on spherical harmonic degree l but not order m.
  • Localized convolutions: Local convolutions represent compactly supported, potentially anisotropic filters using direct spherical discretization and numerical quadrature.These filters are motivated by localized solution operators and directional atmospheric dynamics.
  • Computational structure: Compact support makes the discrete convolution matrix sparse, enabling matrix-vector multiplication in O(nlatnlon) floating-point operations.The number of nonzero entries per row depends on grid resolution and filter support.
  • Filter basis: Learnable localized filters are parameterized as linear combinations of basis functions, using Morlet-like wavelets with smooth compact support.A Hann window on a disk preserves sparsity while constructing the localized basis.
  • Spherical resampling: Spherical signals can be resampled with alias-free spherical harmonic transforms or bilinear interpolation that accounts for longitude periodicity and poles.The interpolation grid is extended to the poles, whose values are set from the nearest latitude ring.

B.8 Implementation in PyTorch

The authors develop torch-harmonics, a publicly available PyTorch library for differentiable signal processing on the sphere.

  • PyTorch implementation: torch-harmonics provides differentiable spherical harmonic transforms for integration with GPU deep-learning pipelines.The library is publicly available under the BSD-3-clause license.

C FourCastNet 3 architecture

FourCastNet 3 is a spherical neural operator that preserves spherical geometry throughout an encoder–processor–decoder architecture. It combines global and local spherical convolutions, separates channel encoding, and supports discretization-agnostic operation.

  • Design principles: FCN3 treats spherical geometry as a guiding principle and supports evaluation and retraining at arbitrary resolutions or discretizations.Its distributed components are also designed for efficient training and inference.
  • Operator blocks: The architecture combines pointwise MLPs, global spectral convolutions, and local discrete-continuous spherical convolutions.Together, these components target atmospheric signals spanning planetary waves to local flows and fronts.
  • Forecast mapping: FCN3 maps atmospheric, surface, auxiliary, and random inputs to the next weather state on a 0.25° spherical grid.Random conditioning is used to produce a probabilistic map, while auxiliary variables include static or readily computed fields.
  • Encoder and processor: The encoder resamples latitude–longitude inputs to a Gaussian grid and retains a spherical representation through global or local operator blocks.The input grid is 721 × 1440 and the internal grid is 360 × 720.
  • Channel handling: Separate grouped encoders and decoders avoid prematurely mixing variables with distinct spatial statistics and spectral signatures.Atmospheric encoders are reused across pressure levels, while channels are encoded and decoded separately.
  • Processor design: Processor blocks alternate global and local spherical convolutions within a ConvNeXt-inspired architecture, followed by pointwise MLPs and residual paths.The supplied architecture description states that residual prediction is less stable in long autoregressive rollouts because artifacts can be amplified.

D Evaluation metrics

FCN3 is evaluated with deterministic and probabilistic metrics defined on spherical fields, including ensemble spread, skill, calibration, and spectral fidelity. The CRPS-based evaluation captures probabilistic accuracy beyond ensemble-mean performance, while the spectral loss addresses spatial correlations.

  • Deterministic metrics: Deterministic scores compare predictions with ground-truth observations using spherical-field errors such as RMSE, MAE, and ACC.Metrics are defined at individual spherical grid points and extended across channels and the sphere using averaging and quadrature.
  • Ensemble metrics: Ensemble skill applies deterministic metrics to the ensemble mean, whereas spread and the spread-skill ratio assess ensemble dispersion and interchangeability.Under the ideal-ensemble assumption, ensemble variance and squared ensemble skill are equal on average.
  • Probabilistic metrics: CRPS evaluates the predictive probability distribution against a ground-truth observation, unlike ensemble skill, which focuses only on the ensemble mean.Its expectation is minimized when ensemble members follow the ground-truth distribution; fair CRPS removes ensemble-size bias in spread estimation.
  • Spectral evaluation: Spatial CRPS alone evaluates marginal distributions at each location and can leave implausible cross-location ensemble members undetected.This limitation motivates a spectral CRPS term that evaluates spectral coefficients to account for spatial correlations across angular modes.
  • Training objective: The fair CRPS can cause early-training instabilities, so FCN3 begins with CRPS on larger ensembles and later switches to fair CRPS with fewer members.The two-member fair-CRPS case can permit one ensemble member to become unbounded when the other matches the observation.

E.2 Pre-training

FCN3 uses staged pre-training that shifts from short-lead performance toward autoregressive rollout quality. Spatial model and data parallelism accommodate the increased memory requirements of later stages.

  • Pre-training stages: Pre-training has two stages: short-lead optimization on hourly ERA5 samples followed by autoregressive optimization on 6-hourly initial conditions.The first stage targets single-step loss, while the second uses four autoregressive steps.
  • Stage 1: 332,800 hourly ERA5 samples from 1979–2016 support the first stage, which uses biased CRPS because fair CRPS can cause occasional instability.The biased objective is used to avoid ambiguity that can produce unbounded spread.
  • Stage 2: 55,400 6-hourly samples support the second stage, where four autoregressive steps are trained and model and data are split spatially across 8 ranks.This stage switches to fair CRPS with two ensemble members to address memory requirements.
  • Training regime: Training progressively trades batch size for model parallelism to enable more autoregressive steps and suppress accumulated rollout errors.The training regime shifts from high learning rate and large batches toward higher model parallelism during fine-tuning.
  • Optimization: All training uses automatic mixed precision in bf16 format with the ADAM optimizer.These settings apply across the summarized training stages.

E.4 Training data

FCN3 is trained on multi-decadal ERA5 atmospheric reanalysis data represented on a global latitude–longitude grid. The dataset includes 72 atmospheric and surface variables and separate training, test, and validation periods.

  • Dataset: ERA5 provides a multi-decade, hourly estimate of Earth’s atmospheric state produced through data assimilation and reanalysis.Its consistent dynamics come from a single modern numerical model combined with historical observations.
  • Variables and resolution: FCN3 predicts 72 ERA5 variables, including surface and pressure-level atmospheric quantities, on a 721 × 1440 latitude–longitude grid.Although the model is 6-hourly, hourly data sampling increases the available training data.
  • Data splits: The training dataset spans 1980–2016, the test period is 2017, and out-of-sample validation covers 2018–2021.Reported metrics are computed on validation data from 2020.
  • Evaluation scope: The results discussion separates probabilistic characteristics, deterministic single-member performance, and physical realism and stability.These analyses are presented in distinct appendix sections covering the model’s evaluation dimensions.

F.1 Scoring methodology

FCN3 is evaluated under the WeatherBench 2 protocol using an online scoring implementation designed for large ensembles and low-cost inference. Results use 2020 out-of-sample initial conditions and compare 50-member ensembles with IFS-ENS.

  • Scoring implementation: The online scoring implementation minimizes I/O operations and addresses scalability limitations in WeatherBench 2’s evaluation implementation.Scoring scripts and inference code are publicly available and designed for tightly coupled HPC systems.
  • Evaluation period: Evaluation uses 12-hourly initial conditions throughout 2020, with spillover predictions scored using observations from 2021.The evaluation window runs from 2020-01-01 00:00 UTC through 2020-12-31 12:00 UTC.
  • Baselines: Unless otherwise specified, FCN3 is evaluated as a 50-member ensemble against an equal-sized IFS ensemble.IFS-ENS and GenCast baselines are taken from WeatherBench 2.
  • Metric computation: Reported results use unbiased CRPS and spread-skill-ratio formulations with spatial integration computed by trapezoidal quadrature.The metric computation is adapted to match the WeatherBench 2 evaluation procedure.

F.2 Probabilistic scores

FourCastNet 3 achieves strong probabilistic and deterministic forecast skill while maintaining calibration, realistic spectra, and stable rollouts through 60 days. It also substantially reduces computational cost relative to conventional and diffusion-based forecasting systems.

  • Probabilistic skill: FCN3 outperforms IFS-ENS across the 16-variable scoring protocol and matches or slightly exceeds GenCast on 15 of 16 channels at short lead times.The exception is 2 meter temperature, where GenCast performs better.
  • Probabilistic skill: FCN3 ensemble-mean ACC outperforms IFS-ENS, measuring spatial correlation between predicted and observed anomalies.ACC values of 1 indicate perfect prediction, while scores above 0.55 are generally considered skillful.
  • Ensemble calibration: FCN3 attains spread-skill ratios near 1, indicating good calibration, although dispersion changes from initially overdispersive to underdispersive between 24 and 200 hours.The paper attributes initial underdispersion relative to IFS to the absence of initial-condition uncertainty.
  • Rollout stability: At 6, 360, and 1440 hours, FCN3 forecasts remain visually realistic and stable without observable blurring, including at the 60-day subseasonal horizon.The 1440-hour horizon corresponds to 240 autoregressive steps.
  • Spectral properties: FCN3 ensemble members faithfully reproduce ERA5 angular and zonal power spectra through 60 days, whereas GraphCast and NeuralGCM exhibit progressive spectral blurring.The FCN3 spectra preserve the ERA5 spectral cutoff up to degree l = 639.

G.1 Simultaneous model-, data- and ensemble-parallelism

Makani scales FCN3 training through simultaneous model, batch, and ensemble parallelism organized around spherical domain decomposition. Its communicator hierarchy and distributed spatial operations divide computation across latitude, longitude, data, and ensemble dimensions.

  • Parallelism strategy: Makani supports simultaneous model-, batch-, and ensemble-parallel training for large FCN3 weather models.Model parallelism uses spatial domain decomposition, while batch and ensemble parallelism distribute data.
  • Communicator hierarchy: Makani organizes GPUs into orthogonal communicator groups so parallel operations communicate only within the relevant group.The hierarchy includes batch and ensemble data-parallel groups and azimuth and polar model-parallel groups.
  • Ensemble parallelism: Ensemble members are computationally independent until loss computation, making ensemble parallelism highly efficient.A single all-to-all communication is performed at the end of each step to compute the CRPS score.
  • HPC design: Makani adapts high-performance-computing domain-decomposition techniques and optimized communication primitives to large-scale ML weather training.The approach targets systems with high-bandwidth, low-latency interconnects.
  • Domain decomposition: Domain decomposition splits spherical signals across polar and azimuth groups, distributing latitude and longitude subdomains across GPUs.An azimuth group of 4 and polar group of 2 produce 8 subdomains.
Loading 2507.12144v2…