Source-linked AI summary
Uncertainty-Aware End-to-End AI Weather Forecasting: Disentangling Observation and Model Contributions
Rodrigo Almeida, Noelia Otero, Jost Arndt, Simon Baur, Wojciech Samek, Jackie Ma
TL;DR
End-to-end weather models are computationally efficient but lack calibrated, attributable uncertainty. The paper adds encoder noise and processor dropout to a pretrained Aardvark model, forming a nested ensemble whose spread is decomposed by source. Probabilistic finetuning improves mean forecasting while producing calibrated forecasts, although performance remains below the operational ECMWF ensemble.
Problem
End-to-end models forecast directly from raw observations but issue neither calibrated uncertainty estimates nor attribution of uncertainty to its sources.
Method
A pretrained Aardvark model is uplifted with learned input-dependent encoder noise and processor Monte Carlo dropout in a nested ensemble.
Results
Probabilistic finetuning improves the mean forecast and yields an ensemble whose spread can be decomposed into observation and model contributions.
Takeaways & Limitations
Component-attributed uncertainty distinguishes observing-system spread from learned-dynamics spread, supporting more transparent end-to-end forecasts.
Takeaways & Limitations
The system trails the operational ECMWF ensemble, with the short-lead CRPS gap reaching 80% in the variable mean at day 1.
Abstract
from arXiv · showhide
End-to-end weather forecasting systems produce skillful global gridded and station forecasts directly from raw Earth observations, replacing the numerical weather prediction pipeline, including data assimilation, at a fraction of its cost. These systems are deterministic and issue no uncertainty. Here we render the Aardvark Weather model probabilistic by attaching one stochastic mechanism to each component: learned, input-dependent noise at the observation encoder, capturing aleatoric uncertainty inherited from the observing system, and Monte Carlo dropout in the processor, capturing epistemic uncertainty in the learned dynamics. The resulting nested ensemble attributes forecast spread to the two sources through a law-of-total-variance decomposition, cross-checked by withholding observation streams. Probabilistic finetuning significantly improves the mean forecast, by 4.2% on average across variables and lead times. The ensemble is calibrated against ERA5 through the medium range (spread-skill ratio 0.98), keeps station RMSE within 2.4% of the deterministic model while beating it in CRPS at every lead time, and trails the operational ECMWF ensemble. The encoder branch behaves as observation-driven uncertainty. Component-attributed uncertainty makes end-to-end forecasts more transparent, a step toward observation-driven digital twins of the atmosphere.
Introduction
End-to-end weather models forecast directly from raw observations at low cost but lack calibrated uncertainty and source attribution. This work adds component-specific stochasticity to address that gap and evaluates whether the resulting uncertainty can be disentangled.
- Motivation: End-to-end neural forecasts can match or surpass numerical weather prediction skill at a fraction of the computational cost.These systems are trained on and initialized from reanalysis datasets such as ERA5.
- Motivation: Probabilistic forecasts support decision-making because users have different cost-loss ratios, making a single deterministic forecast suboptimal across users.Operational assessment therefore emphasizes probabilistic skill and forecast value alongside deterministic scores.
- Research gap: End-to-end models forecast from raw observations at minimal cost but provide neither calibrated uncertainty estimates nor attribution of uncertainty to its sources.This is the gap addressed by the work.
- Uncertainty sources: Predictive uncertainty comprises aleatoric variability inherited from observations and epistemic uncertainty originating in the model.The two components imply different improvement routes: better observations versus additional data, training, or parametrization.
- Approach: The method uplifts pretrained Aardvark Weather into a probabilistic forecaster using encoder noise for observation uncertainty and processor dropout for learned-dynamics uncertainty.A nested ensemble over encoder draws and dropout masks makes forecast spread decomposable without training a new probabilistic system from scratch.
- Study questions: The study tests where end-to-end forecast uncertainty originates, whether its contributions can be disentangled, and whether probabilistic finetuning preserves or improves mean forecasts.Evaluation is performed against ERA5, the operational IFS ensemble, and HadISD station observations.
Results
Probabilistic finetuning upgrades Aardvark into an ensemble that improves deterministic forecast skill, provides calibrated grid forecasts, preserves station RMSE, and attributes spread to observation and model branches.
- 4.2% average RMSE reduction across 60 variable–lead combinations improves the ensemble mean over Aardvark, with no significant degradation.Improvements reach 16%, and 50 of 60 combinations are statistically significant.
- The ensemble remains behind operational IFS ENS in probabilistic skill, with an 80% variable-mean CRPS gap at day 1 narrowing to 18–36% by day 10.The gap is largest for Z500 and MSLP, whose operational forecasts use observing streams absent from the encoder.
- 0.98 variable-mean spread–skill ratio over days 1–10 indicates near-calibration against ERA5 without post-hoc spread adjustment.SSR is 1.21 at day 1, 0.92 around day 5, and 0.97 at day 10.
- 2.4% maximum station RMSE deviation from the deterministic forecast coexists with significantly better CRPS at every lead for T2M and WS10.Station calibration is less favorable, with under-dispersion at all leads.
- 0.14 encoder variance share at day 1 declines to 0.04 by day 10, while dropout variance rises from 0.86 to 0.96.This decomposition assigns observation-driven uncertainty primarily to the encoder and model-driven uncertainty to dropout.
- 96–111% increases in encoder uncertainty after withholding IASI, while dropout changes by at most 8%, support branch-specific attribution.Withholding the least influential IGRA radiosonde stream leaves every component within 1% of baseline.
Discussion
The nested-ensemble uplift separates forecast spread into observation-driven and model-driven components without training a new probabilistic system from scratch. Results indicate that model uncertainty dominates after initialization, while limitations constrain the attribution and evaluation scope.
- Discussion: Two finetuned noise sources distinguish uncertainty inherited from observations from uncertainty arising in learned dynamics.The nested ensemble adds learned, input-dependent encoder noise and processor dropout to a pretrained deterministic model.
- Discussion: At most a seventh of predictive variance traces back to observations at day 1, with model uncertainty becoming relatively more important during rollout.The direction matches NWP expectations, but the level differs: most day-1 variance originates in learned dynamics.
- Discussion: Observation denial provides a low-cost falsification check because withholding an observation stream changes the encoder and dropout variance axes in the predicted way.The experiment re-encodes and reruns the ensemble rather than repeating a full assimilation–forecast cycle over months of dates.
- Discussion: The approach has structural limitations: deterministic decoding omits downscaling uncertainty, dropout uses a fixed p = 0.05, and verification covers one year, one backbone, and modest ensemble sizes.The decoder limitation motivates a third stochastic branch, while deep processor ensembles would provide a stronger epistemic reference at higher training cost.
- Discussion: The reported skill and calibration are lower bounds because the backbone uses a 1.5° grid and nine observation streams, while encoder/dropout attribution is not recovered ground truth.Remaining differences from IFS ENS may partly reflect resolution and observation coverage rather than probabilistic machinery.
- Discussion: Future work includes comparison with ERA5’s Ensemble of Data Assimilations and further use of observation denial to test the encoder attribution.EDA is proposed as a physically grounded reference for assimilation uncertainty, not as ground truth.
Methods
The method converts deterministic Aardvark into a stochastic end-to-end forecaster by placing independent randomness in the observation encoder and processor, then attributing forecast variance to those components with a nested ensemble.
- Architecture: The deterministic Aardvark pipeline comprises an observation encoder, autoregressive processor, and decoder, which together produce forecasts from heterogeneous observations.The encoder maps observations to an initial state, the processor advances it, and the decoder maps forecast states to targets.
- Stochastic components: Learned, input-dependent Gaussian noise makes the encoder stochastic, representing observation-side uncertainty through a heteroscedastic construction.Noise amplitude depends on the input and is injected through per-patch embeddings; the paper uses norm conditioning in its results.
- Stochastic components: MC dropout makes the processor stochastic, with masks active during inference and finetuning so processor spread adapts to the noise-conditioned encoder.The decoder remains deterministic, yielding a strict two-way split between encoder and processor uncertainty.
- Variance attribution: The two independent random sources are encoder noise and processor dropout masks, and their predictive variance is decomposed by conditioning on the encoder draw.The encoder and dropout components are read respectively as observation-driven and model-driven uncertainty in a relative aleatoric/epistemic sense.
- Variance attribution: A nested ensemble crosses M encoder-noise draws with N inner dropout realizations, allowing within- and between-group variation to estimate the two branches.Within-group spread measures dropout variation, while group-mean spread reflects changing encoder draws.
- Validation and evaluation: The MSW/N correction removes finite-inner-ensemble sampling noise so dropout variance does not leak into the encoder estimate.Observation-stream withholding provides a falsifiable check: the encoder component should increase while the dropout component should not respond.
- Validation and evaluation: The experiments use the public Aardvark configuration, finetune components from deterministic weights with fair CRPS, and evaluate a 49-member ensemble against ERA5 in 2018.Inference uses M = 7 encoder draws and N = 7 processor masks; statistical inference aggregates each of n = 337 initializations into one score.
Data availability
The study provides the finetuned weights, evaluation data, source inputs, verification datasets, and reproduction code through public repositories.
- Public resources: Finetuned model weights and generated evaluation data are available through the study’s Hugging Face dataset.The training and verification inputs come from the publicly released Aardvark dataset.
- Public resources: ERA5 fields and IFS ENS verification data were fetched from WeatherBench, and code reproducing the experiments and figures is available on GitLab.
Supplementary information
The supplementary material compares encoder noise-injection variants, verifies regional station performance, and documents notation and statistical quantities.
- Noise-injection variants: Norm conditioning has a 4–5% CRPS advantage over embedding injection at days 1–4, decaying to parity by day 10.
- Noise-injection variants: Figure S1 compares embedding injection and norm conditioning using fair CRPS, spread–skill ratio, and dropout variance fraction against ERA5 for 2018.Lead 0 is the encoder analysis, so the dropout branch and dropout variance fraction are zero by construction.
- Regional station verification: The probabilistic ensemble improves deterministic station CRPS at every lead in all four regions and across all 88 variable–region–lead combinations.The improvement remains significant in sparse West Africa and Pacific station sets, despite wider confidence intervals.
- Notation: The supplementary notation table groups symbols by forecast components, data and states, random sources, nested ensembles, variance decomposition, and verification statistics.
- Notation: The notation defines U_tot, U_enc, and U_drop as total predictive variance and its encoder and dropout components, alongside ANOVA estimates and the dropout fraction.
- Notation: Verification notation includes n = 337 forecast initializations, lag-1 autocorrelation r_1, and a data-adaptive correlation length ℓ used for resampling and HAC truncation.