Source-linked AI summary
Every Fixed Metric Has a Blind Spot: A Learned Atmospheric Critic for Scoring Forecast Realism
Younes Elberkennou, Dmitri Demler, Thierry Meier, Luca Rispoli, Fanny Lehmann, Joel Oskarsson
TL;DR
Fixed distributional metrics can miss atmospheric artifacts because each relies on a representation or transformation chosen in advance. The paper trains a discriminator to distinguish reference weather from forecasts and uses its output as a realism score; it detects and ranks the tested corruptions, while real-forecast scores generally worsen with lead time for machine-learning models and are flatter for numerical models.
Problem
Fixed distributional metrics use representations or transformations chosen in advance, limiting which unrealistic forecast artifacts they detect.
Source: §1 · p. 1Method
Results
The critic detects and ranks all tested synthetic corruptions; unrealism generally rises with lead time for machine-learning forecasts, while numerical-model scores are flat.
Source: §3.1 · p. 4, §3.2 · p. 4, §4 · p. 4Takeaways & Limitations
The learned critic detects unrealistic structure across the tested corruptions and separates real forecast fields from ERA5 in the evaluated models.
Source: §4 · p. 4The practical score depends on the discriminator and the patterns it most easily detects, and separate discriminators are required for each group of compared target distributions.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Despite their high accuracy on point-wise metrics, machine learning weather forecasting models can exhibit different failure modes such as blurring, periodic irregularities, and other unphysical spatial artifacts. This has motivated a variety of metrics to detect known failure cases. Existing metrics fix a representation or transformation in advance, and that choice limits the artifacts they can detect. We propose to train a discriminator for separating reference data from the model's output, and using its output logit to obtain a divergence-like realism score. The discriminator learns whatever separates the model's fields from real weather, adapting to whichever failure mode that model exhibits. We compare our learned atmospheric critic to existing metrics using various synthetic corruptions applied to ERA5 reanalysis data. Our method successfully identifies the corruptions and ranks their severity, while existing metrics fail on at least one corruption. Additionally, we evaluate forecasts from real weather models, and find that the realism score degrades with longer lead times and the metric generally assigns higher realism to numerical models than to machine learning models.
1 Introduction
Weather models can achieve strong point-wise scores while producing unrealistic atmospheric artifacts, and fixed distributional metrics may miss failures outside their chosen representation. The paper proposes a learned critic that adapts its representation to distinguish forecasts from reference weather.
1 Introduction
- Fixed point-wise metrics can favor overly smooth forecasts, while models may also produce periodic, patch, polar, or physically inconsistent artifacts.The introduction argues that identifying such issues quantitatively matters for improving models and building trust for meteorological use.Source: §1 · p. 1
- Existing distributional comparisons fix a representation or transformation, including field spaces, likelihood histograms, pretrained features, or kernels.The supplied related-work passages describe Wasserstein distances, normalizing flows, FID, and MMD.Source: §1 · pp. 1–2
- The learned atmospheric critic distinguishes reference data from forecasts and converts its output into a divergence-like realism score that adapts to model-specific failure modes.The paper reports that the metric detects all tested synthetic corruptions and that machine-learning-model scores worsen with lead time, while numerical-model scores remain constant.Source: §1 · p. 2
2 Method
The method measures how distinguishable forecast fields are from reference atmospheric states. A classifier’s output yields a divergence-like score, with practical and training-sharing qualifications.
2 Method
- A binary classifier is trained to distinguish samples from reference distribution P and forecast distribution Qµ,τ using balanced classes and binary cross-entropy.P represents reference states over the evaluated period, while Qµ,τ contains forecasts from model µ at lead time τ.Source: §2 · p. 2
- With an optimal classifier and true distributions, Sadv(Qµ,τ) equals DKL(Qµ,τ ∥ P), but finite data and limited capacity yield a variational lower bound that need not be calibrated and may be negative.The score applies the variational objective of Nowozin et al. [2016].Source: §2 · p. 2
- To balance discriminator capability and practical usefulness, the study trains one discriminator per model, shared across that model’s lead times.The approach can be extended to a discriminator shared across models; fixed finite capacity still makes scores comparable across models.Source: §2 · p. 2
3 Experiments
On controlled ERA5 corruptions, fixed metrics have complementary blind spots, while the learned critic detects and orders every tested corruption. On real forecasts, unrealism scores generally rise with lead time for machine-learning models, with stated exceptions and flatter numerical-model scores.
3.1 Behavior on artificial corruptions
- The corruption tests use ERA5 surface fields and five perturbations spanning smoothing, periodic artifacts, cross-variable mismatch, spurious small-scale variance, and broken long-range dependence.A critic is trained per corruption type and held fixed across severity levels; comparisons include Wasserstein, spectral, and MMD baselines.Source: §3.1 · p. 3
- The learned critic satisfies detection and severity-monotonicity criteria for all five corruptions, whereas no fixed baseline meets both criteria for every corruption.Subsampled-field MMD is the strongest fixed baseline but misses hemisphere splice and high-frequency noise; global-mean WD and spectral metrics also miss specified corruption types.Source: Table 1 · p. 3, Figure 2 · p. 3, §3.1 · p. 4
3.2 Behavior on real forecast models
- GraphCast, Pangu-Weather, FuXi, and U-Cast receive progressively higher unrealism scores at longer lead times, while SWIFT is roughly flat and IFS HRES and ERA5 Forecast are flat.HRES has an initial deviation attributed to initialization from analysis; ERA5 Forecast stays near the empirical null. The authors note that monotonicity is not generally expected for accurate forecasts.Source: §3.2 · p. 4
- Existing metrics generally struggle to exceed the empirical null on real forecasts, while the learned method separates model-specific shifts; MMD performs more weakly here than on corruptions.The authors interpret the baseline behavior as sensitivity to shifts between realistic data splits rather than forecast-induced shifts.Source: §3.2 · p. 4
4 Conclusion
The learned critic detects unrealistic structure across tested corruptions and generally assigns greater unrealism to machine-learning forecasts at longer lead times. The appendix connects its score to a variational f-divergence construction while noting that practical scores are not exact divergences.
4 Conclusion
- The critic detects all tested corruptions and tracks their increasing severity; machine-learning forecast unrealism generally rises with lead time.The conclusion also reports complementary blind spots among fixed baselines.Source: §4 · p. 4
- In practice, the score depends on the discriminator and detectable patterns, and requires a separate discriminator for each compared target group.Future work could explore critics that transfer across forecast models and comparisons conditioned on an initial state.Source: §4 · p. 4
A Connection to variational f-divergences
- For reverse KL, applying the output map to the optimal BCE logit yields the reverse-KL optimum T*(x) = −q(x)/p(x).The log-density ratio is understood where both densities are positive.Source: A Connection to variational… · p. 6
- At the population optimum, Equation (1) equals DKL(Q ∥ P), while finite-sample, finite-capacity fitting yields a variational estimate rather than an exact divergence.The construction applies the variational f-divergence framework of Nowozin et al. [2016].Source: A Connection to variational… · pp. 6–7
B.1 Data and temporal splits
The corruption experiments use 6-hourly ERA5 states from 2004–2022, while forecast verification is restricted to 2020. Forecast data are matched to ERA5 by valid time and evaluated across available lead times.
B.1 Data and temporal splits
- The corruption experiments use 27,760 ERA5 states from 2004–2022, while the 2020 verification period contains 1,464 states.The states are 6-hourly ERA5 reanalysis samples.Source: §B.1 · p. 7
- Forecasts are evaluated at available leads of 6, 12, 24, 48, 96, and 192 hours, with U-Cast lacking a 6-hour output.A forecast is retained only when ERA5 is available at its exact verification timestamp.Source: §B.1 · p. 7
- Five temporal splits vary the test days within each month to avoid seasonal effects, excluding each test window and four days on either side from training.The first split tests days 5–11; the other four use seven-day test windows.Source: §B.1 · p. 7
B.2 Critic training and sample counts
Separate binary classifiers distinguish ERA5 from each corruption or forecast model, using standardized variables and a deliberately conservative training schedule. Forecast sample construction can reuse timestamp-matched ERA5 states across leads.
B.2 Critic training and sample counts
- A separate binary classifier is trained for every corruption or forecast model, with ERA5 as the positive class and variables standardized from ERA5 training data.Each variable uses its pooled mean and standard deviation.Source: §B.2 · p. 7
- The ImageNet-pretrained SqueezeNet-1.1 critic is adapted to four channels and a scalar logit, then trained for three epochs with balanced binary cross-entropy.Training uses AdamW, batch size 32, learning rate 10−4, and weight decay 10−4; the epoch count is conservative to prevent overfitting.Source: §B.2 · p. 7
- For forecast critics, each retained initialization–lead pair is paired with timestamp-matched ERA5, so the same ERA5 state may recur across forecasts verifying at that time.For corruption critics, each training ERA5 state appears once clean and once corrupted per epoch.Source: §B.2 · p. 7
B.3 Evaluation and aggregation
Each temporal split is a separate evaluation with a newly trained critic, and scores are aggregated across critics or fixed-metric evaluations. The empirical null measures the critic’s response to held-out unperturbed ERA5 variation.
B.3 Evaluation and aggregation
- Each temporal split trains a new critic and scores held-out fields; each forecast model uses one critic across its available lead times.For each corruption type, a critic is trained using the severity definitions in Appendix C and held fixed along its response curve.Source: §B.3 · p. 7
- Learned-critic curves report the mean and full range over five critics, while fixed metrics report the mean and full range over ten evaluations.The plotted evaluations aggregate results across temporal splits.Source: §B.3 · p. 7
- Figure B.1 shows SqueezeNet training and validation curves from a standalone 10-epoch GraphCast run, using days 1–11 for training and 12–15 for validation.Source: Figure B.1 · p. 8
- Table 2 distinguishes eligible states, per-class training counts, and candidate/reference test counts; ranges reflect temporal-window shifts and model-specific coverage.The complete WeatherBench2 row covers GraphCast, Pangu-Weather, and IFS HRES; FuXi has shorter coverage.Source: Table 2 · p. 8
- The empirical null evaluates the fitted critic on held-out unperturbed ERA5 and measures its response to variation between realistic training and test periods.This provides a reference for interpreting shifts detected in corrupted or forecast distributions.Source: §B.3 · p. 8
B.4 Compared metrics
The compared metrics represent weather fields through different fixed transformations, then quantify distributional differences using empirical samples.
B.4 Compared metrics
- All baselines compare joint empirical distributions of four standardized fields; forecast–ERA5 matching selects valid times but is not used by the metrics.Corruption sweeps use 1,000 held-out states per severity; MMD and multi-field SCWD use at most 256 states per distribution.Source: §B.4 · p. 8
- Global-mean Wasserstein distance compares distributions of four cosine-area-weighted global field means on a shared 20-bin-per-variable grid.The score is the exact discrete quadratic Wasserstein distance between the empirical distributions.Source: §B.4 · p. 8
- SCWD computes empirical W2 distances between four filter responses at spherical anchors, then reports their unweighted root-mean-square.Fields are convolved with compact Wendland filters on a regular 60 × 120 anchor grid, with 1000 km support radius.Source: §B.4 · pp. 8–9
- The zonal log-spectrum discrepancy is the root-mean-square difference between candidate and ERA5 longitudinal log powers across variables and wavenumbers.Power is averaged over latitude with cosine-area weights and then over states; 10−12 times the largest positive ERA5 power is added before logarithms.Source: §B.4 · p. 9
- Subsampled-field RBF MMD uses a biased empirical V-statistic on 1,024 approximately uniform coordinates per flattened field, standardized with ERA5 statistics.The RBF bandwidth per field is the median nonzero pairwise distance between ERA5 vectors; the joint kernel averages the four bandwidth-normalized squared field distances in its exponent.Source: §B.4 · p. 9
C Corruption library
The corruption library applies five controlled transformations to standardized ERA5 fields, with severity defined within each corruption family.
C Corruption library
- Seven equally spaced severity levels from 0 to 1 are evaluated, but severity is comparable only within a corruption family.Non-splice critics train on nonzero levels; splice critics train on clean and fully spliced samples because intermediate severity would label some unchanged ERA5 states as fake.Source: C Corruption library · p. 9
- Gaussian blur applies a Gaussian kernel independently to each field, with standard deviation 0.225 + 0.9s grid pixels for s > 0.At s = 0 it is the identity; the mapping is discontinuous at zero and affine over positive severities.Source: C Corruption library · p. 9, Figure C.1 · p. 10
- High-frequency noise adds independent Gaussian noise per grid cell and channel, while checkerboard corruption adds a mean-centered alternating 2 × 2 block pattern.Their standardized amplitudes are 0.05s and 0.2s, respectively; the figure shows controlled corruptions applied to 2 m temperature.Source: C Corruption library · p. 9, Figure C.1 · p. 10
- Hemisphere splice probabilistically replaces the southern hemisphere across all four fields with another ERA5 state, preserving local structure while breaking long-range dependence.Severity controls the probability of complete replacement; during evaluation donor choices remain fixed across severity levels.Source: C Corruption library · p. 9, Figure C.1 · p. 11
- Field splice independently replaces complete variable fields with fields from different ERA5 times, preserving individual fields while disrupting cross-variable dependence.Severity controls the probability of replacement, and each replaced variable uses a separately chosen donor field.Source: C Corruption library · p. 9, Figure C.1 · p. 11
- Splice critics train only on clean and fully spliced samples, then evaluate progressively mixed distributions as replacement probability varies.Training uses new donor choices every epoch, whereas evaluation keeps choices fixed across severity levels.Source: C Corruption library · p. 9
D Extended results
The extended results show full response curves for fixed metrics and the learned critic, alongside forecast-model logit distributions across lead times.
D Extended results
- Across tested models, logit histograms broadly retain their shape across lead times, consistent with degradation not dominated by a few unrealistic samples.IFS HRES and SWIFT histograms remain nearly unchanged and in a similarly realistic regime, aside from the initial shift.Source: D Extended results · p. 12, Figure D.2 · p. 13
- Figure D.1 shows full lead-time responses for fixed distributional baselines; curves are normalized separately within each metric.Diamonds mark empirical nulls, offset horizontally for visibility, and shaded areas show support across repeated evaluations.Source: Figure D.1 · p. 12
- Figure D.2 presents held-out real-versus-fake logits for every forecast model and lead time; positive logits favor ERA5, with the decision boundary at zero.The distributions provide a complementary view of the critic's lead-time responses.Source: D Extended results · p. 12, Figure D.2 · p. 13