PaperlayerView exact revision on arXiv

Source-linked AI summary

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function

Christopher Subich, Syed Zahid Husain, Leo Separovic, Jing Yang

arXiv:2501.19374v2cs.LGphysics.ao-ph

TL;DR

Deterministic data-driven weather forecasts can be overly smooth, with MSE contributing to fine-scale smoothing through its double penalty. The paper introduces a parameter-free spectral adjustment to MSE and fine-tunes GraphCast, yielding finer effective resolution and improvements in ensemble verification and tropical-cyclone intensity predictions.

  • Problem

    Deterministic data-driven forecasts are overly smooth, underpredict localized extremes, and MSE-based training encourages reduced amplitude at less predictable scales.

    Source: §1.1 · p. 1, §2.1 · p. 3

  • Method

    The parameter-free AMSE loss separates spectral amplitude error from decorrelation, and is used to fine-tune GraphCast.

    Source: §2.3 · p. 4, §4 · p. 8

  • Results

    AMSE-fine-tuned GraphCast has about 160 km effective resolution, improves CRPS-based lagged-ensemble verification, and retains HRES-quality tropical-cyclone intensity with control-equivalent track predictions.

    Source: §3.1 · p. 6, §3.2 · p. 7, §3.3 · p. 7

  • Takeaways & Limitations

    The results show that fine-scale sharpness and effective resolution can improve in a deterministic data-driven forecast without directly modeling ensemble uncertainty.

    Source: §1.2 · p. 2

    Autoregressive fine-tuning leaves some mesoscale smoothing, possibly because forecasts must also serve as initial conditions for later steps.

    Source: §4 · p. 8

Abstract

from arXiv · show

Recent advancements in data-driven weather forecasting models have delivered deterministic models that outperform the leading operational forecast systems based on traditional, physics-based models. However, these data-driven models are typically trained with a mean squared error loss function, which causes smoothing of fine scales through a "double penalty" effect. We develop a simple, parameter-free modification to this loss function that avoids this problem by separating the loss attributable to decorrelation from the loss attributable to spectral amplitude errors. Fine-tuning the GraphCast model with this new loss function results in sharp deterministic weather forecasts, an increase of the model's effective resolution from 1,250km to 160km, improvements to ensemble spread, and improvements to predictions of tropical cyclone strength and surface wind extremes.

1. Introduction

Data-driven weather models achieved strong medium-range forecast skill, but deterministic forecasts remain overly smooth. This work modifies the MSE loss and fine-tunes GraphCast to produce sharper forecasts and better predictions of tropical-cyclone intensity and surface wind speed.

1. Introduction

  • GraphCast, FourCastNet, and Pangu-Weather met or surpassed ECMWF’s IFS forecast skill through 10 days, while GraphCast forecasts about 86 million output values per step.The cited NOAA GraphCast configuration predicts six atmospheric variables at 13 levels plus five surface variables on a ¼° grid, using two time levels as input.Source: §1 · p. 1

1.1. The Problem of Forecast Smoothing

  • Deterministic data-driven forecasts are overly smooth, underpredict localized extreme events, and can be less suitable for spectral nudging and data assimilation.For winter storm Eunice, the 3.5-day GraphCast forecast was too weak and overly smooth.Source: §1.1 · p. 1

NEURALGCM

  • NeuralGCM combines a dynamical core with learned parameterizations and multiple losses, so which design choices reduce smoothing remains unclear.Its losses include MSE on coarsened fields and spectral power matching at high wavenumbers, without phase matching.Source: NEURALGCM · p. 2

1.2. This Work

  • A parameter-free modification of MSE enables fine-tuning GraphCast to produce sharp deterministic forecasts, increase effective resolution, and improve tropical-cyclone intensity and surface-wind predictions.The approach interpolates between amplitude preservation and classical MSE.Source: §1.2 · p. 2

2. Method

MSE can favor under-amplitude predictions when features are only partly predictable, contributing to fine-scale smoothing. The method separates spectral amplitude error from decorrelation and uses the adjusted loss to fine-tune GraphCast.

2.1. Smoothing Is Optimal Under Mean Squared Error

  • When predictability is partial, MSE is minimized by reducing forecast variability, and its double penalty can encourage unrealistically smooth predictions.For correlation 0 < ρ < 1, the expected MSE is optimized at forecast standard deviation σX = ρ < 1.Source: §2.1 · p. 3

2.2. Spectral Separation of the Mean Squared Error

  • Fine scales are less predictable, and conventional models learn to reduce their spectral power faster than they learn to predict them, producing smoothing under MSE.During GraphCast training, amplitude ratio and coherence track each other; smoothing increases at longer forecast leads.Source: §2.2 · pp. 3–4

2.3. Spectrally Adjusted Mean Squared Error

  • AMSE separates spectral magnitude from coherence, remains zero only when prediction matches target, and is parameter-free without cutoff-scale or term-weight selection.Its gradients favor increased coherence and correct spectral magnitude, though it does not satisfy the triangle inequality.Source: §2.3 · p. 4

2.4. Fine-Tuning Methodology

  • Fine-tuning starts from the operational GraphCast checkpoint, begins with single-step forecasts, then increases forecast length in stages.Training used HRES-fc0 data from 2016–2021, with 2022 reserved for evaluation.Source: §2.4 · p. 5

3. Results

Across the 2022 evaluation, AMSE AR12 avoids the control model’s fine-scale dissipation, improves lagged-ensemble CRPS with little eRMSE change, and better captures cyclone intensity and wind-speed extremes.

3. Results

  • The primary AMSE AR12 model was trained over 12 forecast steps and evaluated against the control over 2022 using HRES initialization and ground truth.Source: §3 · p. 5

3.1. Effective Resolution

  • The control’s 5-day forecasts reach the 25%-energy-loss cutoff at about 1,250 km, while AMSE AR12’s long forecasts show a noise-based effective resolution of about 160 km.The control’s cutoff-based resolution is sensitive to the chosen threshold; AMSE AR12 instead has excess small-scale energy at long leads.Source: §3.1 · p. 6
  • The AMSE AR1 model’s noise-based effective resolution is about 450 km, and its longer forecasts are less coherent with the analysis and less skillful at all scales.Source: §3.1 · p. 6

3.2. Lagged Ensemble Verification

  • The results report CRPS improvements with little eRMSE change; AR1 has greater ensemble spread and significantly reduced CRPS, while AR1/AR12 spread/error ratios converge for most variables at longer leads.Source: §3.2 · p. 7

3.3. Hurricane Prediction and Extreme Weather

  • AMSE AR12 retains HRES intensity quality while the control’s weak-intensity bias grows with lead time, and their storm-location predictions are equivalent.For Hurricane Ian, both models predicted the five-day storm location within about 125 km, but the control predicted an unrealistically weak storm.Source: §3.3 · p. 7
  • AMSE AR12 provides more realistic surface wind-speed extremes than the control, while both models produce realistic temperature extremes.Source: §3.3 · p. 7

4. Discussion & Limitations

The modified loss separates spectral-amplitude error from decorrelation, yielding finer effective resolution and improvements in ensemble spread and tropical-cyclone intensity predictions. The discussion qualifies these results by the resolution definition, grid structure, and remaining training and evaluation questions.

4. Discussion & Limitations

  • MSE encourages averaging at unpredictable small scales, smoothing forecasts and reducing local variance; AMSE separates spectral-amplitude error from decorrelation.The authors describe the AMSE decomposition as encouraging realistic spectra even when accurate prediction is not possible.Source: §4 · pp. 7–8
  • AMSE fine-tuning of ¼°, 13-level GraphCast improves effective resolution, lagged-ensemble CRPS verification, and tropical-cyclone weak-intensity bias.The authors describe the fine-tuning process as abbreviated.Source: §4 · p. 8
  • Autoregressive multi-step fine-tuning leaves some mesoscale smoothing, possibly because forecasts must also provide good initial conditions for subsequent steps.The authors identify replay-buffer training as a direction for future work on retaining long-range skill while improving amplitude prediction.Source: §4 · p. 8
  • Using AMSE throughout training remains an untested possibility because a thorough test would require substantial computational budget.The authors note that MSE’s coherence-dependent smoothing might regularize gradients and speed training, while AMSE could let the model see fine-scale modes for longer.Source: §4 · p. 8

4.1. Effective Resolution

  • The paper defines effective resolution through fine-scale smoothing, so its resolution claim depends on the chosen definition.Other definitions exist, including diffusion- and dispersion-based definitions.Source: §4.1 · p. 8

4.2. Alternative Grids

  • The spectral method uses Parseval’s theorem and spherical harmonic transforms, which are straightforward on global latitude/longitude grids but may be difficult on local-area grids without regular global structure.Other multiscale decompositions, such as wavelet lifting, may preserve the intuition if suitable scale-dependent variance and correlation measures can be found.Source: §4.2 · p. 9

4.3. Applications to Other Domains

  • AMSE is motivated for weather by a physically meaningful, relatively stable spectrum, where errors in large-scale location need not change small-scale energy.The paper notes that use in other domains is most promising when sub-aggregations of the spectrum are meaningful, particularly in fluid dynamics.Source: §4.3 · p. 9

4.4. Applications to Ensemble Modelling

  • AMSE-based training improves forecast spread in an ensemble setting without dedicated ensemble-based training.The authors propose exploring it as a basis for ensemble forecasts whose individual members have realistic trajectories alongside whole-ensemble CRPS-like optimization.Source: §4.4 · p. 9

Impact Statement

Accurate extreme-weather forecasts can save lives, but forecast improvements should not be judged only by easily calculated scores. The authors present more realistic variability and improved probabilistic scores and tropical-cyclone intensity predictions, while noting that physical plausibility and out-of-distribution behavior remain unestablished.

Impact Statement

  • Accurate forecasts of extremes such as tropical cyclones save lives, motivating improvements that reflect what matters rather than merely chasing scores.The authors caution against confusing easy-to-calculate measures with what is truly important.Source: Impact Statement · p. 9
  • The work reports more realistic moderate- and fine-scale variability, improved probabilistic scores, and better tropical-cyclone intensity predictions, but not guaranteed complete physical plausibility..Source: Impact Statement · p. 9
  • The authors hope the work will ease adoption of better-performing data-driven forecasting systems, while noting that operational centres rigorously evaluate models before deployment.Source: Impact Statement · p. 9

A. Relationship to Maximum Likelihood Estimation

The appendix frames spectral modes as random variables to examine AMSE through a maximum-likelihood lens. Under that formulation, KL minimization gives a numerically determined forecast spread ratio, though the resulting loss has problematic behavior near perfect correlation.

A. Relationship to Maximum Likelihood Estimation

  • The spectral decomposition is physically meaningful because fine-scale atmospheric dynamics are increasingly rotationally symmetric and position-invariant, with amplitudes resembling Gaussian draws.Source: A. Relationship to Maximum Likelihood… · p. 13
  • The analysis treats modes at each total wavenumber as independent Gaussian random variables and models the forecast as correlated with the ground truth, with its own standard deviation.This setup frames deterministic-forecast realism under limited predictability as a maximum-likelihood problem using KL divergence, also called the continuous ignorance score.Source: A. Relationship to Maximum Likelihood… · p. 13
  • The KL objective uses the ground-truth and forecast probability densities, with the forecast density depending on the observed ground truth.Source: A. Relationship to Maximum Likelihood… · p. 13
  • Figure 7 visualizes high-pass-filtered forecast and analysis fields for the forecast shown in Figure 1.Source: Figure 7 · p. 14
  • The KL-derived optimum forecast-to-ground-truth standard-deviation ratio is below one, reaching about 0.66 near ρ = 0.4, but the objective is singular near ρ = 1.The authors say implementing this loss seems impractical because the implied random part of the prediction collapses to zero variance near perfect correlation.Source: A. Relationship to Maximum Likelihood… · p. 14

B.1. Visualization

The visualization isolates high-wavenumber components of a sample forecast and shows that AMSE training enhances forecast variability at those modes. A fourth-order spherical-harmonic filter emphasizes scales of 800 km and shorter.

B.1. Visualization

  • AMSE training enhances high-mode variability while leaving the control and AMSE forecast structures very similar.The visualization applies a fourth-order filter in spherical-harmonic space.Source: §B.1 · p. 14
  • The filter uses a cutoff wavenumber k0 = 50 to emphasize length scales of 800 km and shorter.Source: §B.1 · p. 14
  • Figure 8 shows amplitude spectral density for the variables of Figure 4 at 6h, 120h, and 240h lead times.Source: Figure 8 · p. 15

B.2. Spectra

The spectra show that most variables follow power-law distributions, while energy varies greatly across wavenumbers. The 2m temperature field shows little smoothing, regardless of model, in association with the constant elevation input.

B.2. Spectra

  • Most variables’ spectra follow power-law distributions, while atmospheric energy is ultimately removed by turbulent, frictional dissipation.No practical global atmospheric model can effectively resolve the scales at which this dissipation occurs.Source: §B.2 · p. 15
  • The 2m temperature field shows very little smoothing compared with the analysis, regardless of model.The passage attributes this to the field’s strong dependence on local elevation, supplied as a constant field.Source: §B.2 · p. 15
  • Available energy per total wavenumber varies over several orders of magnitude, so large amplitude-density differences can look small on log-log plots.Source: §B.2 · p. 15

B.3. Quantile/Quantile Plots

The quantile-quantile plots compare forecast and observed percentile values at North American stations to assess forecast distributions independently of skill. They show more realistic surface-wind distributions for AMSE AR12, while the models are essentially equivalent for 2m temperature.

B.3. Quantile/Quantile Plots

  • The plots compare observed percentiles with HRES analysis and forecast percentiles at North American weather stations for winter and summer periods.The y = x line indicates matching aggregated distributions; departures indicate systematic extreme-value bias.Source: §B.3 · p. 15, Figure 9 · p. 16
  • AMSE AR12 represents surface-wind distributions more realistically, matching HRES trends, while control forecasts shift toward weaker winds, particularly in summer.Departures from the y = x line indicate systematic under- or overprediction of extremes.Source: §B.3 · p. 15, Figure 9 · p. 16
  • The models are essentially equivalent in the distribution of 2m temperatures, where the control model shows little smoothing.AMSE AR12’s 2m temperature improvements appear more in forecast skill than in unconditional temperature distributions.Source: §B.3 · pp. 15–16

B.4. Details of the Lagged Ensemble Verification

AMSE AR12 improves CRPS skill for most variables and lead times, with precipitation exceptions; its eRMSE advantage emerges at long lead times. The lagged-ensemble evaluation uses CRPS, eRMSE, and spread-error ratio, with cautions about equal weighting.

B.4. Details of the Lagged Ensemble Verification

  • The evaluation uses fair CRPS, ensemble RMSE (eRMSE), and spread-error ratio (SER) for lagged ensembles across atmospheric and surface variables.SER is the ratio of grid-integrated MSE to unbiased ensemble-spread variance, taken after grid averaging.Source: §B.4 · p. 16
  • Significance is assessed by bootstrapping one-third of the forecast dates, yielding an average 36h gap between sampled dates.The passage treats samples spaced this way as reasonably independent because persistence skill decays quickly over 36h.Source: §B.4 · pp. 16–17
  • AMSE AR12 improves CRPS skill for most variables and lead times, but total precipitation improves only slightly at long leads and degrades at short leads.The passage attributes the precipitation result to spectral-mode separation being unnatural for localized, nonnegative precipitation.Source: §B.4 · p. 17
  • Interpret eRMSE skill cautiously: the scores assume ideal ensemble members, unlike lagged ensembles whose shorter-duration members should be more accurate.The evaluation does not separately weight each term; the authors believe control’s early-lead smoothing may make equal weighting more optimal for its lagged ensemble.Source: §B.4 · p. 17
  • At long lead times, AMSE AR12 begins to show eRMSE skill over control as the relative degradation is stronger from 0.5 to 4.5 days than from 6 to 10 days.Source: §B.4 · p. 17

B.4.1. UNBIASED ENSEMBLE ROOT MEAN SQUARED ERROR

The section corrects finite-ensemble bias in eRMSE by subtracting a standard-error term before spatial averaging and taking the square root. The adjustment slightly improves AMSE-tuned scores over the control model.

B.4.1. UNBIASED ENSEMBLE ROOT MEAN SQUARED ERROR

  • The overestimate is more severe for small ensembles, such as the lagged ensemble, whose size cannot easily be increased.The bias is illustrated using realizations drawn from N(µ, σ²) with ground truth zero.Source: §B.4.1 · p. 17
  • Subtracting the standard-error term corrects eRMSE’s finite-sample overestimate, with the correction applied pointwise before spatial averaging and taking the square root.The correction is proposed for the ensemble mean squared error estimator.Source: §B.4.1 · pp. 17–18
  • The unbiased adjustment slightly improves AMSE-tuned scores over the control in the lagged ensemble.Figure 13 reports the unbiased ensemble RMSE scorecard.Source: §B.4.1 · p. 18, Figure 13 · p. 20

B.5. Ablation Studies

The ablations show that longer training alone is not the comparison of interest: only AMSE retains fine-scale sharpness, while MAE moderately sharpens medium scales and improves lagged-ensemble CRPS.

B.5. Ablation Studies

  • Only AMSE retains fine-scale sharpness; MAE moderately sharpens medium scales and improves lagged-ensemble CRPS despite residual smoothing.The MAE sharpness gain is reported for wavenumbers 32–200, corresponding to 1,250–250 km scales.Source: Figure 14 · p. 21, §B.5 · p. 21
  • The MSE AR12 and MAE AR12 controls use the same fine-tuning schedule, with MAE preserving the per-variable and per-level error weightings.The additional fine-tunings test whether the results are simply an artifact of increased training time.Source: §B.5 · p. 21
  • At longer lead times, the MAE-optimal median can remain smoother because fine scales are less predictable than coarser scales.The paper offers this as an intuitive explanation for smoothing under mean absolute error.Source: §B.5 · p. 21
Loading 2501.19374v2…