PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Day Forecasts from Sparse Observations

Marcin Andrychowicz, Lasse Espeholt, Di Li, Samier Merchant, Alexander Merose, Fred Zyda, Shreya Agrawal, Nal Kalchbrenner

arXiv:2306.06079v3physics.ao-phcs.LG

TL;DR

Observation-trained neural weather models had demonstrated good performance only up to 12 hours and for precipitation, while NWP simulations are computationally expensive. MetNet-3 combines sparse and dense observations with a densification approach to forecast precipitation and surface variables up to 24 hours; it outperforms ENS on reported precipitation and surface-variable comparisons.

  • Problem

    Observation-trained neural weather models had demonstrated good performance only up to 12 hours and for precipitation, while NWP forward simulations require thousands of CPU hours.

    Source: §1 · p. 1

  • Method

    MetNet-3 uses dense and sparse observations, with densification that masks station inputs during training and produces dense forecasts from sparse station data.

    Source: §1 · p. 2

  • Results

    MetNet-3 outperforms ENS on thresholded precipitation CSI comparisons and has better CRPS and MAE than multi-member ENS for all surface variables across the full 24-hour lead-time range.

    Source: §2.1 · p. 4, §2.2 · p. 6

  • Takeaways & Limitations

    MetNet-3 uses observations with latency on the order of minutes and an HRRR state with 55-minute latency, rather than relying on ENS forecasts that may use information up to 12 hours old.

    Source: §3 · p. 8

    MetNet-3 still relies on an assimilated NWP initial state because incorporating all relevant observational sources remains challenging.

    Source: §1 · p. 2

Abstract

from arXiv · show

Deep neural networks offer an alternative paradigm for modeling weather conditions. The ability of neural models to make a prediction in less than a second once the data is available and to do so with very high temporal and spatial resolution, and the ability to learn directly from atmospheric observations, are just some of these models' unique advantages. Neural models trained using atmospheric observations, the highest fidelity and lowest latency data, have to date achieved good performance only up to twelve hours of lead time when compared with state-of-the-art probabilistic Numerical Weather Prediction models and only for the sole variable of precipitation. In this paper, we present MetNet-3 that extends significantly both the lead time range and the variables that an observation based neural model can predict well. MetNet-3 learns from both dense and sparse data sensors and makes predictions up to 24 hours ahead for precipitation, wind, temperature and dew point. MetNet-3 introduces a key densification technique that implicitly captures data assimilation and produces spatially dense forecasts in spite of the network training on extremely sparse targets. MetNet-3 has a high temporal and spatial resolution of, respectively, up to 2 minutes and 1 km as well as a low operational latency. We find that MetNet-3 is able to outperform the best single- and multi-member NWPs such as HRRR and ENS over the CONUS region for up to 24 hours ahead setting a new performance milestone for observation based neural models. MetNet-3 is operational and its forecasts are served in Google Search in conjunction with other models.

1 Introduction

Neural weather models offer fast, high-resolution forecasts, but prior observation-trained models had demonstrated strong performance only up to 12 hours and for precipitation. MetNet-3 extends forecasts to 24 hours and multiple variables, using densification to turn sparse station observations into dense forecasts.

1 Introduction

  • NWP systems assimilate observations into a dense initial atmospheric representation, but their forward simulations require thousands of CPU hours and constrain forecast resolution.Forecast resolution affects computational cost.Source: §1 · p. 1
  • Prior high-resolution neural weather models had demonstrated good performance only up to 12 hours and for precipitation alone.Combining varied observational sources is also an engineering challenge, and sparse inputs must support dense forecasts.Source: §1 · pp. 1–2
  • MetNet-3 extends observation-trained forecasts to 24 hours for precipitation and surface variables, including temperature, dew point, wind speed, and direction.It retains 2-minute temporal frequency and up to 1 km spatial resolution.Source: §1 · p. 2
  • Densification masks station inputs while retaining them as targets, evaluates held-out and observed stations, and uses spatial parameter sharing to produce dense forecasts from all training stations.Figure 1 depicts the training, evaluation, and inference aspects of this procedure.Source: §1 · p. 2, Figure 1 · p. 2
  • MetNet-3 still relies on an assimilated NWP initial state because incorporating all relevant observational sources remains challenging.The state provides a dense, somewhat diverging estimate of the surface variables being forecast.Source: §1 · p. 2

2 Results

MetNet-3 is evaluated over CONUS for precipitation and sparse surface-variable forecasts, with probabilistic comparisons against NWP baselines. It outperforms ENS on instantaneous precipitation CRPS across 24 hours, on hourly accumulated precipitation CRPS up to 19 hours, and on surface-variable metrics across 24 hours.

2 Results

  • The evaluation covers instantaneous and hourly accumulated precipitation plus 2m temperature, 2m dewpoint, 10m wind speed, and 10m wind direction over CONUS.Precipitation and surface-variable forecasts are evaluated against MRMS and OMO station observations, respectively.Source: §2 · p. 3

2.2 Sparse Surface Variables

  • Surface-variable evaluation uses weather-station observations from 942 CONUS locations, with a 20% hold-out set excluded from training and evaluation inputs.The targets are sparse station observations, and the densification procedure is applied before evaluation on the held-out stations.Source: §2 · p. 3, §2.2 · p. 6

2.1 Precipitation

  • MetNet-3 outperforms ENS on instantaneous precipitation-rate CRPS across the full 24-hour lead-time range.It also outperforms ENS on CSI for light precipitation through 15 hours and for heavy precipitation through 24 hours.Source: §2.1 · p. 4, Figure 5 · p. 5

(a) CRPS

  • MetNet-3 outperforms ENS and HREF on hourly accumulated precipitation CRPS through 19 hours and on moderate-precipitation CSI through 18 hours.Hourly MRMS estimates provide substantially fewer training frames than instantaneous precipitation estimates.Source: Figure 6 · p. 5, (a) CRPS · p. 6

2.2 Sparse Surface Variables

  • For all four surface variables, MetNet-3 achieves better CRPS and MAE than ENS across the full 24-hour lead-time range.Most observed values fall within MetNet-3’s 80% confidence interval, while ENS’s predicted distribution is described as peaked and under-dispersed.Source: §2.2 · p. 6
  • A 20-hour MetNet-3 temperature forecast has similar MAE to a five-hour forecast from the best-performing baseline.In the hyperlocal setting, where test-station values are supplied as inputs during evaluation, results improve further, especially at early lead times.Source: §2.2 · p. 6

3 Discussion

The discussion reports a substantial gain over MetNet-2, while noting mismatches between observations and NWP states and differences in operational latency. It also illustrates a newly forming precipitation pattern and compares ground-variable forecasts across probabilistic models.

3 Discussion

  • ERA5 reanalysis precipitation can markedly mismatch gauge-corrected MRMS, and surface-temperature MAE against OMO stations is 1.5 C for HRES and 0.9 C for HRRR.The authors caution that training or evaluating against NWP states can make model accuracy difficult to gauge; ERA5 is also about 25 km resolution at 6-hour frequency.Source: §3 · p. 6
  • The precipitation case study depicts a newly forming large central-US pattern rather than simple extrapolation of existing patterns.It shows probabilities above 1 mm/h, CSI-optimized thresholds, and CSI values on the evaluation mask.Source: Figure 7 · p. 7
  • Figure 8 compares probabilistic MetNet-3 and NWP ground-variable forecasts using CRPS and MAE, with lower values indicating better performance.Its CRPS plots omit deterministic HRRR and HRES baselines; HREF variables were unavailable for these comparisons.Source: Figure 8 · p. 8
  • ENS runs four times daily with up to 12-hour-old atmospheric information, while MetNet-3 uses minute-latency observations and a 55-minute-latency HRRR state.MetNet-3 takes another 10 minutes to forecast all CONUS lead times, and the authors say adjusting for operational latency would increase its gains over NWP baselines.Source: §3 · p. 8
  • MetNet-3 shows a substantial performance gain over MetNet-2, continuing a similar improvement from the original MetNet.The authors attribute the gain to multiple innovations and note that the model still uses only a tiny fraction of available atmospheric data.Source: §3 · pp. 8–9

4 Methods

The methods combine diverse atmospheric inputs with learned topographical context and a multi-scale network to predict precipitation and surface variables. Training addresses sparse station targets, varying loss scales, and the computational demands of large spatial contexts.

4.1 Dataset Creation

  • Training pairs combine recent radar, station, satellite, assimilated-state, and geospatial/time inputs with future radar, station, and state targets, using sequential time splits and held-out weather stations.The data span July 2017 to September 2022; training, validation, and test periods do not overlap, and test-station data are excluded from training.Source: §4.1 · pp. 9–10

4.2 Model and Architecture

  • MetNet-3 combines topographical embeddings, a U-Net backbone, and a MaxVit transformer; learned 4-km embedding-grid vectors are bilinearly interpolated at input pixel centers.Each grid point has 20 scalar parameters, trained alongside the other model parameters.Source: §4.2 · p. 10, §4.2.1 · p. 10

4.2.2 Network Architecture

  • High-resolution small-context and low-resolution large-context inputs feed a U-Net and modified MaxVit, producing categorical forecasts for six ground variables and precipitation plus deterministic assimilated-state predictions.The network crops and upsamples representations; precipitation outputs are further upsampled to 1-km resolution.Source: §4.2.2 · pp. 10–11, Figure 11 · p. 12

4.2.3 Conditioning with Lead Time

  • Lead time is encoded across 0–24 hours at 2-minute intervals and applied additively and multiplicatively to inputs and hidden representations.Training samples lead times exponentially, with 24-hour leads sampled ten times less often than zero-hour leads; the authors report improved results at all lead times.Source: §4.2.3 · p. 12

4.3 Training

  • Training uses cross-entropy for probabilistic targets and MSE for assimilated-state targets; random masking of 25% of OMO stations encourages predictions at unobserved locations.The assimilated-state loss is included to improve forecasts of other variables, not as an evaluated output.Source: §4.3 · p. 12, §4.3.1 · p. 12

4.3.1 Densification

  • A separate ground-variable-specialized model increases OMO loss weight by 100x and disables topographical embeddings, trading some precipitation quality for improved ground-variable forecasts.The authors identify a trade-off between precipitation and ground-variable forecast quality in a single model.Source: §4.3.1 · pp. 12–13

4.3.2 Loss Scaling

  • Dynamic gradient rescaling balances output-channel influence across losses with different magnitudes and bounds the effect of corrupted target channels.The method rescales each sample's gradient so that each output channel has the same L1 norm without changing the sample's overall gradient magnitude.Source: §4.3.2 · p. 13

4.3.3 Hardware Configuration

  • The large spatial context is distributed across a 4-by-4 grid of 16 interconnected TPU cores rather than reducing resolution; the full model trains on 512 TPUv3 cores in seven days.MaxVit attention windows are partitioned so each full window is processed on one core.Source: §4.3.3 · p. 13

Supplemental Material to Deep Learning for Day Forecasts

The supplemental material front matter lists the paper's authors, affiliations, and preprint information.

Supplemental Material

A Data

The data section figure compares hourly accumulated precipitation from gauge-corrected MRMS and ERA5 reanalysis for a specified timestamp.

A Data

  • The figure presents hourly accumulated precipitation from MRMS and ERA5 reanalysis.Source: Figure 1 · p. 16
  • MRMS is the gauge-corrected product, while ERA5 supplies reanalysis data.Source: Figure 1 · p. 16
  • The comparison is for Sat Nov 30 2019 12:00 UTC.Source: Figure 1 · p. 16

B Supplement: Model and Training

The supplement details MetNet-3’s input preprocessing, network components, and lead-time conditioning. It also specifies training regularization and topographical embeddings.

Inputs

  • Inputs combine precipitation channels from high- and low-resolution MRMS with time-offset observations, normalized channels, and 512-dimensional embeddings.Precipitation rates are transformed as tanh(log(r + 1)/4).Source: B Supplement: Model and Training · p. 16

Network

  • The network uses modified MaxVit blocks and a U-Net downsampling and upsampling path, with 512 channels throughout and mixed-precision computation.MaxVit modifications include removing MLP sub-blocks, normalized keys and queries, and skip connections from each block’s output.Source: B Supplement: Model and Training · p. 17

Lead Time Conditioning

  • Lead time is encoded as a one-hot input and applied through additive-multiplicative conditioning at the inputs and after each layer normalization.The conditioning MLP’s second layer is initialized to make conditioning an identity function at initialization.Source: B Supplement: Model and Training · p. 17

Topographical Embeddings

  • Topographical embeddings cover a specified region on a 4 km grid, yielding 3M points and 60M trainable parameters.Each embedding has size 20.Source: Topographical Embeddings · p. 17

Initialization

  • Initialization uses LecunNormal and rescales each MaxVit sub-block’s final linear layer by 1/N, where N is the number of sub-blocks on its residual connection.Source: Topographical Embeddings · p. 17

Regularization

  • Dropout, stochastic depth, and AdamW weight decay provide regularization; stochastic-depth probability increases linearly from 0 to 0.2 through the network.Dropout rate is 0.1 and weight decay coefficient is 0.1.Source: Topographical Embeddings · p. 17

B.1 Outputs, Targets and Losses

The supplement describes how MetNet-3 rescales losses to balance their magnitudes, including separate weighting for HRRR and OMO targets.

B.1 Outputs, Targets and Losses

  • MSE targets are rescaled to approximately zero mean and unit standard deviation per variable, alongside dynamic gradient rescaling.Source: §B.1 · p. 18
  • The HRRR loss is multiplied by 10 and divided by the 617 predicted HRRR channels.Source: §B.1 · p. 18
  • HRRR channels matching OMO ground variables receive 30× weight, while remaining channel weights decrease to preserve the average HRRR-channel weight.Source: §B.1 · p. 18
  • OMO target channels have equal weights whose sum rises from 0.01 for the standard model to 1 for OMO fine-tuning.Source: §B.1 · p. 18

C Supplement: Evaluation

The evaluation supplement filters comparisons to times and locations with appropriate data coverage and explains why precipitation CRPS may decrease with lead time. It also describes evaluation masks and the OMO station split.

C Supplement: Evaluation

  • Evaluation locations and times are filtered to those with historical forecasts for every baseline; ENS availability restricts evaluation to two daily runs, 00 and 12 UTC.Source: C Supplement: Evaluation · p. 18
  • CRPS can decrease counterintuitively with lead time because expected precipitation varies by lead time and higher precipitation amounts generally yield higher CRPS.The supplement says this pattern is not observed in CSI plots because CSI is normalized.Source: C Supplement: Evaluation · p. 19
  • Although all available MRMS data are used for training, evaluation uses only locations with the highest radar-data quality.Figure 2 identifies training and evaluation masks for MRMS and HRRR targets.Source: C Supplement: Evaluation · p. 19, Figure 2 · p. 19

OMO ground truth

  • The OMO weather-station figure distinguishes test stations in blue from training stations in green.Source: Figure 3 · p. 19

D Supplement: Evaluation Metrics

The supplement evaluates forecast quality with CRPS, CSI, and MAE. It describes CSI’s categorical thresholding procedure and CRPS as a discretized comparison of forecast and observed distributions.

D Supplement: Evaluation Metrics

D.1 Critical Success Index (CSI)

  • CSI evaluates precipitation forecasts as a binary categorical score, with true positives divided by true positives, false negatives, and false positives.Because forecast outputs are probability distributions, a validation-set probability threshold is optimized separately for each lead time and precipitation-rate category.Source: §D.1 · p. 19
  • The CSI thresholding procedure classifies a rate category as positive when its predicted probability mass exceeds the optimized threshold.The threshold is selected on held-out validation data separately for each lead time and rate category.Source: §D.1 · p. 19

D.2 Continuous Ranked Probability Score (CRPS)

  • CRPS compares predicted and observed cumulative distributions across possible values, calculated over discretization bins.The bin-wise expression uses the probability that the target is at or below each bin’s upper end and weights by bin size.Source: §D.2 · p. 20

D.3 Mean Absolute Error (MAE)

Mean Absolute Error (MAE) measures the absolute difference between a deterministic prediction and ground truth. For MetNet-3 and ensemble baselines, the forecast median is used; MAE is not applied to precipitation because its distributions are very skewed.

D.3 Mean Absolute Error (MAE)

  • MAE is the absolute difference between a deterministic prediction and ground truth.The paper defines it as MAE = |ŷ − y|.Source: §D.3 · p. 20
  • For MetNet-3 and ensemble baselines, the forecast median is used because it minimizes MAE for a perfect model.Source: §D.3 · p. 20
  • MAE is not applied to precipitation because its distributions are very skewed.Source: §D.3 · p. 20

E Supplement: Additional Results

The supplement lists additional results on precipitation, wind, finetuning, and model comparisons, with figures presenting the associated metrics and comparisons. It also includes ablations of topographical embeddings and large-context inputs.

E Supplement: Additional Results

  • Supplementary figures include precipitation CRPS plots with deterministic baselines and CSI plots for additional instantaneous precipitation rates.Figures 4 and 5 are identified as the CRPS and CSI plots, respectively.Source: E Supplement: Additional Results · p. 20

(f) 20 mm

E Supplement: Additional Results

(a) CRPS

E Supplement: Additional Results

Loading 2306.06079v3…