Source-linked AI summary
Neural networks for post-processing ensemble weather forecasts
Stephan Rasp, Sebastian Lerch
TL;DR
Systematic errors in ensemble forecasts motivate probabilistic post-processing beyond parametric models with pre-specified link functions. The paper uses neural networks with auxiliary predictors and station embeddings, finding significant improvements over benchmark methods in German 2-meter temperature forecasts while being computationally more efficient.
Problem
Parametric post-processing models require choosing a predictive distribution and suitable link functions, limiting flexibility across weather variables.
Method
Neural networks learn mappings from ensemble and auxiliary predictors to Gaussian distribution parameters, using station embeddings and CRPS optimization.
Results
NN-aux-emb improves the best benchmark technique EMOS-loc-bst by 3%, for a total improvement of 29% compared to the raw ensemble.
Takeaways & Limitations
The approach indicates that local information, auxiliary variables, and nonlinear predictor–parameter relationships are important for skillful probabilistic temperature forecasts.
Takeaways & Limitations
Convolutional and recurrent neural networks for temporal dependencies caused more overfitting without improving validation scores in these tests.
Abstract
from arXiv · showhide
Ensemble weather predictions require statistical post-processing of systematic errors to obtain reliable and accurate probabilistic forecasts. Traditionally, this is accomplished with distributional regression models in which the parameters of a predictive distribution are estimated from a training period. We propose a flexible alternative based on neural networks that can incorporate nonlinear relationships between arbitrary predictor variables and forecast distribution parameters that are automatically learned in a data-driven way rather than requiring pre-specified link functions. In a case study of 2-meter temperature forecasts at surface stations in Germany, the neural network approach significantly outperforms benchmark post-processing methods while being computationally more affordable. Key components to this improvement are the use of auxiliary predictor variables and station-specific information with the help of embeddings. Furthermore, the trained neural network can be used to gain insight into the importance of meteorological variables thereby challenging the notion of neural networks as uninterpretable black boxes. Our approach can easily be extended to other statistical post-processing and forecasting problems. We anticipate that recent advances in deep learning combined with the ever-increasing amounts of model and observation data will transform the post-processing of numerical weather forecasts in the coming decade.
1. Introduction
Neural networks are proposed as a flexible approach to probabilistic post-processing that can learn nonlinear relationships and incorporate diverse predictors. The study evaluates this approach for German station-based 2-meter temperature forecasts against benchmark methods.
- Ensemble forecasts quantify flow-dependent uncertainty but retain systematic errors requiring statistical post-processing.
- Distributional regression models estimate predictive-distribution parameters conditionally on predictors, commonly using specified parametric families and link functions.
- Neural networks provide a flexible machine-learning alternative capable of modeling arbitrary nonlinear functions with arbitrary predictors.
- The study applies neural networks to 2-meter temperature forecasts at German surface stations using data from 2007–2016.
- The authors compare neural-network configurations with benchmark post-processing methods across different training-period lengths and examine predictor importance.
2. Data and notation
The study evaluates 48-hour ECMWF ensemble temperature forecasts against German station observations using a held-out 2016 validation year. Predictors combine ensemble summaries with station-specific and auxiliary information.
- The dataset contains daily 50-member ECMWF ensemble forecasts initialized at 00 UTC for 2-meter temperature at a 48-hour lead time.
- Forecast cases span 3 January 2007 through 31 December 2016, with 2016 reserved for validation.
- Evaluation covers 537 German weather stations, with observations available at 499 stations during 2016.
- After missing observations are removed, the validation, long-training, and short-training sets contain 182 218, 1 626 724, and 180 849 samples, respectively.
- The predictor vector Xs,t combines ensemble means and standard deviations with station-specific features and has dimension p = 42.
3. Benchmark post-processing techniques
Benchmark methods include parametric EMOS models, boosted EMOS, and quantile regression forests. They differ in distributional assumptions, predictor handling, estimation procedures, and local or global pooling.
- EMOS: EMOS models represent the conditional weather distribution parametrically, with parameters linked to ensemble predictors through functions estimated from training data.
- EMOS: Standard EMOS temperature models use Gaussian predictive distributions whose mean and standard deviation depend on ensemble mean and spread.
- EMOS: Local EMOS estimates station-specific coefficients, whereas global EMOS pools stations and cannot correct location-specific ensemble deficiencies.
- Boosting: EMOS-loc-bst uses boosting to include additional predictors while selecting variables iteratively according to improvement in model fit.
- Quantile regression forests: Quantile regression forests provide a non-parametric alternative that avoids choosing a parametric distribution family but restrict predicted quantiles to the training observations’ range.
4. Neural networks
The proposed neural networks map forecast predictors to Gaussian distribution parameters and are trained with the CRPS. Auxiliary variables and station embeddings extend the models beyond basic ensemble summaries.
- The networks use input features and hidden layers of weighted nodes with nonlinear activation functions, optimized by stochastic gradient descent using Adam.
- Fully-connected networks model outputs as linear combinations of inputs, whereas neural networks with a hidden layer represent nonlinear relationships.
- For post-processing, the output layer represents Gaussian mean µs,t and standard deviation σs,t, while the Gaussian CRPS is used as the loss function.
- Auxiliary predictors: Auxiliary predictor variables are incorporated through the full predictor vector, producing FCN-aux and NN-aux configurations.
- Station embeddings: Station embeddings map each station ID to learned latent features concatenated with predictors; the study uses nemb = 2.
- Model configurations: The combined station-embedding and auxiliary-input models are denoted FCN-aux-emb and NN-aux-emb, with ensembles of ten networks used to stabilize complex-model estimates.
5. Results
Across the 2016 validation period, neural-network post-processing generally outperformed benchmark methods, especially when combining auxiliary predictors with station embeddings. Longer training data benefited complex nonlinear models, while post-processing improved calibration and remained computationally efficient.
- Overall performance: 13% relative improvement over raw ECMWF forecasts was obtained by EMOS-gl in mean CRPS, while adding local station information improved the global score by another 10%.FCN matched EMOS-gl closely, and FCN-emb mirrored the station-specific EMOS-loc design.
- Overall performance: 29% total improvement over the raw ensemble was achieved by NN-aux-emb, which also improved EMOS-loc-bst by 3%.For the 2015 training period, combining auxiliary variables and station embeddings produced the best reported results.
- Training-period effects: 4–5% skill increases occurred for EMOS-loc-bst and neural-network models with 2007–2015 training data, whereas linear models improved little.QRF also became one of the best models with the longer period, indicating a minimum data requirement for that method.
- Calibration: Post-processed forecast distributions were substantially better calibrated than the under-dispersed raw ensemble forecasts.The raw ensemble placed observations outside its forecast range too frequently, while post-processed PIT histograms showed much smaller deviations.
- Station-wise results: At most stations, all post-processing methods improved on the raw ensemble, but neural networks had more negative outliers than EMOS methods and QRF.Location-specific error characteristics may be easier for locally estimated benchmark models to capture at some stations.
- Statistical significance: 30% of stations showed NN-aux-emb significantly better than EMOS-loc-bst, versus 2% or fewer where it was worse.Significance was assessed using Diebold–Mariano tests with Benjamini–Hochberg adjustment for temporal and spatial dependencies.
- Computational cost: The most complex neural-network methods were at least twice as fast as EMOS-loc-bst, including an ensemble of ten realizations.QRF was roughly ten times slower than EMOS-loc-bst, while complex networks benefited from GPU execution.
6. Feature importance
Permutation importance identifies meteorological and station characteristics associated with the neural networks’ predictive performance. Adding nonlinear layers and embeddings changes both the sensitivity and distribution of feature importance.
- Interpretation caveats: Permutation importance cannot capture feature collinearities, although it avoids refitting the model after omitting each feature.For local boosting and QRF implementations, importance changes from station to station, making interpretation more difficult.
- Linear model: Station altitude and orography were the most important predictors after mean temperature in FCN-aux.The result is consistent with interpolation that does not adjust the model-grid elevation to the surface station height.
- Neural-network comparison: The nonlinear NN-aux-emb model increased overall sensitivity to permuted features and distributed importance more evenly.This comparison examined how feature importance changed after adding station embeddings and a nonlinear layer.
- Neural-network comparison: In NN-aux-emb, station altitude, orography, heat fluxes, and total cloud cover became more important.The reported increases included sensible and latent heat flux as well as total cloud cover.
- Meteorological interpretation: Shortwave flux and 850 hPa humidity were important predictors alongside mean forecast temperature and station altitude.The authors interpret these variables as indicators of insolation, with 850 hPa humidity apparently serving as a cloud-cover proxy.
- Meteorological interpretation: Temperature standard deviation was not an important feature for the post-processing models.This contrasts with the importance assigned to mean forecast temperature and several insolation-related variables.
7. Discussion
The discussion examines unsuccessful extensions and future directions, including richer distributional modeling, temporal learning, data augmentation, grid-wide prediction, and multivariate dependence.
- Distributional modeling: Parametric distribution choices are difficult for variables such as wind speed or precipitation, motivating methods that learn full forecast distributions.For temperature, however, a Gaussian distribution is described as a good approximation.
- Distributional modeling: A bin-based distributional approximation produced skill comparable only to the raw ensemble for temperature.The experiments used uniform components and predicted probabilities for prespecified temperature bins.
- Temporal dependence: Convolutional or recurrent networks increased overfitting without improving validation scores when used to model temporal dependencies.The authors suggest revisiting these approaches for other datasets.
- Data augmentation: Adding random noise to input features did not improve validation, while neighboring stations or grid points are proposed as an alternative augmentation source.The proposed alternative could augment a station's forecasts with spatially related data.
- Spatial and multivariate extensions: Grid-wide post-processing with convolutional networks could use large-scale weather patterns to produce spatially consistent predictions, but adds complexity and likely requires more training data.The extension would replace individual station locations with the entire two-dimensional grid.
- Spatial and multivariate extensions: The study considers one weather variable at one location and look-ahead time, leaving cross-variable, spatial, and temporal dependence as an extension target.Multivariate neural post-processing is identified as a promising starting point for future work.
8. Conclusion
The paper presents neural networks for ensemble-forecast post-processing, emphasizing flexible nonlinear modeling and station embeddings. The models outperform state-of-the-art techniques efficiently, while auxiliary variables and local information are especially valuable.
- Conclusion: Neural networks significantly outperform state-of-the-art post-processing techniques while being computationally more efficient.The conclusion identifies nonlinear relations and station embeddings as central advantages.
- Conclusion: Neural networks model arbitrary predictors and distribution parameters without pre-specified link functions and add station information through embeddings.Parameters are estimated by optimizing the CRPS for probabilistic forecasting.
- Conclusion: Encoding local information is very important for skillful probabilistic temperature forecasts, while auxiliary variables considerably improve results.The improvement is strongest when nonlinear predictor–parameter relations are allowed.
- Conclusion: A trained model can identify meteorological variables most important for correcting systematic temperature forecast errors.The authors therefore characterize neural networks as somewhat interpretable despite intractable individual-parameter interpretation.
- Conclusion: Neural networks are presented as a flexible and efficient addition to the toolkit for statistical post-processing and forecasting.The conclusion connects this role to increasing model and observation data and multifaceted forecast errors.
A. Hyperparameters for benchmark and network models
Appendix tables document the tuning parameters used for benchmark and network models, including definitions of key network-training quantities.
- Benchmark models: Table A1 lists hyperparameters for benchmark models and defines AIC as the Akaike information criterion.The caption is the supplied description of the benchmark-model hyperparameter table.
- Network models: Table A2 lists hyperparameters for network models, with parenthesized values indicating settings for the longer 2007–2015 training period.The table covers network-model settings across training-period configurations.
- Network models: For network models, parameters include weights, biases, and latent embedding features, while an epoch is one pass through all training samples.Batch size denotes the number of random training samples used per SGD gradient update.
B. Forecast evaluation
The paper evaluates probabilistic forecasts using calibration diagnostics and proper scoring rules, with CRPS as a central measure. Statistical comparisons use Diebold–Mariano tests while accounting for temporal and spatial dependence and multiple testing.
- Calibration: Verification-rank and PIT histograms assess calibration by comparing observed distributions of ranks or probability integral transforms with uniformity.Deviations from uniformity indicate systematic errors such as bias or under-represented forecast uncertainty.
- Scoring rules: Proper scoring rules evaluate calibration and sharpness simultaneously, with negatively oriented scores favoring smaller values.The logarithmic score and CRPS are identified as prominent examples.
- Scoring rules: CRPS-based optimum-score estimation determines forecast-distribution parameters by minimizing average CRPS over a training sample.CRPS is presented as a more robust meteorological alternative to likelihood-based estimation.
- Scoring rules: The continuous ranked probability skill score measures improvement relative to a reference forecast using averaged scores.It is introduced for comparative model assessment.
- Statistical testing: Diebold–Mariano tests compare mean forecast scores, with negative statistics favoring the first forecast and positive statistics favoring the second.The procedure accounts for temporal dependence through score-difference autocovariances and controls the false discovery rate across stations.
C. Additional results
Additional results examine verification diagnostics for models trained on different historical periods. The figures aggregate rank and PIT histograms over the 2016 evaluation period.
- Additional diagnostics: Figure C1 reports verification-rank and PIT histograms for raw and post-processed forecasts trained on data from 2015.The histograms aggregate all forecast cases evaluated during calendar year 2016.
- Additional diagnostics: Figure C2 reports verification-rank and PIT histograms for raw and post-processed forecasts trained on data from 2007–2015.The forecasts are aggregated over all cases in the 2016 evaluation period.
C.2. CRPSS results for alternative benchmark models
This section reports alternative benchmark comparisons, computational costs, and station-level significance tests. The results indicate broad significant differences between network models and standard EMOS approaches, with NN-aux-emb generally strongest.
- Alternative benchmarks: CRPSS comparisons are reported relative to EMOS-loc-boost using alternative reference models.Figure C3 presents the corresponding results with different reference models.
- Computational costs: Computation times cover model estimation for both training sets and out-of-sample forecast generation.The comparison is qualified because benchmark implementations used different programming languages and hardware environments.
- Statistical significance: NN-aux-emb has the highest ratios of significant score differences against competitors and is significantly outperformed at very few stations.Pair-wise one-sided Diebold–Mariano tests are performed across 499 stations, with Benjamini–Hochberg adjustment at nominal α = 0.05.