PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison

Benedikt Schulz, Sebastian Lerch

arXiv:2106.09512v1stat.MLcs.LGphysics.ao-phstat.AP

TL;DR

Wind gust postprocessing is important for severe-weather warnings, but few recent studies have focused on it. This paper systematically compares statistical and machine-learning approaches using German operational ensemble forecasts and station observations. All methods calibrate forecasts, while additional meteorological predictors improve skill and locally adaptive neural networks achieve the best forecasts among those compared.

  • Problem

    Despite wind gusts’ importance for severe-weather warnings, recent ensemble-postprocessing research has focused relatively little on gust forecasts.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically compares statistical and machine-learning postprocessing methods for wind gust ensembles, including a locally adaptive neural-network framework with different probabilistic output types.

    Source: §1 · p. 2, §6 · p. 25

  • Results

    All methods yield calibrated forecasts and correct raw-ensemble systematic errors; additional meteorological predictors significantly improve skill, and locally adaptive neural networks provide the best forecasts.

    Source: §6 · p. 23

  • Takeaways & Limitations

    The neural-network framework supports different probabilistic forecast types and learns physically consistent relations, including those associated with the diurnal cycle.

    Source: §6 · p. 25

    Most methods do not preserve spatial, temporal, or multivariable ensemble dependencies, a challenge for operational postprocessing.

    Source: §6 · p. 25

Abstract

from arXiv · show

Postprocessing ensemble weather predictions to correct systematic errors has become a standard practice in research and operations. However, only few recent studies have focused on ensemble postprocessing of wind gust forecasts, despite its importance for severe weather warnings. Here, we provide a comprehensive review and systematic comparison of eight statistical and machine learning methods for probabilistic wind gust forecasting via ensemble postprocessing, that can be divided in three groups: State of the art postprocessing techniques from statistics (ensemble model output statistics (EMOS), member-by-member postprocessing, isotonic distributional regression), established machine learning methods (gradient-boosting extended EMOS, quantile regression forests) and neural network-based approaches (distributional regression network, Bernstein quantile network, histogram estimation network). The methods are systematically compared using six years of data from a high-resolution, convection-permitting ensemble prediction system that was run operationally at the German weather service, and hourly observations at 175 surface weather stations in Germany. While all postprocessing methods yield calibrated forecasts and are able to correct the systematic errors of the raw ensemble predictions, incorporating information from additional meteorological predictor variables beyond wind gusts leads to significant improvements in forecast skill. In particular, we propose a flexible framework of locally adaptive neural networks with different probabilistic forecast types as output, which not only significantly outperform all benchmark postprocessing methods but also learn physically consistent relations associated with the diurnal cycle, especially the evening transition of the planetary boundary layer.

1. Introduction

Wind gusts pose important warning challenges, yet their small-scale, local drivers limit predictability, and prior ensemble-postprocessing research has rarely focused on gusts. This paper reviews and systematically compares statistical and machine-learning postprocessing methods tailored to wind gust forecasts.

1. Introduction

  • Wind gusts are important natural hazards, but small-scale processes and local occurrence limit predictability even for convection-permitting NWP models.Accurate, reliable forecasts support effective warnings and protection of life and property.Source: §1 · p. 1
  • Ensemble forecasts retain systematic errors, so statistical postprocessing is used to improve probabilistic forecast accuracy and reliability.The paper describes EMOS, BMA, member-by-member postprocessing, and isotonic distributional regression among the established approaches.Source: §1 · p. 2
  • Recent ensemble postprocessing has focused more on temperature, precipitation, and mean wind speed than on gusts, motivating a systematic comparison tailored to gust forecasts.The study compares statistical and machine-learning methods, including approaches that incorporate predictors beyond the target-variable ensemble.Source: §1 · p. 2
  • The case study uses operational convection-permitting German weather-service forecasts from 2010–2016 and observations from 175 surface stations.The authors describe this as the first systematic comparison of a wide variety of postprocessing approaches for wind gusts, to their knowledge.Source: §1 · p. 2

2. Data and notation

The study evaluates COSMO-DE-EPS forecasts against hourly observations at German stations, using separate training, validation, and test periods. It defines the forecast and predictor notation used to describe the data and methods.

2.1. Forecast and observation data

  • The dataset contains 20-member COSMO-DE forecasts initialized at 00 UTC for lead times from 0 to 21 hours, spanning December 2010 through 2016.Forecasts were run at 2.8 km horizontal grid spacing; verification uses hourly observations at 175 German SYNOP stations and nearest-grid-point forecasts.Source: §2.1 · p. 3

2.2. Training, testing and validation

  • The models are trained on 2010–2015, with 2010–2014 used for fitting and 2015 for validation before final fitting on the full training period.The independent test set is 2016.Source: §2.2 · p. 3

2.3. Notation

  • The notation uses y for observed gust speed, x for the ensemble vector, and x ∈ Rp for the vector of predictors or features.For meteorological variables, ensemble means are typically used as predictors; the available inputs include 61 meteorological variables and temporal and spatial predictors.Source: §2.3 · pp. 3–4, §2.1 · p. 3, Table 1 · p. 4

3. Postprocessing methods

This section compares statistical, machine-learning, and neural-network approaches to probabilistic wind-gust postprocessing, including a locally adaptive joint neural-network framework. It describes method-specific model structures, predictors, and training, as well as HEN’s binned forecasts, neural-network ensemble combination, practical trade-offs, and predictor availability.

3.1.1. Ensemble Model Output Statistics (EMOS)

  • EMOS links ensemble forecasts to parametric distribution parameters and is implemented with a zero-truncated logistic distribution fitted by CRPS minimization.Its coefficients are estimated separately by station using seasonal training sets spanning the previous, current, and next month.Source: §3.1.1 · pp. 5–6

3.1.2. Member-By-Member Postprocessing (MBM)

  • Sub-ensemble-aware MBM substantially improves performance and mostly removes residual sub-ensemble structure, but increases computation by about 17-fold.MBM also preserves ensemble rank correlations by applying the same linear transformation to each member.Source: §3.1.2 · p. 7

3.1.3. Isotonic Distributional Regression (IDR)

  • IDR provides nonparametric probabilistic forecasts without distributional assumptions or parameter tuning, using isotonic relationships between predictors and the target.The authors describe this as the first IDR application to postprocessing beyond precipitation accumulation (Henzi et al., 2019b).Source: §3.1.3 · p. 7

3.2.1. Gradient-Boosting Extension of EMOS (EMOS-GB)

  • EMOS-GB uses gradient boosting to select predictors and update distributional-regression coefficients iteratively, enabling automated use of additional meteorological variables.The implementation uses a truncated logistic distribution and local station models.Source: §3.2.1 · p. 8

3.2.2. Quantile Regression Forest (QRF)

  • QRF produces probabilistic forecasts by estimating quantiles from analog observations in random-forest partitions of the predictor space.The implementation uses additional variables’ ensemble means, but not standard deviations, and local station models.Source: §3.2.2 · p. 9

3.3.1. A framework for neural network-based postprocessing

  • The proposed neural-network framework uses station embeddings in a joint model across stations and offers three variants with different probabilistic forecast outputs.The common architecture learns nonlinear predictor relationships, with a separate model fitted for each lead time.Source: §3 · p. 5, §3.3.1 · pp. 9–10

3.3.2. Distributional Regression Network (DRN)

  • DRN replaces prespecified links with a neural network that learns nonlinear mappings from predictors to distribution parameters, trained by optimizing CRPS.For wind gusts, the truncated logistic output uses a softplus activation to enforce a nonnegative location parameter and stabilize training.Source: §3.3.2 · p. 11

3.3.3. Bernstein Quantile Network (BQN)

  • BQN represents the forecast quantile function with Bernstein polynomials; its neural network predicts nonnegative increments that ensure increasing quantiles.The authors use degree 12, reporting better validation calibration and predictive performance than degree 8, and train using quantile loss averaged over 99 quantiles.Source: §3.3.3 · pp. 11–12

3.3.4. Histogram Estimation Network (HEN)

  • HEN turns forecasting into probabilistic classification over bins, yielding a piecewise-uniform forecast; here, fixed data-driven 20-bin edges let the network estimate probabilities with categorical cross-entropy.The simple forecast structure also permits analytical solutions for the CRPS and logarithmic score used in parameter estimation.Source: §3.3.4 · pp. 12–13

3.3.5. Combining predictive distributions

  • For NN ensembles, DRN parameters and BQN coefficients are averaged, whereas HEN forecasts use Vincentization to avoid underconfidence and produce smoother, finer-bin distributions.Vincentization averages the HEN quantile functions; the resulting piecewise-linear forecast uses the union of the individual quantile-function edges.Source: §3.3.5 · pp. 13–14

3.4. Summary and comparison of postprocessing methods

  • No approach is universally best in practical use; the authors subjectively compare flexibility, forecast quality, complexity, and interpretability, and summarize predictor availability across methods.Figure 3 overviews method characteristics, while Table 2 summarizes the predictors available to each approach.Source: §3.4 · p. 14

4. Evaluation

On the 2016 test period, raw COSMO-DE-EPS forecasts are biased and underdispersed, while postprocessing improves predictive performance; neural methods lead overall and at most stations.

4.1. Predictive performance of the COSMO-DE-EPS and a climatological baseline

  • The EPS is biased and strongly underdispersed, while EPC is well calibrated but slightly positively biased; overall, the ensemble outperforms the climatological benchmark.EPC uses past observations from the same time of year, with a 15-year database.Source: §4.1 · pp. 14, 17, §4.2 · p. 17

4.2. Comparison of the postprocessing methods

  • Postprocessing substantially improves calibration, but IDR remains underdispersed; parametric forecasts have overly heavy lower tails, while semi- and non-parametric forecasts are slightly left-skewed.QRF forecasts also show minor overdispersion.Source: §4.2 · p. 17
  • All postprocessing approaches substantially improve EPS performance; basic methods improve CRPS by around 26–29%, while DRN and BQN perform best, with sharper distributions driving much of the gain.BQN also has the best coverage and sharpest prediction intervals.Source: §4.2 · pp. 17–18, Table 3 · p. 18
  • Training time varies substantially: EMOS takes 19 minutes, whereas MBM requires over 35 days, though software and parallelization differences complicate direct comparisons.EMOS-GB takes about twice as long as other advanced methods, associated with using almost twice as many predictors.Source: §4.2 · p. 18

4.3. Lead time-specific results

  • Postprocessing rankings remain relatively stable across lead times, but around the evening transition raw EPS bias and coverage worsen while postprocessed CRPS improves again.Postprocessing corrects spin-up biases at 0–1 h; prediction intervals widen with lead time and become smaller at night.Source: §4.1 · p. 16, §4.3 · pp. 18, 20
  • For thresholds above 25 m/s, the EPS loses skill relative to climatology, whereas all postprocessing methods retain positive skill across the considered thresholds.Performance falls above 30 m/s, particularly for EMOS-GB and QRF.Source: §4.3 · p. 20

4.4. Station-specific results and statistical significance

  • Network methods perform best at 162 of 175 stations; DRN and BQN significantly outperform basic methods in over 97% and machine-learning methods in over 80% of station–lead comparisons.QRF significantly improves on neural methods in around 5% of cases; locally estimated methods tend to have lower significance where they perform best.Source: §4.4 · pp. 20–21

5. Feature importance

Feature importance varies across station-specific EMOS-GB models, while wind-related ensemble predictors dominate QRF and neural-network rankings. Neural networks also rank spatial and time-varying meteorological information as relevant, with patterns linked to the diurnal cycle and evening transition.

5.1. EMOS-GB and QRF

  • For EMOS-GB, ensemble mean wind gust is generally the leading location predictor, while ensemble standard deviation leads for scale; other meteorological predictors matter at selected station–lead combinations.Feature importance is assessed separately for location and scale using absolute coefficient size, and rankings vary substantially across stations, especially during spin-up.Source: §5.1 · p. 21
  • QRF’s ten leading predictors relate directly to ensemble wind-speed characteristics, while short-wave radiation and other variables become relevant at longer lead times near sunset.The authors connect short-wave radiation importance to sensitivity of wind around sunset to surface-heating maintenance of turbulence; correlated wind predictors may substitute for one another in tree splits.Source: §5.1 · pp. 21, 23
  • Figure 8 identifies the station-wise CRPS-best method averaged over lead times, with point size encoding how often cross-group CRPS differences are statistically significant.The pairwise test matrix reports the proportion of lead-time–station cases where equal performance is rejected after multiple-testing adjustment.Source: Figure 8 · p. 22, §5.1 · p. 22

5.2. Neural network-based methods

  • Across the three neural-network methods, wind-gust ensemble forecasts are the most important inputs, followed by spatial features and meteorological variables linked to the diurnal cycle.The neural-network importance measure is the relative CRPS decrease when an input is permuted; BQN rankings also place the ensemble minimum and maximum among its most important members.Source: §5.2 · p. 23
  • Neural-network rankings place station embedding, altitude, and station-wise bias among the leading spatial inputs, with day of year, surface short-wave radiation, and temperature also relevant at different times.Spatial-feature importance has a diurnal trend resembling the ensemble’s mean bias; ground-level temperature is important for early-morning forecasts at the first few lead times.Source: §5.2 · p. 23

6. Discussion and conclusion

All considered methods calibrate forecasts and correct raw-ensemble systematic errors, while additional meteorological predictors improve skill; locally adaptive neural networks perform best among the benchmarks. The discussion highlights limits in preserving dependencies and seamlessness, alongside avenues for extending the framework.

6. Discussion and conclusion

  • Locally adaptive neural networks provide the best forecasts and significantly outperform benchmark methods, while additional meteorological predictors improve skill and all methods calibrate forecasts and correct raw-ensemble systematic errors.The NN framework offers different probabilistic output types and learns physically consistent relations, including those associated with the diurnal cycle.Source: §6 · pp. 23, 25
  • The feature-importance figures cover selected predictors across lead times and NN variants, but their captions alone do not state comparative outcomes.Figure 9 shows QRF median station-wise importance with bootstrapped 95% confidence intervals; Figure 10 concerns relative importance, and Figure 10’s caption identifies DRN, BQN, and HEN variants.Source: Figure 9 · p. 24, Figure 10 · p. 24, §6 · p. 24
  • NN-based methods learn physically consistent relations, and the proposed modular framework supports parametric distributions, Bernstein-polynomial quantile functions, or histogram-based predictive densities.The feature-importance analysis illustrates these learned relations.Source: §6 · p. 25
  • Except for MBM, the considered methods do not preserve spatial, temporal, or multivariable dependencies, while reconstructing correlations requires additional steps and complexity.Combining MBM with neural networks is proposed as future work to incorporate predictors while preserving physical characteristics.Source: §6 · p. 25
  • Separate models for each lead time, and for each station in several methods, mean the approaches are not seamless in space and time and may produce physically inconsistent forecast-trajectory jumps.A joint NN model across lead times is suggested as a possible alternative.Source: §6 · p. 25
  • Future work could incorporate spatial information with convolutional networks, temporal information with recurrent networks, and physical information or weather regimes into postprocessing.The authors also call for testing whether specific meteorological conditions affect the more complex methods and for developing dynamical feature-based approaches.Source: §6 · p. 25

A. Forecast evaluation

Forecast evaluation examines calibration and sharpness, uses proper scores for comparison and estimation, and applies statistical tests and permutation importance to assess predictive differences and predictor relevance.

A. Forecast evaluation

  • Forecasts are assessed by the principle of maximizing sharpness subject to calibration, using probabilistic forecasts, observations, and observed wind-gust realizations as the evaluation objects.Calibration concerns statistical consistency with observations; sharpness concerns forecast concentration.Source: Calibration and sharpness · p. 31, A. Forecast evaluation · p. 31

Calibration and sharpness

  • PIT or uPIT histograms, ensemble ranks, and prediction-interval coverage assess calibration, while shorter prediction intervals indicate sharper forecasts.U-shaped PIT histograms indicate underdispersion and hump-shaped histograms indicate overdispersion.Source: Calibration and sharpness · p. 31

Proper scoring rules

  • Proper scoring rules assign penalties to forecast-observation pairs and enable quantitative comparisons; CRPS is in the observation’s units and reduces to absolute error for deterministic forecasts.The log-score is another strictly proper scoring rule; quantile loss and squared error evaluate quantile and mean forecasts, respectively.Source: Proper scoring rules · pp. 31–32
  • Quantile loss evaluates quantile forecasts, squared error evaluates mean forecasts, and forecast error with the median checks distributional bias.These scores address forecasts reduced to a statistical functional such as a quantile, mean, or median.Source: Proper scoring rules · p. 32

Optimum score estimation

  • Optimum score estimation selects parametric forecast parameters by minimizing the mean strictly proper score over a training set, and minimizing log-score is equivalent to maximum likelihood estimation.Source: Optimum score estimation · p. 32

Statistical tests of equal predictive performance

  • Diebold–Mariano tests compare two methods’ predictive performance, with CRPS-based significance tests applied separately by station and lead time and Benjamini–Hochberg correction controlling false discovery rate at 0.05.The correction is applied across method pairs for Table 4 and within locations across lead times for Figure 8.Source: Statistical tests of equal predictive… · pp. 32–33

Permutation importance

  • Permutation importance measures how shuffling a predictor changes test-set CRPS; larger increases indicate greater predictor importance, and multi-pass importance shuffles a feature set together.Relative permutation importance scales the increase by the original-data CRPS.Source: Permutation importance · p. 33

B. Implementation details

Implementation details specify publicly available R code and a quantile grid used when evaluation relies on quantiles.

B. Implementation details

  • The implementation’s R code is publicly available, and quantile-based evaluations use 125 equidistant quantiles per test sample.The grid includes the median and prediction-interval quantiles corresponding to a 20-member ensemble’s approximately 90.48% coverage.Source: B. Implementation details · p. 34

Basic methods

The basic methods use standard R optimization and package implementations, with a tested MBM simplification performing worse; EMOS-GB and QRF are implemented using dedicated packages.

Basic methods

  • EMOS and MBM use gradient-based L-BFGS-B optimization, with Nelder–Mead as a second run after failed or non-converged optimization; IDR uses the isodistrreg package.For EMOS, parameterizing b as exp(˜b) improved estimation stability; tested MBM variants omitting d performed worse and retained systematic histogram deviations.Source: Basic methods · p. 34

EMOS-GB and QRF

  • EMOS-GB and QRF use the crch and ranger packages, respectively, and QRF predictor importance is obtained from ranger on the training set.Source: EMOS-GB and QRF · p. 34

NN-based methods (DRN, BQN, HEN)

The neural-network methods use normalized predictors, tuned network configurations, and model-specific training losses; HEN also uses iteratively merged bins. Separate supplementary figures show MBM forecast verification histograms.

NN-based methods (DRN, BQN, HEN)

  • Predictors are normalized using training-set means and standard deviations, and hyperparameters are tuned before comparing ensembles of ten network models.The validation period is excluded when computing normalization statistics.Source: NN-based methods (DRN, BQN, HEN) · p. 34
  • HEN is trained with categorical cross-entropy, while DRN and BQN use manually implemented losses.DRN uses the CRPS of a truncated logistic distribution; BQN uses quantile loss at specified levels based on Bernstein polynomials.Source: NN-based methods (DRN, BQN, HEN) · p. 34
  • HEN bins are generated iteratively by merging the least-populated bin with the smaller neighboring bin, subject to length limits.The first bin is at most 2 m/s, the last at most 7 m/s, and other bins at most 5 m/s.Source: NN-based methods (DRN, BQN, HEN) · p. 34
  • Tables 5 and 6 present final method hyperparameters and the configurations of the individual neural networks, respectively.Source: Table 5 · p. 35, Table 6 · p. 35

S1.1. Additional illustrations for MBM

  • MBM verification is illustrated with rank histograms for 0-hour and longer lead times and uPIT histograms for lead times from 0 to 3 hours.The figures report empirical interval coverage against nominal coverage of about 90.48%, corresponding to a 20-member ensemble.Source: Figure S1 · p. 36, Figure S2 · p. 36

S1.2. Climatological differences of the training and test dataset

Figure S3 compares observed wind-gust means and standard deviations by hour of day for 2001–2015 and 2016.

S1.2. Climatological differences of the training and test dataset

  • Figure S3 plots the hourly mean and standard deviation of observed wind gusts, comparing 2001–2015 with 2016.Source: Figure S3 · p. 37

S1.3.1. EMOS-GB

The supplementary material visualizes parameter coefficients and predictor importance across forecast lead times for EMOS-GB, QRF, and neural-network models, and describes the CRPS for piecewise uniform distributions.

S1.3.1. EMOS-GB

  • EMOS-GB coefficient plots show station-wise absolute coefficient summaries across lead times, with severe outliers for some station–lead-time combinations.The solid line gives mean coefficients across stations; uncertainty bars in the relevant plots are bootstrapped 95% confidence intervals of the median.Source: S1.3.1. EMOS-GB · pp. 38–39, Figure S4 · p. 38

S1.3.2. QRF

  • QRF feature-importance plots show station-wise median importance for selected predictors as a function of forecast lead time.The error bars indicate bootstrapped 95% confidence intervals of the median.Source: Figure S7 · p. 40

S1.3.3. NN models

S2. CRPS of a piecewise (uniform) distribution

  • For a piecewise uniform distribution, the CRPS includes a penalty when the observation falls outside the range of the bins.The HEN forecast is a special case represented by a uniform distribution over one bin with point masses at its edges.Source: S2. CRPS of a piecewise (uniform)… · p. 43

To show:

The proof treats the piecewise distribution through interval-supported component CDFs and clipped observations, showing how CRPS contributions behave inside and outside their supports.

To show:

  • Each component CDF is supported on one bin interval, and the observation is clipped to that interval before evaluating its CRPS contribution.Source: To show · p. 43

Proof:

  • Within a bin, clipping preserves the indicator in the CRPS integrand; outside the support, the contribution reduces to distance from the support boundary.The proof treats observations below and above the support separately.Source: Proof · pp. 43–44
Loading 2106.09512v1…