A review of predictive uncertainty estimation with machine learning

Hristos Tyralis; Georgia Papacharalampous

Source-linked AI summary

A review of predictive uncertainty estimation with machine learning

Hristos Tyralis, Georgia Papacharalampous

arXiv:2209.08307v2Sep 17, 2022stat.MLcs.LGmath.ST

TL;DR

Probabilistic machine-learning predictions and forecasts are becoming more common, but their concepts and methods lack a holistic structure. This review synthesizes predictive uncertainty methods and assessment metrics across statistical and machine-learning approaches, then classifies the field and discusses emerging research challenges.

Problem
As probabilistic machine-learning prediction and forecasting become more common, their concepts and methods remain insufficiently formalized and structured under a holistic view.
Source: §1 · p. 3
Method
The paper reviews predictive uncertainty estimation and assessment metrics, tracing approaches from early statistical models to recent machine-learning algorithms.
Source: §1 · p. 3
Results
The review spans early statistical approaches through flexible machine-learning algorithms and concludes by classifying the material and discussing emerging research challenges.
Source: §1 · p. 3
Takeaways & Limitations
The review provides an overview of the field’s main concepts, methodologies, and research.
Source: §1 · p. 3
A stated drawback is that the log score is restricted to predictive densities.
Source: §3.4 · p. 16

Abstract

from arXiv · show

Predictions and forecasts of machine learning models should take the form of probability distributions, aiming to increase the quantity of information communicated to end users. Although applications of probabilistic prediction and forecasting with machine learning models in academia and industry are becoming more frequent, related concepts and methods have not been formalized and structured under a holistic view of the entire field. Here, we review the topic of predictive uncertainty estimation with machine learning algorithms, as well as the related metrics (consistent scoring functions and proper scoring rules) for assessing probabilistic predictions. The review covers a time period spanning from the introduction of early statistical (linear regression and time series models, based on Bayesian statistics or quantile regression) to recent machine learning algorithms (including generalized additive models for location, scale and shape, random forests, boosting and deep learning algorithms) that are more flexible by nature. The review of the progress in the field, expedites our understanding on how to develop new algorithms tailored to users' needs, since the latest advancements are based on some fundamental concepts applied to more complex algorithms. We conclude by classifying the material and discussing challenges that are becoming a hot topic of research.

1. Introduction

This review organizes predictive uncertainty estimation with machine learning around probabilistic predictions, assessment metrics, and statistical and machine-learning methods. It synthesizes these concepts to inform algorithms tailored to users’ needs and surveys specialized applications and research challenges.

1. Introduction

Probability distributions communicate more information than point predictions and characterize predictive uncertainty (Dawid 1984, Gneiting and Raftery 2007).The distribution of predictions is termed the predictive distribution, and estimating it is predictive uncertainty estimation.Source: §1 · p. 2
Early predictive distributions were estimated with Bayesian models by assigning parameter priors, updating them with data, and integrating over the posterior (Roberts 1965, Barbieri 2015).The review notes that subsequent progress in time-series forecasting also advanced predictive uncertainty estimation, mostly in Bayesian settings (Chatfield 1996).Source: §1 · p. 2
New loss functions followed the theory of consistent scoring functions and proper scoring rules; minimizing a loss in regression can estimate quantiles of the predictive distribution.Source: §1 · pp. 2–3
Prediction includes forecasting, which conditions future predictions on past events and temporal dependence; IID machine-learning algorithms can be applied to forecasting and often perform well without exploiting that dependence.The review distinguishes probabilistic prediction from forecasting while noting that ideas can transfer between the fields.Source: §1 · p. 3
The review surveys assessment metrics and statistical and machine-learning models, synthesizing their features to inform probabilistic prediction algorithms tailored to users’ needs.It also examines combinations of techniques, time-series forecasting, spatial prediction, extremes, and measurement errors, and discusses challenges in probabilistic prediction.Source: §1 · p. 3

2. Definitions and history

The section defines predictive distributions for future responses and motivates Bayesian treatment of parameter uncertainty, then positions the review as an integrated account of probabilistic prediction methods and assessment.

2.1 Bayesian statistical modelling

A predictive distribution describes future responses given observed data and future predictors; this review examines its estimation with machine learning algorithms (Gelman et al. 2013, Barbieri 2015).The prediction task is to estimate p(z|y, x, x̃) for future response z.Source: §2.1 · p. 5
Point estimates of unknown parameters ignore estimation uncertainty, motivating Bayesian modelling that treats parameters as random and updates their distribution from data.The Bayesian regression model combines a likelihood with a prior to obtain a posterior parameter distribution.Source: §2.1 · p. 5
Independence underlies the predictive derivation, but temporal dependence in time-series forecasting makes that assumption unsuitable and can worsen predictive performance.The review says the time-series problem is reformulated in Section 4.3.Source: §2.1 · p. 6

2.2 Summary of review papers on probabilistic forecasting and prediction

Prior reviews leave the field fragmented: none covers all probabilistic-prediction subtopics, and algorithm-focused and scoring-focused studies often omit the other.Some reviews address narrow algorithm classes, while papers on scoring do not address algorithms.Source: §2.2 · p. 6
This review aims to provide a complete view of the topics and their interplay, including loss functions, which recent deep-learning reviews omit.The supplied passage identifies Abdar et al. (2021) and Zhou et al. (2022) as examples of reviews missing loss functions.Source: §2.2 · pp. 7–8

3. Loss functions for assessing probabilistic predictions · 3.1 Assessment of quantile predictions · 3.2 Assessment of expectile predictions

The section grounds probabilistic prediction assessment in scoring-function theory and explains losses that consistently elicit quantiles and expectiles.

3. Loss functions for assessing probabilistic predictions

The review extends the theory of point-prediction loss functions to scoring functions and rules for probabilistic predictions.It presents loss-function properties as a foundation for developing machine-learning algorithms.Source: §3 · p. 8

3.1 Assessment of quantile predictions

Quantile loss assigns different weights to errors depending on sign, enabling estimation at levels other than 1/2 and ensuring that 100α% of observations lie below the prediction (Koenker and Hallock 2001); at level 1/2, it is symmetric.Source: §3.1 · pp. 9–10
Quantile loss is strictly consistent for distributions with finite first moments, so it optimizes expected score when eliciting a quantile (Murphy and Daan 1985, Gneiting 2011a; Ehm et al. 2016, Taggart 2022a).Source: §3.1 · p. 10
A quantile scoring function is consistent iff it has the generalized piecewise linear form with nondecreasing g, and strictly consistent when g is strictly increasing (Koenker and Bassett Jr 1978, Gneiting 2011a; Raiffa and Schlaifer 1961, p.196; Gneiting 2011b).Under this form, any αth quantile is an optimal point prediction.Source: §3.1 · p. 10

3.2 Assessment of expectile predictions

Expectiles are least-squares analogues of quantiles defined by asymmetric distances around a distributional center; the 1/2 expectile is the mean (Newey and Powell 1987; Daouia et al. 2018).Source: §3.2 · p. 11
An asymmetric piecewise squared-error loss is strictly consistent for expectile eτ, reducing to half squared error when τ = 1/2.Source: §3.2 · p. 11
A scoring function is consistent for the τth expectile iff it has the asymmetric form based on a convex h, and strict convexity ensures strict consistency (Gneiting 2011a).Source: §3.2 · p. 11
Expectiles are elicitable and coherent risk measures that convey loss size, whereas quantiles convey loss frequency; despite these properties, expectiles are used less, possibly because of interpretability concerns (Bellini et al. 2014, Emmer et al. 2015; Taylor 2021; Waltrup et al. 2015).Source: §3.2 · pp. 11–12

3.3 Proper scoring rules

The section defines proper scoring rules for probabilistic forecasts and surveys their decomposition and forms for distributions, quantiles, and prediction intervals.

3.3 Proper scoring rules

A proper scoring rule assesses a forecast distribution against an observation and optimizes expected score when the issued distribution is true F (Gneiting and Raftery 2007).It is strictly proper when s(F, F) = s(P, F) only if P = F (Winkler and Murphy 1968, Gneiting and Raftery 2007).Source: §3.3 · p. 12

3.3.1 Decomposition of proper scoring rules

A general decomposition separates a probabilistic forecast’s score into uncertainty, resolution (or sharpness), and reliability (Bröcker 2009).Larger resolution means a better score, and reliability corresponds to γ = π(γ).Source: §3.3.1 · p. 13

3.3.2 Continuous Ranked Probability Score (CRPS)

The widely used CRPS evaluates cumulative distribution functions and supports sample-based forecasts; for ordered ensembles, it can be expressed using quantile scores, while unordered ensembles are not recommended.The CRPS generalizes absolute error for a point-mass distribution.Source: §3.3.2 · pp. 13–14

3.3.3 Scoring rules for quantiles

For quantiles at levels α1, …, αk, a proper scoring rule sums terms using non-decreasing functions; with k = 1, it reduces to the quantile scoring function (Gneiting and Raftery 2007).The quantile score can be decomposed into reliability, resolution, and uncertainty (Bentzien and Friederichs 2014).Source: §3.3.3 · p. 14

3.3.4 Scoring rules for intervals

The central interval scoring rule adds interval width to penalties for observations below the lower bound or above the upper bound, with each penalty scaled by 2/α.The discussion focuses on prediction intervals for a single observation.Source: §3.3.4 · p. 15
The shortest interval and intervals set by an endpoint or midpoint are not elicitable unless an endpoint is given via a quantile; equal-tailed and modal intervals are elicitable (Brehmer and Gneiting 2021).Mode and modal intervals are not indirectly elicitable (Dearborn and Frongillo 2020).Source: §3.3.4 · p. 15

3.4 Some more proper scoring rules for assessing probabilistic predictions · 3.5 Some more related concepts

Proper scoring rules offer varied ways to assess probabilistic predictions, while related concepts address score comparability, user-specific requirements, and validation of predictive models. The log score is distinctive for being local, but it is restricted to predictive densities.

3.4 Some more proper scoring rules for assessing probabilistic predictions

Proper scoring rules include the log, Tsallis, Brier, Bregman, survival, Hyvärinen, composite, pseudo, diagonal, energy, and variogram-based rules.The cited reviews provide comprehensive lists of proper scoring rules.Source: §3.4 · p. 15
The log score is the only local proper scoring rule, depends only on the predictive distribution’s value at the realized event, connects to likelihood inference, and is restricted to predictive densities.Unlike the CRPS, it ignores probabilities assigned to events that could have happened but did not.Source: §3.4 · p. 16
Other proposed rules address predictive-distribution moments, simultaneous events, Anderson–Darling distance, threshold and quantile weighting, or temporal dependence.The error-spread rule is formulated with respect to predictive-distribution moments.Source: §3.4 · p. 16
Further alternatives include joint scoring rules and the asymmetric continuous probabilistic score (ACPS).Source: §3.4 · p. 16

3.5 Some more related concepts

Because most score values depend on observation scale, skill scores standardize scoring rules for comparisons across cases, but are not necessarily proper.Source: §3.5 · p. 16
Arbitrary scoring rules can be properized, and users may select or construct rules tailored to their requirements.The passages cite properization methods and examples of constructing user-tailored rules.Source: §3.5 · p. 16
Liu et al. (2020a) argue that minimizing a proper scoring rule over the input space is infeasible when a model cannot learn the true data distribution, and propose minimization over all possible true distributions as a minimax problem.Source: §3.5 · p. 16
VaR backtesting checks whether conditional coverage matches the nominal level, with misspecification termed a violation; unlike elicitability’s comparison of multiple models, backtesting validates one model.Source: §3.5 · p. 17

4. Early history and simple models

The section introduces simpler probabilistic models as foundations for understanding how more complex machine-learning models can be built. It covers Bayesian ordinary linear regression, quantile regression, time-series models, and copula models.

4. Early history and simple models

The section presents simple models to clarify how more complex, accurate probabilistic prediction models can be built.It introduces this overview before turning to machine-learning models.Source: §4 · p. 17
The models covered are Bayesian statistical ordinary linear regression, linear-in-parameters quantile regression, time-series models, and copula models.Source: §4 · p. 17
Machine-learning models build on simpler models by adding complexity, which the section associates with improved performance.Source: §4 · p. 17

4.1 Bayesian statistical models

For Bayesian statistical models, the section gives the predictive distribution for Gaussian linear regression and discusses simulation and alternative prediction approaches when explicit distributions, likelihoods, or correct data-generating specifications pose challenges.

4.1 Bayesian statistical models

Under a uniform noninformative prior on (β, logσ), Gaussian Bayesian linear regression has a multivariate Student predictive distribution with n – p degrees of freedom (Gelman et al. 2013, Chapter 14).Its location is X̃u and its squared scale matrix is s2(I + X̃VuX̃ᵀ).Source: §4.1 · pp. 17–18
Beyond Gaussian models, predictive distributions may not be explicitly defined, making MCMC simulation an appropriate and perhaps the only way to proceed (Gelman et al. 2013, Chapter 14).Source: §4.1 · p. 18
Bayesian modeling commonly emphasizes in-sample performance under correct data-generating-process specification, while recent work studies out-of-sample cross-validation with loss functions (Vehtari and Ojanen 2012; Piironen and Vehtari 2017; Vehtari et al. 2017).Bayesian models are mostly applied to small datasets, and cross-validation costs may increase significantly as datasets grow.Source: §4.1 · p. 18
Approximate Bayesian Computation simulates data using prior-drawn parameters and retains cases close to observed data by a distance metric; its predictive distributions are nearly identical to, but inferior to, exact ones (Frazier et al. 2019).It is proposed to address increasing dataset computational costs and intractable likelihoods.Source: §4.1 · p. 18
Focused Bayesian prediction does not require correct specification of the data-generating process and updates predictions using a specified measure, such as a proper scoring rule (Loaiza-Maya et al. 2021).It can also improve combinations of algorithms.Source: §4.1 · p. 18

4.2 Quantile and expectile regression linear in parameters

Linear quantile and expectile regression estimate distributional functionals by optimizing their respective loss functions. The section surveys their use, extensions, and methodological challenges.

4.2.1 Quantile regression

Linear quantile regression estimates parameters by minimizing sample-averaged quantile loss, matching the fitting objective to quantile predictions.Source: §4.2.1 · p. 19
Quantile regression is useful beyond mean prediction when the predictive distribution’s parametric form is unknown, outliers occur, or variance depends on covariates.Source: §4.2.1 · p. 19
When quantile-model estimates are unstable, regularization such as ridge or lasso can improve inference and prediction, including in high-dimensional settings.Source: §4.2.1 · p. 19
Bayesian quantile models support parameter inference or difficult optimization, but use working likelihoods because parametric likelihood specification is unavailable.Source: §4.2.1 · p. 19
Multivariate quantile regression predicts quantiles of multiple outputs, increasing problem complexity; approaches include marginal directional reduction and direct formulations.Source: §4.2.1 · pp. 19–20
Independent fits at different quantile levels can produce quantile crossing, which methods address by sorting estimates or enforcing non-crossing constraints.Source: §4.2.1 · p. 20
Quantile regression also covers censored survival analysis, semiparametric models with nonlinear, spatial, or non-normal random effects, and panel models with multiple random effects.Source: §4.2.1 · p. 20
Because quantile loss is not everywhere differentiable, gradient-based optimization may not apply; approximations to the loss can enable efficient optimization.Source: §4.2.1 · pp. 20–21

4.3 Forecasting with time series models

This section surveys probabilistic time-series forecasting, in which future-value distributions are estimated conditional on past observations, covering Bayesian, bootstrap, quantile-regression, and alternative approaches. It also notes prior work on forecasting-error properties and the stated complexity of multi-step-ahead probabilistic forecasting.

4.3 Forecasting with time series models

Probabilistic forecasting estimates the distribution of a future value conditional on observations to date; one-step forecasts use h = 1, while multi-step forecasts use h > 1.Time-series models can also exploit temporal dependence absent from the usual IID regression setting.Source: §4.3 · p. 21

4.3.1 Bayesian forecasting models

Bayesian models provide a formal way to estimate predictive distributions, but complicated likelihoods can prevent explicit forms, and long series may make Bayesian methods impractical.Approximate Bayesian computation may help with little cost in predictive performance (Frazier et al. 2019).Source: §4.3.1 · pp. 22–23

4.3.2 Bootstrap-based forecasting models

Bootstrap-based probabilistic forecasts offer an alternative to parametric Bayesian models by approximating the innovation distribution through resampling fitted-model residuals.Innovations are the random component that essentially models the time-series model’s error.Source: §4.3.2 · p. 23

4.3.3 Quantile regression-based models

Quantile-loss counterparts extend time-series models to probabilistic prediction; CAViaR models quantiles autoregressively (Engle and Manganelli 2004), while CARE models use expectile loss (Kuan et al. 2009).Quantile crossing has also been addressed in AR models by Gourieroux and Jasiak (2008).Source: §4.3.3 · p. 24

4.3.4 Other forecasting models

Alternative practices include Gaussian analytical approximations of model errors or approximations of error variance, with ideas potentially applicable to general machine-learning frameworks.Examples for VAR models include computationally efficient multidimensional Gaussian-integral algorithms (Chan 1999), Gaussian approximations (De Luna 2000), and first-order Taylor approximations (Snyder et al. 2001).Source: §4.3.4 · pp. 24–25
Prior work considered asymptotic properties (Hansen 2006) and approximation of the error’s distribution (Wu 2012, Lee and Scholtes 2014).Source: §4.3.4 · p. 25
Multi-step-ahead probabilistic forecasting was described as too complex to solve (Regnier 2018).Source: §4.3.4 · p. 25

4.4 Copula-based regression models

Copula-based regression models combine marginal distributions with a copula representing dependence to obtain conditional predictive distributions; the section surveys estimation approaches and extensions to quantiles, survival analysis, high-dimensional data, Bayesian modelling, and missing data.

Copula-based regression models

Copula-based regression models the predictor marginals separately from their dependence structure to obtain the response distribution conditional on predictors and a specified copula (Noh et al. 2013).The dependence model and marginal distributions are separate components, facilitating modelling.Source: §4.4 · p. 25

Estimation procedures

Noh et al. (2013) use parametric copulas and nonparametric marginals, but copula misspecification can make results unreliable because the regression function is monotone in predictors (Dette et al. 2014).Copula-based regression is parametric, so correct specification of the copula is key.Source: §4.4 · p. 25

Direct quantile estimation

Direct quantile estimation from conditional distributions has been studied using vine copulas by Kraus and Czado (2017), with conditional-likelihood improvement by Tepegjozova et al. (2022).Größer and Okhrin (2021) also discuss the usefulness of vine copulas.Source: §4.4 · p. 26

Quantile regression

Copula-based quantile regression has been studied for IID data and time series by Noh et al. (2015), and by Rémillard et al. (2017).Source: §4.4 · p. 26

Survival analysis with copula-based regression

De Backer et al. (2017) investigated copula-based regression for estimating quantiles in survival analysis.Source: §4.4 · p. 26

Regularization in copula-based regression

For high-dimensional Gaussian copula regression, He et al. (2018) transform variable selection into multiple testing, reducing redundant variables through the stopping rule.High-dimensional Gaussian copula-based regression was also investigated by Tony Cai and Zhang (2018).Source: §4.4 · p. 26

Bayesian statistical modelling

Klein and Kneib (2016) proposed Bayesian modelling for copula-based regression, adding nonlinear predictors for model parameters.Source: §4.4 · p. 26

Various copulas

Gaussian copula regression and vine copulas are among the approaches surveyed; vine copulas address mixed data, conditional heteroscedasticity, large-scale, nonlinear, and non-Gaussian settings.Gaussian copula regression is discussed by Masarotto and Varin (2017) and Zhao et al. (2020); vine copulas by Chang and Joe (2019) and Liu and Li (2022).Source: §4.4 · p. 26

5. Machine learning algorithms

This section surveys machine-learning approaches for producing probabilistic predictions, from flexible model families to methods that derive predictive distributions or intervals from deterministic forecasts. It covers distributional and density modelling, random forests and boosting, ensemble, Bayesian, quantile, isotonic, conformal, and kernel-smoothing methods, including approaches for specialized data types.

5. Machine learning algorithms

Machine-learning models can issue probabilistic predictions by combining relevant fundamental concepts introduced earlier.Source: §5 · pp. 26–27

5.1 Gaussian process regression

Gaussian processes have jointly Gaussian finite-dimensional distributions, enabling Bayesian estimation of predictive distributions.Source: §5.1 · p. 27

5.2 Generalized additive models for location, scale and shape (GAMLSS)

GAMLSS models the full predictive distribution by linking its parameters, including those beyond mean and variance, to predictors and fitting by penalized likelihood.Source: §5.2 · p. 27

5.3 Random forests

Quantile regression forests estimate conditional quantiles nonparametrically using tree responses and an indicator function, and are consistent under certain general conditions.Source: §5.3 · pp. 28–29

5.4 Boosting algorithms

Boosting adds base learners sequentially to correct prior errors while minimizing a user-tailored loss, and can combine different types of base learners.Source: §5.4 · pp. 29–30
Boosting supports probabilistic prediction through quantile and distributional regression, including gamboostLSS, NGBoost, and mean-and-variance estimation with PGBM.Quantile boosting can use diverse base learners, while gamboostLSS models location, scale, and shape; PGBM uses estimated means and variances to sample from a specified distribution.Source: §5.4 · pp. 30–31

5.5.1 Bayesian Additive Regression Trees (BART)

BART sums decision trees under a regularization prior and permits probabilistic predictions; multiplicative-tree variants can model heteroscedasticity (Pratola et al. 2020).BART is motivated by boosting algorithms (Chipman et al. 2010).Source: §5.5.1 · p. 31

5.5.2 Transformation models

Conditional transformation models estimate predictive distributions with fewer assumptions than GAMLSS (Hothorn et al. 2014), and extensions include boosting, trees, and autoregressive models.The autoregressive variant combines properties of time-series and transformation models (Rügamer et al. 2023a).Source: §5.5.2 · p. 31

5.5.3 Conformal prediction

Conformal prediction uses a nonconformity measure to form marginal intervals with specified coverage, while conformalized quantile regression models heteroscedasticity and guarantees interval coverage.Marginal coverage is not conditioned on predictors, and comparing intervals with specified coverage probability is difficult.Source: §5.5.3 · pp. 31–32

5.5.4 Support vector regression.

Support vector regression has quantile and expectile variants, and an orthogonal loss makes it suitable for noisy data (Yu et al. 2018).A semiparametric quantile-regression variant was proposed by Shim et al. (2012), and expectile regression with support vector machines by Farooq and Steinwart (2017).Source: §5.5.4 · p. 32

5.5.5 Splines

Spline methods include quantile regression and Bayesian non-crossing quantile regression (He and Ng 1999; Kitahara et al. 2021; Xu and Reich 2021), while local polynomial methods combine quantile trees or use expectile regression.Polynomial quantile regression combined with quantile regression trees supports probabilistic prediction (Chaudhuri and Loh 2002); Adam and Gijbels (2022) propose expectile regression with local polynomial fitting.Source: §5.5.5 · p. 33, §5.5.6 · p. 33

5.5.7 Functional regression

A general framework for functional-data regression includes quantile and distributional regression; models are estimated using penalized likelihood or gradient boosting and may include scalar or functional covariates of multiple types.Source: §5.5.7 · p. 33

5.5.8 Fuzzy logic

Fuzzy time-series methods for probabilistic forecasting rely on specialized techniques for time series (Silva et al. 2016, 2020).Source: §5.5.8 · p. 33
Fuzzy-logic methods discussed here concern forecasting and are described as unrelated to methods presented in earlier sections.Source: §5.5.8 · p. 33

5.5.9 Converting deterministic to probabilistic predictions

Ideas for converting point predictions into probabilistic forecasts, including analog ensembles and dressing, can transfer to machine-learning applications.The passages note that most environmental models issue point predictions and that machine-learning applications do as well.Source: §5.5.9 · p. 33
When machine-learning forecasts make past-pattern analogs difficult to identify, dressing is preferred and estimates uncertainty by dressing past errors to current point forecasts (Roulston and Smith 2003).Analog methods assume current time-series patterns resemble past ones (Lorenz 1969), but the passages say such similarities may not be identifiable for machine-learning forecasts (Yang and van der Meer 2021).Source: §5.5.9 · p. 34

5.5.10 Conditional probability density estimation with kernel smoothing

In regression, a conditional probability distribution can be estimated by kernel smoothing the empirical joint predictor and dependent-variable distribution and marginal predictor distribution (Hyndman et al. 1996), while weighting data points by similarity.Source: §5.5.10 · p. 34

5.5.11 Other models

EMOS uses ensemble predictions as predictors in a linear-regression variant that models the conditional mean and variance of a Gaussian distribution (Gneiting et al. 2005).It resembles distributional regression and exploits properties of weather forecasts issued as ensembles.Source: §5.5.11 · p. 34
Nott (2006) models a possibly non-Gaussian response with a double exponential-family distribution, penalized-spline predictors, and Bayesian inference.Source: §5.5.11 · p. 34
Fasiolo et al. (2021) estimate conditional-response quantiles directly using Bayesian quantile loss, avoiding specification of the response distribution as in GAMLSS.Source: §5.5.11 · pp. 34–35
Isotonic distributional regression respects a spatial partial order on predictors by estimating conditional distributions with suitable loss functions, and is automatic once that order is defined (Henzi et al. 2021a, 2021b).Source: §5.5.11 · p. 35

6. Neural networks and deep learning

This section surveys probabilistic prediction with neural networks, covering Bayesian and simulation-based methods, quantile and distributional regression, GAN-based forecasting, calibration- and interval-oriented regression, distribution construction by classification, and probabilistic federated learning.

6.1 Bayesian methods

Bayesian neural networks place priors on weights and error variance, infer posterior parameter distributions, and derive predictive distributions; Gaussian-mixture outputs can model non-Gaussian responses.The mixture approach increases flexibility, while variational inference approximates the posterior to address the computational burden of Bayesian methods (Swiatkowski et al. 2019).Source: §6.1 · pp. 35–36

6.2 Monte Carlo simulation-based methods

Dropout ensembles approximate Bayesian deep Gaussian-process predictions more quickly, and their predictive uncertainty can be estimated from ensemble variance (Gal and Ghahramani 2016).Batch-normalization ensembles can also approximate Bayesian uncertainty (Teye et al. 2018); other approaches randomize network depth or architecture.Source: §6.2 · p. 36

6.3 Quantile and expectile regression

Averaging quantile-regression losses across levels shares information, mitigates quantile crossing, and produces more accurate predictions.An L1 penalty on the loss gradient is another proposed way to address crossing; neural-network variants also include expectile and Huber quantile regression.Source: §6.3 · p. 37

6.4 Distributional regression

Neural distributional regression ranges from predicting Gaussian means and variances to CRPS-trained Gaussian outputs, copula models for richer distributions, and general deep-learning frameworks.The literature also includes deep-learning transformation models and Von Mises models for directional statistics.Source: §6.4 · pp. 37–38

6.5 Other neural networks models

Other neural-network approaches adapt conformal prediction and transformation models, while generative adversarial networks extend naturally to probabilistic prediction through sample generation and discrimination.The cited conformal work includes neural networks, deep learning, and recurrent networks.Source: §6.5 · p. 38
Pacchiardi et al. (2022) proposed scoring-rule training for GANs, while conditional GANs support probabilistic regression forecasts, including time-series applications (Koochali et al., 2019, 2021).The cited applications include multivariate probabilistic time-series forecasting.Source: §6.5 · p. 38
Kuleshov et al. (2018) use Bayesian deep learning to target regression calibration, departing from maximizing predictive sharpness subject to calibration.Source: §6.5 · pp. 38–39
Thiagarajan et al. (2020) estimate means and prediction intervals with separate models, using interval estimates to improve means in bi-level optimization and reporting good uncertainty estimates.Source: §6.5 · p. 39
Li et al. (2021) turn binned dependent-variable classifications into distributions, while Thorgeirsson and Gauterin (2021) assign probabilities to local model weights for federated learning.Source: §6.5 · p. 39

7. A representative simple example

A simulation compares quantile regression and Gaussian-response GAMLSS, finding that both improve with larger samples while GAMLSS performs better under the correctly specified model. The authors note that nonparametric methods may prevail when real-data distributions are misspecified.

7. A representative simple example

In simulations, y = x1 + x2 + ε with standard-Gaussian ε and x1, x2 used 100 samples split into 50 training and 50 testing, and simulations were repeated 10,000 times.Source: §7 · pp. 39–40
Because quantile scores are not scale free, the authors use coverage probabilities to compare performance across sample sizes, while noting that coverage is not a proper scoring rule.Source: §7 · p. 40
Mean GAMLSS coverage at α = 0.05 was 0.07344 with 100 samples and 0.05211 with 1,000; both methods became more accurate with larger samples.Source: §7 · p. 40
GAMLSS outperformed quantile regression at all quantile levels and sample sizes, though the performance gap narrowed as sample size increased.Quantile regression improved faster as sample size increased but remained worse than GAMLSS in these simulations.Source: Table 2 · p. 41, §7 · p. 41
Because the dependent-variable distribution was correctly specified in the simulation, GAMLSS had an advantage as a parametric model.The authors note that the simulation’s known variable properties favored parametric models.Source: §7 · p. 41
For real data, where the dependent-variable distribution may be misspecified, nonparametric methods could instead prevail.Source: §7 · p. 41

8. Combinations of algorithms

Ensemble combinations improve on individual algorithms across several aspects, including point-prediction performance, and the same holds for algorithms issuing probabilistic predictions. The section reviews weighted averaging, Bayesian model averaging, and stacked generalization.

8. Combinations of algorithms

Algorithm combinations, or ensemble learning, improve on individual algorithms across multiple aspects, including point-prediction performance; the same applies to probabilistic prediction.Source: §8 · p. 41

8.1 Weighted averaging of algorithms

Weighted averaging can combine predictive distributions through linear pools of CDFs or densities, logarithmic pools, or distribution functionals such as quantiles and expectiles.The section identifies weighted averaging as the simplest combination approach.Source: §8.1 · pp. 41–42
The equally weighted linear pool (simple averaging) is hard to beat in practice (Lichtendahl et al. 2013); multiple attempts are more successful as data size increases.Source: §8.1 · p. 42
Mitchell (2007) proposed estimating the weights of the linear pool by minimizing a distance measured by the Kullback–Leibler information criterion (corresponding to log-score), while Hora and Kardeş (2015) proposed minimizing the quadratic scoring rule.Source: §8.1 · p. 42

8.2 Bayesian model averaging

Bayesian model averaging combines model predictions using posterior model probabilities; applications include improving underdispersive forecast ensembles and BART performance with many predictors.BART improvement is reported for settings with a large number of predictors.Source: §8.2 · pp. 43–44

8.3 Stacked generalization

Stacking estimates combination weights from out-of-sample performance using consistent scoring functions or proper scoring rules, rather than in-sample performance.Yao et al. (2018) favor stacking over BMA when the combined algorithms omit the true data-generating model; the combiner may be linear or more flexible.Source: §8.3 · p. 44

9. Special cases

This section surveys specialized probabilistic prediction and forecasting topics, including temporal, spatial, and spatio-temporal dependence, extremes, and observational errors, alongside applications across fields and large-dataset comparisons of machine-learning methods, which could be facilitated by specialized software.

9.1 Temporal models

Temporal forecasting methods include GAM-inspired models for seasonality, trends, and holidays, and hybrids combining neural networks with GARCH variance modeling or spline quantile functions (Taylor and Letham 2018; Khosravi et al. 2013; Gasthaus et al. 2020).Source: §9.1 · p. 45
Modelling multiple time series with one model can exploit additional information and improve predictive performance (Sen et al. 2019), with CNN and autoregressive distributional approaches as examples.Chen et al. (2020) use a CNN for quantile and distributional regression; Salinas et al. (2020) use an autoregressive component.Source: §9.1 · p. 46

9.2 Spatial models

Spatial prediction must address irregular measurement locations; Gaussian processes are prominent, while pure machine-learning models do not straightforwardly exploit spatial dependence.Source: §9.2 · p. 46
Banerjee et al. (2008) developed Bayesian Gaussian process modeling suited to big datasets and addressing relevant weaknesses inherent to Bayesian models.Source: §9.2 · p. 46

9.3 Spatio-temporal models

Spatio-temporal data pose new challenges requiring more complex models; approaches include Bayesian methods, Gaussian processes, expectile models, and deep learning (Wang and Gelfand 2014; Spiegel et al. 2020; Wu et al. 2021).Source: §9.3 · p. 47

9.4 Extremes

For rare-event prediction, model distribution tails or extreme quantiles, often with heavy-tailed distributions; distributional-regression estimates may be unstable, and quantile or consistent-scoring methods may be unsuitable.Source: §9.4 · pp. 47–48
For tail assessment, conditional or censored likelihood scoring targets a specified region, but expected scores cannot distinguish tail properties.Other proposals include CRPS expressions for GEV and GPD distributions and scores emphasizing tails.Source: §9.4 · pp. 48–49

9.5 Uncertainty in measurements (observational errors)

Measurement-error methods adapt quantile regression to model predictor errors and adapt proper scoring rules to errors in the dependent variable (Wei and Carroll 2009; Ferro 2017; Bessac and Naveau 2021).These adaptations address the fact that measured observations may not be the real values.Source: §9.5 · p. 49

9.6 Some representative applications

Probabilistic prediction and forecasting are established application areas, with review coverage spanning economics and finance, energy, and environmental and earth sciences.The paper introduces representative application reviews across these domains.Source: §9.6 · p. 49, §9.6.3 · p. 50

9.6.1 Economics and finance

Economics and finance reviews cover sales forecasting, financial time-series deep learning, and algorithm combinations, although most applications concern point forecasts (Duran 2008; Sezer et al. 2020; Steel 2020).Point-forecast deep-learning surveys can still inform probabilistic forecasting through straightforward extensions.Source: §9.6.1 · pp. 49–50

9.6.2 Energy research

Energy reviews address probabilistic wind and solar forecasting, electricity-price forecasting, and electric-load forecasting (Zhang et al. 2014; Yang et al. 2022; Weron 2014; Hong and Fan 2016).Solar post-processing converts point forecasts, often from numerical weather models, into probabilistic forecasts.Source: §9.6.2 · p. 50

9.6.3 Environmental and earth and planetary sciences

Environmental and earth-science reviews cover weather and hydrological post-processing, probabilistic geology methods, hydroclimatology metrics, and probabilistic random forests (Li et al. 2017; Vannitsem et al. 2021; Albarello and D’Amico 2015; Huang and Zhao 2022; Tyralis et al. 2019b).The section also notes prediction of extremes as a related topic.Source: §9.6.3 · p. 51

9.6.4 Big data comparisons

Large-dataset comparisons of multiple machine-learning methods are essential for understanding their performance and properties, but remain rare relative to point-prediction comparisons.The section points to comparisons using environmental data as examples.Source: §9.6.4 · p. 51
Comparisons cover computer science, data competitions, and spatial problems, and could be facilitated by specialized software (e.g. Ghosh et al. 2022).Source: §9.6.4 · p. 51

10. Summary and future outlook

The section organizes probabilistic prediction around foundational concepts, practical problem types, modeling choices, and scoring methods, emphasizing how prediction targets, data characteristics, and scoring choices shape assessment. It outlines future needs for methods better tailored to users and prediction tasks, including temporal and spatial prediction, extreme quantiles, observational errors, and stronger comparative evidence.

10.1 Setting a probabilistic prediction problem – technical considerations

Bayesian statistical modelling, linear regression, and scoring rules are foundational: scoring-rule training can fit parametric models without Bayesian simulations and nonparametric models without distributional misspecification.Linear regression can also be transformed into Bayesian models to predict probabilities for future dependent variables.Source: §10.1 · pp. 51–52
Probabilistic prediction methods can combine Bayesian modelling, linear regression, scoring-rule training, and machine-learning theory to build algorithms suited to specific tasks.The review illustrates this with distributional boosting, which optimizes a proper scoring rule using a machine-learning algorithm.Source: §10.1 · pp. 52, 54
The three practical problem types include independent responses, temporal dependence, and spatial dependence; most machine-learning models suit the first, while time-series models and LSTMs exploit the second.Spatial problems are usually addressed with spatial statistical models such as kriging and with Gaussian processes.Source: §10.1 · p. 52
Measurement-error and extreme-value modelling are often needed in time-series and spatial settings, whereas ordinary business-as-usual modelling occurs across all three practical problem types.The review notes that extreme-value applications include environmental and finance settings, where temporal or spatial dependence is common.Source: §10.1 · p. 53
A specified functional, such as a quantile or exceedance probability, can be predicted directly, while the full predictive distribution is more informative but harder to estimate.A quantile can be fitted with a linear model using quantile loss; distributional regression with proper scoring rules is a route to the full distribution.Source: §10.1 · pp. 53–54
For a target functional, use a consistent scoring function to fit a related model; to obtain the full distribution, use distributional regression with proper scoring rules.Combining algorithms may improve predictive performance; multiple quantiles can also provide a high-resolution distribution.Source: §10.1 · p. 54
Scoring-rule choice affects finite-sample assessments unless nested information sets, perfect estimation, and correct specification all hold (Patton 2020).These assumptions are rarely met in practice, so the scoring function should be disclosed to the modeller.Source: §10.1 · pp. 55–56

10.2 Differences between point prediction, distributional regression and Bayesian

Distributional regression and Bayesian models risk misspecification because specifying a probability distribution imposes assumptions, while flexible point prediction can struggle with limited data.More flexible distributions can model additional aspects of the dependent variable; prior distributional knowledge can help when data are limited, at the cost of flexibility.Source: §10.2 · pp. 56–57
When only one functional is needed, point prediction avoids simulating and storing a full distribution, which can be computationally prohibitive.For small datasets, Bayesian modeling may be appealing; for high or extreme quantiles, point prediction can be inefficient, while distributional regression may be preferable.Source: §10.2 · p. 57
Estimating multiple quantiles as a substitute for a full predictive distribution reduces point prediction’s speed advantage and can produce quantile crossing.Simultaneous estimation of multiple quantiles with deep learning is cited as an advance addressing these undesirable properties.Source: §10.2 · p. 57

10.3 Future outlook

Existing algorithms often use prespecified scoring rules that may not meet users’ needs, such as when exceedance magnitude matters beyond quantile estimation.The review identifies properizing new scoring functions for functionals and optimizing with proper scoring rules beyond local ones as strategies for improved probabilistic predictions.Source: §10.3 · p. 58
Temporal and spatial prediction still need improvement, particularly for predicting multiple points or quantiles simultaneously, a more challenging task than predicting multiple means.Related scoring rules such as the energy score exist but are rarely used in practice.Source: §10.3 · p. 58
Extreme-quantile prediction remains a research challenge because available scoring functions are unsatisfactory; approaches include extrapolating intermediate functionals with extreme value theory.Distributional regression is an option with previously noted disadvantages, and scoring functions using information beyond exceedance frequency are also trending.Source: §10.3 · p. 59
Modeling observational errors has progressed slowly, perhaps because suitable large datasets are lacking, and work is moving toward developing new scoring functions.Source: §10.3 · p. 59
Large-scale comparisons of probabilistic algorithms remain uncommon; forecasting series vary in magnitude, so skill scores may help compare methods but may not be proper.The authors also note that statistical tests of performance differences have seen little progress, while combining algorithms may improve predictive performance.Source: §10.3 · p. 59
Coverage and interval-width comparisons are informative but inconsistent with predicting the functional; scoring approaches for intervals at varying quantile levels are needed.Related scoring rules mostly address fixed quantile levels.Source: §10.3 · p. 59
New scoring rules may help calibrate machine-learning algorithms to tasks with distinctive properties, including intermittency, multiple modes, or heavy tails.The authors identify prediction of dependent variables with such properties as another challenge.Source: §10.3 · pp. 59–60

10.4 Final remarks

Although point predictions remain dominant and that situation is not expected to change, probabilistic applications are expected to gain share in academia and industry.The authors also anticipate that users’ decision procedures will adapt to the additional information probabilistic predictions provide.Source: §10.4 · p. 60