Source-linked AI summary
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell
TL;DR
Neural networks often produce overconfident predictions, making reliable predictive uncertainty estimation important but difficult. This paper proposes deep ensembles trained with probabilistic objectives and finds strong, well-calibrated uncertainty estimates across classification and regression benchmarks.
Problem
Neural networks are often overconfident, while reliable predictive uncertainty and calibration remain difficult to quantify.
Method
The method trains probabilistic neural networks with proper scoring rules, ensembles their predictions, and optionally applies adversarial training for smoother uncertainty estimates.
Results
The method significantly outperforms or matches MC-dropout and is competitive with existing methods on predictive negative log-likelihood across regression datasets.
Takeaways & Limitations
Deep ensembles provide a simple, scalable non-Bayesian baseline for predictive uncertainty estimation that is suitable for distributed computation.
Takeaways & Limitations
Ensembles require M times as many parameters as a single network, creating memory costs for constrained applications.
Abstract
from arXiv · showhide
Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.
1 Introduction
Deep neural networks achieve strong performance but tend to produce overconfident predictions, making predictive uncertainty difficult to quantify and evaluate. The paper proposes deep ensembles as a simple, scalable approach and investigates proper-scoring-rule training, ensembles, and adversarial training for uncertainty estimation.
- Motivation: Deep neural networks achieve state-of-the-art performance across many machine-learning tasks but are poor at quantifying predictive uncertainty and tend to be overconfident.
- Evaluation: Predictive uncertainty is difficult to evaluate because ground-truth uncertainty estimates are usually unavailable, motivating calibration and proper scoring rules.Calibration measures the discrepancy between subjective forecasts and empirical long-run frequencies.
- Related work: Bayesian neural networks quantify uncertainty through posterior distributions over parameters, but exact Bayesian inference is computationally intractable for neural networks.The paper situates its approach among approximate Bayesian methods.
- Contributions: The paper proposes a simple, scalable uncertainty-estimation method based on probabilistic neural networks trained with a proper scoring rule, while investigating ensembles and adversarial training.These two pipeline modifications are presented as mechanisms that can produce smooth predictive distributions.
- Novelty: The work investigates deep ensembles and adversarial training for predictive uncertainty estimation and compares them with approximate Bayesian methods.The authors describe this as the first study, to their knowledge, to examine these techniques for uncertainty estimation in this way.
2 Deep Ensembles: A Simple Recipe For Predictive Uncertainty Estimation
Deep Ensembles estimate predictive uncertainty by training independently initialized neural networks with proper scoring rules, adversarial smoothing, and uniform mixture averaging. The method uses task-appropriate predictive distributions, including mean–variance Gaussian outputs for regression, and is designed for parallel training.
- Core recipe: The proposed recipe combines proper scoring rules, adversarial training, and an ensemble of M neural networks.The recommended default values are M = 5 and ϵ = 1% of each input dimension’s range.
- Proper scoring rules: Proper scoring rules encourage calibrated predictive distributions, with softmax cross entropy and the Brier score providing classification examples.Properness means the true distribution receives the highest expected score, with equality only when the predictive and true distributions match.
- Regression: For regression, each network predicts a mean µ(x) and positive variance σ^2(x), and training minimizes the negative log-likelihood of a heteroscedastic Gaussian.Variance positivity is enforced with a softplus output and a minimum variance of 10^-6 for numerical stability.
- Adversarial smoothing: Adversarial training smooths predictive distributions by increasing target likelihood within an ϵ-neighborhood of observed training examples.The perturbation is chosen along a direction likely to increase the loss, making it a computationally efficient smoothing procedure.
- Ensemble training: Ensemble members use the entire training dataset, random parameter initialization, and random data shuffling, then train independently in parallel.The ensemble is treated as a uniformly weighted mixture model when combining predictions.
3 Experimental results
Experiments across regression, classification, and ImageNet show that deep ensembles improve predictive uncertainty by learning variance with NLL, combining networks, and expressing greater uncertainty on unfamiliar examples. Ensemble performance generally improves with ensemble size, while adversarial-training benefits are strongest for small ensembles.
- Regression: Learning variance with NLL produces better predictive uncertainty than empirical prediction variance from networks trained with MSE.On the toy regression task, the comparison evaluates empirical variance, NLL training, adversarial training, and ensemble combination.
- Regression: The method outperforms or matches existing approaches on NLL, although it can be slightly worse on RMSE because it optimizes predictive uncertainty rather than MSE.Additional results show benefits from both ensemble combination and learning variance.
- Classification: Ensembles significantly outperform MC-dropout across all three classification metrics, while adversarial training helps mainly for small ensembles.Adversarial training improves MNIST for every ensemble size and SVHN when M = 1, but its effect decreases as M grows.
- ImageNet: On ImageNet, increasing ensemble size significantly improves both accuracy and predictive-uncertainty quality.The evaluation uses single-crop Inception results and isolates the effect of ensembles because of computational constraints.
- Dataset shift: Across unseen-class and out-of-distribution tests, ensembles assign higher uncertainty to unfamiliar examples, with uncertainty improving as ensemble size increases.MC-dropout produces over-confident predictions on unseen examples, whereas ensemble disagreement is higher for visually unfamiliar inputs.
4 Discussion
The method is presented as a simple, scalable non-Bayesian approach that provides a strong baseline for predictive uncertainty quantification by combining target ambiguity with model uncertainty. Future work includes increasing ensemble diversity and optimizing weights, while addressing the ensemble’s larger memory footprint.
- Method and uncertainty sources: The proposed method provides a strong baseline for predictive uncertainty quantification while remaining simple and scalable.It is described as a non-Bayesian solution.
- Method and uncertainty sources: Probabilistic training captures ambiguity in targets y for a given x, while ensembles capture model uncertainty by averaging predictions across models.The two uncertainty sources arise from proper-scoring-rule training and ensemble averaging.
- Future work and limitations: Independent-network training is trivially parallelizable, supporting the method’s scalability.The discussion identifies parallelized independent training as the current focus.
- Future work and limitations: Explicitly decorrelating networks’ predictions could promote ensemble diversity and improve performance further.The passage gives prediction decorrelation as an avenue for future work.
- Future work and limitations: Optimizing ensemble weights through stacking or adaptive mixture-of-experts methods could further improve performance.Both stacking and adaptive mixture of experts are named as possible approaches.
- Future work and limitations: An ensemble uses M times more parameters than a single network, creating a memory constraint for some applications.The discussion flags memory-constrained applications as a limitation.
Supplementary material … A.1 Additional results on regression benchmarks
Table 2 reports additional regression-benchmark results using RMSE and NLL, comparing ensembles of MSE-trained networks with single-network maximum likelihood.
- Supplementary material: Table 2 reports RMSE for the top table and NLL for the bottom table on additional regression benchmark datasets.The passage identifies these as supplementary regression results.
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles: Ensemble-M (MSE) is an ensemble of M networks trained to minimize mean squared error.Its predicted variance is the empirical variance of the individual networks’ predictions.
- A Additional results on regression: ML-1 uses maximum likelihood with a single network trained to predict the mean and variance.The passage refers to the procedure described in Section 2.2.1.
- A.1 Additional results on regression benchmarks: The comparison includes Ensemble-5 (MSE) as an ensemble baseline against ML-1.The passage explicitly names Ensemble-5 (MSE) in the reported significance comparison.
- A.1 Additional results on regression benchmarks: ML-1 is significantly better than Ensemble-5 (MSE) in the reported comparison.The supplied passage states this significance claim without providing the associated metric values.
- A.1 Additional results on regression benchmarks: The table distinguishes ensemble methods by training objective and uncertainty calculation from ML-1’s single-network mean-and-variance prediction.This distinction follows the method definitions provided in the table caption.
A.2 Effect of training using MSE vs training using NLL · B Additional results on classification
On the Year Prediction MSD dataset, training ensembles with MSE and using empirical prediction variance is generally poorly calibrated because MSE does not capture predictive uncertainty. Predicted variance is significantly better calibrated than empirical variance in the reported calibration results.
- A.2 Effect of training using MSE vs training using NLL: MSE-trained ensembles using empirical prediction variance generally do not produce well-calibrated predictive probabilities.The paper attributes this limitation to MSE not being a scoring rule that captures predictive uncertainty.
- A.2 Effect of training using MSE vs training using NLL: On the Year Prediction MSD dataset, predicted variance is significantly better calibrated than empirical variance.Figure 7 compares expected and observed fractions, with the dashed blue line representing ideal calibration.
B.1 Additional results on MNIST
On MNIST, ensembles outperform MC-dropout across alternative architectures, while adversarial training further improves performance. These results are qualitatively similar to those obtained with the original architecture.
- B.1 Additional results on MNIST: Ensembles outperform MC-dropout on MNIST with both a two-hidden-layer MLP and a convolutional network.The alternative setups use different architectures from the original three-layer MLP.
- B.1 Additional results on MNIST: Adversarial training improves MNIST performance for both alternative architectures.This improvement is observed with the two-hidden-layer MLP and the convolutional network.
- B.1 Additional results on MNIST: The alternative-architecture MNIST results are qualitatively similar to those from the original setup.Figures 8 and 9 change the architecture while retaining the setup used for Figure 2(a).
B.2 Qualitative evaluation of uncertainty
The qualitative evaluation measures predictive uncertainty through maximum softmax confidence and ensemble disagreement measured by Jensen-Shannon divergence. On known and unknown classes, both measures capture meaningful ambiguity.
- Measures: Uncertainty is evaluated using predicted-label confidence, defined as the maximum softmax output, and ensemble-member disagreement measured by Jensen-Shannon divergence.These measures are applied to assess uncertainty qualitatively.
- Known classes: On MNIST known classes, examples are organized by high versus low disagreement and by highest versus least confidence.Figures 10 and 11 present these qualitative comparisons.
- Unknown classes: On unseen NotMNIST classes, examples are likewise organized by highest versus least confidence and by ensemble agreement.Figure 12 evaluates a deep ensemble trained on MNIST and tested on NotMNIST.
- Findings: Both confidence and disagreement capture meaningful ambiguity on known and unknown classes.The paper reports this observation across the qualitative results in Figures 10–12.