Source-linked AI summary
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, Dmitry Vetrov
TL;DR
Deep neural networks often produce unreliable probability estimates, motivating better in-domain uncertainty evaluation for image classification. The paper identifies metric pitfalls, proposes calibrated log-likelihood and the DEE score, and studies ensembling techniques. It finds that many sophisticated methods are equivalent to ensembles of only a few independently trained networks, while test-time augmentation is a strong baseline and can improve ensembles.
Problem
Common in-domain uncertainty metrics can be unreliable or unsuitable for comparing different models, complicating ensemble evaluation.
Method
The paper evaluates in-domain image-classification uncertainty, recommends calibrated log-likelihood, and introduces the deep ensemble equivalent (DEE) score for comparing ensembling techniques.
Results
Many popular ensembling techniques require dozens of samples yet are essentially equivalent to ensembles of only a few independently trained models; TTA is a strong baseline that can improve ensembles.
Takeaways & Limitations
Fair ensemble comparisons require temperature scaling and caution with common uncertainty metrics, while TTA is an overlooked practical option.
Takeaways & Limitations
The study focuses on in-domain image classification and does not cover other settings such as regression or out-of-domain uncertainty; TTA calibration can depend on augmentation severity.
Abstract
from arXiv · showhide
Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of only few independently trained networks in terms of test performance.
1 INTRODUCTION
The paper examines unreliable DNN probability estimates and develops a framework for evaluating in-domain uncertainty and ensembles. It proposes calibrated log-likelihood, the DEE score, and highlights test-time augmentation as a strong baseline.
- Motivation: DNN classification accuracy does not guarantee reliable predictive uncertainty estimates.Unreliable probability estimates have motivated increased attention to uncertainty estimation.
- Metric evaluation: Calibrated log-likelihood avoids most identified metric pitfalls and is generally reasonable for in-domain uncertainty estimation.
- Ensembling evaluation: The deep ensemble equivalent (DEE) score measures how many independently trained models match an ensembling technique’s performance.It provides a unified scale for comparing techniques across datasets and architectures.
- Ensembling evaluation: Most popular ensembling techniques average dozens of samples yet are essentially equivalent to ensembles of only a few independently trained models.
- Test-time augmentation: Test-time data augmentation is a strong uncertainty-estimation baseline and a simple way to improve ensembles.The study identifies it as overlooked in uncertainty estimation and ensembling.
2 SCOPE OF THE PAPER
The study is scoped to in-domain uncertainty in standard image-classification benchmarks using conventional training and clean-data augmentation. Other domains, uncertainty settings, and more elaborate training approaches remain outside scope.
- Experimental scope: The experiments use standard image-classification benchmark problems.
- Scope boundaries: Regression, language modeling, image segmentation, active learning, and reinforcement learning are outside the paper’s scope.These settings are identified as promising directions for further research.
- Uncertainty setting: The paper focuses on in-domain rather than out-of-domain uncertainty.Out-of-domain uncertainty concerns different-domain, corrupted, or adversarial inputs and is treated as a separate setting.
- Data and augmentation: Methods are trained on clean data with simple data augmentation.Methods using out-of-domain data, mixup, or adversarial training are excluded.
- Training procedure: The study uses conventional SGD and batch normalization while excluding more elaborate optimization techniques.Super-convergence and stochastic weight averaging are noted as possible alternatives.
3 PITFALLS OF IN-DOMAIN UNCERTAINTY ESTIMATION
In-domain uncertainty metrics can be temperature-sensitive, model-dependent, and unreliable for ranking methods. The section motivates calibrated log-likelihood and parameter-aware evaluation as safer comparison practices.
- Different uncertainty metrics capture different properties, and their evaluation pitfalls can significantly change method rankings.The considered metrics include Brier score, log-likelihood, calibration metrics, misclassification-detection metrics, and threshold–accuracy curves.
- 3.1 LOG-LIKELIHOOD AND BRIER SCORE: Comparing log-likelihood at suboptimal temperatures can produce arbitrary rankings, including changes in the best ensembling method.The authors therefore state that log-likelihood comparisons should be performed at the optimal temperature.
- 3.1 LOG-LIKELIHOOD AND BRIER SCORE: Calibrated log-likelihood evaluates the model and post-training calibration as one system at the optimal temperature, avoiding calibration error removable by temperature scaling.The metric can substantially alter comparisons, reducing differences between Bayesian and conventional networks; an unbiased estimate without held-out validation is also described.
- 3.1 LOG-LIKELIHOOD AND BRIER SCORE: Log-likelihood and Brier score are temperature-sensitive and highly correlated across broad image-classification settings.Log-likelihood correlates with accuracy at ρ > 0.86, rising to ρ > 0.95 after calibration; Brier score behaves similarly.
- Misclassification-detection AUCs cannot be directly compared across models because each model induces a different binary classification problem.Models may produce different confidences and misclassify different objects, so their positive and negative cases differ.
- Confidence-threshold accuracy curves are overly calibration-dependent, whereas rejection-count thresholding is less sensitive to temperature scaling.Different temperatures assign different numbers of objects to confidence levels, undermining direct comparison.
- 3.4 CALIBRATION METRICS: ECE has bias, can favor uniform predictions, focuses on maximum probability, and may yield incompatible estimates across models.These issues motivate caution when using ECE-like scores for cross-model calibration comparisons.
- 3.4 CALIBRATION METRICS: TACE remains unreliable for comparing models because its bias can differ by model and its rankings vary with threshold and bin count.The paper reports this sensitivity for VGG16BN on CIFAR-100 at the optimal temperature.
4 A STUDY OF ENSEMBLING & DEEP ENSEMBLE EQUIVALENT
The study compares diverse ensembling techniques using deep ensembles as a unified reference scale. It finds that snapshot methods can approach deep-ensemble performance, while test-time augmentation often improves ensembles, especially on ImageNet.
- Methods: The study covers stochastic computation-graph and snapshot-based ensembling techniques, including deep ensembles, SSE, FGE, SWAG, cSGLD, VI, K-FAC Laplace, dropout, and TTA.All techniques are represented as parameter distributions whose predictions are averaged during testing.
- Deep Ensemble Equivalent: DEE measures the minimum number of independently trained models needed to match an ensembling method’s performance, providing a unified comparison across datasets and architectures.The study bases DEE on calibrated log-likelihood and uses it to improve interpretability over model- and dataset-specific metric values.
- Experiments: SSE and cSGLD outperform most techniques and show near-linear DEE scaling with sample count on CIFAR datasets, although SSE saturates on ImageNet.The authors suggest that their exploration of different loss-landscape modes explains this behavior, while suboptimal ImageNet cyclic-learning-rate hyperparameters may explain SSE’s saturation.
- Experiments: SSE typically outperforms cSGLD because it uses a threefold larger training budget and collects less-correlated snapshots.SSE collects one snapshot per cycle, whereas cSGLD collects three snapshots, creating a trade-off between training budget and DEE-to-sample efficiency.
- Test-Time Data Augmentation: Test-time data augmentation consistently improves most ensembling methods, with especially clear gains on ImageNet and limited gains for powerful ensembles on CIFAR datasets.The authors attribute the dataset difference to ImageNet’s larger images supporting more diverse augmented samples.
- Test-Time Data Augmentation: TTA can improve accuracy while decreasing uncalibrated log-likelihood, because severe random crops can disrupt calibration; temperature scaling can recover performance.Less extreme crops often avoid this decalibration, and TTA frequently improves calibration rather than breaking it.
5 DISCUSSION & CONCLUSION
The paper identifies evaluation pitfalls in in-domain uncertainty estimation and finds that deep ensembles remain strong under a unified framework. It emphasizes reliable benchmarks and the importance of exploring different loss-landscape modes.
- Temperature scaling is required for fair ensemble comparisons because uncalibrated models can receive misleading log-likelihood rankings.Ensembles are generally better calibrated out of the box but are not perfectly calibrated.
- Many uncertainty metrics are unreliable or unsuitable for comparing methods, including ECE analogues, AUC-ROC, AUC-PR, and accuracy-confidence curves.
- Many popular ensembling techniques need dozens of test-time samples yet match only a handful of independently trained models.
- Deep ensembles with few members outperform methods based on stochastic computation graphs, while methods confined to one loss-landscape mode cannot compete with mode-exploring methods.
- Unreliable metrics inhibit fair comparison, motivating more reliable benchmarks across uncertainty-estimation settings.
A IS ”EFFICIENT” TRAINING OF ENSEMBLES EFFICIENT AT ALL?
Snapshot-based methods can reduce training epochs while matching deep-ensemble performance, but this efficiency shifts costs to inference, memory, and hyperparameter sensitivity. The analysis is restricted to conventional training procedures.
- 2-5× less training epochs lets SSE, cSGLD, and FGE match deep ensembles on average across CIFAR datasets.
- Snapshot-based efficiency can require up to ×25 greater inference expense because many more weight samples must be stored.
- Sub-optimal maximum learning rate, decay-cycle length, or snapshot schedule can make snapshot methods completely inefficient.
- The comparison uses conventional training and excludes super-convergence, stochastic weight averaging, compression, and distillation.
B EXPERIMENTAL DETAILS
The experiments standardize probabilistic assumptions and evaluate several ensembling methods across CIFAR and ImageNet architectures. Implementations largely reuse established procedures and public code.
- All ensembling techniques use the same softmax likelihood and fully factorized zero-mean Gaussian prior, with architecture- and dataset-specific weight-decay parameters.
- Conventional CIFAR networks use SGD, batch normalization, unified learning-rate scheduling, and horizontal-flip plus random-crop augmentation.
- The reported learning-rate schedule decreases from lrinit through a decay interval and then remains at lrinit · 0.01.
- Deep ensembles average predictions from independently initialized networks, using 128 training repetitions on CIFAR and 50 on ImageNet.
- Variational inference uses a fully factorized Gaussian weight approximation and tends to underfit deep learning models in practice.
- cSGLD adds gradient noise and captures several snapshots per cycle, whereas FGE uses pretraining, short cycles, and a piecewise-linear learning-rate schedule.
C ADDITIONAL EXPERIMENTAL RESULTS
Additional results compare uncertainty metrics, calibration, test-time augmentation, and DEE across CIFAR and ImageNet. Metric orderings are generally stable even when their scales differ.
- Log-likelihood and Brier score are not strictly equivalent but show high empirical correlation across CIFAR-10, CIFAR-100, and ImageNet.
- Plain and calibrated log-likelihood are highly correlated with accuracy, especially after calibration.
- On CIFAR-10, one calibrated network approaches dropout, VI, and K-FAC Laplace performance except for VGG.
- On CIFAR-100, calibrated deep ensembles rank first for WideResNet and VGG.
- Across DEE metrics, method ordering and overall line behavior remain consistent despite metric-dependent DEE scales.
- SSE outperforms deep ensembles on CIFAR-10 under the reported DEE comparison.