Source-linked AI summary
Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference
Max-Heinrich Laves, Sontje Ihler, Karl-Philipp Kortmann, Tobias Ortmaier
TL;DR
MC-dropout uncertainty can be miscalibrated and fail to represent model error, a concern for safety-critical predictions. The paper extends temperature scaling to dropout variational inference and defines UCE to measure uncertainty calibration. Experiments report reduced miscalibration and robust rejection of uncertain predictions.
Problem
MC-dropout variational inference can produce uncertainty that does not correspond well to model error, despite the importance of calibrated uncertainty for safety-critical decisions.
Source: §1 · p. 1Method
The paper extends temperature scaling to dropout variational inference and defines perfect uncertainty calibration and expected uncertainty calibration error (UCE).
Source: §1 · p. 2, §2 · p. 2, §2.1 · p. 3Results
45–66%: with temperature scaling, MC dropout reduces ECE, while UCE drops drastically, especially for larger networks.
Source: §3.1 · p. 4Takeaways & Limitations
For Hmax < 0.8, calibrated uncertainty yields an almost linear relationship between rejection threshold and top-1 error, enabling robust rejection without compromising model accuracy.
Source: §3.3 · p. 5, §4 · p. 5
Abstract
from arXiv · showhide
Model uncertainty obtained by variational Bayesian inference with Monte Carlo dropout is prone to miscalibration. The uncertainty does not represent the model error well. In this paper, temperature scaling is extended to dropout variational inference to calibrate model uncertainty. Expected uncertainty calibration error (UCE) is presented as a metric to measure miscalibration of uncertainty. The effectiveness of this approach is evaluated on CIFAR-10/100 for recent CNN architectures. Experimental results show, that temperature scaling considerably reduces miscalibration by means of UCE and enables robust rejection of uncertain predictions. The proposed approach can easily be derived from frequentist temperature scaling and yields well-calibrated model uncertainty. It is simple to implement and does not affect the model accuracy.
1 Introduction
The introduction motivates calibration of MC-dropout uncertainty for safety-critical decisions and presents temperature scaling as an extension for dropout variational inference. It positions the work alongside entropy-regularization and calibration methods, then states its metric, method, and evaluation contributions.
1 Introduction
- MC dropout estimates epistemic uncertainty from limited training data, but its uncertainty can be miscalibrated and fail to correspond to model error.The paper links epistemic uncertainty to uncertainty in model weights caused by limited dataset size.Source: §1 · p. 1
- For safety-critical tasks, miscalibrated model uncertainty matters because it can contribute to decisions with fatal consequences.The introduction names autonomous driving and computer-aided diagnosis as examples.Source: §1 · p. 1
- Label smoothing and confidence penalty address overconfident networks, but the cited works do not apply entropy regularization to calibration of confidence or uncertainty.Label smoothing is linked to overfitting and poor generalization, while confidence penalty penalizes low-entropy outputs [7] [8].Source: §1 · p. 1
- Temperature scaling uses a single scalar to smooth softmax outputs and has been shown to calibrate likelihoods in non-Bayesian deep networks.The introduction also mentions isotonic regression, Platt scaling, and Gumbel-Softmax or Concrete distributions [11] [12].Source: §1 · p. 2
- The paper extends temperature scaling to dropout variational inference, defines perfect uncertainty calibration and UCE, and evaluates architectures on CIFAR-10/100 against confidence penalty.The authors report improved calibration and superiority over confidence penalty.Source: §1 · p. 2
2 Methods
The methods define uncertainty calibration through error rates conditional on predicted uncertainty, introduce UCE to measure miscalibration, and adapt temperature scaling to MC-dropout predictions.
2 Methods
- MC dropout estimates model uncertainty using N stochastic test-time forward passes, while normalized softmax entropy represents uncertainty across all classes.The normalized entropy ranges from 0 to 1.Source: §2 · p. 2
- Uncertainty is perfectly calibrated when the error probability conditional on predicted uncertainty equals that uncertainty for every value from 0 to 1.For example, predictions with uncertainty 0.2 should have a 20% top-1 error rate.Source: §2 · p. 2
2.1 Expected Uncertainty Calibration Error (UCE)
- UCE measures the expected absolute gap between predicted uncertainty and the error rate among predictions at that uncertainty, with a binned approximation.The approximation follows the expected calibration error approach.Source: §2.1 · p. 3
2.2 Temperature Scaling for Dropout Variational Inference
- Uncalibrated MC-dropout uncertainty diverges from perfect calibration in the ResNet-101 CIFAR-100 reliability diagrams.The diagrams use 15 bins and show the uncalibrated results in Figure 1's top row.Source: §2.2 · p. 3, Figure 1 · p. 3
- Temperature scaling is applied before softmax and Monte Carlo integration, with its temperature optimized by validation-set negative log-likelihood under MC dropout.The same temperature is used to calibrate confidence and uncertainty.Source: §2.2 · p. 3, Table 1 · p. 4
3 Experiments & Results
The experiments assess temperature scaling (TS) for MC-dropout uncertainty calibration, compare it with confidence penalty, and examine rejecting uncertain predictions. They report calibration results across networks and datasets, a comparison with confidence penalty, and the error pattern under rejection.
3.1 Uncertainty Calibration
- 45–66%: TS calibration reduced ECE, while UCE dropped drastically, especially for larger networks.The evaluation used 25 MC-dropout forward passes across different networks and datasets.Source: §3.1 · p. 4
3.2 Temperature Scaling vs. Confidence Penalty
- Confidence penalty adds negative softmax entropy to the negative log-likelihood training objective, weighted by β.The experiments compare TS with confidence penalty and examine rejection of uncertain predictions.Source: §3.2 · p. 4, §3 · p. 4
- ECE fell from 5.20% without confidence penalty to 3.37% with it, but TS was more effective and uncertainty remained largely miscalibrated.The comparison trained DenseNet-121 with dropout on CIFAR-10, fixing β = 0.1 and omitting data augmentation.Source: §3.2 · p. 5
3.3 Example: Rejection of Uncertain Predictions
- For Hmax < 0.8, calibrated uncertainty yields an almost-linear relationship between the rejection threshold and top-1 error.Lowering Hmax rejects more uncertain predictions; the plotted comparison contrasts uncalibrated and TS-calibrated uncertainty.Source: §3.3 · p. 5, Figure 3 · p. 5
4 Conclusion
The conclusion reports that temperature scaling effectively calibrates MC-dropout uncertainty without compromising accuracy, while noting that several factors affecting Bayesian uncertainty remain open for future work.
4 Conclusion
- TS effectively calibrates MC-dropout uncertainty, is more effective than confidence penalty, and does not compromise model accuracy.The conclusion also reports substantially better-calibrated softmax probabilities and describes the approach as simple to implement.Source: §4 · p. 5
- Network architecture, weight decay, and dropout configuration are among the factors influencing Bayesian deep-learning uncertainty that the paper leaves for future work.Source: §4 · p. 5
A.1 Expected Uncertainty Calibration Error
This appendix section restates expected uncertainty calibration error (UCE) and identifies error and uncertainty per bin as its components.
A.1 Expected Uncertainty Calibration Error
- The section restates the definition of Expected Uncertainty Calibration Error (UCE) from Eq. (8).Source: §A.1 · p. 7
- The error per bin is defined as a component of the UCE calculation.Source: §A.1 · p. 7
- Uncertainty per bin is defined as a component of the UCE calculation.Source: §A.1 · p. 7
A.2 Temperature Scaling with Monte Carlo Dropout
The appendix derives temperature scaling for MC-dropout variational inference by following the frequentist derivation, then optimizes its scaling parameter on validation data.
A.2 Temperature Scaling with Monte Carlo Dropout
- Temperature scaling for MC-dropout variational inference follows the frequentist derivation using logit vectors from N stochastic passes for each input.Source: §A.2 · p. 7
- The derivation formulates temperature scaling as an entropy-maximization solution.Source: §A.2 · p. 7
- The solution recovers ordinary temperature scaling when λ = T^-1, with T optimized by validation-set negative log-likelihood using MC dropout.Source: §A.2 · p. 7
A.3 Training Settings
Training used a 256-sample batch, AdamW, and a validation set drawn from the training data. Dropout was used during training and in 25-pass MC dropout inference.
A.3 Training Settings
- Implementations used PyTorch 1.2.Source: §A.3 · p. 7
- Models were trained with batch size 256 and AdamW, using an initial learning rate of 0.01 and β1 = 0.9, β2 = 0.999.Weight decay was 0.01; cross-entropy loss and a reduce-on-plateau scheduler were used, with patience 10 and factor 0.1.Source: §A.3 · pp. 7–8
- MC dropout used 25 forward passes with dropout probability 0.5; all models also used dropout probability 0.5 before the last linear layer during training.A 5,000-sample validation set was randomly extracted from the training set.Source: §A.3 · p. 8
A.4 Additional Reliability Diagrams
The additional reliability diagrams cover further dataset–model combinations. The proposed method calibrates all models with respect to UCE and ECE across all bins.
A.4 Additional Reliability Diagrams
- The proposed method calibrates all models with respect to both UCE and ECE across all bins.Source: §A.4 · p. 8
- Figure 4 shows 15-bin reliability diagrams for ResNet-18 on CIFAR-10.Source: Figure 4 · p. 8
- Figure 5 shows 15-bin reliability diagrams for DenseNet-169 on CIFAR-100.Source: Figure 5 · p. 8