Source-linked AI summary
Bin-wise Temperature Scaling (BTS): Improvement in Confidence Calibration Performance through Simple Scaling Techniques
Byeongmoon Ji, Hyemin Jung, Jihyeun Yoon, Kyungyul Kim, Younghak Shin
TL;DR
Modern deep networks can be poorly calibrated, so confidence may not reliably reflect prediction accuracy in applications where output interpretation matters. This paper introduces bin-wise temperature scaling and a validation-augmentation variant; across evaluated models and datasets, these methods generally improve calibration over temperature scaling without changing test error rates.
Problem
Deep neural networks can be overconfident, so their predicted confidence may exceed their accuracy and complicate interpretation.
Source: §1 · p. 1Method
The paper proposes temperatures for separate confidence bins and augments validation samples to obtain more stable bin temperatures.
Source: §1 · p. 2, §2.2 · p. 3, §2.3 · p. 3Results
BTS improves calibration over TS in 10 of 12 confidence-interval-binning cases and 11 of 12 sample-count-binning cases; sample-count ABTS outperforms TS in all listed cases.
Source: §4 · p. 4Takeaways & Limitations
Because TS, BTS, and ABTS are post-processing methods, their test error rates remain the same as the baseline model’s.
Source: §4 · p. 4Both confidence-interval and sample-count binning may yield unstable temperatures in low-confidence regions when validation samples there are sparse.
Source: §2.2 · p. 3, §2.3 · p. 3
Abstract
from arXiv · showhide
The prediction reliability of neural networks is important in many applications. Specifically, in safety-critical domains, such as cancer prediction or autonomous driving, a reliable confidence of model's prediction is critical for the interpretation of the results. Modern deep neural networks have achieved a significant improvement in performance for many different image classification tasks. However, these networks tend to be poorly calibrated in terms of output confidence. Temperature scaling is an efficient post-processing-based calibration scheme and obtains well calibrated results. In this study, we leverage the concept of temperature scaling to build a sophisticated bin-wise scaling. Furthermore, we adopt augmentation of validation samples for elaborated scaling. The proposed methods consistently improve calibration performance with various datasets and deep convolutional neural network models.
1. Introduction
Modern CNNs can be accurate yet overconfident, making their outputs harder to interpret. The paper proposes bin-wise temperature scaling, with validation augmentation, to improve calibration.
1. Introduction
- Deeper networks can become increasingly overconfident, with confidence values exceeding their accuracy despite improved classification performance.The paper notes that this mismatch complicates interpretation in applications such as cancer prediction.Source: §1 · p. 1
- Post-processing calibration adjusts an already trained model, whereas other methods calibrate confidence during training.The cited training-time methods avoid a separate validation dataset, but their calibration performance was not consistently improved compared with temperature scaling.Source: §1 · p. 1
- Temperature scaling uses a validation dataset to calibrate predicted confidence without affecting the test error rate.The paper describes it as a simple and effective post-processing technique.Source: §1 · p. 1
- BTS uses different temperatures for confidence ranges, while ABTS augments validation samples before applying the same bin-wise procedure.The paper evaluates both methods with recent deep models and image-classification datasets.Source: §1 · p. 2, Figure 1 · p. 2
2. Methods
The methods calibrate predicted confidence by scaling softmax outputs, moving from one validation-fitted temperature to confidence-dependent bin temperatures and augmentation of sparse validation bins.
2. Methods
- A calibrated confidence should match the accuracy observed among samples assigned that confidence.For example, ten samples with confidence 0.8 are expected to yield 80% accuracy.Source: §2 · p. 2
2.1. Temperature Scaling
- Temperature scaling (TS) fits one positive temperature on validation data and applies it to all test samples; values above 1 soften softmax for overconfident predictions.The temperature is optimized by minimizing validation negative log likelihood.Source: §2.1 · p. 2
2.2. Bin-wise Temperature Scaling
- Bin-wise temperature scaling (BTS) fits temperatures using validation samples within bins, then selects a test sample’s temperature according to its confidence.Equal-width confidence bins can leave low-confidence bins with very few validation samples, making their temperatures unstable.Source: §2.2 · pp. 2–3
- Sample-count binning equalizes validation samples across bins, except that samples above 0.999 share one bin to avoid redundant temperatures for nearly identical high confidences.ABTS augments sparse validation bins before applying the BTS procedure; this study augments eight bins below 0.8, although the optimal number may vary by dataset and model.Source: §2.2 · p. 3, §2.3 · p. 3
2.3. Augmentation based Bin-wise Temperature Scaling
- ABTS targets low-confidence-bin instability, which can persist with sample-count binning because sparse low-confidence samples span a wider confidence range.The study applies augmentation to bins with few validation samples, then performs the BTS procedure.Source: §2.3 · p. 3
3. Experiments
The experiments compare BTS and ABTS with temperature scaling across image datasets and CNN architectures, assessing calibration with expected calibration error and reliability diagrams.
3. Experiments
- The study evaluates BTS and ABTS against conventional temperature scaling using expected calibration error across four image datasets and five network architectures.Table 1 reports ECE (%) for the baseline model, TS, BTS, and ABTS across datasets and models.Source: §3 · p. 3, Table 1 · p. 4
3.1. Datasets
- CIFAR-10/100, Caltech-UCSD Birds, and Stanford Cars provide the image-classification datasets, with separate training, validation, and test sets.CIFAR-10/100 use 45,000, 5,000, and 10,000 images; Birds use 5,994, 2,897, and 2,897; Cars use 8,041, 4,020, and 4,020.Source: §3.1 · p. 3
3.2. Models
- CIFAR-10/100 use four CNNs, while Birds and Cars use DenseNet and ResNet-101; the latter datasets use ImageNet-pretrained weights.CIFAR models are trained from scratch; the study also describes image normalization and momentum-based stochastic gradient descent.Source: §3.2 · p. 3
3.3. Evaluation Metrics
- Expected calibration error (ECE) is the weighted average of the per-bin difference between accuracy and average confidence.Reliability diagrams bin test samples by confidence range and display each bin’s accuracy and average confidence; perfect calibration aligns them.Source: §3.3 · pp. 3–4
4. Results
BTS and ABTS generally improve calibration over temperature scaling across the evaluated models and datasets, while ABTS with number-of-samples binning performs best overall. Results also examine augmentation choices, bin counts, and reliability on individual examples.
Table 1 results
- BTS improves on TS in 10 of 12 confidence-interval-binning cases and 11 of 12 number-of-samples-binning cases; number-of-samples ABTS beats TS in every Table 1 case.The best and second-best calibration results generally come from number-of-samples ABTS and confidence-interval ABTS, respectively; all three methods preserve the baseline test error rate.Source: §4 · p. 4
Reliability diagrams
- For Wide ResNet on CIFAR-100, ABTS’s bin-specific temperatures yield better overall calibration than TS, which lowers overall confidence but leaves low-confidence samples more under-confident.The figure compares the uncalibrated baseline, TS, and number-of-samples ABTS; the baseline is under-confident below 0.4 and over-confident above 0.4.Source: §4 · p. 4, Figure 2 · p. 5
Example predictions
- For an ambiguous CIFAR-10 image misclassified as Truck, confidence falls from 0.98 with the baseline to 0.83 with TS and 0.68 with ABTS.ABTS also has the lowest ECE for CIFAR-10 with VGG 16 in Table 1 and the lowest confidence on the illustrated incorrectly classified ambiguous images.Source: §4 · p. 5, Figure 3 · p. 5
Augmentation results
- The augmentation comparison evaluates image shifting, brightness adjustment, contrast control, and blur on CIFAR-100 validation samples.Table 2 reports ECE for confidence-interval and number-of-samples ABTS across these augmentation choices.Source: §4 · pp. 5–6, Table 2 · p. 5
- ABTS outperforms TS across four validation-sample augmentation techniques, with no clear augmentation leader and only marginal performance differences between techniques.The authors note that strong augmentation can shift the validation and test distributions and diminish ABTS effectiveness.Source: §4 · pp. 5–6, Table 2 · p. 5
Bin-count results
- BTS and ABTS outperform TS across the tested bin counts, and their ECE is not sensitive to bin-count changes in the CIFAR-100 comparisons.The main results use 50 bins, although the optimal count may vary by dataset and model; the evaluated counts include 5, 10, 20, and 50.Source: §4 · pp. 4, 6, Table 3 · p. 6
5. Conclusion
The conclusion proposes bin-wise temperature scaling to address the limits of a single temperature and adds validation-sample augmentation to stabilize bin temperatures. Across the evaluated deep CNNs and image datasets, the proposed methods outperform baseline temperature scaling.
Conclusion
- BTS applies separate temperatures by confidence-interval or number-of-samples bins, while ABTS augments validation samples to find stable temperatures for each bin.The methods were evaluated on various deep CNNs and image datasets.Source: §5 · p. 6
- The experiments indicate that the proposed methods outperform baseline temperature scaling across the evaluated models and image datasets.The conclusion frames these as post-processing scaling methods.Source: §5 · p. 6