PaperlayerView exact revision on arXiv

Source-linked AI summary

Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks

Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Wilson Leão, Steeven Janny, Christian Gagné

arXiv:1810.11586v3cs.LGstat.ML

TL;DR

DNN confidence is important for reliable decisions, but TS can struggle with small or noisy validation sets and highly accurate networks. The paper proposes ATS, a TS-family calibration method, and reports improved calibration in these challenging settings while retaining TS’s practical advantages.

  • Problem

    Temperature Scaling can fail to calibrate highly accurate networks or models calibrated with small or noisy validation sets, despite the importance of confidence in decision-making applications.

    Source: §1 · p. 1

  • Method

    Attended Temperature Scaling (ATS) is a TS-family post-processing method that rescales model outputs to improve calibration without retraining the network.

    Source: §1 · p. 2, §4 · p. 3, §7 · p. 9

  • Results

    ATS improves calibration while preserving TS’s time and memory advantages and network accuracy, including when validation sets are small or noisy or networks are highly accurate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ATS is presented as a practical option for calibrating already pretrained models in applications where calibration matters.

    Source: §7 · p. 9

    Temperature Scaling can find a suboptimal temperature when validation data contain too few correctly or incorrectly classified samples, as can occur with highly accurate classifiers or small validation sets.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliability to the system. To calibrate the deep networks' confidence, many probabilistic and measure-based approaches are proposed. Temperature Scaling (TS) is a state-of-the-art among measure-based calibration methods which has low time and memory complexity as well as effectiveness. In this paper, we study TS and show it does not work properly when the validation set that TS uses for calibration has small size or contains noisy-labeled samples. TS also cannot calibrate highly accurate networks as well as non-highly accurate ones. Accordingly, we propose Attended Temperature Scaling (ATS) which preserves the advantages of TS while improves calibration in aforementioned challenging situations. We provide theoretical justifications for ATS and assess its effectiveness on wide range of deep models and datasets. We also compare the calibration results of TS and ATS on skin lesion detection application as a practical problem where well-calibrated system can play important role in making a decision.

1. Introduction

DNN confidence matters for reliable decisions, but network outputs can be overconfident and poorly calibrated. The paper proposes ATS to improve calibration where TS struggles while preserving TS’s efficiency and accuracy.

1. Introduction

  • DNN softmax outputs are often far from true class probabilities and tend toward overconfidence, despite strong accuracy on challenging tasks.The paper describes overconfidence as assigning one class a probability near 1 and others near 0.Source: §1 · p. 1
  • DNN confidence is important in decisions such as transferring control in self-driving cars and identifying potentially deadly diseases in medical care.The passage describes high-confidence misclassification as a risk in medical care.Source: §1 · p. 2
  • TS can fail with highly accurate networks or small or noisy validation sets, which are practical challenges when collecting and expert-labeling calibration data.The authors note that non-professional labeling to reduce expenses may introduce noise, particularly in medical applications.Source: §1 · p. 2
  • ATS preserves TS’s time and memory advantages and network accuracy while improving calibration, especially with small or noisy validation sets and highly accurate DNNs.The authors also provide theoretical justifications for why ATS works better in these situations.Source: §1 · p. 2

2. Related Works

Related work distinguishes probabilistic uncertainty estimation from measure-based calibration. The paper focuses on post-processing methods, particularly temperature scaling, as a practical alternative to more computationally demanding approaches.

2. Related Works

  • Bayesian approximations, MC-dropout, and ensembles estimate uncertainty, but MC-dropout has high testing and training costs and ensembles require substantial test-time memory.The passage describes MC-dropout as approximating variational Bayesian inference and ensembles as models constructed by bagging.Source: §2 · p. 2
  • Measure-based methods calibrate by minimizing a calibration measure; NLL, ECE, and Brier score are common, while NLL can overfit and make networks overconfident.MMCE is described as a differentiable surrogate for ECE, which is not smooth.Source: §2 · p. 2
  • Post-processing methods including TS adjust the softmax layer without changing DNN weights, and the paper proposes ATS as a TS-family approach.These methods find softmax-softening parameters by minimizing calibration loss on a validation set.Source: §2 · p. 3

3. Problem Setup

The setup assumes a pretrained classifier and a small validation set, then frames calibration as rescaling logits to reduce calibration error. It introduces NLL and ECE as complementary calibration measures.

3. Problem Setup

  • The setup assumes a pretrained K-class model and a small validation dataset drawn from the same distribution as training and test data.The model produces logits for each validation sample, and its softmax output is interpreted as confidence.Source: §3 · p. 3
  • Calibration aims to rescale logits with parameter T to minimize the model’s calibration error.The setup identifies the model’s softmax output as its confidence.Source: §3 · p. 3

3.1.1 Negative Log Likelihood (NLL) [10]

  • NLL is an empirical estimate of expected negative log probability, and a smaller NLL indicates a closer match between predicted probabilities and the true conditional distribution.The passage presents NLL as a calibration measure comparing the model’s probability function with the data’s true conditional distribution.Source: §3.1.1 · p. 3

3.1.2 Expected Calibration Error (ECE) [31]

  • ECE measures the weighted absolute gap between confidence and accuracy across equally spaced confidence bins; the paper uses NLL for optimization and reports both measures.The passage notes that ECE is not differentiable.Source: §3.1.2 · p. 3

4. Temperature Scaling (TS) [13]

Temperature Scaling (TS) rescales logits with a single positive temperature chosen by minimizing validation-set NLL, which keeps the method efficient and preserves accuracy. Its optimum can be suboptimal when validation data are scarce, highly accurate classifiers have few errors, or labels are noisy.

4. Temperature Scaling (TS) [13]

  • TS rescales logits with one positive temperature T, selected by minimizing validation-set NLL to soften softmax outputs.Optimizing only one parameter gives TS low time and memory complexity and helps avoid overfitting the small validation set.Source: §4 · pp. 3–4

4.1. Analyzing TS Approach

  • Because TS rescales logits without changing their ordering, it preserves the network's accuracy.The paper identifies intact accuracy as an important property of post-processing calibration methods.Source: §4.1 · p. 4
  • When validation data lack enough correctly and misclassified samples, TS can find a suboptimal temperature, especially for highly accurate classifiers or small validation sets.Correctly classified samples push T toward 0, while misclassified samples push it toward infinity; their balance determines the optimum.Source: §4.1 · p. 4
  • TS is sensitive to label noise because its optimal temperature depends strongly on the samples' true labels in the NLL objective.The analysis also describes TS's temperature selection as maximizing confidence for true labels and minimizing it for other classes.Source: §4.1 · p. 4

5. Attended Temperature Scaling (ATS)

Attended Temperature Scaling (ATS) builds class-specific validation subsets by adding likely class-k samples from other labeled classes, then tunes temperature using a binary calibration loss. The paper reports that ATS achieves the best calibration in almost all reported measure-based experiments and is more robust than TS in challenging validation settings.

5. Attended Temperature Scaling (ATS)

  • ATS achieves the best calibration results in almost all reported model–dataset experiments among measure-based methods.The comparison includes TS and ATS as single-parameter methods alongside matrix and vector scaling.Source: Table 1 · p. 5
  • ATS forms each M_k from true class-k samples plus other-class samples whose class-k score exceeds threshold θ.The threshold is tuned on the validation set, and selected surrogate samples are generally near the class decision boundary.Source: §5 · pp. 4–5
  • Because ATS adds surrogate samples and changes the data distribution, it uses a new calibration measure rather than NLL to choose T.The measure is minimized over T for the constructed class-specific subsets.Source: §5 · p. 5
  • Minimizing L_ATS makes each class score approach Q(y = k|x), which the authors state yields calibrated model outputs.The loss casts calibration for each subset as a binary-label problem with labels k or not-k.Source: §5 · pp. 5–6

5.1. Analyzing ATS Approach

  • ATS reuses other-class validation samples to estimate class-conditional probabilities more accurately with small validation sets and reduce label-noise influence.The added samples are mostly near decision boundaries, supporting calibration for highly accurate networks with few misclassified samples.Source: §5.1 · p. 6

6. Experiments

ATS is reported to calibrate networks better in several settings where TS is less reliable, including noisy labels and small validation sets. The experiments also examine calibration methods in a skin-lesion referral application.

6. Experiments

  • The experiments compare calibration methods across diverse models and datasets, using 20% of each test set for validation and the remainder for testing.Methods include TS, matrix and vector scaling, MC-Dropout, and ensembles; ATS’s θ is tuned on validation data.Source: §6 · p. 6

6.1.1 Calibration

  • ATS calibrates networks better than most tested methods in many cases, while TS and ATS preserve accuracy and matrix/vector scaling overfit the validation set.The overfitting affects both accuracy and confidence.Source: §6.1.1 · p. 6

6.1.2 Robustness to Noise

  • Across 10%–50% label noise, TS is sensitive and can miss the optimal T even with modest noise, while ATS remains more robust unless labels are severely corrupted.The experiments report mean and standard deviation of NLL; Figure 3 depicts the behavior.Source: Figure 3 · p. 7, §6.1.2 · p. 7

6.1.3 Robustness to the Size of Validation Set

  • With small validation sets, TS is unstable, whereas ATS can find the optimal T with fewer samples and is more stable across validation sizes.Figure 4 reports mean and standard deviation of NLL across five runs for four model–dataset combinations.Source: Figure 4 · p. 7, §6.1.3 · p. 7

6.1.1 Calibration

  • ATS improves calibration more on highly accurate networks, where too few misclassified validation samples can keep TS from reaching its local optimum.The paper also reports that small changes in T can significantly improve NLL.Source: §6.1.1 · p. 7
  • MC-Dropout and ensembles improve accuracy and calibration simultaneously, but increase time and memory costs and cannot calibrate already pretrained models.These methods are compared with measure-based approaches.Source: §6.1.1 · p. 7

6.2. Application

  • In the skin-lesion referral application, ATS improves decision-making by reducing referrals of correctly classified cases without missing misclassified cases, and it widens their confidence gap more than TS.The application uses a ResNet200 model trained on ISIC data.Source: §6.2 · p. 8, Table 4 · p. 8

7. Conclusion

The conclusion presents Attended Temperature Scaling (ATS) as a post-processing method that finds an optimal temperature in settings where classic Temperature Scaling (TS) fails. The skin-lesion examples show lower confidence from both methods on incorrect predictions, with ATS confidence higher than TS confidence in each listed case.

7. Conclusion

  • Figure 5 describes greater reliability when, at the same rate of correct classifications above a threshold, more misclassified samples fall below it.Source: Figure 5 · p. 8
  • Correct-case confidences are 0.92 ATS versus 0.58 TS for melanocytic nevus, 0.93 versus 0.58 for melanoma, and 0.92 versus 0.58 for BCC.Source: §7 · p. 8
  • For three incorrect predictions, ATS/TS confidence is 0.65/0.41 (benign keratosis→melanoma), 0.68/0.43 (melanoma→nevus), and 0.65/0.45 (Bowen→benign keratosis).Source: §7 · p. 8
  • ATS finds an optimal T for highly accurate networks and small or noisy validation sets, where classic TS fails to find the best T.Source: §7 · pp. 8–9
  • ATS is a post-processing approach that calibrates pretrained models without retraining them.Source: §7 · p. 9

Related papers

Loading 1810.11586v3…