Calibrating Where It Matters: Constrained Temperature Scaling

• Deep classifiers often provide poorly calibrated predictions, while calibration quality varies with neural architecture.Calibration methods include training regularization, post-hoc calibration, and combinations of the two.Source: §1 · p. 1
• Calibrated classifiers can support actions that minimize expected cost, but healthcare settings differ in their cost functions and those functions are unlikely to be known at training time.In binary classification, relative false-positive and false-negative costs and class priors determine a cost-minimizing threshold.Source: §1 · p. 1
• Because deployment cost functions are constrained by prior knowledge, calibration is most important near the decision boundaries they could produce, not uniformly across the probability simplex.The paper illustrates this with triage, where the relative costs of misclassifying malignant and benign cases constrain the possible decision-boundary region.Source: §1 · p. 1
• The paper modifies temperature estimation to tailor calibration to clinically motivated cost constraints and demonstrates it with convolutional classifiers on the ISIC 2019 dermoscopy dataset.The experiments cover binary and multi-class classification.Source: §1 · p. 2

• Temperature scaling post-hoc calibrates deep classifiers by scaling logits with a single temperature fitted on a validation set to minimize negative log-likelihood.The validation set is also called the calibration set.Source: §2 · p. 2
• For binary classification with decision thresholds in (0.0, 0.5), the modified fit minimizes negative log-likelihood using only calibration examples with p < 0.5.The paper denotes this fitted temperature T*; above 0.5, it considers sufficient that observed frequencies be at least 0.5.Source: §2 · p. 2
• For multi-class classification, the modified fit uses calibration examples whose most probable class is benign, reflecting the assumption that mistakes involving benign classes are most costly.Temperature does not change membership in the selected subset; the estimated temperature is denoted T*.Source: §2 · p. 2

3 Experiments

• Experiments used seven ISIC 2019 classes, a fixed 60:20:20 train-validation-test split, and ImageNet-pretrained EfficientNet B7 and ResNet101 architectures.The validation set supported early stopping and temperature estimation.Source: §3 · p. 2
• The EfficientNet B7 binary reliability diagram shows the modified-scaled classifier consistent with reliability and especially well calibrated for p ≤ 0.5, unlike the uncalibrated classifier.Figure 1 compares classifiers without calibration and with modified temperature scaling.Source: §3 · p. 2, Figure 1 · p. 3
• T* further reduced binary ECE for test examples with p < 0.5 while retaining much of temperature scaling's benefit for p ≥ 0.5.Standard temperature scaling reduced ECE in both ranges for both architectures.Source: §3 · p. 2, Table 1 · p. 3
• T* further reduced ECEBen, marginally for ResNet, without increasing ECEMal in the multi-class results.Standard temperature scaling reduced ECE in both benign-predicted and malignant-predicted regions for both architectures.Source: §3 · pp. 2–3, Table 2 · p. 3

4 Discussion

• Temperature-estimation changes can improve calibration near probability-simplex regions likely to be near decision boundaries.The method accounts for constraints on cost functions, with class priors; broader exploration across cost-function priors and calibration methods is future work.Source: §4 · p. 3
• The method may adversely affect estimation of overall expected cost.The preliminary study should be extended to multiple datasets and architectures, with robustness to dataset shift investigated.Source: §4 · p. 3
• When reporting confidences, probabilities can be given in the well-calibrated simplex region, while elsewhere a qualitative comparison may be more appropriate.The example may also be applied to other accuracy-preserving post-hoc calibration methods or combined with decision calibration.Source: §4 · p. 3