Source-linked AI summary
Measuring Calibration in Deep Learning
Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, Dustin Tran
TL;DR
Calibration measurement is challenging because ECE has multiple flaws and calibration-measure choices can alter evaluations. The paper systematically studies these choices through alternative metrics and recalibration experiments, finding that class conditioning, the L2 norm, and adaptive binning improve evaluation robustness, with adaptive binning limited in sparse regions.
Problem
Expected Calibration Error, the dominant calibration metric, has numerous flaws, while the effects of calibration-measure choices on empirical conclusions and recommended remedies remain insufficiently understood.
Method
The paper studies calibration-measure variants covering prediction coverage, thresholding, class conditionality, binning, adaptivity, and norms, evaluating them through recalibration and metric-consistency experiments.
Results
Across multiple image-classification datasets, recalibration-method rankings depend strongly on the calibration measure; adaptive binning has mean rank correlation 0.5927 versus 0.3677 for even binning.
Takeaways & Limitations
The paper recommends class conditioning, adaptive binning, and the L2 norm for more effective and robust calibration evaluation.
Takeaways & Limitations
Adaptive calibration metrics can create excessively large ranges in sparsely populated output regions, reducing interpretability and increasing within-range variance when granularity is increased.
Abstract
from arXiv · showhide
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How one measures calibration remains a challenge: expected calibration error, the most popular metric, has numerous flaws which we outline, and there is no clear empirical understanding of how its choices affect conclusions in practice, and what recommendations there are to counteract its flaws. In this paper, we perform a comprehensive empirical study of choices in calibration measures including measuring all probabilities rather than just the maximum prediction, thresholding probability values, class conditionality, number of bins, bins that are adaptive to the datapoint density, and the norm used to compare accuracies to confidences. To analyze the sensitivity of calibration measures, we study the impact of optimizing directly for each variant with recalibration techniques. Across MNIST, Fashion MNIST, CIFAR-10/100, and ImageNet, we find that conclusions on the rank ordering of recalibration methods is drastically impacted by the choice of calibration measure. We find that conditioning on the class leads to more effective calibration evaluations, and that using the L2 norm rather than the L1 norm improves both optimization for calibration metrics and the rank correlation measuring metric consistency. Adaptive binning schemes lead to more stablity of metric rank ordering when the number of bins vary, and is also recommended. We open source a library for the use of our calibration measures.
1 Introduction
Calibration measures whether predicted probabilities match observed accuracy, but the widely used ECE metric has important shortcomings that can affect calibration conclusions.
- Calibration assesses whether predicted probabilities match the frequency of correct outcomes, such as a class assigned 90% probability appearing 90% of the time.
- Proper scoring rules such as negative log likelihood do not directly measure calibration, and optimizing them can still produce uncalibrated probabilities.
- ECE became a widely used measure for evaluating calibration and motivated subsequent work on recalibrating deep neural networks.
- The paper identifies pathologies in ECE that prevent it from properly evaluating recent calibration methods reported to improve ECE.
- The study examines ECE’s class conditioning, norm, and prediction-coverage choices, analyzes calibration metrics experimentally, proposes alternatives, and recommends class conditioning, adaptivity, and the L2 norm.
2 Background & Related Work
Calibration formalizes whether predicted probabilities match true outcome frequencies, while common evaluation and recalibration methods make different choices about binning, classes, and model outputs. The section introduces ECE and post-processing approaches including Platt scaling, temperature scaling, and isotonic regression.
- Calibration: Calibration error measures the difference between the true probability of a label given prediction p and p itself.A model is calibrated when these quantities match for every probability value and class label.
- Expected Calibration Error: ECE estimates expected calibration error by comparing bin accuracy with mean confidence across fixed probability bins.Each prediction is assigned to a bin, and bin contributions are weighted by their number of predictions relative to the dataset size.
- Expected Calibration Error: Standard ECE implementations leave ambiguity about binning and multiclass calibration, commonly using equally spaced bins and only the maximum predicted probability.This formulation does not evaluate the other class probabilities.
- Recalibration Methods: Post-processing calibrates classifier outputs without retraining, using parametric methods such as Platt scaling or non-parametric isotonic regression.These methods fit calibration mappings on held-out validation data and apply them at test time.
- Recalibration Methods: Multiclass calibration extends these methods through matrix or vector scaling, temperature scaling, or separate one-versus-all isotonic models.Temperature scaling uses a single positive scalar parameter, whereas multiclass isotonic regression calibrates binary problems separately before combining probabilities.
3 Issues With Calibration Metrics
Common calibration metrics can obscure meaningful errors because maximum-probability and fixed-bin formulations omit class-specific behavior and allow opposing errors to cancel. Class-wise evaluation and attention to probability density and bin counts are therefore central concerns in assessing calibration metrics.
- Multiclass Coverage: ECE is reductive for multiclass predictions because it evaluates only the predicted class’s probability, ignoring the remaining K −1 probabilities.This omission matters when secondary or tertiary predictions are relevant to decisions or outcomes.
- Multiclass Coverage: Per-class calibration error is non-uniform across classes, making comprehensive calibration across all class probabilities important for applications with unequal decision costs.Clinical decisions may depend on secondary or tertiary disease predictions, especially under noisy data distributions.
- Binning: Evenly spaced bins can misrepresent calibration when prediction probabilities are unevenly dispersed, particularly under sharp outputs with sparsely populated ranges.Probability distributions may be left-skewed, leaving some fixed bins sparse and others dense.
- Binning: Increasing the number of bins reduces measurement bias but increases variance because fewer data points populate each bin.This bias-variance tradeoff is compounded when sharpness leaves some fixed calibration ranges sparsely populated.
- Metric Reliability: Static binning can produce near-zero calibration error when overconfident and underconfident predictions overlap and cancel within the same bin.Consequently, differences between models may reflect metric cancellation rather than genuine calibration improvement.
4 Analyzing The Space of Calibration with General Calibration Error
General Calibration Error organizes calibration metrics by choices such as class conditioning, binning, thresholding, and norm. The proposed SCE and ACE variants respectively evaluate all class probabilities and adapt bins to prediction density.
- General Calibration Error: General Calibration Error captures calibration metrics by varying five measurement properties.It takes these property choices as inputs and returns the corresponding metric.
- Class conditionality: Class-conditional calibration computes errors independently for each class before averaging, preventing systematic class differences from being washed out.Class weighting remains even when all datapoints are included in the loss.
- Adaptivity: Adaptive Calibration Error spaces bin intervals so each contains an equal number of predictions, focusing estimation where predictions are concentrated.The adaptive design is motivated by the bias-variance tradeoff from allocating ranges across regions with different prediction density.
- Maximum Probability: Static Calibration Error extends ECE to every class probability by binning each class separately, computing within-bin errors, and averaging them.Unlike maximum-probability ECE, this evaluates non-maximum probabilities as well.
- Static Calibration Error: With infinite data and bins, SCE is zero if and only if the model is calibrated.Its bin-level terms compare accuracy and confidence for each class and bin.
- Thresholding: Thresholding addresses inefficient bin use caused by the many infinitesimal softmax probabilities, especially in multiclass settings.Without thresholding, bins can be devoted to unlikely regions.
5 Experiments
The experiments show that calibration-metric design choices substantially affect both metric optimization and the ordering of recalibration methods. Adaptive binning improves robustness to bin-count changes, while different metric properties can produce inconsistent conclusions about calibration quality.
- 5.1 Class Conditionality: Class-conditional histogram binning separates probabilities by predicted class before estimating accuracy, unlike standard histogram binning, which combines classes.This comparison tests whether class conditioning changes measured calibration effectiveness.
- 5.1 Class Conditionality: One-third: on CIFAR-10, ECE estimated one-third the calibration error of its class-conditional counterpart.The two measures differed only in class conditionality.
- 5.2 Metric Properties: Metric optimization was most sensitive to even versus adaptive binning and to the L1 versus L2 norm, with performance strongest on metrics matching the optimized metric’s properties.The experiments optimized temperature-scaling parameters for different calibration-error variants.
- 5.3 Sensitivity to the Number of Bins Hyperparameter: Adaptive binning achieved mean rank correlation 0.5927 versus 0.3677 for even binning across bin-count variations.The comparison used recalibration techniques on a Wide ResNet trained on CIFAR-10.
- 5.4 Recalibration Experiments: Across 32 metrics, calibration rankings for recalibration techniques were dramatically inconsistent despite identical datasets and architectures.The CIFAR-10 comparison evaluated eight recalibration techniques, including temperature, vector, matrix, and neural-network scaling.
- 5.4 Recalibration Experiments: Ignoring calibration-metric properties when evaluating post-processing methods can lead to misleading conclusions about their success.This finding challenges evaluations based on a single metric such as ECE.
6 Discussion
The paper identifies problems with ECE, proposes alternative metrics, and recommends practices for evaluating multiclass calibration. It also notes a limitation of adaptive metrics in sparse output regions.
- It proposes SCE, ACE, and GCE to explore and ameliorate these calibration-measurement problems.
- The study recommends best practices for evaluating calibration.
- The paper recommends GCE over ECE for evaluating a multiclass classifier’s calibration error.
- Adaptive calibration metrics can create excessively large ranges when output is very sparse, reducing interpretability and increasing within-range variance.
A.1 Thresholding & Thresholded Adaptive Calibration Error
Thresholding addresses the dominance of infinitesimal softmax predictions in adaptive calibration, while training-set sparsity can make the lowest thresholded range unstable.
- Most softmax predictions become infinitesimal, which can wash out calibration scores, especially for classifiers with many classes.
- Thresholded Adaptive Calibration Error evaluates only probability values above a threshold ϵ, unlike ACE’s unthresholded evaluation.
- Thresholded predictions overlap with ECE’s maximum-per-datapoint predictions, producing similar reactions to recalibration methods.
- On MNIST training data, sparse small probabilities can make the lowest thresholded range spike to .99 or higher because every datapoint is fit.
- On Fashion-MNIST validation data, the figure compares ACE with 100 ranges against thresholded adaptive calibration with 50 ranges over training.
B Label Noise
Label-noise experiments retrain softmax regression across increasing noise levels and compare how ECE, ACE, and SCE respond as predictions become less certain.
- Softmax regression models are retrained at 40 label-noise levels from 0.0 to 0.05.
- The label-noise value is the percentage of randomly assigned labels, with the correct label included among possible assignments.
- As label noise increases, the fraction of important predictions omitted by ECE rises with modeling difficulty.
- ACE and SCE account for all predictions, whereas ECE evaluates only each sample’s maximum-confidence prediction.
- As label noise increases, model output certainty and accuracy decrease.
C Calibration Metric Labeling
The calibration-metric labeling enumerates variants by binning scheme, maximum-probability inclusion, class conditionality, threshold, and norm.
- Metric variants are ordered by binning scheme, maximum probabilities, class conditionality, threshold, and norm.
D Challenges in Calibration
The paper frames calibration measurement as challenging and introduces the broad challenges before presenting new metrics.
- The draft notes that Sections 5–6.2 and an isolated experiments section may be consolidated into the discussion.
- The authors signal that challenges in calibration measurement precede their proposed metrics.
D.1 Ground Truth & Comparing Calibration Metrics
Calibration lacks a single ground-truth estimation target because it spans probabilities and classes, while metrics make different approximation choices. These choices include which predictions and probability regions to weight, how to bin values, and whether to emphasize uncertain predictions.
- Calibration is a guarantee across every predicted probability p ∈[0, 1] and class label y, making estimation error vary across probability-class regions.
- Different metrics approximate different overall calibration values because they differ in assumptions, prediction coverage, and binning bias.
- Calibration metrics can weight probability regions differently, including thresholding values to emphasize rare events or other selected aspects of performance.
- Out-of-distribution detection places particular importance on calibration near probability 0.5, where predictions represent high uncertainty.