Mitigating Bias in Calibration Error Estimation

1 INTRODUCTION

• Statistical bias in ECEbin—the difference between its expected value and TCE—has largely been unexplored, leaving its magnitude, sign, and dependence on estimator hyperparameters unknown.Relevant choices include the number of bins and how they are formed.Source: §1 · p. 1
• Calibration supports judging whether confidence reflects correctness, but inaccurate error estimates may impair that assessment and the choice of recalibration method.The paper situates the issue in high-stakes applications including self-driving cars and medical diagnosis.Source: §1 · p. 1
• The bias-by-construction (BBC) framework uses simulation to make TCE analytically computable and estimate calibration-metric bias directly.BBC finds systematic, non-negligible ECEbin bias, particularly for perfectly calibrated models.Source: §1 · p. 2
• ECEsweep extends ECEbin by selecting the number of bins to ensure a monotonic calibration histogram, alongside comparisons with ECEdebias and KDE.ECEbin, ECEdebias, and ECEsweep use bins specified either by equal width or by equal numbers of instances.Source: §1 · p. 2
• ECEsweep selects the optimal recalibration method correctly 70% of the time, versus 30% for ECEbin.Optimality is assessed against TCE estimates from numerical integration on maximum-likelihood-fitted curves, with the model family selected by Akaike information criterion.Source: §1 · p. 2
• ECEbin is more biased than ECEdebias and ECEsweep; ECEdebias performs better for perfectly calibrated models, while ECEsweep performs better for miscalibrated models.Equal-mass binning gives more accurate TCE estimates for all binning-based estimators.Source: §1 · p. 2

2 RELATED WORK

• Prior work reports that ECEbin is sensitive to implementation choices, including bin count, binning technique, and the norm used.The paper distinguishes its contribution by quantifying bias in simulation for realistic model outputs across architectures, datasets, and implementation hyperparameters.Source: §2 · pp. 2–3
• For ten models, ECEsweep matches the TCE-based recalibration choice 7/10 times, versus 3/10 for ECEbin.The comparison covers histogram binning, isotonic regression, and temperature scaling.Source: Table 1 · p. 3
• Brocker (2012) and Ferro and Fricker (2012) propose ECEdebias, which uses jackknife estimates of per-bin bias; Zhang et al. (2020) propose smoothed KDE.ECEdebias and KDE have additional hyperparameters, and the paper compares their relative bias-reduction advantages with ECEsweep.Source: §2 · p. 3
• Other calibration-error measures include kernel-based KCE and a KS-inspired metric for worst-case deviation, whereas TCE measures average confidence-accuracy difference.The passage notes that worst-case and average-difference measures may apply in different circumstances.Source: §2 · p. 3
• The paper is the first to propose using calibration-curve monotonicity to improve calibration metrics and provides theoretical and empirical evidence that modern deep networks exhibit monotonic curves.Earlier work by Zadrozny and Elkan (2002) used monotonicity to motivate isotonic regression, based on observations from a handful of pre-deep-learning models.Source: §2 · p. 3

3 BACKGROUND

• TCE is the ℓp-norm difference between predicted confidence and the true likelihood of being correct, with its value determined by both the confidence-score distribution and the true calibration curve.The curve relates confidence scores to empirical accuracy; the outer expectation is over the score distribution.Source: §3 · p. 3

3.1 Estimates of calibration error

• A sample-based estimate approximates TCE by replacing its outer expectation with a sample average and estimating the inner expectation from instances with similar confidence scores.In the experiments, the ℓ2 norm is used to increase sensitivity to extremely poorly calibrated predictions.Source: §3.1 · pp. 3–4
• Label-binned calibration error uses bins to define neighborhoods and estimate empirical accuracy, whereas ECEbin collapses each bin and compares its mean confidence with its empirical accuracy, weighted by bin count.The instances are partitioned into b bins; the figure illustrates how confidence-score distribution and calibration curve affect measurement.Source: §3.1 · p. 4, Figure 2 · p. 4
• For the same bins, ECElb(f) is at least ECEbin(f), as follows from Jensen’s inequality applied to each bin’s inner term.The two metrics differ in whether they retain individual instances or collapse them into bin averages.Source: §3.1 · p. 4

4 THE BBC FRAMEWORK

• BBC estimates bias by comparing an estimator’s average across simulated datasets with the analytically computed TCE.It assumes a confidence distribution and true calibration curve, generates n samples from them, and repeats the process m times.Source: §4 · p. 4
• The framework studies ECEbin with equal-width bins using parametric confidence and calibration curves fitted to ResNet-110 CIFAR-10 outputs.The framework focuses on bias because variance can be estimated from finite samples; too many bins with few samples per bin can impair the calibration-curve estimate.Source: §4 · p. 4
• ECEbin can overestimate or underestimate TCE at finite sample sizes, and increasing the bin count does not always improve its estimate.For each sample size, the empirically optimal bin count grows with sample size.Source: §4 · p. 4

5 MONOTONIC CALIBRATION METRICS

• The optimal ECEbin bin count depends on the binning technique, sample size, confidence distribution, and true calibration curve.This dependence motivates the paper’s adaptive bin-selection method.Source: §5 · p. 5
• For maximum-likelihood models, convex ROC curves in the infinite-data limit imply monotonically increasing calibration curves.The paper relates ROC-curve slopes to calibration curves and states that the implication also holds in reverse.Source: §5 · p. 5
• Finite data and cross-validation domain differences can produce apparent non-monotonicity; the authors assume such domain shifts are negligible for i.i.d. splits.They posit that calibration-error assessment may assume a monotonic true calibration curve because deviations are considered artificial.Source: §5 · p. 5
• ECE_sweep chooses the largest bin count for which the bin heights, including those at all smaller counts, remain monotonic.The sweep increases the count from two until it encounters non-monotonic binning or reaches one bin per sample, then returns the last monotonic count.Source: §5 · p. 5
• For classifiers trained with non-likelihood criteria, non-monotonic calibration may instead signal that a model is unreasonable or inadmissible on the task.Moment matching is given as an example of a non-likelihood-based criterion.Source: §5 · p. 5

6 FITTING THE CALIBRATION CURVE AND SCORE DISTRIBUTION

• Analytic or numerical TCE computation requires parametric forms for the confidence distribution and true calibration curve, which the study fits to empirical logit datasets.The confidence distribution is fit by maximum likelihood, and calibration curves are selected from generalized linear models using AIC.Source: §6 · pp. 5–6
• The study fits 10 public logit datasets from four architectures trained on CIFAR-10, CIFAR-100, and ImageNet, using top-label confidence and correctness as a binary calibration problem.The architectures are ResNet, ResNet-SD, Wide-ResNet, and DenseNet.Source: §6 · p. 5
• Simulated ECEbin scores from the fitted distributions and curves closely match scores on original data, while dataset source influences curves more than architecture.The score distributions are significantly skewed; ImageNet models are least skewed and CIFAR-10 models most skewed.Source: Figure 3 · p. 6, §6 · p. 6
• Confidence scores are modeled with beta densities, while calibration curves use selected generalized linear models with monotonic functions.The models use logit, log, or log(1 − x) link and transformation functions, selected by AIC.Source: §6 · pp. 5–6

7.1 Estimating bias on real models and data

• The BBC framework generates synthetic examples from real-model fits and estimates bias from the difference between estimated ECE and true calibration error across 1,000 samples.Source: §7.1 · p. 6
• Equal-mass binning consistently has smaller-magnitude bias than equal-width binning, and ECEem sweep performs best overall, especially at low sample sizes.Source: §7.1 · pp. 6–7
• 0.347 mean absolute bias for ECEem sweep versus 0.504 for ECEem debias across datasets, models, and sample sizes (t = 5.10, p < 1e−5).Source: §7.1 · p. 7

7.2 How well can we detect miscalibration?

• Detecting 2% miscalibration requires over 10,000 samples; with fewer than 500, reliable detection requires miscalibration greater than 10%.Source: §7.2 · pp. 7–8
• ECEem sweep has significantly lower miss rates than ECEew bin, particularly below 10,000 samples.Source: §7.2 · p. 7, Figure 5

7.3 Perfect calibration

• Under perfect calibration, KDE has the least bias, but its high bias in realistic scenarios makes it nonviable; excluding KDE, ECEem debias is least biased.Source: Figure 6 · p. 8, §7.3 · p. 8
• Choosing ECEem sweep versus ECEem debias depends on prior beliefs about model miscalibration; both outperform common-practice ECEew.Source: §7.3 · p. 8

8 DISCUSSION AND CONCLUSION

• Using ECE_sweep instead of ECEbin to select a recalibration method leads to better choices and better test-set calibration.The conclusion notes that bias may also have affected conclusions in prior calibration studies, including Guo et al. (2017).Source: §8 · p. 8
• ECE_sweep is reported as more sensitive than ECEbin for detecting model miscalibration.The comparison is tied to Figure 5 in the discussion.Source: §8 · p. 8
• Naive calibration-error estimates used with training batches below O(1000) may be flawed, particularly as model score distributions change during training.The discussion cautions that a calibration measure may be more affected by the changing score distribution than by the true calibration curve.Source: §8 · p. 8
• The authors identify unexamined extensions including assumption violations, less-skewed confidence distributions, binary classification, and alternative calibration measures.They also propose studying the interplay between bias and variance.Source: §8 · pp. 8–9
• Statistical bias in calibration-error estimators grows near perfect calibration, but equal-mass binning and ECE_sweep or debiased estimators can mitigate it.The discussion also identifies improved recalibration choices and miscalibration detection as benefits of bias-reduced measurement.Source: §8 · p. 9

A.1 Confidence score distribution fits

• The appendix fits Beta distributions to top-label confidence scores from each of 10 empirical datasets using maximum likelihood.A recursive grid search estimates the Beta parameters, with experiments reporting robustness to its hyperparameters.Source: §A.1 · p. 11, Table 2 · p. 11

A.2 Calibration curve fits

• The appendix fits binary GLMs to calibration functions across link-function and score-transform choices, selecting each dataset's best fit by AIC.The optimized standard loss is used for the candidate combinations.Source: §A.2 · p. 11

A.3 Comparing ECEbin computed on simulated data versus real data

• ECEbin measured on original logits is relatively close to the average from 1,000 simulated datasets, indicating that the parametric models fit the original data well.The comparison is reported for the model and dataset fits summarized in the appendix.Source: §A.3 · p. 11

B.1 Bias

• ECEbin can overestimate true calibration error, and its optimal bin count depends on sample size.The appendix notes that bias is evaluated across binning methods and fixed bin counts; ECE_sweep adapts its bin count to sample size.Source: §B.1 · p. 15
• Equal-mass binning generally yields less-biased calibration metrics than equal-width binning, and ECEbin's least-biased bin count changes with sample size.The bias plots cover fitted outputs for CIFAR-10 ResNet-110, CIFAR-100 Wide ResNet-32, and ImageNet ResNet-152 while varying sample size and bin count.Source: Figure 7 · p. 15, §B.1 · p. 15, Figure 8 · p. 16

• The plotted settings use fitted model outputs from CIFAR-10 ResNet-110, CIFAR-100 Wide ResNet-32, and ImageNet ResNet-152.Source: §B.2 · p. 17
• Variance decreases with sample size, but shows no clear dependence on the number of bins.Source: Figure 10 · p. 17, §B.2 · p. 17, Figure 11 · p. 18
• The variance comparison varies sample size and bin count for calibration metrics using both equal-width and equal-mass binning.Source: §B.2 · p. 17

• The analysis averages estimates across 1,000 simulated datasets at n = 200 and n = 5,000, using uniform and fitted beta score distributions.Source: C Controlling true calibration error… · p. 19
• The true calibration curve is set to EY [Y | f(X) = c] = c^d, with d varied from 1 to 10 to control TCE from perfect calibration upward.Source: C Controlling true calibration error… · p. 19
• Bias is highest at 0% TCE and generally decreases as TCE increases; at n = 5,000, ECEbin can still be off by 2% for perfectly calibrated models, while ECEsweep significantly reduces bias.Source: C Controlling true calibration error… · p. 19

• At n = 200, an ECEew bin estimate of 12% could correspond to 5% or 8% TCE, while ECEem sweep somewhat mitigates the bias and ambiguity.Source: sweep choose? · p. 19
• For uncalibrated models, ECEsweep chooses a bin count close to optimal; for perfectly calibrated models, the optimal count is small (<=4) and its selection is poor.Source: Figure 14 · p. 19
• Poor bin selection may partly explain residual ECEsweep bias for perfectly calibrated models, and any binning method that always returns a positive value cannot be fully unbiased there.Source: Figure 14 · p. 19