PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Uncertainty Calibration via Proper Scores for Classification and Beyond

Sebastian G. Gruber, Florian Buettner

arXiv:2203.07835v4cs.LGstat.ML

TL;DR

The paper addresses the difficulty of reliably quantifying uncertainty calibration and recalibration effects because commonly used calibration-error estimators are biased and inconsistent. It introduces proper calibration errors linked to proper scores and corresponding upper bounds, showing theoretically and empirically that these provide more reliable improvement estimates and extend recalibration beyond classification.

  • Problem

    Calibration-error estimators used to assess probabilistic reliability are often biased and inconsistent, making recalibration effects difficult to quantify reliably in sensitive applications.

    Source: §1 · pp. 1–2

  • Method

    The paper develops proper calibration errors related to proper scores and introduces upper bounds that measure calibration changes for injective transformations.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Theoretical and empirical analyses show shortcomings in common estimators, while the proposed approach reliably estimates recalibration improvement and applies beyond classification.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Proper scores provide a foundation for trustworthy recalibration assessment across classification and other probabilistic prediction tasks.

    Source: §1 · p. 2, §5 · p. 9, §6 · p. 10

    In general, proper calibration errors do not have unbiased estimators because the required conditional expectation cannot be estimated directly.

    Source: §4 · p. 7

Abstract

from arXiv · show

With model trustworthiness being crucial for sensitive real-world applications, practitioners are putting more and more focus on improving the uncertainty calibration of deep neural networks. Calibration errors are designed to quantify the reliability of probabilistic predictions but their estimators are usually biased and inconsistent. In this work, we introduce the framework of proper calibration errors, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties. This relationship can be used to reliably quantify the model calibration improvement. We theoretically and empirically demonstrate the shortcomings of commonly used estimators compared to our approach. Due to the wide applicability of proper scores, this gives a natural extension of recalibration beyond classification.

1 Introduction

The paper targets unreliable calibration-error estimates in sensitive, data-limited applications, where common estimators can mismeasure recalibration benefits. It introduces proper calibration errors and reliable upper-bound-based estimation, including applicability beyond classification.

Motivation

  • Common calibration estimators are biased, inconsistent, and highly sensitive to test-set size, undermining reliable assessment of uncertainty and recalibration.This sensitivity can substantially over- or underestimate recalibration effects, especially when fewer test data are available.Source: §1 · pp. 1–2

Contributions

  • The paper introduces proper calibration errors that relate calibration errors to proper scores and provide important estimation guarantees.The framework supports reliable estimation of injective recalibration improvements through the associated proper score.Source: §1 · p. 2

Scope

  • The framework extends reliable recalibration-improvement quantification to non-classification settings through proper scores.Its stated scope therefore includes arbitrary probabilistic predictions rather than classification alone.Source: §1 · p. 2

Findings

  • Common estimators produce recalibration-improvement estimates that are heavily biased and become monotonically worse as test data decrease.This result motivates replacing them with estimators whose behavior is more robust to test-set size.Source: §1 · p. 2

2 Related Work

The related work covers calibration-error definitions, recalibration methods, and proper scores as the conceptual basis for the paper’s framework. It emphasizes limitations of binning- and kernel-based estimators and adopts a general calibration condition for arbitrary predictive distributions.

2.1 Calibration errors

  • Calibration requires the complete prediction f(X) to equal the conditional target distribution P_Y|f(X), with classification as a special case.The paper distinguishes this stricter condition from top-label calibration, which can provide a false sense of security.Source: §2.1 · p. 3
  • The Brier score jointly captures model fit and calibration, whereas Lp calibration errors isolate calibration through a decomposition.Estimating calibration remains difficult because continuous models rarely provide samples for every possible prediction.Source: §2.1 · p. 3
  • Common calibration estimators include ECE, class-wise and top-label errors, kernel methods, KS, MMCE, and KCE, each with distinct estimation or scope limitations.Fixed binning introduces hyperparameter-dependent lower bounds, while some kernel estimators have computational or statistical caveats.Source: §2.1 · p. 4

2.2 Recalibration

  • Recalibration methods transform model outputs after training, commonly using accuracy-preserving approaches such as temperature scaling, ETS, or DIAG.These methods are typically optimized on validation data and aim to improve calibration without changing the predicted top label.Source: §2.2 · p. 4

2.3 Proper scores

  • Proper scores evaluate probabilistic predictions through expected scores, with properness requiring the target distribution to achieve the best expected value.Their associated divergences and entropies support calibration-sharpness decompositions and the proposed calibration-error framework.Source: §2.3 · pp. 4–5

3 Theoretical analysis of calibration errors

The theoretical analysis organizes existing calibration errors, shows that many can disagree with true calibration, and characterizes how ECE estimation worsens with smaller datasets. These results motivate proper calibration errors with robust estimators.

Theoretical framework

  • The paper’s taxonomy relates existing calibration errors and identifies theoretical shortcomings in commonly used approaches.It presents a novel overview of relationships among calibration errors and analyzes their estimation behavior.Source: §3 · p. 5
  • A model can have zero values for widely used errors such as ECE, KS, MMCE, or related measures while remaining far from calibrated.By contrast, CE2(f)=0 if and only if the model is calibrated.Source: §3 · pp. 5–6

ECE estimation

  • The ECE estimator is approximately strictly convex and monotonically decreasing in data size, with larger changes at smaller sample sizes.The dependence is stronger when the ground-truth ECE is low, and analogous behavior holds for several binning-based estimators.Source: §3 · p. 6

Empirical confirmation

  • Simulations with known calibration levels produce curves consistent with the theoretical ECE analysis and show that better calibration exacerbates ECE bias as data size changes.These results explain why ECE-based calibration-improvement quantification can be unreliable.Source: §3 · p. 6, Figure 2 · p. 6

Motivation

  • The analysis motivates proper calibration errors that are zero exactly at calibration and are paired with robust estimation strategies.This provides the basis for the framework developed in the subsequent section.Source: §3 · p. 6

4 Proper calibration errors

The paper defines proper calibration errors through divergences associated with proper scores, yielding calibration measures that extend beyond classification. Their associated upper bounds provide reliable estimation and comparison of recalibration improvements, while common estimators can be biased by finite data.

4 Proper calibration errors

  • Proper calibration errors are defined through divergences associated with proper scores, and strictly proper versions equal zero exactly when the model is calibrated.Because proper scores apply to arbitrary distributions, the definition naturally extends calibration errors beyond classification.Source: §4 · pp. 6–7
  • For every proper calibration error with a finite entropy infimum, the framework constructs an associated calibration upper bound rather than a potentially misleading lower bound.The construction is motivated by the lack of unbiased estimators for the calibration error itself.Source: §4 · p. 7
  • The upper bound has a minimum-variance unbiased estimator and approaches the true calibration error as classification accuracy approaches one.This provides a theoretically reliable quantity for calibration assessment under the stated conditions.Source: §4 · p. 7
  • For the Brier score, the Brier score itself is an upper bound of its induced calibration error, while its square root yields the root calibration upper bound RBS.The Brier-score entropy has infimum zero, making the upper-bound identity direct.Source: §4 · p. 7
  • For injective recalibration functions, comparing upper bounds is equivalent to comparing the corresponding proper calibration errors, enabling reliable improvement estimation and calibration-gradient comparisons.Injectivity excludes degenerate finite-output solutions while preserving model sharpness and some output-space continuity.Source: §4 · pp. 7–8
  • Figure 3 indicates that the proposed upper bound is stable with test-set size, whereas lower-bound estimators exhibit systematic size dependence.The figure caption identifies the square root of the Brier score as an upper bound of CE2 and the other errors as lower bounds.Source: Figure 3 · p. 8, §4 · p. 8

5 Experiments

Experiments evaluate calibration-estimator robustness across test-set sizes, recalibration improvements, and probabilistic regression. The proposed upper-bound framework is stable across classification settings and improves the alignment and interpretability of predictive variance in regression.

Experimental setup

  • The evaluation compares RBS with binned, kernel-density, and other calibration estimators across multiple model-dataset combinations, including CIFAR10, CIFAR100, and ImageNet.The study includes Wide ResNet and DenseNet models, plus TS, ETS, and DIAG injective recalibration methods.Source: §5 · pp. 8–9

Robustness of calibration errors to test set size

  • Across DenseNet 40, Wide ResNet 32, ResNet 110 SD, ResNet 110, and LeNet 5 trained on CIFAR10 without recalibration, the proposed upper bound and KS are stable across test-set sizes.CWCE2 with 100 bins is constantly the highest estimated lower bound but remains sensitive to test-set size; Appendix G reports similar results across further settings.Source: §5 · p. 9

Quantifying recalibration improvement

  • RBS is the only estimator robustly quantifying improvement from injective recalibration across settings; other estimators overestimate gains with large test sets and can report negative gains with small ones.The authors therefore characterize existing estimators as unreliable for evaluating recalibration methods.Source: §5 · p. 9

Variance regression calibration

  • In variance regression, DSS communicates recalibration-driven changes in predicted variance, whereas SKCE behaves erratically and seemingly ignores the variance trend.The experiment uses a mixture density network on Residential Building data and Platt scaling across training iterations.Source: §5 · p. 9, Figure 4 · p. 10
  • After calibration with a proper calibration error, average predicted variance corresponds to mean squared error across training iterations, unlike before recalibration.Table 1 compares MSE and average predicted variance before and after recalibration.Source: §5 · p. 9, Table 1 · p. 10

6 Conclusion

The paper develops proper calibration errors and upper bounds to quantify recalibration reliably across classification and other probabilistic tasks. It finds that common estimators can mislead under changing data sizes, whereas the proposed upper bound robustly estimates improvement and extends to variance regression.

6 Conclusion

  • Common calibration estimators can give highly misleading improvement estimates as test-set size changes, while the proposed upper bound remains robust for injective recalibration.The approach also succeeds in additional variance-regression experiments.Source: §6 · p. 10
  • The framework relates calibration errors to proper scores and introduces upper bounds that directly measure calibration change for injective transformations.Source: §6 · p. 10
  • Recalibration improves instance-level prediction of squared error relative to predicted variance across training iterations.Source: Table 2 · p. 10

Checklist

The appendix supplies reproducibility, notation, calibration-error definitions, and score-related helpers supporting the paper’s theoretical and experimental claims.

Checklist

  • The appendix reports code, training details, error bars, compute information, asset provenance, and theoretical assumptions or proofs across the checklist.It states that full code is in the supplement and that experiments were repeated until error bars were barely visible, except for Figure 4.Source: Checklist · p. 14

A Overview

  • The overview organizes the appendix around notation, rigorous calibration-error definitions, proofs, recalibration examples, proper U-scores, and additional experiments.Source: A Overview · p. 16

B Notation

  • The notation defines categorical targets, probability vectors, and C as the top-label prediction obtained from the largest model output.Source: B Notation · p. 16

C Definitions

  • The definitions harmonize calibration errors from prior literature and introduce expected scores, associated divergences, and generalized entropy for proper-score analysis.Source: C Definitions · pp. 16–17

D.1 Helpers

  • The helper lemma establishes the score-related identity used in subsequent proofs under the existence of the conditional expected score.Source: §D.1 · p. 17

D.2 Theorem 3.1

The theorem characterizes relationships among calibration errors and shows that several common errors can vanish without full calibration. A construction then demonstrates the insufficiency of common errors for detecting miscalibration.

D.2 Theorem 3.1

  • Brier-score zero, proper calibration-error zero, kernel calibration-error zero, and model calibration are equivalent under the stated result.Source: §D.2 · pp. 17–18
  • Top-label calibration error zero is equivalent to maximum mean calibration error zero and kernel-smirnov calibration error zero, which imply ECE zero.Source: §D.2 · pp. 17–18
  • The theorem establishes norm and estimator relationships, including TCEp(f) ≥ TCE1(f) ≥ KS(f) and TCE1(f) ≥ c · MMCE(f), under stated conditions.Source: §D.2 · pp. 17, 18, 21, 22
  • The appendix corrects an earlier false implication from CWCEp(f)=0 to TCEp(f)=0 and notes a contradiction with a prior theorem.Source: §D.2 · pp. 19–20

D.3 Proposition 3.2

  • For α > 0.5 and surjective models, there exist distributions where MMCE, KS, ECE, TCEp, and CWCEp all vanish while calibration can still fail.Source: §D.3 · pp. 23–24

D.4 Proposition D.2

The appendix constructs an accuracy-preserving recalibration that fools common errors, analyzes ECE’s dependence on data size, and develops proper-score identities for calibration bounds.

D.4 Proposition D.2

  • An accuracy-preserving transformation can achieve perfect ECE and related common-error evaluations while remaining uncalibrated for the full model prediction.Source: §D.4 · pp. 25–26
  • The constructed transformation exposes why common calibration errors can be insufficient when the whole predictive distribution, rather than only the top label, must be calibrated.Source: §D.4 · p. 26

D.5 Proposition 3.3

  • ECE estimates become increasingly sensitive to data size for smaller samples, while this sensitivity diminishes with larger samples.Source: §D.5 · p. 29
  • ECE sensitivity to data size also depends on calibration quality, with better calibration producing worse sensitivity according to the stated analysis.Source: §D.5 · p. 30

D.6 Lemma 4.1

  • The score framework defines generalized entropy through expected proper scores and uses finite-integrability assumptions for the lemma’s expectation manipulations.Source: §D.6 · pp. 30–31

D.7 Theorem 4.3

Theorem 4.3 relates proper calibration errors to proper-score upper bounds, with asymptotic equivalence under classification assumptions. The framework also supports recalibration analyses, including accuracy-preserving binary transformations.

D.7 Theorem 4.3

  • Every proper calibration error with a finite infimum over the score-induced entropy has an associated calibration upper bound U_S(f).Under classification and mild conditions, this upper bound is asymptotically equal to the calibration error as model accuracy approaches one.Source: §D.7 · p. 31

D.8 Proposition 4.5

  • The framework uses proper-score structure to connect calibration improvement with score-based estimation, while accuracy preservation remains a condition for practical recalibration.The supplied passages establish the score comparison and binary construction, but do not provide additional numerical theorem results.Source: §D.7 · p. 31, §D.8 · p. 33, §E.1.1 · p. 34
  • For injective recalibration, comparing upper bounds is equivalent to comparing the corresponding proper calibration errors.The proposition states U_S(h ◦ f) > U_S(h′ ◦ f) exactly when CE_S(h ◦ f) > CE_S(h′ ◦ f).Source: §D.8 · pp. 32–33

E.1.1 Binary case (scalar output)

  • Binary recalibration maps predictions below and above 0.5 to their respective conditional event probabilities, producing a calibrated transformed model.When the conditional probability below 0.5 is smaller than that above 0.5, the transformation preserves accuracy while reducing sharpness.Source: §E.1.1 · pp. 33–34

E.1.2 Multi-class case (vector output)

The multi-class construction maps predictions to their top-label and recalibrates conditional on that label, yielding calibrated predictions while preserving accuracy under a stated condition. The surrounding proper U-score framework extends calibration analysis beyond ordinary single-observation scores.

E.1.2 Multi-class case (vector output)

  • The top-label recalibration transform is calibrated by conditioning on the model’s predicted class.The construction defines the transform through P(Y | r ◦ f(X) = r(p)) and concludes that t_f ◦ f is calibrated.Source: §E.1.2 · p. 34
  • Top-label accuracy is preserved when the most likely outcome for each predicted label is that same label.Under this condition, the transformed and original top-label predictions agree; otherwise recalibration may improve accuracy.Source: §E.1.2 · p. 34
  • Deterministic transformations cannot increase sharpness under a proper score.The proposition states SHARPS(f) ≥ SHARPS(m ◦ f), and the proof relies on the conditions for Fubini’s theorem.Source: §E.1.2 · p. 34
  • For the log score, sharpness is mutual information, so deterministic post-processing cannot add information about the target.The paper interprets sharpness as generalized mutual information in this case.Source: §E.1.2 · p. 35

F Proper U-scores

  • Proper U-scores generalize proper scores using U-statistics, whose estimators are unbiased minimum-variance estimators.The background defines U-statistics for r-sized subsets of iid observations and motivates the extension used later.Source: F Proper U-scores · p. 35, §F.1 · pp. 35–36

F.2 Contributions

The paper defines proper U-scores and extends proper calibration errors to multi-observation scoring rules, enabling unbiased treatment of quantities such as squared KCE. Extended experiments examine estimator stability, recalibration improvement, and regression settings.

F.2 Contributions

  • Proper U-scores extend proper scoring rules to scores that consume multiple predictions and events, with r = 1 recovering ordinary scoring rules.Their U-statistics provide unbiased minimum-variance estimators, and strict propriety characterizes equality with the target distributions.Source: §F.2 · p. 36
  • Every proper U-score induces a proper calibration error, which is zero exactly when the model is calibrated for strictly proper U-scores.The paper cautions that properties established for r = 1 need not automatically extend to r ∈ N.Source: §F.2 · p. 37
  • The squared KCE is proper but not strictly proper because it measures calibration while ignoring sharpness.For predictions sampled from the model, its associated divergence is minimized by any calibrated prediction distribution, not uniquely by the target distribution.Source: §F.2 · p. 37

G.1 Details on the ECE estimator simulation in Figure 2

  • The extended classification experiments use realistic 100-class predictions and calibrated or temperature-scaled targets to vary calibration quality.The simulation targets approximately 87.6% accuracy and constructs perfect, mediocre, and bad calibration conditions.Source: §G.1 · p. 38

G.2 Details on experimental setup of Section 5

  • The experiments span multiple architectures and CIFAR10, CIFAR100, and ImageNet, using validation and predefined test splits with TS, ETS, and DIAG recalibration.Test-set-size experiments use repeated sampling and averaging to reduce variance at small sample sizes.Source: §G.2 · p. 38

G.3 Estimated model calibration

  • The proposed upper-bound estimator remains stable across many model-dataset settings, while the figure compares it with lower-bound estimators as test-set size changes.Figure 5 uses the square root of the Brier score as the upper bound and identifies the other estimators as lower bounds.Source: §G.3 · p. 39, Figure 5 · p. 39

G.4 Recalibration improvement

  • Only RBS reliably quantifies recalibration improvement, and in squared space only the Brier score gives provably unbiased estimates for CE2 improvement.Other estimators are sensitive to test-set size or substantially underestimate the true improvement; KDE bandwidth failures exclude some CIFAR100 cases.Source: §G.4 · p. 40

G.5 Variance regression

  • The regression experiments recalibrate variance outputs through Platt scaling and optimize the DSS score, which is treated as a calibration error for injective recalibration.Each experiment is repeated with five seeds, and DSS is used as a variance-regression extension of MSE.Source: §G.5 · p. 40

G.5.1 UCI dataset Residential Building

The Residential Building and Friedman 1 experiments evaluate calibration estimates under changing test-set sizes and during regression-model training. They show that proper-score-based measures more reliably capture recalibration improvements and variance-calibration changes than common estimators.

G.5.1 UCI dataset Residential Building

  • Aggregating results from different runs with distinct seeds produces evaluations that repeat the findings in the main paper.Source: §G.5.1 · p. 41
  • Figure 7 shows that the Brier score upper-bounds CE2, while the proposed approach estimates injective-recalibration improvements with less bias.The approach captures the true CE2 improvement in an unbiased manner.Source: Figure 7 · p. 42, §G.5.1 · p. 42, §G.5.2 · p. 42

G.5.2 Friedman 1

  • The modified Friedman 1 problem introduces variance depending on X6, making predictive-variance calibration non-trivial.Noise follows ϵ ∼ N(0, 0.5 + X6), with training, validation, and test sets of 100 examples each.Source: §G.5.2 · p. 42
  • Recalibration adjusts overfitted predicted variance during Friedman 1 training and improves uncertainty communication without changing the predicted mean.DSS tracks the variance trend, whereas SKCE is less sensitive to variance calibration and more sensitive to mean calibration.Source: §G.5.2 · p. 42, Figure 9 · p. 44
  • Figure 8 shows DSS communicating recalibration-induced variance changes reasonably, while SKCE behaves erratically and fails to capture the variance trend.As model fit improves, recalibration requires less adjustment to predicted variance.Source: Figure 8 · p. 43

Related papers

Loading 2203.07835v4…