Source-linked AI summary
Calibration tests in multi-class classification: A unifying framework
David Widmann, Fredrik Lindsten, Dave Zachariah
TL;DR
Common calibration analyses can overlook differences among multi-class probability predictions, while calibration measures lack a meaningful scale. The paper develops a kernel-based framework with consistent, unbiased estimators and tests that make calibration error probabilistically interpretable.
Problem
Method
Results
Asymptotic approximations estimate p-values well on average, and tests based on them, particularly SKCEuq, are powerful in experiments; distribution-free bounds are loose.
Source: §7.2 · p. 8Takeaways & Limitations
Viewing estimators as test statistics gives kernel-based calibration errors probabilistic interpretability through p-value bounds and approximations under the calibrated-model null.
Source: §8 · p. 9Using all continuous functions makes calibration error either zero or infinite, so it cannot compare miscalibrated models.
Source: §4.1 · p. 5
Abstract
from arXiv · showhide
In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class classification that generalize existing measures such as the expected calibration error, the maximum calibration error, and the maximum mean calibration error. We propose and evaluate empirically different consistent and unbiased estimators for a specific class of measures based on matrix-valued kernels. Importantly, these estimators can be interpreted as test statistics associated with well-defined bounds and approximations of the p-value under the null hypothesis that the model is calibrated, significantly improving the interpretability of calibration measures, which otherwise lack any meaningful unit or scale.
1 Introduction
Tumour grading contains irreducible uncertainty, and limited training data can leave additional uncertainty about the model. Because clinical decisions depend on its probabilities, a model should be both accurate and calibrated, yet common calibration research has focused on binary or most-confident predictions.
1 Introduction
- A model should express both irreducible aleatoric uncertainty and epistemic uncertainty about the model, which additional training data can reduce.Finite, expensive-to-annotate pathological-image datasets may leave a learned model far from the ideal model.Source: §1 · p. 1
- Clinical use requires meaningful uncertainty: excessive certainty about high or low tumour grades can lead to harmful treatment decisions.If a pathologist intervenes when uncertainty is high, false but highly certain gradings may also keep the pathologist out of the loop, while overly uncertain gradings may demand costly intervention.Source: §1 · pp. 1–2
- Calibration requires predicted probabilities to match empirical frequencies, a property complementary to accuracy.An overconfident model can be accurate but miscalibrated, while predicting only population class proportions can be calibrated but inaccurate.Source: §1 · p. 2
- Calibration in multi-class settings remains insufficiently addressed: common measures focus on binary classification or the most-confident prediction.The ECE, MCE, and MMCE and reliability diagrams are cited as examples of measures developed for binary classification.Source: §1 · p. 2
- Grouping predictions only by their highest confidence can miss differences in the remaining class probabilities, and generalized ECE has limitations in high-dimensional probability spaces.Its histogram-regression estimator requires partitioning the simplex and is asymptotically inconsistent in many cases.Source: §1 · p. 2
2 Our contribution
The paper develops a multi-class calibration framework that encompasses familiar binary-class measures and studies kernel-based measures with consistent, unbiased estimators. These estimator properties support model comparisons and calibration tests with interpretable p-value bounds and approximations.
2 Our contribution
- The framework covers multi-class calibration and encompasses ECE, MCE, and MMCE, measures commonly used for binary classification.It studies measures based on vector-valued reproducing kernel Hilbert spaces and derives consistent and unbiased estimators for them.Source: §2 · p. 2
- Common calibration estimates are random and may be biased, while calibration measures lack a meaningful unit or scale.The paper notes that comparing models without accounting for estimator randomness can be misleading.Source: §2 · p. 2
- Consistent, unbiased estimators support competing-model comparisons and calibration tests with interpretable bounds and approximations of p-values.The paper evaluates its estimators and tests empirically against existing methods and provides Julia packages for calibration estimation and testing.Source: §2 · p. 3
3 Background
The background establishes the probabilistic classification setting, contrasts common and strong calibration, and introduces matrix-valued kernels. Strong calibration conditions the full class distribution on a prediction; universal kernels later provide a function space capable of identifying it.
3 Background
- For readability and space, the main text summarizes definitions and concepts while proofs are left to supplementary material.The supplementary material is intended as a reference for precise statements and proofs.Source: §3 · p. 3
3.1 Probabilistic setting
- A probabilistic classifier maps an input to a vector of probabilities over the m possible classes.The setting treats X and Y as input features and outputs, respectively.Source: §3.1 · p. 3
3.2.1 Common notion
- The common notion checks whether the most-confident class is correct at the rate indicated by its confidence, without accounting for which class was selected.For a prediction of (0.4, 0.3, 0.3), it guarantees only that predictions with 40% top confidence are correct 40% of the time.Source: §3.2.1 · p. 3
3.2.2 Strong notion
- Strong calibration requires the conditional class distribution given the full prediction to equal that prediction, and is stronger than calibrated marginal probabilities.For prediction (0.4, 0.3, 0.3), the corresponding inputs must have that class distribution.Source: §3.2.2 · p. 3
3.3 Matrix-valued kernels
- Matrix-valued kernels map pairs of probability predictions to matrices and correspond one-to-one with vector-valued RKHSs.Universal kernels yield RKHSs dense in continuous functions; the paper notes that such kernels can identify strongly calibrated models and gives constructions from scalar kernels such as Gaussian or Laplacian kernels.Source: §3.3 · p. 4
4 Unification of calibration measures
This section introduces a general measure of strong calibration and relates it to existing calibration measures. It defines calibration error and a kernel-based version, then shows how ECE and MMCE fit into the framework.
4.1 Calibration error
- CE is defined relative to a non-empty function class F; it vanishes for strongly calibrated models, but the converse need not hold.For the class of continuous functions, CE is zero if and only if the model is strongly calibrated, but then it is either 0 or infinite and cannot compare miscalibrated models.Source: §4.1 · pp. 4–5
4.2 Kernel calibration error
- For a universal matrix-valued kernel, KCE is zero if and only if the model is strongly calibrated.KCE uses the unit ball of the kernel's RKHS as the function class.Source: §4.2 · p. 5
4.3 Expected calibration error
- ECE is zero if and only if the model is strongly calibrated when its distance separates distinct probability distributions; cityblock, total variation, and squared Euclidean ECE are special cases of CE.The distance condition holds for the standard choices listed in the passage.Source: §4.3 · p. 5
4.4 Maximum mean calibration error
- MMCE is a special case of KCE, while its original formulation applies only to binary classification.The paper focuses on calibration evaluation and tests rather than methods for obtaining calibrated models.Source: §4.4 · p. 5
5 Calibration error estimators
The section addresses estimating calibration error from an i.i.d. validation set. It presents three consistent estimators of squared KCE and contrasts them with the standard ECE estimation approach.
5 Calibration error estimators
- Histogram-regression ECE estimators can be inconsistent and biased, and may scale poorly with the number of classes.The calibration function needed by the sample-average ECE estimator is not readily available and must itself be estimated.Source: §5 · p. 6
- Three estimators of squared KCE are consistent, as summarized in Table 1.The estimators target SKCE, the squared kernel calibration error.Source: §5 · p. 6, Table 1 · p. 6
- The unbiased SKCE estimators include quadratic- and linear-complexity variants.The subscripts q and l denote their respective computational complexities.Source: §5 · p. 6
6 Calibration tests
This section treats calibration-error estimators as test statistics under the null hypothesis that a model is calibrated. It derives p-value bounds and approximations, giving otherwise scale-free error estimates a probabilistic interpretation.
6 Calibration tests
- Bounds and approximations of the p-value under the calibrated-model null support hypothesis tests and give calibration-error estimates an interpretable probabilistic meaning.The p-value is the probability, under the null, that the estimator is at least as large as the observed estimate.Source: §6 · p. 6
- Distribution-free bounds require no assumptions about the data distribution or model, but are usually loose.The bounds are derived for the biased estimator and either unbiased estimator under a bounded-kernel condition.Source: §6 · p. 6
- The linear estimator is asymptotically normally distributed by the central limit theorem.The stated asymptotic distribution assumes finite variance of h_i,j.Source: §6 · p. 7
- Under strong calibration, the asymptotic distribution of n times the quadratic unbiased SKCE estimator can be approximated by bootstrapping or Pearson curve fitting.The asymptotic distribution is derived in Theorem G.2.Source: §6 · p. 7
7 Experiments
The experiments examine calibration-error estimators and calibration tests on synthetic data from calibrated and uncalibrated models. The kernel-based estimators show favorable bias and testing behavior, while additional experiments expose class-count effects and a multimodality caveat.
7 Experiments
- The synthetic experiments use 250 labeled predictions for 10 classes, with M1 calibrated and M2 and M3 uncalibrated.The SKCE experiments use a matrix-valued exponential kernel with bandwidth chosen by the median heuristic.Source: §7 · p. 7
7.1 Calibration error estimates
- The experiments confirm the unbiasedness of SKCEul across samples from the three generative models.Figure 1 compares distributions of calibration-error estimates with their means and true errors.Source: §7.1 · p. 7, Figure 1 · p. 8
7.2 Calibration tests
- The tests compare distribution-free bounds and asymptotic p-value approximations for SKCE estimators with ECE consistency resampling on data from three generative models.Each model is evaluated on 104 randomly sampled data sets.Source: §7.2 · p. 7, §7 · p. 7
- Asymptotic SKCE tests estimate p-values well on average and are powerful in the experiments, while consistency-resampling tests for ECE over-reject calibrated M1.Distribution-free bounds are loose and yield tests with little power.Source: §7.2 · p. 8, Figure 2 · p. 9
7.3 Additional experiments
- SKCEul is guaranteed to be biased upwards, while SKCE estimators are not affected by increasing class count in the same way as ECE estimates.SKCEul shows a multimodal distribution in some experiments with 100 and 1000 classes.Source: §7.3 · p. 8
- ECE-based p-value approximations become less reliable as the number of classes increases, especially for uniform-size bins.The decrease is smaller for data-dependent bins.Source: §7.3 · p. 8
8 Conclusion
The conclusion presents a unified framework for calibration error, including kernel-based measures with consistent and unbiased estimators. It also frames those estimators as test statistics with p-value bounds and approximations, while leaving kernel and hyperparameter effects for future study.
8 Conclusion
- The framework covers several existing error measures and introduces kernel-based measures with consistent and unbiased estimators.The conclusion contrasts these properties with the more common ECE.Source: §8 · p. 9
- The effect of kernel choice and hyperparameters on the estimators remains an open question that the paper defers to future research.Source: §8 · p. 9
- Bounds and asymptotic p-value approximations give kernel-based calibration-error estimates a probabilistic interpretation and support reliable calibration tests.The conclusion contrasts these tests with previously proposed ECE consistency-resampling tests.Source: §8 · p. 9
B Probabilistic setting
- Strong calibration means the conditional distribution of the label given the model prediction equals that prediction almost surely.The paper also states this equivalently as the calibration discrepancy being zero almost surely.Source: B Probabilistic setting · p. 12
C Calibration error
The calibration error is defined relative to a chosen function class: strong calibration always implies zero error, but the converse requires a particular class. The appendix also describes vector-valued RKHSs and the role of universal kernels.
C Calibration error
- Strong calibration implies zero calibration error for any chosen function class F.The error depends on the choice of F, and symmetric F ensures a nonnegative error.Source: C Calibration error · p. 13
- Using all continuous functions makes zero calibration error equivalent to strong calibration, but that function class is too large and impractical.For other function classes, zero error does not generally imply strong calibration.Source: C Calibration error · p. 13
D Reproducing kernel Hilbert spaces of vector-valued functions on the probability simplex
- A RKHS with a continuous kernel is a subspace of continuous functions, while a universal kernel's RKHS is dense in the continuous functions.The appendix notes that Gaussian and Laplacian kernels on the probability simplex are universal.Source: D Reproducing kernel Hilbert spaces of… · pp. 15–16
E Kernel calibration error
The kernel calibration error (KCE) uses matrix-valued kernels to assess calibration, and universal continuous kernels make zero KCE equivalent to strong calibration. The estimator section provides unbiased alternatives to the plug-in estimator, including a linear-scaling estimator with higher variance.
E Kernel calibration error
- KCE is defined through a vector-valued RKHS embedding, with its value given by the embedding's RKHS norm.The embedding represents expectations of calibration residuals against functions in the RKHS.Source: E Kernel calibration error · pp. 16–17
- For a universal continuous kernel, KCE is zero if and only if the model is calibrated in the strong sense.The result uses the density of the kernel's RKHS in the continuous functions.Source: E Kernel calibration error · p. 16
F Estimators
- The plug-in estimator for KCE is biased, while removing terms involving the same observations yields an unbiased estimator of SKCE.The unbiased estimator is formed from pairs of distinct observations.Source: F Estimators · p. 18
- An alternative unbiased estimator uses only linearly many samples but has higher variance than the quadratic-scaling estimator.Both estimators target SKCE.Source: F Estimators · p. 18
G Asymptotic distributions
This section derives asymptotic behavior and calibration tests for the proposed estimators. Under strong calibration, the unbiased estimator has degenerate U-statistic behavior, while the linear estimator supports a normal-approximation test.
G Asymptotic distributions
- Under strong calibration, the unbiased SKCE estimator is a degenerate U-statistic with an asymptotic limit represented using standard normal variables and eigenvalues.The theorem's stated assumptions include a finite kernel moment.Source: G Asymptotic distributions · pp. 19–20
- Under the null of strong calibration, thresholds for the unbiased estimator can be estimated by Pearson-curve moment matching or bootstrap quantiles.Both procedures are performed under the assumption that the model is strongly calibrated.Source: G Asymptotic distributions · pp. 20–21
- The linear estimator's asymptotic behavior follows from the central limit theorem and yields a one-sided test using a normal quantile and estimated standard deviation.The stated acceptance region uses the (1−α)-quantile of the standard normal distribution.Source: G Asymptotic distributions · p. 21
H Distribution-free bounds
The section develops distribution-free bounds and calibration tests for kernel-based estimators, including a tighter bound under strong calibration. It also relates calibration error to ECE and MCE special cases and notes that bound tightness depends on the chosen norms.
H Distribution-free bounds
- The tightness of the kernel bound depends on p and q; for k = φI_m, its minimum is 2Φ only when p = q or m = 2 and p > q.For other choices of p and q, the bound is non-optimal in this example.Source: H Distribution-free bounds · pp. 21–22
- Under strong calibration, the distribution-free bound can be improved, and the section derives a corresponding hypothesis test using the plug-in KCE estimator.The result compares two bound terms and gives an acceptance region at significance level α.Source: H Distribution-free bounds · pp. 24–25
- Distribution-free deviation bounds and one-sided tests are also derived for the unbiased quadratic and linear SKCE estimators.The section gives separate results for the U-statistic estimator and the linear estimator.Source: H Distribution-free bounds · pp. 25–26
I.1 Expected calibration error and maximum calibration error
- For bounded-function spaces, calibration error recovers ECE with cityblock, total variation, and squared Euclidean distances, while p = 1 gives the MCE special case.The stated ECE relation expresses calibration error as an Lp′ norm of the calibration residual.Source: §I.1 · p. 26
I.2 Maximum mean calibration error
This section relates MMCE to KCE and evaluates calibration-error estimators and SKCE-based p-value bounds and approximations. The experiments cover generative models with known calibration properties, empirical test error across significance levels, and variations in kernels, distance choices, and model parameters.
I.2 Maximum mean calibration error
- KCE reproduces MMCE’s analysis of common calibration when the model is reduced to its most-confident predictions.The reduction treats the top-class probability and its complement as a two-class model, with correctness determining the label.Source: §I.2 · pp. 26–27
J.1 Calibration errors
- In the experiments, the authors evaluate their proposed SKCE estimators and compare them with two ECE estimators.Source: §J.1 · p. 28
J.1.1 Expected calibration error
- The histogram-regression ECE uses either 10 uniform bins per class or data-dependent splits that leave at least five samples in each bin.The data-dependent scheme iteratively splits on the class predictions with the highest variance.Source: §J.1.1 · p. 28
J.1.2 Kernel calibration error
- For SKCE, the experiments use the matrix-valued kernel exp(−∥x − y∥/ν)I_m and total variation distance, with bandwidth set by the median heuristic.The heuristic uses the median pairwise distance in a validation dataset, which need not be separate.Source: §J.1.2 · p. 28
J.2 Generative models
- The generative setup specifies distributions over predictions and labels, enabling calibrated and uncalibrated models with known calibration errors; it is calibrated exactly when π = 0.Labels are drawn from Cat(β) with probability π and Cat(γ) otherwise, conditional on prediction γ.Source: §J.2 · p. 28
J.2.1 Theoretical expected calibration error
- As the number of classes grows, the two considered ECEs tend to π and π exp(−a)a^(a−1)/Γ(a), respectively.The result is stated for the considered generative models, including the symmetric-α case and the case where β is a one-hot vector.Source: §J.2.1 · p. 29
J.2.2 Mean total variation distance
- Additional experiments set the SKCE kernel bandwidth to the mean total variation distance between independent predictions drawn from the same Dirichlet distribution.This theoretical mean does not depend on the number of available samples, though it is usually unavailable.Source: §J.2.2 · p. 29
J.2.4 Calibration tests: Additional figures
- Figures 51–54 evaluate distribution-free p-value bounds for SKCEuq with the median heuristic across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 51 · p. 51, Figure 52 · p. 51, Figure 53 · p. 52
- Figures 55–58 evaluate distribution-free p-value bounds for SKCEuq with the mean total variation distance across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 55 · p. 52, Figure 56 · p. 53, Figure 57 · p. 53
- Figures 59–62 evaluate distribution-free p-value bounds for SKCEul with the median heuristic across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 59 · p. 54, Figure 60 · p. 54, Figure 61 · p. 54
- Figures 63–66 evaluate distribution-free p-value bounds for SKCEul with the mean total variation distance across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 63 · p. 55, Figure 64 · p. 55, Figure 65 · p. 56
- Figures 67–70 evaluate asymptotic p-value approximations for SKCEuq with the median heuristic across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 67 · p. 56, Figure 68 · p. 57, Figure 69 · p. 57
- Figures 71–74 evaluate asymptotic p-value approximations for SKCEuq with the mean total variation distance across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 71 · p. 58, Figure 72 · p. 58, Figure 73 · p. 58
- Figures 75–78 evaluate asymptotic p-value approximations for SKCEul with the median heuristic across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 75 · p. 59, Figure 76 · p. 59, Figure 77 · p. 60
- Figures 79–82 evaluate asymptotic p-value approximations for SKCEul with the mean total variation distance across four generative-model parameter settings.They use α = (1, . . . , 1) or (0.1, . . . , 0.1), paired with uniform or one-hot β.Source: Figure 79 · p. 60, Figure 80 · p. 61, Figure 81 · p. 61
J.3 Modern neural networks
On CIFAR-10, neural-network calibration estimates gave differing model rankings and unclear meaning, while SKCE p-value approximations varied by estimator. The computational-time evaluation showed that two SKCE estimators could be evaluated quickly even with 1,000 samples and classes.
J.3 Modern neural networks
- Calibration-error estimators, including both ECE estimators, ranked the CIFAR-10 neural networks differently, and their values around 10^-2 and 10^-4 were not readily interpretable.The comparison used pretrained ResNet, DenseNet, VGGNet, GoogLeNet, MobileNet, and Inception models on CIFAR-10 test predictions.Source: §J.3 · p. 62
- All SKCE_uq asymptotic p-value approximations were zero, while SKCE_ul approximations ranged from about 0.18 for ResNet18 to 0.91 for GoogLeNet.Most distribution-free p-value bounds were between 0.99 and 1, which the authors describe as loose.Source: §J.3 · p. 63
- Consistency-resampling p-value approximations were always zero, but the authors caution that this approach may underestimate p-values for calibrated models on average.The caution is based on the paper’s controlled generative-model experiments.Source: §J.3 · p. 63
J.4 Computational time
- The authors argue that fair comparisons of calibration-error estimators should account for estimation error as well as computational time.They compare this to work-precision diagrams for numerical differential equation solvers.Source: §J.4 · p. 64
- For 1,000 samples and 1,000 classes, SKCE_b and SKCE_uq with the median heuristic each took around 0.1 seconds to evaluate.The reported timing depends on the Julia implementation and hardware.Source: §J.4 · pp. 63–64