A Consistent and Differentiable Lp Canonical Calibration Error Estimator

• Canonical calibration requires the entire probability vector to be calibrated, whereas top-label and marginal calibration impose weaker constraints.The curse of dimensionality makes canonical calibration difficult to estimate reliably.Source: §1 · p. 2
• ECEKDE uses Beta kernels for binary classification and Dirichlet kernels for multiclass probability simplices.The estimator is designed to capture canonical calibration while scaling to higher dimensions.Source: §1 · p. 2
• The proposed estimator is tractable, consistent, and differentiable for expected Lp canonical calibration error.Source: §1 · p. 2
• The contribution includes second-order debiasing and empirical evaluation for calibration-error estimation and calibration-regularized risk minimization.The paper evaluates the method across multiple network architectures and datasets.Source: §1 · p. 2

• Post-hoc methods such as temperature scaling, binning, isotonic regression, and Dirichlet calibration recalibrate predictions after model training.Temperature scaling is easy to implement and accuracy-preserving but can clamp high confidence scores of correct predictions.Source: §2 · p. 2
• Trainable calibration methods incorporate differentiable surrogates or regularizers, but some suppress correct high-confidence predictions or lack a clear calibration-error quantity.Differentiable binned estimators also retain dependence on the binning scheme and have unclear extension to full probability-vector calibration.Source: §2 · p. 3
• Mix-n-Match is consistent but requires numerical integration that is infeasible in high dimensions and was implemented only for binary calibration.Source: §2 · p. 3
• The paper presents its estimator as the first consistent, differentiable method successfully applied to Lp canonical calibration error in the multiclass setting.Source: §2 · p. 3

Kernel density estimation

• The framework estimates conditional expectations on the probability simplex with kernel density estimates, using Beta kernels for binary classification and Dirichlet kernels for multiclass calibration.The estimator is designed to be both consistent and differentiable for calibration-error regularized risk minimization.Source: §3 · pp. 3–4

Debiasing

• The conditional expectation is estimated through a ratio of unbiased estimators of a squared mean, represented as a ratio of second-order U-statistics.The ratio construction provides the basis for estimating the squared conditional expectation used in calibration error.Source: §3 · p. 5
• A debiasing scheme achieves O(n^-2) bias for p ∈ {1, 2}, improving the estimator’s bias convergence.The schemes apply to the ratios in Equations (7) and (3).Source: §3 · pp. 5–6

Computational properties

• O(n^2) computational cost and O(n^-1/2) convergence make the estimator tractable for mini-batch training, with empirical convergence rates close to the theoretical value.The O(n^2) cost is measured with respect to mini-batch size when used in gradient descent.Source: §3 · pp. 5–6

Multiclass calibration

• Canonical calibration requires calibrating the whole probability vector, and the proposed multiclass estimator targets this strongest calibration notion.The classifier is perfectly calibrated when CEp(f) = 0.Source: §3 · pp. 3, 5

Consistency

• Under Lipschitz continuity of pf(x)(f(x)) over the simplex interior, the Dirichlet-kernel calibration-error estimator is consistent.Consistency follows from uniform convergence of Dirichlet kernel density estimates and averaging consistent summands.Source: §3 · pp. 5–6

Estimator motivation

• The evaluated calibration estimators are motivated by ECEbin’s sensitivity to binning, exponential dimensionality burden, and asymptotic inconsistency.These limitations make canonical calibration error difficult to estimate with histogram-based methods.Source: §4 · p. 6

Simplex comparison

• ECEKDE better approximates finite-sample behavior than the simplex extension of ECEbin and models concentration near low-dimensional simplex faces.The comparison uses an extension of the binned estimator to the probability simplex.Source: Figure 1 · p. 6, §4 · p. 6

Synthetic experiments

• ECEKDE converges to the ground-truth calibration error across class counts and sample sizes, whereas ECEbin remains poor even with 20,000 points.The ground truth is obtained by integration on synthetic data with 4 and 8 classes.Source: §4 · p. 6, Figure 2 · p. 7

Sharpness debiasing

• Sharpness estimation is evaluated with partially and fully debiased ratios across sample sizes, using synthetic data and repeated trials.The experiment varies sample sizes from 32 to 16,384, averages over 10,000 repetitions, and fixes the KDE location at f(x_j) = 0.17.Source: §4 · p. 7

5 Calibration regularized training

• The bandwidth b controls KDE smoothness, with small values causing undersmoothing and large values causing oversmoothing; experiments select b by leave-one-out likelihood.The regularization weight λ is selected from values that provide a good accuracy–calibration trade-off, while p reflects the desired Lp error.Source: §5 · p. 8, Figure 3 · p. 8

5.1 Experiments

• KDE-XE consistently achieves competitive canonical calibration errors while boosting accuracy across the evaluated architectures.Tables 2 and 3 report canonical L1 ECEKDE and accuracy for multiple loss functions and architectures.Source: §5.1 · p. 8, Table 2 · p. 9, Table 3 · p. 9
• Temperature scaling does not improve canonical calibration error, despite improving top-label calibration in prior work.The observation is consistent with Kull et al. [2019], who reported larger class-wise calibration error after temperature scaling than top-label error.Source: §5.1 · p. 8
• KDE-XE Pareto dominates competitors across datasets and architectures for several λ values, reducing DermaMNIST DenseNet ECEKDE from 66% to 45% at λ = 0.2.Figure 4 compares accuracy and L1 ECEKDE across regularization strengths.Source: §5.1 · p. 8, Figure 4 · p. 9
• KDE-XE adds only an insignificant impact on training speed compared with unregularized XE.Table 4 reports per-epoch training times for four architectures on CIFAR-10.Source: §5.1 · p. 9, Table 4 · p. 9
• The estimator’s O(n^2) complexity is measured with respect to mini-batch size during regularized training, and the added cost is less than a couple percent.The paper notes that O(n^2) complexity is also present in related calibration estimators and calls for larger-scale benchmarking.Source: §5.1 · pp. 9–10

• O(n^2) complexity, O(n^-1/2) convergence, and bias reduced from O(n^-1) to O(n^-2) through debiasing establish favorable estimator properties.The estimator can therefore be applied to small data subsets.Source: §6 · p. 10
• The ECEKDE can be optimized alongside any loss function within an existing batch stochastic gradient descent framework.Source: §6 · p. 10
• Canonical calibration evaluates whether the entire probability vector is calibrated, and the estimator is presented as a tractable metric for this strongest multiclass notion.The paper emphasizes its relevance to safety-critical applications using predicted probabilities for downstream decisions.Source: §6 · p. 10
• Across neural architectures and datasets, the estimator was competitive in accuracy and calibration error while offering superior consistency for canonical calibration error.Source: §6 · p. 10

Ethical statement

• Medical-image benchmark results may be misinterpreted with respect to clinical applicability, creating the main stated ethical boundary.The paper cites Varoquaux and Cheplygina [2022] when identifying this risk.Source: Ethical statement · p. 10
• The method addresses canonical calibration error in multiclass settings, a use case not typically covered by existing deployed calibration methods.The paper frames this scope as distinct from common calibration-error estimation practice.Source: Ethical statement · p. 10

Checklist

• The paper reports that it describes its contributions and scope, discusses limitations, and addresses potential negative societal impacts through an ethics statement.These checklist responses are marked affirmative.Source: Checklist · p. 14
• The authors report complete theoretical assumptions and proofs, plus reproducibility materials containing code, data, and experiment instructions.The reproducibility materials are stated to be in the supplementary material.Source: Checklist · p. 14
• Error bars were reported, with Figure 4 cited in connection with this reporting; total compute and resource-use reporting were addressed as a separate criterion in the cited checklist.Source: Checklist · p. 14
• The paper cites creators of existing assets but does not report their licenses, and states that it does not release the data.The checklist also reports no crowdsourcing or human-subject research and notes medical-data compliance with the Declaration of Helsinki.Source: Checklist · p. 14

A.1 Derivation of the MSE decomposition

• The MSE decomposition reduces the residual term to (1 − E[y | f(x)])E[y | f(x)] ≥ 0 for binary y, identifying an expectation over Bernoulli variances.The derivation conditions on f(x) and applies the law of total expectation.Source: §A.1 · p. 15

B Bias of ratio of U-statistics

• The ratio of two U-statistics is generally biased despite unbiased numerator and denominator estimators, and its correction is obtained by expanding a geometric series.The appendix states a convergence result for the ratio bias under estimability and uses the expansion to analyze its expectation.Source: B Bias of ratio of U-statistics · pp. 16–17

C De-biasing of ratios of straight averages

• For ratios of straight sample averages, the geometric expansion is truncated using U-statistic asymptotics, retaining terms needed for an estimator unbiased up to order n^-2.The construction recursively corrects the relevant ratio terms while neglecting sufficiently high-order contributions.Source: C De-biasing of ratios of straight… · p. 18, X Var( ¯X) (64) · p. 20

D De-biasing of ratios of squared means

• The squared-mean ratio is formed from second-order U-statistics for numerator and denominator, then corrected by expanding its biased ratio and organizing covariance and variance contributions.The resulting estimator is stated to be unbiased up to order two, with terms indexed (a)–(f).Source: D De-biasing of ratios of squared means · pp. 20–23

D.1 Term (a) and D.2 Term (b)

• Terms (a) and (b) are corrected analogously by applying the geometric-series method to the remaining biased ratios, while avoiding further recursion for terms already of order O(n^-2).The appendix explicitly treats term (a) through covariance correction and term (b) through an analogous variance correction.Source: §D.1 · p. 24, §D.2 · p. 24

D.3 Term (c)

• Term (c) is approached by representing the squared second-order U-statistic in terms of other U-statistics before deriving its covariance contribution.The supplied derivation introduces this representation as the first step for evaluating term (c).Source: §D.3 · p. 25
• Term (c) neglects lower-order contributions, estimates one surviving term directly, and recursively corrects the final term using equation (96).The resulting correction is rewritten with symmetrized U-statistics and includes covariance and variance components.Source: §D.3 · p. 25

D.4 Term (d)

• Term (d) uses a correction procedure analogous to term (c), so only its resulting variance-based correction is presented.The displayed result contains Var(X) multiplied by a variance-and-moment factor.Source: §D.4 · p. 26

D.5 Term (e)

• Term (e) is reexpressed through several U-statistics, after which all but the last two terms are discarded for a second-order correction.The remaining covariances are rewritten in terms of X and Y, with the recursive contribution taken from equation (96).Source: §D.5 · pp. 26–27

D.6 Term (f)

• Term (f) likewise follows the procedure used for term (e), with only the resulting variance-based correction reported.Its correction is expressed using Var(X) and a variance-and-moment factor.Source: §D.6 · p. 28

E Top-label calibration

• The evaluation uses 15 adaptive-width bins for ECEbin, with 95% confidence intervals obtained from 100 bootstrap samples.Adaptive binning assigns equal numbers of samples to bins; calibration-regularized experiments use biased ECEKDE.Source: E Top-label calibration · p. 28
• KDE-XE consistently reduces calibration error over XE on CIFAR-10 and CIFAR-100 without reducing accuracy.Table 5 reports top-label L1 adaptive-width ECEbin and accuracy across architectures, with the best ECE values highlighted.Source: E Top-label calibration · p. 28, Table 5 · p. 28
• On CIFAR-100 reliability diagrams, KDE-XE lowers overconfident predictions and achieves better average calibration than MMCE with λ = 2 and FL-53.Figure 6 compares ResNet and Wide-ResNet models using 10-bin reliability diagrams.Source: E Top-label calibration · p. 28

F Relationship between ECEbin and ECEKDE

• The comparison selects ECEKDE bandwidths by leave-one-out maximum likelihood and ECEbin bin counts with Doane’s formula.These procedures are used for all three calibration notions.Source: F Relationship between ECEbin and ECEKDE · p. 28, Figure 8 · p. 30
• In a three-class CIFAR-10 example, probability points concentrate near histogram edges, while the corresponding Dirichlet KDE represents the distribution over the simplex.Figure 7 illustrates the binned estimator and the associated Dirichlet KDE surface.Source: F Relationship between ECEbin and ECEKDE · p. 28, Figure 7 · p. 29
• ECEKDE estimates closely correspond to histogram-based ECEbin approximations for canonical, marginal, and top-label calibration.The comparison uses ResNet-56 models trained on three-class CIFAR-10 subsets with varying numbers of points for ECEKDE estimation.Source: F Relationship between ECEbin and ECEKDE · p. 29, Figure 8 · p. 30
• The plotted comparisons vary the ECEKDE sample size across 100, 500, 1000, and all available points, with 13 bins per class in the relationship analysis.Typical bandwidths are 0.001 for 100 points and 0.0001 otherwise.Source: Figure 8 · p. 30

G Bias and convergence rates

• ECEKDE and ECEbin estimates are comparable and both provide reasonable estimates in the three-class setting.Figure 9 varies the number of estimation points and evaluates against a ground truth computed from 3000 test points with ECEKDE.Source: G Bias and convergence rates · p. 30, Figure 9 · p. 31
• The empirical convergence rate is close to the theoretically expected -0.5 for both ECEKDE and ECEbin.Figure 10 averages absolute estimation errors over 120 ResNet-56 models trained on three-class CIFAR-10 subsets.Source: G Bias and convergence rates · p. 30, Figure 10 · p. 31
• Both estimators are biased and have variance, while their combined absolute-error behavior remains in the same order of magnitude.The ground truth uses 3000 test-set probability scores and ECEKDE.Source: G Bias and convergence rates · p. 30, Figure 10 · p. 31