PaperlayerView exact revision on arXiv

Source-linked AI summary

A Ranking Approach for Measuring Calibration

Anirban Chatterjee, Rina Foygel Barber

arXiv:2609.13100v1stat.MEcs.LGstat.ML

TL;DR

Estimating ℓ2-ECE accurately from finite samples requires assumptions, and assumption-free uniform consistency is impossible. The paper introduces rankECE, which compares observations with neighboring predicted probabilities; theoretical and empirical results show it is a better proxy for ℓ2-ECE than binned approximations.

  • Problem

    Accurately estimating ℓ2-ECE from finite samples requires assumptions, and no estimator can be uniformly consistent without them.

    Source: §1.1 · p. 2

  • Method

    RankECE is a tuning-free, rank-based measure that compares observations associated with neighboring predicted probabilities.

    Source: §1.2 · p. 3

  • Results

    RankECE provides a better proxy for ℓ2-ECE than binned approximations, as supported by theoretical and empirical results.

    Source: §6 · p. 18

  • Takeaways & Limitations

    RankECE also offers a computationally inexpensive test for perfect calibration, with empirical power comparable to or higher than existing SKCE methods.

    Source: §6 · p. 18

    The finite-sample guarantee requires bounded total variation of the miscalibration error, and rankECE's construction relies on one-dimensional predictions in [0, 1].

    Source: §6 · p. 18

Abstract

from arXiv · show

When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that $Y=1$) exactly matches the forecasted probability $f(X)$. In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability $f(X)$. Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.

1 Introduction

Probabilistic forecasts inform downstream decisions, but miscalibration can undermine their reliability. The paper proposes rankECE as a rank-based alternative and reports that it approximates ℓ2-ECE more closely than binned approximations.

1 Introduction

  • Calibration makes forecast probabilities interpretable as uncertainty for downstream decisions, yet fitted models can be miscalibrated through misspecification or overfitting.The paper motivates principled calibration measures because reliable decisions depend on well-calibrated probabilities.Source: §1 · pp. 1–2

1.1 Background and Related Works

  • Uniformly consistent estimation of ℓ2-ECE is impossible without assumptions, making guaranteed finite-sample assessment of approximate calibration challenging.Nonasymptotic guarantees for accurate ℓ2-ECE estimation must rely on assumptions about the predictor.Source: §1.1 · p. 2

1.1.1 Binned approximations to ECE

  • Binned ℓ2-ECE estimates calibration within bins but can miss miscalibration and is sensitive to bin count, implementation choices, and bias.Nonparametric plug-in estimators have also been explored, but assumption-free nonasymptotic consistent estimation remains impossible.Source: §1.1.1 · p. 3

1.1.2 Alternatives to ECE

  • Alternatives characterize calibration through kernel embeddings, test-function classes, interval cutoffs, or distance to a calibrated predictor [Kumar et al., 2018; Widmann et al., 2019, 2021; Gopalan et al., 2022; Rossellini et al., 2025; Blasiok et al., 2023].The paper relates distance from calibration to weighted calibration error with 1-Lipschitz test functions and to weak calibration [Kakade and Foster, 2008].Source: §1.1.2 · p. 3

1.2 Our Contributions

  • Across a wide range of settings, rankECE consistently approximates ℓ2-ECE more closely than ℓ2-binECE, and rankECE-based tests perform competitively with SKCE-based tests [Widmann et al., 2019].The paper also validates the approximation finding on Amazon and Yelp Polarity sentiment prediction tasks.Source: §1.2 · p. 4

2 A Rank Measure for Calibration

The paper defines rankECE by comparing labels associated with neighboring predicted probabilities in sorted samples. It also defines a population analogue and establishes that the sample estimator can estimate it consistently without assumptions on the predictor.

2 A Rank Measure for Calibration

  • The framework assumes predicted probabilities Z=f(X) are nonatomic to avoid ties; the definition and theory are extended to remove this assumption in Appendix F.Source: §2 · p. 4
  • RankECE's sample approximation uses labels attached to neighboring predictions in sorted order as approximate independent draws conditional on similar predicted probabilities.The construction treats consecutive order statistics as having approximately equal predicted probabilities.Source: §2 · p. 4
  • The population analogue of rankECE is defined by taking expectations of the sample-based quantity.Source: §2 · p. 4
  • The sample estimator concentrates around population rankECE, which can be estimated consistently without assumptions on the predictor.Proposition 2.1 establishes concentration, and the passage states that the result places no assumptions on f.Source: §2 · pp. 4–5

3 Properties of rankECE

Section 3 establishes rankECE’s approximation properties relative to ℓ2-ECE and develops tests for perfect calibration. It also examines when approximation gaps are controlled and how rankECE can characterize calibration exactly.

3.1 rankECE as an approximation to ℓ2-ECE

  • For every finite n, rankECE is a lower bound on ℓ2-ECE; for each fixed predictor, its empirical estimate converges to ℓ2-ECE as n grows.The fixed-predictor convergence result does not provide uniform control across predictors.Source: §3.1 · p. 5
  • Under bounded total variation of the residual, rankECE tightly approximates ℓ2-ECE, with error decaying at O(1/n) in typical smooth or monotone-classifier settings.The paper notes that bounded variation is essential: without it, rankECE can approach zero while ℓ2-ECE stays bounded away from zero.Source: §3.1 · p. 6
  • Without bounded variation, rankECE can be arbitrarily close to zero while ℓ2-ECE remains bounded away from zero.The paper constructs residual functions whose total variation increases with n to demonstrate this boundary.Source: §3.1 · p. 6
  • The sample-based, tuning-free estimator cannot substantially overestimate ℓ2-ECE, and will not substantially underestimate it under a mild total-variation condition.This is the stated accuracy guarantee in light of the impossibility of assumption-free consistent estimation of ℓ2-ECE.Source: §3.1 · p. 6

3.2 Testing for Perfect Calibration

  • For n ≥ 4, rankECE is zero if and only if the predictor satisfies the calibration condition; binECE can be zero even for a miscalibrated predictor.The exact characterization is presented as a desirable property because small measure values then provide meaningful calibration guarantees.Source: §3.2 · p. 7
  • The finite-sample test controls its type I error at α and is consistent under the alternative.The test is based on rankECE.Source: §3.2 · pp. 7–8
  • Under the null, normalized rankECE is asymptotically standard Gaussian, supporting an asymptotically valid, consistent test with exact asymptotic level control.The paper motivates this asymptotic test by noting that concentration-based finite-sample tests can be conservative and lose power.Source: §3.2 · p. 8

4 Comparing rankECE and ℓ2-binECE

Section 4 characterizes the bias and estimation conditions of binned calibration error, then shows that rankECE can better detect and approximate miscalibration, especially when residuals oscillate.

4.1 ℓ2-binECE: properties and estimation

  • ℓ2-binECE is ℓ2-ECE applied to a predictor averaged within prespecified probability bins, but it can be a weaker calibration notion and severely underestimate ℓ2-ECE.Its empirical estimator concentrates around its expectation, but that expectation can retain statistical bias.Source: §4.1 · p. 9
  • For K = o(n), the bias of ℓ2-binECE becomes asymptotically negligible, and this condition is required for its distribution-free estimation.The bound is tight up to constants independent of K and n.Source: §4.1 · p. 10

4.2 rankECE as an alternative to ℓ2-binECE

  • RankECE dominates ℓ2-binECE up to finite-sample correction terms, while both population measures can underestimate ℓ2-ECE but cannot overestimate it.For K = o(n), the binned approximation underestimates ℓ2-ECE nearly at least as much as rankECE.Source: §4.2 · pp. 10–11
  • When the residual function is L-Lipschitz, rankECE’s approximation error converges at O(L/n).The section contrasts this adaptive behavior with binning schemes whose suitable bin count can depend on the typically unknown residual function.Source: §4.2 · pp. 11–12
  • When K ≪ m ≪ n, rankECE detects non-trivial miscalibration while ℓ2-binECE can miss it because too few bins fail to capture residual oscillations.RankECE behaves like data-adaptive binning with resolution O(1/n), without requiring prior knowledge of the residual structure.Source: §4.2 · p. 12

5 Experiments

Experiments compare rankECE and binned ECE as approximations to ℓ2-ECE in simulations and sentiment models, and evaluate rankECE-based calibration tests against SKCE-based tests. RankECE more closely tracks ℓ2-ECE across the approximation studies, while its tests offer competitive power and faster computation.

5.1 rankECE and ℓ2-binECE as approximations to ℓ2-ECE

  • The evaluation compares ratios to ℓ2-ECE over 100 repetitions, using ℓ2-binECE with K ∈ {10, √n, n^1/3, n/20}.The simulation reference is evaluated numerically on a fine uniform grid.Source: §5.1 · p. 13, §5.1.1 · p. 14
  • Across simulations and four pretrained sentiment models, rankECE more closely approximates ℓ2-ECE than ℓ2-binECE, including at moderate sample sizes.The comparison is shown for the simulated conditional-probability functions and sentiment models.Source: Figure 3 · p. 14, §5.1.1 · p. 14, Figure 4 · p. 15

5.1.1 Simulations

  • As simulated conditional probabilities become more oscillatory, fixed K = 10 can fail, K = n/20 has persistent bias, and rankECE continues to closely track ℓ2-ECE.K = √n and K = n^1/3 eventually approximate ℓ2-ECE, but require larger samples as function complexity increases.Source: §5.1.1 · p. 14, Figure 3 · p. 14

5.1.2 Sentiment Prediction

  • Across four sentiment models, rankECE provides a substantially more accurate approximation; ℓ2-binECE’s performance depends on bin count and model, with several choices retaining bias or inconsistency.The results are shown for DistilBERT SST2, BERT SST2, RoBERTa Twitter Sentiment, and BERT Multilingual Stars.Source: Figure 4 · p. 15, §5.1.2 · p. 16

5.2 Testing for Perfect Calibration

  • All tested methods control type-I error; rankECE(A) has power comparable to SKCE-U, while rankECE(F) is comparable to or sometimes exceeds SKCE-L and both rankECE tests are much faster than SKCE procedures.The asymptotic rankECE test is more powerful than the finite-sample version, reflecting the latter’s conservative concentration inequalities.Source: §5.2 · pp. 16–17, Figure 5 · p. 17

6 Discussion

The discussion presents rankECE as a proxy for ℓ2-ECE and a basis for calibration tests. Appendix A adds experiments comparing it with binning-based approximations in simulated settings and sentiment prediction.

6 Discussion

  • RankECE compares neighboring predicted probabilities to proxy ℓ2-ECE, avoiding the bin-count tradeoff that can make binned ECE underestimate or overestimate calibration error.The discussion reports that theory and experiments support rankECE as a better proxy for ℓ2-ECE.Source: §6 · p. 18

A.1 Comparing rankECE and ℓ2-binECE as approxmations to ℓ2-ECE

  • Across varied simulated settings, rankECE more faithfully approximates ℓ2-ECE than ℓ2-binECE.Figure A.2 compares the measures across six conditional probability functions.Source: Figure A.2 · p. 27, §A.1 · p. 27

A.2 Comparing rankECE and ℓ2-binECE in Sentiment Prediction

  • On Yelp sentiment models, ℓ2-binECE depends on bin count and prediction model, whereas rankECE is invariant to prediction-function choice.The experiments report this pattern for four pretrained sentiment models.Source: Figure A.3 · p. 28, §A.2 · p. 28

A.3 Testing for perfect calibration

Additional tests compare rankECE-based and SKCE-based calibration procedures. The paper also discusses rankECE’s finite-sample testability and its decision-theoretic guarantees under restrictions on transformations.

A.3 Testing for perfect calibration

  • Asymptotic rankECE matched SKCE-U power, finite-sample rankECE exceeded SKCE-L, and both rankECE tests ran significantly faster than SKCE methods, by orders of magnitude versus SKCE-U.The comparison covers additional model-misspecification settings; SKCE-U uses a multiplier bootstrap to determine its test threshold.Source: Figure A.4 · p. 29, §A.3 · p. 29, Figure A.6 · p. 30

B.1 Testability of rankECE

  • A concentration result makes rankECE testable from finite samples; a holdout test can retain a trained model when its rankECE is small or replace it with a constant model.The paper describes this as a procedure that can return a model guaranteed with high probability to have low miscalibration under the given measure.Source: §B.1 · p. 30

B.2 Actionability of rankECE

  • For M-piecewise monotone transformations, the plug-in rule’s excess risk is controlled by rankECE; analogous guarantees hold for ℓ2-ECE without transformation restrictions and for bounded proper scoring rules.The stated rankECE guarantee uses the decision loss defined in (B.1).Source: §B.2 · p. 31

B.3 Proof of Theorem B.1

The proof expresses the risk difference through a weighted residual and bounds it using a partition on which the weight is constant. Piecewise monotonicity limits the number of intervals needed.

B.3 Proof of Theorem B.1

  • The proof rewrites the risk difference using a threshold-crossing weight and the residual, then applies a partition-based ℓ2-binECE bound to complete the rankECE guarantee.The partition is used to apply Cauchy–Schwarz and repeat the earlier approximation argument.Source: §B.3 · pp. 31–32

B.3.1 Proof of Lemma B.1

  • For an M-piecewise monotone transformation, the threshold-crossing weight is constant on a partition of [0, 1] with at most 2(M + 1) intervals.The bound follows by limiting changes within monotone segments and refining at the decision threshold.Source: §B.3 · p. 32, §B.3.1 · pp. 32–33

C An example comparing rankECE and ℓ2-binECE

In the constructed oscillatory-miscalibration example, rankECE detects miscalibration at frequencies that binning can miss, but its resolution is limited by sample size. The accompanying proofs establish bounds and convergence properties used to explain these comparisons.

C An example comparing rankECE and ℓ2-binECE

C.2 Proof of Lemma C.1

  • The binning analysis partitions the prediction range into arbitrary bins and uses the square wave’s periodicity to bound residual behavior within each bin.The proof breaks bin intervals into whole periods and a remaining interval of length less than 1/m.Source: C An example comparing rankECE and… · p. 33, §C.2 · p. 35

C An example comparing rankECE and ℓ2-binECE

  • When m ≪ n, rankECE detects oscillatory miscalibration; when m ≫ n, it fails, consistent with O(1/n) being the finest resolution available from n observations.For ℓ2-binECE, the number of bins must adapt carefully to the unknown residual oscillations.Source: C An example comparing rankECE and… · p. 34

C.1 Empirical Comparison of ℓ2-binECE and rankECE in Example 4.1

  • As sample size grows, rankECE rapidly approaches ℓ2-ECE at fixed oscillation frequency, while ℓ2-binECE remains worse; with fixed n, rankECE’s approximation degrades as frequency rises.The binning choices K = 10 and K = √n remain consistently poor, while K = n/10 has persistent bias.Source: Figure C.7 · p. 34, §C.1 · p. 35

C.3 Proof of Lemma C.2

  • The rankECE bound uses the residual’s total variation and adjacent ordered observations to control detection as oscillation frequency changes.The proof bounds oscillation within short intervals between order statistics, whose scale depends on the sample size.Source: §C.3 · pp. 36–37

D.1 Proof of Proposition 2.1

  • The concentration proof for rankECE applies McDiarmid’s bounded difference inequality to the statistic formed from the ordered observations.Its proof bounds the change when one observation pair is replaced.Source: §D.1 · p. 37

D.2 Proof of Proposition 3.1

  • The upper-bound proof conditions on ordered predictions and uses the rearrangement inequality to relate rankECE to E[r(Z)²] = ℓ2-ECE(f).The passage identifies the rearrangement inequality as the final step in the upper-bound argument.Source: §D.2 · p. 38

D.3 Proof of Proposition 3.2; D.4 Proof of Proposition 3.3

  • The convergence proof approximates the residual by a continuous function on a compact set, then uses uniform continuity to control nearby ordered predictions.The total-variation proof separately assumes finite residual variation and uses that variation to bound the ordered residual changes.Source: §D.3 · pp. 39–40, §D.4 · p. 41

D.5 Proof of Theorem 3.1

The proof reduces the zero-rankECE characterization to a result for uniform order statistics. It establishes that rankECE is zero exactly when the residual vanishes almost everywhere.

D.5 Proof of Theorem 3.1

  • The proof transforms the prediction distribution to uniform variables and reduces Theorem 3.1 to showing that rankECE is zero exactly when the transformed residual vanishes almost everywhere.The inverse probability transform transfers the result from uniform variables back to the original prediction distribution.Source: §D.5 · pp. 41–42

D.5.1 Proof of Lemma D.2

  • For uniform order statistics, the lemma shows that the expected statistic is zero if and only if its function is zero almost everywhere.The proof uses conditional expectations over intervals and continuity to extend an almost-everywhere conclusion.Source: §D.5 · p. 42, §D.5.1 · pp. 42–43

D.6 Proof of Proposition 3.4

The appendix proves concentration and asymptotic results for rankECE-based procedures. Its arguments use conditional independence, Bernstein bounds, and a martingale central limit theorem.

D.6 Proof of Proposition 3.4

  • The concentration proof decomposes rankECE into two sums of conditionally independent terms and applies Bernstein’s inequality to each.The proof uses the union bound to combine the two bounds.Source: §D.6 · pp. 43–44

D.7 Proof of Theorem 3.2

  • The asymptotic proof identifies scaled rankECE as a martingale terminal value and verifies conditions for applying a martingale central limit theorem.The cross-products of residuals form the martingale difference sequence.Source: §D.7 · pp. 44–46

D.8 Proof of Corollary 3.1

  • The corollary proof combines the null distributional convergence with a law of large numbers and Slutsky’s lemma, then establishes consistency under the alternative.The supplied passages identify these proof steps but do not give the resulting test formula.Source: §D.8 · p. 46

D.9 Proof of Remark 3.2

  • The Remark 3.2 proof constructs an empirical resampling distribution from a calibrated distribution and compares the resulting sampling laws.It samples with replacement from a finite collection of size M, chosen much larger than n.Source: §D.9 · pp. 46–47

E.1 Proof of Proposition 4.1

These proofs analyze the binned calibration estimator and its relationship to ℓ2-ECE. They use bounded differences, conditional variance, binwise decompositions, and assumptions on the residual function.

E.1 Proof of Proposition 4.1

  • The concentration proof bounds how much the binned statistic changes when one sample is replaced, then applies McDiarmid’s inequality.Only bins containing the original or replacement prediction are affected.Source: §E.1 · pp. 48–49

E.2 Proof of Lemma 4.1

  • The binwise expectation proof conditions on each bin’s sample count and bounds the conditional variance of the residual.It uses that Y − Z lies in [−1, 1] to bound that variance.Source: §E.2 · pp. 49–50

E.3 Proof of Theorem 4.1

  • The theorem proof separates consecutive ordered-prediction pairs into within-bin and boundary pairs; there are at most K − 1 boundary pairs.The within-bin terms are handled bin by bin, while the boundary contribution is bounded using the number of non-empty bins.Source: §E.3 · pp. 50–51

E.4 Justification of Remark 4.1

  • Under perfect calibration with uniform Z, every bin has probability 1/K and mean residual zero.These identities are substituted into the expression derived from the binwise variance calculation.Source: §E.4 · p. 52

E.5 Justification of Remark 4.2

  • For equal-width bins and an L-Lipschitz residual, the proof bounds within-bin residual variation using the bin width 1/K.It applies a variance bound to the residual within each bin.Source: §E.5 · p. 52

F On non-atomicity of f(X)

The appendix extends rankECE to predictors with tied scores by randomizing their order, showing that the earlier theoretical properties continue to hold. The added randomness can be averaged out to define a derandomized version.

F On non-atomicity of f(X)

  • Independent uniform variables break ties between equal predictions, producing a unique ordering for the empirical rankECE definition.The ordering sorts predictions first and uses the auxiliary variables when prediction values coincide.Source: F On non-atomicity of f(X) · p. 53
  • For n ≥4, rankECE(f) = 0 if and only if f is perfectly calibrated, even without the nonatomicity assumption.The appendix states that the key earlier results extend to this general setting.Source: F On non-atomicity of f(X) · p. 53
  • Averaging over the tie-breaking randomness defines a derandomized rankECE, avoiding different values for the same observations.The appendix notes that random tie-breaking can yield different rankECE values on identical data.Source: F On non-atomicity of f(X) · pp. 53–54

F.1 Proof of Theorem F.1

  • The proof maps randomized rankings to orderings of independent uniform variables, then follows the earlier proof strategy.It uses the inverse probability transform to construct the uniforms.Source: §F.1 · p. 54, §F.1.1 · pp. 54–55

G Additional Details on Sentiment Models

The evaluated sentiment classifiers are four publicly available pretrained models, all using BERT variants and producing class probabilities from logits. Their tasks and prediction scores differ, particularly for the multilingual ordinal model.

G Additional Details on Sentiment Models

  • Four pretrained sentiment classifiers use softmax-transformed class logits as predicted scores for calibration assessment.The models are DistilBERT SST2, BERT SST2, RoBERTa Twitter Sentiment, and BERT Multilingual Stars.Source: G Additional Details on Sentiment Models · p. 57
  • DistilBERT SST2 and BERT SST2 predict positive versus negative sentiment on SST-2 movie-review sentences.For both, the assessed score is the softmax probability of the positive class.Source: G Additional Details on Sentiment Models · p. 58
  • RoBERTa Twitter Sentiment is pretrained on about 124M tweets and fine-tuned for three-class sentiment analysis.The assessed score is the softmax probability of the positive class.Source: G Additional Details on Sentiment Models · p. 58
  • BERT Multilingual Stars predicts one-to-five-star reviews across six languages, with an ordinal fine-tuning task.Its assessed score sums the softmax probabilities for the four- and five-star classes.Source: G Additional Details on Sentiment Models · p. 58

Related papers

Loading 2609.13100v1…