PaperlayerView exact revision on arXiv

Source-linked AI summary

How Much Can Reliability Drift Under a Fixed Confidence Distribution?

Wenhao Liang, Lin Yue, Wei Emma Zhang, Mingyu Guo, Olaf Maennel, Weitong Chen

arXiv:2609.38917v1cs.AI

TL;DR

The paper asks how much reliability can change when a covariate shift leaves the confidence distribution unchanged. It defines and analyzes a fragility profile, then estimates a finite-readout version with a separate lower confidence bound; on ImageNet, that bound is positive for several classifiers as released and fewer after temperature scaling.

  • Problem

    How much can reliability change under covariate shift when the confidence distribution is unchanged, given that confidence-based summaries cannot register such a shift?

    Source: §1 · p. 1

  • Method

    The paper defines a fragility profile for score-preserving reweightings under a χ² budget and develops a finite-readout estimate with a split-sample lower confidence bound.

    Source: §1 · p. 1, §4 · p. 6

  • Results

    At ρ=0.3, bounds were positive in both split arms for four of six primary and nine of twelve additional classifiers as released, versus none of six primary and three of twelve additional classifiers after temperature scaling.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The variance controls only the local profile, while the full profile identifies the centred within-level propensity law, so calibration residual and grouping variance do not determine fragility in general.

    Source: §3.2 · p. 5

    The theory concerns exact confidence levels and the unobserved propensity, whereas empirical objects use bins and a learned readout and do not isolate level-set heterogeneity.

    Source: §8 · p. 9

Abstract

from arXiv · show

A classifier's conditional accuracy can change while its confidence distribution stays exactly the same. We study the worst-case movement of the reliability relation under covariate shifts that preserve the distribution of the confidence score, constraining the reweighting within each confidence level by a $\chi^2$ budget; the resulting worst case, as a function of the budget, is a fragility profile. On an interval of budgets that can be computed from the source distribution, the profile equals exactly the square root of the budget times the within-level variance of the correctness propensity -- the grouping-loss term of calibration-refinement decompositions. Beyond this interval the profile is governed by the tails of the propensity law, and the entire upward profile determines the centred within-level law; consequently, calibration residual and grouping variance do not determine fragility in general, though they do when labels and predictions are deterministic. Since the propensity is not observed, we restrict reweightings to a learned finite readout within confidence bins, bound the part the restriction misses by the grouping variance remaining inside readout cells, estimate the restricted profile with role-separated labels, and provide a separate split-sample lower confidence bound. On ImageNet this bound is positive in both splits for four of six primary classifiers and nine of twelve additional ones as released, and for three of eighteen after temperature scaling. Held-out drift under optimised reweightings fitted without evaluation labels tracks the estimated profile; an exploratory label-permutation diagnostic yields near-zero agreement for this statistic while largely reproducing the correlation observed for unsigned random reweightings.

1 Introduction

The paper asks how reliability can change under covariate shifts that preserve the confidence distribution, a movement confidence-based summaries cannot register. It defines a fragility profile and connects its local behavior to grouping heterogeneity, while showing that the full profile contains more information than variance alone.

1 Introduction

  • Confidence histograms, coverage curves, and accuracy estimators from target confidences cannot register shifts that change reliability while preserving the confidence distribution.The fragility profile measures the largest reliability movement under score-preserving covariate shifts as a function of a within-level χ² budget.Source: §1 · p. 1
  • On a source-computable budget interval, the fragility profile equals the square root of the budget times within-level grouping heterogeneity.This makes the grouping term of calibration–refinement decompositions an exact local rate of reliability drift.Source: §1 · p. 1
  • Beyond that interval, the upward profile identifies the centred within-level propensity law, so calibration residual and grouping variance do not determine fragility in general.The hierarchy collapses when labels and predictions are deterministic functions of the input.Source: §1 · p. 2
  • The paper restricts reweightings to a learned finite readout within confidence bins, bounds the omitted part, and develops role-separated estimation and a split-sample lower confidence bound.Its empirical study covers eighteen ImageNet classifiers and includes held-out tracking, permutation diagnostics, and natural-shift decomposition.Source: §1 · p. 2

2 Setup

The setup fixes a trained classifier and distinguishes its confidence score from the propensity to be correct at each input. It defines score-preserving shifts as input reweightings that retain confidence-level mass, with within-level heterogeneity determining whether reliability can move.

2 Setup

  • The correctness propensity is the probability of correct prediction given the input, while reliability is its conditional mean at a confidence level.The centred propensity has mean zero within a level, and its variance is the grouping heterogeneity measuring what a score cannot express.Source: §2 · pp. 2–3
  • When labels and predictions are deterministic functions of the input, the propensity is binary and the reliability relation determines the profile.The paper assumes neither deterministic labeling nor prediction; empirical group-level correctness rates need not be binary.Source: §2 · p. 3
  • A score-preserving shift reweights inputs within confidence levels, keeps the score distribution fixed, and is constrained by a conditional Pearson χ² budget.The shift reweights inputs rather than changing the conditional label distribution given the input.Source: §2 · p. 3

3 The fragility profile

The fragility profile measures reliability movement under score-preserving reweighting and is governed locally by within-level variance. At larger budgets its tail behavior matters, and the full upward profile identifies the centred propensity law.

3.1 An exact local law, and what lies beyond it

  • Beyond the variance-controlled regime, the full profile is described by a truncated-affine reweighting and the tails of the centred propensity.The profile is concave and non-decreasing; its optimizer is determined by the thresholded propensity and the χ² budget.Source: §3.1 · pp. 3–4
  • On the closed source-computable interval [0, ρ+], the profile equals √(ρGP(s)) exactly, not merely asymptotically.The interval ends when the affine optimizer would assign negative weight to the hardest inputs; later regimes may be empty.Source: §3.1 · p. 4
  • Reliability can move under a score-preserving shift exactly at levels with positive grouping heterogeneity, while threshold-policy coverage remains fixed as selective risk changes.The profile bounds the movement in each direction through a tail-averaged profile above the threshold.Source: §3.1 · p. 4

3.2 The whole profile identifies the centred law

  • The upward profile identifies the centred within-level propensity law, although the theorem establishes no stability for inverting an estimated curve.The curve and its slope trace the stop-loss transform, which determines the law.Source: §3.2 · p. 4
  • Calibration residual and grouping variance do not determine fragility over general propensity laws, but deterministic labels make the residual sufficient to fix the profile.Two laws with the same mean and variance can agree through ρ≤0.22 yet differ by 0.119 at ρ=3.Source: §3.2 · pp. 4–5

4 Restricted fragility: estimation and a lower confidence bound

Because the correctness propensity is unobserved, the paper studies reweightings expressible through a finite readout within fixed confidence bins. It orders this restricted profile against a binned oracle, estimates it with role-separated labels, and constructs a separate split-sample lower confidence bound.

4 Restricted fragility: estimation and a lower confidence bound

  • For fixed bins, the restricted profile is a lower bound on the binned oracle profile, with approximation error bounded by the square root of budget times residual grouping variance within readout cells.The bound is not an estimate or confidence statement, and nested readouts are ordered while learned readouts and confidence quantile groups are not.Source: §4 · p. 5
  • Role-separated rate labels avoid letting labels choose their own readout cells, but the plug-in profile remains upward biased in expectation and is neither confidence bound.FAUC is a lossy ranking summary; the reported grouping variance receives a binomial-noise correction, whereas FAUC uses uncorrected rates.Source: §4 · p. 6
  • The lower-bound construction uses feasible within-bin transfers between groups, combining simultaneous bounds on group rates and masses across radii.The transfer moves mass from one group to another and converts the rate difference into a reliability movement under a conditional χ² cost.Source: §4 · p. 6
  • A split-sample construction gives a 95% lower confidence bound for the restricted profile of the partition learned on the fitting half, with each split arm targeting its own partition.A positive value establishes nonzero restricted fragility, not a matching movement in overall accuracy; a zero bound does not show fragility is small.Source: §4 · p. 6

5 Experiments

The split-sample lower confidence bound is positive in both arms for four of six primary and nine of twelve additional ImageNet checkpoints as released, but only three of eighteen after temperature scaling. Experiments also test held-out drift, estimation-noise explanations, large-budget profiles, and calibration comparisons.

Held-out drift

  • Optimised reweightings fitted on rate labels yield median held-out drift agreement of 0.62 across 36 checkpoint–state combinations, checking the fitted construction out of sample rather than predicting natural shifts.Agreement is weakest where G_R is smallest.Source: §5 · p. 6

Estimation-noise diagnostics

  • The signed-drift permutation null gives median agreement −0.01, compared with 0.62 observed, while the unsigned random-shift null largely reproduces observed correlations.The signed check was exploratory, ignores cross-fold dependence, and supports no formal p-value.Source: §5 · p. 7

Full profile

  • At ρ=3, the variance law overstates the full profile by a median 44% and the capped version by 10%, while rankings remain nearly identical (FAUC Spearman 0.996).The full profile changes estimated large-budget movement sizes, not model rankings in these comparisons.Source: §5 · p. 7

Known-propensity simulations

  • For a four-point law at ρ=3, mean absolute errors are 0.022 (plug-in) and 0.148 (cap) with 250 rate labels, and 0.006 and 0.102 with 5000.Both estimates are unreliable under weak heterogeneity at n=250, and filtering rare cells can remove the practical advantage.Source: §5 · p. 7

Calibration comparisons

  • After temperature scaling, ECE and FAUC associations reverse across models, while a fixed-partition monotone map lowers median cross-fitted ECE by 75.1% on primary and 76.2% on additional checkpoints.The fixed-partition map leaves the restricted profile, FAUC, and selective-risk fragility unchanged; refitting the readout changes FAUC by up to 27%.Source: §5 · p. 8

Lower confidence bound

  • At ρ=0.3, the lower bound is positive in both split arms for four of six primary and nine of twelve additional checkpoints as released, versus three of eighteen after temperature scaling.Each entry has a 95% guarantee for its own arm; the counts imply no joint coverage.Source: §5 · p. 8

6 Natural shift: what the source profile bounds

For natural benchmarks, binwise drift splits exactly into composition and conditional components, but the source profile bounds only the composition component under a support condition. The conditional component is larger in the reported ImageNet comparisons.

Decomposition and scope

  • Binwise drift decomposes exactly into composition and conditional terms, and the source profile bounds the absolute composition term when target groups have source support.The conditional term is not controlled by the source profile and can reflect changed labeling or unresolved within-cell variation.Source: §6 · p. 8

Results

  • The conditional component is larger across ImageNet-C, ImageNetV2, and ImageNet-Sketch; mean L_comp/L_total is 41.8%, 11.1%, and 9.6%, respectively.On ImageNet-C the ratio falls from 50.6% at severity 1 to 33.0% at severity 5.Source: §6 · p. 8

Scope and estimation

  • The utilization median is 0.64 across 9,177 fold-pooled bin records, but the bound uses plug-in source rates and has no coverage guarantee.The utilization is undefined and excluded when a target group lacks source mass or its denominator vanishes.Source: §6 · p. 8

Associations

  • Across 75 domains, source FAUC is associated with the composition term (mean within-domain Spearman +0.63), while the preregistered outcome is mixed because single-domain benchmarks disagree.The authors interpret this as consistent with the decomposition, not as a validated forecast.Source: §6 · p. 9

7 Related work

The paper positions fragility alongside calibration-refinement, distributionally robust evaluation, and calibration under shift. Its distinction is a score-marginal-preserving shift budgeted within confidence levels.

Grouping loss

  • Calibration-refinement work studies static grouping loss, whereas this paper asks what that heterogeneity permits a shift to do.Related work estimates grouping loss for neural networks and relates it to decisions and epistemic uncertainty.Source: §7 · p. 9

Distributionally robust evaluation

  • Unlike robust risks over attribute-defined subpopulations or selected conditional distributions, this framework fixes the confidence marginal and budgets reweighting within each confidence level.A learned readout serves as the within-bin attributes, and the local law holds exactly on an interval.Source: §7 · p. 9

Calibration under shift

  • Unlike shift estimators that use target inputs, estimators observing only target confidences return the same estimate under every score-preserving shift.Multicalibration controls calibration on specified groups, while fragility measures heterogeneity left within a confidence level.Source: §7 · p. 9

8 Limitations and conclusion

The conclusion distinguishes exact level-set theory from empirical bin-and-readout quantities and warns that the reported source profile does not bound total natural drift. The appendices establish the transport identity, score-marginal-invisible drift, and the full profile’s budget regimes.

8 Limitations and conclusion

  • The empirical objects condition on bins and a learned readout, so they do not isolate exact-level heterogeneity; natural shifts need not attain the worst case or be bounded in total by a source quantity.The eighteen checkpoints are not a sample of models, and additions and post-hoc analyses were designed with earlier results known.Source: §8 · p. 9

B Conventions, measurability and what “at a score level” means

  • Score-preserving shifts are covariate reweightings: their density depends on inputs, preserving the label mechanism, and score preservation is equivalent to conditional mean weight one given the score.This restriction is used in every transport statement; label-dependent reweighting could change the correctness propensity.Source: B Conventions, measurability and what… · p. 14

C Proof of the reliability transport identity

  • Under score-preserving reweighting, the reliability change at a score equals the conditional expectation of the reweighting deviation times centred correctness propensity.The identity holds almost everywhere in the source score distribution.Source: C Proof of the reliability transport… · p. 14

D Proof of score-marginal-invisible drift (Proposition 2)

  • Where within-score propensity variance is positive, some admissible shift can move reliability while preserving the score marginal; where variance is zero, every such shift leaves reliability unchanged.The proposition is stated on measurable sets of positive score-distribution measure.Source: D Proof of score-marginal-invisible… · p. 14

E Proof of the full fragility profile (Theorem 1)

  • The fragility profile is nondecreasing and concave, and its full upward curve identifies the centred within-level propensity law, unlike variance alone.The general-law witnesses and deterministic-label collapse delimit when calibration residual and grouping variance do or do not determine fragility.Source: E Proof of the full fragility profile… · p. 15

E.1 A verification argument

  • For a thresholded propensity, the truncated-affine reweighting r*=(Z−τ)+/M1(τ) attains the profile bound while spending the prescribed conditional χ² budget.The verification establishes optimality and attainment together.Source: §E.1 · p. 15

E.2 The three regimes

  • For budgets below ρ+, the profile follows the square-root variance law; between ρ+ and ρsat it is tail-governed, and at or above ρsat it reaches the top propensity value.The middle regime may be empty, and saturation is absent when the top atom has zero mass.Source: §E.2 · p. 16
  • When within-level variance is zero, every admissible reweighting has zero movement and the fragility profile is identically zero.Source: §E.2 · p. 16

F Proof of profile identifiability (Theorem 2)

This section establishes that the upward fragility profile identifies the centred within-score propensity law, while not identifying its mean or providing a stable estimate of that law, and examines how fragility varies with direction, divergence choice, and readout resolution. It also compares accepted-set risk at matched thresholds under preserved score distributions, reporting envelope checks and qualifying exploratory readout, calibration, risk-ratio, and score-only comparisons.

F Proof of profile identifiability (Theorem 2)

  • The upward profile alone identifies the full centred propensity law, including its lower tail, but does not identify the conditional mean or guarantee stable recovery.The proof reconstructs the survival function from the profile and its derivative; it recovers the lower tail from where the variance-controlled regime ends, without using the downward profile. The reliability relation supplies the conditional mean, and the exact-curve identifiability result gives no stability guarantee for inversion.Source: F Proof of profile identifiability… · pp. 17–18

G Downward fragility and directional asymmetry

  • For asymmetric centred propensity laws, upward and downward profiles can saturate at different levels, although either profile alone identifies the law.The saturation thresholds differ whenever the conditional law is asymmetric; reporting both profiles is a convenience, not a requirement.Source: G Downward fragility and directional… · p. 18

H Divergence-universality of the local modulus

  • Smooth f-divergences share the same local fragility modulus, while Pearson chi-square yields an exact finite-radius square-root law; chi-square is not claimed to be unique.The exact interval follows from the quadratic generator and its affine extremal reweighting; other smooth divergences share the modulus only asymptotically. A non-quadratic generator can also reproduce the exact law on the same interval, so uniqueness is not asserted.Source: H Divergence-universality of the local… · p. 19

I Readout hierarchy (Proposition 5)

  • Refining a readout increases its restricted heterogeneity and fragility profile, but does not necessarily increase realised composition drift.For nested readouts, the restricted quantities are bounded above by their unrestricted counterparts; realised drift is not monotone under refinement.Source: I Readout hierarchy (Proposition 5) · pp. 19–20

I.1 Readout approximation (Proposition 6)

  • The readout approximation error is bounded using the within-cell residual variance, and the bound is sharp on the oracle profile’s variance-controlled interval.The residual variance is the oracle grouping variance not exposed by the readout; a readout constant within a bin attains the simple bound on that interval. Outside it, the bound need not be tight, and the residual variance is not identified from one label per input without assumptions.Source: §I.1 · p. 20

J Readout refinement in the frozen pilot

  • 6.5-fold: plug-in fragility rises from the null readout to the richest pilot readout, and realised held-out drift tracks the estimates induced by frozen weights.The pilot averages results over twenty confidence bins and five folds; its table reports cross-fitted plug-in estimates, not confidence bounds.Source: J Readout refinement in the frozen pilot · p. 21

J.1 ECE and FAUC across checkpoint–state combinations: an exploratory description

  • Pearson 0.91: source ECE and FAUC are marginally associated across 24 checkpoint–state combinations, but their partial correlation is 0.00 conditioning on grouping variance.Grouping variance retains partial correlation 0.86 given ECE; these exploratory correlations do not establish that ECE fails to determine fragility.Source: §J.1 · pp. 21–22

J.2 A score-only comparator, and why it is not a floor

  • A score-only readout is neither a floor nor a bound for the learned-readout profile, and their numerical difference does not isolate selection bias.Their partitions are not nested, and the protocols also differ in rate-estimation samples and readout-fitting data.Source: §J.2 · p. 22
  • The score-only comparator provides context for what a readout exposes at J=20, but the split-rate and same-fold comparator ratios are not quantitatively compared.Their difference mixes selection effects with a changed estimator and partition; the reported ratio ranges are 1.06–3.27 and 1.39–4.21, respectively.Source: §J.2 · p. 22

K Selective-risk transport (Proposition 3)

  • Bin-boundary thresholds preserve coverage under Q_B=P_B, and relative accepted-set risk increase is evaluated at the same threshold using the downward optimiser.The reported quantity is undefined when source risk is nonpositive.Source: K Selective-risk transport… · p. 23
  • 5.05 × 10^-16 was the maximum identity error across 576 combinations; 39/576 cells exceeded the envelope by a median 2.2 × 10^-4, attributed to estimated cell rates.The median exceedance was 0.7% of the bound, not a failure of the population statement.Source: K Selective-risk transport… · p. 23
  • The risk-grid analysis was descriptive, not preregistered; among 96 selected temperature-scaled combinations, the relative-risk-increase ratio had median 1.30 and reached 2 in only 6.The highlighted Swin-T/ConvNeXt-Tiny ratio of 2.29 at radius 0.3 and 90% coverage lay in the upper tail; the envelope exceeded realised held-out increases.Source: K Selective-risk transport… · p. 23

L Composition–conditional decomposition (Proposition 8) 𝑘𝑞𝑘= Í

The section decomposes binwise drift into composition and conditional terms, then clarifies how those quantities are aggregated and how estimator choices constrain their interpretation. It also gives witness laws separating calibration residual and grouping variance from fragility, and documents cross-fitting and sensitivity checks.

L Composition–conditional decomposition (Proposition 8) 𝑘𝑞𝑘= Í

  • The composition term is a feasible reweighting for the binwise programme and is therefore bounded by its fragility envelope under the stated support conditions.The envelope check closes numerically up to error; utilization, rather than containment, is the informative quantity.Source: L Composition–conditional… · p. 23
  • Domain-level Lx is the target-mass-weighted L2 norm of binwise δx; exact additivity holds binwise, while domain norms obey only the triangle inequality.Equality requires one component to be a non-negative multiple of the other.Source: L Composition–conditional… · pp. 23–24
  • 77.9% of target mass remains after excluding flagged ImageNet-C bins; the mean composition-to-total ratio shifts from 41.8% to 43.9%.The conditional norm remains larger in 401/450 units, but this post-hoc higher-count-bin check changes the estimand and has no interval or coverage statement.Source: Table 2 · p. 24, L Composition–conditional… · p. 24

M The witness laws of Proposition 4

  • Two finite-support witness laws agree in calibration residual and grouping variance, yet their profiles separate by 0.100 for ρ ≥ 2.Both have GP = 0.08 and profiles equal to 0.08√ρ on [0, 1.125].Source: M The witness laws of Proposition 4 · p. 25

N Cross-fitting, leakage test and estimator conventions

  • Reported FAUC integrates the upward plug-in profile over log-radius without normalizing by log 4; it is not computed from debiased GR.The downward integral is recorded separately as a directional-asymmetry diagnostic.Source: N Cross-fitting, leakage test and… · p. 26
  • Separating fit, rate-estimation, and evaluation data prevents rate labels from choosing their own cells; the earlier protocol reused labels to fit the readout and temperature.The revised roles fit supervised objects on data disjoint from the data used to estimate cell rates.Source: N Cross-fitting, leakage test and… · p. 26
  • Under the null, the same-fold estimator is positive at 3.2–4.6 standard errors, while the inner-split estimator is within one.The randomization uncertainties are conditional on cached predictions, not dataset sampling error; simulations detect no bias at their stated precision, while Proposition 9 gives conditional unbiasedness under its assumptions.Source: N Cross-fitting, leakage test and… · pp. 27–28
  • Unbiased cell rates do not yield an unbiased plug-in profile: convexity makes its expectation at least the true profile, while the split-sample lower bound alone carries a coverage guarantee.The inequality is in expectation, not a guarantee for a single realization.Source: N Cross-fitting, leakage test and… · p. 28

N.1 Split-sample lower confidence bound

The split-sample procedure gives a separate lower confidence bound for the restricted profile, with the preregistered analysis finding positive bounds for at least four of six architectures in a primary state. Its target is the learned finite readout, and the paper distinguishes this certification from plug-in estimates and from broader empirical claims.

N.1 Split-sample lower confidence bound

  • For each readout-group pair, moving mass from a less-correct to a more-correct group preserves bin mass and yields movement equal to the correctness gap times the feasible step.The step is limited by both the χ² budget and the source mass available in the group losing mass.Source: §N.1 · p. 29
  • The two split arms learn different partitions and readouts, so they certify different population targets and are reported separately.The 95% guarantee applies per analysis, not jointly across the two arms.Source: §N.1 · p. 30
  • The split-sample bound targets the restricted profile of the learned finite readout; it is not an error bar on the retained-cell FAUC estimator.All eight groups enter; small or empty cells receive vacuous intervals and a zero lower bound.Source: §N.1 · p. 30
  • Outcome A was observed: at ρ = 0.3, at least four of six architectures had a positive model-level bound in at least one primary state.The reported 95% statement is per model, state, and arm, not joint across architectures, states, or arms.Source: §N.1 · p. 30

O Extended scope, assumptions and claim boundaries

  • The population and algebraic claims are unchanged by the estimator revision, but numerical quantities using source cell rates were recomputed or withdrawn.The paper does not assume that a shared estimator produces a shared bias across models, so rankings had to be re-derived.Source: O Extended scope, assumptions and… · p. 31
  • The empirical heterogeneity and fragility targets are partition-restricted bounds for binned quantities, not the continuous-score oracle quantities; plug-in estimates have no finite-sample conservativeness guarantee.The controlled shifts are adversarial constructions, not forecasts of natural shift; at finite readout, the conditional term also conflates mechanism change with coarseness.Source: O Extended scope, assumptions and… · p. 31
  • The profile identifies the centred within-level law of correctness propensity, while deterministic labels and predictions form a special case where the reliability relation fixes the profile.The paper makes no claim about whether any particular labeling mechanism is deterministic, and its empirical cell rates need not be binary.Source: O Extended scope, assumptions and… · p. 31

P Preregistered natural-target disambiguation

The preregistered natural-target comparison tests whether source metrics track the composition component of drift, while a separate intervention examines what strictly increasing score maps preserve. ImageNet-C supports the preregistered direction, but the broader benchmark result is mixed; the monotone-map results apply fold by fold.

P Preregistered natural-target disambiguation

  • The three source metrics rank the six as-released checkpoints identically, limiting the comparison’s design power rather than showing that the metrics are equivalent.The pairwise Spearman correlations are +1.000; temperature scaling breaks the ranking but also reorders examples, so it does not isolate calibration from resolution.Source: P Preregistered natural-target… · p. 33
  • The analysis averages within-domain correlations across 75 ImageNet-C domains, with composition drift as its primary endpoint because the source profile bounds that channel.ImageNet-Sketch and ImageNetV2 each provide only one target domain, so their domain-bootstrap quantities are descriptive, not usable intervals.Source: P Preregistered natural-target… · pp. 33–34
  • On ImageNet-C, FAUC correlates with the composition term at +0.634, while ECE is −0.093 with an interval containing zero; corruption-block intervals preserve this contrast.The preregistered overall outcome was B (mixed), because the direction was not stable across all three benchmarks.Source: P Preregistered natural-target… · pp. 33, 35
  • The ImageNet-C composition-to-total norm ratio falls from 50.6% at severity 1 to 33.0% at severity 5.These are ratios, not additive shares; the analysis used the original fixed partition.Source: Figure 4 · p. 34

Q Monotone score reparameterisation

  • A fixed strictly increasing bijection preserves the score-preserving feasible set, within-level propensity law, grouping heterogeneity, fragility profile, coverage, and selective risk, but calibration is not invariant.The score value changes while its ordering and conditioning σ-field remain unchanged.Source: §Q.1 · p. 36

Q.2 Cross-fitted implementation, and why the lemma is applied fold by fold

  • The lemma is applied within each held-out fold because each fold has its own fitted map; the pooled score is not one global reparameterisation.No σ-field claim is made about the pooled score without fold identity.Source: §Q.2 · p. 36

Q.3 Primary structural analysis

The primary structural analysis tests whether a fold-wise strictly increasing score map preserves the fixed partition and readout, while calibration changes. The downstream invariance is an arithmetic consequence of identical assignments; later diagnostics distinguish this result from refitting the readout or using finite-precision stress maps.

Q.3 Primary structural analysis

  • Every Table 4 entry is exactly zero across the six architectures, confirming identical bin and readout assignments under the fold-wise map.The table counts 250,000 train and evaluation assignments per model; the preregistered tolerance was 10^-12.Source: §Q.3 · p. 36, Table 4 · p. 37

Q.4 Calibration endpoints

  • The median relative ECE reduction is 75.1%, and binary NLL improves for all six architectures.The result uses the original-protocol analysis; its baseline is distinct from the post-audit Table 1 baseline.Source: §Q.4 · p. 37

Q.5 Fitted maps and numerical diagnostics

  • Pooled ties fall from 29 to 4 because fold-specific maps can separate equal raw scores across folds, while within-fold tie breaks remain zero.For ResNet-50 V1, 4 duplicate-value groups lie within one fold and 25 span multiple folds, predicting the four pooled ties.Source: §Q.5 · p. 38
  • Pushed-forward bin edges need not equal independently refitted transformed-score quantiles because nonlinear maps do not commute with linear interpolation between order statistics.The primary analysis uses pushed-forward edges.Source: §Q.5 · p. 38

Q.6 Deterministic stress maps

  • The deterministic stress maps act on scalar confidence alone; the strictly increasing sharp map merges 146 ResNet-50 V1 scores in finite precision without changing bin assignments or structural quantities.The authors do not claim every such implementation is one-to-one in floating point.Source: §Q.6 · p. 38

Q.7 Secondary analysis: refitting the readout

  • Refitting the readout after changing the confidence coordinate can alter the restricted plug-in profile, although the population fragility target remains invariant under a bijective reparameterisation.This analysis is diagnostic only and is not used to support a claim that arbitrary learned readouts are invariant.Source: §Q.7 · p. 38

Q.8 Preregistration

  • The preregistered phase met its prespecified Outcome A criterion after fixing the transformation family, fitting procedure, endpoints, and outcome criteria in advance.Synthetic validation tests were required to pass before the six-model summary was computed.Source: §Q.8 · p. 39

R.1 Random score-preserving shifts

Across these post-audit analyses, natural-shift comparisons are qualitatively similar under the original and revised partitions, while CIFAR-10H offers a reproducible but approximate human-label proxy. Random score-preserving shifts show strong rank association with held-out movement, but much of that ordering is also reproduced without within-bin heterogeneity.

R.1 Random score-preserving shifts

  • 0.9010: the two-sided plug-in profile's Spearman correlation with median held-out random movement across 7,200 cells, but most of the association is reproduced by the label-permutation null.The cells are dependent and the ordering is described as a weak test of the profile.Source: §R.1 · p. 39, Figure 5 · p. 40
  • 96.1% of random-shift ratios are below one; among exceedances, the mean excess is 0.30, and exceedance is not a coverage failure.The ratios are descriptive over dependent shifts and use a plug-in profile.Source: §R.1 · pp. 39–40
  • The ratio above the profile falls from 9.8% to 0.63% across restricted-heterogeneity quartiles, and from about 5% at radii up to 0.3 to 0.55% at radius 3.These random shifts are not natural shifts and do not show how close natural shifts come to the worst case.Source: §R.1 · p. 40, Table 9 · p. 40

R.2 A human-label proxy for the correctness propensity

  • 0.9995: human-proxy heterogeneity from the two annotator halves has this Pearson correlation across 150 model-fold-bin values.The cross-half covariance is used to correct for vote-noise inflation; heterogeneity in bins also includes variation across scores within each bin.Source: §R.2 · p. 41
  • 3.3%: the pooled median same-proxy heterogeneity ratio; at radius 0.01, the profile ratios are 0.26, 0.16, and 0.18 for the three models.These ratios are descriptive: both profiles are estimated from finite data and their ratio's bias direction is not established.Source: §R.2 · p. 42
  • The CIFAR-10H case study finds reproducible within-bin heterogeneity in a human-label proxy, not the correctness propensity itself, and computes no lower confidence bound.It covers three convolutional classifiers on one dataset; repeated labels of the same images do not meet the lower-bound assumptions.Source: §R.2 · p. 42

R.3 Natural shift under the revised partition

  • 12.2% → 12.0% on ImageNetV2 and 9.8% → 10.0% on ImageNet-Sketch: the composition-to-total norm ratio is essentially unchanged under the revised partition.The conditional norm exceeds the composition norm in all 30 units on both benchmarks under both partitions.Source: §R.3 · p. 43, Table 11 · p. 43
  • ImageNet-C was not rerun, so this recomputation does not address its per-corruption comparison, severity trend, or channel-specific association.Source: §R.3 · p. 43

S Architecture-breadth extension

The post-audit extension tests the pipeline on twelve additional ImageNet checkpoints from twelve further families. Controlled-shift agreement and positive lower bounds recur in the broader cohort, while source-metric rankings and temperature-scaled results differ from the primary cohort in some respects.

S Architecture-breadth extension

  • The extension is descriptive: checkpoints are not i.i.d. samples from a model population, so no p-value or interval accompanies its comparisons.Source: S Architecture-breadth extension · p. 43
  • On all eighteen checkpoints, source-metric correlations are +0.75 for ECE–G_R, +0.85 for ECE–FAUC, and +0.95 for G_R–FAUC; the three metrics are no longer rank-identical.With temperature scaling, ECE–FAUC changes sign to −0.36, while G_R–FAUC remains +0.95.Source: S Architecture-breadth extension · p. 44, Table 13 · p. 44, Figure 7 · p. 45
  • 0.81: median controlled-shift rank agreement across all 72 checkpoint-state combinations, with the weakest agreement where restricted heterogeneity is smallest.The breadth cohort's median is 0.80 across its 48 combinations, versus 0.82 for the primary 24.Source: S Architecture-breadth extension · pp. 44–45, Figure 8 · p. 46
  • 0.9007: across all eighteen checkpoints, the random-shift plug-in profile's correlation with median held-out movement; the permutation null reproduces most of this statistic.The shifts are non-natural, and the plug-in has no coverage guarantee.Source: S Architecture-breadth extension · p. 46, Figure 9 · p. 46
  • The fold-wise monotone map lowers cross-fitted ECE by a median 76.2% for ten checkpoints while leaving structural quantities unchanged exactly.It is undefined for ResNeXt-50 and RegNetX-4.0GF because some top-label confidences equal 1.0 in float32.Source: S Architecture-breadth extension · p. 47
  • Family-level deletions keep as-released ECE–FAUC correlations between +0.79 and +0.90 and controlled-shift agreement between 0.81 and 0.82.In the temperature-scaled state, ECE–FAUC correlations range from −0.50 to −0.28 across family deletions; these are ranges, not intervals.Source: S Architecture-breadth extension · p. 47

T Post-hoc diagnostics added in the final revision

The post-hoc diagnostics test how random-shift ordering behaves under label permutation, compare the plug-in profile with variance summaries, and enumerate similar-ECE checkpoint pairs. The appendix also reports provenance constraints and notes that its known-propensity magnitude study uses synthetic laws.

T Post-hoc diagnostics added in the final revision

  • Similar estimated ECE does not fix FAUC: in every enumerated matched pair, the FAUC difference has the same sign as the G_R difference.No primary-cohort pair matches the ECE tolerance, and the comparison is not a test of fragility at fixed G_R.Source: T Post-hoc diagnostics added in the… · p. 47, Table 16 · p. 49
  • 0.877 observed versus 0.901 under the two-role permutation null: the null reproduces most of the pooled random-shift correlation across all eighteen checkpoints.At fixed radius, the corresponding correlations are 0.75 and 0.83; the proposed mechanism is binomial rate noise shared by both statistics.Source: T Post-hoc diagnostics added in the… · p. 48, Table 17 · p. 49
  • The controlled-shift null median is centred at zero, with maxima at most 0.07 across cohorts, versus 0.59–0.63 observed.The statistic is signed held-out drift from frozen maximizing weights.Source: T Post-hoc diagnostics added in the… · p. 48
  • Variance and capped-variance baselines exceed FAUC by median factors 1.17 and 1.09, respectively, but do not change rankings or held-out ordering on these readouts.The profile and variance differ in magnitude at radius ≥1 and between matched-variance bins; the effect of readout granularity was not tested.Source: T Post-hoc diagnostics added in the… · p. 48, Table 18 · p. 49

U Known-propensity magnitude and resolution study

  • The known-propensity magnitude and resolution study uses synthetic laws, showing when full-profile magnitude is estimable from rate labels, not whether it matters on a benchmark.Source: U Known-propensity magnitude and… · p. 50

U.1 Finite-sample magnitude with a known propensity

These appendices test finite-sample profile estimation, readout refinement, and ImageNet group merges, and audit a solver used in two permutation nulls. They report readout-resolution patterns on ImageNet alongside estimation and approximation results on known-propensity laws.

U.1 Finite-sample magnitude with a known propensity

  • The known-propensity study uses four-cell laws, ρ∈{0, 0.01, 0.03, 0.1, 0.3, 1, 3}, and 2000 replicates per condition, comparing plug-in and capped-variance estimates.The primary arm uses proportional stratified labels with at least one label per cell.Source: §U.1 · p. 50, Table 20 · p. 51
  • For law B at ρ=3, capped-variance error falls only 0.148 to 0.102 from n=250 to 5000, while plug-in error falls from 0.022 to 0.006.The capped estimate retains population slack as sample size grows; the plug-in error shrinks.Source: Table 19 · p. 50, §U.1 · p. 50
  • At n=250, law B with variance divided by 16 has 0.027 upward bias and over 56% relative errors for both estimators; the 30-label rule also truncates rare cells.The dropped cells include law B endpoints and, for the asymmetric downward profile, a rare hard cell.Source: §U.1 · p. 51

U.2 Nested readouts on known atoms

  • Nested readouts use smooth, rare, and wave laws on 32 atoms, with feature-ordered readout sizes K∈{1, 2, 4, 8, 16}; propensity ordering is only an oracle diagnostic.The finite-sample rates are derived from stratified labels on the 16 finest cells.Source: §U.2 · p. 52
  • At ρ=1 and ρ=3, the middle bound’s excess over the restricted profile is median 1.5 and 1.9 times the true gap; the wave law’s two-cell split has GR=0.0016 versus GP=0.050.The wave result illustrates how a coarse feature readout can miss much of the propensity heterogeneity.Source: Table 22 · p. 52, §U.2 · p. 52

U.3 Nested merges of the ImageNet readout

  • ImageNet merges reuse frozen bins, readout, rates, and masses; adjacent quantile groups from the eight-group readout create nested coarser partitions.The support is fixed at eight groups and inherited by the merges.Source: §U.3 · p. 52
  • Across 18 checkpoints, four groups retain 73%–94% of eight-group FAUC; only median 35% (released) and 14% (temperature-scaled) of the four-to-eight-group GR rise survives correction.The remaining share is the binomial-noise-corrected plug-in increase.Source: Table 24 · p. 53, §U.3 · p. 53

U.4 A defect in a closed-form solver used by two permutation nulls

  • The tied-rate solver defect affected two permutation nulls: after correction, no paper number or table entry changed, and the controlled-shift null was unchanged.The corrected solver agrees with frozen bisection to 10^-13 on 480,000 random tie-heavy cases.Source: §U.4 · p. 53
Loading 2609.38917v1…