PaperlayerView exact revision on arXiv

Source-linked AI summary

Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration

Daehwan Kim, Haejun Chung, Ikbeom Jang

arXiv:2609.01072v2cs.LGcs.CV

TL;DR

Multiclass post-hoc calibration can change the prediction associated with reported confidence, while accuracy does not measure how often such changes occur. CORD repairs the calibrated probability vector after fitting to recover the original top-1, attaining zero TPCR and improving mean ECE, NLL, and Brier across the evaluated datasets.

  • Problem

    Multiclass calibration can change the associated top-1 prediction, whereas accuracy captures only the net correctness effect and not the frequency of prediction changes measured by TPCR.

  • Method

    CORD uses the original and calibrated outputs to repair the full probability vector, preserving the calibrated conditional distribution while restoring the original top-1 without refitting or tuning.

  • Results

    CORD attains zero TPCR by construction and lowers mean ECE, NLL, and Brier relative to direct outputs across CIFAR-10, CIFAR-100, and ImageNet-1K, with gains persisting under shift and varied calibration-set sizes.

  • Takeaways & Limitations

    CORD removes prediction preservation from calibrator fitting and assigns exact recovery of the original decision to subsequent output repair.

  • Takeaways & Limitations

    CORD uses order-preserving stabilization for boundary outputs when required, while the supplied original and calibrated outputs remain unchanged.

Abstract

from arXiv · show

Post-hoc calibration corrects reported confidence, yet a multiclass calibrator can also change the associated top-1 prediction. Accuracy captures only the net effect of these changes on correctness, not how often predictions change; the Top-1 Prediction Change Rate (TPCR) instead measures this frequency. We propose Calibrator-Output Repair for Top-1 Decision Preservation (CORD), the first post-fit adapter to impose exact prediction preservation by repairing the full calibrated probability vector. From the original and calibrated outputs alone, CORD determines the mass assigned to the original top-1. The calibrated conditional distribution allocates the remaining mass over the other classes, yielding a repaired vector whose own argmax recovers the original prediction. On the calibration split, CORD coordinates the repaired masses to retain the calibrated outputs' mean mass on original predictions whenever attainable. The adapter alters neither the fitted calibrator nor its direct output, fits no additional supervised map, and requires no user- or validation-tuned hyperparameter. Across CIFAR-10/100 and ImageNet-1K, CORD attains zero TPCR by construction and lowers mean ECE, NLL, and Brier relative to the corresponding direct outputs in every dataset; paired gains persist under distribution shift and across calibration-set sizes. CORD thus removes the preservation constraint from calibrator fitting and assigns exact recovery of the original decision to subsequent output repair. Our code is available at https://github.com/labhai/CORD.

Introduction

Multiclass post-hoc calibration can change the top-1 prediction associated with reported confidence, while accuracy records only the net correctness effect. CORD instead repairs calibrated probability vectors after fitting to preserve the original decision exactly.

  • Motivation: A calibrated probability vector jointly determines reported confidence and the top-1 class through its argmax.Changing the vector can therefore alter both the confidence report and the prediction it describes.
  • Motivation: TPCR counts all top-1 revisions, including accuracy-neutral changes between incorrect classes, unlike accuracy's net correctness change.Figure 1 illustrates this distinction for Vector Scaling on ImageNet-1K with ResNet-50.
  • CORD: CORD repairs a fitted calibrator's full probability vector so its argmax matches the original prediction.The adapter uses the original and calibrated outputs while leaving the fitted calibrator and direct output unchanged.
  • CORD: CORD reallocates mass between the original prediction and the remaining classes while preserving the calibrated conditional distribution among those remaining classes.It coordinates repairs over the calibration split to retain the calibrated outputs' mean mass on original predictions whenever attainable.

Related Work

Prior calibration methods preserve predictions during fitting by restricting or structuring the calibration map. CORD instead repairs calibrated outputs after fitting while preserving conditional class allocation and coordinating repaired masses across the calibration split.

  • Fit-time Prediction Preservation: Fit-time methods preserve class ordering through constrained or structured calibration maps, including TS, IRM, order-preserving neural maps, PTS, and AdaTS.These approaches impose preservation within the fitted calibration mechanism, often through shared or input-dependent temperature transformations or order-preserving mappings.
  • Post-fit Repair: CORD separates preservation from fitting by repairing the full calibrated probability vector using only original and calibrated outputs.The repaired vector recovers the original prediction while leaving the fitted calibrator and its direct output unchanged.
  • One-Dimensional Repair Family: CORD preserves the calibrated conditional distribution over non-original classes, changing only the mass split between the original class and the remaining classes.Renormalizing the non-original coordinates preserves their pairwise ratios and minimizes KL divergence on the corresponding simplex slice.
  • Prediction-Preserving Interval: Prediction preservation reduces each input's remaining repair freedom to a scalar mass s_i within a nonempty interval that makes the original class uniquely top-ranked.The interval is determined by the largest conditional probability among the other classes and a fixed numerical offset 10^-12.
  • Coordinating Repairs on the Calibration Split: CORD coordinates per-input repairs through one aggregate mean-mass constraint, retaining the calibrated mean mass on original predictions whenever attainable.A single Lagrange multiplier couples the repairs, and the resulting scalar equation can be solved by bisection.
  • Structural Guarantees: The structural guarantees ensure repaired vectors remain probability vectors, uniquely recover the original top-1, preserve conditional allocation, and retain the inherited feasible mean on the calibration split.These guarantees assume stabilized interior outputs, a fixed deterministic tie rule, and 0 < ϵ_num < 1/4.

Experiments

Experiments evaluate CORD across datasets, calibrator families, corruption severities, and calibration-set sizes. CORD preserves every original top-1 prediction while generally improving calibration and full-distribution metrics relative to direct outputs.

  • Experimental setup: CORD is evaluated on CIFAR-10/100 and ImageNet-1K using diverse classifiers, shared five-split calibration/evaluation protocols, and multiple calibrator families.The Base set includes seven non-preserving calibrators, with a reduced set on ImageNet-1K because of computational cost.
  • Prediction preservation: CORD attains zero TPCR for every evaluated classifier–calibrator pair, recovering every original top-1 prediction and therefore the original accuracy.Direct-output TPCR spans 0.07%–31.05%, while top-1 accuracy changes range from −10.06 to +0.22 percentage points across dataset–calibrator pairs.
  • Main results: CORD lowers mean ECE, NLL, and Brier relative to corresponding direct outputs in every dataset, with confidence intervals above zero for the reported reductions.The NLL and Brier reductions extend the observed gains beyond confidence calibration to the full probability report.
  • Repair ablations: CORD outperforms a minimal pointwise repair on ECE, NLL, and Brier while both repairs preserve the calibrated conditional distribution and attain zero TPCR.This comparison isolates the choice of repaired mass assigned to the original prediction.
  • Robustness under distribution shift: Under corruption, CORD maintains zero TPCR and positive paired reductions in ECE, NLL, and Brier at every severity on CIFAR-10-C and CIFAR-100-C.Direct-output TPCR rises from 1.65% to 4.58% on CIFAR-10-C and from 12.75% to 18.70% on CIFAR-100-C across severities 1–5.
  • Calibration-size sensitivity: On ImageNet-1K, paired reductions persist across calibration-set fractions from 10%–100%, while direct-output TPCR declines with more calibration data but remains nonzero at 100%.CORD maintains zero TPCR throughout, and paired gains widen with higher direct-output TPCR and narrow as TPCR falls.

Conclusion

CORD moves exact top-1 preservation from calibrator fitting to post-fit repair, making preservation a deployment choice while retaining the unchanged direct output option.

  • Conclusion: CORD relocates exact top-1 prediction preservation from calibrator fitting to post-fit output repair.This opens post-fit preservation as a distinct calibration design space.
  • Conclusion: Preservation can be selected at deployment: use direct outputs when prediction changes are acceptable, or repaired outputs when the original top-1 must be retained.Whether a change improves or degrades correctness is knowable only retrospectively from the label.
  • Conclusion: CORD provides a lightweight post-fit route without an auxiliary supervised map or a user- or validation-tuned hyperparameter.The broader design question concerns when preservation is required and how to select a probability report within the feasible family.

Derivations and Numerical Details for CORD

The appendix supplies derivations and fixed numerical conventions deferred from the main paper's CORD section.

  • Derivations and Numerical Details for CORD: The appendix records fixed numerical conventions deferred from the main paper.These conventions accompany the derivations for CORD.
  • Derivations and Numerical Details for CORD: The supplied subsection is supplementary material for the main paper's CORD section.It provides derivations and conventions rather than introducing a separate method.

Boundary Outputs and Fixed Numerical Conventions

CORD stabilizes boundary probability outputs internally while preserving pairwise ordering, maximizers, normalization, and the fixed tie rule.

  • Boundary Outputs and Fixed Numerical Conventions: Exact ties select the smallest maximizing class index for original, direct, and repaired outputs.CORD uses ϵnum = 10^-12 and δstab = 10^-10 throughout.
  • Boundary Outputs and Fixed Numerical Conventions: Order-preserving stabilization keeps every pairwise order relation and the complete maximizer set unchanged while placing all coordinates in (0, 1).The stabilized vector is normalized, so the fixed tie rule selects the same original prediction.
  • Boundary Outputs and Fixed Numerical Conventions: Stabilization is internal, and the supplied outputs remain unchanged.For q, the internal copy differs by at most 2δstab in L1 distance.

KL Characterization of the Repair Family

For a fixed mass on the original prediction, CORD's repaired vector uniquely preserves the calibrated conditional distribution over all remaining classes.

  • KL Characterization of the Repair Family: CORD fixes the original predicted class a, its calibrated mass b = q_a, and the calibrated conditional allocation α_j = q_j/(1−b) for j ≠ a.The repair is analyzed on the simplex slice with prescribed original-class mass s.
  • KL Characterization of the Repair Family: The KL objective separates into a constant term and a nonnegative divergence that vanishes only when the repaired conditional distribution equals α.Thus, once the original-prediction mass is fixed, the calibrated conditional distribution determines the remaining coordinates.
  • KL Characterization of the Repair Family: For fixed s, the reconstruction u = s e_a + (1−s)α uniquely minimizes D_KL(q∥·) on the simplex slice u_a = s.The remaining-class conditional distribution is inherited rather than re-estimated.

Shared-Scalar Characterization

CORD characterizes the calibration-split repair through a single shared scalar that coordinates per-example repaired masses while preserving feasibility and uniqueness.

  • Shared optimization: The coordinated repair minimizes a strictly convex objective over interval-bounded masses with an aggregate mean constraint.The feasible set is nonempty and compact because the target mean lies in the attainable interval.
  • KKT reduction: A single multiplier η enforces the aggregate equality, while local bound multipliers are eliminated by interval-constrained per-example minimization.The shared scalar is the only multiplier coupling repairs across examples.
  • Per-example response: The interval-constrained response is determined by endpoint clipping or the unique interior solution satisfying h_g(s) = η.The response uses lower and upper endpoints when η falls outside the corresponding stationarity range.
  • Existence: The mean response is continuous and nondecreasing, attaining every value in the feasible interval [s̄−, s̄+].Consequently, the set E_µ of scalars achieving the target mean is a nonempty closed interval.
  • Uniqueness: Every η in E_µ yields the unique calibration-split repair through the KKT conditions, while new inputs use a unique nearest-zero scalar selected from E_µ.The nearest-zero convention resolves possible nonuniqueness in the shared scalar.

Closed-Form Response and Numerical Solver

CORD evaluates each interval-constrained response in closed form and solves for the shared scalar with finite-bracket, plateau-aware bisection.

  • Closed-form response: The interior stationarity equation is quadratic, and CORD uses a cancellation-safe root in (0, 1).At η = 0, the closed-form response equals g before projection.
  • Closed-form response: Projecting the closed-form response onto [s−, s+] produces the constrained per-example response.The projection handles endpoint-active cases as well as interior stationarity.
  • Numerical bracketing: A finite bracket [L, U] contains zero and the nearest-zero multiplier because endpoint responses attain the lower and upper feasible means.The bracket guarantees that the target mean lies between the response means at its endpoints.
  • Numerical solver: When the solution set is a plateau, CORD uses a generalized-inverse interpretation and selects the nearest-zero solution.The solver updates the target-side endpoint on equality and returns that endpoint after no distinct binary64 midpoint remains.

Proof of the Structural Guarantees

The structural proof shows that CORD repairs calibrated vectors without changing the original decision, while experiments report preserved accuracy and improved calibration metrics across datasets and conditions.

  • Structural guarantees: CORD’s repaired vector is a probability distribution constructed from the repaired original-class mass and the calibrated conditional distribution over other classes.The proof establishes existence and uniqueness of the calibration-split repair before verifying its prediction properties.
  • Structural guarantees: The original class is uniquely top-ranked in the repaired vector, establishing exact prediction preservation independently of the tie rule.Every non-original class receives at most the remaining mass allocated through the calibrated tail.
  • Identity characterization: CORD returns the calibrated output unchanged on the calibration split exactly when every direct original-class mass lies within its feasible interval.For new inputs, identity holds when the calibrated original-class mass lies in the corresponding interval; stabilization bounds the internal-output deviation by 2δ_stab.
  • Sensitivity: The equal-weight choice λ = 1/2 is selected because it is the only fixed weight symmetric to exchanging the direct and reference original-class masses.All seven tested fixed weights produced positive mean reductions in ECE, NLL, and Brier, while the largest point estimate varied by metric across λ ∈ [0.3, 0.9].
  • Empirical results: Across classifier–calibrator pairs, CORD exactly recovers original top-1 predictions and therefore original accuracy while repair effects on ECE, NLL, and Brier vary by direct output.Tables 7 and 8 report five-split means for CIFAR-10 and CIFAR-100, with positive ∆R_M indicating improvement.
  • Reliability diagrams: Reliability diagrams average curves over five splits, with rows for classifiers, columns for Uncal. and Base calibrators, and shading for pointwise 95% CIs.Parenthetical values report ECE for Uncal. or Base → Base + CORD.
  • Distribution shift: Under corruption, CORD maintains zero TPCR and lower ECE, NLL, and Brier than Base at every severity on CIFAR-10 and CIFAR-100.The separation widens as corruption severity and direct-output TPCR increase.
  • Calibration-set size: Across calibration-set fractions, CORD maintains zero TPCR and persistent mean paired reductions in ECE, NLL, and Brier on both CIFAR datasets.Changes are smaller on CIFAR-10 and larger on CIFAR-100, consistent with their direct-output TPCR levels.
Loading 2609.01072v2…