PaperlayerView exact revision on arXiv

Source-linked AI summary

Signal-Routed Temperature Scaling: Low-Capacity Risk-Conditioned Calibration for Small Validation Budgets

Wenhao Liang, Liangwei Nathan Zheng, Lin Yue, Wei Emma Zhang, Mingyu Guo, Olaf Maennel, Weitong Chen

arXiv:2609.38936v1cs.LGcs.AI

TL;DR

With small calibration splits, a scalar map may leave structured miscalibration, while a highly adaptive map may be difficult to estimate reliably. The paper proposes SRTS-BCE, a low-capacity risk-conditioned calibrator, and finds its relative advantage over higher-capacity calibration grows at small budgets, with rankings depending on budget and dataset.

  • Problem

    With only a few thousand held-out examples, calibration capacity must balance correcting structured residual miscalibration against fitting noise.

    Source: Introduction · p. 1

  • Method

    SRTS-BCE cross-fits a correctness-risk score from logit statistics, routes examples into three groups, and fits one top-label-BCE temperature per group.

    Source: Introduction · p. 1, Method: SRTS-BCE · p. 3

  • Results

    At n=250, SRTS-BCE beats SMART+BCE on all three CIFAR-100 backbones, and the Swin-T ranking reverses between the largest evaluated budget and n=250.

    Source: RQ1b: Does capacity become decisive as… · p. 5

  • Takeaways & Limitations

    The results indicate that the preferred calibrator capacity shifts with calibration budget, with low capacity helping when structured residuals are present but data are limited.

    Source: Introduction · p. 2, RQ1b: Does capacity become decisive as… · p. 5

    A routing signal alone does not ensure ECE gains: transfer also requires a BCE-to-ECE link and router stability, so the method does not provide a rule for choosing capacity from validation data alone.

    Source: Discussion and Limitations · p. 7

Abstract

from arXiv · show

When a classifier is recalibrated from only a few thousand held-out examples, the capacity of the calibration map becomes a statistical design choice rather than a purely architectural one: a scalar map can underfit structured residual miscalibration, while a highly adaptive map can be hard to estimate reliably from so small a split. We disentangle the calibration objective from adaptive capacity and propose signal-routed temperature scaling (SRTS-BCE), a 10-parameter, argmax-preserving calibrator that cross-fits a correctness-risk score over six logit statistics and fits one top-label-BCE temperature per $K=3$ risk groups, recovering TvA-TS as its $K=1$ limit. On fine-tuned CIFAR-100 / ViT-B/16, SRTS-BCE reduces $\mathrm{ECE}_{15}$ from 1.65 (scalar TvA-TS) to 0.96, matching the higher-capacity SMART+BCE head (0.95) at the full calibration budget. The two regimes separate as the budget shrinks: at $n=250$ SRTS-BCE beats SMART+BCE on all three CIFAR-100 backbones (the seed-to-draw hierarchical interval excludes zero), whereas the flagship comparison against the scalar remains directional. A protocol-frozen Tiny-ImageNet follow-up reproduces the small-budget separation and exhibits a budget-dependent ranking reversal on Swin-T; matched routing and map controls show that the effect is tied neither to the learned router nor to discrete grouping. Together the results identify post-hoc calibrator capacity as a finite-sample design choice whose preferred level shifts with the amount of available calibration data.

Introduction

With only a few thousand held-out examples, calibration-map capacity becomes a statistical design choice: scalar maps may leave structured miscalibration, while adaptive heads may fit noise. The paper introduces SRTS-BCE as a low-capacity middle ground and studies its behavior as calibration budgets shrink.

Introduction

  • On CIFAR-100 / ViT-B/16, fine-tuning reaches mean top-1 91.00 while retaining ECE15 = 9.69, motivating post-hoc recalibration.The authors frame calibration capacity as a trade-off between correcting structured residual miscalibration and avoiding noise fitting on a small split.Source: Introduction · p. 1
  • Existing comparisons often vary calibration objective and adaptive capacity together, making their individual effects difficult to separate.The paper distinguishes NLL from top-label BCE objectives and global, reliability-conditioned, and per-sample temperature capacity.Source: Introduction · p. 1
  • SRTS-BCE is a 10-parameter, argmax-preserving calibrator that cross-fits a correctness-risk router and fits a top-label-BCE temperature for each of three groups.It recovers TvA-TS at K=1.Source: Introduction · pp. 1–2
  • At n=250, SRTS-BCE beats SMART+BCE on all three CIFAR-100 backbones, while the scalar comparison remains directional.The paper also reports two protocol-frozen Tiny-ImageNet replication cells and controls across objective, routing signal, and map family.Source: Introduction · p. 2

Related Work

Prior calibration methods range from a single global temperature to sample-dependent heads and grouped maps. SRTS-BCE combines a learned correctness-risk ordering with group-specific top-label BCE temperatures, placing it between those capacity extremes.

Related Work

  • A global temperature is economical to estimate but applies the same correction to every example, while adaptive heads predict input-conditional temperatures.The cited approaches range from relatively unrestricted per-sample heads to low-parameter maps conditioned on reliability statistics.Source: Related Work · p. 2
  • Grouped calibration methods share parameters within input regions, using partitions or learned groupings rather than unrestricted sample-wise temperature functions.Examples include confidence-based piecewise maps, local maps, ensembles, and learned semantic or topology-based groupings.Source: Related Work · p. 2
  • SRTS-BCE groups examples by a cross-fitted correctness-risk score and fits one top-label BCE temperature per group, with TvA-TS as its K=1 limit.It conditions on several logit statistics while retaining less adaptive capacity than a sample-wise head.Source: Related Work · p. 3

Background and Problem Setup

The paper studies fit-once post-hoc calibration from a small clean validation split, with the calibrator frozen for evaluation on clean and corrupted test data. It uses ECE15 as its primary deployment-facing summary while reporting additional calibration and risk-coverage metrics.

Background and Problem Setup

  • The setup fits an argmax-preserving calibrator once from a stratified clean validation set of about 2,500 examples, then freezes it for clean and corrupted test evaluation.Shift evaluations use no corrupted labels, clean-test pairings, or test-set model selection.Source: Background and Problem Setup · p. 3
  • Positive scalar temperature rescaling preserves the predicted class, and the paper reports that no evaluated post-hoc method changes top-1 accuracy.Calibration is assessed with ECE15, AdaECE15, CW-ECE15, smECE, NLL, and Brier score; lower is better for all.Source: Background and Problem Setup · p. 3
  • App. D visualises the structured residual left by scalar TvA-TS.Source: Background and Problem Setup · p. 3

Method: SRTS-BCE

SRTS-BCE uses a cross-fitted correctness-risk score to route examples into risk groups, then fits a top-label BCE temperature per group. Its design constrains adaptive capacity while preserving the classifier's predicted class.

Method: SRTS-BCE

  • SRTS-BCE routes examples into three equal-frequency risk groups and fits one top-label-BCE temperature per group.The router selects a temperature group rather than directly outputting calibrated probabilities.Source: Method: SRTS-BCE · p. 3
  • A five-fold stratified logistic router uses six standardized logit statistics plus an intercept to produce out-of-fold correctness-risk scores.The statistics are maximum probability, logit margin, probability margin, entropy, logit norm, and top logit.Source: Method: SRTS-BCE · p. 3
  • For deployment, the router is refit on the full clean split, and each test sample is routed using its logits with thresholds and temperatures fixed.Groups with fewer than 50 samples use the pooled temperature.Source: Method: SRTS-BCE · p. 3
  • The fixed-partition approximation says grouped fitting can improve pooled top-label BCE when between-group gradient variance exceeds the variance penalty associated with K.The relation is local and approximate, predicts BCE rather than ECE, and excludes router- and threshold-estimation error.Source: Method: SRTS-BCE · p. 4
  • The design fixes K=3 before evaluation because increasing K raises O(K/n) estimation variance, while hold-out K-selection is unreliable at this budget.The stated components address objective mismatch, in-sample routing bias, feature-head overfitting, and shift deployability.Source: Method: SRTS-BCE · p. 4

Experimental Setup

The evaluation uses fine-tuned image classifiers, a fixed stratified calibration split, and held-out test evaluation. Objective-matched baselines and fixed-K cross-regime comparisons help separate calibration objective from capacity.

Experimental Setup

  • The flagship CIFAR-100 / ViT-B/16 evaluation uses five fine-tuning seeds and a 5% stratified validation split of about 2,500 samples, with no test-set selection.Cross-backbone, mechanism-ablation, and shift diagnostics generally use three seeds.Source: Experimental Setup · p. 4
  • Comparisons include scalar TS-NLL and TvA-TS, unrestricted adaptive heads, and low-parameter HTS and SMART maps.HTS and SMART are also fit with top-label BCE to match the objective used by SRTS-BCE and TvA-TS.Source: Experimental Setup · p. 4
  • The cross-regime evaluation covers CIFAR-100, CIFAR-10, and Tiny-ImageNet, with transformer and ConvNet backbones; every main-table SRTS-BCE entry uses fixed K=3.For CIFAR-100-C, calibrators are fit on clean validation logits and applied unchanged using each corrupted sample’s logits for rerouting.Source: Experimental Setup · p. 4

Results

SRTS-BCE improves on scalar calibration in the flagship clean comparison and separates from a higher-capacity head as calibration data shrink. Matched controls and shift evaluations qualify where that advantage appears.

RQ1: Does SRTS-BCE improve over scalar and unrestricted calibration?

  • 0.96 ± 0.32 ECE15 for SRTS-BCE improves on TvA-TS at 1.65 ± 0.44, while SMART+BCE has a statistically unresolved lower clean point estimate (0.9483 vs. 0.9559).The SRTS-BCE–TvA-TS difference is 0.70 pp; the matched SMART+BCE comparison has hierarchical CI [−0.19, +0.12].Source: RQ1: Does SRTS-BCE improve over scalar… · pp. 4–5, Table 1 · p. 5

RQ1b: Does capacity become decisive as the calibration budget shrinks?

  • At n=250, SRTS-BCE beats SMART+BCE on all three CIFAR-100 backbones, with the seed-to-draw interval excluding zero; the flagship comparison against TvA-TS remains directional.On ViT-B/16, SRTS-BCE is lower on 81% of draws, while SMART+BCE beats the scalar on 47%; a protocol-frozen Tiny-ImageNet follow-up reproduces the small-budget separation, with a Swin-T ranking reversal at 2,500 samples.Source: RQ1b: Does capacity become decisive as… · p. 5, Figure 2 · p. 6

RQ2: Which components of SRTS-BCE matter?

  • At K=3 with BCE, risk and margin routing outperform confidence-only and random partitions but are not distinguished; under NLL, all rules collapse to one band (ECE15 ∈[1.71, 1.82]).Source: RQ2: Which components of SRTS-BCE… · p. 5, Figure 3 · p. 6
  • PWLinear-3 is statistically indistinguishable from SRTS-BCE on clean CIFAR-100 and CIFAR-100-C, showing the result is not specific to discrete grouping.QaTS and HTS also show BCE-over-NLL improvements; QaTS trails SRTS-BCE on CIFAR-100 and CIFAR-100-C, but comparisons are unresolved in some other regimes.Source: Table 2 · p. 6, RQ2: Which components of SRTS-BCE… · pp. 6–7
  • On CIFAR-100-C, SRTS-BCE has a lower mean ECE15 than SMART+BCE, but the 0.12-pp difference is directional and its primary hierarchical interval includes zero.With deployable rerouting, the high-risk share grows from 44% to 66% as corruption severity increases; forcing equal test-time occupancy worsens ECE monotonically.Source: RQ3: When does clean-fitted SRTS-BCE… · p. 7, Figure 4 · p. 7

RQ3: When does clean-fitted SRTS-BCE transfer?

  • Routing gains concentrate where clean-fitted reliability ordering separates groups with distinct calibration optima and retains that ordering at deployment; under natural shift, calibration harms pretrained ImageNet-R/-A.The objective transfers on fine-tuned ImageNet-V2, whereas the pretrained ImageNet-R/-A result marks a boundary to transfer.Source: RQ3: When does clean-fitted SRTS-BCE… · p. 7

Discussion and Limitations

The discussion frames conditional calibration as useful only within supported transfer conditions and leaves validation-only capacity selection unresolved. The appendix supplies implementation, metric, and objective-capacity details, alongside reproducibility boundaries.

Discussion and Limitations

  • Routing gains are positive only where a transferable residual dominates, and selecting capacity from validation data alone remains open; the 1-SE selector still fails in the low-data regime.SRTS-BCE primarily improves top-label confidence, while selective risk is nearly unchanged versus TvA-TS.Source: Discussion and Limitations · p. 7

Appendix

  • The appendix collects implementation details, additional ablations, statistical analyses, distribution-shift results, and extended experiments supporting the main text.Its stated organization separates objective, capacity, routing, calibration-budget, shift, and cross-backbone analyses.Source: Appendix · p. 10

A Reproducibility and Implementation Notes

  • The five-fold cross-fitted router uses six z-scored logit signals, and groups with fewer than 50 validation samples fall back to the global TvA-TS temperature.The router is refit on the full validation split for deployment; the package documents calibration-stage reproduction but excludes raw logits for several regimes.Source: A Reproducibility and Implementation… · p. 10, Metric definitions · p. 11

Metric definitions

  • ECE-family metrics are reported in percentage points; ECE15 uses 15 equal-width confidence bins, while AdaECE15 uses 15 equal-mass bins.TopBCE is the test-set top-label binary cross-entropy, the method’s fitting objective, evaluated on calibrated probabilities.Source: Metric definitions · p. 10

B Objective and Capacity

  • The BCE K-ablation gives mean ECE15 values 1.65/1.07/0.96/0.94/1.01 at K=1/2/3/5/10, selecting K=3 as default without stable evidence of added benefit at K=5.The surrounding analyses compare fitting objective and temperature-map capacity as separate factors.Source: B Objective and Capacity · p. 11

Objective-conditional routing ablation

Routing helps under BCE but not NLL, and additional diagnostics show that calibration performance depends on both the fitting objective and the capacity allowed by the small validation split. The section also compares alternative low-capacity maps and unrestricted adaptive heads.

Objective-conditional routing ablation

  • Under BCE, risk routing reaches ECE15 0.96 versus confidence 1.29, shuffled 1.59, and random 1.68; under NLL, deployable routes cluster at 1.71–1.82.The OOF risk router also beats the label-informed true-label-risk router at 1.69; the interpretation is specific to the fitting objective.Source: Objective-conditional routing ablation · p. 11, Table 4 · p. 12

Mixed-objective diagnostic

  • As the BCE weight rises in mixed-objective fits, the K=3 risk-routed variant moves toward the pure-BCE result while retaining the same OOF risk router.The pure-BCE reference is 0.83±0.28 in the three-seed diagnostic.Source: Mixed-objective diagnostic · p. 12, Table 6 · p. 13

Adaptive-head fairness sweep (capacity-control diagnostic)

Cross-backbone adaptive-head failure summary

BCE-trained unrestricted adaptive-head fairness check

  • With BCE-matched training and selection, SATS-Logit-BCE reaches ECE15 8.12±0.84 and Feature-SATS-BCE 9.61±0.44, versus 1.54 for TvA-TS and 0.83 for SRTS-BCE.The evaluated heads also substantially degrade NLL; the authors limit this failure claim to the heads and protocols tested.Source: BCE-trained unrestricted adaptive-head… · p. 13, Table 9

NLL-objective diagnostics

Scaling

  • On CIFAR-100-C, the SRTS-BCE versus PWLinear-3 hierarchical interval includes zero, while cell-level and corruption-type intervals favor PWLinear-3; changing router-score scale leaves conclusions unchanged.The QaTS extension reports that QaTS-BCE trails SRTS-BCE on CIFAR-100, IN-100/DeiT-S, and CIFAR-100-C, but is statistically unresolved on Tiny-ImageNet/ViT-B/16.Source: Scaling · pp. 15–17, Table 14 · p. 17
  • On clean CIFAR-100/ViT-B/16, HTS-BCE (1.63) matches scalar TvA-TS (1.65), while SRTS-BCE reaches 0.96; at 250 samples, SRTS beats HTS on all three backbones.The clean flagship SRTS-BCE versus TvA-TS interval excludes zero, while comparisons with matched-capacity continuous maps and SMART+BCE include zero; proper-score gains are chiefly in TopBCE.Source: Scaling · pp. 14, 15, 17, Table 13 · p. 17

D Routing-Signal Ablations

The section frames SRTS-BCE as isolating routing signal from the fixed BCE objective and K=3 capacity. Its motivating diagnostics show why risk-conditioned scaling is considered.

D Routing-Signal Ablations

  • The routing-signal section holds the top-label BCE objective and K=3 budget fixed to isolate which conditioning signal carries the benefit.The subsection explicitly distinguishes this signal ablation from changes to objective or group count.Source: D Routing-Signal Ablations · p. 17

Motivating diagnostics: why risk-conditioned scaling is needed

  • At matched scalar confidence, empirical accuracy varies across OOF-risk groups; scalar TvA-TS leaves its largest residual in the high-risk group.The K-ablation indicates K=3 captures most of the available gain.Source: Motivating diagnostics: why… · p. 17

Routing-signal ablation (which signal carries the benefit)

With K=3 and group-wise BCE fitting held fixed, the routing-signal comparison finds that genuine reliability signals outperform random or shuffled partitions. Margin routing is competitive with the six-signal OOF risk router.

Routing-signal ablation (which signal carries the benefit)

  • Random routing (ECE15 1.68) and shuffled routing (1.59) are near scalar TvA-TS (1.65), while margin routing reaches 0.97 and the six-signal risk router 0.96.The paired comparison does not statistically separate margin-only from the six-signal router; the latter is a sensible default, not a demonstrated improvement.Source: Routing-signal ablation (which signal… · pp. 17–18

Margin-only vs. learned-router routing: fixed-budget ablation

The fixed-budget ablations find that margin is a strong routing signal and that the full OOF risk router does not significantly outperform it. Alternative routing targets trade off metrics rather than providing universal gains.

Margin-only vs. learned-router routing: fixed-budget ablation

  • At fixed K=3 and BCE, margin tertiles are strong; the full six-signal router has slightly lower point estimates, but its paired advantage over margin is not statistically resolved.Across seven cells, the mean risk-minus-margin ECE15 difference is −0.17 pp with 95% CI [−0.51, +0.15].Source: Margin-only vs. learned-router… · pp. 18–19, Table 17 · p. 19

Alternative Low-Capacity Routing Targets

  • The OOF wrongness-risk router remains best on mean ECE15; margin routing is competitive, while signed-residual routing improves NLL, Brier, and eAURC but worsens ECE15.These are metric-specific trade-offs in the small-validation ID-clean setting, not universal improvements.Source: Alternative Low-Capacity Routing Targets · p. 19

E Budget Sensitivity

  • The budget-sensitivity subsection motivates a working rule of a few hundred effective validation samples per group for risk grouping.It points to a paired resampling analysis and a Tiny/ViT-B/16 boundary.Source: E Budget Sensitivity · p. 19

Calibration-resample budget experiment

Calibration-split resampling shows that low-capacity conditional maps retain an advantage at small budgets, while their relative performance and the useful map form depend on the setting. The evidence supports a small-budget separation on CIFAR-100, with limits from metric choice, dataset scope, and matched-map comparisons.

Calibration-resample budget experiment

  • At 250 samples, SRTS-BCE beats TvA-TS on 81% of resamples; SMART+BCE has mean ECE15 2.03 versus the scalar's 1.96 and wins 47%.As budgets grow, the adaptive methods converge, and all beat the scalar at 2500 samples.Source: Calibration-resample budget experiment · pp. 19–20
  • Among K=3 routing targets, wrongness-risk routing is best on ECE15, while margin routing leads on AdaECE15/smECE and signed-residual routing improves NLL/Brier/eAURC.These are metric trade-offs in the small-validation clean CIFAR-100 setting, not a universal ranking.Source: Table 20 · p. 20
  • At n=250, SRTS−SMART seed-level intervals exclude zero on ViT [−0.59, −0.28], DeiT-S [−1.34, −0.30], and Swin-T [−0.59, −0.25].These are the primary per-backbone replication intervals.Source: Calibration-resample budget experiment · p. 20
  • At n=250, SRTS−TvA excludes zero on DeiT-S and Swin-T, while the ViT-B/16 interval [−0.72, +0.01] remains directional.The ViT comparison has four of five seeds negative and an 81% draw-level win rate.Source: Calibration-resample budget experiment · p. 20
  • The n=250 SRTS−SMART separation excludes zero on all backbones under AdaECE15 and smECE, but the ViT-B/16 SRTS−TvA intervals include zero under both alternatives.The flagship scalar gap therefore depends on the binned estimator; the cross-backbone replication does not.Source: Calibration-resample budget experiment · p. 20
  • On Tiny-ImageNet/ViT-B/16, conditioning offers no advantage at any budget, and every conditional map is within noise of or worse than TvA-TS.For TopBCE at 250 samples, SRTS-BCE wins only 0.21 of cases; the authors frame routing gain as structured-residual benefit minus finite-budget variance cost.Source: Calibration-resample budget experiment · p. 21

Small-budget matched continuous-map test

  • At n=250, SRTS-BCE is statistically unresolved against matched PWLinear-3, but beats matched linear and spline risk-score maps on all three backbones.The evidence supports the three-level risk-conditioned family, not discrete grouping specifically; map shape matters more than raw parameter count.Source: Small-budget matched continuous-map test · p. 21
  • SRTS-BCE has the lowest ECE at every tested calibration fraction, but its gap over TvA-TS shrinks from 0.73 pp at 100% to 0.36 pp at 10%.At 10%, ECE is 1.45 for SRTS-BCE, 1.81 for TvA-TS, and 1.95 for SMART+BCE; TvA-TS is the most rate-stable single-parameter fallback.Source: Table 21 · p. 21

Protocol-frozen cross-dataset small-budget follow-up (Tiny-ImageNet)

The follow-up evaluates risk-conditioned calibration across clean-data settings and natural shift, finding that small-budget separation replicates on Tiny-ImageNet but the preferred calibrator varies by regime and backbone. Corruption and large-scale evaluations further mark limits and ranking reversals rather than uniform superiority.

Protocol-frozen cross-dataset small-budget follow-up (Tiny-ImageNet)

  • At n=250, SRTS-BCE beat TvA-TS and SMART+BCE on frozen Tiny-ImageNet DeiT-S and Swin-T, with all four hierarchical CIs excluding zero.For DeiT-S, the SRTS−TvA-TS and SRTS−SMART+BCE intervals were [−1.36, −0.80] and [−0.73, −0.29]; for Swin-T they were [−1.01, −0.59] and [−1.26, −0.58]. Scalar win rates were 95% and 92%. On Swin-T, SMART+BCE was significantly better at 2,500 samples, while SRTS-BCE was significantly better at n=250; DeiT-S tied at 2,500. ViT-B/16 remained the excluded boundary cell, so this is a two-backbone replication, not a universality claim.Source: Protocol-frozen cross-dataset… · pp. 21–22

Fitted temperatures and routing stability

  • At n=250, SRTS-BCE retained 0% saturation and fallback, with empirical risk ordering monotone in 93% of fits, but strict temperature ordering in only 64%.The low-risk group temperature was volatile (median 0.17, IQR [0.13, 0.73]); under shift, the high-risk share rose from roughly uniform thirds on clean test to 44% at severity 1 and 66% at severity 5 while temperatures and thresholds stayed fixed.Source: Fitted temperatures and routing… · p. 22

CIFAR-100-C corruption-shift evidence

  • Under deployable CIFAR-100-C rerouting, SRTS-BCE had ECE15 4.63 versus SMART+BCE 4.75 and TvA-TS 5.72, but the full hierarchical gain was directional.Its severity-4–5 ECE was 6.79 versus SMART+BCE 7.03 and TvA-TS 8.38; its aggregate eAURC was 0.0431 versus 0.0441 and 0.0457. Re-quantizing thresholds degraded aggregate ECE15 from 4.63 ± 0.84 to 5.73 ± 1.08, supporting clean-anchored thresholds that let shifted samples move into the high-risk group.Source: CIFAR-100-C corruption-shift evidence · pp. 22–23

Deployment-facing selective metrics

  • On clean CIFAR-100 and CIFAR-100-C, SRTS-BCE and SMART+BCE reduced HCFP@0.95 versus TvA-TS, while selective risk at 80/90/95% coverage was nearly identical.Clean HCFP@0.95 was 7.95/9.56 versus 12.00; shifted HCFP@0.95 was 5.35/6.44 versus 7.97. At matched coverage, SRTS-BCE had the lowest AURC on both settings: clean 15.49 versus 18.38 for TvA-TS and 18.80 for HTS-BCE; CIFAR-100-C 62.24 versus 65.29 and 65.77 (×10^3).Source: Deployment-facing selective metrics · pp. 23–24

Where deployable rerouting helps, and why

  • Corruption benefits were conditional on transfer: per-corruption SRTS-BCE gains tracked miscalibration, while ConvNet rerouting improved on TvA-TS but its ordering against SMART+BCE depended on backbone.The per-corruption gain anti-correlated with NoTS ECE15 (Spearman ρ = −0.68, p = 0.001); SRTS-BCE beat both peers on 12 of 19 corruptions. On ResNet-50 and RegNetY-1.6GF, it improved over TvA-TS by 4.44 and 2.16 pp, respectively, but no clean-fit calibrator was uniformly safe under corruption.Source: Where deployable rerouting helps, and… · p. 24, ConvNet CIFAR-100-C Deployable Rerouting · pp. 24–25

Generated Tiny-C and CIFAR-10-C Collapse-Regime Diagnostics

  • Generated Tiny-ImageNet-C corruptions put every calibrator in severe collapse (ECE15 above 60%), whereas CIFAR-10-C was a tie between SRTS-BCE and SMART+BCE.On generated Tiny-ImageNet corruptions, ordering was consistent across three backbones and bootstrap-supported SRTS−SMART deltas were −0.68 [−0.75, −0.62] for ViT, −0.51 [−0.57, −0.45] for DeiT-S, and −0.54 [−0.59, −0.49] for Swin-T. On CIFAR-10-C, aggregate ECE15 was 3.00 versus 3.20, with a cell-level CI crossing zero [−0.55, +0.14].Source: Generated Tiny-C and CIFAR-10-C… · p. 25

ImageNet-C, ImageNet-Sketch, and ImageNet-scale boundaries

  • ImageNet evaluations mark boundaries: routing added no gain on fine-tuned ImageNet-100 or pretrained ImageNet-C under BCE, while clean pretrained calibration gains were modest.On ImageNet-Sketch, SRTS-BCE ECE15 was 1.41 ± 0.06 versus TvA-TS 1.44 ± 0.03, a 0.03 pp gain, while SMART+BCE had lower eAURC. On clean ImageNet-100, methods were statistically indistinguishable across three backbones; on clean pretrained ImageNet-1K, low-capacity BCE calibrators modestly improved on TvA-TS for three of four backbones. Under NLL, K=3 did not beat K=1 on pretrained ImageNet-C.Source: ImageNet-C, ImageNet-Sketch, and… · pp. 25–27

BCE-objective ImageNet-C evaluation

  • On pretrained ImageNet-1K under corruption and natural shifts, clean-fitted SRTS-BCE offered no improvement over scalar TvA-TS, and all compared clean-fitted calibrators worsened both metrics on ImageNet-R and ImageNet-A.On ImageNet-C, TvA-TS had the lowest ECE (2.47 ± 0.08) and severity-4–5 ECE (3.48 ± 0.12), versus 2.56 ± 0.23 for SRTS-BCE. On ImageNet-R and ImageNet-A, TvA-TS had lower mean ECE than SRTS-BCE by paired deltas of +0.66 [+0.65, +0.66] and +1.40 [+1.37, +1.42], respectively.Source: BCE-objective ImageNet-C evaluation · p. 27, Natural-shift boundary: ImageNet-R and… · pp. 27–28

Natural-shift boundary: ImageNet-R and ImageNet-A

  • On ImageNet-V2, BCE-objective calibrators reduce ECE15 from NoTS at 8.4–10.1 to 2.9–3.9, while SRTS-BCE's advantage over TvA-TS remains unresolved at 1,000 images.The BCE objective improves over NLL at both scalar and routed capacity on all three backbones, but every SRTS-BCE versus TvA-TS interval crosses zero.Source: Natural-shift boundary: ImageNet-R and… · p. 28

Backbone generalisation: DeiT-S and Swin-T

  • On CIFAR-100 DeiT-S and Swin-T, SRTS-BCE ECE15 is 0.94 and 1.04, respectively, below SMART+BCE at 1.17 and 1.13 and TvA-TS at 3.37 and 2.87.SRTS-BCE has lower point-estimate ECE than both comparators on all three seeds for each backbone; the Tiny-ImageNet extension shows SMART+BCE leads on Swin-T, while SRTS-BCE has a narrow mean edge on DeiT-S.Source: Backbone generalisation: DeiT-S and… · pp. 28–29

G Cross-Backbone Scope

  • Across nine fixed-K=3 clean dataset-backbone cells, SRTS-BCE has lower mean ECE in five and SMART+BCE in four, supporting backbone-level rather than dataset-level selection.Fine-tuned CIFAR-100 favours SRTS-BCE, while clean CIFAR-10 and Tiny-ImageNet require choosing between SRTS-BCE and SMART+BCE by backbone.Source: G Cross-Backbone Scope · p. 28, Backbone generalisation: DeiT-S and… · p. 29

CIFAR-10 clean calibration (stability check)

  • On clean CIFAR-10, SMART+BCE leads ViT-B/16 and Swin-T, while SRTS-BCE leads DeiT-S; on the two ConvNets, SMART-SoftECE has the lowest ECE15 but the worst and least stable AdaECE15.CIFAR-10 is treated as a near-saturated stability check, not a main contribution or corruption-shift claim.Source: CIFAR-10 clean calibration (stability… · pp. 29–30

Tiny-ImageNet clean calibration

  • On Tiny-ImageNet ViT-B/16, SMART+BCE ECE15 is 0.97 ± 0.22, below SRTS-BCE at 1.12 ± 0.17 and TvA-TS at 1.26 ± 0.18.SMART+BCE leads SRTS-BCE on all three seeds, reversing the CIFAR-100 ViT-B/16 ordering; the follow-up on DeiT-S and Swin-T finds the flip is backbone-dependent.Source: Tiny-ImageNet clean calibration · p. 30

Low-data fine-tuning check (protocol-frozen negative result)

The protocol-frozen low-training-data check did not support the hypothesis that larger residual miscalibration would favor SRTS-BCE: with the calibration split held fixed, lower-capacity calibrators ranked ahead. This adds a boundary to the regime map, alongside a Tiny-ImageNet follow-up with a budget-dependent ranking reversal.

Low-data fine-tuning check (protocol-frozen negative result)

  • At 10% training data, TvA-TS (1.37±0.51) ranked ahead of SMART+BCE (1.51 ± 0.29) and SRTS-BCE (1.71 ± 0.34); SRTS–SMART deltas were bootstrap-indistinguishable.At 25%, SMART+BCE 0.96 and TvA-TS 0.98 were near-tied ahead of SRTS-BCE 1.11.Source: Low-data fine-tuning check… · p. 30
  • The calibration split stayed at ≈2,500 samples while CIFAR-100 training data fell to 10% or 25%, so the negative result concerns training-data regime, not a smaller calibration budget.The protocol-frozen hypothesis that lower-data fine-tuning would favor SRTS-BCE was not supported.Source: Low-data fine-tuning check… · p. 30
  • The authors describe a regime boundary: heavily pretrained and under-trained heads favor scalar or continuous-margin calibrators, while routed gains concentrate in well-trained, fully fine-tuned models.The low-data residual is larger but apparently less signal-trackable.Source: Low-data fine-tuning check… · p. 31
  • At n=250, SRTS-BCE and the matched three-anchor PWLinear-3 were statistically unresolved on all three backbones, while linear and spline maps using the same risk score were significantly worse.Source: Table 25 · p. 32
  • On Tiny-ImageNet Swin-T, SMART+BCE was significantly better at 2,500 calibration samples, but the ordering reversed as the budget shrank.The protocol-frozen follow-up used three training seeds and 20 draws per budget.Source: Table 26 · p. 32

A conservative validation-only capacity selector

A repeated cross-validation one-standard-error rule selects K=3 only when its top-label-BCE gain over K=1 clears the uncertainty threshold. Across 21 regimes it reduced mean test-ECE excess against the per-seed best-of-2, though validation-only capacity selection remained imperfect.

A conservative validation-only capacity selector

  • Mean test-ECE excess was 0.09 pp against the per-seed best-of-2, versus 0.18 for always-K=3 and 0.59 for always-K=1.The selector retained 7 of 8 regimes where K=3 helped by >0.2 pp and avoided 2 of 3 regimes where it hurt by >0.2 pp.Source: A conservative validation-only… · p. 32, Table 29 · p. 33
  • The selector can still choose K=3 when K=1 is better on test in the low-data fine-tune, and tested validation-internal statistics did not detect that regression.Selection also cannot detect covariate shift absent from validation data.Source: A conservative validation-only… · p. 33
  • The fold-level one-standard-error variant made the same selections as the repeat-means rule on all 55 evaluated cells.On regimes used to form the rule, always-K=3 had lower average regret than either selector (0.048 vs. 0.061 pp), and selectors chose K=3 in 62% of cells where the oracle preferred K=1.Source: A conservative validation-only… · p. 33
  • On natural-shift datasets, selectors’ conservatism reduced average regret to 0.25 pp versus 1.44 pp for always-K=3, with worst-case regret 0.94 versus 5.70.No evaluated validation-only rule was reliable at this budget, so fixed K=3 remained the deployment choice.Source: A conservative validation-only… · p. 33

Extended limitations

This section qualifies SRTS-BCE’s gains by regime and metric, distinguishing evidence for a routing mechanism in BCE gains from guarantees about ECE and noting that comparisons with SMART+BCE are less conclusive than those with TvA-TS. It also covers validation-only capacity selection, routing-signal diagnostics, and comparisons with argmax-changing baselines, while noting that the supplied passages lack cell values for the cross-backbone selection and full-metric tables.

Extended limitations

  • SRTS-BCE’s advantage is regime- and metric-dependent: it favors fine-tuned and deployable CIFAR-100, while scalar TvA-TS favors heavily pretrained ImageNet-C and SMART+BCE wins Tiny-ImageNet ECE on ViT-B/16 and Swin-T.On clean CIFAR-100, classwise ECE does not distinguish calibrated maps (all 0.12 ± 0.01); smECE favors SRTS-BCE (1.08 vs. TvA-TS 1.56), while SMART+BCE leads AdaECE15. Reliable out-of-fold risk estimation is required, so routing can narrow or become unstable on very small splits.Source: Extended limitations · pp. 33–34

Pooled Scaling

  • The routing diagnostic supports objective-gradient signal: risk-router B/within-group variance exceeds the random control by 3–12× in ViT and ImageNet regimes, but falls below the floor and control on ResNet-56.The random partition sits at the finite-sample floor; the ratio diagnoses BCE gain, not ECE, which also depends on the BCE-to-ECE gap and finite-sample router noise.Source: Pooled Scaling · pp. 35, 37

Argmax-changing classical baselines

  • Under the ≈2,500-sample validation budget, regularized vector scaling and Dirichlet calibration are worse on ECE15 than scalar TS, SRTS-BCE, SMART+BCE, and TvA-TS.These argmax-changing methods occupy a different capacity regime from the constrained 10-scalar family; vector scaling also introduces class-dependent parameters and can change the argmax.Source: Argmax-changing classical baselines · p. 37

Bootstrap uncertainty

  • −0.62 pp mean clean CIFAR-100 ECE15 difference favors SRTS-BCE over TvA-TS, with a seed×sample hierarchical 95% CI of [−0.88, −0.36].SRTS-BCE matches SMART+BCE on clean ECE15: mean difference −0.03 pp, hierarchical 95% CI [−0.19, +0.12], not statistically distinguishable.Source: Bootstrap uncertainty · p. 38
  • −1.09 pp mean CIFAR-100-C ECE15 difference favors deployable SRTS-BCE over TvA-TS, with a full hierarchical 95% CI of [−2.02, −0.36].Against SMART+BCE, the CIFAR-100-C ECE15 difference is −0.12 pp, but its full hierarchical 95% CI [−0.33, +0.03] includes zero; the corruption-type eAURC interval is [−0.0012, −0.0008].Source: Bootstrap uncertainty · pp. 38–39
  • Cross-backbone SRTS-BCE-Select chooses K per seed by Cal-Select ECE15 from {1, 2, 3, 5, 10}, then refits on the full calibration split.The supplied table caption identifies the selected-K and test-ECE15 columns but does not include their values.Source: Table 50 · p. 46
  • Mean test-ECE excess over the per-seed best-of-2 was 0.09 for the one-standard-error selector, versus 0.18 for always-K=3 and 0.59 for always-K=1.The selector uses only the clean calibration split, choosing K=3 when repeated cross-validated top-label-BCE gain exceeds one standard error; it has no cross-regime fitted parameters.Source: Table 51 · p. 46
  • The risk router cleared the BCE grouping floor by 3–12× the random control in ViT and ImageNet regimes, while random partitions sat at the floor.This signal-to-noise diagnostic predicts BCE gain, not ECE; ImageNet-pretrained cells cleared the BCE floor yet SRTS lost in ECE.Source: Table 52 · p. 47
  • Under the matched ≈2,500-sample validation budget and val-NLL selection, vector and Dirichlet scaling had worse ECE15 than the argmax-preserving calibrators listed.These classical baselines change the predicted class and occupy a different capacity regime; the reference rows use an archived three-seed diagnostic subset.Source: Table 54 · p. 47
Loading 2609.38936v1…