PaperlayerView exact revision on arXiv

Source-linked AI summary

A Canary in the Neural Mine: Catching an Image Classifier's Silent, Confident Failures

Wonseok Shin, Songkuk Kim

arXiv:2610.33063v1cs.CVcs.AI

TL;DR

Deployed vision systems can silently propagate confident errors, and MSP overlooks feature variation outside its scalar confidence signal. The paper analyzes which feature-space information complements MSP and proposes CANARY to measure and fuse that signal. Across 385 shift cells, CANARY matches or improves MSP in 382 on e-AURC and 379 on AUROC, with gains concentrated on natural shifts.

  • Problem

    MSP, the standard failure predictor, collapses feature geometry and is first-order insensitive to variation orthogonal to its confidence gradient, while confident errors can pass silently in deployment.

    Source: §1 · p. 1

  • Method

    A factorial analysis identifies the complementary signal as class-conditional, non-parametric typicality on the off-manifold residual, which CANARY measures with per-class kNN distance and fuses with MSP.

    Source: §1 · p. 2

  • Results

    Across 385 shift cells, CANARY matches or improves MSP in 382 on e-AURC and 379 on AUROC, with worst-case regression of −0.0002 AUROC.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Fusion gains grow as baseline MSP quality falls, so clean-validation MSP quality forecasts the gain before deployment.

    Source: §1 · p. 2, §7 · p. 14

    CANARY is structurally blind to purely on-manifold confident failures, and the headroom law is an empirical regularity whose exact form may differ beyond the studied regime.

    Source: §7 · p. 14

Abstract

from arXiv · show

Deployed vision models auto-accept confident predictions, so high-confidence errors propagate silently. Maximum Softmax Probability (MSP), the standard failure predictor, has a structural blind spot: as a scalar read from the classifier head, it is first-order blind to feature variation orthogonal to its confidence gradient. Through a factorial analysis of feature-space detectors, we identify a conditional-redundancy principle explaining why they fail to complement MSP: the class-separability geometry they measure is already encoded in MSP and collapses under fusion. The signal that survives is class-conditional typicality on the off-manifold residual---and because this residual's distribution varies by backbone, from near-Gaussian to heavy-tailed, it must be estimated non-parametrically. We instantiate these findings as CANARY, a training-free detector that computes a per-class kNN distance on the PCA residual of the penultimate feature, percentile-calibrates it on validation data, and fuses it with MSP through a single validation-tuned weight. CANARY's benefit follows a headroom law: gains grow as baseline MSP quality falls, so a single clean-validation number forecasts the improvement. With every parameter selected on in-distribution data alone, CANARY matches or improves 382 of 385 shift cells (e-AURC; 379 on AUROC) across five backbones, with worst-case regression within seed noise ($-0.0002$ AUROC) and the largest gains on natural distribution shifts (ImageNet-A/R, ObjectNet)---where deployed models are most exposed.

1 Introduction

Confident errors can pass silently in deployed selective-classification systems, while MSP collapses feature geometry and misses variation orthogonal to its confidence signal. The paper identifies the feature-space signal that complements MSP and introduces CANARY, whose gains are largest when baseline MSP is weaker.

1 Introduction

  • MSP is first-order insensitive to feature variation orthogonal to its confidence gradient, leaving a structural blind spot for failure prediction.On ViT-B/16, this blind spot spans 767 of 768 feature dimensions.Source: §1 · pp. 1–2
  • CANARY’s AUROC gains rise as clean-validation MSP AUROC falls across synthetic and natural shifts, with the largest gains on natural shifts.The headroom relationship is shown across ImageNet-C, ImageNet-A, and ImageNet-R.Source: Figure 1 · p. 2, §1 · p. 2
  • Factorial analysis finds class-separability geometry redundant with MSP under fusion; the surviving signal is non-parametric, class-conditional off-manifold typicality.The residual distribution varies by backbone, while global typicality breaks under natural shifts.Source: §1 · pp. 2–3
  • CANARY estimates per-class kNN distance on the PCA residual of penultimate features, percentile-calibrates it, and fuses it with MSP using one validation-tuned weight.It is a training-free, post-hoc detector.Source: §1 · pp. 2–3
  • Across 385 shift cells, CANARY matches or improves MSP in 382 on e-AURC and 379 on AUROC, with worst-case regression of −0.0002 AUROC.Its gains concentrate on natural shifts, including ImageNet-A, ImageNet-R, and ObjectNet.Source: §1 · p. 3

2 Related Work

Prior failure-prediction work includes logit corrections, activation-shaping heuristics, feature-space detectors, and dedicated predictors. The paper frames a gap in understanding which feature signal can add information when fused with MSP.

2 Related Work

  • Logit-based corrections inherit MSP’s blind spot, while activation-shaping heuristics can degrade when transferred from CNNs to structurally different Vision Transformers.Examples include MaxLogit, Energy, Doctor, and GEN for logit corrections, and ReAct, ASH, and DICE for activation shaping.Source: §2 · p. 3
  • Feature-space methods designed for OOD detection often underperform MSP on failure prediction, and heuristic combinations assume independent signals despite conditional redundancy.The paper argues that a feature-based partner contributes through its off-manifold component; SIRC instead combines softmax with a class-agnostic feature norm.Source: §2 · pp. 3–4
  • Dedicated predictors such as TrustScore, ConfidNet, and OpenMix score misclassifications, but none isolates the signal that survives fusion with MSP.ConfidNet and OpenMix train auxiliary heads, whereas TrustScore is post-hoc.Source: §2 · p. 4

3 Preliminaries

The preliminaries define selective classification as accepting predictions above a confidence threshold and evaluate failure predictors by how well they rank correct predictions above incorrect ones. They also specify MSP and the ranking-focused evaluation metrics.

3 Preliminaries

  • In selective classification, a model accepts its prediction when confidence reaches threshold τ and abstains otherwise; an ideal score ranks correct predictions above incorrect ones.As τ increases, errors should be rejected first.Source: §3 · p. 4
  • MSP is the highest class probability from the softmax distribution, computed from logits produced by a linear head over penultimate features.Because it uses only the logit vector, it collapses the higher-dimensional feature geometry.Source: §3 · p. 4
  • The evaluation uses e-AURC as its primary ranking metric because raw AURC is confounded by base accuracy; lower e-AURC is better.AUROC is reported as a rank-sanity check, while AUGRC provides secondary gaming-resistant evidence.Source: §3 · p. 4

4 What Complements MSP? A Factorial Analysis of the Feature Residual

The factorial analysis isolates what feature geometry contributes beyond MSP, which is first-order blind to variation transverse to its confidence gradient. Under fusion, class-separability geometry is redundant; non-parametric, class-conditional typicality on the residual is the signal that survives.

4 What Complements MSP? A Factorial Analysis of the Feature Residual

  • The controlled factorial experiment separates feature-based detectors’ basis, distance, and reference-conditioning choices to identify what survives fusion with MSP.Its resulting blueprint is non-parametric, class-conditional typicality on the orthogonal residual.Source: §4 · p. 5, §4.5 · p. 8

4.1 The Orthogonal Blind Spot of Linear Heads

  • MSP’s scalar confidence is first-order blind to the feature-space hyperplane orthogonal to its gradient—767 of 768 dimensions for ViT-B/16.The gradient varies by input, so this is a local necessary condition for a complementary signal, not evidence about which transverse component survives fusion.Source: §4.1 · p. 5

4.2 The Conditional-Redundancy Principle

  • Conditioning’s standalone AUROC advantage (∆5.6 pp) collapses to ≈0.2 pp under MSP fusion, while per-class conditioning remains positive on ImageNet-A across backbones.Global conditioning weakens and reverses on ViT-B/16, supporting class-conditional residual typicality for natural shifts.Source: §4.2 · p. 5, Figure 2 · p. 6

4.3 Distance: The Necessity of a Non-Parametric Estimator

  • Residuals range from near-Gaussian on ViT-B/16 and DINOv2 to heavy-tailed on Swin-B and CNNs, so no parametric density transfers uniformly; kNN is uniformly near-best.kNN is within ∼0.01 AUROC of the best parametric density on near-Gaussian backbones and is the single best estimator where tails are heavy.Source: §4.3 · pp. 6–7, Figure 3 · p. 6

4.4 Basis: PCA and the Coordinate Problem

  • Under per-class kNN, PCA, AE, and VAE perform within 0.2 pp, while overcomplete TopK-SAE trails by ∼2 pp.PCA is adopted as training-free, deterministic, and parameter-free; its fused AUROC changes by at most 0.0007 with cosine-kernel PCA across four backbones.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5 Conditioning: Class-Conditional vs. Global Typicality

  • On structured natural shifts, global residual scoring can be misled, while per-class scoring retains robustness; per-class residual variance correlates moderately with failure rate (Spearman 0.44–0.64).For synthetic near-isotropic corruptions, a global score suffices.Source: §4.5 · p. 7

5 Method

CANARY scores residual typicality among correctly classified training examples, then calibrates that score and MSP on correctly classified validation inputs. It combines the calibrated signals with a single weight tuned by cross-fitting on clean validation data.

5.1 Low-Rank PCA Residual

  • CANARY fits only on correctly classified training examples and uses a low-rank PCA residual of the frozen classifier’s penultimate feature.The residual projects centered features onto the orthogonal complement of the top-K PCA directions; K is fixed at 512.Source: §5 · p. 8, §5.1 · p. 8

5.2 Per-Class kNN Distance

  • For each predicted class, CANARY scores the residual by its Euclidean distance to the 100th nearest neighbor in that class’s correctly classified training residuals.The reference sets store residuals and require no covariance fitting or auxiliary training.Source: §5.2 · p. 8

5.3 Percentile Calibration

  • CANARY maps residual distance and MSP through their respective validation-correct empirical CDFs, producing p-values on a common footing.A higher residual score marks a more typical, likely-correct input; the fixed validation references require no test labels.Source: §5.3 · pp. 8–9

5.4 Weighted-Fisher Fusion

  • A weighted log-linear combination fuses the calibrated MSP and residual p-values, with its single weight selected by cross-fitting on clean validation data.The weight is tuned using AUROC, averaged across five random seeds, then frozen for shift evaluations; the score is used for ranking rather than as an independence-based hypothesis test.Source: §5.4 · p. 9

6 Experiments

Across diverse backbones and shifts, CANARY generally matches or improves on MSP, with larger gains when baseline MSP is weaker. Experiments also test transfer across label spaces and representations and report low inference overhead.

6.1 Setup

  • The evaluation spans ImageNet shifts, held-out ObjectNet, CIFAR-100 and its corruptions, multiple backbone families, and 19 post-hoc baselines.e-AURC is the primary metric; AUGRC and AUROC are secondary metrics.Source: §6.1 · pp. 9–10

6.2 Experimental Results

  • CANARY outperforms all 19 baselines on e-AURC, AUGRC, and AUROC except for a within-seed-noise near-tie with SIRC (Residual) on ResNet-50.The e-AURC reduction over MSP ranges from 9.5 (×103) on EfficientNet-B4 to 0.4 on ViT-B/16.Source: §6.2 · p. 10, Table 1 · p. 11
  • Fusion gains track MSP headroom across architectures and shifts, and within ViT-B/16 the weaker-MSP fine-tuning variant gains more than the stronger one.Across shifts, the AUROC gains are +0.42 vs. +0.08 pp; on ObjectNet they are +1.04 vs. ≈0.Source: §6.2 · pp. 11–12
  • Across 385 shift cells, CANARY matches or exceeds MSP on 382 e-AURC cells (379 AUROC), and Table 2 identifies it as the only detector with a bounded worst case.The 14 CIFAR-100 exceptions across 150 cells are also within a −0.0004 AUROC margin on WRN-28-10.Source: §6.2 · p. 10, Table 2 · p. 12, CIFAR-100: the never-worse property… · p. 12

CIFAR-100: the never-worse property transfers to a different label

  • On CIFAR-100, CANARY matches or exceeds MSP on 136 of 150 cells; all 14 exceptions on WRN-28-10 are within a −0.0004 AUROC margin.The gains are small because the CNN residual is nearly uninformative, with standalone residual AUROC ≈0.55–0.57.Source: CIFAR-100: the never-worse property… · p. 12, Table 3 · p. 13
  • On self-supervised DINOv2, CANARY improves over MSP on every shift cell, with gains increasing with corruption severity.SIRC (L1) has a marginally higher DINOv2 shift-mean, 0.8550 vs. 0.8538, but trails CANARY on all five main backbones.Source: CIFAR-100: the never-worse property… · p. 13, Table 3 · p. 13
  • Across ViT scales, CANARY never falls meaningfully below MSP, though cells at or above MSP decrease from 77 to 74 to 66 as headroom shrinks.On ViT-L/16, 11 of 77 cells marginally trail MSP, all within seed noise; the worst case is −0.0005 AUROC.Source: CIFAR-100: the never-worse property… · p. 13

6.4 Computational Cost

  • On ViT-B/16, CANARY adds ≈50 µs per unbatched query and averages under 2 µs batched on an NVIDIA GeForce RTX 4080.Fitting requires ≈1.9 TFLOP given extracted features, and the reference set occupies ≈0.6 GB.Source: §6.4 · p. 13

7 Conclusion

The conclusion presents CANARY as a training-free way to measure class-conditional typicality on off-manifold feature residuals and fuse it with MSP. Its empirical gains follow the headroom law, but purely on-manifold confident failures remain beyond its reach.

7 Conclusion

  • Conditional redundancy explains why class-relevant feature geometry adds no novel information under MSP fusion; the off-manifold component supplies the complementary signal.CANARY measures class-conditional, non-parametric typicality on the PCA residual and fuses it with MSP using one validation-tuned weight.Source: §7 · p. 14
  • CANARY’s gains are largest where MSP is weakest, while its performance degrades gracefully toward MSP where the baseline is strong.The reported evaluation uses a deployment-realistic protocol without retraining or OOD-data exposure.Source: §7 · p. 14
  • The empirical never-worse finding and headroom law lack rigorous mathematical formalization, and the law’s exact form may differ beyond the nine backbones and shift families studied.CANARY is structurally blind to purely on-manifold confident failures, though many challenging ImageNet-A samples retain an off-manifold component.Source: §7 · p. 14
  • Future work includes formalizing when never-worse holds, deriving a headroom-law bound, adding external signals for on-manifold failures, and extending to dense prediction, adversarial settings, and medical imaging.Source: §7 · p. 14

A Non-Gaussianity of the Residual Across All Backbones

Residual distributions differ across backbones: some are heavy-tailed, while others are near-Gaussian, with bimodality varying independently.

A Non-Gaussianity of the Residual Across All Backbones

  • Swin-B and the CNNs have heavy tails even in typical classes, while ViT-B/16 remains near-Gaussian and typical DINOv2 classes are near-Gaussian despite a bimodal minority.DINOv2’s extreme class is explicitly bimodal, while 24% of its classes are flagged as bimodal.Source: A Non-Gaussianity of the Residual… · p. 18
  • The regime summary places Swin-B and the CNNs in the heavy-tailed group and ViT-B/16 and DINOv2 near-Gaussian, while DINOv2 has the highest bimodal fraction.Table 4 summarizes per-class residual statistics across backbones.Source: A Non-Gaussianity of the Residual… · p. 18, Table 4 · p. 18
  • Figure 5 contrasts each backbone’s most non-Gaussian class with its median class using standardized residual densities, fitted Gaussians, and excess kurtosis.The densities are plotted on a logarithmic y-axis.Source: A Non-Gaussianity of the Residual… · p. 18, Figure 5 · p. 19

B Density-Model Ablation

The density-model ablation finds that kNN adapts across backbone-dependent residual regimes; further tests explain why reconstruction quality and multimodality alone do not determine detector performance.

B Density-Model Ablation

  • On heavy-tailed backbones, kNN leads the best parametric density; on near-Gaussian backbones, it matches the best within ∼0.01 AUROC and gains 0.003–0.022 AUROC over shared-covariance Mahalanobis.The best parametric choice flips by regime: full covariance works best on near-Gaussian backbones but overfits heavy tails.Source: B Density-Model Ablation · p. 20

C Why the TopK SAE basis underperforms

  • A smaller TopK SAE reconstruction residual still yields lower AUROC than PCA: 0.7264 versus 0.7662 at matched K = 256.PCA leaves 27.8% of feature variance unexplained, compared with 26.2% for the SAE; against deployed K = 512 PCA (AUROC 0.747), the SAE still trails by about 2 pp.Source: C Why the TopK SAE basis underperforms · p. 21, Figure 6 · p. 21

D Per-Class Residual Variance Predicts Failure

  • Per-class residual variance and validation failure rate have consistently positive, moderate correlations across backbones, ranging from 0.44 to 0.64.The analysis measures residual variance on ID-correct training samples and class-specific validation failure rates across 1,000 classes; a global reference pools away this class-specific information.Source: D Per-Class Residual Variance Predicts… · p. 22, Table 6 · p. 22

E Robustness to the Basis Rank K

CANARY’s fused score is robust to PCA rank, and the results across shifts and backbones support its headroom-based performance pattern, including on ObjectNet.

E Robustness to the Basis Rank K

  • Across K/D from 0 to 0.9, CANARY’s fused AUROC varies by at most 0.0021; a global ViM-style residual norm instead swings by 0.014–0.151 across backbones.The comparison averages AUROC across 77 shift cells; CANARY’s K-insensitivity also holds on the large-gain Swin-B.Source: E Robustness to the Basis Rank K · p. 22, Figure 7 · p. 23

F Backbone Checkpoints and Hyperparameters

  • The backbone rule fixes k = 100 and chooses K = min(512, D−128), retaining at least a 128-dimensional residual rather than tuning these per backbone.Table 7 lists the checkpoints, feature dimensions, basis ranks, and cross-fitted fusion weights.Source: F Backbone Checkpoints and… · p. 23, Table 7 · p. 23

G.1 Main Backbones

  • Tables 8–13 report all 20 detectors on the five main backbones, with ImageNet-C grouped into four corruption groups and ImageNet-A/R reported as full sets.Their captions identify the listed backbone tables, but do not provide cell-level outcomes in the supplied passages.Source: §G.1 · p. 24, Table 8 · p. 24, Table 10 · p. 25

G.2 ObjectNet

  • CANARY matches or exceeds MSP on ObjectNet across all five main backbones, and no baseline shares this never-worse property.Table 14 reports failure prediction on the held-out ObjectNet set of 16,825 images.Source: Table 14 · p. 27, §G.2 · p. 27

G.3 DINOv2 (self-supervised)

  • Table 15 reports DINOv2 results by ImageNet-C corruption group and on ImageNet-A/R, among all 20 detectors.The supplied passage gives the table’s scope, not cell-level outcomes.Source: §G.3 · p. 28, Table 15 · p. 28

G.4 CIFAR-100

  • Tables 16–17 cover CIFAR-100-C for two CIFAR-native backbones, reporting corruption groups and overall average in a label space disjoint from ImageNet.The tables report e-AURC ×10^3; the supplied passages specify scope but not outcomes.Source: §G.4 · p. 28, Table 16 · p. 29, Table 17 · p. 29

H SIRC per-backbone behavior

  • SIRC (Residual)’s top-1 results concentrate on ResNet-50—31 of 35 e-AURC cells and 51 of 56 AUROC cells—while CANARY has top-1 cells on every backbone.On ResNet-50, SIRC (Residual) and CANARY tie within seed noise on both metrics.Source: H SIRC per-backbone behavior · p. 30, Table 18 · p. 30

I The Headroom Law on ObjectNet

  • On ObjectNet, the headroom-law relationship reproduces: CANARY’s AUROC gain over MSP is plotted against validation MSP AUROC and has the same negative slope as the Figure 1 panels.Figure 8 uses each backbone’s ImageNet-1k validation MSP AUROC as the forecastable x-axis.Source: Figure 8 · p. 30, I The Headroom Law on ObjectNet · p. 30

J Comparison with Kernel PCA

The Kernel PCA comparison tests reconstruction error as a detector, in fusion with MSP, and as CANARY’s basis; the model-scale results show gains tracking available MSP headroom.

J Comparison with Kernel PCA

  • Fused KPCA reconstruction error regresses below MSP in 6 of 8 cases by up to 0.027 AUROC, while CANARY improves over MSP across all four backbones.CoP+MSP slightly exceeds CANARY only on DINOv2, 0.8553 versus 0.8538; the comparison uses the same fusion rule.Source: Table 19 · p. 31, J Comparison with Kernel PCA · p. 31
  • KPCA’s native multiplicative fusion with MSP drops AUROC by 5.6–10.6 pp on every backbone; its Energy variant falls 6.8–14.8 pp below MSP.The cited explanation is that class-agnostic reconstruction error perturbs the within-distribution confidence ranking.Source: J Comparison with Kernel PCA · pp. 31–32, Table 20 · p. 32
  • Replacing PCA with the cosine-kernel basis changes CANARY’s AUROC by at most 0.0007, with no tangible benefit from the kernel’s non-linearity.The per-class kNN, calibration, and fusion are held constant.Source: Table 21 · p. 32, J Comparison with Kernel PCA · p. 32

K Model scale: ViT-S/16 and ViT-L/16

  • CANARY improves on all 77 shift cells for ViT-S, while on ViT-L it marginally trails MSP on 11 cells with a worst-case drop of −0.0005 AUROC, within seed noise.The ViT-L mean improvement is +0.10 pp; its residual is more correlated with MSP on validation (Pearson +0.52).Source: K Model scale: ViT-S/16 and ViT-L/16 · p. 33, Table 22 · p. 33
Loading 2610.33063v1…