PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialUQ: Post-Hoc Uncertainty Quantification from Spatial Consistency in Black-Box Vision Models

Md Kawsher Mahbub, Milon Biswas, Mirza Niaz Morshed, Wei Yu

arXiv:2610.09498v1cs.CVcs.AIcs.LG

TL;DR

Frozen clinical vision models need inference-time failure detection without access to internals or ground truth. SpatialUQ scores disagreement between global and spatial-crop predictions, achieving 0.784 AUC versus 0.664 for MC-Dropout on NIH ChestX-ray14 with fewer forward passes.

  • Problem

    Frozen black-box vision models need inference-time failure detection without ground truth, while many existing uncertainty methods require training changes, compute, model internals, or calibration data.

    Source: §1 · p. 1

  • Method

    SpatialUQ uses Jensen-Shannon divergence between global output probabilities and the mean predictions from five fixed spatial crops in six deterministic passes.

    Source: §1 · p. 1

  • Results

    On NIH ChestX-ray14, MUS achieves 0.784 AUC versus 0.664 for MC-Dropout (p < 10^-6) using 6 versus 30 forward passes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MUS is suited to low-cost failure detection for overconfident multi-label classifiers under moderate shift when retraining is infeasible.

    Source: §6 · p. 8

    MUS performs poorly on small focal lesions such as nodules, and generalization beyond radiography and RGB natural images remains untested.

    Source: §5.7 · p. 7, §6 · p. 8

Abstract

from arXiv · show

Clinical vision models are often deployed as frozen black boxes with no access to internals, retraining, or ground truth at inference time. We introduce \textbf{SpatialUQ}, a post-hoc uncertainty method using only output probabilities. It measures the Jensen-Shannon divergence between the global prediction and the mean of five fixed spatial crops in six deterministic forward passes. The premise is simple, trustworthy predictions are spatially consistent. On NIH ChestX-ray14 (DenseNet-121, $N{=}25{,}596$), our Multicrop Uncertainty Score (MUS) reaches $0.784$ failure-detection AUC versus $0.664$ for MC-Dropout ($p{<}10^{-6}$) at one-fifth the compute, with native calibration ($\text{SCE}{=}0.049$ vs.\ $0.127$ for $\ell_1$), the best-calibrated among methods above 0.78 AUC. A supervised fusion of MUS with entropy, confidence, and $\ell_1$ reaches $0.832$, outperforming a five-member ensemble ($0.813$). MUS scales with model quality, reaching $0.899$ with BiomedCLIP ($\rho = 0.846$), while this relationship remains meaningful in-distribution ($\rho = 0.523$) but breaks down under severe distribution shift (VinBigData, $\rho = 0.027$). MUS is well-suited to diffuse findings but is less dependable for small focal lesions such as nodules. Code and experimental materials are publicly available at https://huggingface.co/datasets/kawsher11/SpatialUQ.

1 INTRODUCTION

SpatialUQ addresses inference-time failure detection for frozen medical vision models using only output probabilities. It measures spatial inconsistency, with strong NIH results and limitations under severe shift and for some spatially uniform failures.

1 INTRODUCTION

  • Frozen black-box models limit existing uncertainty methods: some require training changes, added compute, model internals, feature distributions, or calibration data.The introduction identifies geometric evidence as largely unexplored by related consistency and augmentation-sensitivity methods.Source: §1 · p. 1
  • SpatialUQ measures Jensen-Shannon divergence between the global prediction and the mean prediction from five fixed spatial crops in six deterministic passes.The signal uses output probabilities without model internals, sampling, or reference data.Source: §1 · p. 1
  • MUS reaches 0.899 AUC with BiomedCLIP (ρ = 0.846), extending the approach to frozen foundation models.The introduction describes MUS as natively calibrated and usable without architectural access.Source: §1 · p. 1
  • The MUS–model-quality relationship remains meaningful in-distribution (ρ = 0.523) but collapses under severe shift on VinBigData (ρ = 0.027).A small labeled calibration sample is proposed as a deployment diagnostic.Source: §1 · p. 2
  • On NIH ChestX-ray14, MUS reaches 0.784 AUC versus 0.664 for MC-Dropout (p < 10^-6) using 6 versus 30 forward passes.Fusion reaches 0.832 AUC, exceeding a five-member ensemble (0.813) without retraining.Source: §1 · pp. 1–2
  • Spatial inconsistency is complementary rather than universal: it excels on overconfident multi-label tasks but degrades when failures are spatially uniform.The introduction also notes lower dependability for small focal lesions such as nodules.Source: §1 · p. 2

2 RELATED WORK

Related work spans stochastic uncertainty methods, test-time augmentation, post-hoc feature scores, and calibration approaches. SpatialUQ instead uses spatial variation in output probabilities as a scalar failure-triage signal.

2 RELATED WORK

  • MC-Dropout requires dropout layers during training, Bayesian approximations can scale poorly, and deep ensembles multiply compute.A chest X-ray benchmark cited in the passage finds no single method universally optimal.Source: §2 · p. 2
  • Test-time augmentation measures variation across transformed inputs, while SpatialUQ tests prediction stability across spatial sub-regions as evidence-localization consistency.Geometric and photometric augmentations can confound spatial ambiguity with color or contrast sensitivity.Source: §2 · p. 2
  • Feature- and logit-based scores such as Mahalanobis, DDU, energy, and ODIN use training-set feature distributions or logits.Conformal prediction constructs prediction sets rather than ranking inputs by failure probability.Source: §2 · p. 2
  • SpatialUQ produces a scalar triage score and applies JSD between global and local spatial predictions as a failure-detection proxy without model internals.The passage describes this geometrically grounded interpretation as, to the authors' knowledge, the first such use of JSD.Source: §2 · p. 2

3 OUR APPROACH

SpatialUQ operationalizes spatial consistency by comparing global and local predictions from fixed image crops. It offers an unsupervised JSD score, optional supervised fusion, and a bound connecting the score to probability shifts.

3.1 Spatial Decomposition

  • SpatialUQ decomposes each input into four non-overlapping quadrants and one overlapping center crop, fixed across inputs and architectures.Each crop is upsampled before inference, alongside one pass on the original image.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 Uncertainty Score via Jensen-Shannon Divergence

  • The uncertainty score is the JSD between the global prediction and the aggregated local prediction; it is bounded in [0, log 2] and symmetric.The approach specifies categorical JSD for single-label tasks and Bernoulli JSD for multi-label tasks.Source: §3.2 · p. 3
  • The deterministic pipeline uses one global and five crop passes, averages crop predictions, and returns a scalar score without additional overhead beyond those six passes.The score is applied to medical multi-label datasets, ImageNet classification, and MS COCO detection.Source: §3.2 · p. 3

3.3 Linear Fusion (Supervised Extension)

  • Optional logistic fusion combines MUS with entropy, confidence, and ℓ1 distance; MUS itself requires no labels, while fusion fits weights on a labeled hold-out.The hold-out is the NIH validation set or, in zero-shot settings, a random 10% of the test set.Source: §3.3 · pp. 3–4

3.4 Information-Theoretic Bound

  • Lemma 1 bounds mean absolute global–local class-probability shift using the Bernoulli-JSD score, with the per-class bound tight as probabilities approach 1/2.The bound concerns probability shift; failure prediction is supported separately by positive, significant correlations with per-image Brier score across medical benchmarks.Source: §3.4 · p. 4

4 EXPERIMENTAL SETUP

The evaluation spans medical and natural-image classification and object detection, with failure labels and metrics defined for cross-method comparison. It includes multiple model architectures and post-hoc, augmentation-based, feature-based, and ensemble baselines.

4.1 Datasets and Tasks

  • Evaluation covers NIH ChestX-ray14, CheXpert, VinBigData, ImageNet-1k, and MS COCO across medical and natural-image classification and object detection.NIH includes three independently trained architectures, while other models include frozen-backbone and zero-shot evaluations.Source: §4.1 · p. 4

4.2 Failure Definition

  • A prediction is labeled a failure when its per-image Brier score exceeds the test-set 75th percentile, fixing the failure rate at 25%.The passage states that the Brier score uses ground-truth labels and is epistemically independent of MUS.Source: §4.2 · p. 4

4.3 Training Protocol

  • NIH models use two-stage fine-tuning, first with a frozen backbone and then with the backbone unfrozen; ImageNet and COCO models are evaluated zero-shot.The NIH protocol uses BCE with label smoothing and class-balanced positive weights.Source: §4.3 · p. 4

4.4 Baselines

  • Baselines include output-space, crop-based, augmentation-based, feature/logit-based, and ensemble uncertainty methods on the same frozen model without retraining.The compared methods include MC-Dropout with 30 passes and five-model deep ensembles.Source: §4.4 · pp. 4–5

4.5 Evaluation Protocol

  • AUC is the primary binary failure-detection metric, with AUPR, AURC, E-AURC, FPR, and calibration also reported.Metrics use 95% bootstrap confidence intervals; pairwise AUC significance uses DeLong tests and two-tailed bootstrap p-values.Source: §4.5 · p. 5

5 RESULTS

SpatialUQ detects NIH test-set failures using six deterministic passes, with strong performance also reported for frozen foundation models. Its native calibration supports deployment when labeled hold-out data are unavailable.

5.1 NIH ChestX-ray14: In-Distribution Failure Detection

  • 0.784 AUC with six deterministic passes beats MC-Dropout’s 0.664 (Δ=+0.119, p<10−6) while reducing wall-clock time 5×.Fusion reaches 0.832 AUC, exceeding a five-member ensemble at 0.813 (Δ=+0.019, p<10−6).Source: Figure 2 · p. 5, §5.1 · p. 5

5.2 Extension to Frozen Foundation Models

  • MUS reaches 0.825 (ρ=0.631) with CLIP ViT-B/32 and 0.899 (ρ=0.846, p<10−6) with BiomedCLIP; fusion reaches 0.883/0.925.MUS is better calibrated than ℓ1 on these models (SCE 0.133 vs. 0.232).Source: §5.2 · p. 5

5.3 Calibration as a Deployment Signal

  • MUS is natively calibrated, with SCE 0.049 versus 0.127 for raw ℓ1; labeled hold-out data allow isotonic regression to calibrate ℓ1 (0.006/0.789) and fusion (0.004/0.830).Source: §5.3 · p. 5, §5.1 · p. 5

Method AUC 95% CI SCE

Across NIH, external medical datasets, and natural-image tasks, failure-detection performance depends on the uncertainty signal and setting. Ablations, transfer results, and triage analyses identify both complementary signals and limits for focal lesions.

Method AUC 95% CI SCE

  • 0.832 AUC for fusion exceeds the five-member ensemble’s 0.813, while unsupervised MUS reaches 0.784 AUC with SCE 0.049.ℓ1 reaches 0.790 AUC with SCE 0.127; the reliability diagram compares MUS with baseline methods.Source: Method AUC 95% CI SCE · p. 6, Figure 3 · p. 6

5.4 Ablation: Fusion Components and Spatial Design

  • Removing entropy drops fusion AUC by 0.044, compared with 0.005 for MUS, 0.007 for ℓ1, and 0.001 for confidence; five fixed crops are optimal.Nine crops add 0.007 AUC at 67% additional cost, while three crops lose 0.007 and 1,000 random placements reach 0.739.Source: §5.4 · p. 6, Table 2 · p. 6

5.4.1 Spatial vs. Photometric Perturbation

  • 0.784 AUC for spatial MUS exceeds photometric JSD’s 0.755 at six passes, although photometric JSD has slightly better calibration (SCE 0.043 vs. 0.049).The authors attribute the AUC advantage to the perturbation’s geometric structure, not the divergence measure.Source: §5.4.1 · p. 6

Zero-Shot Generalization to External Medical Datasets

  • On CheXpert, MUS reaches 0.708 AUC versus MC-Dropout’s 0.599; under severe VinBigData shift, MC-Dropout leads 0.764 to 0.614.MUS’s correlation with error falls from ρ=0.298 on CheXpert to ρ=0.027 on VinBigData as failures become spatially uniform.Source: Zero-Shot Generalization to External… · p. 6, Table 3 · p. 6

Natural Image Classification and Object Detection

5.7 Clinical Triage Simulation

  • Referring the 9.2% highest-MUS images recovers 20% of Brier-defined failures at 54.5% precision; retained Brier falls to 0.507 at 30% rejection.MUS performs best for Pneumonia (JSD AUC 0.979) and Edema (0.971), but poorly for Nodule (0.368), Fibrosis (0.553), and Mass (0.558); a 16-crop extension raises Nodule AUC to 0.589. Figure 4 plots retained-set Brier against rejection rate; Figure 5 compares top-to-bottom decile Brier ratios, with little coupling for Nodule, Mass, and Fibrosis.Source: §5.7 · p. 7, Figure 4 · p. 7

6 DISCUSSION

MUS is most useful when multi-label classifiers are overconfident and confidence is weak, but spatial inconsistency is not a universal failure signal. Its calibration and shift robustness come with defined operating boundaries.

Complementarity and Regime Dependence

  • On NIH DenseNet-121, confidence has AUC 0.401, while MUS is most useful in overconfident multi-label settings and is complementary to confidence on well-calibrated single-label tasks.On ImageNet, confidence dominates and fusion combines the two signals.Source: §6 · p. 8

Uncertainty type and OOD positioning

  • MUS’s correlation with model quality can serve as a pre-deployment diagnostic: ρ ≲0.15 flags severe shift, as on VinBigData (ρ = 0.027), while in-distribution correlation remains meaningful (ρ = 0.523).MUS is framed as an epistemic signal of learned spatial grounding, rather than irreducible label noise.Source: §6 · p. 8

Why MUS over ℓ1?

  • MUS has slightly lower AUC than ℓ1 (0.784 vs. 0.790; ∆= 0.006; p < 10−6), but lower SCE (0.049 vs. 0.127) and better VinBigData AUC (0.614 vs. 0.583).Isotonic regression lowers ℓ1 SCE to 0.006 but requires labeled hold-out data; fusion’s SCE of 0.246 is attributed to its uncalibrated entropy component.Source: §6 · p. 8

Limitations

  • MUS degrades under severe domain shift, has limited sensitivity to small focal lesions, and can weaken with ViT global attention or globally pooled generative VLMs; fusion also requires a labeled hold-out set.The authors identify low-cost failure detection for overconfident multi-label classifiers under moderate shift as the primary operating regime; generalization beyond radiography and RGB natural images remains untested.Source: §6 · p. 8

7 FINAL REMARKS

SpatialUQ offers inference-time failure detection using six deterministic forward passes and output probabilities alone. Its performance and diagnostic have a defined scope, and broader validation remains future work.

7 FINAL REMARKS

  • SpatialUQ uses six deterministic forward passes and output probabilities alone, without model internals, labels, or retraining.The conclusion reports AUCs of 0.784 on DenseNet-121 and 0.899 on BiomedCLIP, with ρ = 0.846.Source: §7 · p. 8
  • The ρ ≲0.15 diagnostic is presented as a pre-deployment check for observed distribution-shift failure regimes.Future work includes broader validation on MIMIC-CXR, other modalities, temporal shift, population-level AURC, and intersectional subgroup analyses.Source: §7 · p. 8
  • The authors used generative AI for writing assistance and checking reported numbers against experimental code, and state that authors reviewed the AI-assisted content.They report no AI use to generate experimental results, fabricate data, or conduct literature surveys.Source: AI use statement · p. 9

CHECKLIST

The checklist records the paper’s reproducibility materials, theoretical assumptions and proof location, and backbone evaluation setup.

A PROOF OF LEMMA 1

  • Lemma 1 states its assumptions, and the full proof is in Appendix A; the bound concerns probability shift, while its connection to failure detection is empirical.The checklist specifies pc, qc ∈ [0, 1] and JSD in nats.Source: CHECKLIST · p. 11

CHECKLIST

  • Code, data splits, and fixed seeds are supplied in the supplement; the implementation notebooks are also publicly available, and the NIH split and training protocol are documented.The checklist reports a patient-wise 73,891/12,633/25,596 split and experiments on a single NVIDIA Tesla T4 GPU.Source: CHECKLIST · pp. 11–12

B BACKBONE CLASSIFICATION PERFORMANCE

  • On NIH ChestX-ray14, DenseNet-121 has mean AUC 0.7885, EfficientNet-B4 0.7347, and ViT-B/16 0.7921; these backbones are used as frozen black boxes for subsequent uncertainty evaluation.The checklist says the models’ training details and experimental results are documented.Source: B BACKBONE CLASSIFICATION PERFORMANCE · pp. 14–15

C.1 DenseNet-121: All Methods Comparison

Across the comparisons, SpatialUQ’s MUS provides a competitive, natively calibrated failure score, with performance depending on model architecture, spatial extent, and distribution shift. Its fixed crops outperform random placement, while severe shift and globally pooled representations can weaken the signal.

C.1 DenseNet-121: All Methods Comparison

  • MUS reaches AUC 0.7837 and SCE 0.0487 on NIH DenseNet-121; Fusion CV has the highest AUC (0.8319), while five-member Deep Ensemble reaches 0.8130.MUS exceeds MC-Dropout by ∆AUC = +0.1194 (p < 10−6), and fusion exceeds Deep Ensemble by +0.0189 (p < 10−6); seed runs keep MUS AUC between 0.7826 and 0.7902.Source: §C.1 · p. 15, Table 9 · p. 16, Table 10 · p. 16

C.2 Foundation Model Evaluation with Linear Probes

  • With frozen foundation models and linear probes, MUS AUC rises from 0.825 with CLIP to 0.899 with BiomedCLIP, alongside stronger correlation with Brier score (ρ = 0.631 and 0.846, respectively).Figure 6 shows MUS outperforming entropy and confidence baselines; its calibration remains moderate (SCE 0.10–0.13) and better than raw L1 (0.115–0.232).Source: §C.2 · pp. 16–17, Table 11 · p. 17, Figure 6 · p. 17

C.3 EfficientNet-B4: Full Failure Detection Comparison

  • On EfficientNet-B4, Fusion CV leads at 0.8714 AUC and MUS reaches 0.7842, while MC-Dropout is near chance at 0.5357.The MUS–MC-Dropout difference is +0.2485 (95% CI [0.2381, 0.2589], p < 10−6).Source: §C.3 · p. 17, Table 12 · p. 18

C.4 ViT-B/16: Full Failure Detection Comparison

  • On ViT-B/16, Fusion CV leads at 0.8314 AUC, ODIN is second at 0.8005, and MUS reaches 0.7497; global attention partially reduces the failure signal.Masking gives ViT a marginal AUROC improvement over upsampling (0.7465 vs. 0.7421, ∆= +0.0045), unlike the reported CNN results.Source: §C.4 · p. 18, Table 13 · p. 19, Table 14 · p. 19

C.5 BioViL-T: Generative VLM Failure Regime

  • BioViL-T’s MUS reaches 0.533 AUC with ρ = −0.187, a regime flagged by the ρ ≲0.15 diagnostic; globally pooled representations weaken the crop-level inconsistency signal.Fusion recovers to 0.838 AUC, driven by entropy and confidence signals.Source: §C.5 · pp. 18–19, Table 15 · p. 19

D.1–D.3 Per-class JSD AUC

  • Per-class MUS performance varies by pathology and architecture: DenseNet-121 excels on diffuse findings but reaches 0.368 AUC for Nodule, whereas EfficientNet-B4 reaches 0.8822 for Nodule.ViT-B/16 retains strong results for diffuse Pneumonia (0.88) and Edema (0.87), but drops to 0.3037 for Nodule, 0.4182 for Mass, and 0.4024 for Emphysema; the discussion attributes this to global attention compressing the inconsistency signal.Source: §D.1 · p. 20, Table 16 · p. 20, §D.2 · p. 20

D.4–D.5 Zero-Shot Per-Class JSD AUC

  • Under zero-shot transfer, CheXpert Nodule AUC falls to 0.1446, while VinBigData performance ranges from 0.9467 for Atelectasis to 0.1608 for Nodule.On VinBigData, Consolidation and Mass also fall to 0.3375 and 0.1644; the text describes the degradation as broader under severe shift.Source: §D.4 · p. 21, Table 19 · p. 21, §D.5 · p. 21

E.1 Sensitivity to Crop Placement (Random vs. Fixed)

  • The standard five fixed crops reach AUC 0.784, compared with 0.7389 for 1,000 randomly sampled crops, while random placement retains some efficacy.The authors report the fixed geometry as a more efficient signal and interpret random-placement performance as evidence that spatial inconsistency is robust to crop positions.Source: §E.1 · p. 22

E.2 Crop Count Ablation

Across crop-count, robustness, fusion, and transfer experiments, SpatialUQ’s performance depends on the spatial and distributional setting. Five crops balance coverage and cost, while complementary signals and moderate-shift transfer improve failure detection.

E.2 Crop Count Ablation

  • Five crops provide the best cost-efficiency trade-off: nine crops add +0.007 AUC at 67% more inference cost, while three crops reduce AUC by 0.007.Table 21 reports MUS AUC on NIH DenseNet-121; Figure 8 shows the crop-count trade-off.Source: §E.2 · p. 22, Table 21 · p. 22, Figure 8 · p. 23
  • Adding a center crop gives K=5 fuller central-region coverage than K=4, while their AUC difference of 0.0022 lies within the bootstrap confidence interval.The center crop overlaps the quadrants and provides double coverage of the image center without extra forward passes.Source: §E.2 · p. 22, Figure 7 · p. 23

E.3 Failure Threshold Robustness

  • MUS AUC rises from 0.770 to 0.784 to 0.823 across the 50th, 75th, and 90th percentile Brier thresholds, respectively.The passage attributes the trend to stronger spatial-inconsistency signal among the most severely incorrect predictions; CheXpert shows the same increase, from 0.647 to 0.708 to 0.757.Source: §E.3 · p. 24, Table 22 · p. 24

E.4 Leave-One-Out Fusion Ablation

  • For NIH DenseNet-121, removing entropy lowers full-fusion AUC of 0.8319 by 0.0437, while removing ℓ1 Distance and MUS lowers it by 0.0071 and 0.0053.Removing confidence lowers AUC by 0.0010.Source: §E.4 · pp. 24–25, Table 23 · p. 25
  • For NIH ViT-B/16, removing entropy lowers full-fusion AUC of 0.8314 by 0.0780, while removing ℓ1 Distance and MUS lowers it by 0.0022 and 0.0015.Removing confidence slightly improves AUC by 0.0014.Source: §E.4 · p. 25, Table 24 · p. 25
  • Across three ImageNet-1k architectures, confidence removal lowers fusion AUC by 0.0934, 0.0407, and 0.0168, while MUS contributes a smaller but consistent signal.The confidence penalties correspond to ConvNeXt-Tiny, ViT-B/16, and EfficientNet-B4; MUS removal penalties range from 0.0035 to 0.0056.Source: §E.4 · p. 25, Table 25 · p. 25

E.5 Permutation Feature Importance (Learned Auditor)

  • In the learned-auditor analysis, entropy is the primary discriminator, while MUS and confidence provide secondary independent contributions.The stated baseline test AUC for the three-feature learned auditor is 0.8250.Source: §E.5 · pp. 25–26

F.1 CheXpert Zero-Shot: Complete Evaluation

  • On CheXpert, MUS reaches 0.7076 AUC and exceeds MC-Dropout by +0.1085 AUC without retraining.At 20% referral, selective prediction improves system AUC from 0.6860 to 0.7478.Source: §F.1 · p. 26, Table 26 · p. 26

F.2 VinBigData Zero-Shot: Complete Evaluation

Zero-shot results show that spatial inconsistency is unreliable under severe distribution shift, while ImageNet-1k results favor confidence as a standalone signal and fusion for near-ceiling performance.

F.2 VinBigData Zero-Shot: Complete Evaluation

  • On VinBigData, MUS reaches 0.6141 AUC versus MC-Dropout’s 0.7641, and its Brier correlation is near zero at ρ = 0.0269.Fusion reaches 0.750 AUC, but its gain is driven by MC-Dropout and entropy rather than MUS.Source: §F.2 · p. 26, Table 27 · p. 27

G.1 ImageNet-1k: Full Metric Breakdown

  • Across three ImageNet-1k architectures, Confidence-inv achieves 0.9132–0.9233 AUC alone, while Fusion CV reaches 0.9173–0.9369.MUS alone is weaker, with AUC ranging from 0.6415 to 0.7171.Source: §G.1 · p. 27, Table 28 · p. 27

G.2 MS COCO 2014: Full Metric Breakdown and Score Threshold Sensitivity

On MS COCO, MUS contributes to failure detection alongside entropy, especially for RetinaNet, and its AUC is robust to conservative detection thresholds. Across the additional sections, results cover lesion scale, referral and triage, demographics, computational cost, and the spatial mechanism behind MUS.

G.2 MS COCO 2014: Full Metric Breakdown and Score Threshold Sensitivity

  • On COCO, entropy leads standalone AUC (0.8839 Faster R-CNN; 0.9121 RetinaNet), while fusion reaches 0.8851 and 0.9133, with MUS chiefly complementing RetinaNet.MUS standalone AUC is 0.8003 for Faster R-CNN and 0.7469 for RetinaNet.Source: §G.2 · p. 27, Table 29 · p. 28
  • MUS AUC is stable from confidence thresholds 0.05–0.10, with modest drops at 0.30 (Faster R-CNN −0.0127; RetinaNet −0.0446).The reported sensitivity analysis describes performance as fully stable in the 0.05–0.10 range.Source: §G.2 · p. 27, Table 30 · p. 28

H MULTI-SCALE MUS FOR NODULE DETECTION

  • Diffuse pathologies have mean JSD AUC 0.944 versus 0.603 for localized lesions; fine-grid crops raise Nodule AUC from 0.368 to 0.589, while prevalence is uncorrelated with MUS AUC (r = 0.112, p = 0.702).The 16-crop 56×56 fine grid is the primary multi-scale result; averaging it with the coarse score lowers AUC to 0.495.Source: H MULTI-SCALE MUS FOR NODULE DETECTION · pp. 28–29, Table 31 · p. 30

I.1 When to Use SpatialUQ

  • Table 32 consolidates findings from Sections 5–6 about when to use SpatialUQ.The supplied passage identifies the table as a summary of prior findings.Source: Table 32 · p. 30

I.2 Selective Prediction Coverage (NIH DenseNet-121)

  • At a 10% recall budget, MUS refers 4.3% of the test set at 58.7% precision; backbone AUC gains rise from +0.0017 at 5% referral to +0.0092 at 30%.The baseline AUC is 0.8080; the table reports images referred in descending MUS order.Source: §I.2 · p. 30, Table 33 · p. 30

I.3 Clinical Triage Simulation

  • For NIH DenseNet-121, referring 9.2% of images recovers 20% of Brier-defined failures at 54.5% precision, while the 95%-sensitivity rule refers 97.7% of EfficientNet-B4 images.ViT-B/16 referral under the same sensitivity rule is 96.4%.Source: §I.3 · pp. 30–31

I.4 Demographic Fairness

  • MUS gender AUC is nearly equal for NIH DenseNet-121 (female 0.7838; male 0.7837), while age trends reverse between DenseNet-121 and EfficientNet-B4.DenseNet-121 AUC declines from 0.8199 for patients under 40 to 0.7512 for ages 60–80; EfficientNet-B4 rises from 0.7723 to 0.7943 across those groups.Source: §I.4 · pp. 31–32, Table 34 · p. 31, Table 35 · p. 32

J IMPLEMENTATION DETAILS

  • MUS uses six forward passes per image and runs about five times faster than 30-pass MC-Dropout on the NIH test set.For DenseNet-121, the reported times are approximately 8 minutes for MUS and 39 minutes for MC-Dropout.Source: J IMPLEMENTATION DETAILS · p. 32

K.2 Per-Method SOTA Analysis

MUS compares favorably with several post-hoc uncertainty methods, but its performance varies by architecture and task. The section also reports computational trade-offs, future validation needs, and the class mapping used for transfer evaluation.

K.2 Per-Method SOTA Analysis

  • MUS beats MC-Dropout in four of five settings, with AUC gains of +0.119 on NIH DenseNet, +0.249 on EfficientNet, +0.100 on ViT, and +0.109 on CheXpert; VinBigData is the exception (−0.150).On NIH DenseNet, fusion reaches 0.832 AUC versus 0.813 for a five-member ensemble; DDU and Mahalanobis each reach 0.754.Source: §K.2 · pp. 33–34

K.3 Per-Setting Performance Interpretation

  • MUS AUC is 0.784 for DenseNet-121 and EfficientNet-B4 and 0.750 for ViT-B/16; EfficientNet-B4 reaches 0.882 for Nodule versus DenseNet-121’s 0.368.Raw entropy reaches AUC 0.177 on NIH DenseNet; on COCO, entropy dominates standalone signals with AUCs of 0.884 and 0.912.Source: §K.3 · pp. 34–35

K.4 Computational Cost

  • MUS requires six forward passes and yields a fivefold wall-clock speedup over 30-pass MC-Dropout, while avoiding the ensemble’s fivefold training cost.A five-member ensemble uses five inference passes but cannot be applied to a frozen checkpoint without retraining.Source: §K.4 · p. 35

K.5 Future Directions

  • Evaluation beyond radiography and RGB natural images remains untested; proposed extensions include MIMIC-CXR validation and broader demographic audits.The authors also identify adaptive crop placement, transformer segmentation, and 3D CT as future directions.Source: §K.5 · p. 35

CheXpert Label Mapping (10 classes)

Related papers

Loading 2610.09498v1…