Bound-Aware Per-Organ Recall Risk Control for Multi-Organ CT Segmentation under Clinical Domain Shift

Souraj Adhikary; Negar Chabi; Andre Mastmeyer

Source-linked AI summary

Bound-Aware Per-Organ Recall Risk Control for Multi-Organ CT Segmentation under Clinical Domain Shift

Souraj Adhikary, Negar Chabi, Andre Mastmeyer

arXiv:2608.18193v1Aug 18, 2026cs.CVcs.AIcs.LG

TL;DR

Per-organ recall guarantees for frozen segmentation may fail under clinical domain shift, raising the question of how to re-certify them with local cases. This paper audits AMOS-calibrated thresholds on RAOS and compares concentration bounds for local re-certification. WSR re-certified six Tier-1 organs with 25 local cases, versus 30–40 for HB, while CRC required 10–15 but had a heavier individual-case tail.

Problem
Clinical domain shift can break exchangeability, leaving the local case count and over-segmentation cost needed to re-certify organ-specific recall guarantees uncertain.
Method
The study audits per-organ FNR thresholds from an AMOS-trained nnU-Net on RAOS and compares RCPS, CRC, HB, and WSR for local re-certification.
Results
WSR re-certified six Tier-1 organs with 25 local cases, versus 30–40 for HB; CRC required 10–15 but had a heavier individual-case tail.
Takeaways & Limitations
AMOS-calibrated recall floors did not reliably transfer to RAOS, making concentration-bound selection a reportable deployment-time re-certification decision in this audit.
Takeaways & Limitations
The study examines one model and one source–target pair, and its recalibration uses one site with labels of unmeasured cost and budget estimates without confidence intervals.

Abstract

from arXiv · show

Distribution-free risk control adds organ-specific recall guarantees to frozen segmentation. We calibrate per-organ thresholds for an AMOS-trained nnU-Net, audit transfer to RAOS, and estimate local re-certification cost using case-level voxel false-negative rate (FNR). The AMOS control passes, but $7/12$ organs exceed $α{=}0.10$ after transfer; smaller calibration sets can mask exceedances with conservative or vacuous thresholds. Risk-Controlling Prediction Sets (RCPS) give high-probability control of population-mean risk, whereas Conformal Risk Control (CRC) gives weaker expectation control. Both require exchangeability; fixed and global thresholds give no per-organ guarantee. The Waudby--Smith--Ramdas (WSR) betting bound re-certifies six Tier-1 organs with 25 local cases, versus 30--40 for Hoeffding--Bentkus (HB). CRC needs 10--15 but has a heavier individual-case tail. No Tier-2 organ meets our illustrative precision criterion with 25 cases.

1 Introduction

The section motivates per-organ, distribution-free recall-risk control because mean Dice and softmax scores do not establish reliable organ-at-risk recall. It frames AMOS-to-RAOS transfer as a test of exchangeability, calibration-size effects, and local re-certification needs.

Motivation: Per-organ voxel FNR control uses post-hoc thresholds with finite-sample guarantees, studying RCPS and CRC rather than relying on mean Dice or softmax scores.FNR is defined as 1−recall for each organ.
Problem: Exchangeability between calibration and test cases can fail under disease, scanner, or imaging-protocol shifts, motivating local re-certification and measurement of over-segmentation.The section asks how many labelled local cases are needed to re-certify guarantees.
Study design and contributions: The study evaluates an AMOS-trained nnU-Net on RAOS across a graded surgical-shift ladder, auditing cohort shift and calibration size while assessing fixed, global, and local thresholds.The contributions include an in-distribution positive control and evidence that fixed and global thresholds provide no per-organ guarantee.

2 Related Work

Prior work provides distribution-free risk-control rules and segmentation-specific conformal methods, but this study targets per-organ recall guarantees under clinical shift. It contrasts high-probability RCPS with expectation-level CRC, tests both beyond exchangeability, and avoids retraining.

Distribution-free risk control: RCPS controls bounded risk with high probability, whereas CRC controls risk in expectation; WSR adds a variance-adaptive betting bound.Learn-then-Test addresses multiplicity across a grid, while covariate-shift and beyond-exchangeability methods relax standard assumptions without directly solving this setting.
Conformal/risk control for segmentation: Segmentation conformal work predicts mask sets, Dice ranges, or CRC-based FNR/FDR-style risks.These methods establish related prediction-set, quality-control, and risk-control applications for segmentation.
The two closest works: The study differs from sem-CRC by selecting organ-specific softmax thresholds, comparing RCPS with CRC, and testing both after exchangeability fails.COMPASS instead forms intervals for fixed mask-derived quantities under covariate shift, and its weighting may not cover changes in P(Y | X).
Segmentation quality control and hallucination: Unlike HALOS, this work does not retrain; it certifies recall only for organs known to be present and studies transfer of that guarantee.RAOS and related work motivate concern about failures of AMOS-trained or frozen nnU-Net models on shifted clinical scans, while Dice and HD95 do not express this guarantee.

3 Method

The method calibrates organ-specific thresholds to control population-mean missed-voxel risk under exchangeable case-level losses, while distinguishing RCPS high-probability guarantees from CRC expectation control. It also evaluates precision and sample-size usability during local RAOS re-certification.

Risk definition: Case-level FNR measures the fraction of ground-truth voxels missed for present organ k at threshold τ, with guarantees applying to population mean R_k rather than individual loss L_k.Cases, not voxels, are the exchangeable units; absent-organ hallucinations are not controlled.
Risk definition: Separate organ-channel thresholding trades fewer missed voxels at lower τ against additional predicted-positive voxels, so recall and over-segmentation are reported together.Softmax argmax instead assigns one class per voxel.
Threshold calibration: RCPS selects the largest organ-specific threshold whose (1−δ) upper confidence bound on population-mean risk is at most α.Monotonicity makes acceptable thresholds a prefix of the grid, avoiding grid-wide multiplicity correction.
Guarantees: RCPS guarantees population-mean future loss ≤α for at least a 1−δ fraction of calibration samples, whereas CRC guarantees only E[L_k(λ_k)]≤α without δ.CRC can require fewer cases, but its expectation guarantee is not interchangeable with RCPS high-probability control.
Finite-sample bounds: HB and WSR are RCPS bounds applied to the same calibration losses, while CRC uses a separate expectation criterion; WSR can be tighter when losses have low variance.Calibration is marginal by organ, not simultaneous across 12 organs; HB and WSR can use δ/12 for union-bound control.
Local re-certification: Local RAOS re-certification defines usable budget as the smallest n controlling held-out FNR while achieving mean whole-volume precision ≥0.5, with HB/WSR using 1−δ draw control and CRC using mean control.The precision cutoff is illustrative rather than clinically validated, and the single-site RAOS estimates are optimistic.

4 Experimental Setup

The experiments use a frozen 3D full-resolution nnU-Net trained on AMOS CT, with cross-fitted AMOS calibration and cached-softmax transfer analysis on all 163 RAOS cases. Evaluation is restricted to 12 shared organs and uses prespecified risk-control, bootstrap, and re-certification protocols.

Model and data: A 3D full-resolution nnU-Net is trained on AMOS CT, with 236 of 243 cases usable for out-of-fold calibration after seven corrupted exports are skipped.The cross-fitted calibration set does not match the single fixed predictor assumed by the formal guarantee.
Model and data: 163 RAOS cases comprise Set 1 with 67 oncology cases, Set 2 with 22 partial-excision cases, and Set 3 with 74 full-excision cases.All analyses use cached softmax outputs after training; the model is not retrained.
Shared label space: The study evaluates 12 organs shared by AMOS and RAOS, excluding merged prostate_uterus, aorta, and IVC classes.FNR is defined only when an organ is present, so eligibility is below 163 cases and may be reduced by resection.
Protocol: The protocol uses α ∈{0.05, 0.10}, δ = 0.10, 150/86 AMOS calibration/test splits over 20 seeds, 1001 thresholds, and 2000 bootstrap samples.Transfer-audit CIs use seed 2026, re-certification uses 200 random draws per budget, and the illustrative precision criterion is 0.5.

5 Results

The AMOS positive control passes, but frozen thresholds do not transfer reliably to RAOS: 7/12 organs exceed α=0.10, with organ-specific shifts and vacuous thresholds among apparent non-exceedances. Local re-certification shows WSR can control Tier-1 recall with 25 cases, while HB needs 30–40 and CRC’s 10–15-case guarantee is weaker; Tier-2 organs remain unusable at 25 cases.

AMOS positive control: Every organ meets the AMOS split-wise target in at least 95% of splits, exceeding the 90% coverage requirement.Mean held-out FNR is 0.028, versus in-sample ˆR(λk)≈0.06; CRC makes no 1−δ claim.
Transfer to RAOS: 7/12 organs exceed α=0.10 on RAOS after frozen AMOS transfer, while only bladder, liver, and stomach remain below α non-vacuously.Both adrenals use the vacuous predicted-everything threshold λk=0.000, so non-exceedance does not prove transfer of control.
Transfer to RAOS: 0.063→0.297 for duodenum and 0.064→0.143 for right kidney, with non-overlapping CIs, show heterogeneous RAOS degradation despite loose AMOS fit being absent.The predictor mismatch changes per-organ FNR by 0.002 on average, at most 0.006, and flips no exceedance verdicts.
Local re-certification: 25 local RAOS cases suffice for WSR to re-certify all Tier-1 organs, versus 30–40 for HB under the illustrative precision criterion.At α=0.10, HB and WSR require about 22–25 local cases before any threshold can be certified; CRC meets a separate expectation-level criterion with 10–15 cases.
Local re-certification: 0.62–0.90 WSR precision at 25 cases contrasts with HB’s 0.39 for liver and ≤0.01 for the rest, producing a 4–1800× false-positive-cost difference.WSR’s re-certified floor still controls recall, whereas HB selects a near-pass-through threshold.
Tier-2 organs: 6 Tier-2 organs do not reach a usable floor at 25 cases, with pancreas and esophagus requiring larger sets and four organs never crossing the precision criterion within the grid.WSR needs approximately 40/80 cases for pancreas/esophagus, versus CRC’s approximately 20/40; gallbladder, duodenum, and both adrenals never qualify.

6 Discussion

The discussion frames per-organ risk control as an audit requiring transparent calibration, deployment, guarantee, and false-positive reporting rather than a standalone shift warning. It motivates prospective Tier-1 testing while emphasizing limitations and that recall control is not a complete safety case.

Audit implications: A useful audit must report calibration size, vacuous thresholds, the deployed predictor, guarantee type, and false-positive cost.The RCPS bound changed the observed local sample size needed for a useful threshold.
Audit implications: ∼25 local cases are proposed for prospective Tier-1 workflow testing, with manual review for Tier-2 organs, not clinical referral validation.The illustrative results motivate testing but do not validate a clinical referral rule.
Limitations: The study covers one model and source–target pair, omits comparisons with retraining and adaptation, and uses one-site labels with budget estimates lacking confidence intervals.The main calibration and deployment predictors differ, while the matched check covers one fold of one model.
Safety limitations: Per-organ recall control is marginal, CRC is expectation-level, and guarantees exclude hallucination, resection error, and absent-organ cases.Overlapping independently thresholded masks also mean whole-volume precision does not establish label-map usability; ROI effects remain untested.

7 Conclusion

AMOS-calibrated per-organ recall floors did not reliably transfer to RAOS, making local re-certification necessary after domain shift. WSR used fewer local cases than HB under the high-probability criterion, while CRC required fewer under weaker expectation control but had a heavier individual-case tail.

7 Conclusion: AMOS-calibrated per-organ recall floors did not reliably transfer to RAOS, so local re-certification was needed after distribution shift.The local annotation budget was determined by the concentration bound rather than the frozen model once the guarantee failed.
7 Conclusion: WSR met the high-probability criterion with fewer local cases than HB.The comparison concerns the local cases needed to restore control after transfer failure.
7 Conclusion: CRC required fewer local cases under its weaker expectation guarantee but had a heavier individual-case tail, while Tier-2 organs needed larger sets or never met the illustrative precision criterion.Thus, fewer cases under CRC came with weaker control and greater individual-case tail risk.