Source-linked AI summary
TRUST: Threshold-Recalibrated Uncertainty-Safe Training for Certified Dismissal in Breast Cancer Screening
Parham Hajishafiezahramini, Matthew Hamilton, Edward Kendall, Gregory Doyle, Oscar Meruvia Pastor
TL;DR
Breast screening needs ways to reduce review of clearly negative examinations without compromising cancer detection or relying only on conventional discrimination metrics. TRUST trains a classifier with a dynamically recalculated dismissal threshold and evaluates case-level dismissal using an independent one-sided 99% Clopper–Pearson certificate. Across NLBS, RSNA, and external RSNA→NLBS evaluation, the proposed model achieved the strongest high-recall dismissal results, supporting certificate-aware training for selective dismissal.
Problem
The central problem is how to dismiss screening examinations from radiologist review while controlling the cancer prevalence among dismissed cases with a finite-sample guarantee.
Method
TRUST repeatedly updates a provisional dismissal threshold during training and uses it to shape a positive-class penalty, while keeping final certification independent.
Results
Across NLBS and RSNA, the proposed model achieved the highest case-level dismissal rates at both 98% and 95% recall targets and the lowest CPU99 at each operating point.
Takeaways & Limitations
The results support dynamically threshold-aware training for high-recall selective dismissal across in-domain and external evaluations.
Takeaways & Limitations
Clinical validation is retrospective and does not measure effects on radiologist workflow, reading time, recall decisions, or patient outcomes.
Abstract
from arXiv · showhide
Reducing the review of clearly cancer-negative screening mammograms could lower radiologist workload without compromising cancer detection. We propose a closed-loop threshold-aware training strategy in which the dismissal threshold is recalculated during training and used to penalize cancer-positive images that approach the dismissal region. We evaluated the method on NLBS and RSNA using five controlled training configurations, with case-level assessment based on a one-sided 99\% Clopper--Pearson upper bound for cancer prevalence among dismissed cases. The proposed model achieved the highest case-level dismissal rates at both 98\% and 95\% recall targets. On NLBS, dismissal reached 19.74\% and 21.70\%, while the cross-entropy baseline did not meet either recall target. On RSNA, dismissal improved from 7.04\% to 14.31\% and from 13.49\% to 19.69\%. In external RSNA$\to$NLBS evaluation, the proposed model achieved dismissal rates of 12.95\% and 19.87\% at the 98\% and 95\% recall targets, respectively. These results support closed-loop threshold-aware training for high-recall selective dismissal.
1 Introduction
Breast screening generates substantial benign recalls and workload, motivating selective dismissal with an independently auditable cancer-risk guarantee. TRUST incorporates the dismissal criterion into training rather than applying it only after classifier training.
- Clinical motivation: Most recalled and biopsied findings are ultimately benign, increasing radiologist workload, imaging use, patient anxiety, and healthcare costs.
- Clinical motivation: Conventional metrics such as AUROC, sensitivity, and specificity do not directly quantify safe removal of examinations from manual review.
- Clinical motivation: The study targets the proportion of screening examinations that can be dismissed while maintaining a finite-sample upper bound on cancer prevalence among dismissed cases.
- Study approach: The proposed objective repeatedly updates the dismissal threshold during optimization and penalizes unsafe dismissal of cancer-positive examinations.
- Study approach: The study evaluates dynamic threshold coupling, controlled alternatives, and certified case-level dismissal across two independent mammography datasets with internal and external validation.
2 Related Work
Prior work has addressed AI-assisted screening, selective prediction, and statistical risk control, but commonly applies thresholds after training or uses separate selection mechanisms. This study instead guides classifier optimization with a dynamically updated dismissal boundary and tests safe-dismissal behavior under domain shift.
- Screening AI: AI screening studies have reported radiologist-comparable performance and workflow workload reductions, but conventional evaluations do not establish independently auditable dismissal guarantees.
- Rule-out and selective classification: Statistical rule-out frameworks evaluate caseload reduction and missed-cancer risk using scores from an already trained model.
- Rule-out and selective classification: Selective classification methods use confidence thresholds or jointly learn prediction and selection functions, whereas this method guides the classifier directly without a separate selection head.
- Risk control: Conformal and related risk-control approaches motivate incorporating statistical criteria into optimization, while this study focuses specifically on high-recall mammography dismissal with independent case-level certification.
- Domain shift: The study examines whether certificate-aware training retains safe-dismissal benefits when evaluated on data from an independent institution.
3 Materials and Methods
The study uses two mammography datasets, standardized preprocessing, a frozen Mammo-CLIP encoder with a trained classification head, and conservative maximum aggregation for case-level decisions.
- Datasets: NLBS contains 26,988 images from 5,997 screening cases, including 149 cancer-positive cases.
- Datasets: RSNA contains 54,706 images from 11,913 examinations, including 486 cancer-positive examinations after preprocessing.
- Preprocessing: Both datasets undergo consistent DICOM-to-PNG conversion, orientation correction, breast cropping, resizing, normalization, augmentation, and three-channel Mammo-CLIP standardization.
- Model: EfficientNet-B5 initialized with Mammo-CLIP weights is frozen, while only a LayerNorm-dropout-linear classification head is trained to produce sigmoid scores.
- Case-level aggregation: The case-level score is the maximum image score, so a case remains for review when any constituent image receives a high cancer score.
3.5 Certification unit and the independence assumption
Certification is performed at the case level because images within an examination are not independent. Thresholds are selected on a separate search subset and evaluated once on an independent case-level subset using a one-sided 99% Clopper–Pearson bound.
- Certification unit: Each certification observation must be an independent binary outcome indicating whether a dismissed case contains cancer.
- Certification unit: Multiple images from one examination are dependent, so treating them as separate observations would artificially narrow the confidence bound.
- Certification unit: Maximum aggregation produces one score and one binary outcome per case for the Clopper–Pearson calculation.
- Certification unit: Image-level dismissal rates are descriptive only and receive no Clopper–Pearson confidence bound or formal certification.
- Evaluation design: Five-fold case-level cross-validation yields out-of-fold predictions for every case before threshold search and final evaluation.
3.8 Proposed closed-loop dismissal loss
The proposed loss combines standard classification objectives with a one-sided dismissal penalty that protects cancer-positive images near the dynamically recalculated dismissal threshold.
- The objective combines binary cross-entropy, focal loss, and a one-sided dismissal loss.The dismissal term is added alongside classification signals rather than replacing them.
- The dismissal penalty applies only to cancer-positive examples scoring below τt + m and grows as their scores move farther below that boundary.Positive examples above the boundary receive no dismissal penalty, while cross-entropy and focal loss continue to train both classes.
- At each epoch, τt is recalculated from training-calibration predictions as the largest threshold whose one-sided 95% Clopper–Pearson upper bound on dismissed positive images is at most 1%.If no threshold satisfies the criterion, τt is set to zero; this threshold is used only as a training-time signal.
- The margin creates a buffer above the dismissal boundary, while the small dismissal weight keeps the penalty auxiliary to cross-entropy and focal-loss objectives.The reported fixed hyperparameters include γ = 2.0, λfocal = 1.0, λdismiss = 0.05, and m = 0.10.
- The fixed-τ ablation holds τt at 0.05, isolating the effect of dynamic threshold updating from the effect of adding dismissal loss.The closed-loop configuration instead replaces the initial value with the threshold recalculated at each epoch.
3.9 Training protocol
Training used matched configurations and fixed data splits to compare classification, calibration, focal, fixed-threshold dismissal, and dynamically recalibrated closed-loop objectives.
- Optimization and sampling: Training used AdamW for 20 epochs with minibatches containing 20 cancer-positive and 60 cancer-negative samples.Because balanced sampling changes the training distribution, sigmoid outputs were treated as model scores rather than population-calibrated cancer probabilities.
- Training configurations: All five configurations used the same data splits, training schedule, and evaluation procedure for controlled comparison.None was selected or excluded based on test-set performance.
- Training configurations: The CE-only baseline used binary cross-entropy as its sole training objective and served as the primary baseline.
- Training configurations: The CE + Brier configuration combined binary cross-entropy with Brier-score regularization using λBrier = 0.1.It was included to test whether calibration-oriented training improves safe-dismissal performance.
- Training configurations: The CE + Focal configuration combined binary cross-entropy with focal loss.
- Training configurations: The fixed-τ and proposed configurations used the full classification and dismissal objective, with τt fixed at 0.05 or recomputed every epoch, respectively.The proposed configuration used γ = 2.0, λfocal = 1.0, λdismiss = 0.05, and m = 0.10.
3.11 External evaluation
External evaluation kept model weights trained on RSNA fixed while calibrating operating thresholds on a separate NLBS subset and evaluating performance on independent cases.
- External protocol: One final CE-only model and one final closed-loop model were trained on RSNA and applied to NLBS without retraining model weights.The external analysis focused on these primary baseline and proposed models.
- External protocol: Threshold selection used a stratified 20% case-level search subset, while evaluation used an independent 80% subset.The largest threshold retaining empirical recall at or above each target was selected and then fixed for evaluation.
- External protocol: For external NLBS evaluation, the operating threshold was determined from NLBS rather than transferred unchanged from RSNA.Model weights remained entirely RSNA-trained, while dismissal, recall, AUROC, and confidence bounds were computed on independent NLBS cases.
- Interpretation: The experiment tests whether RSNA-learned behavior is retained on an independent screening population when only target-domain threshold calibration is performed.
- Evaluation measures: Case-level AUROC was calculated on the independent evaluation subset using maximum aggregation of image scores.Formal certification applied only at the case level; image-level dismissal was descriptive and received no Clopper–Pearson bound.
4 Results
Across NLBS, RSNA, and external RSNA→NLBS evaluation, the proposed closed-loop model achieved the strongest case-level dismissal results at the 98% and 95% recall targets, with lower CPU99 bounds in the reported comparisons.
- NLBS: 19.74% and 21.70% were the proposed model’s NLBS case-level dismissal rates at the 98% and 95% recall targets, respectively.The corresponding CPU99 bounds were 0.70% and 0.81%.
- RSNA: 14.31% and 19.69% were the proposed model’s RSNA case-level dismissal rates at the 98% and 95% recall targets, respectively.The corresponding CPU99 bounds were 1.06% and 1.28%.
- RSNA: The proposed model’s case-level AUROC was 0.7136 on RSNA, while other configurations ranged from 0.7215 to 0.7280.Despite this AUROC ranking, the proposed model had the highest dismissal rate and lowest CPU99 bound at both recall targets.
- External validation: 12.95% was the proposed model’s external RSNA→NLBS dismissal rate at the 98% recall target, which the CE-only baseline did not reach.Thresholds were selected on the NLBS search subset and applied to the separate NLBS evaluation subset without retraining model weights.
- External validation: 19.87% versus 13.55% was the proposed model’s external RSNA→NLBS dismissal rate versus the baseline at the 95% recall target.The proposed model’s CPU99 was 1.37% versus 1.77% for the baseline.
- Summary across settings: Across the two in-domain datasets, the proposed model had the highest case-level dismissal rate at both recall targets, while CPU99 was lower than for the strongest alternatives.On NLBS, dismissal was 19.74% and 21.70%; on RSNA, it was 14.31% and 19.69%.
- Summary across settings: AUROC and dismissal performance were not necessarily aligned across datasets and high-recall operating points.The proposed model’s dismissal advantage persisted despite differing AUROC rankings.
5 Discussion
The proposed closed-loop model consistently improved certified high-recall dismissal across datasets, operating points, aggregation levels, and external evaluation. Its advantage over fixed-threshold and cross-entropy alternatives supports threshold-aware training, although broader clinical validation remains necessary.
- Evidence in support of the main claim: The proposed model achieved the highest case-level dismissal rates and lowest CPU99 bounds at both recall targets on NLBS and RSNA.This consistency held despite differences in prevalence, population, and image acquisition.
- Why dynamic threshold recomputation may matter: At 98% recall, closed-loop recomputation increased dismissal from 15.45% to 19.74% on NLBS and from 9.91% to 14.31% on RSNA.The gains were 4.29 and 4.40 percentage points, respectively, relative to the fixed-threshold ablation.
- Evidence in support of the main claim: AUROC and high-recall dismissal were not interchangeable: the proposed model had the lowest RSNA AUROC among five configurations but the strongest dismissal performance.Dismissal evaluates specific operating points where recall and cancer prevalence among dismissed cases are controlled.
- Consistency between case- and image-level results: The proposed model also achieved the highest image-level dismissal rates: 14.80% and 30.88% on NLBS, and 11.03% and 23.24% on RSNA at the two recall targets.The same advantage before aggregation suggests the result is not limited to the case-level maximum-aggregation rule.
- External validation: In external RSNA→NLBS evaluation, the proposed model dismissed 12.95% of cases at 98% recall and 19.87% at 95% recall, while retaining an advantage over the baseline.Both models recalibrated thresholds on target-domain data, so the comparison tests transfer of the learned advantage rather than reuse of an unchanged threshold.
- Potential economic implications of selective dismissal: At 98% recall, 19.74% dismissal on NLBS corresponds to approximately 197 fewer manual interpretations per 1,000 screening examinations.This reduces manual interpretation rather than mammographic acquisition, and illustrative economic figures are not realized health-system savings.
6 Conclusion
The closed-loop approach improved certified selective dismissal across NLBS and RSNA, including external RSNA→NLBS evaluation. Results also indicate that dynamic threshold updating benefits dismissal beyond a fixed-threshold objective, although the certificate is not a guarantee of future clinical safety.
- The approach repeatedly updates a provisional dismissal threshold during training to shape a positive-class penalty while separating final threshold selection and statistical evaluation from training.This aligns optimization with the high-recall operating region used for selective dismissal.
- The proposed model achieved the highest case-level dismissal rates and lowest CPU99 at both 98% and 95% recall targets across NLBS and RSNA.The fixed-τ comparison further suggests that dynamic boundary updating adds benefit beyond a dismissal-oriented loss alone.
- The proposed model maintained the strongest dismissal performance despite AUROC varying across datasets.It achieved the highest AUROC on NLBS but the lowest among evaluated configurations on RSNA, while retaining the highest case-level dismissal rates at both recall targets.
- External RSNA→NLBS evaluation reproduced the overall pattern with fixed RSNA-learned weights and thresholds applied to an independent NLBS evaluation subset.At 98% recall, dismissal was 12.95% with CPU99 1.06%; at 95% recall, dismissal was 19.87% versus 13.55% for the baseline, with CPU99 1.37% versus 1.77%.
- The statistical certificate applies specifically to the one-sided upper confidence bound on cancer prevalence within dismissed evaluation cases, not to future clinical safety.Further evaluation is needed across additional populations, architectures, and prospective workflows.
Compliance with Ethical Standards
This retrospective computational study used publicly available, de-identified mammography datasets and required no additional institutional ethical approval.
- The study used only publicly available, de-identified mammography datasets.The datasets were NLBS and the RSNA Screening Mammography Breast Cancer Detection dataset.
- No additional institutional ethical approval was required because the analysis used secondary data that were publicly available and de-identified.
Data and Code Availability
The study used publicly available NLBS and RSNA mammography datasets, and generated no new clinical data.
- The NLBS dataset is available through the Federated Research Data Repository.
- The RSNA Screening Mammography Breast Cancer Detection dataset is publicly available through the RSNA/Kaggle challenge.
- No new clinical data were generated as part of this study.