Source-linked AI summary
Uncertainty-Informed Deep Learning Models Enable High-Confidence Predictions for Digital Histopathology
James M Dolezal, Andrew Srisuwananukorn, Dmitry Karpeyev, Siddhi Ramesh, Sara Kochanny, Brittany Cody, Aaron Mansfield, Sagar Rakshit, Radhika Bansa, Melanie Bois, Aaron O Bungum, Jefree J Schulte, Everett E Vokes, Marina Chiara Garassino, Aliya N Husain, Alexander T Pearson
TL;DR
Clinical deployment of digital-histopathology models is hindered by unpredictable performance under domain shift and limited case-wise uncertainty assessment. The paper introduces training-data-calibrated, dropout-based whole-slide uncertainty thresholding and finds that high-confidence predictions outperform non-UQ predictions across internal and external evaluations, including non-lung cohorts.
Problem
Domain shift and absent case-wise uncertainty assessment limit confidence in deploying deep-learning models for clinical digital histopathology.
Method
The study uses dropout-based uncertainty estimation with thresholds determined from training data to classify whole-slide images as low or high confidence.
Results
High-confidence predictions outperformed non-UQ predictions across cross-validation and external CPTAC and Mayo Clinic evaluations, including testing on non-lung cancers.
Takeaways & Limitations
Training-data-calibrated slide-level uncertainty can identify more accurate high-confidence predictions across institutions and domain shifts.
Takeaways & Limitations
The reported gains require abstaining on part of the data, potentially up to one-third, and were extensively tested only for binary classification of a histologically well-defined outcome.
Abstract
from arXiv · showhide
A model's ability to express its own predictive uncertainty is an essential attribute for maintaining clinical user confidence as computational biomarkers are deployed into real-world medical settings. In the domain of cancer digital histopathology, we describe a novel, clinically-oriented approach to uncertainty quantification (UQ) for whole-slide images, estimating uncertainty using dropout and calculating thresholds on training data to establish cutoffs for low- and high-confidence predictions. We train models to identify lung adenocarcinoma vs. squamous cell carcinoma and show that high-confidence predictions outperform predictions without UQ, in both cross-validation and testing on two large external datasets spanning multiple institutions. Our testing strategy closely approximates real-world application, with predictions generated on unsupervised, unannotated slides using predetermined thresholds. Furthermore, we show that UQ thresholding remains reliable in the setting of domain shift, with accurate high-confidence predictions of adenocarcinoma vs. squamous cell carcinoma for out-of-distribution, non-lung cancer cohorts.
Main
Deep learning shows promise in digital histopathology, but unpredictable behavior under domain shift and limited uncertainty assessment impede reliable clinical use. This work addresses these issues with slide-level uncertainty thresholds determined from training data for whole-slide predictions.
- Problem: Domain shift can degrade deep-learning performance on data outside the training distribution, complicating clinical deployment and assessment of generalization.External testing is important, but limited clinical data diversity makes generalization to other institutions and patient populations difficult.
- Motivation: Most digital-pathology applications do not assess case-wise uncertainty, despite clinical cases sometimes being diagnostically ambiguous or insufficient.
- Uncertainty quantification: Monte Carlo dropout estimates uncertainty by generating multiple predictions with randomly dropped model nodes during inference.The resulting prediction distribution can be summarized using its standard deviation as an uncertainty estimate.
- Gap: Prior uncertainty estimates were vulnerable to domain shift, and manually selected thresholds could leak validation-label information.The authors therefore emphasize thresholds determined from training data and uncertainty estimates at whole-slide rather than smaller tile level.
- Approach: The study develops clinically oriented slide-level uncertainty for lung adenocarcinoma versus squamous cell carcinoma and evaluates it on two large external datasets.The contribution includes training-data thresholding, nested cross-validation, assessment of training-data requirements, and multi-institution external evaluation.
Results
Uncertainty thresholding improved high-confidence classification across internal and external evaluations, while low-confidence regions often corresponded to ambiguous or non-diagnostic morphology. The approach also identified unreliable out-of-distribution and synthetic examples through low-confidence predictions.
- Cross-validation: High-confidence predictions outperformed non-UQ models for a 1:3 class-imbalance ratio with dataset sizes of at least 200 slides, but not at a 1:10 ratio.Increasing class imbalance reduced AUROC, while larger datasets partially alleviated the deterioration.
- External evaluation: AUROC reached 0.99 with 97.5% accuracy for high-confidence CPTAC predictions, versus 0.93 AUROC and 85.3% accuracy without UQ.High-confidence sensitivity and specificity were 98.4% and 96.7%, respectively; 66–100% of patients received high-confidence predictions.
- External evaluation: Mayo Clinic high-confidence predictions reached AUROC 1.0 and 100% accuracy, compared with 0.98 AUROC and 94.1% accuracy without UQ.High-confidence sensitivity and specificity were both 100%, and 70.9–94.6% of patients received high-confidence predictions.
- Out-of-distribution evaluation: In 4015 non-lung, non-adenocarcinoma, non-squamous tumors, 78.5% of slides were assigned low confidence rather than a diagnosis.Among the remaining high-confidence predictions, 10.3% were called squamous and 11.2% adenocarcinoma despite no correct diagnosis being expected.
- Morphologic interpretation: Low-confidence image regions near the decision boundary commonly contained ambiguous tumor morphology, while distant low-confidence regions often contained benign background lung and stroma.
- Synthetic-image evaluation: Adding randomly labeled synthetic intermediate images reduced non-UQ AUROC to 0.811 ± 0.024, while UQ high-confidence AUROC remained between 0.945 and 0.966.Only 0–0.8% of synthetic intermediate images received high-confidence predictions, compared with 70.6–93.0% of real cancer slides.
Discussion
The study presents training-data-derived uncertainty thresholds for separating high- and low-confidence histopathology predictions, then evaluates their reliability across institutions and domain shifts. High-confidence predictions generally improve diagnostic performance, while limitations include abstention, binary-task scope, out-of-distribution errors, and weaker benefits with severe class imbalance.
- Clinical and cross-domain performance: High-confidence predictions consistently outperform low-confidence predictions across cross-validation, multiple institutions, and domain-shifted cancer cohorts.The approach was tested on TCGA, CPTAC, Mayo, and non-lung cancer datasets.
- Limitations: The performance gains require abstaining on some cases, and the tested evidence is limited to a binary, histologically well-defined task.The authors also report erroneous high-confidence predictions among some out-of-distribution slides and weaker benefits under severe imbalance.
- Methodological contribution: The method uses dropout prediction distributions and training-fold thresholds selected by Youden’s index to identify high-confidence cases without data leakage.Nested cross-validation determines tile- and slide-level thresholds within training folds.
- Biological interpretation: High-confidence tiles were enriched for unambiguous morphology, supporting a biological interpretation of uncertainty estimates.Expert pathologist review found clear adenocarcinoma or squamous morphology among high-confidence regions.
Methods
The study trains Xception-based models on annotated and external whole-slide datasets to classify lung adenocarcinoma versus squamous cell carcinoma. Monte Carlo dropout estimates uncertainty, while nested training-only thresholding separates high- and low-confidence tile- and slide-level predictions.
- Dataset sources: 941 TCGA slides, 1,306 CPTAC slides, and 190 Mayo Clinic slides support training and external evaluation of lung adenocarcinoma versus squamous cell carcinoma classification.The datasets include H&E-stained whole-slide images with diagnoses assigned from clinical records or institutional pathology review.
- Uncertainty estimation: Each tile receives 30 dropout-enabled forward passes, producing a mean prediction and standard deviation that represents tile-level uncertainty.The distribution of stochastic predictions supplies both the tile-level prediction and its uncertainty estimate.
- Tile-level thresholding: Tile confidence is binary: predictions are high-confidence when uncertainty is below the threshold and low-confidence otherwise.The threshold is selected by maximizing the Youden index for detecting incorrect predictions.
- Slide-level aggregation: Slide predictions average high-confidence tile predictions, while slide uncertainty averages tile uncertainties before applying a slide-level confidence threshold.The slide-level threshold is likewise selected using the Youden index and then used for all model predictions.
- Threshold determination: Nested five-fold cross-validation determines uncertainty thresholds from training data only, preventing validation-label leakage before evaluation.The resulting thresholds are applied to validation and external datasets, alongside the prediction threshold determined during cross-validation.
- Evaluation design: The experiments assess uncertainty thresholding across dataset sizes, class imbalance, external slides, activation landscapes, non-lung cancer cohorts, and synthetic boundary images.Out-of-distribution testing includes 7,171 whole-slide images from 28 non-lung cancer cohorts.
Data Availability
The internal Mayo Clinic dataset is restricted because it was used with institutional permission and may be subject to intellectual-property or patient-privacy obligations.
- Data access: Requests for the internal Mayo Clinic data are evaluated under institutional and departmental policies governing intellectual property and patient privacy.Academic users must email the corresponding author, and approved data sharing remains subject to those restrictions.
Code Availability
The study provides code and scripts for reproducing its experiments through a public GitHub repository.
- Reproducibility: All reproduction code was implemented in Python with TensorFlow and is available at the BISCUIT GitHub repository.The repository includes the code and scripts used for the manuscript’s experiments.
Ethics Declaration
The authors disclose employment, funding, and outside consulting relationships, while noting that the listed employee holds no financial conflict for this work.
- Disclosures: DK reports employment at DV Group, LLC, without a financial conflict of interest for this work, while ATP reports multiple grants and outside advisory or consulting fees.The remaining authors report no competing financial interests.