Source-linked AI summary
Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis
Benjamin Lambert, Florence Forbes, Alan Tucholka, Senan Doyle, Harmonie Dehaene, Michel Dojat
TL;DR
Clinical adoption of high-performing DL medical-image models remains limited by rough, black-box predictions and insufficient confidence information. This review surveys UQ methods and evaluation protocols for medical image classification and segmentation, concluding that UQ is an active route toward more trustable clinical AI while facing implementation and evaluation challenges.
Problem
DL medical-image models often lack reliable confidence estimates and robust, interpretable predictions, limiting confidence in their clinical use.
Method
The review synthesizes UQ methods, evaluation protocols, challenges, and future directions for DL-based medical image analysis.
Results
MC dropout is the most popular UQ approach, representing 44.73% of implemented methods, including standard MC dropout and sampling Heteroscedastic models.
Takeaways & Limitations
UQ is presented as key to facilitating human-machine collaboration and reducing the black-box effect in clinical AI.
Takeaways & Limitations
UQ evaluation often relies on proxy tasks because uncertainty ground truth is usually unavailable, and misclassification detection should not directly rank methods.
Abstract
from arXiv · showhide
The full acceptance of Deep Learning (DL) models in the clinical field is rather low with respect to the quantity of high-performing solutions reported in the literature. Particularly, end users are reluctant to rely on the rough predictions of DL models. Uncertainty quantification methods have been proposed in the literature as a potential response to reduce the rough decision provided by the DL black box and thus increase the interpretability and the acceptability of the result by the final user. In this review, we propose an overview of the existing methods to quantify uncertainty associated to DL predictions. We focus on applications to medical image analysis, which present specific challenges due to the high dimensionality of images and their quality variability, as well as constraints associated to real-life clinical routine. We then discuss the evaluation protocols to validate the relevance of uncertainty estimates. Finally, we highlight the open challenges of uncertainty quantification in the medical field.
1 Introduction
Clinical deployment of DL medical-image models requires robust, trustable predictions because errors can affect patient care. This review surveys UQ methods, evaluation metrics, and open challenges relevant to classification and segmentation.
- DL models are often black boxes with unreliable confidence, overconfidence on outliers, and sensitivity to adversarial attacks.These limitations complicate failure detection and raise reliability and safety concerns in clinical practice.
- The review covers UQ methods for DL-based medical image classification and segmentation.
- It focuses on metrics used to evaluate uncertainty estimates.
- The review discusses current UQ challenges and limitations and suggests directions for future work.
- The report is organized around DL and UQ concepts, employed methods, evaluation procedures, and challenges in medical image analysis.
2 Framework
Supervised medical-image DL learns mappings from annotated images to class labels or segmentation masks, while UQ supplements predictions with confidence information. Predictive uncertainty is commonly separated into epistemic, aleatoric, and segmentation-related label uncertainty.
- Supervised DL learns p(y|x) from paired training images x and ground-truth labels y, representing classes or segmentation masks.
- CNNs extract image features with convolutional layers and support classification and segmentation architectures.
- During training, predictions are compared with ground truth through loss functions such as cross-entropy or focal loss.
- UQ augments softmax-based class or pixel/voxel predictions with uncertainty estimates to improve interpretation of model confidence.
- Epistemic uncertainty reflects model knowledge gaps and is considered reducible with additional data.
- Aleatoric uncertainty reflects intrinsic data noise and is considered irreducible with more data.
- Label uncertainty in segmentation is linked to inter-rater variability in manual delineations.
3 Review of Uncertainty-Quantification methods for medical image analysis using Deep Learning
The review systematically identifies supervised DL studies using UQ for medical image classification and segmentation, then organizes methods by prediction-based estimates and uncertainty categories. It notes that softmax-only estimates capture aleatoric rather than epistemic uncertainty.
- The systematic search covered Google Scholar and PubMed studies published from 2015 through June 2022.Included studies used supervised DL for medical image classification or segmentation and quantified algorithmic uncertainty.
- Predicted probabilities p(y|x) provide a simple uncertainty estimate, with calibration assessing whether probabilities match observed correctness.
- Figure 1 classifies the 130 selected papers by UQ method class, sampling versus single-step implementation, and modeled uncertainty type.
- Softmax-probability UQ considers output distributions but not weight distributions, so it models only aleatoric uncertainty.
3.2 Bayesian Neural Network methods
Bayesian and sampling-based neural-network methods estimate uncertainty by representing or perturbing model parameters and aggregating predictions. These approaches differ in implementation burden, while BNNs can model both aleatoric and epistemic uncertainty.
- Bayesian neural networks replace each fixed NN weight with a distribution learned from a prior and training data.Inference repeatedly samples weights and averages predictions to obtain a predictive distribution and uncertainty estimates.
- BNNs can estimate entropy, variance, or mutual information and model both aleatoric and epistemic uncertainty.
- MC dropout approximates Bayesian inference by retaining dropout during inference and aggregating predictions from multiple randomly masked forward passes.
- Deep Ensembles sequentially train neural networks with random initialization, producing diverse predictions without changing architecture or training paradigm.
3.5 Heteroscedastic model-based methods
Heteroscedastic models estimate aleatoric uncertainty from the data, using either sampling-based or deterministic formulations. Sampling variants paired with MC dropout can model both aleatoric and epistemic uncertainty, whereas deterministic variants produce predictions and uncertainty in one step.
- Heteroscedastic models: Heteroscedastic models learn aleatoric uncertainty directly from data without requiring ground-truth uncertainty labels.They include sampling methods extending MC dropout and deterministic approaches that produce predictions with associated uncertainty in a single step.
- Sampling heteroscedastic models: Sampling formulations model network-output logits as Gaussian-noise corrupted and predict both mean logits and noise variance.Higher predicted variance represents higher aleatoric uncertainty; training integrates the loss over multiple noise samples.
- Sampling heteroscedastic models: Paired with MC dropout, sampling heteroscedastic models capture both epistemic and aleatoric uncertainties.
- Deterministic heteroscedastic models: Deterministic heteroscedastic variants avoid integrating the loss over multiple noise samples while jointly learning prediction and uncertainty.They add a dedicated uncertainty output and use an uncertainty-augmented loss for classification or segmentation.
3.6 Label-distribution model-based methods
Label-distribution models address segmentation settings with multiple expert annotations by learning and sampling a distribution of plausible masks. However, inter-rater variability may reflect contextual annotation biases rather than true label uncertainty.
- Label-distribution models: Label-distribution models use multiple expert segmentations to approximate a ground-truth label distribution for image segmentation.Generative segmentation networks learn this distribution, and inference sampling produces diverse masks reproducing inter-rater variability.
- Inference: Sampling from the learned label distribution produces diverse segmentation masks that reproduce observed inter-rater variability.
- Caveat: Inter-rater variability may arise from partial-volume effects and experts’ over- or under-segmentation, making it uncertain as ground-truth uncertainty.
- Caveat: Inter-rater variability is linked more to contextual biases, such as radiologist experience or annotation habits, than to true label uncertainty.
3.9 Evidential Deep Learning
Evidential Deep Learning uses the Dempster–Shafer Theory of Evidence to represent epistemic uncertainty through class-specific belief masses and an overall uncertainty mass. Deep Learning implementations fit Dirichlet distributions to outputs and train with Bayes-Risk loss.
- Dempster–Shafer Theory of Evidence: Dempster–Shafer Theory of Evidence represents epistemic uncertainty using belief masses for each class and an overall uncertainty mass.With no evidence favoring any class, beliefs are minimal at 0 and overall uncertainty is maximal at 1.
- Evidential Deep Learning: Evidential Deep Learning fits a Dirichlet distribution to model outputs instead of a standard categorical distribution.
- Evidential Deep Learning: Bayes-Risk loss replaces standard cross-entropy to train the evidential model.
4 How to evaluate uncertainty quantification approaches
The review identifies seven protocols for evaluating uncertainty estimates because ground-truth uncertainty is typically unavailable and proxy metrics are therefore needed. These protocols include quantitative calibration and error-detection measures, qualitative inspection, downstream assessment, rejection, OOD detection, segmentation-quality correlation, and label-distribution comparison.
- Evaluation protocols: Seven evaluation-protocol types are identified because uncertainty estimates usually lack direct ground-truth values.The reviewed papers therefore rely on proxy metrics to assess uncertainty-quantification performance.
- Qualitative and downstream assessment: Qualitative assessment visually inspects cases deemed certain or uncertain, while downstream evaluation measures effects on tasks such as image selection or predictive performance.
- Calibration: Calibration metrics test correspondence between predicted probabilities and error rates using ECE, Brier Score, or NLL.
- Misclassification detection: Misclassification detection converts uncertainty into a binary classifier distinguishing correct from incorrect predictions through an uncertainty threshold and confusion matrix.This enables standard classification metrics based on the resulting counts.
- Rejection protocol: Rejection evaluation orders predictions by certainty, rejects the most uncertain fraction, and measures performance on the remainder.Its resulting curve summarizes whether uncertainty identifies cases more likely to be incorrect; this protocol highlights the same trends as misclassification detection.
- OOD and segmentation protocols: OOD evaluation tests whether uncertainty distinguishes in-distribution from out-of-distribution images, while segmentation evaluation correlates aggregated image-wise uncertainty with quality metrics such as Dice.
- Label-distribution protocol: Label-distribution evaluation compares predicted and expert label distributions using metrics such as Generalized Energy Distance.
5 Discussion
The review surveys uncertainty quantification methods and evaluation protocols for deep-learning medical image analysis, emphasizing clinical usability, computational trade-offs, and unresolved evaluation and deployment challenges. It identifies sampling-based methods as prevalent but costly, while highlighting deterministic, out-of-distribution-aware, and explainability-linked approaches as important research directions.
- Methods and computational costs: MC dropout represents 44.73% of implemented methods, but its dropout-rate sensitivity and repeated inference can limit clinical practicality.Its popularity is attributed to easy integration with dropout-trained neural networks and potential protection against overfitting in small medical datasets.
- Methods and computational costs: Ensembling can improve predictive performance while providing uncertainty estimates, but requires multiple model trainings and predictions, increasing computational cost and time.The review describes ensembles as less common than MC dropout models.
- Evaluation protocols: UQ evaluation commonly uses proxy tasks such as misclassification, out-of-distribution, and quality-control detection because uncertainty ground truth is usually unavailable.When multiple expert delineations exist for segmentation, inter-rater variability can serve as ground-truth uncertainty.
- Evaluation protocols: Misclassification-detection results should not directly rank UQ methods because each model generates its own model-specific set of correct and incorrect predictions.The review therefore cautions against direct comparison of these evaluation results.
- Future directions: Single-step UQ methods compute uncertainty efficiently but only partially model uncertainty by estimating either aleatoric or epistemic uncertainty.Although currently niche, their practical efficiency may support wider adoption in medical applications.
- Methods and computational costs: 81.15% of implemented UQ methods use sampling protocols, increasing computational burden and motivating greater exploration of deterministic single-step methods.Sampling generates multiple predictions for each query input, which may hinder adoption in automated medical pipelines.
6 Conclusion
The review surveys uncertainty quantification methods for deep-learning medical-image applications and identifies uncertainty sources that make trustworthy AI an active research topic.
- The review covers popular uncertainty quantification methods for deep-learning medical-image applications.
- Predictive uncertainty can arise from noisy images, imperfect ground-truth labels, incomplete data, and inter-site image variability.
- Developing trustworthy AI solutions that integrate uncertainty quantification remains an active research topic.
7 Declaration of competing interest
The declaration reports professional affiliations between several authors and Pixyl.
- BL, AT, SD, and HD are employees of Pixyl.
- MD and FF serve on Pixyl’s scientific advisory board.
- The disclosed affiliations involve employment or scientific-advisory roles at Pixyl.