Source-linked AI summary
Assessing Reliability and Challenges of Uncertainty Estimations for Medical Image Segmentation
Alain Jungo, Mauricio Reyes
TL;DR
Medical image segmentation systems can be accurate yet insufficiently robust for reliable patient-level clinical use, motivating uncertainty-based failure detection. The paper evaluates common voxel-wise uncertainty measures on two medical image segmentation datasets using calibration, overlap, and correction analyses. Methods are similarly calibrated at the dataset level but often miscalibrated at the subject level; auxiliary networks are a valid alternative applicable to previously trained segmentation models.
Problem
Deep learning segmentation systems remain insufficiently robust for clinical integration, while the benefits and challenges of uncertainty estimation for medical image segmentation are not well established.
Method
The study evaluates common voxel-wise uncertainty measures on two medical image segmentation datasets using calibration, uncertainty-error overlap, correction, and Dice metrics.
Results
Current uncertainty methods perform similarly, are relatively well-calibrated at the dataset level, but tend to be miscalibrated at the subject level.
Takeaways & Limitations
Reliable failure detection requires subject-level uncertainty estimation and calibration assessment; auxiliary networks are a valid alternative applicable to previously trained segmentation models.
Takeaways & Limitations
Findings may differ for datasets containing very small structures, and uncertainty assessment remains influenced by differences in segmentation performance.
Abstract
from arXiv · showhide
Despite the recent improvements in overall accuracy, deep learning systems still exhibit low levels of robustness. Detecting possible failures is critical for a successful clinical integration of these systems, where each data point corresponds to an individual patient. Uncertainty measures are a promising direction to improve failure detection since they provide a measure of a system's confidence. Although many uncertainty estimation methods have been proposed for deep learning, little is known on their benefits and current challenges for medical image segmentation. Therefore, we report results of evaluating common voxel-wise uncertainty measures with respect to their reliability, and limitations on two medical image segmentation datasets. Results show that current uncertainty methods perform similarly and although they are well-calibrated at the dataset level, they tend to be miscalibrated at subject-level. Therefore, the reliability of uncertainty estimates is compromised, highlighting the importance of developing subject-wise uncertainty estimations. Additionally, among the benchmarked methods, we found auxiliary networks to be a valid alternative to common uncertainty methods since they can be applied to any previously trained segmentation model.
1 Introduction
Medical image segmentation has improved substantially, but robustness and patient-level failure detection remain important clinical challenges. Uncertainty measures may help users assess confidence, prioritize corrections, and monitor failures.
- Clinical integration requires mechanisms to detect and respond to segmentation failures at the individual-patient level.
- Uncertainty measures provide confidence information that can support user decision-making and focus corrections on high-uncertainty regions.
- Common approaches include test-time dropout, aleatoric uncertainty from a second output, and network ensembling.
- The study evaluates common uncertainty measures for reliability, benefit, and limitations, while developing practical evaluation recommendations.
2 Material & Methods2
The study evaluates several voxel-wise uncertainty methods on two medical image segmentation datasets using standardized segmentation models and multiple quality metrics. Methods include entropy-based baselines, MC dropout, aleatoric uncertainty, ensembles, and auxiliary networks.
- 2.1 Data: Experiments use BraTS and ISIC datasets with standardized preprocessing and distinct segmentation tasks.BraTS contains 265 subjects; ISIC contains 2000 training, 150 validation, and 600 testing images.
- 2.2 Segmentation model: A U-Net-like architecture was selected to reduce architectural influences while evaluating uncertainty reliability.The model uses four pooling/upsampling steps, dropout regularization, batch normalization, cross-entropy loss, and Adam optimization.
- 2.3 Uncertainty methods: The benchmark includes normalized softmax entropy, two MC-dropout configurations, aleatoric variance, ensembles, and two auxiliary-network designs.The ensemble combines K = 10 networks, while MC dropout uses T = 20 stochastic samples.
- 2.3 Uncertainty methods: Auxiliary networks learn voxel-wise uncertainty from segmentation errors and either use segmentation features or independently process images and predicted masks.
- 2.4 Assessing quality of uncertainties: Uncertainty quality is assessed using calibration, uncertainty-error overlap, and correction benefit, alongside Dice segmentation performance.Calibration uses reliability diagrams and expected calibration error; uncertainty-error overlap compares thresholded uncertainty with segmentation errors.
3 Results
Across both datasets, uncertainty methods showed good dataset-level calibration but substantial subject-level miscalibration, while no method consistently outperformed the others. Method behavior varied, with auxiliary networks performing well and excessive dropout harming results.
- Calibration: 28%/46% of BraTS/ISIC subjects were underconfident, while 32%/18% were overconfident, despite good dataset-level calibration.Subject-level calibration errors can therefore be masked when aggregated across all voxels.
- Method comparison: No uncertainty method consistently outperformed the others across the reported ECE, U-E, BnF, and Dice metrics.The comparison used average subject-level ECE, uncertainty-error overlap, correction-benefiting subjects, and Dice coefficient.
- Method comparison: Auxiliary networks were typically well-calibrated and benefited from strong segmentation performance of their baseline segmentation network.
- Method comparison: MC dropout generally improved ECE, U-E, and Dice over non-MC versions, whereas larger dropout amounts worsened performance.The reported pattern suggests that increasing dropout can reduce segmentation quality and calibration gains.
- Method comparison: The aleatoric method produced low uncertainty at segmentation-error locations and therefore provided little correction benefit.
4 Discussion
The study concludes that dataset-level calibration can conceal unreliable subject-level uncertainty estimates, motivating subject-level methods and calibration assessment. Ensembles and auxiliary networks are practical alternatives, but the findings are bounded by dataset structure and model-performance differences.
- Reliability: Voxel-wise uncertainty measures were generally well-calibrated at dataset level but tended to fail at subject level, where errors can average out.
- Implications: Subject-level uncertainty estimation should address the High-Dimension-Low-Sample-Size condition to improve practical reliability.
- Method choice: Ensembles ranked as the most reliable method when computational resources permit, while auxiliary networks offered comparable performance for previously trained segmentation models.
- Correction: For correction, maximum benefit was observed when combining low-precision segmentation models with uncertainty-based false-positive removal.
- Limitations: Findings may differ for datasets containing very small structures, and uncertainty assessment cannot fully decouple differences in segmentation performance.