Source-linked AI summary
Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation
Tanya Nair, Doina Precup, Douglas L. Arnold, Tal Arbel
TL;DR
Small MS lesions are clinically important, but deep-learning systems provide deterministic predictions and have limited evidence for small-lesion detection. This paper evaluates MC-dropout uncertainty measures in a 3D MS lesion CNN, aggregates voxel uncertainties into lesion scores, and finds that uncertainty-based filtering provides superior operating points to sigmoid thresholding.
Problem
Small MS lesions must be detected for clinical assessment and treatment evaluation, while deterministic deep-learning outputs provide no prediction uncertainty and small-lesion superiority remains unestablished.
Method
The paper trains a 3D MS lesion CNN with dropout, computes four voxel-based uncertainty measures, and aggregates voxel uncertainties into lesion-level scores.
Results
Uncertainty-based filtering improves remaining-prediction performance across operating points and provides more robust, principled operating points than sigmoid-output thresholding.
Takeaways & Limitations
Uncertainty estimates can help clinicians assess whether to accept or reject high-uncertainty lesions or examine uncertain lesion boundaries.
Abstract
from arXiv · showhide
Deep learning (DL) networks have recently been shown to outperform other segmentation methods on various public, medical-image challenge datasets [3,11,16], especially for large pathologies. However, in the context of diseases such as Multiple Sclerosis (MS), monitoring all the focal lesions visible on MRI sequences, even very small ones, is essential for disease staging, prognosis, and evaluating treatment efficacy. Moreover, producing deterministic outputs hinders DL adoption into clinical routines. Uncertainty estimates for the predictions would permit subsequent revision by clinicians. We present the first exploration of multiple uncertainty estimates based on Monte Carlo (MC) dropout [4] in the context of deep networks for lesion detection and segmentation in medical images. Specifically, we develop a 3D MS lesion segmentation CNN, augmented to provide four different voxel-based uncertainty measures based on MC dropout. We train the network on a proprietary, large-scale, multi-site, multi-scanner, clinical MS dataset, and compute lesion-wise uncertainties by accumulating evidence from voxel-wise uncertainties within detected lesions. We analyze the performance of voxel-based segmentation and lesion-level detection by choosing operating points based on the uncertainty. Empirical evidence suggests that uncertainty measures consistently allow us to choose superior operating points compared only using the network's sigmoid output as a probability.
1 Introduction
MS lesion analysis must detect lesions of every size, including 3–5-voxel lesions, because lesion counts support clinical assessment and treatment evaluation. The paper investigates MC-dropout uncertainty measures as an alternative to deterministic predictions and sigmoid thresholding.
- MS lesions can be as small as 3–5 voxels, yet detecting and segmenting lesions of all sizes supports disease staging, prognosis, and treatment-efficacy evaluation.
- Current deep-learning methods have not yet been shown to outperform other machine-learning methods for detecting small lesions, risking errors in patient lesion counts.
- Deterministic deep-learning predictions lack the uncertainty information provided by traditional Bayesian models, while Bayesian deep learning faces implementation and training-time challenges.
- The paper compares multiple MC-dropout uncertainty measures for deep-learning lesion segmentation and detection in medical images.
- Choosing operating points with dropout uncertainty measures is more robust and principled than thresholding the network’s sigmoid output.
2 Proposed Method
The proposed method uses a 3D CNN with dropout to produce segmentation predictions and four voxel-level uncertainty measures, then aggregates voxel evidence into lesion-level uncertainty. Uncertainty thresholds filter predictions at voxel and lesion levels.
- Network architecture: A 3D fully convolutional CNN processes multimodal MRI volumes with dropout, using the mean of T stochastic segmentations as the single prediction.The network assigns lesion/non-lesion labels to all voxels.
- Network architecture: Additive skip connections replace concatenation to reduce the network’s parameter count.
- Dropout as a Bayesian Approximation: Variational inference approximates the intractable weight posterior by minimizing KL divergence, with dropout sampling approximating the posterior over lesion-label predictions.
- Uncertainty measures: The method computes four voxel-based uncertainties: predictive variance, MC sample variance, predictive entropy, and mutual information.
- Uncertainty measures: Predictive variance is learned during training from Gaussian output noise without explicit uncertainty labels, while MC sample variance uses the variance across T stochastic segmentations.
- Uncertainty measures: Predictive entropy summarizes information in the voxel-wise predictive density, and mutual information compares expected-prediction entropy with expected sample-wise entropies.
- Prediction filtering: Uncertainty filtering retains predictions only when voxel uncertainty falls below a chosen threshold in addition to the segmentation threshold.
- Lesion-level uncertainty: Lesion-level uncertainty is computed by summing log voxel uncertainties under a conditional-independence assumption, then rescaled separately for each measure before lesion-wise filtering.
3 Experiments and Results
The experiments evaluate uncertainty-based filtering for voxel and lesion predictions on a large clinical RRMS MRI dataset. Across uncertainty measures, filtering improves retained-prediction performance, especially for small lesions, while medium and large lesions decline slightly.
- Dataset: 1064 RRMS patients comprised the proprietary, multi-site, multi-scanner clinical trial evaluation dataset.MRI sequences were acquired annually over 24 months at 1mm x 1mm x 3mm resolution.
- Uncertainty-based filtering: Uncertainty-based filtering improved performance at all operating points, even when excluding only 2% of the most uncertain lesions.The improvement resulted from reducing both false-positive and false-negative assertions among retained predictions.
- Lesion-size analysis: Small lesions benefited from uncertainty filtering regardless of the uncertainty measure, and small lesions constituted 40% of the dataset.The model performed less well on small lesions, so removing uncertain predictions improved overall performance for this size group.
- Lesion-size analysis: Medium and large lesions showed slightly reduced performance after uncertainty filtering compared with non-thresholded segmentations.The reported size-stratified analysis contrasts this decrease with the improvement observed for small lesions.
- Uncertainty measures: At lesion level, the four uncertainty measures tended to rank lesions in the same certainty order despite operating on different scales.This produced similar lesion filtering at different thresholds across measures.
4 Conclusion
The conclusion presents a 3D MS lesion CNN that estimates voxel- and lesion-level uncertainty and reports improved detection accuracy for small lesions. These uncertainty estimates can help clinicians review uncertain predictions and boundaries.
- 4 Conclusion: The study developed a 3D MS lesion CNN providing four voxel-based uncertainties and accumulated them to estimate lesion-level uncertainty.The four measures were predictive variance, MC sample variance, predictive entropy, and mutual information.
- 4 Conclusion: Uncertainty filtering greatly improved lesion detection accuracy for small lesions, which comprised 40% of the dataset.The conclusion interprets high uncertainty as reflecting incorrect predictions in this lesion group.
- 4 Conclusion: Uncertainty measures can help clinicians or radiologists accept or reject high-uncertainty lesions and further analyze uncertain lesion boundaries.The authors state that this could facilitate wider adoption of deep learning in clinical workflows.