PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Uncertainty Estimates for Predicting Segmentation Quality

Terrance DeVries, Graham W. Taylor

arXiv:1807.00502v1cs.CV

TL;DR

Deep-learning segmentation can fail silently, making it difficult for clinicians to identify unreliable results and understand where errors occur. The paper uses spatial uncertainty maps to predict image-level segmentation quality and finds that explicit uncertainty improves quality prediction; its evaluation is limited by the small ISIC 2017 dataset.

  • Problem

    Medical segmentation models may fail silently, while clinicians need spatial clues about errors and a way to identify poor segmentations for follow-up.

    Source: §1 · p. 1

  • Method

    The framework produces per-pixel uncertainty maps and uses them as an intermediate representation for image-level segmentation-quality prediction.

    Source: §1 · p. 1

  • Results

    Explicit uncertainty reduced RMSE by up to 0.03 points and detection error by up to 8 percentage points versus using no uncertainty information; maximum softmax probability, MC-dropout, and LCE produced similar improvements.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    Spatial uncertainty can aid image-level segmentation-quality prediction, and several recent uncertainty methods worked well in the skin-lesion setting.

    Source: §5 · p. 8

    The quality network was trained on the same dataset as the segmentation network, and a separate held-out dataset was not explored because ISIC 2017 was small.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

The use of deep learning for medical imaging has seen tremendous growth in the research community. One reason for the slow uptake of these systems in the clinical setting is that they are complex, opaque and tend to fail silently. Outside of the medical imaging domain, the machine learning community has recently proposed several techniques for quantifying model uncertainty (i.e.~a model knowing when it has failed). This is important in practical settings, as we can refer such cases to manual inspection or correction by humans. In this paper, we aim to bring these recent results on estimating uncertainty to bear on two important outputs in deep learning-based segmentation. The first is producing spatial uncertainty maps, from which a clinician can observe where and why a system thinks it is failing. The second is quantifying an image-level prediction of failure, which is useful for isolating specific cases and removing them from automated pipelines. We also show that reasoning about spatial uncertainty, the first output, is a useful intermediate representation for generating segmentation quality predictions, the second output. We propose a two-stage architecture for producing these measures of uncertainty, which can accommodate any deep learning-based medical segmentation pipeline.

1. Introduction

Deep-learning segmentation can fail silently, while clinicians need both spatial clues about errors and a way to identify poor cases. The paper links per-pixel uncertainty maps to image-level segmentation-quality predictions, reporting improved performance over alternative deep-learning approaches.

1. Introduction

  • Uncertainty estimates could help reviewers respond appropriately when predictions may be faulty, a concern for medical decisions based on model outputs.The paper notes that neural networks often fail silently, without indicating that a prediction may be wrong.Source: §1 · p. 1
  • For medical segmentation, spatial uncertainty maps could show clinicians where and why a model failed and inform manual correction.Poor automated segmentations may be referred to a qualified human for follow-up.Source: §1 · p. 1
  • The framework learns spatial uncertainty maps and uses them to improve image-level segmentation-quality prediction, outperforming alternative deep-learning techniques.The authors identify per-pixel uncertainty as a useful intermediate representation for predicting segmentation quality.Source: §1 · p. 1
  • The work unifies uncertainty estimation and image-level segmentation-quality prediction in a modular framework agnostic to the deep-learning architecture and uncertainty technique.The authors also extend Learning Confidence Estimates [4] from scalar to pixel-level output.Source: §1 · pp. 1–2

2. Related Work

Related work covers uncertainty estimation and segmentation-quality prediction as largely separate pursuits. It describes spatial uncertainty maps for pixel-level review, alongside approaches to image-level quality estimation that do not exploit uncertainty information.

2.1. Uncertainty Estimation

  • Uncertainty estimates can indicate likely incorrect predictions or out-of-distribution inputs, but Bayesian neural networks are computationally intractable in practice.The review describes sampling-based approximations and Deep Ensembles as resource-intensive, while learned uncertainty approaches are presented as more computationally efficient.Source: §2.1 · p. 2

2.2. Segmentation Quality Prediction

  • Spatial uncertainty maps visualize per-pixel confidence and can help users identify problematic regions when reviewing or modifying a segmentation.The cited works [11] associate uncertain regions with likely misclassification.Source: §2.2 · p. 2
  • Image-level quality prediction is useful in automated pipelines, where acceptable segmentations may still contain pixel-level uncertainty.Earlier approaches used hand-crafted features, which the review describes as limited in expressiveness and transferability across imaging modalities.Source: §2.2 · p. 2
  • Adversarial quality estimation can improve segmentation, but its output offers little human-interpretable meaning beyond whether a segmentation looks realistic.Other methods predict metrics such as Jaccard or Dice, including QualityNet and Reverse Classification Accuracy.Source: §2.2 · p. 2
  • The proposed system passes an image, segmentation prediction, and uncertainty map to a quality network that estimates segmentation quality; the map supports interpretation while the estimate can flag poor cases.The review notes that prior quality-prediction methods do not exploit uncertainty information that could improve their accuracy.Source: §2.2 · p. 3

3. Estimating Segmentation Quality with Uncertainty Information

The framework uses a segmentation network to produce a mask and pixelwise uncertainty, then a second network predicts segmentation quality from the image, mask, and uncertainty. It supports different quality metrics and four uncertainty-estimation approaches.

3. Estimating Segmentation Quality with Uncertainty Information

  • A segmentation network f produces class logits and uncertainty estimates, while a second network g uses the image, predicted mask, and uncertainty map to predict segmentation quality.The predicted quality can represent any segmentation-based evaluation metric or multiple metrics.Source: §3 · p. 3
  • If f performs very well on its training set, g may need holdout examples because too few poor segmentations can bias it toward predicting good quality.Source: §3 · p. 3

3.1. Maximum Softmax Probability

  • Maximum softmax probability uses the largest classwise softmax output as confidence, calculated per pixel to form an input-resolution uncertainty map.Source: §3.1 · p. 3

3.2. Monte-Carlo Dropout

  • MC-dropout samples a dropout-trained network at inference to approximate a distribution over outputs and estimate epistemic uncertainty, associated with model parameters.The described implementation averages softmax probabilities across T = 20 samples and estimates uncertainty using entropy.Source: §3.2 · p. 4

3.3. Heteroscedastic Classifier Neural Network

  • The heteroscedastic classifier neural network learns per-pixel variance alongside class logits to model aleatoric uncertainty, adding variance-scaled Gaussian noise to probability logits during training.The variance estimate is passed through softplus to ensure it is non-negative.Source: §3.3 · p. 4

3.4. Learned Confidence Estimates

  • Learned Confidence Estimates interpolate between predicted and target distributions during training according to confidence; a log penalty discourages uniformly low confidence.Low-confidence predictions are pushed toward the correct answer, while high-confidence predictions remain unchanged.Source: §3.4 · p. 4

4. Experiments

Experiments on ISIC 2017 skin-lesion segmentation evaluate spatial uncertainty maps and use uncertainty to predict image-level Jaccard quality. Explicit uncertainty improves quality prediction, while several methods show similar gains.

4. Experiments

  • The evaluation uses 2,750 ISIC 2017 dermoscopic images, with expert binary masks, resized to 224 × 224 pixels.The official splits contain 2,000 training, 150 validation, and 600 test images; segmentation models score 0.73±0.02 Jaccard index.Source: §4 · p. 4

4.2. Segmentation Quality Prediction

  • The quality network predicts Jaccard index from the image, predicted mask, and confidence, with variants testing different uncertainty measures and a no-uncertainty input.RCA and QualityNet are the comparison methods.Source: §4 · pp. 4, 6, §4.2 · p. 8

4.1. Uncertainty Maps

  • Maximum softmax probability, MC-dropout, and LCE highlight incorrectly segmented regions, whereas HCNN usually marks only a low-uncertainty band around its prediction.When predictions closely match ground truth, all methods produce a tight uncertainty ring along segmentation borders.Source: Figure 2 · p. 5, §4.1 · p. 6

4.2. Segmentation Quality Prediction

  • Explicit uncertainty reduced RMSE by up to 0.03 points and detection error by up to 8 percentage points versus providing no uncertainty information.Maximum softmax probability, MC-dropout, and LCE produced similar improvements; HCNN improved detection error by only 5 points and did not improve RMSE.Source: §4.2 · p. 8
  • QualityNet performs roughly like the no-uncertainty baseline, while RCA performs only slightly better than random on the varied ISIC skin lesions.The authors suggest RCA's design for registered images with little variation may explain its poor performance on lesions varying in color, texture, shape, size, and location.Source: §4.2 · p. 8
  • Maximum probability, MC-dropout, and LCE identify poor-quality segmentations better than HCNN or the no-uncertainty baseline.Most false positives—poor segmentations rated highly—are attributed to corrupted labels or lazy annotations.Source: Figure 3 · p. 7, §4.2 · p. 8
  • Quality estimates rarely fall below 0.2, likely because poor-quality segmentations are rare in the data used to train the estimator.The quality estimator was trained on the same dataset as the segmentation network; a held-out dataset was not explored because ISIC 2017 is small.Source: §4.2 · p. 8

5. Conclusion

The conclusion presents a unified framework for pixel- and image-level uncertainty in automated segmentation. On skin-lesion segmentation, explicit spatial uncertainty aided Jaccard quality prediction, and several uncertainty methods worked well.

5. Conclusion

  • The framework derives pixel- and image-level uncertainty within one end-to-end system and demonstrates the approach on skin-lesion segmentation.Explicit spatial uncertainty aided prediction of segmentation Jaccard index, and several recent uncertainty methods worked well in this setting.Source: §5 · p. 8
Loading 1807.00502v1…