PaperlayerView exact revision on arXiv

Source-linked AI summary

Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation

Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, Jeffrey Chiang, Zhihao Wu, Xiaowei Ding

arXiv:1908.10454v2eess.IVcs.CVcs.LG

TL;DR

Medical image segmentation rarely has the large, high-quality annotated datasets required by advanced models, because data and annotation acquisition are expensive. This survey reviews solutions for scarce and weak annotations, summarizes technical and empirical findings, and compares their requirements and benefits. It concludes by highlighting cost-gain trade-offs and available strategies for imperfect datasets.

  • Problem

    Medical image segmentation lacks consistently large, representative, high-quality annotations, with scarce and weak annotations common because medical data and labeling are expensive.

  • Method

    The paper reviews methods for scarce and weak annotations, covering annotation, learning, regularization, and refinement strategies while comparing their data requirements and implementation difficulty.

  • Results

    The survey identifies diverse techniques for handling scarce and weak annotations and compares them by performance gains, implementation difficulty, and required data resources.

  • Takeaways & Limitations

    The review provides recommended solutions based on cost-gain trade-offs and increases awareness of strategies for imperfect medical image segmentation datasets.

  • Takeaways & Limitations

    Semi-supervised methods without pseudo-annotations improve Dice scores mainly for limited labeled training sets and become less effective as training-set size grows.

Abstract

from arXiv · show

The medical imaging literature has witnessed remarkable progress in high-performing segmentation models based on convolutional neural networks. Despite the new performance highs, the recent advanced segmentation models still require large, representative, and high quality annotated datasets. However, rarely do we have a perfect training dataset, particularly in the field of medical imaging, where data and annotations are both expensive to acquire. Recently, a large body of research has studied the problem of medical image segmentation with imperfect datasets, tackling two major dataset limitations: scarce annotations where only limited annotated data is available for training, and weak annotations where the training data has only sparse annotations, noisy annotations, or image-level annotations. In this article, we provide a detailed review of the solutions above, summarizing both the technical novelties and empirical results. We further compare the benefits and requirements of the surveyed methodologies and provide our recommended solutions. We hope this survey article increases the community awareness of the techniques that are available to handle imperfect medical image segmentation datasets.

1. Introduction

Medical image segmentation models have advanced substantially, but their success depends on datasets that are rarely large, representative, and carefully annotated. This review examines techniques for handling scarce and weak annotations and summarizes their technical and empirical contributions.

  • Advanced segmentation models still depend on large, representative, carefully annotated datasets that are rarely available in medical imaging.Data and annotations are expensive to acquire, making imperfect training datasets common.
  • The review distinguishes scarce annotations from weak annotations as the two major dataset limitations.Scarce annotations provide limited annotated data, whereas weak annotations may be sparse, noisy, or image-level.
  • The article reviews techniques for handling these limitations while summarizing their technical novelties and empirical results.

2. Related works

Earlier surveys addressed broad medical imaging applications, GANs, or general learning paradigms. This survey instead focuses specifically on scarce and weak annotations in medical image segmentation.

  • Earlier surveys covered medical imaging applications, GANs, or semi-supervised, multi-instance, and transfer learning across medical image analysis.
  • This survey focuses on scarce and weak annotations specifically in medical image segmentation.
  • Its scope is motivated by segmentation’s stronger supervision requirements and greater vulnerability to annotation quality and quantity.

3. Organization of survey

The survey organizes dataset limitations into scarce and weak annotations, then groups methods by how they address each limitation and compares them by performance, implementation difficulty, and data requirements.

  • Dataset limitations: Scarce annotations cover datasets where only a small fraction of images are densely annotated, while weak annotations include sparse, noisy, or image-level annotations.
  • Scarce annotations: Scarce-annotation methods enlarge the training set through augmentation, external labeled data, cost-effective annotation, or unlabeled data.
  • Scarce annotations: Other scarce-annotation methods strengthen regularization in the input space, output space, or gradients through representation changes, shape priors, or multi-task supervision.
  • Weak annotations: Weak-annotation methods are grouped by image-level, sparse, and other annotation weaknesses, including class activation maps and selective losses.
  • Comparison: The survey compares methods by performance gain, implementation difficulty, and required data resources, then recommends solutions using a cost-gain trade-off.

4. Problem I: Scarce annotation

Scarce annotation is a common obstacle for supervised medical image segmentation. Traditional augmentation, natural-image transfer learning, and weight regularization only partially address it, motivating reactive refinement and proactive training-set expansion or paradigm changes.

  • Scarce annotation is common in supervised deep learning for medical image segmentation.
  • Traditional solutions: Traditional data augmentation is limited because augmented examples remain highly correlated with the original training set.
  • Traditional solutions: Transfer learning from natural images benefits 2D medical segmentation but not the common 3D segmentation setting.
  • Modern approaches: Reactive methods refine predictions after segmentation, whereas proactive methods enlarge training data through cost-effective annotation or synthetic generation, or alter the training paradigm.

4.1. Data augmentation

Data augmentation addresses scarce annotations by creating additional training examples through traditional transforms, label mixing, or synthetic image generation. More expressive synthesis methods can improve performance and cover rare conditions, but they demand more complex training resources.

  • Traditional data augmentation: Traditional augmentation manipulates image quality, appearance, or layout to generate additional medical-image training examples.Examples include noise, blur, intensity changes, rotation, scaling, and deformation.
  • Mixing augmentation: Mixup creates new images and corresponding masks by convexly combining pairs of labeled examples.The mixing coefficient λ is sampled from a beta distribution, and the resulting labels are typically soft.
  • Synthetic augmentation: Synthetic augmentation includes same-domain generation and cross-domain projection of labeled data into the target domain.Cross-domain synthesis is reviewed with domain adaptation, while same-domain methods use CycleGANs, conditional GANs, and transformation networks.
  • Synthetic augmentation: Spatially constrained CycleGANs generated microscopy images that were more effective than CycleGAN-generated images for improving the base segmentation model.The spatial constraints guide nuclei toward desired locations and orientations.
  • Synthetic augmentation: 2 points increase in Dice was reported when synthetic pleural nodules were added to training for lung segmentation.The comparison was against training using only real images, evaluated on 34 LIDC images with peripheral nodules.
  • Synthetic augmentation: 3 points increase in Dice was reported over atlas-based augmentation in a 1-shot medical image segmentation setting.The passage states that only one labeled example was available for training and that persistence with larger labeled sets was unclear.
  • Summary: Image synthesis can provide greater appearance variability and help with data scarcity and rare conditions, but usually requires adversarial training and additional labeled or unlabeled data.Traditional transforms are simpler but produce augmented images correlated with the originals.

4.2. Leveraging External Labeled Datasets

External labeled datasets address scarce annotations through transfer learning, domain adaptation, and dataset fusion. These approaches differ in their data requirements, handling of target labels, and mechanisms for bridging or modeling domain differences.

  • Transfer learning: Transfer learning uses a large external labeled dataset to pre-train a model before fine-tuning it on the target dataset.Natural-image transfer to medical segmentation is less explored partly because 3D medical images do not align naturally with 2D natural-image models.
  • Domain adaptation: Domain adaptation bridges distribution gaps by learning common latent representations or translating images between domains.Distribution shifts can arise from scanners, acquisition protocols, or patient populations.
  • Domain adaptation: GANs and CycleGANs are frequently used for image reconstruction and domain translation through competing generator-discriminator networks and cycle consistency.CycleGAN uses two inverse mappings, while the cycle-consistency loss encourages recovery of the input after both mappings.
  • Domain adaptation: 9% improvement in Dice over existing state-of-the-art domain adaptation techniques was reported for one reviewed approach.The cited passage reports this improvement in the context of preventing encoder over-fitting.
  • Domain adaptation: When both source and target labels are available, domain adaptation can place shared latent representations at different locations within the network.Evaluated designs included shared decoders, shared encoders and decoders, shared encoders with separate decoders, and shared intermediate representations.
  • Dataset fusion: 1.5% improvement in Dice over state-of-the-art single-dataset approaches was reported for a multi-dataset training scheme.The cited experiment used an 80/20 training and testing split.
  • Dataset fusion: Dataset fusion jointly leverages multiple datasets by learning to discriminate between domains and conditioning segmentation on the input domain.The review distinguishes this mechanism from domain adaptation, which minimizes domain shift.
  • Summary: Models trained with target-domain annotations generally generalize better to that domain, whereas methods without target labels remain applicable to unlabeled target images.The review notes that many domain-adaptation methods require adversarial training, which makes them tricky to train.

4.3. Cost-effective Annotation

Cost-effective annotation methods reduce the burden of obtaining dense labels by selecting informative unlabeled samples or refining predictions through limited user interaction. The reviewed approaches trade annotation cost and interaction effort against computational requirements and segmentation quality.

  • Active Learning: Active learning iteratively selects unlabeled images for annotation using model-based criteria, then fine-tunes the segmentation model with the augmented training set.Each iteration runs the model on unlabeled images, selects a batch, obtains annotations, and updates the model.
  • Active Learning: Suggestive annotation combines ensemble-based pixel uncertainty with visual-diversity filtering to select informative, nonredundant images.Pixel uncertainty is computed from prediction variance and averaged over each image before visually similar samples are removed.
  • Active Learning: Annotation-cost-aware active learning formulates sample selection as a 0-1 knapsack problem that maximizes uncertainty under a cost threshold.The method estimates uncertainty with patch-level FCNs and predicts image annotation cost from mask perimeter and connected-component counts.
  • Active Learning: Ensemble-based uncertainty estimation is computationally expensive because segmentation-model ensembles must be retrained after each annotation round.Dropout-based repeated inference offers a more efficient alternative for estimating uncertainty.
  • Summary: One-shot active learning can perform comparably to an iterative alternative, while one user click enabled unseen-structure segmentation at Dice 0.64 versus 0.56 for the best automated model.The interactive result concerns colon cancer segmentation in 126 abdominal CT scans.

4.4. Leveraging Unlabeled Data

The survey reviews three ways to use unlabeled medical images for segmentation: self-supervised pre-training, pseudo-annotation learning, and semi-supervised learning without pseudo annotations. These approaches can improve performance with limited labels, but pseudo-label noise and increasing architectural complexity or dataset size constrain their benefits.

  • Self-supervised Pre-training: Self-supervised learning pre-trains segmentation networks on unlabeled medical images using proxy tasks before fine-tuning on labeled target data.Reviewed proxy tasks include image-to-scalar classification or regression and image-to-image restoration such as denoising or colorization.
  • Self-supervised Pre-training: With 25 fine-tuning images, self-supervised pre-training achieved a Dice score of 0.61 versus 0.57 for both COCO-pre-trained and scratch-trained models.Its advantage over randomly initialized weights diminished when the fine-tuning set increased from 25 to 400 images.
  • Self-supervised Pre-training: Models Genesis extends self-supervision by combining a library of image-restoration schemes that share an encoder and decoder during training.The shared architecture makes the framework scalable to a large library of surrogate tasks.
  • Semi-supervised learning with pseudo annotations: Pseudo-annotation methods train on model-generated masks for unlabeled images, with approaches differing in model initialization, mask generation, and treatment of unreliable regions.Examples include K-means initialization without labeled data, iterative refinement, data distillation, and model or two-stream consensus.
  • Semi-supervised learning with pseudo annotations: Four points in Dice separated a pseudo-labeling method from labeled-only training with 50 labeled and 100 unlabeled CT scans, but the gain fell to 1 point when 100 labeled scans were used.Another study using 12 labeled and 749 unlabeled MR images reported a 1.6-point Dice increase over its labeled-only base model.
  • Semi-supervised learning without pseudo annotations: Semi-supervised learning without pseudo annotations combines segmentation on labeled data with an unsupervised task on unlabeled data, avoiding direct vulnerability to annotation noise.These methods generally improve Dice by a couple of points, but use complex architectures and become less effective as the training set grows.

4.5. Regularized Training

Regularized training addresses scarce annotations by changing image representations, adding auxiliary tasks, or imposing shape priors on predictions. These approaches can improve segmentation with limited additional annotation requirements, although multi-task learning requires extra labels.

  • Altered image representation: Altered image representations simplify the learning problem by making inputs more informative or compact, reducing dependence on large training sets.The review covers multi-scale and multi-view 2D patches, fused orthogonal-view predictions, 2.5D projections, and handcrafted feature augmentation for 3D segmentation.
  • Altered image representation: 1% Dice improvement over majority-voting fusion was reported for pancreas and multi-organ segmentation using fused predictions from three orthogonal views.The pancreas experiments used 82 abdominal CT volumes, while multi-organ experiments used 50 high-resolution abdominal CT scans.
  • Altered image representation: 6% Dice improvement over a single-scale baseline was achieved by augmenting registered T1 and FLAIR MR representations with dense handcrafted features for white matter hyperintensity segmentation.The result was reported on a test set of 46 cases.
  • Multi-task learning: Multi-task learning uses shared representations with parallel task heads, allowing tasks such as classification, detection, reconstruction, and segmentation to regularize one another.The additional task may require annotations, whereas image reconstruction is described as an unsupervised auxiliary task.
  • Shape regularization: Shape regularization imposes geometric or structural priors on predicted regions, including star-shape and boundary constraints.A star-shape loss improved U-Net Dice by 3.0% on ISIC 2017; partial-supervision methods with shape priors improved Dice by 1.69% with ResNet-101 and 0.45% with 3D U-Net.
  • Summary: Input-space and shape regularization require no further annotations, while multi-task learning does; the three forms may provide complementary advantages.The review notes that combining these forms had not yet been studied in prior work.

4.6. Post segmentation refinement

Post-segmentation refinement uses CRF variants to impose spatial and appearance consistency on coarse predictions, producing sharper and more coherent masks. Local and fully connected CRFs show mixed results in 3D, while recurrent formulations enable end-to-end integration with CNNs.

  • CRF refinement: CRFs refine coarse masks through smoothness and appearance terms that encourage coherent labels, remove isolated regions, and sharpen boundaries.The CRF produces a maximum a posteriori estimate using unary and pairwise potentials over image-conditioned pixel labels.
  • CRF variants: Local CRFs restrict pairwise interactions to neighboring pixels, whereas fully connected CRFs model all pixel pairs using efficient mean-field inference.The fully connected formulation reduces inference complexity from quadratic to linear in the number of pixels.
  • Fully connected CRF: 3.7% Dice improvement over Random Forests, 0.3% over an ensemble, and 0.7% over a patch-based CNN were reported for 3D FC-CRF on BraTS.The authors also noted that configuring 3D FC-CRF was laborious.
  • RNN-CRF: RNN-CRF reformulates mean-field inference as CNN layers and recurrent hidden states, making the CRF-CNN pipeline trainable end to end.This approach achieved state-of-the-art vessel segmentation performance on three public fundus datasets in the reviewed study.
  • Empirical results: Six to seven Dice points above a baseline U-Net and other CRF implementations were reported when learned CNN features were used for refinement on WMH scans.The evaluation used 60 MR scans from the WMH 2017 Challenge.
  • Summary: CRF refinement is effective in many 2D applications, but 3D extensions have produced mixed results.The review attributes reduced 3D effectiveness partly to extensive manual tweaking and configuration difficulty.

5. Problem II: Weak Annotations

Weak annotations reduce the cost of creating segmentation training data, especially for 3D images, but provide less detailed supervision than strong masks. The reviewed literature indicates that models can still be trained effectively with image-level, sparse, or noisy annotations.

  • Weak annotation types: Weak annotations comprise image-level labels, sparse annotations covering only some slices or pixels, and noisy or model-generated annotations.Strong annotations are fully contoured segmentation masks, whereas weak annotations are obtained at significantly lower annotation cost.
  • Motivation: Strong manual segmentation masks are time consuming and tedious, particularly for 3D images.This annotation burden motivates the use of weaker alternatives.
  • Supported training: Recent research has shown that fairly effective segmentation models can be trained without strong annotations.The review summarizes methods using image-level, sparse, and noisy supervision.

5.1. Learning with Image Level Labels

Image-level supervision is used for segmentation through class activation maps or multiple-instance learning. These approaches convert image- or slide-level information into pixel-level predictions and improve performance over their respective baselines in the reviewed studies.

  • Overview: Weakly supervised segmentation commonly uses bounding boxes or image-level labels with class activation maps or multiple-instance learning.Both approaches are reviewed as ways to exploit supervision weaker than dense masks.
  • Class activation maps: Class activation maps combine feature maps to produce class-specific saliency maps that can be binarized into segmentation masks.The reviewed CAM methods use intermediate outputs of classification networks to generate segmentation cues.
  • Class activation maps: 10% Dice improvement over a CAM baseline was achieved for diagnostic brain tumor segmentation in confocal laser endomicroscopy images.The study used 8,345 slices per class from 1,010 patients with a 4:1:1 train-validation-test split.
  • Multiple-instance learning: Multiple-instance learning assigns labels to bags rather than individual instances, leaving positive bags without known positive-instance locations.Directly assigning a positive bag label to every instance would introduce noisy supervision.
  • Multiple-instance learning: 6% F-measure improvement around boundaries and 2% overall improvement were reported over a baseline using area constraints and multi-level predictions for histopathology cancer segmentation.The study used 250 cancer and 500 non-cancer images for training, with 80 images of each class for testing.
  • Summary: The reviewed image-level-label methods improve performance over their respective baselines using either CAM-generated masks or MIL-based treatment of label noise.The review presents CAMs and MIL as distinct strategies for the weakest form of annotation.

5.2. Learning with Sparse Labels

Sparse-label methods address incomplete pixel annotations either by reconstructing missing mask regions or by modifying the training objective so only labeled pixels guide learning.

  • Incomplete annotations may cover only some slices of a 3D volume or some pixels of a 2D image.
  • Selective-loss methods avoid mask completion by allowing only labeled pixels to contribute to training loss.
  • Mask-completion methods artificially reconstruct unlabeled regions and use the completed masks for training.
  • Sparse-label methods include active learning, iterative mask refinement, and artificially generated masks from points, lines, scribbles, or diameter markings.
  • 0.4→0.86 IoU was reported when annotation coverage increased from 2.5% to 8.9% in sparse kidney-tubule annotations.
  • 1% and 4% Dice improvements resulted from instance-based and mini-batch class balancing, respectively, when training entirely on sparsely annotated images.

5.3. Learning with Noisy Labels

Noisy-label methods address inaccurate or ambiguous segmentation boundaries using quality-aware losses, pseudo-annotation filtering, and either iterative mask refinement or direct robust training.

  • Segmentation label noise arises from random annotator errors and disagreements between readers about ambiguous lesions.
  • Quality-aware weighted cross entropy scales each pixel’s contribution according to the model’s perceived annotation quality.
  • Pseudo-annotation methods reduce the influence of noisy model-generated labels through gradient downweighting or reliability assessment.
  • Noise-handling approaches either refine noisy masks iteratively or leave them unchanged while adjusting training.

6. Discussion

The discussion organizes methods by annotation scarcity, annotation weakness, data requirements, annotation cost, implementation difficulty, and performance, then offers strategy recommendations for different constraints.

  • Combining scarce and weak-annotation solutions can exploit datasets that contain dense masks for some images and image-level labels for many others.
  • Table 7 groups methodologies by strategy and uses colors to indicate their additional data requirements.
  • Low-data-requirement methods use only the available labeled segmentation dataset and should be used wherever possible.
  • Shape regularization, mixed-image augmentation, same-domain synthesis, and 2D CRF post-processing are recommended alongside traditional augmentation for scarce annotations.
  • Self-supervised pre-training is identified as promising among medium-data-requirement methods because it needs unlabeled data and usually minor architectural changes.
  • Active learning and interactive segmentation are prioritized when experts must annotate more data, because they use expert input cost-effectively.
  • Sparse annotations can approach strong-supervision performance but may be unsuitable for small lesions and remain iterative and insufficiently user-friendly.
  • Modified CAM methods are recommended for image-level labels because they improve Dice over direct CAM approaches while remaining only a few Dice points below full supervision.

7. Conclusion

The survey reviews solutions for scarce and weak medical-image segmentation annotations and compares them by resources, implementation difficulty, and performance gains to identify cost–gain trade-offs.

  • Scarce-annotation solutions range from expert-in-the-loop active learning and interactive segmentation to automated use of unlabeled, synthetic, or related-domain labeled data.
  • Weak-annotation solutions address sparse, noisy, and image-level annotations while highlighting methodologies with favorable cost–gain trade-offs.
Loading 1908.10454v2…