PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Semantic Segmentation of Natural and Medical Images: A Review

Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, Ghassan Hamarneh

arXiv:1910.07655v4cs.CVcs.LGeess.IV

TL;DR

Semantic image segmentation assigns image pixels to classes and supports scene understanding and medical applications, but its deep learning literature spans diverse methods and settings. This review comprehensively organizes natural and medical image segmentation approaches into six categories, examines their variants and loss functions, and discusses limitations and future directions. Across the reviewed natural-image results, architectural advances are prominent, with mean IoU rising from 62.2% for FCN to 89.0% for DeepLabV3+ on PASCAL VOC 2012.

  • Problem

    Semantic segmentation is important for scene understanding and medical applications, while accurate pixel-level annotation is expensive and segmentation methods span diverse modalities and supervision settings.

  • Method

    The review categorizes natural and medical deep learning segmentation methods into six groups and analyzes their variants, loss functions, limitations, and future directions.

  • Results

    Architectural improvements dominate the reviewed natural-image literature, with mean IoU increasing from 62.2% for FCN to 89.0% for DeepLabV3+ on PASCAL VOC 2012.

  • Takeaways & Limitations

    The review provides a framework for comparing segmentation contributions across natural and medical images and identifies future research directions.

  • Takeaways & Limitations

    Limited medical imaging data and legal, ethical, and privacy constraints make universal solutions across imaging modalities difficult to develop.

Abstract

from arXiv · show

The semantic image segmentation task consists of classifying each pixel of an image into an instance, where each instance corresponds to a class. This task is a part of the concept of scene understanding or better explaining the global context of an image. In the medical image analysis domain, image segmentation can be used for image-guided interventions, radiotherapy, or improved radiological diagnostics. In this review, we categorize the leading deep learning-based medical and non-medical image segmentation solutions into six main groups of deep architectural, data synthesis-based, loss function-based, sequenced models, weakly supervised, and multi-task methods and provide a comprehensive review of the contributions in each of these groups. Further, for each group, we analyze each variant of these groups and discuss the limitations of the current approaches and present potential future research directions for semantic image segmentation.

1 Introduction

This review examines deep learning-based semantic segmentation for natural and medical images, motivated by its roles in scene understanding and medical image analysis. It organizes the literature into six contribution categories and surveys methods, loss functions, datasets, and future directions.

  • Motivation: Semantic segmentation partitions an image into nonoverlapping regions whose union covers the entire image, supporting scene understanding and medical applications.Medical uses include image-guided interventions, radiotherapy, and improved radiological diagnostics.
  • Scope: The review covers segmentation research for natural images and both 2D and volumetric medical images across multiple imaging modalities.Medical modalities discussed include X-ray, visible-light imaging, MRI, PET, CT, and ultrasound.
  • Review organization: The literature is grouped into architectural, optimization-function, data-synthesis, weakly supervised, sequenced, and multi-task improvements.Figure 1 presents these categories alongside a timeline of influential papers.
  • Review organization: The review studies popular segmentation loss functions, including their behavior under varying levels of false-positive and false-negative predictions.Figure 2 maps the reviewed categories onto a high-level deep semantic segmentation pipeline.
  • Review organization: For each category, the review first discusses non-medical datasets and then surveys corresponding improvements for medical images.The paper also identifies limitations and proposes future research directions.

2 Network Architectural Improvements

This section reviews CNN-based architectural advances for dense semantic segmentation, from fully convolutional networks and encoder-decoder designs to multi-scale context, attention, and adversarial approaches.

  • Fully Convolutional Networks: Fully convolutional networks produce pixel-wise predictions by upsampling convolutional activation maps.They fuse outputs from shallower layers to preserve contextual spatial information during segmentation.
  • Encoder-decoder Networks: SegNet decodes low-resolution encoder features using max-pooling indices and trainable decoder filters, unlike FCN’s learned deconvolution and skip-feature addition.SegNet has no fully connected layers and is entirely convolutional.
  • Encoder-decoder Networks: U-Net combines a contracting path for context with a symmetric expanding path and skip connections for precise localization.The skip connections improved accuracy and addressed vanishing gradients.
  • Encoder-decoder Networks: Tiramisu adapts densely connected networks to segmentation with dense blocks, separate downsampling and upsampling transitions, and skip connections.Feature-map concatenation grows linearly in the downsampling path but not in the upsampling path.
  • Multi-scale and Later Improvements: Spatial pyramid networks encode multi-scale contextual information, while encoder-decoder networks progressively recover spatial information for sharper object boundaries.These approaches modify feature fusion and contextual representation in deep segmentation networks.
  • Multi-scale and Later Improvements: DeepLabV3+ combines dilated convolutions and feature pyramid pooling with a decoder that refines segmentation, especially along object boundaries.DeepLabV3+ outperformed many state-of-the-art segmentation networks on PASCAL VOC 2012 and Cityscapes.

3 Architectural Improvements Applied to Medical Images

Medical-image segmentation architectures address volumetric processing, computational cost, contextual preservation, data synthesis, attention, adversarial learning, and temporal dependencies across image sequences.

  • Efficient Architectures: Medical segmentation methods compress deep models to process larger images or volumes and support real-time inference.Neural architecture search produced a smaller U-Net with better organ or tumor segmentation performance on CT, MR, and ultrasound images.
  • Architectural Modifications: Architectural modifications preserve context and improve medical segmentation across electron microscopy, CT liver, and MRI prostate datasets.Examples include CNN-based input normalization and dilated convolution near the network bottleneck.
  • Data Synthesis: Image-to-image translation can remove a tumor from a diseased image and add it to a healthy image to capture detailed tumor structure for segmentation.The framework transforms between presence and absence domains before learning tumor addition.
  • Attention-based Segmentation: Attention-based architectures improve or refine segmentation of prostate, abdominal organs, and other medical targets in MRI and 3D images.Reported approaches include multi-level attention and dilated convolution blocks for detailed attention.
  • Adversarial Segmentation: Adversarial training has been applied to pancreas, retinal, and brain-tumor segmentation across CT and medical imaging datasets.Several studies use GANs or adversarial frameworks to support medical image segmentation.
  • Sequenced Models: RNNs and LSTMs model temporal dependencies in image sequences and combine with CNNs or U-Net outputs for segmentation in 4D volumes and 3D CT.Applications include brain MRI slices and pancreas segmentation.

4 Optimization Function based Improvements

Loss-function improvements target segmentation accuracy and class imbalance by changing how pixels, classes, object boundaries, and difficult examples contribute to training.

  • Cross Entropy: Pixel-wise cross entropy evaluates each pixel’s class-prediction vector against its one-hot target and averages over image pixels.For binary segmentation, predictions are obtained from the logistic or sigmoid function.
  • Class Imbalance: Equal pixel weighting can let prevalent classes dominate training when class representations are unbalanced.Weighted cross entropy assigns class-specific weights to counteract dataset imbalance.
  • Class Imbalance: Balanced cross entropy changes the weighting of positive and negative pixels to reduce false negatives or false positives.β greater than 1 decreases false negatives, whereas β less than 1 decreases false positives.
  • Boundary-aware Losses: Distance-based terms augment cross entropy to enforce separation between nearby components and improve segmentation when objects are close.The distance functions measure proximity to the borders of the nearest and second-nearest cells.

4.3 Focal Loss

Focal loss modifies cross entropy so that easy examples contribute less and difficult examples receive greater focus during CNN training.

  • Focal Loss: Focal loss adds the term (1 − p̂)^γ to cross entropy to reduce the contribution of easy examples.The formulation also includes α to weight the two classes.

4.4 Overlap Measure based Loss Functions

Overlap-based losses optimize agreement between predicted and ground-truth regions, offering alternatives to pixel-wise objectives for class-imbalanced segmentation. The review covers Dice, Jaccard, Tversky, boundary-aware, and other variants, including behavior across object sizes.

  • Dice Loss: Dice coefficient measures overlap between predicted and ground-truth segmentations and is equivalent to the F1 score.Its value ranges from 0 to 1, with 1 indicating perfect overlap.
  • Dice Loss: The Dice coefficient is computed from true positives, false positives, and false negatives as 2TP/(2TP + FP + FN).The review also states that Dice is at least as large as IoU.
  • Jaccard Loss: Jaccard or IoU measures intersection over union as TP/(TP + FP + FN) for predicted and ground-truth segmentations.The metric is expressed through the intersection and union of the two masks.
  • Tversky Loss: Tversky loss generalizes Dice loss by weighting false positives and false negatives to control their relative penalties.The weighting parameter determines how strongly each error type contributes.
  • Specialized Variants: Loss variants combine overlap objectives with exponential-logarithmic terms, convex Jaccard surrogates, boundary losses, or conservative penalties for specialized segmentation needs.These variants address small structures, multiclass Jaccard optimization, boundary alignment, or domain adaptation.
  • Motivation and Comparison: Class imbalance, particularly small foregrounds against large backgrounds, motivates overlap-based losses and modified objectives in medical image segmentation.The review visualizes loss behavior for large and small objects using author-reported parameter settings.

5 Optimization Function based Improvements Applied to Medical Images

Medical image segmentation methods modify optimization objectives to address imbalance, boundary errors, smoothness, and geometric structure. The reviewed approaches combine cross-entropy with weighting, distance-based regularization, level-set terms, or overlap-based penalties.

  • Class Imbalance: Optimizing Dice loss can outperform original cross-entropy when foreground objects are very small relative to the background.The cited discussion identifies this setting as a case where Dice optimization works better.
  • Boundary and Smoothness Regularization: Distance-based regularization encourages smooth outputs by incorporating distances between predicted and ground-truth boundaries.Distance maps represent unsigned distances to segmentation boundaries, while α controls the penalty for larger errors.
  • Loss Behavior: The comparison of cross-entropy and Dice losses reports different Dice values for one false negative on large and small foregrounds, while cross-entropy gives the same value in both cases.For the large foreground, Dice returns 0.96 versus cross-entropy 0.83; for the small object, Dice returns 0.66 versus cross-entropy 0.83.
  • Boundary and Smoothness Regularization: Convolutional and soft-thresholding terms use predicted and ground-truth segmentations to regularize errors in local neighborhoods.The formulation uses a circular convolutional kernel whose normalized elements sum to one.
  • Geometric Priors: Level-set energy minimization can be summed with multiclass cross-entropy as a regularizer for semi-supervised brain MRI segmentation.Other approaches measure voxel distances to object boundaries and use weights to penalize boundary errors.
  • Class Imbalance: Weighted and regularized cross-entropy objectives are used to address imbalance and improve segmentation behavior in medical images.The reviewed methods include weighted penalties and regularizers alongside standard cross-entropy.

6 Image Synthesis based Methods

Image synthesis methods augment limited segmentation datasets by generating synthetic images or image-mask pairs. The reviewed GAN-based approaches report sufficient variety and improved or nearly preserved segmentation performance when synthetic data are used.

  • Motivation: Deep CNNs depend on large datasets to reduce overfitting and class-imbalance problems, motivating data augmentation for limited data.The review distinguishes standard geometric and color-space augmentation from data-space synthesis methods.
  • GAN-Based Augmentation: A Wasserstein GAN with gradient penalty generated labeled image-segmentation mask pairs for chest X-ray and Cityscapes datasets.The generated images had sufficient variety, and segmentation trained only on GAN-based augmentation outperformed training with geometric transformations alone.
  • Cross-Modality Synthesis: Conditional GANs generated cardiac MR images from CT images, and synthetic data increased segmentation accuracy.Using only synthetic data caused only a marginal decrease in segmentation accuracy in the reported experiment.

7 Weakly Supervised Methods

Weakly supervised and unsupervised segmentation methods seek alternatives to expensive pixel-level annotation by using weaker labels or no annotations. The reviewed work spans supervision levels, weakly supervised architectures, and differentiable losses for weak labels.

  • Motivation and Supervision Levels: Pixel-level annotation is time-consuming and expensive, whereas unlabeled and weakly labeled images can be collected more quickly and cheaply.The supervision spectrum includes pixel-wise, image-level, bounding-box, and no-annotation settings.
  • Weakly Supervised Architectures: Weakly supervised segmentation networks use operations such as unpooling, deconvolution, and feature-map aggregation to learn scale-invariant or localization features.Reported evaluations include PASCAL VOC and chest X-ray image datasets.
  • Medical Image Segmentation: Scarcity of richly annotated medical images limits supervised solutions, motivating unsupervised and weakly supervised models for medical image analysis.The cited work introduced a differentiable loss term for weakly supervised labels that reduced training computational demand while also achieving an additional reported benefit.

8 Multi-Task Models

Multi-task segmentation models learn related visual tasks simultaneously, with applications spanning natural and medical imagery. Reviewed examples combine segmentation with detection, classification, tracking, or boundary preservation.

  • Multi-task learning trains multiple related tasks simultaneously to exploit commonalities and improve learning efficiency and task performance.
  • Cascaded multi-task loss preserves boundary information from segmentation masks for building-footprint labeling.
  • Mask R-CNN jointly predicts object masks, class labels, and bounding boxes, and has been applied extensively to segmentation.
  • Clinical neonatal images were processed with patient detection and skin segmentation, remaining robust to lighting, skin tone, and pose changes.
  • Mask R-CNN supports medical applications including cell tracking, nuclei detection, oral-disease segmentation, ulcer segmentation, and rib labeling.

9 Segmentation Evaluation Metrics and Datasets

Segmentation evaluation combines pixel-wise confusion-matrix metrics with overlap measures, while dataset characteristics shape benchmarking. Natural-image benchmarks are comparatively standardized, whereas medical segmentation spans many modalities and smaller datasets.

  • Evaluation Metrics: Binary segmentation evaluation uses TP, TN, FP, and FN pixels to calculate precision, recall, specificity, and pixel-wise accuracy.
  • Evaluation Metrics: Precision is defined as TP divided by TP + FP, while pixel-wise accuracy is defined as TP + TN divided by TP + TN + FP + FN.
  • Evaluation Metrics: Dice coefficient and Jaccard index, also called intersection over union, are popular overlap-based measures for segmentation performance.
  • Evaluation Metrics: In a 5 × 5 overlap example, precision = 0.9333, pixel-wise accuracy = 0.84, Dice coefficient = 0.7778, and Jaccard index = 0.6364.
  • Natural-Image Datasets: PASCAL VOC 2012 contains segmentation annotations for 2,913 images across 20 object classes, supporting natural-image benchmarking.
  • Natural-Image Datasets: Natural-image datasets include PASCAL Context, Cityscapes, ADE20K, and CamVid, covering diverse scenes and annotation settings.
  • Natural-Image Datasets: DeepLabV3+ improved mean IoU from 62.2% with FCN to 89.0%, an approximately 27% improvement on PASCAL VOC 2012.
  • Medical-Image Datasets: Medical segmentation uses 13 imaging modalities, and expensive modalities such as EM, PET, and MRI have smaller datasets than RGB, ultrasound, and X-ray.

10 Discussion and Future Directions

The review identifies architectural, optimization, data, and modeling challenges that motivate future research in semantic segmentation. Key boundaries include computational costs, limited and heterogeneous medical data, weak supervision noise, and incomplete spatial context.

  • Architectures: Encoder-decoder networks with skip connections are prevalent, but transferring richer features increases memory and computation and may transmit non-discriminative maps.
  • Sequenced Models: Sequenced models process temporal data and 3D medical volumes slice by slice, potentially missing geometrical information captured by volumetric convolutions.
  • Optimization Functions: Medical segmentation commonly combines cross-entropy with distance- or overlap-based losses, because overlap-only losses can hinder smooth optimization for small foregrounds.
  • Other Potential Directions: Noise, limited data, acquisition costs, and legal, ethical, and privacy constraints make universal solutions across imaging modalities difficult.
  • Other Potential Directions: Large medical images and volumes often require patches or sub-volumes, making it difficult for models to capture spatial relationships.
  • Other Potential Directions: Deep CNNs can lose local border information, while conditional random fields restore it at the cost of additional computation time.
  • Weak Supervision: Weakly supervised methods use image-level labels or few segmentation annotations, but attention maps may be noisy and multiple-instance-learning window selection is difficult.
  • Other Potential Directions: Medical models often omit multimodal patient data and metadata that could provide valuable information for segmentation.
Loading 1910.07655v4…