PaperlayerView exact revision on arXiv

Source-linked AI summary

Medical Image Segmentation Using Deep Learning: A Survey

Risheng Wang, Tao Lei, Ruixia Cui, Bingtao Zhang, Hongying Meng, Asoke K. Nandi

arXiv:2009.13120v3eess.IVcs.CV

TL;DR

Medical image segmentation research using deep learning is extensive, but existing surveys do not organize the field around its technical branches and task characteristics. This paper provides a coarse-to-fine thematic survey of supervised and weakly supervised methods, covering architecture, learning strategies, datasets, and future directions. It reports representative advances including NAS-UNet's performance with only 6% of U-Net's parameters and attention-based segmentation improvements, while noting that NAS architecture-performance mechanisms remain unexplained.

  • Problem

    Existing surveys often summarize development chronologically or by technical methods without focusing on deep-learning branches and task characteristics such as few-shot and imbalance learning.

  • Method

    The survey classifies supervised-learning literature by backbones, network blocks, and loss functions, and weakly supervised literature by data augmentation, transfer learning, and interactive segmentation.

  • Results

    NAS-UNet outperforms U-Net and its variants with training time close to U-Net and only 6% of its parameters; attention-based ScleraSegNet won SSBC2019.

  • Takeaways & Limitations

    The survey's conceptual organization is intended to help researchers understand improvement strategies and identify innovations for improving medical image segmentation accuracy.

  • Takeaways & Limitations

    The reasons particular NAS architectures perform well cannot be explained, and whether their properties generalize across tasks remains to be explored.

Abstract

from arXiv · show

Deep learning has been widely used for medical image segmentation and a large number of papers has been presented recording the success of deep learning in the field. In this paper, we present a comprehensive thematic survey on medical image segmentation using deep learning techniques. This paper makes two original contributions. Firstly, compared to traditional surveys that directly divide literatures of deep learning on medical image segmentation into many groups and introduce literatures in detail for each group, we classify currently popular literatures according to a multi-level structure from coarse to fine. Secondly, this paper focuses on supervised and weakly supervised learning approaches, without including unsupervised approaches since they have been introduced in many old surveys and they are not popular currently. For supervised learning approaches, we analyze literatures in three aspects: the selection of backbone networks, the design of network blocks, and the improvement of loss functions. For weakly supervised learning approaches, we investigate literature according to data augmentation, transfer learning, and interactive segmentation, separately. Compared to existing surveys, this survey classifies the literatures very differently from before and is more convenient for readers to understand the relevant rationale and will guide them to think of appropriate improvements in medical image segmentation based on deep learning approaches.

I. INTRODUCTION

Medical image segmentation supports clearer visualization of anatomical and pathological structures and plays an important role in computer-aided diagnosis and smart medicine. This survey organizes deep-learning research by conceptual branches and focuses on task-relevant supervised and weakly supervised approaches.

  • Motivation: Medical image segmentation clarifies anatomical or pathological structures and supports computer-aided diagnosis and smart medicine.The paper identifies liver, brain, optic disc, cell, lung, and cardiac segmentation among common tasks.
  • Segmentation Tasks: Semantic segmentation assigns a category to each pixel, whereas instance segmentation additionally distinguishes separate instances within categories.
  • Learning Settings: Supervised learning uses carefully labeled data, while weakly supervised learning requires only a small part of the data to be labeled.The paper contrasts these settings with unsupervised learning, which does not require labeled data but is more difficult to learn.
  • Research Gap: Earlier surveys often summarized medical image segmentation chronologically or by technical development without emphasizing deep-learning branches and task characteristics such as few-shot and imbalance learning.
  • Survey Organization: This survey classifies deep-learning methods from coarse to fine across supervised and weakly supervised learning, following conceptual developments.It also discusses neural architecture search, graph convolutional networks, multi-modality data fusion, and medical transformers.

2. On supervised learning approaches we analyze literature

The survey structures supervised and weakly supervised medical-image-segmentation literature around distinct improvement strategies and supplements the review with datasets and future directions.

  • Supervised Learning: Supervised-learning literature is analyzed through backbone selection, network-block design, and loss-function improvement.
  • Weakly Supervised Learning: Weakly supervised-learning literature is organized around data augmentation, transfer learning, and interactive segmentation for few-shot or class-imbalanced data.
  • Datasets and Future Directions: The survey collects common public medical image segmentation datasets and discusses future research trends and directions.
  • Paper Organization: The paper organizes the remaining discussion into supervised-learning development and unsupervised or weakly supervised strategies for few-shot data.

II. SUPERVISED LEARNING

Because medical image segmentation usually requires high accuracy, supervised learning is the most popular approach discussed in this section. The review focuses on improving neural-network architectures.

  • Supervised Learning: Supervised learning is the most popular method for medical image segmentation because these tasks usually require high accuracy.
  • Architecture Improvements: The section reviews architecture improvements involving network backbones, network blocks, and loss-function design.

A. Backbone Networks

Backbone research adapts encoder-decoder networks to medical image characteristics, including blurred boundaries, 3D volumes, and limited computational resources. Surveyed designs include U-Net variants, recurrent networks, cascade models, GAN-based methods, and anatomy-informed networks.

  • Encoder-decoder architectures: Encoder-decoder architectures extract features with an encoder and restore image resolution with a decoder for final segmentation.FCN, U-Net, and Deeplab are representative end-to-end architectures.
  • U-Net: U-Net fuses low- and high-level features through skip connections, becoming a benchmark for medical image segmentation.Its symmetric structure combines low-resolution semantic information with high-resolution detail.
  • 3D networks: 3D U-Net processes volumetric medical data directly, while V-Net uses residual connections and four down-samplings to build a deeper network.3D architectures improve spatial modeling but face computational and GPU-memory constraints.
  • Recurrent networks: RNN-based segmentation accumulates features through recursive residual convolutional layers and models temporal or contextual relationships.Its use is uncommon when medical image quality is insufficient for complete temporal information.
  • Cascade networks: Cascade networks train multiple models for coarse-to-fine, detection-to-segmentation, or mixed segmentation workflows.Hybrid 2D-and-3D cascades can improve segmentation accuracy while reducing learning burdens.
  • Other backbone strategies: GANs and anatomically constrained networks address blurred medical-image boundaries and incorporate prior knowledge about organ shape and position.Anatomical constraints guide predictions when image data are insufficiently informative or consistent.

1) Dense Connection:

Dense connections link each layer to preceding layers or connect multiple U-Net layers, enriching feature reuse while increasing parameters and potentially reducing robustness.

  • 2) Dense Connection:: Dense connections provide each layer with outputs from all preceding layers, enabling richer image-feature extraction.An improved U-Net replaces each subblock with dense connections.
  • 2) Dense Connection:: Dense connections increase parameter counts and can reduce feature-representation robustness, motivating pruning during optimization.U-Net++ integrates pruning to reduce parameters.
  • 2) Dense Connection:: U-Net++ connects layers across depths and aggregates different semantic scales in the decoder for flexible feature fusion.The structure can learn the importance of features at different layers automatically.

2) Inception:

The supplied passage introduces inception as a structure intended to address limitations of deep CNNs, while the accompanying figures identify dense-connection and U-Net++ architectures.

  • 2) Inception:: Deep CNNs can provide better performance than shallow networks but face vanishing gradients, convergence difficulty, and high memory usage.These issues motivate alternative network structures.
  • 2) Inception:: The inception structure is presented as overcoming several problems associated with deep CNNs.The supplied passage does not specify its internal operations or quantitative effects.
  • 2) Inception:: The associated figures depict dense connection architecture and U-Net++ architecture as network designs discussed in the survey.Their captions identify the architectures but do not report comparative outcomes.

4) Attention Mechanism:

Attention mechanisms improve medical image segmentation by weighting spatial positions, channels, or both, while non-local attention addresses the limits of local convolution by aggregating global information.

  • Attention Mechanism: Attention blocks selectively weight input variables so networks focus on important image regions or features.Spatial attention targets pixel positions, whereas channel attention emphasizes useful feature channels and suppresses less useful ones.
  • Local Spatial Attention: Attention U-Net modifies encoder outputs before encoder-decoder fusion using a gating signal that controls pixel importance at different spatial positions.
  • Channel Attention: Channel attention recalibrates features using learned global information; SE-Net performs squeezing, excitation, and sigmoid-based channel weighting.The resulting weights rescale the original input features.
  • Mixture Attention: Mixture attention combines spatial and channel attention because each mechanism overlooks information represented by the other.FocusNet uses an SE block for channel attention alongside a spatial-attention branch for medical image segmentation.
  • Attention Mechanism: Channel-centric attention was most effective among tested attention configurations in ScleraSegNet, which won the SSBC2019 sclera segmentation competition.
  • Non-local Attention: Non-local U-Net uses self-attention and global aggregation during up-sampling and down-sampling to overcome local convolution's missing global information.The non-local block can be embedded in different convolutional neural networks.

5) Multi-scale Information Fusion:

Multi-scale fusion methods enlarge contextual coverage while attempting to preserve local detail and relationships across object scales. The survey also describes loss-function designs that address class imbalance and boundary or overlap behavior.

  • Multi-scale Information Fusion: Medical images contain objects spanning widely different scales, while standard convolution and pooling use single kernels and limited receptive fields.The receptive field determines available context information.
  • Pyramid Pooling: Spatial pyramid pooling extracts multi-scale features by aggregating local features from fine to coarse spatial partitions.Residual multi-kernel pooling extends this idea with four pooling kernels for global context, but up-sampling can lose detail.
  • Atrous Spatial Pyramid Pooling: Atrous convolution enlarges the receptive field without increasing parameters, and ASPP combines atrous convolution with spatial pyramid pooling.The approach is intended to improve segmentation while reducing pooling-related detail loss.
  • Atrous Spatial Pyramid Pooling: ASPP can lose local information through gridding and capture irrelevant information across large distances.Hybrid expansion convolution uses variable dilation sequences to access a wider pixel range and suppress the gridding effect.
  • Non-local and ASPP: A combined ASPP and Non-local block uses atrous convolution for richer semantic context and pixel associations to improve detail handling.The survey presents this combination for human body-part segmentation.
  • Loss Functions: Cross-entropy loss treats pixels equally, so class imbalance can make it perform poorly for small target segmentation.Weighted and balanced variants adjust positive and negative class contributions.
  • Loss Functions: Generalized Dice Loss assigns class weights so different areas contribute similarly, making training more stable and robust than Dice loss.Dice loss is suitable for uneven samples but can produce unstable gradients and training difficulty when prediction and target values are small.

B. Transfer Learning

Transfer learning addresses limited medical annotation by reusing pretrained parameters or adapting models across domains. The survey also covers interactive and semi-supervised strategies, while noting important scope and performance limitations.

  • B. Transfer Learning: Transfer learning initializes medical segmentation models with trained parameters, enabling faster training when labeled data are limited.Common approaches fine-tune ImageNet-pretrained models or transfer training across domains.
  • Pre-trained Model: ImageNet-pretrained encoders within U-Net-like networks can improve medical-image segmentation accuracy after fine-tuning.The survey reports this pattern for shoulder muscle MRI segmentation and notes that pretrained models may learn features shared by natural and medical images.
  • Pre-trained Model: Pretrained-model benefits are unstable across datasets and tasks, and 2D natural-image pretraining is poorly suited to many 3D medical applications.The survey recommends using pretraining empirically when it improves segmentation accuracy.
  • Domain Adaptation: When target-domain labels are unavailable, domain adaptation transfers a classifier trained on a labeled source domain to the unlabeled target domain.CycleGAN uses two generators and two discriminators for bidirectional image translation with reconstruction and discrimination constraints.
  • Domain Adaptation: 15% improvement was obtained for myocardial segmentation by training with synthetic MR and real MR images after CycleGAN-based CT-to-MR conversion.The comparison is against a model trained on real MR images.
  • Interactive Segmentation: Interactive segmentation lets clinicians correct model-generated results using clicks, lines, bounding boxes, or feature-map adjustments.The survey describes DeepIGeoS, BIFSeg, and GM interacting as examples, and reports reduced user interactions and time.
  • D. Others Works: Weakly supervised methods use partial labels or small annotated sets, but their performance is rarely acceptable in practice, especially for 3D medical images.The survey identifies this as a future research direction.

A. Network Architecture Search

The survey presents network architecture search as a response to the time and expertise required for manual design, while highlighting performance gains and unresolved interpretability and generalization questions.

  • Manual architecture design is time-consuming and demands substantial domain knowledge, motivating neural architecture search (NAS).NAS searches network structures through search spaces, strategies, and performance estimation.
  • NAS-UNet outperforms U-Net and its variants with training time close to U-Net while using only 6% of the parameters.Its searched DownSC and UpSC cells are built within a U-shaped backbone.
  • Compressed neural networks and NAS target real-time segmentation of high-resolution 2D medical images by reducing model parameters while retaining segmentation performance.The survey identifies this direction as popular for CT, MRI, and histopathology images.
  • NAS performance can be strong, but the reasons particular architectures work well remain unexplained and their generalization across tasks requires investigation.
  • Graph convolutional approaches model medical data as non-Euclidean graphs and can provide efficient feature encoding and promising segmentation results.Graph U-Net performs better than popular U-Nets with a small parameter increase, while its performance does not improve beyond depth 4.
  • Interpretability methods including shape streams, attention maps, and guided back propagation expose features or regions associated with segmentation predictions.SAU-Net uses a secondary shape stream and built-in saliency maps to support shape-feature interpretation.

D. Multi-modality Data Fusion

Multi-modality fusion combines information from different imaging modalities for medical segmentation, while the survey also situates this topic within datasets, evaluation metrics, and broader clinical challenges.

  • Multi-modality data fusion provides richer object features that can improve medical image detection and segmentation.The survey describes a scheme for segmenting anatomical structures from unpaired CT and MRI images.
  • CNNs can encode multi-modality data for a single system that executes segmentation across modalities without extra task-specific training.
  • Fusion strategies are classified into input-level, layer-level, and decision-level fusion, although multi-modality networks introduce new design problems.
  • Public medical segmentation datasets and competitions support the development and evaluation of computer-aided diagnosis techniques.The survey provides a dataset list and example benchmark images.
  • Large datasets remain a challenge for constructing holistic anatomical models that capture the complexity of human anatomy.
  • Segmentation performance is evaluated through pixel quality, region quality, and surface-distance quality metrics.Pixel accuracy measures correctly classified pixels, while Dice measures overlap between predicted and ground-truth maps.

1) Design of Network Architecture:

The survey reviews network-architecture designs for medical segmentation, including multi-scale modules, attention, NAS, graph convolutions, and transformers, alongside task-specific loss and transfer-learning considerations.

  • Design of Network Architecture:: U-shaped encoder-decoder networks with skip connections are widely used as a basic framework for medical image segmentation.The survey identifies network-structure innovation as a popular and transferable improvement direction.
  • Design of Network Architecture:: Spatial pyramid and atrous-convolution modules enlarge receptive fields and support multi-scale feature extraction.
  • Design of Network Architecture:: Spatial and channel attention modules make feature extraction more targeted, motivating task-specific network modules.
  • Design of Network Architecture:: The survey recommends combining a manually designed backbone with NAS-based searches for smaller modules because large-network search is limited by memory and GPU constraints.
  • Design of Network Architecture:: Task-specific loss functions can focus learning on small targets, but their design depends strongly on the segmentation task.Domain knowledge, regularization terms, and automated loss search are presented as related directions.
  • Design of Network Architecture:: Transfer learning is presented as a weakly supervised strategy that shares learned parameters or knowledge to accelerate learning with limited labels.Medical imaging noise and costly annotation motivate pretrained models and transfer-learning research.
  • Design of Network Architecture:: Graph convolution extends convolutional modeling to non-Euclidean medical data by representing objects as nodes and their relationships as edges.This approach has been applied to brain, vessel, prostate, and coronary-artery segmentation.
  • Design of Network Architecture:: Transformer architectures address limitations of convolutional operations in learning global and long-range semantic interactions.CNNs remain comparatively advantageous for extracting low-level image features, while transformers model relationships among visual elements.
Loading 2009.13120v3…