PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality specific U-Net variants for biomedical image segmentation: A survey

Narinder Singh Punn, Sonali Agarwal

arXiv:2107.04537v4eess.IVcs.CV

TL;DR

Biomedical image segmentation needs automated identification and localization across diverse medical modalities, while the mechanics, modality-specific challenges, and solutions of U-Net methods require systematic analysis. This paper surveys U-Net architectures and variants through inter- and intra-modality categorization, including COVID-19 applications, and synthesizes their strengths, similarities, challenges, and future directions.

  • Problem

    Biomedical image segmentation requires automated identification and localization of target regions across diverse modalities for computer-aided diagnosis, but modality-specific U-Net challenges and solutions require systematic understanding.

  • Method

    The paper describes the U-Net framework and categorizes U-Net-based biomedical segmentation approaches by modality across inter-modality and intra-modality analyses.

  • Results

    The survey finds that U-Net variants address challenges associated with different imaging modalities and highlights their application to COVID-19 diagnosis.

  • Takeaways & Limitations

    U-Net-based models provide a foundation for developing architectures that identify and localize biomedical target regions or sub-regions across modalities.

  • Takeaways & Limitations

    Local, non-public datasets limit the reusability and reachability of some reviewed segmentation approaches, while data acquisition, annotation expense, and security constrain data availability.

Abstract

from arXiv · show

With the advent of advancements in deep learning approaches, such as deep convolution neural network, residual neural network, adversarial network; U-Net architectures are most widely utilized in biomedical image segmentation to address the automation in identification and detection of the target regions or sub-regions. In recent studies, U-Net based approaches have illustrated state-of-the-art performance in different applications for the development of computer-aided diagnosis systems for early diagnosis and treatment of diseases such as brain tumor, lung cancer, alzheimer, breast cancer, etc., using various modalities. This article contributes in presenting the success of these approaches by describing the U-Net framework, followed by the comprehensive analysis of the U-Net variants by performing 1) inter-modality, and 2) intra-modality categorization to establish better insights into the associated challenges and solutions. Besides, this article also highlights the contribution of U-Net based frameworks in the ongoing pandemic, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) also known as COVID-19. Finally, the strengths and similarities of these U-Net variants are analysed along with the challenges involved in biomedical image segmentation to uncover promising future research directions in this area.

1 Introduction

Biomedical imaging has expanded across multiple modalities as deep learning supports computer-aided diagnosis, creating a need to understand and systematically compare the available U-Net-based segmentation methods.

  • Medical imaging spans MRI, X-ray, CT/CAT, ultrasound, and PET, with scans differing in acquisition procedures and use cases.
  • Deep learning-based biomedical image analysis is presented as important for developing computer-aided diagnosis systems and understanding their underlying methods.
  • Existing reviews address multi-modality fusion or standard deep learning approaches, whereas this survey focuses on modality-specific U-Net methods.
  • The survey categorizes U-Net approaches across inter-modality and intra-modality dimensions to clarify technological solutions and modality-specific challenges.
  • The review process identifies U-Net biomedical segmentation studies through Google Scholar searches and evaluates selected venues using indicators including impact factor, h-index, peer review, and indexing.
  • U-Net-based biomedical image segmentation approaches have grown exponentially in usage, motivating an exhaustive review of their architectural variants.

2 Biomedical image analysis

Biomedical image segmentation supports diagnosis by identifying disease or anatomical regions from medical images, but remains challenging because targets vary in dimension, shape, and location. U-Net and related architectures automate pixel-level segmentation through encoder-decoder processing, while training strategies and evaluation metrics shape their practical use.

  • Biomedical image analysis: Segmentation provides refined boundaries around diseased or infected regions, offering information beyond classification and localization for biomedical analysis.The paper connects segmentation with applications including COVID-19 analysis and computer-aided diagnosis.
  • Biomedical image analysis: Biomedical segmentation remains difficult because target tissues vary in dimensions, shape, and location, motivating automation of traditionally manual clinical annotation.Deep learning models are categorized as manual, semi-automatic, or fully automatic approaches.
  • U-Net: U-Net encodes and decodes images at the original resolution, propagating multi-scale encoder features through skip connections and a bottleneck layer.Its architecture combines feature extraction with localization through a symmetrical encoder-decoder arrangement.
  • U-Net: U-Net uses 3 × 3 convolution-ReLU operations, 2 × 2 max-pooling for downsampling, channel expansion across layers, and up-convolution for reconstruction.The final output contains background and foreground channels, and weighted pixel-wise cross entropy addresses touching or overlapping objects.
  • Related architectures: U-Net achieved state-of-the-art results by winning the ISBI 2015 challenge, while other architectures such as FCN, DeepLab, and Mask R-CNN address segmentation with different designs.Mask R-CNN accurately detected nuclei with bounding boxes but struggled to generate better masks, motivating integration with U-Net.
  • Implementation strategies: Segmentation architectures are trained either from scratch or through transfer learning, with transfer learning using a pretrained backbone and typically producing better results than random initialization.Transfer learning may freeze the base model or retrain selected high-level layers with prediction layers.
  • Evaluation: Biomedical segmentation systems require task-appropriate metrics because no standard metric set captures time, computational cost, memory requirements, and overall performance.Common metrics use quantities including TP, TN, FP, FN, predicted masks, ground truth, distance, and volume.

3 U-Net variants for medical imaging

The survey categorizes U-Net variants across medical-imaging modalities to clarify how architectural modifications address limitations of vanilla U-Net. It organizes variants by improvements such as attention, multi-scale fusion, and model ensembling, supporting more efficient computer-aided diagnosis.

  • Motivation: Vanilla U-Net can learn more slowly in deeper layers, motivating architectural improvements across modality-specific biomedical segmentation applications.The survey uses inter-modality and intra-modality categorization to organize these developments.
  • Better U-Nets: Better U-Nets introduce modifications including FCN or SegNet integration, transfer learning, dense or residual blocks, multi-stage training, and multi-tasking.These approaches are defined as improvements over the raw U-Net model.
  • Attention U-Nets: Attention U-Nets vary feature-map attention mechanisms using spatial, channel, mixed, or non-local attention to filter relevant features.The category focuses on selecting informative representations within U-Net-based models.
  • Inception U-Nets: Inception U-Nets use multi-scale feature-fusion strategies to learn feature representations effectively.The category is defined by its use of multi-scale processing within U-Net variants.
  • Ensemble U-Nets: Ensemble U-Nets combine multiple models or sub-models, with or without other enhancements, to improve segmentation performance.The survey treats ensembling as a distinct intra-modality category.
  • Survey scope: The survey reviews wide varieties of U-Net-based approaches across modalities to support faster and more efficient computer-aided diagnosis practices.Table 6 summarizes the variants discussed in the following sections.

3.1 X-ray

X-ray studies apply U-Net variants to segment anatomical structures and support computer-aided diagnosis, using refinement, transfer learning, residual encoders, attention, and ensemble designs.

  • Better U-Nets: U-Net lung segmentation from CXR images improved significantly over traditional approaches across multiple datasets.Post-processing used flood fill and morphological operations to refine lung masks.
  • Better U-Nets: Transfer learning with a pre-trained VGG-16 encoder supported multi-class segmentation of lungs, heart, and clavicles in chest X-rays.The decoder used sequential upsampling and standard convolution operations, with fine-tuned pre-trained weights.
  • Better U-Nets: Several X-ray variants modify U-Net with dense, inception, recurrent, residual, attention, or selective-kernel components to improve feature extraction.The reviewed table includes DEFU-Net, MDU-Net, AU-Net, SKU-Net, and related architectures.
  • Better U-Nets: 2STU-Net combines a pre-trained ResNet-34 encoder with a two-stage, multi-scale training scheme for pneumothorax segmentation.Images are prepared at 256×256 and 512×512 resolutions, with later fine-tuning initialized from earlier learned weights.
  • Ensemble U-Nets: CardioXNet uses two parallel U-Nets to segment cardiac and thoracic regions before computing the cardiothoracic ratio for cardiomegaly detection.Data augmentation and largest-area contour post-processing address limited samples and noisy output masks.

3.2 Computed tomography

CT applications use U-Net variants with residual, attention, multi-scale, context, deformable, and cascaded designs for organ, lesion, and anatomical segmentation.

  • Attention U-Nets: Attention-gated U-Net focuses on pancreas regions by suppressing irrelevant features and highlighting target-relevant features through filtered skip connections.The architecture associates each pixel with a gating vector to determine regions of focus.
  • Context and multi-scale U-Nets: U-NeXt integrates attention up-sampling, spatial-pyramid pooling in skip connections, and dense multi-scale feature extraction for CT gallstone segmentation.Its architecture is similar to U-Net++ with design variations.
  • Context and multi-scale U-Nets: MC-Net combines multi-scale context extraction with context residual attention to model local and global semantic information in CT target regions.Its design addresses limited capture of long-range dependencies in many U-Net models.
  • Other U-Nets: CT segmentation studies also employ cascaded localization-and-segmentation networks, graph-cut refinement, and Bayesian U-Nets with uncertainty modeling.These approaches target vertebrae, liver, and individual-muscle segmentation, respectively.

3.3 Magnetic resonance imaging

MRI-based U-Net variants address brain, prostate, and multi-organ segmentation through modality fusion, attention, residual connections, separable 3D convolutions, and deep supervision.

  • Better U-Nets: Multi-modality fusion combines T1, T2, T1c, and FLAIR MRI sequences to provide richer information for multi-class glioma segmentation.The modified U-Net adds residual blocks and replaces max pooling with strided convolutions, but lacks 3D volumetric analysis.
  • Better U-Nets: DAU-Net uses CycleGAN-generated synthetic MRI from CT scans, followed by attention U-Net segmentation for multi-organ prostate-cancer diagnosis.The transformation is trained to add soft-tissue information to CT-derived images.
  • Attention U-Nets: USE-Net incorporates squeeze-and-excitation blocks in skip connections to calibrate channel correlations for multi-class prostate segmentation.The approach is reported to improve generalization across imaging conditions.
  • 3D U-Nets: S3DU-Net extends 3D U-Net with spatiotemporal-separable convolutions, inception modules, and residual connections to learn spatial and temporal patterns.Its separable convolution applies 2D spatial extraction followed by 1D temporal learning.
  • 3D U-Nets: Deep supervision and group dilation produced DSC values of 0.86 on an institutional prostate dataset and 0.88 on PROMISE12.The approach addresses intensity inhomogeneity and anatomical variation but requires complex computations.

3.4 Positron emission tomography

PET segmentation studies combine U-Net architectures with 3D processing, PET-CT fusion, spatial attention, physics guidance, and surrogate-mask refinement for tumor delineation.

  • Better U-Nets: 3D U-Net was applied to dynamic 18F-FET PET/CT lesion detection and segmentation using data from 37 patients.Ground-truth masks were generated through manual delineation and binary thresholding.
  • Multi-modality U-Nets: PET-CT fusion supplies metabolic information from PET and anatomical information from CT for tumor segmentation.PET contrast helps separate soft tissue near tumor boundaries identified with CT’s higher spatial resolution.
  • Multi-modality U-Nets: A fused PET-CT V-Net framework achieved a dice value of 0.85 for lung-cancer tumor segmentation and outperformed traditional unary-modality models.Two independent sub-networks extract modality-specific features before intermediate-level fusion.
  • Physics-guided U-Nets: A physics-guided framework uses stochastic tumor synthesis, an improved dropout U-Net, and radiologist-provided surrogate masks for PET segmentation.The three modules respectively generate realistic tumors, learn segmentation features, and fine-tune the network.
  • Physics-guided U-Nets: Dice scores were 0.87 on simulated images and 0.73 on patient images for primary-tumor segmentation.The method outperformed several semi-automated approaches but avoided modeling 3D voxel correlations.

3.5 Ultrasound

Ultrasound U-Net variants address modality-specific challenges including ambiguous boundaries, variable lesion size, limited data, low signal-to-noise ratio, and small-structure segmentation. These approaches use shape-aware labels, adaptive receptive fields, attention, multi-branch processing, and feature enrichment.

  • Ultrasound applications: Ultrasound imaging supports U-Net segmentation of regions such as breast masses, pelvic-floor muscles, eyeballs, needles, vascular structures, and bone surfaces.Ultrasonography visualizes internal organs by processing reflected acoustic signals.
  • Shape-aware segmentation: SDF labels and semantic embedding help SSU-Net identify ambiguous and discontinuous eyeball boundaries.The model fuses low-level and corresponding-stage features after convolution and bilinear interpolation.
  • Attention U-Nets: Attention skip connections and total-variation regularization improve multi-needle segmentation while maintaining needle spatial continuity in 3D TRUS images.Deep supervision uses patches of needle masks to compute cross-entropy loss.
  • Adaptive convolutions: SKU-Net replaces U-Net convolutions with selective-kernel blocks that dynamically adapt receptive fields to varying breast-mass sizes and image properties.The selective-kernel design uses parallel branches including dilated convolutions.
  • Multi-branch U-Nets: DPU-Net uses parallel encoding and decoding branches with different kernel sizes to handle shape and size variation in IVUS lumen and media-adventitia regions.Real-time augmentation accompanied Jaccard measures of 0.87 at 40 MHz and 0.90 at 20 MHz.
  • Feature enrichment: A multi-feature guided CNN enriches ultrasound inputs using B-mode scans and filtered phase and shadow features before classification and segmentation.The pre-enhancing network synthesizes scans highlighting bone surfaces.

4 Other U-Net variants and imaging

The survey covers U-Net variants that combine recurrent, residual, nested, context-preserving, inception, non-local, ensemble, and transformer designs. These variants target multi-modality generalization, spatial-information preservation, global-context modeling, and automated adaptation.

  • Hybrid convolutional variants: RU-Net and R2U-Net integrate recurrent, residual, and U-Net components for biomedical image segmentation across retina, skin-cancer, and lung modalities.The models were trained and evaluated on multiple modality-specific datasets.
  • Nested U-Nets: UNet++ uses dense nested skip connections and deep supervision to narrow the gap between encoded and decoded feature maps.Its design aims to capture coarser details more effectively than standard U-Net.
  • Context-preserving U-Nets: CE-Net preserves spatial information with pretrained ResNet blocks, dense atrous convolution, residual multi-kernel pooling, and convolutional-deconvolutional decoding.Its context extractor is positioned at the bottleneck.
  • Inception U-Nets: Inception U-Net replaces standard convolutions with parallel varying-filter-size inception blocks and hybrid spectral-spatial pooling.The hybrid pooling operation uses a Hartley transform and max pooling.
  • Non-local U-Nets: NL-Unet adds self-attention-based global context aggregation to address the limited long-range dependencies of local convolutions.The module can integrate with feature extraction and reconstruction operations in U-Net models.
  • Ensemble U-Nets: nnU-Net ensembles 2D U-Net, 3D U-Net, and 3D cascade configurations with automated preprocessing, augmentation, and post-processing for different modalities.It achieved state-of-the-art segmentation results in the medical segmentation decathlon without manual intervention.
  • Transformer U-Nets: Swin-Unet is a pure transformer, U-Net-like architecture designed to capture global context that vanilla U-Net’s local convolutions cannot fully learn.Its motivation is the limitation in modeling long-range dependencies.

5 U-Net in COVID-19 diagnosis

The survey describes U-Net-based methods for COVID-19 diagnosis using chest CT and X-ray imaging. These approaches support infected-region segmentation and combine segmentation with classification to complement limitations of RT-PCR testing.

  • Diagnostic motivation: RT-PCR is described as the most reliable COVID-19 diagnostic approach but as time-consuming and less sensitive during early infection stages.
  • Imaging modalities: Deep-learning approaches use CT and X-ray modalities to assist COVID-19 diagnosis and help control the spread of infection.
  • CT-based approaches: Wu et al. proposed a U-Net-based JCS framework for joint classification and segmentation of COVID-19 from chest CT scans.The framework is described as similar to cU-Net.
  • X-ray-based approaches: X-ray screening methods include a framework combining IRRCNN transfer-learning classification with NABLA-N localization of infected regions in coronavirus images.The framework covers both X-ray and CT images.

6 Analysis

The survey positions U-Net as a prominent, modular basis for biomedical segmentation while identifying challenges in automation, comparability, dataset accessibility, modality diversity, and target-structure variation. Recent variants increasingly emphasize multi-scale fusion, global context, hybrid losses, and computational efficiency.

  • Challenges: Segmentation is critical to computer-aided diagnosis, yet remains challenging because target tissues vary in dimensions, shape, and location.The survey frames automation as desirable because manual segmentation is performed by expert clinicians.
  • U-Net design: U-Net’s modular topology integrates with architectures such as ResNet, VGGNet, DenseNet, and MobileNet across biomedical applications.The survey attributes this flexibility to U-Net’s mutability and modularity.
  • Recent trends: Recent U-Net developments emphasize multi-scale feature extraction and fusion to model global and long-range feature dependencies.
  • Recent trends: Transformer-based U-Net variants use self-attention to enhance global-context capture.
  • Training strategies: Hybrid losses commonly combine binary cross entropy with Dice or Jaccard loss to better penalize false-positive and false-negative predictions.
  • Implementation strategies: Depthwise separable convolutions are used to reduce model computations and training parameters, while multi-modality fusion targets improved feature representation.
  • Evaluation limitations: Local datasets limit reusability and reachability, while differing datasets and evaluation metrics make effective cross-study comparison difficult.The Dice similarity coefficient is the most widely used segmentation metric in the reviewed studies.
  • Future challenges: Generic architectures such as nn-UNet, U-Net++, and MR-Unet may not achieve optimal performance across all tasks because modalities and target structures differ.Noise and artifacts provide an additional source of modality-specific difficulty.

7 Scope and challenges

U-Net models face general challenges involving computation, data availability, image quality, and interpretability, alongside modality-specific difficulties in X-ray, CT, MRI, and ultrasound segmentation.

  • General challenges: Computational power requirements limit U-Net feasibility, while compression, acceleration, cloud computing, and architecture automation address efficiency but leave real-time benchmarks needed.EfficientNet-style scaling and automated architecture design are identified as possible directions, but concrete real-time benchmark results remain necessary.
  • General challenges: Limited training data constrains robust performance because acquisition, delineation, and security are costly, while augmentation and transfer learning offer partial remedies.Augmented-data diversity remains limited by the available data and may result in overfitting.
  • General challenges: Noise and artefacts reduce model performance by obscuring image features or adding irrelevant patterned features, motivating denoising and reconstruction methods.The passage identifies wavelet thresholding, PDE-based methods, NL-means, anisotropic diffusion, and metal artefact reduction as preprocessing options.
  • General challenges: Deep CNNs lack transparency because hidden-layer processing is difficult to interpret, and existing explanation methods still lack a concrete benchmark scheme.Examples include LIME, SHAP, partial dependence plots, and anchor-based explanations.
  • Modality-specific challenges: X-ray, CT, MRI, and ultrasound each present distinct segmentation obstacles involving projection overlap, anatomical variation, tumor heterogeneity, or heterogeneous organ appearance.CT segmentation must distinguish structures with similar features, MRI tumor segmentation faces variation in location, size, shape, and heterogeneity, and ultrasound appearance varies with depth, neighboring tissues, and location.
  • Modality-specific challenges: Architectural and operational designs, including inception, cascaded, attention, dense, atrous, spectral, and hybrid approaches, are integrated to address modality-dependent challenges.Multi-modality fusion is widely studied for inconsistent tumor features in MRI-based segmentation.

8 Conclusion

The survey examines U-Net variants for biomedical image segmentation across modalities and relates their developments to modality-specific challenges. It concludes that U-Net-based models have substantial potential for automated clinical imaging segmentation and computer-aided diagnosis systems.

  • Conclusion: The survey covers U-Net variants and their developments for biomedical image segmentation across different imaging modalities.Each variant is discussed in relation to challenges incurred by different modalities.
  • Conclusion: U-Net-based models have substantial potential for automated biomedical image segmentation and real-world computer-aided diagnosis systems.The conclusion connects this potential to identifying and localizing target regions or sub-regions.
Loading 2107.04537v4…