PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Review of U-Net and Its Variants: Advances and Applications in Medical Image Segmentation

Wang Jiangtao, Nur Intan Raihana Ruhaiyem, Fu Panpan

arXiv:2502.06895v1eess.IVcs.CV

TL;DR

Medical image segmentation must handle low contrast, variable lesion shapes, costly annotation, limited model generalization, and weak robustness. This review classifies medical imaging datasets, examines U-Net variants through structural modifications, and synthesizes four core enhancement mechanisms. It concludes that U-Net-based models provide a systematic foundation for improving medical image segmentation, while important dataset, generalization, robustness, and applicability limitations remain.

  • Problem

    Medical image segmentation faces low contrast, variable lesion shapes, costly expert annotation, limited cross-dataset generalization, and weak robustness on blurred or shadowed images.

  • Method

    The paper classifies seven medical imaging dataset types and reviews U-Net variants through structural modifications, including connections, 3D U-Net, transformers, and domain-adaptation designs.

  • Results

    The review synthesizes U-Net and its variants across medical imaging modalities, datasets, application contexts, architectural compositions, and performance enhancements.

  • Takeaways & Limitations

    The review provides a systematic reference for designing more efficient and stable medical image segmentation networks based on U-Net.

  • Takeaways & Limitations

    Medical datasets often lack sufficient samples for ViT pretraining, require costly expert annotations, and still constrain model transfer and generalization across new domains.

Abstract

from arXiv · show

Medical images often exhibit low and blurred contrast between lesions and surrounding tissues, with considerable variation in lesion edges and shapes even within the same disease, leading to significant challenges in segmentation. Therefore, precise segmentation of lesions has become an essential prerequisite for patient condition assessment and formulation of treatment plans. Significant achievements have been made in research related to the U-Net model in recent years. It improves segmentation performance and is extensively applied in the semantic segmentation of medical images to offer technical support for consistent quantitative lesion analysis methods. First, this paper classifies medical image datasets on the basis of their imaging modalities and then examines U-Net and its various improvement models from the perspective of structural modifications. The research objectives, innovative designs, and limitations of each approach are discussed in detail. Second, we summarize the four central improvement mechanisms of the U-Net and U-Net variant algorithms: the jump-connection mechanism, residual-connection mechanism, 3D-UNet, and transformer mechanism. Finally, we examine the relationships among the four core enhancement mechanisms and commonly utilized medical datasets and propose potential avenues and strategies for future advancements. This paper provides a systematic summary and reference for researchers in related fields, and we look forward to designing more efficient and stable medical image segmentation network models based on the U-Net network.

1. Introduction

Medical imaging spans diverse modalities and datasets, creating distinct segmentation conditions for deep networks. U-Net addresses these conditions through an encoder–decoder architecture and has become widely applied in medical image segmentation.

  • Preceding framework: FCN introduced pixel-level semantic segmentation and became an important framework for constructing subsequent semantic segmentation models.The review presents FCN as a pioneering fully convolutional approach preceding U-Net.
  • Dataset diversity: Medical imaging datasets differ across modalities, producing distinct characteristics and varying performance across deep network models.The review covers modalities including MRI, PET, and OCT and presents applicable datasets and network models.
  • U-Net in medical imaging: U-Net and its variants are extensively applied in medical image segmentation and are described as predominant methodologies in the field.The review identifies U-Net-based methods as widely used across medical imaging datasets.
  • U-Net architecture: U-Net uses an encoder to extract features through convolution and pooling, a decoder to restore feature maps through upsampling, and 1x1 convolution for segmentation mapping.The architecture follows a U-shaped encoder–decoder design with corresponding feature-processing stages.
  • U-Net architecture: U-Net improves on the FCN framework by introducing a U-shaped network architecture for medical image segmentation.The review attributes the U-Net proposal to Ronneberger et al. and references its network structure in Figure 1.

2. Diversified forms of U-Net and its variants

U-Net variants diversify the architecture by changing encoder organization, combining multiple U-Nets, processing volumetric data, and incorporating domain-adaptation or transformer-based designs. These modifications target richer feature extraction, spatial context, multimodal information use, and more reliable segmentation across medical-imaging settings.

  • 2. Diversified forms of U-Net and its variants: U-Net variants modify encoder organization, combine multiple U-Nets, or extend the architecture to 3D data for more complex segmentation tasks.The review groups representative variants into encoder optimization, multiple U-Net networks, and 3D U-Net models.
  • 2.1 Improvement in the number of encoders: Y-Net addresses inverse-model mapping for complex targets, including reconstruction of initial photoacoustic pressure distributions or beamformed images.
  • 2.1 Improvement in the number of encoders: Ψ-Net uses three parallel encoders and one decoder to process a target slice with adjacent slices for intracranial hemorrhage segmentation.Self-attention is used in encoding layers, while contextual attention is used in decoding layers.
  • 2.1 Improvement in the number of encoders: Multipath dense U-Net uses four encoding-path modules to extract and integrate information from diverse brain-imaging modalities.The multimodal inputs are described as enhancing capture of critical information and segmentation performance.
  • 2.2 Multiple U-Net networks: Multiple-U-Net designs obtain additional features and multiscale information by passing one network’s output to another or assigning distinct roles to parallel branches.Examples include unsupervised W-Net and Triple U-Net for pathological cancer-nuclei segmentation.
  • 2.3 3D U-Net Networks: 3D U-Net applies 3D convolutions within a U-shaped encoder–decoder, using skip connections to transfer multiscale information across volumetric data.V-Net further combines 3D convolutions, 3D pooling, and convolutional residual units for 3D medical-image segmentation.
  • 2.4 Semi-supervised and unspuervised Domain Adaptation: Transformer-based and domain-adaptation approaches extend U-Net-related segmentation research toward unpaired translation and cross-domain medical-image segmentation.The reviewed DDSP framework uses inter-channel feature alignment and semantic-consistency loss, while UVCGAN combines a U-Net generator with a Vision Transformer bottleneck.
  • 2.4 Semi-supervised and unspuervised Domain Adaptation: Manual parameter tuning and limited computational-efficiency analysis constrain the scalability and clinical applicability of the reviewed domain-adaptation frameworks.The review recommends automated parameter optimization and validation across additional imaging modalities and datasets.

3. Improvement methods and strategies

The reviewed U-Net improvements modify training, connections, feature extraction, attention, and encoder-decoder design to improve segmentation performance, efficiency, accuracy, and generalizability. The section also covers residual, dense, transformer, hybrid, and computationally efficient architectures, including their reported strengths and limitations.

  • General improvement strategies: U-Net enhancements span data augmentation, convolution, sampling, optimization, and skip-connection modifications to improve performance, efficiency, accuracy, and generalizability.Data augmentation expands training diversity through transformations such as rotation, translation, scaling, and flipping.
  • Residual and dense mechanisms: Residual connections, dense layers, attention mechanisms, and their combinations are reported to increase U-Net segmentation performance.Residual U-Net variants address degradation and computational concerns, while MultiRes blocks combine features from 3×3, 5×5, and 7×7 convolutions.
  • Jump-connection mechanism: Jump-connection variants enrich U-Net feature fusion through structures such as U-Net++, MultiResU-Net, and UNet3+ with extensive deep supervision.The supplied passages identify complete skip connections and multi-scale feature integration as central design choices.
  • Transformer mechanism: Transformer-based models use global-context modeling alongside convolutional features, including TransUNet, TransFuse, and Swin UNet.TransFuse uses parallel DeiT-S and ResNet encoder branches, while Swin UNet applies double transformer blocks across the encoder, decoder, and bottleneck.
  • Hybrid CNN-Transformer designs: BRAU-Net++ combines dynamic sparse attention with skip-connection channel-space attention to learn local and global semantics while reducing computational complexity.Its SCCSA module integrates multi-scale features and compensates for spatial-information loss.
  • Reported results and limitations: BRAU-Net++ reports state-of-the-art performance across nearly all evaluation metrics on Synapse, ISIC-2018, and CVC-ClinicDB, while DCSAU-Net identifies 3D segmentation as future work.DCSAU-Net also uses ablation studies to examine the contributions of its components; EMCAD reports 79.4% fewer parameters and 80.3% fewer FLOPs but remains focused on 2D segmentation.

4 Evaluation Metrics.

The review describes metrics for assessing segmentation overlap, contour accuracy, classification correctness, and sensitivity to positive samples. Metric choice should match the application’s evaluation priority.

  • Metric selection: Boundary-focused applications may prefer Hausdorff or average surface distance, while overall consistency may favor Dice or Jaccard.
  • Overlap metrics: Dice similarity measures image similarity using the ratio of intersection and union, with values from 0 to 1.A value of 1 indicates completely identical images.
  • Overlap metrics: The Jaccard index measures the predicted segmentation area that matches the ground-truth segmentation area using a related calculation.
  • Overlap metrics: MIoU evaluates segmentation performance by averaging intersection-over-union values across all categories.IoU compares overlapping and combined regions, using real and predicted image areas.
  • Boundary metrics: Hausdorff distance measures how closely predicted object contours coincide with ground-truth contours.
  • Confusion-matrix metrics: Accuracy reports correctly predicted samples as a proportion of all samples, whereas recall measures correctly predicted positive samples among actual positives.Recall is essential when false negatives are critical in medical diagnosis.

5. Challenges and Solutions

The review identifies data scarcity, class imbalance, limited global or boundary modeling, computational cost, and weak cross-dataset generalization as major challenges. It discusses training strategies and architectural changes that address feature extraction, receptive fields, spatial information, and efficiency.

  • Data and annotation challenges: Expert annotation is time-consuming, expensive, and susceptible to subjective errors, while ViT-based models require more data than many medical datasets provide.Data augmentation is commonly used to address insufficient sample sizes.
  • Data and annotation challenges: Small regions of interest create extreme positive-negative imbalance, complicating stable segmentation across medical datasets.
  • Training strategies: Regularization, dropout, Dice loss, focal loss, learning-rate scheduling, and early stopping are presented as strategies for generalization, imbalance handling, difficult samples, convergence, and overfitting.
  • Remaining limitations: Limited annotation data remains a fundamental drawback that the proposed discussion does not completely eliminate, potentially restricting deep-network development.
  • Architectural challenges: Convolutional architectures may capture global features poorly, whereas transformer-based models can have weak fine-grained boundary localization.The review identifies these limitations in models including H-DenseUNet, UNet++, TransUNet, CoTr, TransFuse, CS-net, and Swin-Unet.
  • Architectural challenges: Complex CNN, transformer, and multi-branch architectures can require substantial computation because high-resolution medical images demand many convolutions and parameters.
  • Architectural challenges: Dataset-specific neighborhood differences can prevent models trained on one dataset from generalizing effectively to others.
  • Architectural solutions: The review groups architectural improvements into enhanced feature extraction, expanded receptive fields, enhanced spatial information extraction, and reduced computational complexity.Examples include residual or dense connections, ASPP, transformer attention, multiscale fusion, depthwise separable connections, pruning, and parameter quantization.

6. Future Directions.

The review frames future work around improving U-Net-based medical segmentation through four mechanisms: jump connections, residual connections, 3D U-Nets, and transformers. It emphasizes balancing their strengths, weaknesses, and implementation challenges.

  • Research directions: Future research should continue examining jump connections, residual connections, 3D U-Nets, and transformers as central U-Net improvement mechanisms.
  • Research directions: The strengths, weaknesses, and challenges of these four enhancement methods are compared in Table 7.

3D U-Net

The reviewed U-Net variants address medical image segmentation challenges through architectural changes, including residual connections, improved convolutions, 3D processing, and enhanced feature utilization. These approaches also target fuzzy boundaries, information loss, robustness, annotation cost, interpretability, and computational limitations.

  • The reviewed methods target fuzzy noise boundaries, feature distortion, poor generalization, and large parameter counts through architectural and feature-representation improvements.
  • Residual connections, recursive residual units, and improved convolutional layers are among the architectural mechanisms used to refine U-Net models.
  • Skip connections and shared network parameters mitigate information loss and increase pixelwise classification accuracy.
  • Feature visualization and multimodal inputs can support clinical interpretation and improve integration of critical information for segmentation.
  • Despite broad applicability across segmentation tasks, the reviewed models still require optimization of boundary processing, computational efficiency, dataset adaptation, and clinical integration.

7 Conclusion

The conclusion identifies medical image segmentation as a central challenge and reviews U-Net variants across imaging modalities, datasets, applications, architectures, and performance improvements. It presents U-Net as a promising framework whose continued development may improve medical image analysis.

  • Accurate medical image segmentation supports 3D visualization, surgical simulation, image-guided surgery, diagnosis, and treatment planning.
  • The review evaluates U-Net and four major variant mechanisms across medical imaging modalities and datasets.
  • The paper analyzes U-Net-based application contexts, architectural compositions, and performance enhancements to clarify development directions and improvement strategies.
  • U-Net-based network designs retain innovative potential for improving medical dataset analysis and processing despite ongoing image-analysis challenges.
Loading 2502.06895v1…