PaperlayerView exact revision on arXiv

Source-linked AI summary

Medical Image Segmentation Review: The success of U-Net

Reza Azad, Ehsan Khodapanah Aghdam, Amelie Rauland, Yiwei Jia, Atlas Haddadi Avval, Afshin Bozorgpour, Sanaz Karimijafarbigloo, Joseph Paul Cohen, Ehsan Adeli, Dorit Merhof

arXiv:2211.14830v1eess.IVcs.CV

TL;DR

Medical image segmentation needs flexible methods that can handle diverse clinical data and task complexity, while evidence about U-Net variants is dispersed across the literature. This paper proposes a taxonomy, reviews practical design dimensions, evaluates selected variants, and provides implementations; transformer variants achieved higher Dice scores but involved a performance–efficiency trade-off. The review also identifies computational and supervised-learning limitations that constrain deployment and generalization.

  • Problem

    Medical image segmentation requires methods suited to diverse modalities and complex tasks, but the many U-Net variants need organized comparison to guide model selection.

  • Method

    The paper reviews U-Net variants through a modular taxonomy, analyzes data modalities, losses, metrics, and architecture changes, and evaluates selected methods on medical datasets.

  • Results

    Transformer variants UCTransNet and MISSFormer achieved 2% higher Dice scores than CNN extensions across all evaluated datasets, with higher complexity than U-Net.

  • Takeaways & Limitations

    The taxonomy and comparative results help researchers and practitioners weigh segmentation performance against computational and memory efficiency when choosing U-Net variants.

  • Takeaways & Limitations

    The reviewed extensions use supervised learning, which imposes mask bias and may reduce generalization to unseen objects.

Abstract

from arXiv · show

Automatic medical image segmentation is a crucial topic in the medical domain and successively a critical counterpart in the computer-aided diagnosis paradigm. U-Net is the most widespread image segmentation architecture due to its flexibility, optimized modular design, and success in all medical image modalities. Over the years, the U-Net model achieved tremendous attention from academic and industrial researchers. Several extensions of this network have been proposed to address the scale and complexity created by medical tasks. Addressing the deficiency of the naive U-Net model is the foremost step for vendors to utilize the proper U-Net variant model for their business. Having a compendium of different variants in one place makes it easier for builders to identify the relevant research. Also, for ML researchers it will help them understand the challenges of the biological tasks that challenge the model. To address this, we discuss the practical aspects of the U-Net model and suggest a taxonomy to categorize each network variant. Moreover, to measure the performance of these strategies in a clinical application, we propose fair evaluations of some unique and famous designs on well-known datasets. We provide a comprehensive implementation library with trained models for future research. In addition, for ease of future studies, we created an online list of U-Net papers with their possible official implementation. All information is gathered in https://github.com/NITR098/Awesome-U-Net repository.

1 INTRODUCTION

Medical image segmentation supports clinical visualization, measurements, and monitoring, while U-Net has become widely used because it performs effectively with limited annotated data and across imaging modalities. This review organizes more than 100 U-Net-based segmentation methods, analyzes their design and practical aspects, and provides comparative resources for future research.

  • Clinical motivation: Automated medical image segmentation accelerates data processing and provides task-specific visualizations and measurements for clinical practice.Semantic segmentation can highlight task-relevant regions such as organs and abnormal tissue.
  • U-Net background: U-Net was developed for biomedical segmentation and has been applied to CT, MRI, ultrasound, X-ray, OCT, and PET modalities.Its use spans diverse medical imaging settings.
  • U-Net background: U-Net can exploit limited annotated datasets through data augmentation while extracting detailed image features and maintaining good segmentation performance.Random elastic deformation is given as an example of augmentation.
  • Review scope: The review covers more than 100 segmentation algorithms proposed through September 2022 and groups them into six categories.The categories include skip connection, backbone, bottleneck, transformer, rich representation, and probabilistic design enhancements.
  • Review scope: The paper analyzes architecture refinements, data modalities, loss functions, evaluation metrics, and critical contributions, then compares selected methods and shares code and pretrained weights.The comparative experiments use popular datasets, and implementation resources are offered on GitHub.

2 TAXONOMY

The review taxonomy organizes U-Net extensions by the modular component or design idea they modify, while its architectural discussion traces the transition from 2D to volumetric U-Net models. The paper also highlights the information–compute trade-off of 3D data and the practical relevance of U-Net-based systems.

  • Taxonomy: The taxonomy groups U-Net modifications by underlying design idea and identifies where architectural improvements occur, such as in skip connections.Because U-Net is modular, a method may belong to multiple taxonomy divisions.
  • 2D U-Net: The 2015 U-Net architecture won the ISBI cell tracking challenge by a large margin and was designed for segmentation with insufficient data and data augmentation.The review presents this architecture as a cornerstone of biomedical image segmentation.
  • 2D U-Net: The original U-Net uses an encoder-decoder structure with contracting and expanding paths connected through a bottleneck.The encoder captures contextual information, while the decoder progressively restores feature-map resolution.
  • 3D U-Net: 3D U-Net replaces 2D operations with 3D counterparts to use volumetric context and reduce redundancy from slice-by-slice processing.The approach was applied to sparsely annotated Xenopus kidney embryo samples and compared with 2D U-Net using IoU.
  • Practical relevance: The review connects U-Net-based architectures with real-world and commercial medical image analysis, including end-to-end designs such as nnU-Net.The paper frames model selection as dependent on the application setting and available resources.

3 U-NET EXTENSIONS

The review organizes U-Net variants into systematic categories and presents the architecture as a modular foundation for medical image segmentation and clinical use.

  • 3 U-NET EXTENSIONS: U-Net’s modular design supports systematic modification through six categories of architectural enhancements.The categories include skip connections, backbones, bottlenecks, transformers, rich representations, and probabilistic design.
  • 3 U-NET EXTENSIONS: The clinical pipeline spans data preparation, architecture search, output refinement, and application of software results to assist specialists.
  • 3 U-NET EXTENSIONS: The taxonomy is intended to provide practical information for vendors and researchers selecting or studying U-Net variants.

3.1 Skip Connection Enhancements

Skip connections combine coarse semantic information with fine spatial information, and U-Net extensions redesign their number, connectivity, or feature processing to support multiscale segmentation.

  • 3.1 Skip Connection Enhancements: U-Net skip connections combine same-resolution encoder and decoder features to merge global semantic information with local spatial information.
  • 3.1 Skip Connection Enhancements: U-Net++ redesigns skip connections so features from different semantic scales can be aggregated within nested decoder structures.
  • 3.1.1 Increasing the Number of Skip Connections: UNet3+ connects each decoder level with encoder levels and preceding decoder features through full-scale skip connections.
  • 3.1.1 Increasing the Number of Skip Connections: BiO-Net adds backward skip connections that pass decoded high-level features back to the corresponding encoder level, creating a bidirectional recursive architecture.
  • 3.1.2 Processing Feature Maps within the Skip Connections: Attention U-Net uses attention gates in skip connections to suppress irrelevant regions and highlight task-relevant regions.
  • 3.1.2 Processing Feature Maps within the Skip Connections: Projective skip connections support N-dimensional inputs mapped to lower-dimensional segmentation outputs by preserving some dimensions in compressed form.

3.2 Backbone Design Enhancements

Backbone enhancements modify encoder-decoder building blocks to improve depth, volumetric context, multiscale representation, connectivity, or adaptive receptive fields, while often trading against computation.

  • 3.2 Backbone Design Enhancements: Residual backbones enable deeper U-Net architectures, and residual U-Net variants are reported to converge faster than the original U-Net.
  • 3.2 Backbone Design Enhancements: Three-dimensional convolutions capture spatial context across entire image volumes, unlike slice-wise two-dimensional processing.
  • 3.2 Backbone Design Enhancements: 3D convolutional backbones have high computational and GPU-memory costs that limit network depth, filter size, and field-of-view.
  • 3.2 Backbone Design Enhancements: V-Net uses 3D residual blocks for volumetric segmentation, while H-DenseUNet combines 2D and 3D dense backbones to reduce computational burden.
  • 3.2 Backbone Design Enhancements: MultiResUNet uses inception-like multiscale blocks built from consecutive 3 × 3 convolutions and residual connections to reduce memory requirements.
  • 3.2 Backbone Design Enhancements: Deformable convolutions adapt sampling locations to different shapes, scales, and orientations but require additional computation to learn offsets.

3.3 Bottleneck Enhancements

Bottleneck enhancements recalibrate compressed features and expand contextual coverage through attention, modality distillation, and multiscale atrous processing.

  • 3.3 Bottleneck Enhancements: The U-Net bottleneck learns a compressed representation intended to retain information needed by the decoder to restore the input.
  • 3.3.1 Attention Modules: Position-wise attention models spatial dependencies between bottleneck pixels and selectively aggregates contextual information across positions.
  • 3.3.1 Attention Modules: The position-wise attention module starts with α set to zero and learns increasing weighting during training.
  • 3.3.1 Attention Modules: Other bottleneck attention designs enhance relevant features, compress unimportant features, or distill information from complete modalities into missing-modality networks.
  • 3.3.2 Multi-scale Feature Representation: Atrous convolutions insert holes into kernels to enlarge receptive fields, and ASPP processes bottleneck features with parallel sampling rates for multiscale representation.
  • 3.3.2 Multi-scale Feature Representation: JCS combines group atrous convolutions with squeeze-and-excitation recalibration to obtain diverse receptive fields and recalibrated channel features.

3.4 Transformers

Transformer-based U-Net variants extend CNN representations with long-range context, while hybrid and standalone designs address multi-scale feature inconsistency, local-context limitations, and computational cost. The reviewed architectures apply these strategies across 2D and 3D medical segmentation.

  • Transformer motivation: CNNs capture local semantic and texture information but have limited receptive fields for modeling long-range geometrical and structural relations in medical images.TransUNet addresses this limitation by combining Transformer global-context modeling with high-resolution CNN features for localization.
  • Hybrid CNN-Transformer designs: TransUNet tokenizes abstract encoder features, applies stacked Transformers for global context, and combines them with high-resolution CNN features in the decoder.
  • Hybrid CNN-Transformer designs: UCTransNet uses channel-wise Cross-Covariance Transformers and a channel-cross attention module to reduce encoder-decoder feature inconsistency, achieving state-of-the-art Dice results on GlaS, MoNuSeg, and Synapse.
  • Hybrid CNN-Transformer designs: ScaleFormer couples CNN features with intra-scale Dual-Axis MSA and spatial-aware inter-scale Transformer interactions to model local details, long-range affinities, and multi-scale target regions.The review identifies ScaleFormer as the state-of-the-art method in 2D modality during the study.
  • Transformer limitations: Transformer adoption in dense vision tasks is constrained by quadratic computational complexity and by data scarcity that can impair positional-encoding learning.
  • Standalone Transformer designs: Standalone Transformer U-Nets include Swin-Unet, MISSFormer, and D-Former, which target global dependencies while using hierarchical or specialized blocks for spatial representation.MISSFormer emphasizes local continuity and efficient attention, whereas D-Former alternates local- and global-scope modules for 3D segmentation.

3.5 Rich Representation Enhancements

Rich-representation enhancements use multi-scale and multi-modal inputs, specialized losses, attention, polar transformations, and multi-path encoders to retain complementary information. The reviewed methods target class imbalance, structural overlap, and modality-specific feature learning.

  • Multi-scale Fusion: Multi-scale methods resize inputs into image pyramids, allowing encoders to access features at different spatial resolutions and learn scale-specific representations.U-Net variants can aggregate image-pyramid inputs or side outputs in encoder or decoder stages.
  • Multi-modality Fusion: Multi-modal fusion integrates complementary information from imaging techniques whose individual views reveal only partial organ details.Cascaded U-Net uses multiple encoders to process modalities separately, and its multi-modal architecture outperforms a network relying on one modality.
  • Multi-scale Fusion: Focal Tversky Attention U-Net combines multi-scale inputs, deep supervision, attention gates, and focal Tversky loss to address imbalance and improve precision-recall balance.The architecture reports a 3% Dice score improvement on a multi-class CT abdominal segmentation task.
  • Multi-scale Fusion: Polar Transformation M-Net localizes the optic disc, transforms the region into polar coordinates, and uses multi-scale U-Net processing with multi-label Dice-based loss for optic-disc and optic-cup segmentation.The polar representation models the radial relationship between optic cup and optic disc and can expand the cup region through interpolation.
  • Multi-modality Fusion: Dense Multi-path U-Net and HyperDenseNet improve multi-modal representation learning through modality-specific streams, cross-path information sharing, and inception-module extensions.

3.6 Probabilistic Design

Probabilistic U-Net variants extend segmentation models to represent ambiguity, improve robustness beyond the training distribution, or regularize learning with auxiliary generative objectives.

  • Probabilistic U-Net: Probabilistic U-Net learns a distribution over plausible segmentations rather than producing only the most likely hypothesis for ambiguous medical images.Different latent-space samples generate alternative segmentation maps for the same input image.
  • Probabilistic U-Net: A posterior network maps observed segmentation variants into latent-space positions so sampled variants can reproduce the ground-truth segmentation during training.The posterior network complements the prior and U-Net by learning a useful embedding of segmentation variants.
  • VAE regularization: Adding a VAE branch to a 3D U-Net combines segmentation with image reconstruction to guide and regularize the shared encoder for small training sets.The objective combines Dice, reconstruction, and KL-divergence losses as L = Ldice + 0.1 · LL2 + 0.1 · LKL.
  • Graphical-model extensions: MRF-UNet combines accurate U-Net predictions with a lightweight first-order MRF prior, using iterative mean-field inference because their product is intractable directly.The review reports improved performance on both in-distribution and out-of-distribution data without adding MRF parameters.
  • Graphical-model extensions: A Bayesian skip connection reintegrates a threshold-based ischemic-core prior at the final layer of an attention-gated 3D U-Net and achieved better Dice performance with faster convergence.The design targets small, patchy stroke-lesion segmentations with varying shapes.

3.7 Comparative Overview

U-Net extensions are organized by architectural modifications that improve feature processing, representation, context, or uncertainty modeling, but several introduce computational costs.

  • Skip Connection Enhancement: Skip-connection enhancements combine or redesign encoder–decoder feature pathways to exploit local and semantic information across different scales.These changes increase architectural complexity, parameter counts, and computational expense.
  • Backbone Design Enhancement: Backbone enhancements use residual, inception, or dense structures to alleviate vanishing-gradient and degradation problems and facilitate faster convergence.These designs replace or augment the original U-Net backbone rather than changing only its skip connections.
  • Bottleneck Enhancement: Bottleneck enhancements use attention or atrous spatial pyramid pooling to model long-range dependencies and represent compressed features at multiple sampling scales.
  • Transformer: Transformer-based U-Net models supplement U-Net features with global information extracted from tokenized images, although their large parameter counts can lengthen computation.The review identifies Transformer influence as especially prominent after 2021 following the emergence of ViT.
  • Probabilistic Design: Probabilistic extensions include conditional-variational and MRF-based designs for plausible predictions and reduced overfitting, respectively.

4 QUANTITATIVE COMPARISON

The review evaluates selected U-Net variants under a common implementation framework across datasets from microscopy, dermoscopy, and multi-organ CT, while emphasizing comparability constraints.

  • Evaluation considerations: Across-the-board comparisons remain difficult because some studies omit standard metrics or reproducibility details such as hyperparameters, execution time, and memory footprint.
  • 4.1 Implementation details: Cross-model comparison uses a 2D U-Net baseline alongside Attention U-Net, UNet++, and Residual U-Net to assess architectural modifications.The implementations use Python, PyTorch, and a single Nvidia RTX 3090 GPU.
  • 4.2 Datasets: The evaluation includes SegPC plasma-cell microscopy, ISIC 2018 skin-lesion dermoscopy, and Synapse multi-organ 3D CT datasets.These datasets span cancer screening, dermatological diagnosis, and abdominal-organ segmentation.
  • 4.2.1 SegPC: SegPC contains 775 plasma-cell images with nucleus and cytoplasm annotations, split into 70% training, 10% validation, and 20% testing after resizing to 224 × 224.The study applies the networks to cytoplasm segmentation for this dataset.
  • 4.2.3 Synapse: Synapse provides 30 abdominal CT scans containing 3,779 axial images and annotations for 13 organs.The scans contain 85–198 slices each at 512 × 512 resolution.
  • 4.3 Loss functions: The comparison uses Cross Entropy and Dice Sørensen Coefficient losses as the selected traditional objectives for medical image segmentation.

4.4 Evaluation Metrics

The review describes complementary segmentation metrics that quantify overlap, contour error, classification quality, and pixel accuracy, while noting sensitivity to class imbalance.

  • Classification metrics: Precision and recall quantify class-level segmentation accuracy from true-positive, false-positive, and false-negative fractions, while F1 combines them harmonically.Recall is also called sensitivity in the segmentation context.
  • Pixel accuracy: Accuracy measures the ratio of correctly classified pixels to the total mask on a class basis.The review warns that class imbalance in medical datasets can produce undesirable performance.
  • Overlap metrics: Intersection over Union, or Jaccard Index, measures predicted-mask and ground-truth overlap as intersection divided by union, ranging from 0 to 1.
  • Overlap metrics: Dice Coefficient measures overlap between ground-truth and predicted regions and, in binary segmentation, is equivalent to the F1 score.It emphasizes foreground accuracy and penalizes incorrect prediction labels.
  • Boundary metrics: Hausdorff Distance measures segmentation error through the longest Euclidean distance between points on the ground-truth and predicted contours.

4.5 Experimental Results

The review compares U-Net variants across challenging skin-lesion, cell, and multi-organ segmentation tasks, combining quantitative metrics with visual assessment. Transformer-based methods often produce smoother boundaries, while architectural choices affect feature discrimination and contextual modeling.

  • 4.5.1 Skin lesion: Skin-lesion segmentation shows nearly identical Dice scores among CNN- and Transformer-based methods despite substantial image artifacts.Transformer methods may produce smoother boundaries through multi-scale global contextual modeling.
  • 4.5.2 Cell segmentation: Transformer-based methods produce softer cell contours on SegPC 2021, where mIoU complements Dice for overlapping and spiky cell boundaries.The review attributes this visual difference to global long-range contextual information capturing cell shape.
  • 4.5.3 Multi organ (Synapse): On Synapse, vanilla U-Net performs well relative to its parameter count but exhibits miss-labeling, over-segmentation, and under-segmentation.The dataset is difficult because it contains many classes and multi-scale structures.
  • 4.5.3 Multi organ (Synapse): Attention U-Net achieves almost 1% higher Dice than U-Net on Synapse and produces smoother segmentation boundaries.U-Net++ and other dense or residual variants provide only minimal Dice gains over U-Net in this comparison.

4.6 Discussion

The discussion finds that U-Net extensions generally improve performance, but gains depend on task characteristics and architectural strategy. Model selection therefore requires balancing segmentation quality against parameters, GFLOPS, memory, and computational efficiency.

  • Performance discussion: U-Net extensions achieve performance gains over the original U-Net, although vanilla U-Net remains competitive for binary skin-lesion segmentation.Its multi-organ performance is less promising, with qualitative over-segmentation and under-segmentation.
  • Architectural effects: Attention-based strategies perform well for overlapped objects, while U-Net++ benefits from hierarchical skip connections that boost performance.The cited discussion links attention to a broader feature-extraction view and nested connections to improved feature reuse.
  • Efficiency analysis: CNN plug-and-play modifications increase trainable parameters and GFLOPS, making computational complexity a key assessment factor.The review compares model parameters and GFLOPS when evaluating U-Net extensions.
  • Efficiency analysis: Figure 37 combines Dice performance with computational efficiency to support choosing an appropriate clinically applicable model.The comparison is intended to expose the trade-off between segmentation performance and efficiency.

5 CHALLENGES AND OPPORTUNITIES

The review identifies efficiency, interpretability, data diversity, privacy, and training paradigms as important challenges for deploying U-Net variants clinically. Future work must address resource constraints while preserving accuracy and generalization.

  • 5.1 Memory Efficient Models: Adding parameters can improve feature reuse and representation but may require too much memory for clinical devices with limited computational capacity.Suggested directions include pruning, quantization, low-rank approximation, knowledge distillation, and neural architecture search.
  • 5.2 Balance Between Accuracy and Efficiency: Clinical deployment requires balancing accuracy with efficiency because computational restrictions can reduce network quality and prediction accuracy.Running time, memory, and computational complexity are particularly important for real-time or resource-limited applications.
  • 5.3 Interpretability: Interpretability remains important because clinicians need to understand how deep models recognize disease-related patterns and incorporate prior knowledge.Feature visualization is identified as one route toward examining the diagnostic process.
  • 5.5 Privacy and Generalization: Single-center clinical datasets limit diversity and generalizability, motivating privacy-preserving multi-center data strategies such as federated learning.Authorized multi-center datasets are described as more realistic for clinical purposes.
  • 5.7 Data Driven Decisions: All reviewed U-Net extensions use supervised learning, whose ground-truth masks impose mask bias and may reduce generalization to unseen objects.The review proposes incorporating unlabeled data to encourage richer and more data-representative features.

6 CONCLUSION

The paper organizes U-Net variants by architectural modifications and evaluates representative models across medical datasets. Transformer variants achieve higher Dice scores than CNN extensions, while the reported results expose an accuracy–efficiency trade-off for model selection.

  • 6 CONCLUSION: The taxonomy groups U-Net extensions by architectural changes involving skip connections, backbones, bottlenecks, and other feature-processing components.The review also covers uncertainty modeling for annotation uncertainty and out-of-distribution samples.
  • 6 CONCLUSION: Transformer variants UCTransNet and MISSFormer achieve 2% higher Dice scores than CNN extensions across the evaluated datasets.This is the review’s principal cross-dataset performance finding.
  • 6 CONCLUSION: MISSFormer reaches 0.81 DSC with 42M parameters on Synapse, compared with U-Net’s 0.79 DSC and 1.9M parameters.The comparison illustrates that higher performance can require substantially greater model size.
Loading 2211.14830v1…