Source-linked AI summary
From CNN to Transformer: A Review of Medical Image Segmentation Models
Wenjian Yao, Jiajun Bai, Wei Liao, Yuheng Chen, Mengjuan Liu, Yao Xie
TL;DR
Medical image segmentation is important for diagnosis and treatment but remains difficult across diverse images and tasks. The paper surveys four representative models, analyzes their designs, evaluates them on two benchmark datasets, and discusses challenges and future trends. TransUNet achieves the best performance across six metrics on the Tuberculosis Chest X-rays dataset, while the paper also highlights dataset difficulty and annotation constraints.
Problem
Comprehensive reviews covering recent deep learning segmentation models and quantitative comparisons among them remain limited, although segmentation is important for medical image analysis.
Method
The paper surveys U-Net, UNet++, TransUNet, and Swin-Unet, theoretically analyzes their characteristics, and quantitatively evaluates them on two benchmark datasets.
Results
TransUNet achieves the best performance across all six metrics on the Tuberculosis Chest X-rays dataset, including 96.45% (DSC↑).
Takeaways & Limitations
The survey and shared experimental resources are intended to help researchers understand these models and develop models for new segmentation tasks.
Takeaways & Limitations
Ovarian mass segmentation is highly challenging because segmented regions vary in size, shape, location, and texture, whereas lung segmentation is easier due to distinct lung-background features.
Abstract
from arXiv · showhide
Medical image segmentation is an important step in medical image analysis, especially as a crucial prerequisite for efficient disease diagnosis and treatment. The use of deep learning for image segmentation has become a prevalent trend. The widely adopted approach currently is U-Net and its variants. Additionally, with the remarkable success of pre-trained models in natural language processing tasks, transformer-based models like TransUNet have achieved desirable performance on multiple medical image segmentation datasets. In this paper, we conduct a survey of the most representative four medical image segmentation models in recent years. We theoretically analyze the characteristics of these models and quantitatively evaluate their performance on two benchmark datasets (i.e., Tuberculosis Chest X-rays and ovarian tumors). Finally, we discuss the main challenges and future trends in medical image segmentation. Our work can assist researchers in the related field to quickly establish medical segmentation models tailored to specific regions.
1 Introduction
Medical image segmentation is challenging because medical images and tasks vary substantially, motivating deep learning approaches that improve feature extraction and adaptability. This paper reviews four representative models, compares them quantitatively, and discusses future directions.
- Motivation: Medical image segmentation identifies organ or lesion pixels and supports extracting information about tissue shape and volume.Traditional approaches use thresholding, edge detection, morphological operations, or other handcrafted designs.
- Deep Learning: Deep learning models address limitations of handcrafted methods through deep feature learning, improving accuracy and adapting to different datasets and tasks.CNN-based models have achieved strong segmentation results, including U-Net’s first-place result in the ISBI 2015 Cell Segmentation Challenge.
- Transformer-Based Models: TransUNet embeds Transformer modules within a U-shaped architecture to model long-range dependencies and integrate global image information.Self-attention computes relationships between positions in the input sequence, strengthening semantic representation for large, high-resolution images.
- Transformer-Based Models: Swin-Unet combines a U-shaped architecture with hierarchical Swin Transformer attention to improve computational efficiency while reducing computational complexity and memory consumption.Its attention operates on small patches and uses shifted windows to process local regions hierarchically.
- Research Gap: Existing reviews insufficiently cover recent deep learning segmentation models and quantitative comparisons across models.Prior literature cited in the paper focuses on traditional CNN-based models or model structure without quantitative evaluation.
- Paper Scope: The paper surveys U-Net, UNet++, TransUNet, and Swin-Unet, theoretically analyzes their characteristics, evaluates them on two benchmark datasets, and discusses challenges and future trends.The authors also share experimental source code and detailed model configuration parameters.
2 Typical Medical Image Segmentation Models
The reviewed architectures extend U-Net’s encoder-decoder design in different ways: UNet++ adds dense connections, TransUNet combines CNNs with Transformers, and Swin-Unet uses hierarchical shifted-window attention. These designs target richer multiscale representations, global context, or more efficient computation.
- 2.1 U-Net: U-Net uses contracting and expanding paths with skip connections to combine deep feature extraction and pixel-level segmentation.The expanding path upsamples features and concatenates them with corresponding cropped feature maps from the contracting path.
- 2.2 UNet++: UNet++ extends U-Net with dense connections, additional short-skip paths, and upsampling blocks that fuse same-scale encoder features.These connections extract and integrate features from different layers and scales for more discriminative representations.
- 2.3 TransUNet: TransUNet uses a hybrid CNN-Transformer encoder to address CNN limitations in modeling long-range dependencies and handling large images.CNN feature maps are divided into patches and processed by Transformer modules, while the decoder combines encoded features with high-resolution CNN maps.
- 2.3 TransUNet: TransUNet’s multi-head self-attention captures dependencies between image positions and establishes global contextual information in feature representations.This design is intended to improve long-range dependency modeling, semantic information capture, representation capacity, and generalization performance.
- 2.4 Swin-Unet: Swin-Unet is a purely Transformer-based U-shaped architecture that extracts hierarchical features using two-dimensional blocks and shifted-window attention.It applies Swin Transformer modules in encoding, bottleneck, and decoding, while replacing convolutional modules and reducing model parameters.
3 Experimental Setup
The experiments evaluate four segmentation models on tuberculosis chest X-rays and ovarian tumor data, using specified datasets, losses, training settings, and metrics.
- 3.1 Datasets: The evaluation uses a Tuberculosis Chest X-rays dataset and a physician-annotated Ovarian Tumors CT dataset.The ovarian dataset contains 2D CT images and segmentation labels; the tuberculosis dataset includes normal cases and cases with TB manifestations.
- 3.1 Datasets: The ovarian dataset contains 4050 images from 123 patients, split into 3092 training images from 98 patients and 958 test images from 25 patients.The split prevents images from the same patient appearing in both training and test sets.
- 3.2 Implementation Details: The experiments use binary cross-entropy and Dice coefficient losses for segmentation training and evaluation.Binary cross-entropy evaluates binary pixel classification, while the Dice coefficient measures overlap between predictions and ground truth.
- 3.2 Implementation Details: The experimental implementation uses Python and PyTorch, with model-specific input and patch sizes summarized in the parameter settings.The reported settings include 512x512 inputs for U-Net, UNet++, and Trans-UNet, and 224x224 inputs for Swin-UNet.
- 3.2 Implementation Details: U-Net and UNet++ use 512x512 inputs with Adam and a 1e-3 learning rate, whereas Trans-UNet and Swin-UNet use SGD with a 1e-2 learning rate.The listed patch sizes are 1 for U-Net and UNet++, 16 for Trans-UNet, and 4 for Swin-UNet.
4 Experimental Results
The study evaluates four segmentation models using multiple metrics on tuberculosis chest X-rays and ovarian tumors, while also examining qualitative outputs and very low-Dice cases. TransUNet performs best overall, although dataset difficulty affects segmentation outcomes.
- Tuberculosis Chest X-rays: 96.45% DSC, 10.75 HD, 93.25% IoU, 98.16% accuracy, 97.36% precision, and 95.72% recall make TransUNet best across all six tuberculosis metrics.All four methods exceed 91% mIoU on the lung segmentation task, and the remaining less unclear test samples exceed 82% DSC.
- Ovarian Tumors: 89.18% DSC, 22.35 HD, 82.73% IoU, 99.02% accuracy, and 92.28% recall make TransUNet best across the five reported ovarian-tumor metrics.Across the overall ranking, Swin-Unet follows TransUNet, then U-Net and UNet++; Swin-Unet achieves the best average recall.
- Ovarian Tumors: Transformer-equipped models produce fewer ovarian-tumor cases with Dice below 20%, indicating fewer completely erroneous main-region judgments.The analysis treats Dice below 20% as completely unacceptable segmentation.
- Discussion: U-Net’s encoder-decoder with skip connections preserves high-resolution information, supports multi-scale segmentation, and remains adaptable to other medical imaging models.The study reports that all evaluated models use a U-shaped architecture.
- Discussion: TransUNet combines convolutional networks with Transformers, whereas using Transformers without convolutional networks produces less ideal results in Swin-Unet.The discussion links the combination to improved localization and segmentation performance while describing Transformer-only localization as limited.
- Discussion: Ovarian-mass segmentation is more difficult than lung segmentation because target regions vary in size, shape, location, and texture.Lung boundaries are comparatively easy to identify because they differ distinctly from the background.
5 Dataset Challenges and Issues
Medical image segmentation faces major challenges in obtaining expert annotations and handling imbalanced class distributions. Data augmentation and transfer learning are presented as methods for addressing these constraints.
- Dataset Challenges: Medical image labeling is a major supervised-learning challenge because clinicians must interpret, collect, label, and annotate medical images.Good performance and stable generalization require many annotated samples, but assembling them is difficult.
- Dataset Challenges: Class imbalance can significantly affect model performance, including the natural imbalance between benign and malignant ovarian-mass cases.The paper gives an ovarian dataset example in which abnormal patients outnumber normal patients.
- Potential Solutions: Data augmentation addresses limited and imbalanced training data through affine transformations such as flipping, rotating, mirroring, and gray-value enhancement.These transformations can increase the training set and balance positive and negative samples.
- Potential Solutions: Transfer learning from successful models in the same or other domains is presented as another solution to the annotation and imbalance problems.The passage contrasts transfer learning with data enhancement as approaches to the stated dataset challenges.
6 Conclusions
The paper surveys four representative medical image segmentation models, evaluates them quantitatively on two benchmark datasets, and provides implementation details to support researchers. It also identifies large model-based segmentation, including SAM and MedSAM, as a promising future direction for medical image analysis.
- The study investigates U-Net, UNet++, TransUNet, and Swin-Unet as representative medical image segmentation models.
- It quantitatively evaluates these four models on two benchmark datasets and shares experimental source code and detailed model parameters.
- SAM introduced foundation models and prompt-based zero-shot migration to image segmentation, unlike models limited to particular image classes.
- MedSAM applies SAM to medical image segmentation and outperforms the default SAM model on medical segmentation tasks.
- Large model-based image segmentation is presented as a promising future research direction for medical image analysis.