Source-linked AI summary
HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation
Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah Aghdam, Julien Cohen-Adad, Dorit Merhof
TL;DR
Medical image segmentation needs both local detail and global context, but CNNs and transformers each omit one of these capabilities. HiFormer combines hierarchical CNN and Swin Transformer representations through Double-Level Fusion, and experiments report superior performance across diverse datasets with low model complexity. Its design selects shallow and deep features for fusion while excluding middle-level features to reduce computational cost.
Problem
CNNs have limited long-range modeling, whereas transformers can miss low-level features needed for medical image segmentation.
Method
HiFormer combines hierarchical CNN and Swin Transformer encoders with a Double-Level Fusion module that fuses shallow and deep multi-scale features.
Results
HiFormer surpasses CNN-based, transformer-based, and hybrid methods across five challenging datasets in most cases, with reported advantages in quantitative and qualitative evaluations.
Takeaways & Limitations
The reported results indicate that combining CNN local representations with transformer global features preserves low-level details while modeling long-range interactions.
Takeaways & Limitations
The DLF fusion excludes middle-level features to reduce computational cost because their effect on model accuracy was imperceptible.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) have been the consensus for medical image segmentation tasks. However, they suffer from the limitation in modeling long-range dependencies and spatial correlations due to the nature of convolution operation. Although transformers were first developed to address this issue, they fail to capture low-level features. In contrast, it is demonstrated that both local and global features are crucial for dense prediction, such as segmenting in challenging contexts. In this paper, we propose HiFormer, a novel method that efficiently bridges a CNN and a transformer for medical image segmentation. Specifically, we design two multi-scale feature representations using the seminal Swin Transformer module and a CNN-based encoder. To secure a fine fusion of global and local features obtained from the two aforementioned representations, we propose a Double-Level Fusion (DLF) module in the skip connection of the encoder-decoder structure. Extensive experiments on various medical image segmentation datasets demonstrate the effectiveness of HiFormer over other CNN-based, transformer-based, and hybrid methods in terms of computational complexity, and quantitative and qualitative results. Our code is publicly available at: https://github.com/amirhossein-kz/HiFormer
1. Introduction
Medical image segmentation requires precise, robust delineation to support anatomical analysis and clinical decision-making. HiFormer addresses limitations in CNNs, transformers, and existing hybrids by combining local and global representations with multi-scale fusion.
- Medical image segmentation supports injury depiction, disease monitoring, and treatment assessment, making precise and robust results increasingly vital.
- CNNs succeed in segmentation through encoder-decoder structures and skip connections, but their convolution-based modeling limits long-range dependencies and spatial correlations.
- Transformers capture global contexts and long-range dependencies, yet self-attention can omit low-level features important for dense prediction.
- Existing hybrid CNN-transformer methods combine locality and long-range dependencies but do not effectively combine low-level and high-level features.
- HiFormer uses hierarchical CNN and Swin Transformer modules with a Double-Level Fusion module to merge local and global multi-scale representations.
2. Related Works
Related work develops CNN, transformer, and hybrid architectures for segmentation, while progressively emphasizing global context, multi-scale representations, and efficiency. These approaches motivate combining complementary local and global modeling capabilities.
- CNN-based FCNs and U-shaped networks improve segmentation by fusing features from different layers, but CNNs have limited receptive fields for long-range dependencies.
- Vision Transformers use self-attention to learn global information, while later variants reduce their dependence on large datasets and improve efficiency.
- Pyramid and hierarchical vision transformers extract information across scales, including Swin Transformer’s shifted-window approach for efficient local attention.
- Medical segmentation methods include pure transformer models such as Swin-UNet and DS-TransUNet, alongside hybrids such as TransUNet that combine low- and high-level features.
3. Method
HiFormer uses hierarchical CNN and Swin Transformer encoders to combine local, global, and multi-scale representations. Its DLF module fuses shallow and deep features before decoding them into a segmentation map.
- Encoder: The encoder integrates three-level CNN and Swin Transformer hierarchies, transferring CNN local features into corresponding transformer levels through skip connections.
- Swin Transformer Module: Swin Transformer uses windowed and shifted-window attention to provide linear local attention while enabling cross-window connections.
- Double-Level Fusion Module (DLF): DLF uses class tokens, self-attention, and cross-attention to exchange information between the two levels and back-project recalibrated features.
- Double-Level Fusion Module (DLF): The smallest and largest feature maps, P s and P l, are selected for fusion because shallow features provide localization while deeper features provide semantics.
- Decoder: The decoder upsamples and convolves the fused P s and P l features into a unified H × W feature map, then produces the final segmentation map.
4. Experiments
HiFormer is evaluated across multiple medical image segmentation datasets against CNN, transformer, and hybrid methods using task-specific quantitative and qualitative comparisons. The reported results show strong segmentation performance with relatively low model complexity.
- The experiments cover Synapse multi-organ CT segmentation and skin-lesion datasets, using task-specific metrics including Dice, Hausdorff Distance, sensitivity, specificity, accuracy, and mIOU.
- Synapse Multi-Organ Segmentation: 2.91% and 1.26% increases in Dice score, with 16.99 and 6.85 decreases in average HD, are reported for HiFormer-B versus TransUnet and Swin-Unet, respectively.
- Skin Lesion Segmentation: HiFormer reports better performance across ISIC 2017, ISIC 2018, and PH2, with finer contours, stronger boundary performance, and greater robustness to noisy items.
- Multiple Mylomia Segmentation: HiFormer consistently outperformed the challenge leaderboard across tested configurations using mean IoU and modeled multi-scale representations for overlapped backgrounds.
- The lightweight HiFormer shows lower model complexity while attaining eminent or on-par performance compared with published medical image segmentation models.
5. Ablation Study
The ablation study examines CNN backbones, DLF configurations, feature consistency, model parameters, and qualitative segmentation outputs across the evaluated datasets.
- CNN backbone ablation: ResNet backbones produced the best performance, while increasing CNN backbone size did not necessarily improve results.The comparison uses the Synapse dataset and keeps other HiFormer-B configurations identical.
- DLF configuration ablation: The DLF configuration with (2, 1) depths for the small and large levels and six heads for both levels performed best.Increasing the number of heads did not necessarily improve performance.
- DLF configuration ablation: Doubling the MLP expanding ratio increased DSC by 1.04% and reduced HD by 1.82% in the reported comparison.The expanding ratio r significantly affected performance.
- Qualitative evaluation: HiFormer predictions visually matched provided ground-truth masks on SegPC and captured finer structures with more precise contours on skin-lesion images.The visual comparisons use ground-truth boundaries in green and predicted boundaries in blue.
- Feature consistency: Feature visualizations and module-level experiments were used to assess feature consistency before and after applying DLF.The supplementary experiments report each module’s contribution to producing more consistent features.
6. Discussion
Across five challenging medical image segmentation datasets, HiFormer generally surpassed state-of-the-art CNN and transformer-based methods, while retaining failure cases on some anatomical and low-contrast images.
- 6. Discussion: HiFormer surpassed state-of-the-art methods in most cases across five challenging medical image segmentation datasets.The reported quantitative results were consistent with visual analyses of organ segmentation.
- 6. Discussion: Visual results showed noiseless segmentation for organs such as the liver and kidney, consistent with the quantitative benchmarks.The discussion also reports failure cases for the aorta.
- 6. Discussion: Low-contrast skin images remained difficult for HiFormer.This limitation is explicitly reported alongside the model’s successes and failure cases.
7. Conclusions
HiFormer combines CNN local representations with Swin Transformer global features and uses Double-Level Fusion for finer feature integration in medical image segmentation.
- 7. Conclusions: HiFormer is a hybrid CNN-transformer method that combines Swin Transformer global features with local representations from a CNN encoder.The method targets medical image segmentation.
- 7. Conclusions: The Double-Level Fusion module performs finer fusion of features derived from the CNN and transformer representations.The paper presents this design as balancing low-level detail preservation with long-range interaction modeling.
- 7. Conclusions: HiFormer achieved superior performance over CNN-based, vanilla transformer-based, and hybrid models.The conclusion attributes this outcome to retaining low-level feature details while modeling long-range interactions.
Supplementary Material
The supplementary material adds experiments on DLF impact, DLF optimality, multi-level feature fusion, and feature consistency.
- Supplementary Material: The supplementary material expands Table 6 and adds experiments on the proposed DLF module in Tables 9–10.It also discusses the optimality and intuition of DLF feature fusion.
- Supplementary Material: Additional material explains feature fusion across different levels and the intuitions behind feature consistency.These additions complement the paper’s main DLF analysis.
A. Impact and justification of the DLF Module
HiFormer uses a CNN backbone before successive Swin Transformer modules to combine local representations with multi-scale global dependencies. Variable S, L, and B designs address different accuracy–speed trade-offs.
- Impact and justification: A rich CNN backbone is used before the transformer to preserve local features that transformers may miss.The choice is motivated by transformers’ local-feature deficiency and the medical domain’s limited data.
- Impact and justification: Successive Swin Transformer modules capture multi-scale global dependencies for organs and tissues with diverse sizes and scales.
- Impact and justification: S, L, and B HiFormer variants support model selection according to the accuracy-speed trade-off.The variants are intended to provide a general and stable model across different scales.
A.2. Hyper-parameter Optimization
The study examines model scaling, transformer-head count, and CNN backbones to balance representation quality against computational cost. It selects six transformer heads and considers ResNet and DenseNet alternatives.
- Hyper-parameter Optimization: Increasing the S, L, r pairs beyond 3 was hypothesized to cause overparameterization and redundant feature extraction.The authors link larger pairs to substantial computational cost, conflicting with their low-parameter design goal.
- Hyper-parameter Optimization: Six transformer heads were selected as the ideal choice through cross-validation on the Synapse dataset.Configurations with 3 and 12 heads were also evaluated in the ablation study.
- Hyper-parameter Optimization: ResNet and DenseNet were considered as CNN backbones because ResNet skip connections facilitate multi-level representation.The passage also notes that deeper backbones may produce overly high-level representations for later transformer feature extraction.
C. Feature Consistency
Feature visualizations and module-removal experiments indicate that DLF improves feature consistency and that both hierarchical levels contribute complementary information. The module sequence improves segmentation metrics relative to the CNN-only configuration.
- Feature Consistency: Before DLF, attention is diffuse, whereas after DLF it more clearly emphasizes the desired organ.This visualization is reported for HiFormer-B using Grad-CAM.
- Feature Consistency: The larger feature level provides fine-grained information, while the smaller level supplies additional information; both are required.
- Feature Consistency: Adding the Swin Transformer produces a 9.93 drop in HD, indicating predictions become closer to their corresponding labels.
- Feature Consistency: Applying DLF subsequently increases the dice score and decreases HD, supporting its role in making features more consistent.