PaperlayerView exact revision on arXiv

Source-linked AI summary

HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation

Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah Aghdam, Julien Cohen-Adad, Dorit Merhof

arXiv:2207.08518v2cs.CVcs.AI

TL;DR

Medical image segmentation needs both local detail and global context, but CNNs and transformers each omit one of these capabilities. HiFormer combines hierarchical CNN and Swin Transformer representations through Double-Level Fusion, and experiments report superior performance across diverse datasets with low model complexity. Its design selects shallow and deep features for fusion while excluding middle-level features to reduce computational cost.

  • Problem

    CNNs have limited long-range modeling, whereas transformers can miss low-level features needed for medical image segmentation.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    HiFormer combines hierarchical CNN and Swin Transformer encoders with a Double-Level Fusion module that fuses shallow and deep multi-scale features.

    Source: §1 · p. 2, §3.1 · p. 3, §3.1.3 · p. 4

  • Results

    HiFormer surpasses CNN-based, transformer-based, and hybrid methods across five challenging datasets in most cases, with reported advantages in quantitative and qualitative evaluations.

    Source: §1 · p. 2, §6 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    The reported results indicate that combining CNN local representations with transformer global features preserves low-level details while modeling long-range interactions.

    Source: §4.3.2 · p. 6, §7 · p. 9

    The DLF fusion excludes middle-level features to reduce computational cost because their effect on model accuracy was imperceptible.

    Source: §3.1.3 · p. 4

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have been the consensus for medical image segmentation tasks. However, they suffer from the limitation in modeling long-range dependencies and spatial correlations due to the nature of convolution operation. Although transformers were first developed to address this issue, they fail to capture low-level features. In contrast, it is demonstrated that both local and global features are crucial for dense prediction, such as segmenting in challenging contexts. In this paper, we propose HiFormer, a novel method that efficiently bridges a CNN and a transformer for medical image segmentation. Specifically, we design two multi-scale feature representations using the seminal Swin Transformer module and a CNN-based encoder. To secure a fine fusion of global and local features obtained from the two aforementioned representations, we propose a Double-Level Fusion (DLF) module in the skip connection of the encoder-decoder structure. Extensive experiments on various medical image segmentation datasets demonstrate the effectiveness of HiFormer over other CNN-based, transformer-based, and hybrid methods in terms of computational complexity, and quantitative and qualitative results. Our code is publicly available at: https://github.com/amirhossein-kz/HiFormer

1. Introduction

Medical image segmentation requires precise, robust delineation to support anatomical analysis and clinical decision-making. HiFormer addresses limitations in CNNs, transformers, and existing hybrids by combining local and global representations with multi-scale fusion.

  • Medical image segmentation supports injury depiction, disease monitoring, and treatment assessment, making precise and robust results increasingly vital.Source: §1 · p. 1
  • CNNs succeed in segmentation through encoder-decoder structures and skip connections, but their convolution-based modeling limits long-range dependencies and spatial correlations.Source: §1 · pp. 1–2
  • Transformers capture global contexts and long-range dependencies, yet self-attention can omit low-level features important for dense prediction.Source: §1 · p. 2
  • Existing hybrid CNN-transformer methods combine locality and long-range dependencies but do not effectively combine low-level and high-level features.Source: §1 · p. 2
  • HiFormer uses hierarchical CNN and Swin Transformer modules with a Double-Level Fusion module to merge local and global multi-scale representations.Source: §1 · p. 2

2. Related Works

Related work develops CNN, transformer, and hybrid architectures for segmentation, while progressively emphasizing global context, multi-scale representations, and efficiency. These approaches motivate combining complementary local and global modeling capabilities.

  • CNN-based FCNs and U-shaped networks improve segmentation by fusing features from different layers, but CNNs have limited receptive fields for long-range dependencies.Source: §2.1 · p. 2, §2.3 · p. 3
  • Vision Transformers use self-attention to learn global information, while later variants reduce their dependence on large datasets and improve efficiency.Source: §2.2 · pp. 2–3
  • Pyramid and hierarchical vision transformers extract information across scales, including Swin Transformer’s shifted-window approach for efficient local attention.Source: §2.2 · p. 3
  • Medical segmentation methods include pure transformer models such as Swin-UNet and DS-TransUNet, alongside hybrids such as TransUNet that combine low- and high-level features.Source: §2.3 · p. 3

3. Method

HiFormer uses hierarchical CNN and Swin Transformer encoders to combine local, global, and multi-scale representations. Its DLF module fuses shallow and deep features before decoding them into a segmentation map.

Encoder

  • The encoder integrates three-level CNN and Swin Transformer hierarchies, transferring CNN local features into corresponding transformer levels through skip connections.Source: §3.1 · p. 3, §3.1.1 · p. 3

Swin Transformer Module

  • Swin Transformer uses windowed and shifted-window attention to provide linear local attention while enabling cross-window connections.Source: §3.1.2 · pp. 3–4

Double-Level Fusion Module (DLF)

  • DLF uses class tokens, self-attention, and cross-attention to exchange information between the two levels and back-project recalibrated features.Source: §3.1.3 · pp. 4–5, Figure 1 · p. 4
  • The smallest and largest feature maps, P s and P l, are selected for fusion because shallow features provide localization while deeper features provide semantics.Source: §3.1.3 · p. 4, §3.1.2 · p. 4

Decoder

  • The decoder upsamples and convolves the fused P s and P l features into a unified H × W feature map, then produces the final segmentation map.Source: §3.2 · p. 5

4. Experiments

HiFormer is evaluated across multiple medical image segmentation datasets against CNN, transformer, and hybrid methods using task-specific quantitative and qualitative comparisons. The reported results show strong segmentation performance with relatively low model complexity.

  • The experiments cover Synapse multi-organ CT segmentation and skin-lesion datasets, using task-specific metrics including Dice, Hausdorff Distance, sensitivity, specificity, accuracy, and mIOU.Source: §4.1 · p. 5, §4.3 · p. 5

Synapse Multi-Organ Segmentation

  • 2.91% and 1.26% increases in Dice score, with 16.99 and 6.85 decreases in average HD, are reported for HiFormer-B versus TransUnet and Swin-Unet, respectively.Source: §4.3.1 · p. 6

Skin Lesion Segmentation

  • HiFormer reports better performance across ISIC 2017, ISIC 2018, and PH2, with finer contours, stronger boundary performance, and greater robustness to noisy items.Source: §4.3.2 · p. 6

Multiple Mylomia Segmentation

  • HiFormer consistently outperformed the challenge leaderboard across tested configurations using mean IoU and modeled multi-scale representations for overlapped backgrounds.Source: §4.3.3 · p. 6
  • The lightweight HiFormer shows lower model complexity while attaining eminent or on-par performance compared with published medical image segmentation models.Source: §4.4 · p. 6

5. Ablation Study

The ablation study examines CNN backbones, DLF configurations, feature consistency, model parameters, and qualitative segmentation outputs across the evaluated datasets.

CNN backbone ablation

  • ResNet backbones produced the best performance, while increasing CNN backbone size did not necessarily improve results.The comparison uses the Synapse dataset and keeps other HiFormer-B configurations identical.Source: §5 · p. 6, Table 7 · p. 8

DLF configuration ablation

  • The DLF configuration with (2, 1) depths for the small and large levels and six heads for both levels performed best.Increasing the number of heads did not necessarily improve performance.Source: §5 · p. 7, Table 8 · p. 8
  • Doubling the MLP expanding ratio increased DSC by 1.04% and reduced HD by 1.82% in the reported comparison.The expanding ratio r significantly affected performance.Source: §5 · p. 7

Qualitative evaluation

  • HiFormer predictions visually matched provided ground-truth masks on SegPC and captured finer structures with more precise contours on skin-lesion images.The visual comparisons use ground-truth boundaries in green and predicted boundaries in blue.Source: Figure 3 · p. 8, Figure 4 · p. 8

Feature consistency

  • Feature visualizations and module-level experiments were used to assess feature consistency before and after applying DLF.The supplementary experiments report each module’s contribution to producing more consistent features.Source: §5 · p. 8, Table 6 · p. 8

6. Discussion

Across five challenging medical image segmentation datasets, HiFormer generally surpassed state-of-the-art CNN and transformer-based methods, while retaining failure cases on some anatomical and low-contrast images.

6. Discussion

  • HiFormer surpassed state-of-the-art methods in most cases across five challenging medical image segmentation datasets.The reported quantitative results were consistent with visual analyses of organ segmentation.Source: §6 · p. 8
  • Visual results showed noiseless segmentation for organs such as the liver and kidney, consistent with the quantitative benchmarks.The discussion also reports failure cases for the aorta.Source: §6 · p. 8
  • Low-contrast skin images remained difficult for HiFormer.This limitation is explicitly reported alongside the model’s successes and failure cases.Source: §6 · p. 8

7. Conclusions

HiFormer combines CNN local representations with Swin Transformer global features and uses Double-Level Fusion for finer feature integration in medical image segmentation.

7. Conclusions

  • HiFormer is a hybrid CNN-transformer method that combines Swin Transformer global features with local representations from a CNN encoder.The method targets medical image segmentation.Source: §7 · p. 9
  • The Double-Level Fusion module performs finer fusion of features derived from the CNN and transformer representations.The paper presents this design as balancing low-level detail preservation with long-range interaction modeling.Source: §7 · p. 9
  • HiFormer achieved superior performance over CNN-based, vanilla transformer-based, and hybrid models.The conclusion attributes this outcome to retaining low-level feature details while modeling long-range interactions.Source: §7 · p. 9

Supplementary Material

The supplementary material adds experiments on DLF impact, DLF optimality, multi-level feature fusion, and feature consistency.

Supplementary Material

  • The supplementary material expands Table 6 and adds experiments on the proposed DLF module in Tables 9–10.It also discusses the optimality and intuition of DLF feature fusion.Source: Supplementary Material · p. 12
  • Additional material explains feature fusion across different levels and the intuitions behind feature consistency.These additions complement the paper’s main DLF analysis.Source: Supplementary Material · p. 12

A. Impact and justification of the DLF Module

HiFormer uses a CNN backbone before successive Swin Transformer modules to combine local representations with multi-scale global dependencies. Variable S, L, and B designs address different accuracy–speed trade-offs.

Impact and justification

  • A rich CNN backbone is used before the transformer to preserve local features that transformers may miss.The choice is motivated by transformers’ local-feature deficiency and the medical domain’s limited data.Source: §A.1 · p. 12
  • Successive Swin Transformer modules capture multi-scale global dependencies for organs and tissues with diverse sizes and scales.Source: §A.1 · p. 12
  • S, L, and B HiFormer variants support model selection according to the accuracy-speed trade-off.The variants are intended to provide a general and stable model across different scales.Source: §A.1 · p. 12

A.2. Hyper-parameter Optimization

The study examines model scaling, transformer-head count, and CNN backbones to balance representation quality against computational cost. It selects six transformer heads and considers ResNet and DenseNet alternatives.

Hyper-parameter Optimization

  • Increasing the S, L, r pairs beyond 3 was hypothesized to cause overparameterization and redundant feature extraction.The authors link larger pairs to substantial computational cost, conflicting with their low-parameter design goal.Source: §A.2 · p. 12
  • Six transformer heads were selected as the ideal choice through cross-validation on the Synapse dataset.Configurations with 3 and 12 heads were also evaluated in the ablation study.Source: §A.2 · p. 12
  • ResNet and DenseNet were considered as CNN backbones because ResNet skip connections facilitate multi-level representation.The passage also notes that deeper backbones may produce overly high-level representations for later transformer feature extraction.Source: B. Clarification on CNN backbones · p. 12

C. Feature Consistency

Feature visualizations and module-removal experiments indicate that DLF improves feature consistency and that both hierarchical levels contribute complementary information. The module sequence improves segmentation metrics relative to the CNN-only configuration.

Feature Consistency

  • Before DLF, attention is diffuse, whereas after DLF it more clearly emphasizes the desired organ.This visualization is reported for HiFormer-B using Grad-CAM.Source: C. Feature Consistency · p. 12
  • The larger feature level provides fine-grained information, while the smaller level supplies additional information; both are required.Source: C. Feature Consistency · p. 12
  • Adding the Swin Transformer produces a 9.93 drop in HD, indicating predictions become closer to their corresponding labels.Source: C. Feature Consistency · p. 13
  • Applying DLF subsequently increases the dice score and decreases HD, supporting its role in making features more consistent.Source: C. Feature Consistency · p. 13

Related papers

Loading 2207.08518v2…