PaperlayerView exact revision on arXiv

Source-linked AI summary

HiFuse: Hierarchical Multi-Scale Feature Fusion Network for Medical Image Classification

Xiangzuo Huo, Gang Sun, Shengwei Tian, Yan Wang, Long Yu, Jun Long, Wendong Zhang, Aolun Li

arXiv:2209.10218v1eess.IVcs.CV

TL;DR

Medical image classification remains challenging because medical images exhibit intra-class variation, inter-class similarity, and diverse local and global information requirements. HiFuse addresses this challenge with a three-branch hierarchical fusion model combining local and global feature extraction with adaptive multi-scale fusion, and reports the best results across three medical image datasets. The model is presented as scalable with linear computational complexity.

  • Problem

    Medical image classification is challenging because diverse imaging and clinical pathology produce substantial intra-class variation and inter-class similarity, making global semantic information important.

  • Method

    HiFuse combines parallel local and global feature blocks with an adaptive hierarchical feature fusion block to fuse multi-scale representations from CNN and Transformer-inspired branches.

  • Results

    HiFuse reports the best results on the ISIC2018, Covid-19, and Kvasir medical classification datasets compared with other advanced models.

  • Takeaways & Limitations

    The hierarchical design provides rich scalability and linear computational complexity while comprehensively combining local, global, deep, and shallow lesion features.

  • Takeaways & Limitations

    The model could be further improved by assigning branch depth and width according to each task so local features and global representations become more directional.

Abstract

from arXiv · show

Medical image classification has developed rapidly under the impetus of the convolutional neural network (CNN). Due to the fixed size of the receptive field of the convolution kernel, it is difficult to capture the global features of medical images. Although the self-attention-based Transformer can model long-range dependencies, it has high computational complexity and lacks local inductive bias. Much research has demonstrated that global and local features are crucial for image classification. However, medical images have a lot of noisy, scattered features, intra-class variation, and inter-class similarities. This paper proposes a three-branch hierarchical multi-scale feature fusion network structure termed as HiFuse for medical image classification as a new method. It can fuse the advantages of Transformer and CNN from multi-scale hierarchies without destroying the respective modeling so as to improve the classification accuracy of various medical images. A parallel hierarchy of local and global feature blocks is designed to efficiently extract local features and global representations at various semantic scales, with the flexibility to model at different scales and linear computational complexity relevant to image size. Moreover, an adaptive hierarchical feature fusion block (HFF block) is designed to utilize the features obtained at different hierarchical levels comprehensively. The HFF block contains spatial attention, channel attention, residual inverted MLP, and shortcut to adaptively fuse semantic information between various scale features of each branch. The accuracy of our proposed model on the ISIC2018 dataset is 7.6% higher than baseline, 21.5% on the Covid-19 dataset, and 10.4% on the Kvasir dataset. Compared with other advanced models, the HiFuse model performs the best. Our code is open-source and available from https://github.com/huoxiangzuo/HiFuse.

1 Introduction

Medical image classification remains challenging because images exhibit intra-class variation, inter-class similarity, and a need for global semantic modeling. HiFuse addresses this by hierarchically fusing CNN-based local features and Transformer-based global representations across scales.

  • Medical image classification supports computer-aided diagnosis, image retrieval, and medical-data mining.
  • Intra-class variation and inter-class similarity make medical images difficult to classify and require global semantic information.
  • HiFuse introduces a three-branch parallel hierarchical fusion network combining CNN and Transformer advantages for medical image classification.
  • Parallel local and global feature blocks capture local spatial context and global semantic representations at different scales.
  • The adaptive HFF block fuses different-scale semantic features using spatial attention, channel attention, residual inverted MLP, and shortcut connections.
  • HiFuse achieves relatively good results on the ISIC2018, Covid-19, and Kvasir medical image datasets.

2 Related Works

Related work combines convolutional and self-attention features to address Transformers’ limited local inductive bias. However, medical images remain challenging because of their diverse appearances and strong intra-class and inter-class variations.

  • Traditional methods classify medical images using color, texture, shape, and combined handcrafted descriptors.
  • Deep convolutional methods improve classification accuracy while reducing reliance on manual feature extraction for clinical auxiliary diagnosis.
  • Deep learning studies include large-scale skin-cancer diagnosis, transfer learning, and ensembles of pretrained convolutional architectures.
  • Vision Transformers extend self-attention to image classification and related vision tasks by modeling images as token sequences.
  • Hybrid models couple convolutional local features with Transformer global representations, but medical images still pose challenges through diverse imaging and pathological characteristics.

3 Proposed Method

HiFuse uses parallel local and global branches to extract multi-scale features, then adaptively fuses hierarchical representations with HFF blocks for medical image classification.

  • Multi-stage Design: HiFuse combines parallel local and global feature blocks across four stages to preserve and extract information at different scales.The local branch captures spatial context, while the global branch models semantic representations; their aligned hierarchy supports feature fusion.
  • Global Feature Block: The global branch uses window-based multi-head self-attention to model semantic information while reducing attention computation through local windows.Each stage applies LayerNorm, W-MSA, linear transformation, residual connections, relative position bias, and shifted windows.
  • Local Feature Block: The local feature block uses 3 × 3 depthwise convolution and linear layers to capture local spatial features with reduced FLOPs.Depthwise convolution groups channels independently, followed by cross-channel interaction using Transformer-inspired normalization and activation.
  • Hierarchical Feature Fusion: HiFuse addresses the challenge of effectively combining different-scale features within aligned local and global branches.The branches share similar hierarchical structures and channel designs, providing a basis for multi-scale fusion.
  • Hierarchical Feature Fusion: The HFF block adaptively fuses local features, global representations, and previous-stage semantics using spatial attention, channel attention, and residual inverted MLP processing.Its inputs include local and global branch outputs plus downsampled features from the preceding hierarchy.

4 Experiments

The experiments use three medical-image datasets and evaluate classification with accuracy, F1, precision, and recall derived from confusion-matrix counts.

  • Datasets: ISIC2018 contains 10,015 dermoscopic images across seven categories, resized from 650 × 450 to 224 × 224 pixels.
  • Datasets: COVID19-CT contains 349 positive and 397 normal or negative CT scans, resized to 224 × 224 and split 0.6:0.15:0.25 for training, validation, and testing.
  • Datasets: Kvasir contains 4,000 gastrointestinal images across eight classes, with original resolutions ranging from 720 × 576 to 1920 × 1072 pixels.
  • Metrics: The evaluation uses ACC, F1, precision, and recall, all calculated from true-positive, true-negative, false-positive, and false-negative counts.
  • Metrics: Accuracy measures the percentage of correctly identified samples as (TP + TN) / (TP + TN + FP + FN).

4.3 Implementation Details

The implementation compares open-source models under shared image, environment, and hyperparameter settings, using PyTorch and mmcv on an NVIDIA RTX 3090 GPU.

  • Experimental setup: Comparative experiments select open-source state-of-the-art classification models with parameter counts similar to HiFuse.
  • Experimental setup: Training uses an NVIDIA RTX 3090 GPU with 24 GB of video memory, a 1e-4 learning rate, batch size 32, and 100 epochs.
  • Experimental setup: The experiments use 224×224 images, the same operating environment and hyperparameters, and identical training, validation, and test sets for fairness.
  • Implementation: The implementation uses PyTorch within the mmcv framework and adopts cosine annealing for the learning-rate schedule.
  • Implementation: Table 2 provides additional parameter settings for the experiments.

4.4 Ablation Study

An ablation study on ISIC2018 progressively adds global and HFF components, showing performance gains from global features and hierarchical fusion.

  • Ablation results: Adding the global path increased Acc by 2.47% and F1 by 10.2% in the HiFuse-Tiny ablation.
  • Ablation results: Adding HFF components increased Acc by 7.4% and F1 by 8.67%.
  • Ablation design: The ablation progresses from the local path through the global path, channel and spatial attention, inverted residual MLP, and shortcut to final HiFuse-Tiny.
  • Interpretation: The study reports that global features improve representation ability and that the HFF block provides better fusion of global-local features.

4.5 Visual Inspection of HiFuse

Grad-CAM comparisons examine dermoscopy and upper gastrointestinal endoscopy, contrasting local and global attention with HiFuse’s lesion-focused representations.

  • Visualization method: Grad-CAM visualizations compare ConvNeXt-T, Swin Transformer-T, and HiFuse-T using the last model layer before the linear layer.
  • Visualization data: The visualization is presented for selected dermoscopy and upper gastrointestinal endoscopy images in Figure 4.
  • Model comparison: ConvNeXt emphasizes local features, whereas Swin Transformer focuses more on global features.
  • Model comparison: HiFuse produces higher thermal values in lesion areas and more accurately covers lesions than the compared models.

4.6 Results on ISIC2018 Dataset

On ISIC2018, HiFuse outperformed the compared advanced classification models, combining improved accuracy with modest computational cost.

  • HiFuse-Base used 10.97 G Flops while achieving the best reported ISIC2018 accuracy of 84.12%.

4.8 Results on Kvasir Dataset

On Kvasir, HiFuse-Small achieved the best reported accuracy and F1 value, while the authors note that lower depth may suit smaller medical datasets.

  • 85% accuracy and 84.96% F1 value made HiFuse-Small the best-performing model on Kvasir.
  • HiFuse-Base accuracy decreased slightly as depth increased on this dataset.The authors state that an appropriate drop path hyperparameter could slightly improve it.
  • Across the reported experiments, HiFuse achieved higher accuracy with lower computational complexity by capturing global-local features.

5 Discussion

The discussion frames medical image classification as difficult because of limited, diverse data and the need to model both global and local information. It also identifies several directions for improving HiFuse’s adaptability.

  • Medical images have diverse characteristics and fewer data, making efficient and robust backbone design challenging.
  • HiFuse fuses global-local representations across scales through a three-branch hierarchical feature fusion model.The architecture is described as having linear computational complexity and rich scalability.
  • The hierarchical fusion method is designed to be extended and upgraded beyond medical image classification.
  • Future work should assign branch depth and width according to each task so local and global features become more directional.
  • Future work should design dynamic hierarchical feature selection tailored to different datasets.
  • The authors also propose continuing HiFuse research for medical image segmentation and multimodal tasks.

6 Conclusion

HiFuse is presented as a scalable, linear-complexity hierarchical fusion model that combines local, global, shallow, and deep features for medical image classification.

  • HiFuse uses local feature blocks, global feature blocks, and HFF blocks to fuse representations at different scales.The model is described as having rich scalability and linear computational complexity.
  • Experiments on three medical image datasets reported good results for the proposed method.
  • The authors suggest that HiFuse may contribute to various downstream tasks in medical imagery.
Loading 2209.10218v1…