Source-linked AI summary
TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation
Yundong Zhang, Huiye Liu, Qiang Hu
TL;DR
Medical image segmentation needs global context without sacrificing local detail or efficiency. TransFuse addresses this with parallel CNN and transformer branches plus BiFusion feature fusion. The method reports state-of-the-art performance across multiple medical segmentation tasks while reducing parameters and improving inference speed.
Problem
CNN-based segmentation struggles to model global context efficiently, while deep encoders and aggressive downsampling can lose localized details.
Method
TransFuse uses parallel shallow CNN and transformer branches, combining their multi-level features with the BiFusion module.
Results
TransFuse achieves state-of-the-art performance across polyp, skin lesion, hip, and prostate segmentation while improving parameter and inference efficiency.
Takeaways & Limitations
The architecture combines CNN spatial inductive bias with transformer global-relation modeling through late fusion.
Takeaways & Limitations
The evaluation excludes ISIC2018 because its missing test-set annotations prevent fair comparison with existing methods.
Abstract
from arXiv · showhide
Medical image segmentation - the prerequisite of numerous clinical needs - has been significantly prospered by recent advances in convolutional neural networks (CNNs). However, it exhibits general limitations on modeling explicit long-range relation, and existing cures, resorting to building deep encoders along with aggressive downsampling operations, leads to redundant deepened networks and loss of localized details. Hence, the segmentation task awaits a better solution to improve the efficiency of modeling global contexts while maintaining a strong grasp of low-level details. In this paper, we propose a novel parallel-in-branch architecture, TransFuse, to address this challenge. TransFuse combines Transformers and CNNs in a parallel style, where both global dependency and low-level spatial details can be efficiently captured in a much shallower manner. Besides, a novel fusion technique - BiFusion module is created to efficiently fuse the multi-level features from both branches. Extensive experiments demonstrate that TransFuse achieves the newest state-of-the-art results on both 2D and 3D medical image sets including polyp, skin lesion, hip, and prostate segmentation, with significant parameter decrease and inference speed improvement.
1 Introduction
CNNs perform strongly in medical image segmentation but struggle to capture global context efficiently. Existing solutions deepen networks and downsample aggressively, risking washed-out low-level features, lost local information, and unstable training on small datasets.
- CNNs build hierarchical task-specific representations end-to-end and have achieved strong performance across medical image segmentation tasks.
- Large receptive fields require deep stacks of convolutions and repeated downsampling, increasing network depth and reducing spatial resolution.
- Very deep CNNs can wash out low-level features through diminishing feature reuse, weakening information important for pixel-wise segmentation.
- Downsampling discards local information, while parameter-heavy models trained on small medical datasets can be unstable and prone to overfitting.
- Non-local self-attention models global context, but its quadratic complexity in spatial size generally restricts use to low-resolution feature maps.
- TransFuse instead runs shallow CNN and transformer segmentation branches in parallel and fuses their features with BiFusion.
2 Proposed Method
TransFuse processes images through parallel CNN and transformer branches, then combines same-resolution features with BiFusion and gated skip connections. This design targets global context and local detail while avoiding very deep encoders.
- Architecture: The CNN and transformer branches process information differently in parallel, with corresponding feature maps later combined for segmentation.
- Transformer Branch: The transformer branch divides the image into patches, adds positional embeddings, and applies stacked self-attention and MLP layers.
- Transformer Branch: Self-attention updates each embedded patch by aggregating information globally at every transformer layer.
- Transformer Branch: Progressive upsampling reconstructs spatial feature maps at multiple scales for late fusion with corresponding CNN features.
- CNN Branch: The CNN branch removes its final block and uses the transformer branch to obtain global context, producing a shallower model with richer local information.
- BiFusion Module: BiFusion combines CNN and transformer features using self-attention, spatial filtering, and a bilinear Hadamard product before residual processing.
- BiFusion Module: Channel attention promotes transformer global information, while spatial attention enhances CNN local details and suppresses irrelevant regions.
- Prediction and Training: The fused features are combined through attention-gated skip connections, and the network is trained end-to-end with weighted IoU and binary cross-entropy losses.
3 Experiments and Results
TransFuse is evaluated across diverse 2D and 3D medical image segmentation tasks using task-specific metrics and comparisons. Results report strong accuracy alongside reduced model size and faster inference, while ablations examine the parallel branches and BiFusion module.
- Evaluation setup: TransFuse is evaluated on polyp, skin lesion, hip, and prostate segmentation using task-specific quantitative metrics.The experiments include both 2D and 3D datasets, with selected visualizations reported for the evaluated tasks.
- Polyp segmentation: 5.2% average mDice improvement is achieved by TransFuse-S on unseen polyp datasets ColonDB, EndoSene, and ETIS.TransFuse-S also uses 26.3M parameters and reaches 98.7 FPS, compared with 33.3M and 85.3 FPS for HarDNet-MSEG and 32.5M and 63.4 FPS for PraNet.
- Skin lesion segmentation: 1.7% higher Jaccard score is reported for TransFuse-S than SLSDeep on the ISIC 2017 skin-lesion test set.TransFuse-S converges in less than one-third the epochs and outperforms Unet++ with a comparable parameter count.
- Hip segmentation: 30% and 34% average HD reductions are reported against HRNetV2 and Unet++, respectively, on the hip segmentation task.TransFuse-S performs best on both Hausdorff Distance and Average Surface Distance, indicating more precise contours in the reported comparison.
- Prostate segmentation: 4.2% higher mean Dice is reported than nnUNet-2d, while comparison with nnUNet-3d reduces parameters by ∼41% and increases throughput by ∼50%.The prostate results use 5-fold cross validation and report throughput on GTX1080.
- Ablation studies: Ablations show that combining CNN and Transformer branches, using parallel rather than sequential models, and applying BiFusion improves the reported mean Dice results.The proposed model exceeds the double-branch CNN baseline by 2.2% on Kvasir and 18.7% on ColonDB; separate experiments find benefits from each BiFusion component.
4 Conclusion
TransFuse combines Transformers and CNNs through late fusion for medical image segmentation. The paper reports state-of-the-art performance across varied tasks while remaining efficient in parameters and inference speed, and proposes future evaluation on additional medical tasks.
- Conclusion: TransFuse combines Transformers and CNNs with late fusion for medical image segmentation.The architecture uses CNN inductive bias for spatial correlation and Transformers for global relationships.
- Conclusion: TransFuse achieves state-of-the-art performance across varied segmentation tasks while improving parameter and inference-speed efficiency.The conclusion describes this outcome across the paper's evaluated medical image segmentation tasks.
- Future work: Future work will improve vanilla Transformer efficiency and test TransFuse on landmark detection and disease classification.These directions are stated as planned extensions beyond the reported segmentation experiments.