Source-linked AI summary
MISSFormer: An Effective Medical Image Segmentation Transformer
Xiaohong Huang, Zhifang Deng, Dandan Li, Xueguang Yuan
TL;DR
Medical image segmentation requires long-range dependencies, which CNNs model poorly, while transformers have limited local-context modeling. MISSFormer addresses both issues with Enhanced Transformer Blocks and an Enhanced Transformer Context Bridge operating on hierarchical multi-scale features. The authors report effective and robust performance across two dataset forms, with a 3.63 DSC improvement over the U-SegFormer baseline in an ablation.
Problem
CNNs struggle with long-range dependencies because convolution is local, whereas transformers have limited ability to model local context in medical image segmentation.
Method
MISSFormer is a position-free hierarchical U-shaped transformer using Enhanced Transformer Blocks and a context bridge to capture local, global, and multi-scale feature correlations.
Results
The authors report effective and robust segmentation results across two dataset forms; an ablation shows a 3.63 DSC improvement over the U-SegFormer baseline.
Takeaways & Limitations
The paper supports combining long-range dependency modeling, local-context enhancement, and hierarchical multi-scale feature integration for medical image segmentation.
Abstract
from arXiv · showhide
The CNN-based methods have achieved impressive results in medical image segmentation, but they failed to capture the long-range dependencies due to the inherent locality of the convolution operation. Transformer-based methods are recently popular in vision tasks because of their capacity for long-range dependencies and promising performance. However, it lacks in modeling local context. In this paper, taking medical image segmentation as an example, we present MISSFormer, an effective and powerful Medical Image Segmentation tranSFormer. MISSFormer is a hierarchical encoder-decoder network with two appealing designs: 1) A feed-forward network is redesigned with the proposed Enhanced Transformer Block, which enhances the long-range dependencies and supplements the local context, making the feature more discriminative. 2) We proposed Enhanced Transformer Context Bridge, different from previous methods of modeling only global information, the proposed context bridge with the enhanced transformer block extracts the long-range dependencies and local context of multi-scale features generated by our hierarchical transformer encoder. Driven by these two designs, the MISSFormer shows a solid capacity to capture more discriminative dependencies and context in medical image segmentation. The experiments on multi-organ and cardiac segmentation tasks demonstrate the superiority, effectiveness and robustness of our MISSFormer, the experimental results of MISSFormer trained from scratch even outperform state-of-the-art methods pre-trained on ImageNet. The core designs can be generalized to other visual segmentation tasks. The code has been released on Github: https://github.com/ZhifangDeng/MISSFormer
1 Introduction
MISSFormer addresses complementary limitations in medical image segmentation: CNNs struggle with long-range dependencies, while transformers struggle with local context. It combines an enhanced transformer architecture with multi-scale context modeling for accurate segmentation.
- Accurate medical image segmentation supports preoperative evaluation and auxiliary diagnosis by providing a basis for subsequent analysis and treatment.
- CNN-based methods capture detailed information through U-shaped skip connections but struggle to model long-range dependencies because convolution is local.
- Transformer-based methods offer long-range dependency modeling, but self-attention has limited ability to capture local context.
- MISSFormer is a position-free, hierarchical U-shaped transformer whose encoder, bridge, decoder, and skip connections use Enhanced Transformer Blocks.
- Enhanced Mix-FFN improves feature discrimination while modeling long-range dependencies and local context.
- The Enhanced Transformer Context Bridge captures local and global correlations among hierarchical multi-scale features.
2 Related Work
Related work spans CNN-based medical segmentation, vision transformers, and transformer-assisted medical segmentation. These approaches motivate MISSFormer's combination of hierarchical processing, long-range dependency modeling, and local-context representation.
- Medical image segmentation: U-shaped networks became foundational for medical image segmentation because skip connections preserve detailed information and support strong performance.
- Medical image segmentation: Medical segmentation methods have used residual backbones, dilated convolutions, pyramid pooling, and attention mechanisms to enlarge receptive fields or refine features.
- Vision transformers: Vision transformers introduced transformer-based visual processing and pursued global dependencies, efficient training, reduced complexity, and hierarchical representations.
- Transformers for medical image segmentation: Transformer-based medical segmentation methods place transformers in encoders, decoders, or bridges to capture long-range dependencies and improve segmentation.
3 Method
MISSFormer uses a hierarchical encoder-decoder with Enhanced Transformer Blocks and a context bridge to combine efficient long-range modeling with local and multi-scale context. Its decoder produces pixel-wise segmentation from enhanced features and skip connections.
- Overall pipeline: MISSFormer divides images into overlapping 4*4 patches, encodes hierarchical multi-scale features, bridges them, and decodes pixel-wise predictions.Patch merging builds encoder stages, while patch expanding restores resolution in the decoder.
- Enhanced Transformer Block: The Enhanced Transformer Block combines LayerNorm, Efficient Self-Attention, and Enhanced Mix-FFN.
- Enhanced Transformer Block: Efficient self-attention applies spatial reduction to lower complexity and support processing of high-resolution feature maps.The original attention complexity is O(N^2), while spatial reduction reduces the attention cost to O(N^2/R).
- Enhanced Transformer Block: Enhanced Mix-FFN adds a skip connection before depth-wise convolution and applies LayerNorm to improve feature alignment and discrimination.The depth-wise 3×3 convolution captures local context while the modified feed-forward structure retains the input through a skip connection.
- Enhanced Transformer Block: Recursive skip connections extend Enhanced Mix-FFN so repeated transformations produce more expressive feature representations.
- Enhanced Transformer Context Bridge: The Enhanced Transformer Context Bridge flattens, channel-aligns, concatenates, transforms, splits, and restores hierarchical features before decoding.It models local and global correlations across multi-scale encoder features; the reported bridge depth is 4.
4 Experiments
Experiments evaluate MISSFormer through controlled ablations and comparisons on Synapse and ACDC datasets. Results support the U-shaped architecture, Enhanced Mix-FFN, multi-scale context bridge, and strong segmentation performance, including training from scratch.
- 4 Experiments: Experiments use Synapse multi-organ CT and ACDC cardiac MRI segmentation datasets, with ablations conducted on Synapse under matched settings.Synapse contains 30 CT scans and 3779 axial images; 18 scans are used for training and 12 for testing, with DSC and HD evaluation.
- Architecture selection: The U-shaped transformer outperforms SegFormer because skip connections fuse corresponding detail information across stages.SegFormer B5 does not achieve breakthrough results, which the authors attribute to its parameter count and limited medical-data size.
- Enhanced Mix-FFN ablation: A 3.63 DSC promotion over the U-SegFormer baseline follows the redesigned feed-forward network, after summation skips and layer normalization also improve performance.Summation skip connections improve performance by more than 2.6%, while adding layer normalization yields a 1% improvement over U-SegFormer with skips.
- Gradient and convergence analysis: Directly embedding a 3×3 convolution slows middle-layer updates to one-third of U-mlpFormer, whereas the proposed components improve convergence and evaluation results.The analysis links the slower gradient updates to the middle layer’s potentially poorer weights and contrasts this with the proposed components.
- Context bridge ablation: The Enhanced Transformer Context Bridge improves MISSFormer S by up to 2.26% DSC at recursive step 1, while four bridge layers are suitable for the limited medical data.Increasing recursive steps eventually reduces gains, and integrating more scale features is reported to improve learning of long-range dependencies and local context.
- Context bridge ablation: Enhanced Mix-FFN in the context bridge provides better segmentation performance by combining discriminative global and local features.The compared MLP bridge gives more accurate edges, while Mix-FFN improves segmentation through local information supplementation.
- Comparison with state-of-the-art methods: MISSFormer achieves state-of-the-art performance on almost all Synapse measures and better edge predictions and difficult-case segmentation than TransUNet and Swin-Unet despite training from scratch.The compared TransUNet and Swin-Unet encoders are ImageNet-pretrained, while MISSFormer is randomly initialized and trained on Synapse.
- Comparison with state-of-the-art methods: On ACDC, MISSFormer maintains first position in segmentation accuracy, indicating reported generalization and robustness across the evaluated cardiac task.The authors attribute this result to MISSFormer’s feature extraction capability.
5 Conclusion
MISSFormer captures global dependencies and local context through enhanced feature modeling and multi-scale feature integration, achieving effective and robust medical image segmentation across two dataset types.
- MISSFormer is a position-free, hierarchical U-shaped transformer for medical image segmentation.
- Its Enhanced Mix Block addresses feature-discrimination limitations from directly embedding convolution in the feed-forward network.
- The method integrates multi-scale features from its hierarchical transformer encoder, which the paper identifies as essential for accurate segmentation.
- Experiments on two different dataset forms show the effectiveness and robustness of MISSFormer.