Source-linked AI summary
ColonFormer: An Efficient Transformer based Method for Colon Polyp Segmentation
Nguyen Thanh Duc, Nguyen Thi Oanh, Nguyen Thi Thuy, Tran Minh Triet, Dinh Viet Sang
TL;DR
Automatic polyp segmentation is difficult because existing CNN, Transformer, and hybrid approaches have limitations in modeling global context and multi-level decoding features. ColonFormer addresses these issues with a hierarchical Transformer encoder, pyramid CNN decoder, and boundary-refinement module. The paper reports that ColonFormer significantly outperforms existing state-of-the-art models on popular benchmark datasets.
Problem
Polyp segmentation is challenging because polyps have varied appearances and unclear boundaries, while Vision Transformers can be limited by scarce medical training data.
Method
ColonFormer combines a lightweight hierarchical Transformer encoder, hierarchical pyramid CNN decoder, and residual axial-attention refinement module for multi-scale representation and boundary correction.
Results
ColonFormer significantly outperforms existing state-of-the-art models on popular polyp-segmentation benchmark datasets.
Takeaways & Limitations
The architecture provides an efficient unified Transformer-CNN approach for accurate polyp segmentation while retaining multi-scale hierarchical feature representation.
Takeaways & Limitations
The authors identify reducing computational complexity with lightweight or sparse self-attention as future work.
Abstract
from arXiv · showhide
Identifying polyps is challenging for automatic analysis of endoscopic images in computer-aided clinical support systems. Models based on convolutional networks (CNN), transformers, and their combinations have been proposed to segment polyps with promising results. However, those approaches have limitations either in modeling the local appearance of the polyps only or lack of multi-level features for spatial dependency in the decoding process. This paper proposes a novel network, namely ColonFormer, to address these limitations. ColonFormer is an encoder-decoder architecture capable of modeling long-range semantic information at both encoder and decoder branches. The encoder is a lightweight architecture based on transformers for modeling global semantic relations at multi scales. The decoder is a hierarchical network structure designed for learning multi-level features to enrich feature representation. Besides, a refinement module is added with a new skip connection technique to refine the boundary of polyp objects in the global map for accurate segmentation. Extensive experiments have been conducted on five popular benchmark datasets for polyp segmentation, including Kvasir, CVC-Clinic DB, CVC-ColonDB, CVC-T, and ETIS-Larib. Experimental results show that our ColonFormer outperforms other state-of-the-art methods on all benchmark datasets.
1 Introduction
Colon polyp segmentation remains difficult because appearance and boundaries vary, while existing CNN and Transformer approaches have limitations in capturing global context and multi-level features. ColonFormer combines hierarchical Transformer and pyramid CNN components with refinement, and is evaluated on five benchmark datasets.
- Colorectal cancer causes over 694,000 fatalities annually, and each 1% increase in adenoma detection is linked to a 3% reduction in colon cancer risk.
- Polyp segmentation is challenging because polyps vary in shape, size, texture, and color, while boundaries may be unclear under difficult imaging conditions.
- CNNs capture mainly local information because their limited receptive fields restrict modeling of spatial context and global information.
- Transformers capture long-range dependencies through attention, motivating architectures that combine Transformer and CNN strengths for semantic segmentation.
- ColonFormer integrates a hierarchical Transformer encoder, hierarchical pyramid CNN decoder, and refinement module for efficient and accurate polyp segmentation.
- The refinement technique uses residual axial attention for feature fusion and smoothing, while experiments compare ColonFormer with state-of-the-art methods on five benchmark datasets.
2 Related Work
Related work spans CNN-based encoder-decoder models, attention mechanisms, and Transformer-based segmentation architectures. These approaches motivate ColonFormer through limitations involving local context, data requirements, computational cost, and the need for improved efficiency and robustness.
- Convolutional Neural Networks: CNNs extract increasingly abstract multi-level features, but UNet-based architectures can retain noisy low-level information through skip connections.
- Convolutional Neural Networks: UNet variants such as UNet++ and DoubleUNet improve segmentation using nested networks, deep supervision, ASPP, and squeeze-and-excitation blocks.
- Attention Mechanism: Attention modules help networks focus on necessary information and polyp boundaries, but CNN locality limits modeling of long-range spatial dependencies.
- Vision Transformer: Vision Transformers model images through attention but require extensive training data and are limited on small medical datasets; Kvasir contains 1000 images.
- Vision Transformer: Swin Transformer and MixFormer use hierarchical or efficient attention designs to reduce tokens, preserve local continuity, and lower computational complexity.
- Vision Transformer: TransUNet and TransFuse apply Transformers to polyp segmentation, but hybrid CNN-Transformer designs can incur high computational costs.
- The paper targets reduced network size and latency together with improved accuracy and robustness for Transformer-based polyp segmentation.
3 ColonFormer
ColonFormer combines a hierarchical Transformer encoder, hierarchical pyramid CNN decoder, and refinement module to model multi-scale context and improve polyp-boundary prediction. Its compound weighted loss emphasizes difficult pixels and neighboring-pixel regions during multi-scale training.
- Architecture: ColonFormer uses a hybrid encoder, decoder, and refinement module for polyp segmentation.The encoder is based on MiT, while the decoder and refinement stages process multi-scale representations.
- Encoder: The MiT encoder extracts hierarchical multi-level features while reducing self-attention complexity through efficient attention.MiT also uses a 3 × 3 convolution in its feed-forward network for positional information.
- Decoder: The UPer Decoder fuses a PPM global map with multi-scale encoder features to produce a richer semantic map.Convolutional layers emphasize coherence between neighboring feature elements during fusion.
- Refinement Module: The refinement module combines CFP with three RA-RA blocks to incrementally correct polyp boundaries.The residual connection can bypass axial attention when retaining edge information is beneficial.
- Loss Function: ColonFormer trains with weighted focal and weighted IoU losses, combining pixel-level difficulty with neighboring-pixel relationships.The total loss is applied to multi-scale outputs after each output is upsampled to the ground-truth image size.
- Loss Function: The weighting scheme assigns larger βij values to pixels differing from their 31 × 31 neighborhoods, emphasizing polyp edges.This focuses learning on boundary regions that are harder to classify.
- Experiments: Table 1 compares methods on five polyp-segmentation test sets, with ColonFormer results averaged over five runs.The reported datasets are Kvasir, ClinicDB, ColonDB, CVC-T, and ETIS-Larib.
4 Experiments
ColonFormer is evaluated through six experimental setups spanning benchmark, cross-validation, cross-dataset, efficiency, and ablation analyses. Across these evaluations, it generally achieves strong segmentation performance, stability, generalization, and computational efficiency, with some low-resolution limitations.
- Experimental setup: Experiments use five benchmark datasets: Kvasir, CVC-ClinicDB, CVC-ColonDB, CVC-T, and ETIS-Larib.The evaluation includes standard splits, 5-fold cross-validation, and cross-dataset configurations.
- Benchmark comparison: ColonFormer generally outperforms benchmark models, with ColonFormer-S and -L improving over TransFuse-L* by 3% in mDice and 2.7% in mIOU on ColonDB.On ETIS-Larib, ColonFormer-S improves over CaraNet by 5.2% in mDice and 4.8% in mIOU, while ColonFormer-L improves by 6.4% and 5.9%.
- Cross-validation: Both ColonFormer variants outperform other methods in mDice, mIOU, precision, and recall on both CVC-ClinicDB and Kvasir cross-validation datasets.ColonFormer also achieves the lowest standard deviation for every reported metric, indicating the strongest stability in this experiment.
- Cross-dataset evaluation: ColonFormer-S improves over PraNet by 7−8% across all metrics in one cross-dataset configuration, while ColonFormer-L improves mDice by 10.1% and recall by 18.3% in another.The reported configurations use different training and testing datasets, including CVC-ColonDB, CVC-ClinicDB, and ETIS-Larib.
- Efficiency: ColonFormer-S has competitive size and computational complexity with lightweight CNN models, while ColonFormer-L is more efficient in GFLOPs than other Transformer-based methods.The comparison includes PraNet and HarDNet-MSEG among CNN-based models.
- Ablation studies: Replacing the MLP decoder with the UPer Decoder reduces computation from 33.68 GFLOPs to 20.99 GFLOPs while producing similar metrics across test datasets.The refinement module improves performance across all datasets, although ColonFormer-L slightly underperforms on CVC-ColonDB and CVC-T in the reported comparison.
5 Conclusion
ColonFormer combines Transformer and CNN components with hierarchical feature learning and boundary refinement for colon polyp segmentation. The authors report that it outperforms existing state-of-the-art models, while computational complexity remains a future concern.
- ColonFormer combines Transformer and CNN architectures to learn a powerful multi-scale hierarchical feature representation for colon polyp segmentation.
- Its reverse-attention mechanism is enhanced with axial attention and a residual connection.
- The refinement module incrementally corrects polyp boundaries from the decoder’s coarse global map.
- ColonFormer significantly outperforms existing state-of-the-art models on popular benchmark datasets.
- The authors identify reducing computational complexity with lightweight or sparse self-attention layers as future work.