Source-linked AI summary
FCN-Transformer Feature Fusion for Polyp Segmentation
Edward Sanderson, Bogdan J. Matuszewski
TL;DR
Polyp segmentation is needed to support colorectal cancer screening, but deep-learning methods can struggle to generalise across colonoscopes and transformer models typically produce reduced-resolution maps. FCBFormer addresses this by fusing transformer and fully convolutional branches for full-size prediction, achieving state-of-the-art benchmark and generalisability performance. The authors also identify implementation sensitivity to spatial resolution as a limitation in comparisons with some prior methods.
Problem
Polyp segmentation can support lesion detection and classification, but deep-learning solutions may overfit and generalise poorly, while transformer models typically predict reduced-size maps.
Method
FCBFormer runs transformer and fully convolutional branches in parallel, upsamples and concatenates their outputs, then uses a prediction head to produce a full-size segmentation map.
Results
FCBFormer achieves state-of-the-art performance across mDice, mIoU, mPrecision, and mRecall on Kvasir-SEG, CVC-ClinicDB, and cross-dataset generalisability tests.
Takeaways & Limitations
The results highlight FCBFormer’s strengths in generalisability and handling polyps with challenging morphology within automated colonoscopy-image processing.
Takeaways & Limitations
Some prior methods produced worse results than originally reported, potentially because their implementations were optimised for 256 × 256 rather than the 352 × 352 images used here.
Abstract
from arXiv · showhide
Colonoscopy is widely recognised as the gold standard procedure for the early detection of colorectal cancer (CRC). Segmentation is valuable for two significant clinical applications, namely lesion detection and classification, providing means to improve accuracy and robustness. The manual segmentation of polyps in colonoscopy images is time-consuming. As a result, the use of deep learning (DL) for automation of polyp segmentation has become important. However, DL-based solutions can be vulnerable to overfitting and the resulting inability to generalise to images captured by different colonoscopes. Recent transformer-based architectures for semantic segmentation both achieve higher performance and generalise better than alternatives, however typically predict a segmentation map of $\frac{h}{4}\times\frac{w}{4}$ spatial dimensions for a $h\times w$ input image. To this end, we propose a new architecture for full-size segmentation which leverages the strengths of a transformer in extracting the most important features for segmentation in a primary branch, while compensating for its limitations in full-size prediction with a secondary fully convolutional branch. The resulting features from both branches are then fused for final prediction of a $h\times w$ segmentation map. We demonstrate our method's state-of-the-art performance with respect to the mDice, mIoU, mPrecision, and mRecall metrics, on both the Kvasir-SEG and CVC-ClinicDB dataset benchmarks. Additionally, we train the model on each of these datasets and evaluate on the other to demonstrate its superior generalisation performance.
1 Introduction
Polyp segmentation can support colorectal cancer screening, but existing approaches face generalisation and full-resolution prediction challenges. FCBFormer combines transformer and fully convolutional branches to produce full-size segmentation while achieving state-of-the-art results across benchmark and cross-dataset evaluations.
- Motivation: Colorectal cancer is a leading cause of cancer mortality, and polyp segmentation can support lesion detection and classification during screening.Colonoscopy enables detection and treatment in the same procedure, but 17%-28% of polyps may be missed.
- Existing approaches: Deep learning methods automate polyp segmentation, while transformer architectures have recently shown superior semantic-segmentation performance over traditional FCNs.These methods are intended to assist clinicians performing colonoscopy procedures.
- Proposed architecture: FCBFormer combines parallel fully convolutional and transformer branches, fusing their outputs through a prediction head for a full-size segmentation map.The fully convolutional branch returns h×w feature maps, while the transformer branch returns reduced-size maps that are upsampled before fusion.
- Proposed architecture: The transformer branch uses a pre-trained PVTv2 encoder, while the fully convolutional branch extracts features for converting transformer outputs into full-size predictions.The design assigns complementary roles to the two branches before prediction-head processing.
- Architectural contributions: FCBFormer updates SSFormer’s decoder with improved local emphasis and stepwise feature aggregation, while using residual-block-based convolutional components and dense skip connections.The prediction head uses residual blocks and a final 1×1 convolutional pixel-wise prediction layer.
- Evaluation scope: The method achieves state-of-the-art mDice, mIoU, mPrecision, and mRecall performance on Kvasir-SEG and CVC-ClinicDB, including cross-dataset generalisability tests.The paper evaluates training on each dataset and testing on the other.
2 FCBFormer
FCBFormer’s transformer branch uses a PVTv2 encoder and an improved progressive locality decoder, while its fully convolutional branch extracts full-size multi-scale features. A prediction head fuses the upsampled transformer output with convolutional features to produce the final segmentation map.
- Transformer branch (TB): PVTv2-B3 provides a four-level feature pyramid for the transformer branch and was pretrained on ImageNet.The encoder uses overlapping patch embeddings, zero-padding position information, and linear spatial reduction attention.
- Improved progressive locality decoder (PLD+): The improved PLD+ replaces SSFormer’s locality and aggregation components with deeper residual blocks using group normalisation, SiLU activations, convolutions, and residual connections.The modification targets limitations identified in the original LE and SFA modules.
- Fully convolutional branch (FCB): The fully convolutional branch combines residual blocks, strided convolutions, nearest-neighbour upsampling, and dense U-Net-style skip connections.These components extract fused multi-scale features at full spatial size.
- Prediction head (PH): The prediction head concatenates the upsampled transformer output with the FCB output before producing a pixel-wise segmentation prediction.Each prediction-head layer returns 64 channels, except the final single-channel prediction layer.
3 Experiments
Experiments evaluate FCBFormer on two polyp-segmentation datasets, compare it with established architectures, test cross-dataset generalisation, and isolate contributions from FCB and PLD+.
- Datasets and implementation: Experiments use Kvasir-SEG and CVC-ClinicDB, containing 1000 and 612 ground-truth input-target pairs, respectively.Kvasir-SEG samples vary spatially, whereas CVC-ClinicDB samples are 288 × 384.
- Datasets and implementation: Models are trained for binary h × w segmentation after resizing RGB images and masks to h, w = 352.The training setup uses an 80%/10%/10% random train/validation/test split.
- Primary assessment: FCBFormer is compared with U-Net, ResUNet, ResUNet++, PraNet, and MSRF-Net under the same training and evaluation basis.SSFormer is excluded from the main comparison because no official codebase was available and it does not predict full-size maps.
- Primary assessment: FCBFormer predictions are generally more consistent with targets and better represent boundaries in challenging polyp morphologies than existing models.The authors associate this strength with combining transformer and fully convolutional branches, with FCB supporting reliable full-size boundaries.
- Primary assessment: FCBFormer outperformed existing models on mDice, mIoU, mPrecision, and mRecall in the primary assessments.The evaluation averages each metric over the test set.
- Evaluation considerations: Previously proposed methods sometimes perform worse than originally reported, potentially because implementations were optimised for 256 × 256 rather than 352 × 352 inputs.Retraining MSRF-Net at 256 × 256 produced results similar to those in its original paper.
- Generalisability tests: Cross-dataset tests show FCBFormer significantly outperforming existing models on most metrics when evaluated on a distribution different from training.The authors report similar superiority over other models in externally reported generalisability comparisons.
- Ablation study: Ablations indicate significant benefits from FCB and generally additional benefits from PLD+ across primary and generalisability experiments.SSFormer-I+FCB generally outperforms SSFormer-I, while FCBFormer generally outperforms SSFormer-I+FCB.
4 Conclusion
FCBFormer combines transformer and fully convolutional networks for polyp segmentation, achieving reported state-of-the-art performance and strengths in generalisability and challenging morphology. The architecture also motivates further study of neural-network designs for dense prediction.
- FCBFormer is a novel polyp-segmentation architecture that combines transformer and fully convolutional networks for dense prediction.
- The experiments report state-of-the-art performance, outperforming existing models across several popular metrics while showing strengths in generalisability and challenging polyp morphology.
- The method runs transformer and FCN branches in parallel, concatenating their outputs for processing by a prediction head.
- The authors present the architecture’s strengths as evidence that dense-prediction networks and the basis of their performance require further understanding.