Source-linked AI summary
TGANet: Text-guided attention for improved polyp segmentation
Nikhil Kumar Tomar, Debesh Jha, Ulas Bagci, Sharib Ali
TL;DR
Polyp segmentation is challenged by operator-dependent colonoscopy, missed lesions, and variable polyp size and occurrence. TGANet addresses these issues with text-guided attention and auxiliary classification of polyp attributes, and experiments across four datasets report improved segmentation performance, including size- and number-based cases.
Problem
Operator dependence, missed polyps, and variable polyp size can limit automated segmentation methods that do not adequately represent size and occurrence attributes.
Method
TGANet uses auxiliary classification to weight text embeddings representing polyp number and size, alongside feature enhancement and multi-scale feature aggregation.
Results
TGANet outperformed state-of-the-art methods across four polyp datasets, with reported improvements on standard, cross-dataset, and size- or number-based evaluations.
Takeaways & Limitations
The authors report that text-guided attention improves segmentation for variable-sized polyps, including clinically important flat and sessile polyps.
Abstract
from arXiv · showhide
Colonoscopy is a gold standard procedure but is highly operator-dependent. Automated polyp segmentation, a precancerous precursor, can minimize missed rates and timely treatment of colon cancer at an early stage. Even though there are deep learning methods developed for this task, variability in polyp size can impact model training, thereby limiting it to the size attribute of the majority of samples in the training dataset that may provide sub-optimal results to differently sized polyps. In this work, we exploit size-related and polyp number-related features in the form of text attention during training. We introduce an auxiliary classification task to weight the text-based embedding that allows network to learn additional feature representations that can distinctly adapt to differently sized polyps and can adapt to cases with multiple polyps. Our experimental results demonstrate that these added text embeddings improve the overall performance of the model compared to state-of-the-art segmentation methods. We explore four different datasets and provide insights for size-specific improvements. Our proposed text-guided attention network (TGANet) can generalize well to variable-sized polyps in different datasets.
1 Introduction
Polyp segmentation is motivated by operator-dependent colonoscopy and substantial missed rates, while existing methods often rely mainly on localized spatial context. TGANet adds text-guided attention based on polyp size and occurrence to improve segmentation across variable cases.
- Missed polyp rates reach 26.8% in the right colon, 21.4% in the left colon, and nearly 32.7% for flat or sessile polyps.
- Existing segmentation methods often exploit localized spatial context, although polyp appearances can be confused with colonic folds.
- The proposed text-guided attention uses polyp number and size attributes to learn features for one or many polyps and small, medium, or large polyps.
- TGANet was evaluated on four publicly available polyp datasets against five state-of-the-art medical image segmentation methods.
2 Method
TGANet combines a ResNet50 encoder with auxiliary classification of polyp number and size, feature enhancement, label attention, decoder refinement, and multi-scale aggregation. These components use learned text-based attributes to modulate decoder features and produce the segmentation map.
- TGANet uses a pre-trained ResNet50 encoder whose four blocks support both auxiliary attribute classification and the main segmentation task.
- The auxiliary classifiers predict whether images contain one or many polyps and whether polyps are small, medium, or large.
- Feature enhancement modules strengthen encoder features using parallel dilated convolutions and channel attention.
- Label attention: Label attention fuses attribute probabilities with byte-pair-encoded text embeddings to provide soft channel-wise attention to decoder outputs.
- Decoder outputs are refined with label features, while multi-scale feature aggregation fuses decoder representations before predicting the segmentation mask.
- The training objective jointly minimizes two auxiliary cross-entropy losses and segmentation binary-cross-entropy and Dice losses with equal weights.
3 Experiments and results
TGANet is evaluated on four public polyp-segmentation datasets using standard metrics and comparisons with five established methods. It achieves strong overall and cross-dataset performance, including improvements for varied polyp sizes and counts, while ablation results support its text-guided attention component.
- Experimental setup: TGANet was evaluated on Kvasir-SEG, CVC-ClinicDB, BKAI, and Kvasir-Sessile using mIoU, mDSC, recall, precision, F2score, and FPS.The experiments compared TGANet with UNet, HarDNet-MSEG, ColonSegNet, DeepLabv3+, and PraNet.
- Dataset results: 0.8330 mIoU and 0.8982 mDSC were achieved on Kvasir-SEG, exceeding PraNet by 1.58% and 1.45%, respectively.Table 1 reports TGANet as outperforming all compared state-of-the-art methods on this dataset.
- Dataset results: 0.8990 mIoU and 0.9457 mDSC were achieved on CVC-ClinicDB, the highest values among the compared methods.The reported gains over DeepLabV3+ were 0.17% in mIoU and 0.66% in mDSC.
- Dataset results: TGANet obtained 0.8409 mIoU and 0.9023 mDSC on BKAI, outperforming DeepLabV3+ by 0.95% and 0.86%, respectively.
- Dataset results: On Kvasir-Sessile, TGANet surpassed all other methods across evaluation metrics, exceeding PraNet by 2.39% in mIoU and 2.44% in mDSC.This dataset contains clinically relevant flat and sessile polyps, including polyps smaller than 10 mm.
- Generalization and targeted cases: Cross-dataset testing improved over all compared methods, while size- and count-based evaluation showed nearly 1–2% gains for small, medium, and many-polyp cases.The model was trained on Kvasir-SEG and tested on CVC-ClinicDB; qualitative results also showed improved handling of varied sizes and multiple polyps.
- Ablation study: 2.26% mIoU and 1.96% mDSC improvements over the baseline without the label and classifier were reported in the ablation study.The authors attribute the boost to introducing text-guided attention along with the label.
4 Conclusion
TGANet combines text-guided attention, auxiliary size-and-number representations, feature enhancement, and multi-scale decoder fusion to address variable polyp attributes. Experiments show improved segmentation, particularly for clinically important flat and sessile polyps.
- TGANet uses an auxiliary task to learn size- and number-based feature representations as label attentions in decoder blocks.The model targets polyps that vary in size and occur singly or multiply.
- Multiple feature enhancement modules connect encoder blocks to strengthen representations passed toward decoding.
- Multi-scale decoder fusion helps the network handle changes in polyp size and number.
- TGANet provided higher segmentation performance on clinically important flat and sessile polyps.
Supplementary material
The supplementary material documents the datasets, multi-scale fusion design, and complexity measures used in the experiments.
- Datasets: The experiments use polyp datasets with reported image counts, input sizes, and availability; BKAI data were split 80:10:10 when test data were unavailable.Kvasir-SEG includes small, flat, and sessile polyps.
- Multi-scale fusion: The multi-scale fusion module concatenates features from three decoder blocks using upsampling, convolution, batch normalization, and ReLU activations.Residual connections are also applied across layers.
- Complexity analysis: The complexity comparison reports model parameters, FLOPs, image size, and frames per second for TGANet and SOTA methods.