PaperlayerView exact revision on arXiv

Source-linked AI summary

Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers

Bo Dong, Wenhai Wang, Deng-Ping Fan, Jinpeng Li, Huazhu Fu, Ling Shao

arXiv:2108.06932v8eess.IVcs.CV

TL;DR

Polyp segmentation methods commonly use CNN backbones but face challenges in exchanging and fusing features across levels. Polyp-PVT replaces the backbone with a pyramid vision transformer and adds three feature-processing modules, achieving stronger benchmark performance while retaining failure cases under overlapping illumination and reflection.

  • Problem

    CNN-based polyp segmentation methods struggle to account for different-level feature contributions and to fuse those features effectively.

  • Method

    Polyp-PVT uses a transformer encoder with CFM, CIM, and SAM to extract robust representations and fuse high- and low-level polyp features.

  • Results

    Polyp-PVT consistently outperforms current cutting-edge models on five challenging datasets without pre- or post-processing.

  • Takeaways & Limitations

    The framework improves robustness to challenging polyp appearances and acquisition conditions, including camouflage, small objects, and rotation.

  • Takeaways & Limitations

    Polyp-PVT can misidentify overlapping light and shadow or salient reflective points as polyp regions.

Abstract

from arXiv · show

Most polyp segmentation methods use CNNs as their backbone, leading to two key issues when exchanging information between the encoder and decoder: 1) taking into account the differences in contribution between different-level features and 2) designing an effective mechanism for fusing these features. Unlike existing CNN-based methods, we adopt a transformer encoder, which learns more powerful and robust representations. In addition, considering the image acquisition influence and elusive properties of polyps, we introduce three standard modules, including a cascaded fusion module (CFM), a camouflage identification module (CIM), and a similarity aggregation module (SAM). Among these, the CFM is used to collect the semantic and location information of polyps from high-level features; the CIM is applied to capture polyp information disguised in low-level features, and the SAM extends the pixel features of the polyp area with high-level semantic position information to the entire polyp area, thereby effectively fusing cross-level features. The proposed model, named Polyp-PVT, effectively suppresses noises in the features and significantly improves their expressive capabilities. Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations (e.g., appearance changes, small objects, rotation) than existing representative methods. The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.

I. INTRODUCTION

Polyp segmentation remains difficult because conventional methods struggle with low-quality features, poor generalization, image noise, camouflage, and boundary localization. Polyp-PVT addresses these challenges with a transformer-based framework and three modules for cross-level feature processing.

  • Traditional polyp segmentation methods rely on low-level texture, geometric, or superpixel features, but produce low-quality results and generalize poorly.
  • Deep learning methods improve accuracy and generalization, yet boundary localization remains challenging because acquisition noise and camouflage obscure polyp edges.Acquisition noise includes motion blur and reflector problems, while camouflage arises from similar color and texture between polyps and surrounding tissue.
  • Polyp-PVT uses a pyramid vision transformer encoder to extract more robust features than CNN-based backbones.
  • CFM progressively integrates high-level features to collect semantic and location information, while CIM captures polyp cues disguised in low-level features.
  • SAM uses non-local and convolutional graph layers to combine local pixels and global semantic cues from polyp regions.
  • On ColonDB and ETIS, Polyp-PVT achieves mean Dice scores of 0.808 and 0.787, respectively, exceeding SANet by 5.5% and 3.7%.

B. Vision Transformer

The method uses a pyramid vision transformer encoder and dedicated modules to process multi-scale features for polyp segmentation. Its architecture combines high-level semantic fusion, low-level noise filtering, and cross-level feature integration before prediction.

  • Transformers model long-term dependencies with multi-head self-attention, whose dynamic weights and global receptive field differ from convolutional processing.
  • Pyramid transformer backbones use hierarchical stages and can match CNN versatility while improving detection and segmentation performance.
  • Polyp-PVT contains a PVT encoder, CFM, CIM, and SAM for multi-scale extraction, high-level aggregation, low-level enhancement, and feature fusion.
  • The encoder extracts four pyramid features, while CFM processes high-level features and CIM converts low-level features into complementary representations.
  • SAM aligns and fuses the CFM and CIM outputs into a final feature map that is passed to a convolutional layer for segmentation prediction.

B. Transformer Encoder

The transformer encoder is selected for robustness to acquisition disturbances and adapted to dense polyp segmentation with multi-scale outputs. The CFM progressively fuses high-level features through cascaded upsampling, multiplication, concatenation, and convolution.

  • B. Transformer Encoder: Polyp images contain acquisition noise such as motion blur, rotation, and reflection, motivating a vision transformer backbone for robust feature extraction.
  • B. Transformer Encoder: PVT uses a pyramid architecture with spatial-reduction attention to reduce resource consumption, and Polyp-PVT adopts the stronger PVTv2 backbone.
  • C. Cascaded Fusion Module: The CFM is implemented as a cascaded fusion module to balance segmentation accuracy and computational resources.
  • C. Cascaded Fusion Module: In its first part, CFM upsamples the highest-level feature, processes it with convolutional units, multiplies it with another high-level feature, and concatenates the result.
  • C. Cascaded Fusion Module: A subsequent convolutional unit smooths the concatenated feature to produce an intermediate fused map, and later convolutions reduce its dimension to T1.
  • C. Cascaded Fusion Module: The fusion equations use the Hadamard product for element-wise multiplication and concatenation along the channel dimension.

D. Camouflage Identification Module

The camouflage identification module captures polyp details from low-level features when polyps resemble their background. It uses channel and spatial attention to emphasize relevant information and reduce incorrect low-level signals.

  • Polyps can closely resemble the background, making low-level texture, color, and edge features difficult to identify reliably.
  • CIM captures camouflaged polyp details from different dimensions of the low-level feature map X1.
  • The attention mechanism focuses on potential polyps and reduces incorrect information in lower-level features.
  • Channel attention uses adaptive maximum and average pooling with Softmax weighting to recalibrate the input tensor.
  • Spatial attention complements channel attention by using channel-wise maximum and average responses followed by a 7 × 7 convolution.

E. Similarity Aggregation Module

SAM aggregates low-level local features and high-level semantic cues through non-local attention in a graph-convolution domain. It reconstructs the resulting graph features and combines them with the original high-level feature map.

  • SAM explores high-order relations between CIM local features and CFM high-level cues using non-local operation under graph convolution.
  • SAM injects detailed appearance features into high-level semantic features through global attention.
  • The module maps high-level feature T1 into Q and K, while processing detailed feature T2 into an aligned attention representation.
  • Hadamard weighting assigns different pixel weights and increases the weight of edge pixels before pooling and cropping produce V.
  • An inner product establishes correlations between pixels in V and K, producing the correlation attention map f.
  • The attention-weighted Q features pass through GCN, are projected to match Y, and combine with T1 to form SAM output Z.

F. Loss Function

The loss function supervises both the final and intermediate segmentation outputs. Weighted IoU and BCE losses constrain global structure and local details while emphasizing hard pixels.

  • The loss function combines main and auxiliary losses to supervise the final result P2 and intermediate CFM result P1 against ground truth G.
  • The main loss Lmain is computed between final segmentation result P2 and ground truth G.
  • The auxiliary loss Laux supervises intermediate result P1 generated by CFM against ground truth G.
  • Weighted IoU and weighted BCE losses constrain prediction maps from global object-level and local pixel-level perspectives.
  • Weighted BCE assigns higher weights to hard pixels, while weighted IoU also pays more attention to hard pixels.

G. Implementation Details

Polyp-PVT is trained with multi-scale inputs and AdamW under fixed image, batch, epoch, learning-rate, and weight-decay settings. Performance is evaluated with six metrics covering regional, pixel-level, structural, and image-level properties.

  • Implementation Details: Training uses multi-scale inputs, AdamW optimization, learning rate 1e-4, weight decay 1e-4, 352 × 352 images, batch size 16, and 100 epochs.
  • Evaluation Metrics: The evaluation uses Dice, IoU, MAE, weighted F-measure, S-measure, and E-measure.
  • Evaluation Metrics: Mean Dice and mean IoU measure regional similarity and focus on internal consistency of segmented objects.
  • Evaluation Metrics: MAE measures the average pixel-by-pixel absolute error between predictions and ground truth.
  • Evaluation Metrics: Weighted F-measure combines recall and precision while reducing the effect of treating every pixel equally.
  • Evaluation Metrics: S-measure evaluates structural similarity at region and object levels, while E-measure evaluates segmentation at pixel and image levels.

B. Datasets and Compared Models

The evaluation uses five public datasets and compares Polyp-PVT with nine open-source polyp-segmentation models. Experiments assess learning ability, generalization to unseen multicentric data, visual robustness, and FROC performance.

  • Datasets: Five challenging public datasets—Kvasir-SEG, ClinicDB, ColonDB, Endoscene, and ETIS—are used to evaluate the framework.
  • Compared Models: Nine open-source models, including U-Net, UNet++, PraNet, SFA, MSEG, ACSNet, DCRNet, EU-Net, and SANet, provide comparison baselines.
  • Learning Ability: ClinicDB and Kvasir-SEG evaluate learning ability using fixed training and test splits, with 900 and 548 training images respectively.
  • Learning Ability: On ClinicDB, Polyp-PVT’s mDic is 2.1% higher than SANet and 3.8% higher than PraNet.
  • Generalization Ability: On unseen multicentric datasets, Polyp-PVT exceeds SANet by 5.5% on ColonDB, 3.7% on ETIS, and 1.2% on EndoScene.
  • Visualization Results: Visualizations show stable recognition and segmentation across lighting, contrast, reflection, motion blur, small objects, and rotation, with edges closer to ground truth.
  • FROC Evaluation: On ColonDB, the proposed model achieves the top FROC result among the compared methods.

F. Ablation Study

The ablation study evaluates the complete Polyp-PVT against variants that remove or replace its components. Results show that CFM, CIM, and SAM each contribute to segmentation performance, noise suppression, or rotation robustness.

  • Setup: The complete model is defined as Polyp-PVT with the PVT encoder, CFM, CIM, and SAM, and is compared against component variants.
  • Effectiveness of CFM: Removing CFM sharply reduces performance across all five datasets, including a ClinicDB mDic drop from 0.937 to 0.915.
  • Effectiveness of CIM: Removing CIM decreases Endoscene mDic by 1.8% and introduces significant noise in the resulting features.
  • Effectiveness of CIM: CIM’s channel and spatial attention mechanisms distinguish polyp details and edges from redundant low-level information.
  • Effectiveness of SAM: SAM improves ColonDB performance by 2.9% in mDic and 3.1% in mIoU compared with its ablated variant.
  • Effectiveness of SAM: GCN provides a larger receptive field and stronger rotation adaptability than convolution within SAM under 15-degree rotation.

G. Video Polyp Segmentation

Polyp-PVT is evaluated on three video polyp-segmentation benchmarks against six competing approaches. It outperforms the strongest comparison model on two reported benchmarks, while failure cases remain under challenging illumination and reflection.

  • Experimental Setup: Experiments compare Polyp-PVT with six approaches on CVC-300-TV, CVC-612-T, and CVC-612-V using matched training and testing data.
  • Results: Polyp-PVT exceeds PNS-Net by 3.1% mDice on CVC-612-V and 6.7% mDice on CVC-300-TV.
  • Failure Cases: Failure cases include inaccurate boundaries where light and shadow overlap, plus reflective points incorrectly predicted as polyps.

V. CONCLUSION

Polyp-PVT combines a pyramid vision transformer encoder with CFM, CIM, and SAM to extract robust features and fuse high- and low-level cues. Extensive experiments report consistent improvements across challenging image datasets and video segmentation.

  • Conclusion: Polyp-PVT uses a pyramid vision transformer backbone to extract more powerful and robust features for polyp segmentation.
  • Conclusion: CFM, CIM, and SAM separately extract high- and low-level cues and fuse them for the final segmentation output.

Nordic Machine Intelligence, vol. 10, pp. 11–13, 2021

The described medical-image segmentation pipeline uses transformer-based models, including Polyp-PVT, with multi-model fusion and shared preprocessing and post-processing. Five-fold training, ensemble voting, and morphological cleanup are used to stabilize predictions, while evaluations and failure cases are reported.

  • Motivation: Automatic real-time polyp segmentation is motivated by a 6%–27% missed-diagnosis rate during colonoscopy.The stated clinical aim is to help clinicians identify missed lesions and prevent further colorectal-cancer development.
  • Model: The pipeline integrates Polyp-PVT, Sinv2-PVT, and Transfuse-PVT for model fusion across segmentation tasks.Polyp-PVT uses a pyramid vision transformer encoder, while the other models are incorporated into the same fused method.
  • Results: The reported instrument-segmentation table uses IoU, Dice, and pixel accuracy, while the discussion reports IoU results above 0.92 across the three models.The discussion also notes that Polyp-PVT, Sinv2-PVT, and Transfuse-PVT have the same performance and relatively stable five-fold results.
  • Training and inference: Training uses five-fold cross-validation, retaining the weights with the best validation IoU for each model and producing 15 model weights overall.Each of the three models is trained with four folds and validated on the remaining fold.
  • Training and inference: Inference resizes inputs to 352 × 352, upsamples outputs to the original feature size, and combines 15 predictions using minority voting.No data enhancement is used during inference.
  • Post-processing: Morphological opening removes isolated noise points, followed by area-based removal of small predicted blocks.The cleanup is applied to obtain the final prediction result after voting.
Loading 2108.06932v8…