Source-linked AI summary
Progressively Normalized Self-Attention Network for Video Polyp Segmentation
Ge-Peng Ji, Yu-Cheng Chou, Deng-Ping Fan, Geng Chen, Huazhu Fu, Debesh Jha, Ling Shao
TL;DR
Video polyp segmentation must handle low boundary contrast, large shape variation, and limited CNN exploitation of successive-frame information. PNS-Net uses normalized self-attention with progressive refinement to model temporal-spatial cues, achieving state-of-the-art performance at about 140fps. The authors conclude that its components are effective, while evaluation on larger VPS datasets remains future work.
Problem
Accurate and real-time video polyp segmentation is challenging because polyps have low boundary contrast and large shape variation, while CNN-based approaches do not fully exploit successive-frame temporal and spatial information.
Method
PNS-Net uses a normalized self-attention block with query-dependent spatial-temporal processing and stacked progressive residual learning.
Results
PNS-Net achieves the best performance across publicly available datasets under six metrics while running at approximately 140fps.
Takeaways & Limitations
The normalized self-attention blocks can be plugged into existing CNN-based architectures, and the reported core components are effective for video polyp segmentation.
Takeaways & Limitations
Performance on a larger VPS dataset remains an open direction for future work.
Abstract
from arXiv · showhide
Existing video polyp segmentation (VPS) models typically employ convolutional neural networks (CNNs) to extract features. However, due to their limited receptive fields, CNNs can not fully exploit the global temporal and spatial information in successive video frames, resulting in false-positive segmentation results. In this paper, we propose the novel PNS-Net (Progressively Normalized Self-attention Network), which can efficiently learn representations from polyp videos with real-time speed (~140fps) on a single RTX 2080 GPU and no post-processing. Our PNS-Net is based solely on a basic normalized self-attention block, equipping with recurrence and CNNs entirely. Experiments on challenging VPS datasets demonstrate that the proposed PNS-Net achieves state-of-the-art performance. We also conduct extensive experiments to study the effectiveness of the channel split, soft-attention, and progressive learning strategy. We find that our PNS-Net works well under different settings, making it a promising solution to the VPS task.
1 Introduction
PNS-Net addresses video polyp segmentation by replacing conventional CNN-only processing with self-attention designed to use temporal and spatial cues. The framework is evaluated against established segmentation models and reports state-of-the-art performance at real-time speed.
- Motivation: Accurate and real-time polyp segmentation is difficult because polyps have low boundary contrast and large shape variation.Automatic segmentation is motivated by colonoscopy’s dependence on physician experience and high polyp miss rate.
- Proposed framework: PNS-Net introduces a self-attention framework for video polyp segmentation, providing an alternative to existing CNN-based models.The paper positions self-attention as a new perspective for the VPS task.
- Proposed framework: The normalized self-attention block is designed to exploit temporal and spatial cues and can be embedded into CNN-based encoder-decoder architectures.The block is described as flexible, backbone-free, and efficient.
- Evaluation: PNS-Net achieves state-of-the-art performance with real-time speed when compared with classical and cutting-edge polyp segmentation models.The evaluated baselines include UNet, UNet++, ResUNet, ACSNet, and PraNet.
2 Method
PNS-Net combines CNN-based spatial encoding with progressively stacked normalized self-attention to model spatial-temporal relationships in video polyp segmentation. Its NS block uses channel splitting, query-dependent constrained neighborhoods, normalization, soft-attention, aggregation, and residual refinement before decoding predictions.
- Channel Split Rule: The NS block generates query, key, and value features with 1×1×1 convolutions, then splits each into N channel groups for multi-group attention.Each group contains Qi, Ki, and Vi features with C/N channels.
- Query-Dependent Rule: Query-dependent relevance blocks measure spatial-temporal relationships within constrained neighborhoods rather than across all positions.The neighborhood varies with kernel size k, dilation rate d_i, and the number of frames T.
- Normalization Rule: Layer normalization along the temporal dimension maintains a fixed query-feature distribution before affinity computation.The normalization is introduced to address internal covariate shift in the feed-forward processing of Qi.
- Soft-Attention: Soft-attention synthesizes affinity matrices and aggregated features by enhancing relevant spatial-temporal patterns and suppressing less relevant ones.Affinity matrices are concatenated across channels to produce a soft-attention map, which is applied channel-wise after feature aggregation.
- Encoder and Decoder: The encoder extracts low- and high-level spatial features, while a two-stage UNet-like decoder combines decoded low-level and refined spatial-temporal features to produce frame predictions.The encoder uses Res2Net-50 and an RFB-like module reduces feature channels to limit computational burden.
- Progressively Normalized Self-attention: PNS-Net reshapes high-level features from consecutive frames into a temporal feature and progressively refines them with stacked normalized self-attention blocks.The refinement uses R stacked blocks and a residual connection so the block can be plugged into pretrained networks.
3 Experiments
PNS-Net is evaluated on multiple video polyp segmentation datasets using standardized baselines, metrics, qualitative comparisons, and ablation studies. It achieves strong performance across datasets, while channel-split experiments show that receptive-field scale affects temporal-motion modeling.
- Datasets and training: The experiments use Kvasir, CVC-300, CVC-612, and ASU-Mayo, with a two-stage training strategy that pre-trains the static backbone before video segmentation.Kvasir supplies large-scale image data, while the video datasets provide training, validation, and test clips.
- Evaluation protocol: Five contemporary segmentation baselines are retrained under the same data and default settings, and evaluation uses six metrics covering overlap, specificity, structure, alignment, and pixel error.The metrics include maxDice, maxSpe, maxIoU, S-measure, enhanced-alignment measure, and mean absolute error.
- Quantitative comparison: PNS-Net consistently outperforms other state-of-the-art methods on both CVC-612-V and CVC-612-T.The paper also reports qualitative success across difficult polyp sizes, homogeneous regions, and textures.
- Ablation studies: Using four channel splits performs best across all metrics on CVC-300-TV, whereas overly large or small receptive fields weaken temporal-information capture.The ablation attributes this pattern to the balance between global context and local motion, including multi-scale polyp movement.
4 Conclusion
The conclusion presents PNS-Net as a fast self-attention framework for colonoscopy-video polyp segmentation, with plug-in normalized self-attention blocks and strong performance across available datasets. The authors identify evaluation on larger VPS datasets as future work.
- Main conclusion: PNS-Net segments polyps from colonoscopy videos at approximately 140fps while achieving the best performance across publicly available datasets under six metrics.The conclusion combines the reported speed with the paper-wide evaluation outcome.
- Main conclusion: Normalized self-attention blocks can be plugged into existing CNN-based architectures, and ablations support the effectiveness of PNS-Net’s core components.The conclusion describes the blocks as basic and reusable within existing architectures.
- Future work: Evaluation on a larger VPS dataset remains future work, leaving the method’s performance on broader video data unexplored.This is the explicit scope boundary identified by the authors.