PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Polyp Segmentation: A Deep Learning Perspective

Ge-Peng Ji, Guobao Xiao, Yu-Cheng Chou, Deng-Ping Fan, Kai Zhao, Geng Chen, Luc Van Gool

arXiv:2203.14291v3eess.IVcs.CV

TL;DR

Video polyp segmentation lacks large-scale, fine-grained annotations and is challenged by dynamic colonoscopy imagery. The paper introduces SUN-SEG, proposes PNS+, and benchmarks 13 approaches; PNS+ achieves the strongest reported performance, while difficult attributes remain unresolved.

  • Problem

    VPS lacks large-scale, densely annotated video data and a community-agreed benchmark amid dynamic colonoscopy conditions.

  • Method

    The paper introduces SUN-SEG and PNS+, using diversified frame annotations and global/local encoders with normalized self-attention to model long- and short-term cues.

  • Results

    PNS+ achieves the best performance against the evaluated competitors, including Dice 0.756 on SUN-SEG-Easy (Unseen).

  • Takeaways & Limitations

    The benchmark provides a basis for VPS comparison and highlights room for further exploration.

  • Takeaways & Limitations

    Existing models, including PNS+, remain insufficiently robust for appearance changes, artifacts, small or camouflaged polyps, and some occlusion settings.

Abstract

from arXiv · show

We present the first comprehensive video polyp segmentation (VPS) study in the deep learning era. Over the years, developments in VPS are not moving forward with ease due to the lack of large-scale fine-grained segmentation annotations. To address this issue, we first introduce a high-quality frame-by-frame annotated VPS dataset, named SUN-SEG, which contains 158,690 colonoscopy frames from the well-known SUN-database. We provide additional annotations with diverse types, i.e., attribute, object mask, boundary, scribble, and polygon. Second, we design a simple but efficient baseline, dubbed PNS+, consisting of a global encoder, a local encoder, and normalized self-attention (NS) blocks. The global and local encoders receive an anchor frame and multiple successive frames to extract long-term and short-term spatial-temporal representations, which are then progressively updated by two NS blocks. Extensive experiments show that PNS+ achieves the best performance and real-time inference speed (170fps), making it a promising solution for the VPS task. Third, we extensively evaluate 13 representative polyp/object segmentation models on our SUN-SEG dataset and provide attribute-based comparisons. Finally, we discuss several open issues and suggest possible research directions for the VPS community.

1 Introduction

The paper frames video polyp segmentation as important for earlier colorectal-cancer intervention but constrained by limited dense annotations and dynamic colonoscopy conditions. It addresses these gaps with SUN-SEG, PNS+, and a broad benchmark of existing methods.

  • Colorectal-cancer survival exceeds 95% at stage one but falls below 35% at stages four and five, making early screening important.
  • Physician-dependent polyp removal has a reported missing rate of 22% ∼28%.
  • VPS development is constrained by limited densely annotated video data, absent community benchmarks, and dynamic imaging conditions.The cited conditions include varied polyp appearance, artifacts, and imaging degradation.
  • SUN-SEG contains 158,690 frames with attribute, object-mask, boundary, scribble, and polygon labels, while PNS+ combines global/local encoders with normalized self-attention blocks.The encoders model long- and short-term spatial-temporal representations from anchor and successive frames.
  • The benchmark evaluates 13 image- and video-based segmentation approaches and concludes that VPS remains insufficiently addressed.

2 Related Works

Related work spans colonoscopy datasets and image/video polyp-segmentation methods. Existing efforts progress from classification and image segmentation toward temporal modeling, while annotation density and temporal context remain important distinctions.

  • 2.1 Colonoscopy-related datasets: Prior colonoscopy datasets support classification, detection, localization, and segmentation, with varying image, video, and annotation coverage.Table 1 summarizes 20 human-colonoscopy datasets using image, video, dense-label, class-label, bounding-box, and pixel-mask statistics.
  • 2.1 Colonoscopy-related datasets: Earlier video datasets commonly use sparse or sampled annotations, whereas SUN-database provides a large densely annotated detection resource.Examples include ETIS-Larib, EDD2020, PICCOLO, and SUN-database.
  • 2.2 Polyp-segmentation approaches: Hand-crafted methods often struggle with heterogeneous polyps and hard mimics, motivating data-driven AI approaches.
  • 2.2 Polyp-segmentation approaches: Image polyp-segmentation methods include CNN-based and transformer-based approaches, with boundary-aware and uncertainty-based strategies for difficult regions.
  • 2.2 Polyp-segmentation approaches: Existing image-segmentation methods overlook temporal cues, while hybrid 2/3D CNNs aggregate consecutive-frame information but face restricted spatial correlation.

3 VPS Dataset

SUN-SEG is constructed from colonoscopy videos with dense, quality-controlled, multi-level annotations and statistics designed to characterize challenging VPS data. Its labels span clinical attributes, precise masks and boundaries, and weaker annotation forms.

  • Data collection and reorganization: SUN-SEG reorganizes SUN-database videos into consecutive positive and negative clips of roughly 3∼11 seconds at 30 fps.The source contains 113 videos, including 49,136 polyp frames and 109,554 non-polyp frames before reorganization.
  • Professional annotations: Ten experienced annotators create labels and three colonoscopy researchers re-verify their quality and correctness.The pipeline includes restricted quality controls for rejected and passed samples.
  • Professional annotations: The dataset provides five annotation hierarchies: visual attribute, object mask, boundary, scribble, and polygon.Object masks are pixel-wise, boundaries derive from masks, and scribble and polygon labels provide weaker supervision.
  • Professional annotations: SUN-SEG includes ten video-level visual attributes alongside frame-level localization and segmentation annotations.
  • Dataset statistics: SUN-SEG sub-datasets exhibit lower center bias than CVC-300 and CVC-612 and are analyzed through polyp-size, contrast, and distribution statistics.

4 VPS Baseline

PNS+ is presented as a VPS baseline built around normalized self-attention and a global-to-local pipeline. The design is intended to model spatial-temporal dependencies across the anchor frame and successive frames.

  • The method organizes VPS modeling around normalized self-attention and global-to-local learning strategies.The section covers the normalized self-attention block, global-to-local strategy, and implementation details.
  • PNS+ uses a global encoder, a local encoder, and normalized self-attention blocks within its proposed network pipeline.

4.1 Task Formulation

Video polyp segmentation is formulated as binary-class video object segmentation that assigns each pixel a polyp probability between 0 and 1.

  • VPS identifies polyp and non-polyp areas in colonoscopy video frames.
  • The model produces a non-binary probability mask for every pixel in each frame.
  • Video polyp detection is left for future exploration.

4.2 Normalized Self-attention Block

The normalized self-attention block reduces the cost of modeling spatial-temporal relationships by splitting channels, sampling query-dependent neighborhoods, and normalizing query features.

  • Naive self-attention is computationally expensive and performs unsatisfactorily for polyps captured at varied scales, angles, and speeds.
  • Channel Split Rule: Channel splitting divides query, key, and value features into N groups along the channel dimension.The attention features are generated using 1×1×1 convolutional embeddings before splitting.
  • Query-Dependent Rule: Query-dependent relevance blocks compute spatial-temporal affinity within constrained neighborhoods rather than across all key positions.The neighborhood is sampled around each query pixel using a point sampling function.
  • Query-Dependent Rule: The constrained neighborhood varies with kernel size k, dilation rate d_i, and the number of frames T.
  • Normalization Rule: Layer normalization along the temporal dimension maintains a fixed distribution for query features.The normalized query is denoted ˆQ_i = Norm(Q_i).
  • Soft attention enhances relevant spatial-temporal patterns while suppressing less relevant ones during feature synthesis.Affinity matrices and aggregated features are concatenated before computing the soft-attention map.

4.3 Global-to-Local Learning

PNS+ addresses the short temporal range of normalized self-attention with a global-to-local design that models long- and short-term dependencies using anchor and successive frames.

  • The original normalized self-attention mechanism struggles with long-term dependencies because limited resources restrict processing to a short frame span.
  • PNS+ combines a global encoder, local encoder, and two normalized self-attention blocks to propagate information across arbitrary temporal distances.
  • Global and Local Encoders: The global encoder uses the first frame as an anchor and relates it to sampled consecutive frames within a sliding window.The anchor serves as a global reference for long-term modeling.
  • Global Spatial-Temporal Modeling: The first normalized self-attention block uses the anchor feature as query and high-level short-term features as key and value to model global spatial-temporal dependencies.
  • Global Spatial-Temporal Modeling: Residual addition in the first block improves gradient-propagation stability and supports integration into pretrained networks.
  • Global-to-Local Propagation: The second normalized self-attention block propagates global dependencies into local frames, after which a decoder combines low-level local and spatial-temporal features for prediction.The decoder uses a two-stage UNet-alike design and training uses binary cross-entropy loss.

4.4 Implementation Details

The experiments train PNS+ on a 40% SUN-SEG split and test it on easy and hard seen/unseen subsets, using fixed hardware and an anchor-based inference procedure.

  • Data Splits: 40% of SUN-SEG forms the training set, containing 112 clips and 19,544 frames.
  • Data Splits: The remaining data form SUN-SEG-Easy with 119 clips and SUN-SEG-Hard with 54 clips, each divided into seen and unseen scenarios.
  • Training: Training uses ImageNet-pretrained Res2Net-50 weights, batch size 24, and 15 epochs requiring about five hours for convergence.
  • Inference: Inference samples five frames with the first frame as anchor and resizes inputs to 256×448.
  • Inference: 170fps is achieved on a single V100 GPU without heuristic post-processing.

5 VPS Benchmark

The benchmark evaluates 13 image- and video-based segmentation methods across seen and unseen colonoscopy scenarios using six complementary metrics. PNS+ performs strongly on learning ability, generalization, and challenging visual attributes.

  • Benchmark Setup: 13 segmentation methods are compared, including eight video-based and five image-based approaches, under a common dataset and convergence protocol.The benchmark includes COSNet, MAT, PCSA, 2/3D, AMD, DCF, FSNet, PNSNet, UNet, UNet++, ACSNet, PraNet, and SANet.
  • Evaluation Metrics: Six metrics evaluate prediction-ground-truth agreement, sensitivity, precision-recall balance, structural similarity, and enhanced alignment.The evaluation includes Dice, sensitivity, F-measure, weighted F-measure, structure measure, and enhanced-alignment measure.
  • Learning Ability: 0.888 Dice on SUN-SEG-Easy (Seen) exceeds PNSNet’s 0.861, while PNS+ reaches 0.929 enhanced-alignment on SUN-SEG-Hard (Seen) versus PNSNet’s 0.892.These seen-scenario results indicate stronger segmentation performance than the compared top video-based approach.
  • Generalization Capability: 0.756 Dice on SUN-SEG-Easy (Unseen) exceeds ACSNet’s 0.713 and 2/3D’s 0.722, while PNS+ reaches 0.653 weighted F-measure on SUN-SEG-Hard (Unseen).PNSNet drops substantially on the unseen datasets, whereas the authors attribute PNS+’s stronger generalizability to its global-to-local learning strategy.
  • Attribute-based Performance: PNS+ achieves Sα = 0.667 on the challenging IB attribute of SUN-SEG-Easy (Unseen), where most methods struggle with fuzzy polyp boundaries.Across the analyzed attributes, PNS+ consistently outperforms rivals on IB, GH, FM, and SV, while HO and LO are easier scenarios.

5.3 Qualitative Comparison

PNS+ produces more complete polyp segmentation than representative competitors in difficult visual cases, while ablations support its channel split, soft-attention, normalization, and global-to-local design choices. The analysis also identifies persistent failures under challenging visual attributes.

  • PNS+ accurately segments polyps with different sizes and homogeneous textures, whereas four competitors often miss complete regions with camouflaged textures.
  • Contribution of Base Network: 7.7% improvement in Sα is observed for the proposed model over the image-based base variant on SUN-SEG-Easy (Unseen).
  • Contribution of Soft-attention: 1.9% higher Dice is obtained with soft-attention than without it on SUN-SEG-Easy (Unseen).
  • Effectiveness of Normalization: 4.1% higher Dice is obtained with temporal normalization on SUN-SEG-Hard (Unseen), while global-to-local learning improves Fβ by 3.5% over local-to-global learning.
  • Issues and Challenges: Existing models, including PNS+, remain insufficiently robust for appearance changes, surgical instruments, optical flares, and limited polyp-size diversity.

6 Potential Directions

The paper identifies high-precision diagnosis, data-insufficient learning, privacy-preserving AI, and trustworthy AI as directions for advancing colonoscopy research. These directions remain substantially unresolved for VPS.

  • High-precision Diagnosis: Leading VPS approaches remain unsatisfactory for high-precision diagnosis, with sensitivity below 0.63 on SUN-SEG-Hard.
  • Data-insufficient Learning: Data-insufficient learning could use weakly supervised, unsupervised, self-supervised, and knowledge-distillation strategies under limited clinical data.
  • Privacy-preserving AI: Privacy-preserving AI should protect colonoscopy data throughout training, production, and governance, motivating techniques such as federated learning.
  • Trustworthy AI: Trustworthy VPS should support causal, transparent, explainable, and interactive AI-guided decisions.
  • The proposed directions remain far from solved for VPS.

7 Conclusion

The paper establishes a comprehensive deep-learning study of video polyp segmentation by introducing SUN-SEG, proposing PNS+, and benchmarking 13 segmentation approaches.

  • SUN-SEG extends the SUN-database into a large-scale VPS dataset with attribute, object mask, boundary, scribble, and polygon annotations.These diversified annotations support colonoscopy diagnosis, localization, and related tasks.
  • PNS+ uses normalized self-attention within a global-to-local learning strategy to exploit long-term and short-term spatial-temporal cues.The approach is designed to segment colon polyps from colonoscopy video.
  • PNS+ achieves the best performance against the 13 evaluated polyp/object segmentation competitors.
  • The study concludes by outlining potential directions for future colonoscopy-related research and related medical video analyses.
Loading 2203.14291v3…