Source-linked AI summary
Polyp-VSSD: A Pre-trained Transformer with Local-Global Joint VSSD Decoding for Polyp Segmentation
Xiaobo Zhu, Shiwei Wu, Yu An, Yuechen Tao, He Sun, Song Zhang, Zhenyu Liu
TL;DR
Polyp segmentation is challenging because endoscopic images contain variable polyp appearances and domain shifts, and existing methods have limited representational capacity and global-local integration. Polyp-VSSD combines a self-supervised pre-trained Transformer with a local-global VSSD decoder. It reports state-of-the-art results across five public benchmarks, including mIoU 0.903 and mDice 0.946 on EndoScene.
Problem
Endoscopic polyp segmentation remains challenged by variable morphology, imaging artifacts, low contrast, irregular shapes, and domain shift across datasets.
Source: §I · p. 1Method
Polyp-VSSD pre-trains a Transformer encoder on large-scale private endoscopic video and combines it with LG-VSSD decoding for global semantic modeling and local detail reconstruction.
Source: §I · p. 2Results
Across five public benchmarks, Polyp-VSSD reports state-of-the-art results; on EndoScene, it achieves mIoU 0.903 and mDice 0.946.
Source: §I · p. 2Takeaways & Limitations
The authors present Polyp-VSSD as a framework for robust polyp segmentation in complex endoscopic environments.
Source: §IV · p. 6
Abstract
from arXiv · showhide
Early detection of colorectal polyps is essential for preventing colorectal cancer, making accurate polyp segmentation a key component in computer-aided diagnosis systems. However, existing endoscopic segmentation methods often exhibit limited representational capacity and insufficient integration of global and local contextual information. To address these limitations, we propose Polyp-VSSD, a novel local-global joint decoding framework built upon a pre-trained Transformer backbone for high-precision polyp segmentation. Polyp-VSSD leverages the strong feature expressiveness of large-scale visual pre-training and unifies global semantic modeling with local fine-grained reconstruction to enhance segmentation fidelity. Extensive experiments on multiple public endoscopic datasets demonstrate that our approach significantly outperforms state-of-the-art methods in both Dice and IoU. On the EndoScene dataset, Polyp-VSSD achieves mIoU and mDice scores of 0.903 and 0.946, respectively, highlighting its robustness and strong generalization capability. This work presents an effective and scalable solution for intelligent endoscopic image analysis.
I. INTRODUCTION
Polyp segmentation remains challenging amid variable polyp appearances, imaging artifacts, and cross-dataset domain shift, while existing encoders and decoders have limitations in generalization and joint global-local modeling. Polyp-VSSD combines endoscopic self-supervised pre-training with local-global decoding and reports state-of-the-art performance across five public benchmarks.
I. INTRODUCTION
- Accurate automated segmentation is motivated by the role of early colonoscopic polyp detection and removal in preventing colorectal cancer.The introduction also frames robust segmentation as fundamental to computer-aided diagnosis that can alleviate clinicians’ workload and enhance diagnostic efficiency and accuracy.Source: §I · p. 1
- Existing polyp-segmentation encoders often lack robust generalization, while decoders may fail to unify long-range global context with fine-grained boundary reconstruction.The cited challenges include unseen or rare polyp appearances, low contrast, irregular shapes, and domain shift across clinical datasets.Source: §I · pp. 1–2
- Across five public benchmarks, Polyp-VSSD reports state-of-the-art results; on EndoScene, it achieves mIoU 0.903 and mDice 0.946.The authors describe the results as showing consistent superiority across all five datasets and strong generalization on EndoScene.Source: §I · p. 2
- Polyp-VSSD combines a large-scale pre-trained Transformer backbone with LG-VSSD decoding to model global semantics and reconstruct local detail.Its CNN branch recovers fine-grained structures, while its VSSD branch performs efficient linear decoding of long-range global features.Source: §I · p. 2
- DINOv2 self-supervised pre-training on a private endoscopic video dataset exceeding 20,000 minutes produces an encoder tailored to diverse endoscopic morphologies.The contribution summary describes the resulting Transformer encoder as having strong generalization and enhanced polyp-feature representation.Source: §I · p. 2
A. Endoscopic Foundation Model Pre-training
The section describes building an endoscopic foundation encoder through self-supervised pre-training on large-scale video. Its learned visual priors and contextual dependencies are then used for downstream segmentation.
A. Endoscopic Foundation Model Pre-training
- The framework has two stages: pre-train an endoscopic foundation encoder, then decode its features with LG-VSSD for polyp segmentation.The decoder jointly enhances global semantics and local boundary representation.Source: Figure 1 · p. 3
- Over 20,000 minutes of raw endoscopic video are uniformly sampled at two frames per second to yield approximately 1.4 million unlabeled frames for DINOv2 pre-training.DINOv2 learns dense features without manual annotations through knowledge distillation between teacher and student networks.Source: A. Endoscopic Foundation Model… · p. 3
- A standard Vision Transformer encoder is optimized with the DINOv2 objective, which enforces consistency between student predictions and the teacher’s smoothed output distribution.The pre-training is intended to mitigate limited generalization from conventional approaches trained on small, heterogeneous public datasets.Source: A. Endoscopic Foundation Model… · pp. 2–3
- The pre-trained encoder internalizes endoscopy-specific visual priors and contextual dependencies, enhancing representations of polyp morphology, boundaries, texture transitions, and background suppression.The resulting feature space is integrated into Polyp-VSSD as the downstream segmentation backbone.Source: A. Endoscopic Foundation Model… · p. 3
B. SSD Preliminaries
The preliminaries introduce state-space models and their SSD reformulation, including discrete dynamics and quadratic and linear forms. They also describe why conventional causal SSMs are constrained for visual tasks.
B. SSD Preliminaries
- A classical state-space model maps an input signal to a latent state, which then produces an output.The section presents continuous-time dynamics before describing discretization into discrete counterparts.Source: B. SSD Preliminaries · p. 3
- Discretization introduces a time-step parameter and applies zero-order hold to convert continuous SSM parameters into discrete counterparts.The passage identifies Δ as the time-step parameter.Source: B. SSD Preliminaries · p. 3
- Mamba2’s SSD reduces the state-transition matrix A to a scalar, yielding a special case of the selective SSM with linear and quadratic formulations.The section presents a quadratic form and a corresponding linear form.Source: B. SSD Preliminaries · pp. 3–4
- Traditional SSM causality limits each token to past inputs, while flattening 2D features into a 1D sequence disrupts spatial topology and local relationships.The passage identifies these properties as unsuitable for visual tasks and potentially detrimental to segmentation performance.Source: B. SSD Preliminaries · p. 4
C. Local-Global Visual Spatial State Duality Module
The LG-VSSD decoder couples VSSD’s non-causal, linear-complexity global context aggregation with CNN-based local detail recovery. Multi-scan fusion gives spatial tokens holistic context, while the local pathway refines fine-grained structure.
C. Local-Global Visual Spatial State Duality Module
- VSSD uses bidirectional or multidirectional scans over flattened image sequences, then fuses scan outputs so each token integrates non-causal context from the whole image.This provides long-range global feature fusion with linear complexity.Source: C. Local-Global Visual Spatial State… · p. 4
- The CNN-based Local Pixel Block progressively upsamples multi-scale encoded representations and injects them into the VSSD decoder as local cues for fine-grained spatial detail.Its pyramid structure leverages CNN local-feature perception to improve segmentation precision.Source: C. Local-Global Visual Spatial State… · p. 4
- LG-VSSD combines LPB’s local structural-detail recovery with VSSD’s linear-complexity global scanning to reconstruct fine-grained polyp morphology.The two pathways jointly support more accurate feature restoration and segmentation.Source: C. Local-Global Visual Spatial State… · p. 4
A. Datasets and Experimental Settings
The study pre-trained its encoder on filtered private endoscopy video and evaluated downstream segmentation on five public benchmarks. Training followed a two-stage protocol with a frozen encoder during decoder optimization.
A. Datasets and Experimental Settings
- Filtering more than 20,000 minutes of private endoscopy video and sampling at 2 FPS produced 1,451,670 cleaned images for DINOv2 self-supervised pretraining.Non-clinical, blurry or low-quality, and visually redundant content was removed; frames with minimal temporal variation were also discarded.Source: A. Datasets and Experimental Settings · p. 4
- Segmentation training used 1,450 labeled images from Kvasir-Seg and CVC-ClinicDB, with generalization assessed on five public datasets comprising 698 images.The evaluation sets were CVC-ClinicDB, Kvasir-Seg, CVC-ColonDB, EndoScene, and ETIS-LaribPolypDB.Source: A. Datasets and Experimental Settings · p. 4
- The encoder was pre-trained for 100 epochs on four A100 GPUs, then frozen while the decoder was trained for 100 epochs on one A100 GPU.Both LPB and VSSD used three layers; the stages used different learning rates and weight decay settings.Source: A. Datasets and Experimental Settings · p. 4
B. Comparative Study with Multiple Models
Polyp-VSSD achieves the best performance across five public benchmarks against a broad set of segmentation approaches. Its results are especially strong on ETIS and EndoScene, while the authors attribute its generalization advantage to modality-aware representations.
B. Comparative Study with Multiple Models
- Table I compares methods across five public datasets; Polyp-VSSD ranks best on all five and records mDice scores of 0.899 on ETIS and 0.946 on EndoScene.The comparison includes CNN-, Transformer-, Mamba-, and SAM-based methods.Source: B. Comparative Study with Multiple… · pp. 4–5, Table I · p. 5
- The authors suggest polyp-specific architectures outperform SAM-derived variants because of stronger endoscopic-image prior knowledge and more informative prompts.They identify modality-aware encoder representations as important for downstream segmentation decoders.Source: B. Comparative Study with Multiple… · p. 5
- Unlike LSSNet, which scores mIoU 0.900 and mDice 0.945 on ClinicDB but performs less strongly elsewhere, Polyp-VSSD achieves the highest scores on every dataset.The authors report more precise boundary delineation and structural perception, visually confirmed in Fig. 2.Source: B. Comparative Study with Multiple… · p. 5
C. Ablation Study
The ablation results attribute the final model’s performance to combining global VSSD decoding with local LPB refinement. The baseline’s limited decoder capacity and the separate contributions of each pathway motivate the joint design.
C. Ablation Study
- The projection-head baseline yields relatively low mean Dice scores across datasets, which the authors attribute to limited capacity for restoring fine-grained polyp features.The baseline directly connects the pre-trained ViT encoder’s linear projection head to the segmentation output.Source: C. Ablation Study · p. 5
- Adding the global VSSD path in place of conventional CNN upsampling produces consistent, notable gains and models long-range dependencies with linear complexity.The authors connect this global context modeling to stronger semantic comprehension of polyp morphology and spatial distribution.Source: C. Ablation Study · p. 5
- The pyramid-structured CNN LPB refines local features, strengthens boundary-aware representation, and improves reconstruction of subtle contour structures.With VSSD active, its local refinement boosts edge-level accuracy while retaining global awareness.Source: C. Ablation Study · p. 6
- The final architecture with both VSSD and LPB achieves state-of-the-art performance across evaluation metrics, balancing global semantic reasoning with local detail restoration.The global branch captures broad context, while the local branch delineates precise structure for diverse morphologies and complex boundaries.Source: C. Ablation Study · p. 6
IV. CONCLUSION
Polyp-VSSD combines endoscopic foundation-model pretraining with local-global VSSD decoding, and achieves state-of-the-art performance across five public polyp datasets. Ablations show substantial synergistic gains from both components; future work will explore real-time processing and broader downstream tasks.
IV. CONCLUSION
- Polyp-VSSD achieves state-of-the-art performance across five challenging public polyp datasets, particularly on ETIS-LaribPolypDB and EndoScene.The paper describes ETIS-LaribPolypDB as morphologically irregular and EndoScene as having complex backgrounds.Source: §IV · p. 6
- The framework uses a DINOv2-pretrained endoscopic foundation model as encoder and LG-VSSD decoding to combine global context modeling with lightweight convolutional local refinement.The foundation model is pretrained on a large-scale private endoscopic video dataset using self-supervision.Source: §IV · p. 6
- Ablation studies show substantial synergistic gains in final performance from foundation-model pretraining and the LG-VSSD mechanism.Source: §IV · p. 6
- Future work will optimize VSSD for real-time video processing and extend the endoscopic base model to polyp detection, classification, and multimodal fusion.Source: §IV · p. 6