Source-linked AI summary
WSPolypNet: Weakly Supervised Polyp Localization in Colonoscopy Videos
Giseong Hwang, Minjae Jo, Yeonghyeon Park, Kyeonghun Kim, Seoyeon Han, Donghoon Han, Haneul Kim, Yului Jeong, Insung Hwang, Pa Hong, Ken Ying-Kai Liao, Nam-Joon Kim
TL;DR
Dense frame-level spatial annotation is costly and variable, motivating WSPolypNet, which uses video-level supervision for polyp localization. Its multi-view CAM strategy with MedSAM2 refinement improved localization over single-view processing, especially for small polyps.
Problem
Dense frame-level bounding-box or pixel-level annotations for colonoscopy videos are time-consuming and can vary across observers, motivating localization from video-level labels.
Method
WSPolypNet uses a 3D CNN to generate CAM-based localization cues from video-level labels, enhances them across multiple views, and prompts MedSAM2 to refine and propagate masks.
Results
WSPolypNet achieved CorLoc scores of 47.80%, 43.68%, and 35.01% at IoU thresholds of 0.3, 0.5, and 0.7, compared with 36.87%, 33.72%, and 27.94% for single-view processing.
Takeaways & Limitations
Multi-view localization particularly improved small-polyp CorLoc@0.5 from 16.01% to 30.97%, supporting weakly supervised polyp localization with reduced spatial annotation requirements.
Takeaways & Limitations
The framework propagates a single high-confidence point prompt, restricting localization to one polyp when multiple polyps occur in the same video.
Abstract
from arXiv · showhide
Because dense frame-level annotation of colonoscopy videos is costly, we propose WSPolypNet, a weakly supervised framework for polyp localization using only video-level labels. WSPolypNet employs a 3D convolutional neural network trained with video-level supervision to generate class activation maps (CAMs), which identify candidate polyp regions without requiring frame-level spatial annotations. The CAM-derived localization cues are further enhanced using a multi-view strategy and provided to MedSAM2 as point prompts. MedSAM2 then propagates segmentation masks across the video, refining the coarse localization cues according to polyp boundaries. WSPolypNet achieved CorLoc scores of 47.80%, 43.68%, and 35.01% at IoU thresholds of 0.3, 0.5, and 0.7, respectively, compared with 36.87%, 33.72%, and 27.94% in the single-view setting. For small polyps, the multi-view strategy improved CorLoc@0.5 from 16.01% to 30.97%. The framework also achieved a recall of 94.51%. These results demonstrate the potential of weakly supervised spatiotemporal learning to substantially reduce spatial annotation requirements for polyp localization in colonoscopy videos.
I. INTRODUCTION
Colonoscopic polyp localization is clinically important but difficult because videos contain temporally varying appearances and complex tissue, while dense spatial annotation is costly. WSPolypNet addresses this gap using video-level supervision, 3D spatiotemporal modeling, CAMs, and MedSAM2 refinement.
- Up to 20%–26% of polyps may be missed during routine colonoscopic examinations.
- Frame-wise methods may produce temporally inconsistent predictions because they ignore correlations between consecutive video frames.
- Detailed bounding-box or pixel-level annotations require substantial effort and can vary across observers because polyp boundaries are ambiguous.
- Spatially localizing polyps from video-level supervision remains relatively underexplored amid changing viewpoints, intestinal structures, and visual disturbances.
- WSPolypNet uses a 3D CNN and CAMs to estimate candidate regions from video-level labels, then prompts MedSAM2 to refine their boundaries.
A. AI-based Polyp Detection and Localization
Prior polyp localization methods commonly process frames independently and rely on strong spatial supervision. Weakly supervised video localization reduces annotation requirements, while MedSAM2 provides dense mask refinement from CAM-derived point prompts.
- Most existing methods process individual frames independently, underuse temporal information, and require substantial annotation effort for long videos.
- Weakly supervised object localization estimates object locations from image-level labels without dense spatial annotations.
- Video extensions combine video-level supervision with temporal information for weakly supervised object localization.
- CAM localization can be incomplete or spatially coarse because it often emphasizes only the most discriminative object regions.
- CAM-derived candidate locations serve as MedSAM2 point prompts, allowing coarse localization to be refined into masks without frame-level target annotations.
III. PROPOSED METHOD
The proposed pipeline combines 3D CNN video classification, CAM-based localization, multi-view evidence aggregation, and MedSAM2 mask propagation. ROI preprocessing and spatial randomization reduce irrelevant peripheral and source-specific visual cues.
- A 3D CNN predicts video-level polyp presence and produces CAM-based coarse localization cues from preprocessed clips.
- CAM cues are enhanced with multiple views, converted into point prompts, and propagated bidirectionally by MedSAM2 to determine final localization.
- Source-specific differences in resolution, black borders, and interface overlays can introduce visual bias unrelated to polyp appearance.
- FOV-based ROI preprocessing crops the effective endoscopic region and masks pixels outside its detected convex hull.
- Random ROI resizing and placement reduce reliance on fixed padding patterns or ROI locations during training.
- The preprocessing removes irrelevant peripheral regions while preserving the effective endoscopic field of view and visible mucosal area.
C. Spatiotemporal Feature Learning and CAM Generation
WSPolypNet extracts spatiotemporal features from consecutive frames with a 3D CNN, classifies video-level polyp presence, and derives CAMs for initial localization cues.
- The input clip X has dimensions C×T×H×W, and the 3D CNN produces a feature tensor with K channels and temporal-spatial dimensions T′×H′×W′.
- Global average pooling feeds the feature tensor into a binary classifier trained with video-level labels and binary cross-entropy with logits loss.
- The CAM is computed from the final convolutional feature tensor to obtain spatial localization cues from the trained classifier.
- The CAM response map indicates each spatiotemporal location’s contribution to the polyp-present prediction and initializes refinement.
D. MedSAM2-based Localization Refinement
WSPolypNet enhances coarse CAM localization through five-view aggregation, candidate-point selection, and MedSAM2 propagation, with fallback predictions to address empty or drifting masks.
- Multi-view CAM aggregation: Five views combine a full 224×224 input with four overlapping 144×144 corner crops to improve localization, particularly for small polyps.Corner crops are resized to 224×224 before CAM computation, enlarging boundary regions that the full view may under-resolve.
- Multi-view CAM aggregation: The five view-specific localization maps are mapped to original-frame coordinates and fused using a pixel-wise maximum operation.
- Candidate point generation: The fused seed-frame map is normalized, Gaussian-smoothed, and greedily suppressed to produce five candidate points ordered by response.Smoothing uses σ = 4 px, while non-maximum suppression removes locations within a 16-px radius until five points remain.
- MedSAM2 propagation: Each candidate point prompts MedSAM2 to generate a seed-frame mask and propagate a segmentation track bidirectionally across the video.
- Candidate track selection: The first-ranked candidate is selected by default unless another candidate’s track score exceeds it by more than 20%.Track scores average frame-level CAM-derived localization responses over the video sequence.
- Fallback refinement: Propagated masks are replaced by independent frame-wise predictions when empty or when mean foreground confidence falls below τconf = 0.755.The fallback uses that frame’s fused-CAM peak as a positive MedSAM2 point, helping address tracking drift and refine spatial boundaries.
A. Dataset
The study constructs temporally ordered positive and negative colonoscopy clips for video-level classification and applies consistent preprocessing and training settings across backbones.
- Clip construction: 871 positive clips were generated from 100 LDPolypVideo frame sequences divided into non-overlapping clips of up to 30 consecutive frames.
- Clip construction: 615 negative clips were generated from 60 polyp-free endoscopic videos using 30 temporally consecutive, non-overlapping frames per clip.
- Dataset composition: The final dataset contained 1,486 clips, comprising 871 positive and 615 negative samples, at an output frame rate of 1 FPS.ROI preprocessing was applied to all input frames to reduce source-specific visual bias.
- Training and evaluation: Figure 3 qualitatively compares five 3D CNN backbones, the single-view setting, and full WSPolypNet using ground-truth and prediction boxes.
C. Evaluation Metrics
The evaluation compares localization metrics across 3D CNN backbones and assesses the proposed multi-view WSPolypNet against single-view localization. Results show stronger overall localization with X3D-based WSPolypNet, while multi-view gains for small polyps involve a trade-off for large polyps.
- Evaluation metrics: CorLoc was evaluated at IoU thresholds of 0.3, 0.5, and 0.7, with Recall additionally measuring identification of positive samples.
- Backbone comparison: X3D achieved the highest localization performance among evaluated backbones, with CorLoc scores of 21.03%, 9.32%, and 1.63%.R(2+1)D-18 achieved the highest Recall, but X3D was selected because spatial localization was the primary objective.
- Overall localization: WSPolypNet achieved CorLoc scores of 47.80%, 43.68%, and 35.01%, compared with 36.87%, 33.72%, and 27.94% for single-view localization.The comparison uses IoU thresholds of 0.3, 0.5, and 0.7, respectively.
- Polyp-size analysis: The multi-view strategy increased small-polyp CorLoc@0.5 from 16.01% to 30.97%, but decreased large-polyp CorLoc@0.5 from 71.84% to 65.75%.This indicates a size-dependent trade-off in multi-view localization performance.
V. CONCLUSION
WSPolypNet combines X3D-based CAM localization with MedSAM2 refinement for weakly supervised polyp localization in colonoscopy videos. It achieves 43.68% CorLoc at IoU 0.5 and improves small-polyp CorLoc from 16.01% to 30.97%, but currently supports only one polyp per video.
- WSPolypNet combines X3D-based CAM localization with MedSAM2 refinement for polyp localization in colonoscopy videos.
- WSPolypNet achieved 43.68% CorLoc at an IoU threshold of 0.5, while multi-view localization improved small-polyp CorLoc from 16.01% to 30.97%.
- The framework selects and propagates a single high-confidence point prompt through MedSAM2, restricting localization to one polyp when multiple polyps occur in the same video.
- The framework is currently more suitable for annotation assistance than complete annotation replacement.