Source-linked AI summary
ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation
Jiaxin Wen, Ming Yin, Lu Liu, Zeyu Fu
TL;DR
Referring surgical video segmentation must ground language expressions to targets despite similar instruments, occlusion, and complex interactions, while SAM2 pipelines can propagate incorrect initial anchors. ReGround-Surg uses a shared text-conditioned reliability map in two complementary grounding modules, improving J &F over ReSurgSAM2 across three splits with minimal overhead. The method strengthens Stage I without changing SAM2 tracking, but remains limited on diffuse tissue targets, requires gate-floor tuning, and does not address later memory drift or long-term occlusion.
Problem
SAM2-based two-stage referring segmentation is highly sensitive to initial anchor grounding, which is difficult under instrument similarity, occlusion, and tissue-tool interactions.
Method
ReGround-Surg predicts a shared text-conditioned spatial reliability map used by a Gated Side Adapter and Reliability-Weighted Vision-to-Text attention for Stage I grounding.
Results
+3.77, +3.09, and +0.94 J &F improvements over ReSurgSAM2 are reported across three evaluation splits with only 0.5 M additional parameters.
Takeaways & Limitations
Reliability-aware Stage I grounding improves referring surgical video segmentation while leaving the SAM2 encoder, decoder, and tracking components unchanged.
Takeaways & Limitations
Improvements may be weaker for spatially diffuse tissue regions, gate floor α requires manual tuning, and unchanged Stage II leaves memory drift and long-term occlusion unaddressed.
Abstract
from arXiv · showhide
Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.
1 Introduction
Referring surgical video segmentation enables language-based targeting beyond category-level methods, but SAM2 two-stage pipelines remain vulnerable to incorrect initial anchor grounding. ReGround-Surg addresses this bottleneck with shared reliability-guided feature modulation and reports consistent gains across evaluation splits.
- Motivation: Referring segmentation specifies instruments or tissue regions using object category, location, action, or tissue relations expressed in natural language.This flexibility supports interactive surgical video analysis, robotic assistance, and downstream skill assessment.
- Motivation: SAM2-based methods first ground the referred target and select an anchor mask, then propagate it through the video with SAM2 tracking memory.This two-stage design makes the initial grounding stage central to subsequent segmentation quality.
- Problem: Incorrect anchor selection persists even when Stage II tracking is disabled, indicating that the primary failure originates in initial grounding rather than temporal propagation.The failure is especially relevant under multi-instrument confusion, occlusion, and tissue-tool interactions.
- Approach: ReGround-Surg predicts a text-conditioned spatial reliability map and reuses it to enhance expression-relevant visual regions while suppressing off-target evidence during prompt-token aggregation.The Gated Side Adapter operates before text-to-vision attention, whereas Reliability-Weighted Vision-to-Text attention weights visual keys during aggregation.
- Results: +3.77, +3.09, and +0.94 J &F improvements over ReSurgSAM2 are reported across three evaluation splits, with only 0.5 M additional parameters and negligible speed overhead.The results are reported on Ref-EndoVis17 and Ref-EndoVis18.
2 Related Work
Prior work spans surgical adaptations of SAM and SAM2, referring segmentation methods, and cross-modal feature modulation. These approaches establish useful foundations but leave challenges from surgical visual similarity, occlusion, and reliability estimation.
- Surgical segmentation: SurgicalSAM adapts SAM for class-level instrument segmentation, while SurgicalSAM2 and MA-SAM2 target real-time processing and long-sequence tracking stability.These methods primarily address surgical segmentation without the full referring-video grounding problem described here.
- Referring segmentation: General-domain RVOS methods provide language-guided video segmentation baselines but generalise poorly to surgical settings with instrument similarity, occlusion, and challenging imaging conditions.Surgical-domain work such as RSVIS introduced referring surgical video instrument segmentation.
- Cross-modal modulation: LAVT suppresses irrelevant visual regions through text-guided spatial gating, whereas XMem weights elements explicitly but estimates reliability from visual cues without language conditioning.These limitations motivate reliability estimation that is both spatially selective and conditioned on the referring expression.
3 Method
ReGround-Surg improves Stage I anchor grounding by estimating a text-conditioned spatial reliability map and reusing it to modulate visual features and weight visual evidence. The resulting anchor mask is passed to the unchanged SAM2 tracking stage.
- Motivation: Stage I grounding is critical because inaccurate anchor selection persists even when temporal propagation is disabled, showing that strong tracking cannot compensate for poor initial masks.The method therefore targets the grounding stage rather than modifying SAM2 tracking.
- Reliability-Guided Anchor Grounding: ReGround-Surg predicts a spatial reliability map indicating alignment between visual content and the referring expression, then shares it across two complementary modules.The map is reused by the Gated Side Adapter and Reliability-Weighted V2T Attention.
- Gated Side Adapter: The Gated Side Adapter uses a soft residual gate before text-to-vision attention, preserving contextual information when expressions are ambiguous or targets are partially occluded.The gate has a floor α∈[0,1), with α=0.3 selected by grid search.
- Reliability-Weighted V2T Attention: Reliability-Weighted V2T Attention weights visual keys with the reliability map so expression-relevant positions contribute more strongly during prompt-token aggregation.The residual is added to the original visual keys, retaining unattenuated visual content downstream.
- Training and Inference: The proposed modules are used only during Stage I detection, while the resulting anchor mask is propagated by the original SAM2 tracking stage unchanged.Only the CrossModalFusionModule is updated; the SAM2 image encoder and tracking memory remain frozen.
4 Experiments
Experiments evaluate ReGround-Surg on two referring surgical video datasets using standard segmentation and initial-grounding metrics, ablations, difficult-case analyses, and qualitative comparisons. The method improves final and anchor-frame performance over ReSurgSAM2 while adding little computational cost.
- Datasets and Metrics: The method uses region similarity J, contour accuracy F, their mean J &F, and AnchorAcc to evaluate final segmentation and initial grounding.AnchorAcc is the fraction of anchor masks whose IoU with ground truth exceeds 0.5.
- Ablation Studies: GSA improves J &F by +1.81 and RW-V2T by +0.60 individually, while their combination yields a +3.09 gain, exceeding the summed individual gains by 0.68.The results support complementary effects from applying both modules to the shared reliability map.
- Ablation Studies: Text conditioning is central to the gate: static learned and visual-only gates yield only +0.17 and +0.40, while removing the residual structure reduces J &F from 82.43 to 81.94.The residual text-guided design therefore outperforms direct gating in the reported ablation.
- Further Analysis: Initial grounding improves by +9.52 in Init. J &F and +8.98 in AnchorAcc, translating into a +3.09 final video J &F gain after tracking.Initial anchor IoU shows a clear positive trend with final video J &F, although occlusion and scene complexity leave residual variance.
- Further Analysis: Qualitative comparisons cover multi-instrument confusion, partial occlusion, and complete grounding failure, with the method reducing distractor responses and improving first-frame localisation.The reliability gate focuses responses on the referred region, while RW-V2T reduces occluded or unreliable evidence during prompt-token aggregation.
5 Conclusion
ReGround-Surg improves referring surgical video segmentation by strengthening Stage I anchor grounding with reliability-guided modules while leaving SAM2 tracking unchanged. It reports consistent gains over ReSurgSAM2 with minimal added parameters, but remains limited for diffuse targets and post-anchor tracking failures.
- ReGround-Surg enhances Stage I detection using a Gated Side Adapter and Reliability-Weighted V2T attention that share a text-conditioned reliability map.Both modules improve grounding without modifying the SAM2 encoder, decoder, or tracking components.
- The reliability gate may improve less on spatially diffuse tissue targets, while its floor α requires manual tuning across surgical domains.Unchanged Stage II tracking also leaves memory drift and long-term occlusion errors after the anchor frame unaddressed.