PaperlayerView exact revision on arXiv

Source-linked AI summary

Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

Yiqing Shen, Mathias Unberath

arXiv:2610.28776v1cs.CV

TL;DR

Implicit surgical queries often require reasoning about instrument roles, workflow, and time, beyond closed-set labels or explicit target names. SurgReasonSeg combines surgical digital twins with complexity-aware adapters, adaptive context, and historical-case retrieval; it reports the best performance across reasoning and referring benchmarks at 18.8 FPS averaged over queries.

  • Problem

    Closed-set and referring segmentation do not resolve implicit surgical queries that require reasoning about workflows, instrument functions, and temporal relationships.

    Source: §1 · p. 1

  • Method

    SurgReasonSeg reasons over digital twins built from surgical foundation models, using complexity-routed RL LoRA adapters, adaptive context windows, and retrieval from surgical context memory.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Results

    SurgReasonSeg achieves the best performance across reasoning and referring benchmarks at 18.8 FPS averaged over queries.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper presents SurgReasonSeg as an online method for DT-based reasoning segmentation in surgical video.

    Source: §5 · p. 8

    Accuracy is bounded by SurgSAM2 candidate recall, and procedure-level questions depending on events minutes earlier are not evaluated.

    Source: §4 · pp. 6, 8

Abstract

from arXiv · show

Conventional surgical segmentation relies on closed-set categories or explicit text queries, which cannot resolve implicit queries that require reasoning about workflows, instrument functions, and temporal relationships. To address these limitations, we propose SurgReasonSeg, a complexity-aware method that brings reasoning segmentation over digital twin (DT) representations to surgical video. Specifically, SurgReasonSeg constructs DT representations from surgical foundation models and trains two low-rank adapters (LoRA) of one large language model with reinforcement learning (RL), each specialized for a different query complexity. A lightweight router, also trained with RL, selects between the two adapters to balance latency against accuracy. SurgReasonSeg also sets the temporal context with an adaptive sliding window conditioned on query complexity and workflow phase, and retrieves relevant historical cases from a surgical context memory by workflow similarity and procedural relevance. We further introduce EndoVis-ReasonSeg, a surgical reasoning segmentation benchmark with 1,460 training and 961 test samples across semantic, spatial, and temporal categories. On EndoVis-ReasonSeg, SurgReasonSeg improves J&F over the strongest baseline, ReSurgSAM2, by 0.178, 0.237, and 0.297 on semantic, spatial, and temporal queries, and it also achieves the best results on the Ref-EndoVis17 and Ref-EndoVis18 referring segmentation benchmarks. A backbone-matched attribution protocol separates the contribution of the upstream segmenter from that of the reasoning modules. Our method runs at 18.8 FPS averaged over the query distribution, enabling real-time reasoning segmentation in surgical videos.

1. Introduction

Surgical queries often describe an instrument’s function or procedural context rather than name a target, which existing segmentation approaches do not fully address. SurgReasonSeg uses digital-twin representations, complexity-aware reasoning, adaptive temporal context, and surgical case memory to address this gap.

Motivation

  • Implicit surgical queries can require inferring targets from workflow, instrument function, and temporal relationships rather than explicit anatomical labels.Closed-set segmentation uses a fixed vocabulary, while referring segmentation still requires the surgeon to name the target.Source: §1 · p. 1
  • Existing reasoning segmentation methods lack surgical knowledge, cannot retrieve relevant historical context, and spend the same computation on queries of different complexity.Surgical fine-tuning could help, but requires costly annotations and leaves knowledge fixed at training time.Source: §1 · p. 1

Proposed approach

  • SurgReasonSeg builds digital twins from surgical foundation models to provide intermediate abstractions between raw video and reasoning.The DT incorporates surgical domain knowledge through models such as SurgSAM2 and RASO.Source: §1 · p. 1
  • Two RL-trained LoRA adapters, a router, an adaptive sliding window, and surgical context memory tailor reasoning and context to query demands.The memory retrieves past cases by workflow similarity and procedural relevance; the window depends on query complexity and workflow phase.Source: §1 · p. 2

Contributions

  • The paper introduces EndoVis-ReasonSeg and reports backbone-matched attribution and template-disjoint evaluation to assess component contributions and template-level leakage.The benchmark covers semantic, spatial, and temporal reasoning queries.Source: §1 · p. 2

2. Related Work

Prior reasoning segmentation and surgical video segmentation address parts of the problem, while digital-twin reasoning provides a structured interface. The paper positions its work at the intersection of these approaches.

Reasoning Segmentation

  • Reasoning segmentation identifies objects from implicit queries, with prior methods adding segmentation decoding, multi-target handling, interleaved reasoning, or reinforcement learning.Video approaches select a keyframe for mask propagation or compress frames into visual tokens, but general-domain training leaves surgical vocabulary and procedural priors unmodeled.Source: §2 · p. 2

Digital Twin Representations

  • Digital twins serve as structured scene abstractions for LLM reasoning; SurgReasonSeg inherits the DT formulation and distinguishes inherited, adapted, and new components.The paper’s Table 1 describes the inherited DT encoding and four added components.Source: §2 · p. 2, Table 1 · p. 2

Surgical Video Segmentation

  • Surgical referring methods such as RSVIS and ReSurgSAM2 require named targets, while prior temporal-context methods are not conditioned on query complexity.These approaches do not reason over workflow phases or instrument function.Source: §2 · p. 2

3. Methods

SurgReasonSeg represents surgical scenes as digital twins and combines complexity-aware routing, adaptive temporal context, and retrieval from a surgical context memory. EndoVis-ReasonSeg supplies implicit-query training and test data across three reasoning categories.

Method overview

  • The method constructs digital twins, selects temporal context with an adaptive window, retrieves similar past cases, and routes queries to one of two LoRA adapters for reasoning and mask-index output.The overview figure describes the pipeline from surgical video and implicit query through DT construction, context selection, and complexity-matched reasoning.Source: §3 · p. 2, Figure 1 · p. 3

Digital Twin Construction

  • SurgSAM2 supplies instrument and tissue masks, DepthAnything2 estimates instance depth, and RASO assigns surgical semantic labels for the digital-twin representation.The DT also stores centroids and bounding boxes; JSON serialization turns segmentation into retrieval of the target object’s index.Source: §3 · p. 3

Complexity-Aware Reasoning

  • Two LoRA adapters produce extended or concise reasoning chains, while an RL-trained router selects between them using query and DT-window inputs.The router reward balances segmentation accuracy against inference time; reasoning-length thresholds induce adapter specialization without hand-labeled complexity classes.Source: §3 · pp. 3–4

Adaptive Sliding Window

  • The adaptive window combines a base span with query-complexity and workflow-phase extensions; temporal queries at phase transitions have the largest mean window, 6.1 frames.Semantic queries remain near wbase = 2; the router reads the base-plus-phase window before the selected adapter receives the full window.Source: §3 · p. 4, Figure 2 · p. 4

Surgical Context Memory

  • The surgical context memory stores DT sequences, queries, and segmentation outcomes, then retrieves examples using workflow similarity, procedural relevance, and outcome quality.The bank is built only from training videos and frozen during evaluation; retrieved cases are supplied as few-shot examples.Source: §3 · p. 4

Dataset Construction

  • EndoVis-ReasonSeg uses manually written semantic, spatial, and temporal implicit queries paired with clips and ground-truth masks from Ref-EndoVis17, Ref-EndoVis18, and RSVIS.Training and test videos are disjoint by sequence; a separate template-disjoint split holds out 10 of the 47 shared query templates from training.Source: §3 · pp. 4–5

Results

  • SurgReasonSeg achieves the best J, F, and J&F on all three EndoVis-ReasonSeg reasoning categories.Source: Table 2 · p. 5

4. Experiments

SurgReasonSeg outperforms ReSurgSAM2 across semantic, spatial, and temporal reasoning queries, and also leads on the three referring-segmentation settings. Ablations, routing analyses, and attribution tests examine accuracy, latency, and the roles of its components.

Reasoning Segmentation Performance

  • SurgReasonSeg improves J &F over ReSurgSAM2 by 0.178, 0.237, and 0.297 on semantic, spatial, and temporal reasoning, respectively.Baselines used released weights without EndoVis-ReasonSeg fine-tuning or DT access; the comparison is end-to-end. Results report mean±standard deviation over three seeds.Source: §4 · pp. 5–6

Efficiency Analysis

  • At the empirical routing distribution, SurgReasonSeg averages 18.8 FPS, with 48.8 ms for Lsimple and 63.2 ms for Lcomplex inference paths.DT construction for each incoming frame is included; cached entries mean its 10.6 ms cost does not grow with window size.Source: Figure 3 · p. 5, §4 · p. 6

Qualitative Results and Failure Cases

  • Across three qualitative examples, SurgReasonSeg has the highest J on every frame; reported failure sources include missing candidates, attribute errors, and boundary ambiguity.Missing candidates account for 29.7% of test failures and attribute errors for 42.3%; memory sometimes corrects attribute errors by retrieving a contrasting unsuccessful case.Source: §4 · pp. 6–7, Figure 4 · p. 7

Referring Segmentation Performance

  • SurgReasonSeg achieves the best J &F on all three referring settings, improving over ReSurgSAM2 by 0.024, 0.023, and 0.027.The settings are Ref-EndoVis17 (tool), Ref-EndoVis18 (tool), and Ref-EndoVis18 (tissue).Source: Table 3 · p. 6, §4 · p. 7

Ablation Study

  • Removing either the adaptive sliding window or surgical context memory reduces J &F most on temporal queries; their temporal contributions are 0.053 and 0.043, respectively.The full model with DT, ASW, and SCM reaches 0.726, 0.663, and 0.590 J &F at 12.5 FPS.Source: Figure 3 · p. 5, Table 4 · p. 6, §4 · pp. 7–8

Routing Analysis

  • Routing achieves 0.771, 0.717, and 0.654 J &F at 18.8 FPS, while always using Lcomplex reaches 0.758, 0.706, and 0.643 at 16.0 FPS.The accuracy differences fall within seed-to-seed deviations, so the paper reads them as parity; routing selects Lsimple for 69% of queries and Lcomplex for 31%.Source: Figure 5 · p. 7, §4 · p. 8

Attribution and Protocol Analysis

  • With identical DT masks, surgical training, and an oracle comparison, the attribution analysis assigns 84%, 85%, and 90% of the total margins to reasoning modules.The protocol supplies identical masks to controls, removes training-data asymmetry with fine-tuning, and uses an oracle to bound attainable accuracy.Source: §4 · p. 8, Table 5 · p. 8

Limitations

  • Accuracy is bounded by SurgSAM2 candidate recall; the study does not evaluate procedure-level questions or continual memory adaptation across institutions.The window spans only a few frames, and the memory is validated on EndoVis videos only.Source: §4 · p. 8

5. Conclusion

SurgReasonSeg brings digital-twin reasoning segmentation to surgical video, combining complexity-aware routing with workflow-aware temporal modeling. It achieves the best performance across the reasoning and referring benchmarks at 18.8 FPS averaged over queries.

5. Conclusion

  • 18.8 FPS averaged over queries accompanies the best performance across all reasoning and referring benchmarks.Tab. 5 decomposes the gain into segmenter and reasoning contributions.Source: §5 · p. 8
  • SurgReasonSeg constructs surgical-video digital twins from surgical foundation models, with complexity-aware LoRA routing and workflow-aware temporal context.The adaptive window and surgical context memory provide workflow-aware temporal modeling.Source: §5 · p. 8
  • EndoVis-ReasonSeg is introduced as a benchmark for surgical reasoning segmentation.Future work targets open surgery and faster LoRA inference, identified as the main per-frame cost.Source: §5 · p. 8

and Context Memory

The supplementary material examines memory-bank and backbone scaling alongside other design choices and evaluation details. Its table captions identify a 500-entry memory peak and limited gains from the 8B backbone relative to 4B.

and Context Memory

  • SurgReasonSeg's J &F peaks at 500 memory entries.Latency is a cost and is not ranked.Source: Table S1 · p. 10
  • Qwen3-8B adds at most 0.005 J &F over Qwen3-4B at less than half the FPS.FPS is end-to-end, a cost, and is not ranked.Source: Table S2 · p. 10
  • The supplement reports adaptive-window sensitivity, memory-bank and backbone scaling, design ablations, per-video and per-template results, and a Cholec80 transfer subset.The judge sees only the overlaid mask and query, not the ground truth or reasoning chain.Source: and Context Memory · p. 10

S1. Adaptive Window Sensitivity

Adaptive-window hyperparameter settings yield little variation in J &F, while resulting window sizes differ by query type and workflow phase.

S1. Adaptive Window Sensitivity

  • J &F varies by at most 0.015 across the tested adaptive-window hyperparameter settings.The tested ranges are wbase ∈ {1, 2, 3}, wquery ∈ {1, 2, 3}, and wphase ∈ {2, 3, 4}.Source: S1. Adaptive Window Sensitivity · p. 10
  • Semantic queries stay near wbase, while temporal queries during phase transitions receive larger windows.Fig. 2 shows the resulting window sizes.Source: S1. Adaptive Window Sensitivity · p. 10

S2. Memory Bank and Backbone Scalability

Memory removal affects temporal reasoning more than semantic reasoning, and memory performance peaks at 500 entries. Scaling the backbone to 8B gives marginal gains over 4B while reducing inference speed; the tested design choices outperform their replacements.

S2. Memory Bank and Backbone Scalability

  • Without memory, temporal reasoning drops by 0.043 J &F, compared with a 0.018 J &F decrease for semantic reasoning.Semantic queries are often resolved from single-frame digital-twin attributes.Source: S2. Memory Bank and Backbone Scalability · p. 10
  • Every tested design choice outperforms its replacement, and the VLM judge is within 0.004 of the ground-truth IoU upper bound.The comparisons cover RL versus supervised fine-tuning adapters, router alternatives, and VLM-judge weighting alternatives.Source: Table S3 · p. 10
  • Performance improves consistently up to 500 memory entries and slightly decreases at 1,000.At 500 entries, lookup takes 1.2 ms; at 1,000 entries, retrieval latency is 2.1 ms, both reported as compatible with real-time operation. All memory results come from EndoVis videos.Source: S2. Memory Bank and Backbone Scalability · p. 10
  • Scaling from 0.6B to 4B parameters improves all reasoning categories, with the largest gain in temporal reasoning at +0.056 J &F.The passage attributes this pattern to larger models better capturing complex procedural dependencies.Source: S2. Memory Bank and Backbone Scalability · p. 10
  • Qwen3-8B adds only +0.003–0.005 J &F over 4B while lowering FPS from 18.8 to 8.6, below the real-time threshold.The authors use Qwen3-4B as the balance between reasoning accuracy and real-time inference.Source: S2. Memory Bank and Backbone Scalability · p. 10

S3. Design Choice Ablations

The ablations separately probe the training objective, router design, and judge used for reward weighting. The VLM judge is compared with human annotations and a ground-truth IoU upper bound.

S3. Design Choice Ablations

  • The ablations test whether RL is needed by replacing GRPO with supervised fine-tuning on the same rollouts.This comparison asks whether supervision can imitate the output-length selection attributed to the reward.Source: S3. Design Choice Ablations · pp. 10–11
  • The router ablation compares the RL-trained router with a category rule and a supervised classifier under a latency-aware reward.It tests whether the router must be learned against that reward.Source: S3. Design Choice Ablations · pp. 10–11
  • The VLM judge agrees with human annotators at Cohen’s κ of 0.82, and its performance approaches the ground-truth IoU upper bound.The ablation varies reward weighting among uniform weights, a ground-truth IoU bound, and the external judge.Source: S3. Design Choice Ablations · pp. 10–11

S4. Per-Video and Per-Template Breakdown

The per-video and per-template analyses report that the performance ordering persists across test sequences and that held-out phrasings retain comparable accuracy to seen ones.

S4. Per-Video and Per-Template Breakdown

  • 0.728, 0.663, and 0.591 J &F on the weakest sequence exceed ReSurgSAM2 fine-tuned on the training split by 0.107, 0.148, and 0.203, respectively.The reported sequence-by-sequence margins show that no single video carries the aggregate margins.Source: S4. Per-Video and Per-Template Breakdown · p. 11
  • 0.746, 0.692, and 0.628 on 205 held-out-template test samples are only 0.025 to 0.026 below the full model on the standard split.Ten held-out templates define the template-disjoint split; the authors interpret the result as evidence of resolving queries through the DT rather than template memorization.Source: S4. Per-Video and Per-Template Breakdown · p. 11
  • The 0.075–0.110 per-sequence spread tracks the source dataset, with EndoVis17 sequences below EndoVis18 sequences.The passage says this is consistent with the older subset’s lower annotation density.Source: S4. Per-Video and Per-Template Breakdown · p. 11

S5. Cholec80 Transfer Subset

The Cholec80 transfer evaluation tests the model without retraining on a procedure and instrument set disjoint from EndoVis. Performance drops from the in-domain result, but the transferred model still exceeds every baseline of Tab. 2.

S5. Cholec80 Transfer Subset

  • 0.050, 0.052, and 0.071 J &F are the respective drops from the in-domain result, while the transferred model still exceeds every baseline of Tab. 2.The evaluation uses a reasoning segmentation subset built from Cholec80, disjoint from EndoVis in procedure and instruments.Source: S5. Cholec80 Transfer Subset · p. 11
  • The model is evaluated without retraining, with its memory bank populated only from EndoVis training videos, measuring cross-procedure transfer rather than adaptation.The subset comprises 10 sequences segmented into 20 to 40 frame clips at instrument-change boundaries, with 420 implicit queries from a held-out template pool (Cohen’s κ = 0.85).Source: S5. Cholec80 Transfer Subset · p. 11

Related papers

Loading 2610.28776v1…