PaperlayerView exact revision on arXiv

Source-linked AI summary

STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs

Jing Li, Duygu Sarikaya

arXiv:2609.08543v1cs.CV

TL;DR

Surgical VQA supervision remains largely frame-centric, while meaningful surgical reasoning requires tracking entities and interactions as scenes evolve. The paper builds event-centered spatio-temporal graphs and evidence-grounded temporal questions; structured supervision improves all seven temporal categories and increases reliance on visual input.

  • Problem

    Existing frame-centric scene graphs and coarse surgical VQA questions provide limited evidence of whether models can track surgical entities and interactions across time.

    Source: §I · p. 2

  • Method

    The paper augments frame-level surgical scene graphs with object continuity, event-level interaction continuity, and procedure-level connectivity to derive evidence-grounded temporal supervision.

    Source: §I · p. 2

  • Results

    STSG-VQA supervision improves all seven temporal categories, increases reliance on visual input, and complements static supervision in hybrid training.

    Source: §V · p. 9

  • Takeaways & Limitations

    The findings support treating surgical-state recognition and temporal state-transition reasoning as complementary supervisory signals.

    Source: §VI · p. 10

    The benchmark contains 45 videos from a single procedure, so video-level splitting cannot establish generalization across institutions, acquisition systems, patient populations, or other procedures.

    Source: §V · p. 9

Abstract

from arXiv · show

Despite recent advances in surgical vision-language models (VLMs), temporal reasoning remains limited because existing supervision is largely frame-centric. Frame-level scene graphs (SGs) have proven effective in providing structured representations of surgical environments but do not explicitly model the dynamics of surgical workflows. To explicitly model how surgical states evolve across time, we introduce a multi-level structured temporal supervision methodology that augments frame-level surgical SGs with object-level continuity, event-level interaction continuity, and procedure-level connectivity. We then execute temporal queries over the resulting spatio-temporal scene graphs (STSGs) to generate evidence-grounded question-answer pairs, which together form the STSG-VQA benchmark. Each question is linked to the temporal interval and STSG evidence used to derive its reference answer, enabling traceable verification. The benchmark contains 18,458 question-answer pairs across seven temporal categories. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B with STSG-derived supervision improves question-level micro accuracy by 24.39 and 19.56 percentage points over their zero-shot baselines and by 16.50 and 14.25 points over static scene-graph supervision, respectively. These gains span all temporal categories, indicating that STSG-derived supervision helps surgical VLMs reason over temporally grounded interactions rather than isolated frames. The code and dataset will be made publicly available upon acceptance.

I. INTRODUCTION

Existing surgical VQA supervision often remains frame-centric or asks coarse questions that do not test temporal relations, despite the need to track entities and interactions as surgical scenes evolve. The paper extends frame-level scene graphs into event-centered temporal supervision and introduces STSG-VQA to evaluate temporal reasoning.

I. INTRODUCTION

  • Surgical VLMs need temporal understanding to track object movements, ongoing interactions, and contextual changes before and after phase transitions.The introduction identifies scarcity of domain-specific datasets assessing temporal understanding and reasoning as a challenge.Source: §I · p. 1
  • Existing surgical VQA offers limited fine-grained temporal supervision: some questions are frame-level, while multi-frame questions may not explicitly probe relations across frames.Frame-level scene graphs also leave corresponding objects and interaction triplets unassociated across frames, limiting generated questions to frame-local properties.Source: §I · pp. 1–2, Figure 1 · p. 1
  • STSG-VQA extends frame-level surgical scene graphs into an event-centered representation of temporal evolution and supports seven temporal question categories.The categories include count, duration, ordering, extreme, boundary, phase transition, and concurrency.Source: §I · p. 2
  • The benchmark contains 18,458 question–answer pairs and uses evidence-grounded temporal queries to operationalize seven reasoning capabilities.The benchmark also provides a unified evaluation protocol for deterministic and open-ended answers.Source: §I · p. 2
  • Structured temporal supervision benefits temporal and static surgical VQA, increases the contribution of visual evidence, and yields the strongest frame-level performance when combined with static supervision.The reported benefits span both temporal reasoning and static surgical VQA.Source: §I · p. 2

II. RELATED WORK

Prior surgical VLM and VQA work advances image and video understanding, spatial grounding, and structured reasoning, but leaves explicit cross-frame object and interaction continuity underexplored. The paper positions STSGs as an event-centered, evidence-grounded approach to that gap.

A. Surgical Vision-Language Models

  • Extending surgical VLMs from image tasks to video remains challenging because they must integrate transient visual evidence and instrument–tissue interactions across frames while retaining procedural context.Temporal encoders and video-language objectives have been explored, but fine-grained temporal reasoning supervision and evaluation remain limited.Source: A. Surgical Vision-Language Models · p. 2

B. Surgical Visual Question Answering

  • Surgical VQA progresses from image-based answering and spatial bounding-box grounding to scene-graph reasoning, while video-level latent representations still lack explicit object tracks and temporally bounded interaction events.The paper instead introduces an event-centered STSG representation and derives evidence-grounded temporal supervision.Source: B. Surgical Visual Question Answering · pp. 2–3

C. Structured Knowledge for Enhanced Reasoning

  • Video and surgical scene graphs represent structured relations, but cross-frame identity association remains underexplored in surgery, where occlusions, camera motion, and abrupt appearance changes can destabilize it.Source: C. Structured Knowledge for Enhanced… · p. 3

III. METHODOLOGY

The methodology converts frame-level surgical graphs into event-centered spatio-temporal graphs by connecting objects across frames, aggregating persistent interactions, and relating events to procedural phases. The resulting graph supports traceable temporal question-answer supervision.

A. Overview

  • The STSG representation connects object identity across frames, consolidates repeated interactions into extended events, and relates those events to surgical phases.Questions and answers are traceable to the graph relations and temporal intervals used to generate them.Source: A. Overview · p. 3

B. Multi-Level Spatio-Temporal Representation

  • Each frame-level surgical scene graph contains detected entities and intra-frame relations, including spatial predicates and instrument–verb–target action triplets.Entities store visual and semantic attributes such as name, type, and bounding box.Source: B. Multi-Level Spatio-Temporal… · p. 3
  • The video-level STSG composes chronologically ordered, phase-aware frame graphs and augments them with cross-frame temporal links and event-level abstractions.The composition preserves entity nodes and intra-frame relations while associating frames with their procedural phases.Source: B. Multi-Level Spatio-Temporal… · p. 3
  • Temporal augmentation proceeds through object association, frame-level action grounding, and event aggregation to form persistent entities and temporally coherent interactions.Object matching first restricts candidates to observations sharing a component label and semantic type, then measures bounding-box overlap.Source: B. Multi-Level Spatio-Temporal… · p. 3, 1) Object-Level Persistence · p. 3

1) Object-Level Persistence:

Object continuity links frame-level detections into persistent tracks, allowing interactions to be grounded to specific entities and repeated action edges to be consolidated as events. The method distinguishes visually supported entities from implicit graph evidence.

1) Object-Level Persistence:

  • One-to-one object correspondences maximizing admissible bounding-box overlap create temporal coreference edges and persistent tracks across frames.The candidate matches share a component label and semantic type.Source: 1) Object-Level Persistence · p. 3

2) Frame-level Action Grounding:

  • Action triplets are resolved to specific participating entities, since semantic labels alone may not distinguish multiple same-label instruments in a frame.If an annotated endpoint lacks visual detection support, it remains implicit graph evidence rather than a reliable visual entity.Source: 2) Frame-level Action Grounding · pp. 3–4
  • Compatible action edges repeated across consecutive frames are accumulated into one event state rather than treated as independent observations.Continuous gallbladder retraction is represented as a single temporally grounded event.Source: 3) Event-level temporal aggregation · p. 4

3) Event-level temporal aggregation:

The method builds video-level spatio-temporal scene graphs by tracking entities and consolidating interactions into phase-aware events. Temporal queries over those graphs produce varied, evidence-grounded QA pairs with explicit answer scopes and supporting graph evidence.

3) Event-level temporal aggregation:

  • Object tracks preserve entity persistence and event nodes encode interaction evolution, supporting reasoning about event timing, duration, ordering, overlap, trajectories, and phases.Together, these structures preserve frame-level observations while adding temporal evidence.Source: 3) Event-level temporal aggregation · p. 4
  • Object observations are associated across frames into tracks, while compatible interaction states are extended into events and closed after temporal tolerance is exceeded.Completed events retain participants, temporal boundaries, phase context, and frame-level evidence; short observation gaps can leave an event open.Source: 3) Event-level temporal aggregation · p. 4

C. Evidence-Grounded Temporal Supervision

  • Queries specify an interaction, answer scope, and temporal property; the presented video segment may include context beyond the interval whose evidence determines the answer.Comparison questions retain only fully grounded interactions with explicit instrument and target instances.Source: C. Evidence-Grounded Temporal… · p. 5
  • Count and duration use merged temporal episodes, with a two-second merge tolerance and a one-frame-per-second sampling rate.Count refers to distinct episodes, while duration is their cumulative length.Source: C. Evidence-Grounded Temporal… · p. 5
  • Temporal categories use distinct evidence rules: ordering compares first occurrences, concurrency requires sustained interval overlap, and extreme questions rank activity by cumulative duration or episode frequency.Unambiguous comparisons are retained; phase-boundary and transition questions use dedicated windows around adjacent phases.Source: C. Evidence-Grounded Temporal… · p. 5
  • The benchmark uses plausible negatives, diversity-aware sampling, and 108 language templates; each QA pair retains its answer scope and supporting temporal and graph evidence.The resulting 18,458 pairs span seven temporal categories across 45 videos.Source: C. Evidence-Grounded Temporal… · p. 5

IV. EXPERIMENTS

The experiments examine whether structured temporal supervision improves VLM reasoning about evolving surgical events. They also test evidence use, transfer to frame-level VQA, and zero-shot limitations.

IV. EXPERIMENTS

  • The experiments compare zero-shot, static scene-graph, and structured temporal supervision across seven temporal reasoning categories.The central question is whether structured temporal supervision improves reasoning over evolving surgical events.Source: §IV · p. 6
  • A second evaluation tests whether observed gains reflect increased use of visual evidence and whether temporal supervision transfers to and complements frame-level surgical VQA.Source: §IV · p. 6
  • The experiments also use STSG-VQA to characterize zero-shot VLM limitations and assess the effect of increasing the frame budget.Source: §IV · p. 6

A. Dataset and Experimental Setting

The experiments pair a temporal benchmark with a frame-level benchmark to distinguish event reasoning from recognition of individual surgical states. They evaluate multiple VLMs under a shared input protocol and use deterministic scoring rules for answer types.

A. Dataset and Experimental Setting

  • STSG-VQA evaluates temporally extended surgical-event reasoning, while SSGVQA evaluates compositional understanding of individual frames.The two datasets allow the experiments to distinguish recognizing a surgical state from reasoning about its persistence, change, or interaction over time.Source: A. Dataset and Experimental Setting · p. 6
  • STSG-VQA is built from 45 laparoscopic cholecystectomy videos, with video-level partitioning intended to prevent visual and procedural leakage.The source videos span 90,089 seconds; the resulting graphs contain 87,399 frame nodes, 634,460 object observations, and 27,143 temporally aggregated event nodes.Source: A. Dataset and Experimental Setting · p. 6

B. Evaluation Protocol

  • Evaluation includes Qwen2.5-VL-7B, Qwen3-VL-4B, LLaVA-NeXT-Video-7B, and Hulu-Med-4B, using 16 chronologically sampled frames unless otherwise stated.Models receive the segment span, question, answer format, and output prefix, but not the reference answer or STSG evidence.Source: B. Evaluation Protocol · p. 6
  • Deterministic answers use exact integer matching and duration-dependent tolerance, while open-ended answers receive a binary semantic-equivalence judgment from a local Qwen3.5-4B judge.Video-balanced accuracy prevents videos with more questions from disproportionately determining the aggregate result.Source: B. Evaluation Protocol · p. 6

C. Implementation Details

The implementation fine-tunes Qwen3-VL-4B and Hulu-Med-4B with multimodal QLoRA and evaluates transfer between temporal and frame-level surgical VQA. The supplied table captions identify additional temporal-category and visual-input-control evaluations.

C. Implementation Details

  • Qwen3-VL-4B and Hulu-Med-4B are adapted with multimodal QLoRA using 4-bit NormalFloat quantization, double quantization, and bfloat16 computation.Training lasts 1.5 epochs, and the main training and test evaluation use 16 uniformly sampled chronological frames.Source: C. Implementation Details · pp. 6–7
  • Table IV covers question-level accuracy by temporal category in the 16-frame setting, while Table V covers micro accuracy under visual-input controls.Source: Table IV · p. 7, Table V · p. 7
  • The evaluation tests static-supervision transfer to STSG-VQA, temporal-supervision transfer to SSGVQA, and hybrid training on SSGVQA.Hybrid training combines SSGVQA and STSG-VQA examples.Source: C. Implementation Details · p. 7

D. Experimental Results

Structured temporal supervision improves temporal question answering over zero-shot and static scene-graph supervision, with gains across all seven categories. Fine-tuning also widens the advantage of visual input over text-only input.

1) Structured Temporal Supervision Improves Temporal Reasoning:

  • 24.39 and 19.56 percentage-point gains over zero-shot Qwen3-VL and Hulu-Med, respectively, also exceed static scene-graph supervision by 16.50 and 14.25 points.Improvements extend across all seven temporal categories for both model families.Source: 1) Structured Temporal Supervision… · p. 7
  • 29.16 and 28.98-point gains for Qwen3-VL, and 28.96 and 31.88-point gains for Hulu-Med, are the largest zero-shot improvements in concurrency and boundary reasoning.These tasks involve simultaneous interactions or states on opposite sides of a procedural boundary.Source: 1) Structured Temporal Supervision… · p. 7
  • 18.67% for Qwen3-VL and 16.67% for Hulu-Med make duration the lowest-scoring category after STSG-VQA fine-tuning.The passage attributes the remaining difficulty in exact temporal accumulation partly to sparse video sampling.Source: 1) Structured Temporal Supervision… · p. 7

2) Structured Temporal Supervision Increases the Contribution of Visual Evidence:

  • 16.90 points for Qwen3-VL and 13.49 points for Hulu-Med are the expanded 16-frame advantages over text-only input after STSG-VQA fine-tuning.Text-only accuracy also rises, but the larger improvement when frames are present makes a purely linguistic-shortcut account insufficient.Source: 2) Structured Temporal Supervision… · pp. 7–8

3) Temporal Reasoning Complements Static Surgical VQA:

Temporal supervision transfers to static surgical VQA, and hybrid supervision improves every reported SSGVQA metric over static-only training. Qualitative attention patterns and zero-shot results also show that temporal reasoning depends on more than frame count or salient-frame recognition.

3) Temporal Reasoning Complements Static Surgical VQA:

  • 33.90% versus 31.69% for Qwen3-VL and 30.85% versus 26.10% for Hulu-Med are SSGVQA accuracy results after STSG-VQA-only fine-tuning versus the backbones.Weighted F1 also increases for both models, while mAR decreases slightly, so temporal QA supervision does not replace direct frame-level training.Source: 3) Temporal Reasoning Complements… · p. 8
  • 3.83 points for Qwen3-VL and 3.04 points for Hulu-Med are the accuracy gains from adding STSG-VQA to SSGVQA-only training.Hybrid training also raises macro F1 from 55.58% to 59.09% for Qwen3-VL and from 49.00% to 50.47% for Hulu-Med, improving every reported SSGVQA metric for both models.Source: 3) Temporal Reasoning Complements… · p. 8

4) Qualitative Evidence of Temporally Selective Attention:

  • Hybrid-model attention is qualitatively more concentrated on frames and regions associated with grasper retraction for the temporal-ordering question.The comparison contrasts this pattern with SSGVQA-only attention dominated by frame-level visual saliency.Source: 4) Qualitative Evidence of Temporally… · pp. 8–9

5) Current VLMs Remain Limited in Zero-Shot Temporal Reasoning:

  • 45.80% micro accuracy is Hulu-Med-4B’s strongest overall zero-shot result with 8 frames; under 16 frames, Hulu-Med and Qwen2.5-VL are nearly tied at 45.51% and 45.47%.Increasing the frame budget does not consistently improve models: Qwen3-VL improves from 40.79% with 8 frames to 44.67% with 32, while other models plateau or decrease slightly.Source: 5) Current VLMs Remain Limited in… · p. 9, Table VII · p. 9
  • 40.85% to 49.38% is Qwen3-VL’s binary-accuracy change with additional frames, while its open-ended accuracy decreases from 28.30% to 25.79%.Across models, count, duration, and open-ended questions remain harder than binary and multiple-choice decisions.Source: 5) Current VLMs Remain Limited in… · p. 9

V. DISCUSSION

STSG-derived supervision improves temporal reasoning across all seven categories, while the discussion cautions that dataset scope, textual priors, and temporal sampling constrain interpretation. The conclusion presents structured temporal supervision as a scaffold connecting surgical-state recognition with modeling how states evolve.

V. DISCUSSION

  • STSG-VQA supervision improves all seven temporal categories, increases visual-input dependence, and complements static supervision in hybrid training.Because STSGs are withheld at inference, the findings are consistent with models internalizing event persistence, concurrence, and transition rather than retrieving graph evidence directly.Source: §V · p. 9
  • The benchmark's 45 videos come from one procedure, so video-level splitting cannot establish generalization across institutions, acquisition systems, patient populations, or procedures.Source: §V · p. 9
  • Generated templates and phase-conditioned questions retain textual and workflow priors despite grounded negatives and template diversification.External validation, held-out paraphrases, phase-name masking, and temporally shuffled or reversed inputs would provide stricter tests of generalization and temporal dependence.Source: §V · p. 10
  • One-hertz STSG construction and uniform 16-frame sampling can miss brief interactions and repeated-event boundaries, contributing to difficulty with counting and duration.The benchmark is intended for model development and evaluation and is not clinically validated for decision support.Source: §V · p. 10

VI. CONCLUSION

  • The framework augments frame-level scene graphs with object continuity, interaction continuity, and procedure-level connectivity, and supervises seven forms of temporal reasoning.Zero-shot models rely substantially on textual and procedural priors, whereas structured temporal supervision improves every temporal category and increases the contribution of visual evidence.Source: §VI · p. 10
  • Structured temporal supervision serves as a reasoning scaffold connecting recognition of surgical states with modeling their evolution, although duration estimation and event counting remain challenging.The framework is a step toward models reasoning over persistent entities, interactions, and procedural transitions rather than isolated frames.Source: §VI · p. 10
Loading 2609.08543v1…