PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task

Sunqi Fan, Jiashuo Cui, Meng-Hao Guo, Shuojin Yang

arXiv:2512.10359v1cs.CVcs.AI

TL;DR

MLLMs struggle to model spatial relationships and temporal evolution together, while tool-augmented approaches can have limited tool diversity and ineffective scheduling. The Video Toolkit provides varied video-analysis tools, and STAR alternates spatial and temporal tool calls to progressively localize question-relevant regions. With lightweight models under 3B parameters, STAR boosts GPT-4o, yielding an 8.2% gain on VideoMME and 4.6% on LongVideoBench, while outperforming related baselines in accuracy and efficiency.

  • Problem

    MLLMs struggle to model spatial relationships and temporal evolution together, while tool-augmented approaches can have limited tool diversity and ineffective scheduling.

    Source: §1 · p. 1

  • Method

    The Video Toolkit provides varied video-analysis tools, and STAR alternates spatial and temporal tool calls to progressively localize question-relevant regions.

    Source: §1 · pp. 2–3

  • Results

    With lightweight models under 3B parameters, STAR boosts GPT-4o, yielding an 8.2% gain on VideoMME and 4.6% on LongVideoBench, while outperforming related baselines in accuracy and efficiency.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The toolkit and STAR provide a step toward autonomous video-analysis assistants while improving accuracy and reducing computational cost.

    Source: §5 · pp. 9–10

    STAR relies on OpenAI GPT models, and repeated invocations can incur API costs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously modeling spatial relationships within video frames and understanding the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA task. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM's spatiotemporal reasoning capabilities and ensure the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at https://github.com/fansunqi/VideoTool.

1 Introduction

The introduction identifies gaps in tool-augmented video reasoning and presents a Video Toolkit and STAR framework to coordinate spatial and temporal analysis. It reports benchmark gains and describes STAR’s progressive localization approach.

1 Introduction

  • Video-LLMs can process many redundant frames, while tool-augmented LLMs often use only one reasoning dimension and lack effective scheduling.Increasing tool counts can also introduce execution issues, and disordered calls may impair efficiency and accuracy.Source: §1 · pp. 1–2
  • Spatial localization tools and basic zooming or cropping can address localization shortcomings, while filtering unimportant regions can reduce image-processing cost.Source: §1 · p. 2
  • STAR alternates temporal and spatial tools, adapting to intermediate results to progressively localize question-relevant 3D Regions of Interest.Its invocation strategy adapts to video length, content, and question characteristics, and expands the number of processed frames as needed.Source: §1 · pp. 2–3
  • STAR consistently achieves the highest accuracy on EgoSchema.Source: §1 · p. 3
  • The Video Toolkit contains 22 plug-and-play tools spanning varied functions, designed to balance tool quantity and diversity.Source: §1 · pp. 2–3
  • The toolkit-augmented GPT-4o gains 8.2% on VideoMME and 4.6% on LongVideoBench, and outperforms leading 7B Video-LLMs.The introduction also reports stronger scalability for STAR.Source: §1 · p. 3

2 Related Work

Related work covers the development of VideoQA methods and tool-augmented language models, including tool orchestration and applications to video understanding.

2.1 Video Question Answering

  • VideoQA research spans transformer-based cross-modal and spatiotemporal methods, Video-LLMs, adaptive keyframe selection, and lightweight keyframe extraction.Other work builds scene or event graphs for stepwise video understanding.Source: §2.1 · p. 3

2.2 Tool Learning

  • Tool-learning research includes API use, sequential tool combinations, fine-tuning or plug-and-play integration, and orchestration; visual tools have also been applied to video understanding [60].Prior orchestration research includes large-scale tool-network datasets.Source: §2.2 · p. 3
  • DoraemonGPT uses text-to-SQL over an intermediate tool-output database, which can fail and harm accuracy and efficiency; STAR uses a more reliable execution pipeline.Source: §2.2 · p. 4

3 Method

The Video Toolkit combines temporal, spatial, and general-purpose tools; STAR schedules them to progressively localize video regions relevant to a question. Its tools alternate across temporal and spatial dimensions, with general tools reserved for cases where those tools are insufficient.

3.1 Video Toolkit Construction

  • The Video Toolkit comprises about 22 temporal, spatial, and general-purpose tools, designed for segment- and frame-level processing through natural-language interfaces.Tools include specialized models, image and video manipulations, and encapsulated prompting pipelines; standardized tool cards support plug-and-play extensibility.Source: §3.1 · p. 4
  • Object-detection tools based on YOLO or Grounding DINO return bounding boxes that can be described in text, zoomed, or visually marked for further reasoning.These options let the planner use detected regions in different ways during question answering.Source: §3.1 · p. 4
  • The LLM-based Frame Selector uses the question and accumulated information to choose informative frames, then can direct processing to a key frame or a video segment.It begins with sparse uniform sampling and can discard irrelevant frames as information accumulates.Source: §3.1 · p. 4

3.2 Spatiotemporal Reasoning Framework

  • STAR alternates temporal and spatial tools, allowing general tools only as a last resort when the other tools cannot answer the question.The first tool may be temporal or spatial; the planner alternates from that starting point to avoid shortcut chains.Source: §3.2 · pp. 5–6
  • STAR maintains a Visible Frame Dictionary that temporal tools update by selecting frame indices and spatial tools update with information about those frames.The planner checks whether the dictionary contains enough information to answer; STAR progressively narrows the relevant 3D region of interest.Source: §3.2 · pp. 5–6
  • Interleaving lets spatial results inform later temporal choices; the paper reports that decoupling time and space reduces video-understanding accuracy and efficiency.STAR is described as training-free, user-friendly, and extensible.Source: §3.2 · pp. 4, 6

4 Experiments

Experiments evaluate STAR across four VideoQA datasets, using accuracy and computational efficiency to compare it with several baseline families. Results report gains over GPT-4o on VideoMME and LongVideoBench, strong NExT-QA performance, and the best accuracy and frame efficiency among the tested toolchain strategies.

4 Experiments

  • Evaluation covers VideoMME, NExT-QA, LongVideoBench, and EgoSchema, measuring multiple-choice accuracy alongside processed frames and runtime.STAR and STAR-MINI are designed for different computational budgets; STAR-MINI excludes tools above 500M parameters and tools requiring GPT-4o APIs.Source: §4 · pp. 6–7

4.1.1 Results on VideoMME

  • 8.2% improvement over GPT-4o at nearly the same input-frame count; STAR also improves 3.7% over InternVL3-8B.Compared with 72B Qwen2-VL, STAR reduces runtime from 6–8 minutes to 15.8 seconds.Source: Table 1 · p. 7, §4.1.1 · p. 8

4.1.2 Results on LongVideoBench

  • 4.6% improvement over GPT-4o at nearly the same input-frame count; STAR also outperforms 7B models on long and extra-long videos while reducing frames and runtime.The comparison notes that Gemini and GPT-4o with 256 frames use subtitles; STAR currently focuses on video content, with subtitle and audio integration left for future work.Source: Table 2 · p. 7, §4.1.2 · p. 8

4.1.3 Results on NExT-QA

4.2 Ablation Studies

  • STAR achieves the best accuracy and frame efficiency among the tested toolchain strategies, with longer, more diverse toolchains.No Constraints often produces short, repetitive toolchains and high frame-processing costs; prompting alone gives only marginal improvement, while in-context learning reduces processed frames but falls short in accuracy.Source: Table 5 · p. 9, §4.2 · p. 9

5 Conclusion

STAR combines a comprehensive video toolkit with alternating temporal and spatial tool use to localize 3D regions of interest. Experiments across four VideoQA datasets show improved accuracy and efficiency, while repeated GPT-model calls remain a source of API costs.

5 Conclusion

  • STAR alternates temporal and spatial tools to progressively localize 3D regions of interest, aiming to improve accuracy and reduce computational cost.The toolkit is designed to address temporal and spatial reasoning limitations in VideoQA.Source: §5 · p. 9
  • With tools under 3B parameters, STAR significantly boosts GPT-4o’s video understanding and reasoning and consistently outperforms related baselines in accuracy and efficiency.The conclusion reports experiments on four widely used VideoQA datasets.Source: §5 · pp. 9–10
  • STAR still relies on OpenAI GPT models, whose repeated invocations can incur API costs.The authors plan to explore lighter planners and better integration of tool use into the planner’s reasoning.Source: §5 · p. 10

A.1 Temporal Tools

  • Temporal tools select frames, ground described events in time, describe content within temporal spans, trim video, and localize actions to frames.Frame selection has three variants: a vanilla LLM, an AKeyS-inspired prompted LLM, and an MLLM that also receives frame images.Source: §A.1 · p. 17

A.2 Spatial Tools

  • Spatial tools include bbox marking, frame captioning and question answering, OCR, RoI zooming and cropping, and semantic segmentation.The bbox marker annotates 2D regions of interest on frames for MLLM question answering.Source: §A.2 · pp. 17–18

A.3 Tools that can be regarded as both temporal and spatial tools.

  • Tools spanning temporal and spatial roles include search, object identification and tracking, action recognition, image-grid and multiple-image QA, Python code generation, and video summarization.The object identifier finds question-relevant objects ahead of detection and tracking; action recognition uses both temporal dynamics and spatial cues.Source: §A.3 · p. 18

A.4 General-Purpose Tools

  • General-purpose tools summarize textual outputs maintained in the Visible Frame Dictionary or answer video questions directly.Both tools use Qwen2.5-VL-3B.Source: §A.4 · p. 18

B Datasets

The evaluation covers benchmarks for varied video lengths and reasoning demands, including causal and temporal questions, long-form video, and broad domain coverage.

VideoMME

  • VideoMME contains 2,700 multiple-choice examples spanning short, medium, and long videos, six domains, and diverse question types.It is used as a test-only benchmark for video analysis capabilities.Source: B Datasets · p. 18

NExT-QA

  • NExT-QA tests causal, temporal, and descriptive questions across relatively short videos averaging about 0.7 minutes.The work uses its multiple-choice subset, with 34K training, 5K validation, and 9K testing samples.Source: B Datasets · pp. 18–19

LongVideoBench

  • LongVideoBench evaluates referring-reasoning questions on videos averaging around 8 minutes; this work uses 1,337 validation questions without subtitles.Source: B Datasets · p. 19

EgoSchema

  • EgoSchema provides over 5,000 multiple-choice pairs for challenging long-form video QA, with human accuracy at 76% and current Video-LLMs below 70%.Source: B Datasets · p. 19

C Baselines

The baselines cover image-based MLLMs, Video-LLMs, frame-selection methods, and LLM-driven tool-learning systems, with comparisons tailored to the approach category.

Image-based MLLMs

  • Image-based MLLMs turn videos into image sequences and use prompts, but strong models are often proprietary and expensive, have limited spatiotemporal reasoning, and may degrade with too few frames.Source: C Baselines · p. 19

Video-LLMs

  • The Video-LLM comparison includes Qwen-VL, InternVL, and VideoLLaMA at roughly 7B and 72B parameters, aiming to exceed 7B models and approach 72B models.Video-LLM performance is also influenced by the number of input frames.Source: C Baselines · p. 19

(M)LLM-based Frame Selection Methods

  • Frame-selection baselines include sparse-to-dense LLM search, CLIP-guided video trees and keyframe hierarchies, adaptive sampling, T*, and AKeyS; comparisons focus mainly on NExT-QA.Source: C Baselines · pp. 19–20

LLM-driven Tool Learning Methods

  • Tool-learning baselines include ViperGPT, VideoChat, and DoraemonGPT; STAR-MINI uses comparably scaled tools for NExT-QA comparisons.DoraemonGPT builds space- and time-dominant memory and uses MCTS to search for a toolchain.Source: C Baselines · p. 20

D Ablation on each tool

Tool ablations measure how removing each toolkit component changes accuracy and processed-frame counts on VideoMME and LongVideoBench.

D Ablation on each tool

  • The ablation removes one tool at a time while holding other conditions fixed, measuring accuracy changes and processed-frame counts on VideoMME and LongVideoBench.Table 6 highlights the largest change within each tool category.Source: D Ablation on each tool · p. 20
  • In most cases, removing a tool lowers accuracy and increases processed frames, indicating that nearly all tools contribute positively to overall performance.Source: D Ablation on each tool · p. 20
  • In a few cases, removing tools that require many frames reduces accuracy but also reduces the number of processed frames.Source: D Ablation on each tool · p. 20

E Scalability with more frames

With denser frame sampling, STAR continued to improve accuracy; at 1 fps with up to 384 frames, it achieved a 5.2% accuracy gain on a sampled Video-MME evaluation.

E Scalability with more frames

F Generalizability with different base LLMs

Across different LLM planners, STAR generally improved accuracy over tool-free counterparts, with reported gains of 7%–8%.

F Generalizability with different base LLMs

G Analysis on Tool Balance

STAR aims to balance tool use by curbing over-reliance on tools that dominate unconstrained planning; tool-use-count variance generally decreased within categories under STAR.

G Analysis on Tool Balance

H Failure Cases

Reported STAR failure cases include ambiguous visual cues, sparse sampling that misses a video's theme, and difficulty inferring motivations. Case studies cover counting, action recognition, and action reasoning questions.

H Failure Cases

  • Ambiguous visual information can require subtitles or audio cues, which the authors plan to support with future inputs and tools.Source: H Failure Cases · p. 21
  • Sparse frame sampling sometimes fails to capture a video's main theme, motivating planned denser sampling for global reasoning tasks.Source: H Failure Cases · pp. 21–22
  • STAR struggles to infer motivations behind human actions, reflecting difficulty with fine-grained causal reasoning.Source: H Failure Cases · p. 22

I Case Study

Related papers

Loading 2512.10359v1…