Source-linked AI summary
VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation
Yiming Zhao, Yu Zeng, Wenxuan Huang, Zhen Fang, Qing Miao, Qisheng Su, Jiawei Zhao, Jiayin Cai, Lin Chen, Zehui Chen, Yukun Qi, Yao Hu, Xiaolong Jiang, Feng Zhao
TL;DR
Instance-level video understanding remains difficult because text-only queries and language-centered, passive perception provide imprecise references and can miss fine-grained evidence. VideoSeeker addresses this with visual prompts, native agentic tool invocation, automated data synthesis, and two-stage training, achieving +13.7% average improvement over baselines while surpassing GPT-4o and Gemini-2.5-Pro.
Problem
Existing video methods struggle with precise instance-level spatiotemporal localization because text queries provide imprecise references and perception is often passive and language-centered.
Method
VideoSeeker combines visual prompts, proactive tool-based perception, a four-stage automated data pipeline, and SFT-plus-Agentic-RL training.
Results
+13.7% average improvement over Qwen3-VL-8B on instance-level tasks, surpassing Gemini-2.5-Pro and GPT-4o; general benchmarks improve by +3.2% and +3.3% over three tasks.
Takeaways & Limitations
VideoSeeker integrates native tool calling with instance-level video understanding and transfers effectively to general video understanding benchmarks.
Takeaways & Limitations
RL on multiple-choice data suffers reward hacking, reducing performance to 43.8%, whereas open-ended training reaches 74.5%.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) have shown significant progress in video understanding, yet they face substantial challenges in tasks requiring precise spatiotemporal localization at the instance level. Existing methods primarily rely on text prompts for human-model interaction, but these prompts struggle to provide precise spatial and temporal references, resulting in poor user experience. Furthermore, current approaches typically decouple visual perception from language reasoning, centering reasoning around language rather than visual content, which limits the model's ability to proactively perceive fine-grained visual evidence. To address these challenges, we propose VideoSeeker, a novel paradigm for instance-level video understanding through visual prompts. VideoSeeker seamlessly integrates agentic reasoning with instance-level video understanding tasks, enabling the model to proactively perceive and retrieve relevant video segments on demand. We construct a four-stage fully automated data synthesis pipeline to efficiently generate large-scale, high-quality instance-level video data. We internalize tool-calling and proactive perception capabilities into the model via cold-start supervision and RL training, building a powerful video understanding model. Experiments demonstrate that our model achieves an average improvement of +13.7% over baselines on instance-level video understanding tasks, surpassing powerful closed-source models such as GPT-4o and Gemini-2.5-Pro, while also showing effective transferability on general video understanding benchmarks. The relevant datasets and code will be released publicly.
1 Introduction
VideoSeeker addresses limitations in precise instance-level video understanding by combining visual prompts, agentic reasoning, and automated data synthesis. Its experiments report stronger instance-level performance and transfer to general video understanding.
- Motivation: Existing methods often separate visual perception from language reasoning, use passive uniform sampling, and rely on text queries for spatial-temporal references.These limitations make fine-grained instance localization difficult, especially in complex multi-object videos.
- Approach: VideoSeeker uses visual prompts to specify target instances more precisely than lengthy textual descriptions.Users can annotate target regions directly on video frames.
- Training: A two-stage training strategy combines cold-start SFT with Agentic RL to integrate tool calling, proactive perception, and instance-level reasoning.The strategy guides exploration toward high-information visual evidence.
- Data synthesis: The four-stage pipeline filters text QA, verifies target uniqueness, generates pixel-level masks, and renders visual prompts with rewritten QA.The stages use lightweight language filtering, video understanding verification, SAM3 segmentation, and diverse prompt rendering.
- Results: VideoSeeker significantly outperforms open-source and proprietary baselines on instance-level tasks and transfers effectively to general video understanding.The introduction reports broad gains across both instance-level and general video understanding scenarios.
2 Related Works
Prior work extends LVLMs with reinforcement learning and visual tools for multimodal reasoning, including dynamic video retrieval and dense resampling. VideoSeeker differs by targeting specific instances with visual prompts and native tool calling rather than holistic semantics or text-only queries.
- Reinforcement Learning for Vision Language Models: Recent multimodal RL methods improve reasoning for image question answering, grounding, and video understanding.Examples include object-matching and IoU rewards for grounding and RL-driven visual tool invocation.
- Tool-Augmented Agentic Vision Language Models: Tool-augmented vision-language models use external visual tools to extend reasoning beyond pure text understanding.Prior systems support image reasoning, spatial drawing, temporal retrieval, dense video resampling, and dynamic tool calling.
- Video understanding: LongVT and VITAL primarily model holistic video semantics, whereas VideoSeeker tracks specific target instances within videos.The distinction concerns instance-level localization versus global or coarse-grained video understanding.
- VideoSeeker: VideoSeeker uses bounding boxes, points, and masks as queries, providing more precise spatial and temporal references than pure text queries.Its four-stage data pipeline and two-stage training paradigm internalize native tool calling for instance-level understanding.
3 Method
VideoSeeker combines visual-prompt data synthesis with interactive tool use and staged training for instance-level video understanding. Its agentic policy actively retrieves visual evidence, while rewards jointly target correctness, output format, and efficient perception.
- Task Formulation: Instance-level video understanding requires associating a visual prompt with a target instance and reasoning about that instance’s temporal dynamics.
- Environmental Interaction: The environment provides view_visual_prompt and crop_video tools for maintaining the target appearance and selectively retrieving fine-grained video segments.
- Data Construction: The four-stage data pipeline filters text QA, verifies unique targets, generates pixel-level masks, and renders diverse visual prompts without manual annotation.
- Training Data: Reject sampling produces 34.2k trajectories for supervised fine-tuning and 4.1k pass-k-filtered samples for GRPO training.
- Training Strategy: Supervised fine-tuning establishes multi-turn active-perception behavior, followed by GRPO training in which the policy autonomously decides when and how to invoke tools.
- Training Strategy: The reward design combines answer accuracy, format compliance, and parsimony to support correct, structured, and resource-efficient tool use.
4 Experiments
Experiments show that VideoSeeker improves instance-level video understanding through its data, training, and agentic tool-calling design, while transferring to general video tasks and reducing inference costs. Additional analyses identify diminishing returns from more data, heterogeneous distillation losses, and reward hacking under multiple-choice RL.
- Main Results: +13.7% average improvement over Qwen3-VL-8B establishes VideoSeeker-8B as stronger than GPT-4o and Gemini-2.5-Pro on instance-level video understanding.VideoSeeker-4B also improves over Qwen3-VL-4B by +11.4% on average.
- Data Ablation: Performance improves with larger SFT datasets but shows diminishing marginal returns as data volume expands toward saturation.The data-scale study evaluates progressively larger subsets of the full training corpus.
- Stage Ablation: 74.5% is achieved by the cascaded SFT-plus-agentic-RL pipeline, with SFT contributing +9.6% and agentic RL contributing +5.1%.Agentic RL improves over single-turn RL by +3.3%, while single-turn RL contributes +1.8%.
- Generalization: +3.2% and +3.3% average improvements show transfer from instance-level training to three general video understanding tasks.The model was trained exclusively on instance-level video understanding tasks.
- Distillation Analysis: Teacher accuracy does not proportionally transfer: Qwen3-VL-235B-A22B-Thinking and Gemini-3.1-Pro yield students scoring 70.4% and 64.7% on V2P-Bench.The corresponding relative degradations are 8.0% and 19.1%, respectively, under heterogeneous distillation.
- Time Efficiency: VideoSeeker reduces inference costs by converging with fewer total action steps through streamlined tool calling and more compact reasoning chains.The comparison evaluates inference costs in Agent mode across generation and tool-calling phases.
5 Conclusion
VideoSeeker combines native tool invocation, agentic reasoning, and visual prompts for instance-level video understanding. It uses automated data synthesis and two-stage training, achieving strong benchmark performance and transferability.
- VideoSeeker enables LVLMs to perform instance-level video understanding through native tool invocation.The paradigm integrates agentic reasoning with instance-level tasks.
- Models proactively perceive and retrieve relevant video segments on demand for more precise spatial and temporal references.This addresses limitations of traditional text-only approaches.
- A four-stage automated data synthesis pipeline generates large-scale instance-level video data.
- A two-stage training strategy internalizes tool-calling capabilities into LVLMs.
- +13.7% average improvement on V2P-Bench enables VideoSeeker-8B to surpass GPT-4o and Gemini-2.5-Pro.The model also shows effective transferability to broader video understanding scenarios.
Appendix Overview
The appendix provides supplementary sections covering datasets, benchmarks, hyperparameters, limitations and social impacts, training curves, case studies, and prompts.
- Dataset Details and Benchmark Information provide supplementary material on data and evaluation resources.
- Hyperparameters and Training Curves document training configuration and optimization behavior.
- Limitations and Social Impacts address scope and broader considerations of the work.
- Case Study and Prompts provide supplementary examples and prompt specifications.
A Dataset Details
The dataset construction uses LLaVA-Video-178K as source material and transforms selected video QA samples into visual-prompt-dependent data through an automated pipeline.
- Table 7 reports video source distribution, while Table 8 reports data pipeline retention statistics.
- Data Source: LLaVA-Video-178K contributes 178k videos and approximately 1.3 million instruction samples across ten mainstream video sources.Domains include activity recording, cooking, film, and first-person perspective.
- Data Source: Multi-dimensional filtering selects unedited raw videos with rich temporal variations and narrative completeness.
- Data Construction Pipeline: The four-stage pipeline starts from 147,245 raw video QA samples and transforms them into visual-prompt-dependent QA data.GPT-4o performs text filtering to remove samples unsuitable for visual prompts.
B Benchmark Information
Evaluation covers one visual-prompt-driven instance-level benchmark and three general video understanding benchmarks, spanning precise localization, long-context reasoning, and fine-grained retrieval.
- Evaluation uses four video understanding benchmarks and uniformly segments videos into 256 frames on average.
- V2P-Bench: V2P-Bench contains 980 videos and 1,172 QA pairs for visual-prompt-driven instance-level video understanding.It covers three core tasks across twelve evaluation dimensions.
- Video-MME: Video-MME contains approximately 900 manually curated videos and 2.7K multiple-choice QA pairs for long-video and complex-reasoning evaluation.The evaluation uses video modality exclusively.
- LongVideoBench: LongVideoBench contains 3,763 web videos and 6,678 human-annotated multiple-choice questions, with videos up to 60 minutes.Its referring-reasoning paradigm requires precise retrieval of relevant segments.
- LongVT: LongVT contains 244 long videos and 1,280 manually verified QA pairs in a needle-in-a-haystack setting.Supporting evidence appears only in narrow time windows, testing fine-grained localization and reasoning.
C Hyperparameters
This section identifies the SFT and RL hyperparameter tables and reports the main runtime settings used during agentic RL training.
- During agentic RL training, the authors set α = 0.8, β = 0.15, and γ = 0.05.
- RL training uses a sampling frame rate of 1, a maximum of 256 frames, and a maximum single-frame resolution of 112896.
- Table 9 lists the key hyperparameters used for supervised fine-tuning.
D Limitations and Social Impacts
The paper reports a limitation in its data construction pipeline and discusses potential accessibility benefits alongside risks from inherited and model biases.
- Limitations: VideoSeeker’s data construction pipeline relies on LLaVA-Video, so generated data may inherit that dataset’s domain bias and imbalance.
- Social Impacts: The authors identify potential accessibility benefits for visually impaired users through intuitive visual prompts.
- Social Impacts: The authors caution that model outputs may reflect training-data biases and recommend thorough evaluation before critical applications.
E Training Curves
This section presents the reinforcement-learning training curves.
- Figure 6 shows the RL training curves.
F Case Study
The case studies and prompt specifications illustrate how VideoSeeker constructs visual-prompted video QA data and adaptively selects perception tools and annotations for instance-level understanding.
- Prompt Pipeline: The text-filtering prompt removes questions unsuitable for visual prompting, including camera movement, scene-background, and counting questions.
- Case Studies: Case Study 1 shows tool invocation for proactive instance perception and retrieval of relevant video segments.
- Case Studies: Case Study 2 shows adaptive invocation of only the visual cue tool when the question requires visual-cue information.
- Prompt Pipeline: The verification prompt checks whether a question targets a specific, uniquely identifiable visual entity and produces structured metadata for visual-prompted QA construction.
- Prompt Pipeline: The pipeline localizes the target temporally, rewrites the question with <vp>, and recommends a prompt type based on target properties.
- Prompt Pipeline: The prompt types include rectangle, mask_contour, ellipse, triangle, scribble, point, arrow, and set_of_mark annotations.
- Prompt Pipeline: Rendering and rewriting replaces <vp> with a natural generic visual-prompt phrase while preserving question meaning, option semantics, and correctness.