Source-linked AI summary
Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents
Wenqi Liu, Shijie Ma, Yunxiao Wang, Meng Liu, Qile Su, Han Liu, Bohan Hou, Xuanyu Zheng, Changyi Liu, Tianke Zhang, Haonan Fan, Kaiyu Jiang, Yingxin Li, Jiankang Chen, Xu Wang, Bin Wen, Tingting Gao, Han Li, Jianhua Yin, Yinwei Wei, Xuemeng Song
TL;DR
Open-world video understanding needs both sparse visual grounding and external knowledge, yet Thinking-with-Videos and Deep Research are usually developed separately. VideoRover unifies them through iterative video cropping, multimodal search, and webpage browsing, with each result guiding the next action; its 8B RL model performs comparably to proprietary models in direct answering and surpasses larger open-source tool-using models.
Problem
Open-world video questions require linking sparse visual evidence to external knowledge absent from the video or the model’s parametric memory.
Method
VideoRover coordinates video grounding, multimodal retrieval, webpage browsing, and evidence synthesis through a shared iterative research process.
Results
VideoRover-8B-RL achieves performance comparable to proprietary models under direct answering while outperforming newer, larger open-source models using the same tools.
Takeaways & Limitations
The results support combining active video grounding, external retrieval, and long-horizon reinforcement learning for Video Deep Research.
Abstract
from arXiv · showhide
Open-world video understanding often requires a model to locate sparse visual evidence and acquire external knowledge that is absent from the video and its parametric memory. While Thinking-with-Videos enables active temporal perception and Deep Research supports multi-step information seeking, the two capabilities are typically developed in isolation. We introduce VideoRover, a unified Video Deep Research framework that iteratively coordinates video cropping, multimodal search, and webpage browsing. Given a video-question pair, VideoRover uses each tool result to select the next action, so localized video clips guide external retrieval and retrieved evidence triggers further video inspection and verification. To develop this capability, we construct an automated data curation pipeline, producing 26K verified SFT trajectories and 3K challenging RL instances. We also introduce VideoRover-Bench, a benchmark stratified by video duration and research difficulty. Experiments on VideoDR and VideoRover-Bench show that our VideoRover-8B-RL achieves performance comparable to proprietary models in the direct-answer setting without tool use while outperforming larger open-source models equipped with the same tool suite. Ablation studies and training dynamics further validate the complementary roles of active video grounding, external retrieval, and long-horizon reinforcement learning.
1 INTRODUCTION
VideoRover addresses open-world video tasks that require both sparse visual grounding and external knowledge by iteratively coordinating video inspection, multimodal retrieval, and webpage browsing.
- Open-world video tasks require fine-grained visual evidence and knowledge beyond fixed inputs or parametric memory.
- One-way pipelines cannot use retrieved evidence to revisit uncertain video content, motivating bidirectional coordination between observation and retrieval.
- VideoRover uses tool results to update a shared research state, selecting among video re-examination, image retrieval, text search, webpage inspection, and answering.
- 26K verified SFT examples and 3K challenging RL instances provide supervision for Video Deep Research behaviors.
- VideoRover-Bench stratifies evaluation by video duration, task difficulty, and reasoning complexity.
- Experiments and ablations validate strong Video Deep Research performance and the complementary roles of tool design and long-horizon RL coordination.
2 RELATED WORK
Prior work separately advances adaptive visual reasoning and iterative multimodal or web retrieval, but these lines of research remain distinct from unified video-centered deep research.
- Long-video understanding methods improve context capacity, token efficiency, or hierarchical modeling but generally fix visual inputs before inference.
- Tool-augmented visual methods treat visual content as an active reasoning workspace, enabling observation to adapt during inference.
- Early search agents support iterative retrieval and reasoning but remain primarily text-centric.
- Recent multimodal systems add visual search and interaction, while other efforts support long-horizon multi-tool visual research.
3 TASK FORMULATION
Video Deep Research formulates a video-question task whose solution combines relevant video evidence with missing external knowledge through iterative stateful tool use.
- A task instance consists of an N-frame video V and a research question Q, with the goal of producing an answer Y.
- The agent maintains accumulated video and web evidence together with interaction history as its research state.
- VideoRover crops selected intervals, retrieves visually related information from keyframes, searches text queries, and visits webpages for detailed evidence.
- Tools may be invoked repeatedly until the agent executes the terminal answer action.
- An interaction trajectory records the initial state, alternating actions and observations, and the final answer.
- The process terminates when accumulated evidence supports an answer or the interaction budget Tmax is reached.
4 DATA SYNTHESIS AND BENCHMARK CONSTRUCTION
The paper builds Video Deep Research data by coupling reliable visual grounding with external evidence collection, then verifies and stratifies tasks into a benchmark spanning video length and research difficulty.
- Question Construction: The data pipeline constructs tasks in which video identifies an indirectly referenced target while external web knowledge supplies information needed for the answer.Neither video-only reasoning nor question-only web search is sufficient.
- Question Construction: Candidate targets are extracted from Wikipedia with supporting text and source URLs, then paired with YouTube videos whose visual grounding is independently verified.Qwen3.5-27B localizes intervals and keyframes from densely sampled frames and subtitles, while GPT-5.4-mini verifies the selected keyframe against target identity.
- Question Construction: Questions refer to targets indirectly and are filtered out when a video-only model answers them correctly, ensuring dependence on both video evidence and external retrieval.The generated questions ask about related background, history, or recent developments while withholding explicit target identity.
- Trajectory Generation: Trajectory synthesis uses a text-only planner and a video observer that first localizes and crops a candidate segment before dense visual analysis grounds subsequent retrieval.The observer supplies temporally grounded observations about entities, actions, text, and temporal cues.
- Trajectory Generation: Subsequent actions follow the evidence gap: the planner crops additional segments, uses image search for visual identity, and uses text search for missing background or current facts.Retrieved evidence updates the research state before the system answers.
- Training Data: 26K SFT trajectories and 3K RL instances are produced by retaining verified shorter interactions for supervision and balancing challenging longer examples for reinforcement learning.SFT data contain at most 10 tool calls, while RL candidates require more than 10 tool calls and are filtered and balanced by difficulty factors.
- Benchmark Construction: VideoRover-Bench contains 300 balanced evaluation instances across two video-length subsets and three temporal-grounding or retrieval difficulty levels.The length categories are short (0–10 minutes) and long (>10 minutes), with 50 examples in each of six combinations.
5 MODEL TRAINING
VideoRover training combines supervised learning on verified tool-use trajectories with reinforcement learning on challenging instances, moving from stable research behavior toward adaptive decisions under tool feedback.
- Training Strategy: SFT establishes a reliable tool-use prior linking video observation with external retrieval, while RL improves autonomous decisions using tool feedback and final-answer supervision.The two stages are designed to provide complementary training signals.
- SFT Training: SFT fine-tunes the base model on verified multi-round trajectories demonstrating localization, search selection, webpage inspection, evidence updates, and stopping decisions.These trajectories provide a stable initialization for the complete Video Deep Research process, including valid tool invocation.
- SFT Training: SFT excludes tool outputs from the loss and trains on model-generated reasoning, tool calls, and answer tokens for each target trajectory.The task input consists of the video-question pair, while tool outputs serve as environmental context.
- RL Training: GSPO trains on challenging data by sampling eight trajectories per input, rewarding correct answers with 1 and incorrect answers with 0 while reinforcing complete research trajectories.The policy can explore alternative observation, retrieval, and verification decisions under final-answer supervision.
- RL Training: GSPO balances trajectories by averaging valid-token objectives within each trajectory and then across trajectories, preventing longer outputs from dominating optimization.The resulting policy learns when to gather more evidence, revisit the video, or answer.
6 EXPERIMENT
Experiments evaluate VideoRover across direct answering, agentic tool use, ablations, and RL training dynamics. VideoRover-8B-RL performs strongly among open-source agents, while results show that coordinated tool composition and long-horizon RL matter.
- Main Results: VideoRover is evaluated on VideoDR and VideoRover-Bench against proprietary direct-answering models and open-source models with direct or ReAct-style tool use.The benchmark reports accuracy across short and long videos and six VideoRover-Bench subsets.
- Main Results: Open-source models equipped with the same tools substantially outperform their direct-answer counterparts.Tool use enables revisiting segments, verifying visual entities, retrieving missing knowledge, and inspecting source webpages.
- Main Results: Tool access alone is insufficient because effective Video Deep Research requires selecting and composing tools according to the evolving evidence state.The comparison distinguishes structured coordination from an unstructured tool pipeline.
- Main Results: VideoRover-8B-RL achieves the strongest overall performance among evaluated open-source agentic models and leads or matches them on most subsets.It reaches performance comparable to the strongest proprietary models under direct answering and surpasses newer, larger open-source models using the same tools.
- Ablation Studies: Removing crop video, image search, text search, or webpage visits reduces performance, with text search producing the largest overall decline.The joint removal of image search, text search, and webpage visits causes a substantially larger degradation than removing any single tool.
- RL Training Dynamics: After the SFT cold start, smoothed reward and interaction length both increase, suggesting that longer rollouts support sustained evidence acquisition.The authors interpret the trend, together with ablations, as evidence that RL strengthens long-horizon coordination.
7 CONCLUSION
The paper presents VideoRover as a unified framework for iterative video observation and open-world retrieval. Its experiments, ablations, and training dynamics support the roles of complementary tools and RL-based long-horizon coordination.
- Conclusion: VideoRover iteratively coordinates video observation and open-world retrieval, using each result to guide the next action.The framework is developed and evaluated with verified SFT and RL data and VideoRover-Bench across video durations and research difficulties.
- Conclusion: VideoRover-8B-RL performs comparably to proprietary models under direct answering and outperforms newer, larger open-source models equipped with the same tools.These results are reported on VideoDR and VideoRover-Bench.
- Conclusion: Ablations and training dynamics demonstrate complementary roles for video grounding and external retrieval, along with the importance of RL for long-horizon coordination.The conclusion links these findings to the system’s unified research process.
A EXPERIMENTAL DETAILS
The experimental setup uses an 8B vision-language backbone trained with SFT followed by RL and evaluates direct-answering and agentic baselines under a shared protocol.
- Training Details: During RL, each training example produces eight rollout trajectories, with each rollout limited to at most 25 tool calls.Initial video observation samples at 1 FPS, retains at most 256 frames, and limits each frame to 224 × 224 pixels.
- Evaluation Details: Evaluation compares proprietary direct-answering baselines with open-source baselines under both direct answering and ReAct-style agentic tool use.Agentic models autonomously invoke the same tools under a shared evaluation protocol.
B QUALITATIVE CASES
The qualitative cases show VideoRover grounding external research in localized video evidence and adapting its searches when the evidence gap changes.
- Qualitative Cases: In both qualitative examples, VideoRover first localizes and crops a relevant segment, then uses visual cues to guide search and webpage verification.The trajectories illustrate coordination between video grounding and external retrieval.
- World Cup Case: In the World Cup case, VideoRover inspects interview and match segments, combines image and text search, and visits a webpage to verify the answer.The task requires identifying the player and retrieving his World Cup scoring history.
- Roman Coin Case: In the Roman coin case, VideoRover switches from unsuccessful text searches to image search on a localized keyframe before visiting a webpage.The webpage is used to confirm the historical relationship associated with the relevant gens.