Source-linked AI summary
Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoning
Chengwen Liu, Xiaomin Yu, Zhuoyue Chang, Zhe Huang, Shuo Zhang, Heng Lian, Jisheng Dang, Rui Xu, Sen Hu, Jianheng Hou, Chengwei Qin, Xiaobin Hu, Kunyi Wang, Zhi Yang, Hao Peng, Hong Peng, Ronghao Chen, Huacan Wang
TL;DR
VideoDR addresses the lack of benchmarks for questions whose clues are distributed across video frames while answers require open-web verification. It constructs and evaluates a video-conditioned benchmark combining visual-anchor extraction, interactive retrieval, and multi-hop reasoning, finding that Agentic is not consistently better than Workflow and that goal drift and long-horizon consistency remain bottlenecks.
Problem
Existing evaluations do not systematically test using cross-frame video clues to verify open-domain facts distributed across the web.
Method
VideoDR combines multi-frame visual-anchor extraction, interactive web search, and multi-hop reasoning over joint video-web evidence, with human testing supporting difficulty stratification.
Results
Agentic performance varies across models, difficulty levels, video durations, and domains rather than consistently surpassing Workflow.
Takeaways & Limitations
Goal Drift and Long-horizon Consistency are the core bottlenecks identified for next-generation video deep research agents.
Abstract
from arXiv · showhide
In real-world video question answering scenarios, videos often provide only localized visual cues, while verifiable answers are distributed across the open web; models therefore need to jointly perform cross-frame clue extraction, iterative retrieval, and multi-hop reasoning-based verification. To bridge this gap, we construct the first video deep research benchmark, VideoDR. VideoDR centers on video-conditioned open-domain video question answering, requiring cross-frame visual anchor extraction, interactive web retrieval, and multi-hop reasoning over joint video-web evidence; through rigorous human annotation and quality control, we obtain high-quality video deep research samples spanning six semantic domains. We evaluate multiple closed-source and open-source multimodal large language models under both the Workflow and Agentic paradigms, and the results show that Agentic is not consistently superior to Workflow: its gains depend on a model's ability to maintain the initial video anchors over long retrieval chains. Further analysis indicates that goal drift and long-horizon consistency are the core bottlenecks. In sum, VideoDR provides a systematic benchmark for studying video agents in open-web settings and reveals the key challenges for next-generation video deep research agents.
1 Introduction
VideoDR addresses a gap between video benchmarks that use closed evidence and deep-research benchmarks that begin with text. It evaluates open-domain question answering requiring video cues, web search, and joint evidence reasoning.
- Research gap: Existing video evaluations largely use closed evidence, while deep-research benchmarks mostly begin with textual queries.Neither setting systematically characterizes using video clues to verify and synthesize facts from open webpages.
- VideoDR task: VideoDR defines video-conditioned open-domain factoid answering that combines multi-frame visual anchors, browser search, and multi-hop reasoning.The final answer must be unique and verifiable from joint video-web evidence.
- Benchmark construction: The benchmark uses human annotation and strict quality control to ensure answers depend on combined video understanding, web search, and evidence-based reasoning.Samples answerable from the video alone or webpages alone are explicitly removed.
Video Deep Research
VideoDR illustrates video deep research through a museum question requiring visual identification, multi-hop web search, and retrieval of an accession number. The benchmark also studies capability boundaries across agentic approaches.
- Video Deep Research: A representative task identifies a museum from video cues, finds the closest recommended exhibit to the entrance, and returns its accession number.The example answer is WB.67.
- Agent Capability Boundaries: VideoDR analysis compares Workflow and Agentic paradigms to characterize their performance and error patterns.The benchmark uses varied semantic domains, question lengths, and video durations for this analysis.
- Agent Capability Boundaries: Goal Drift and Long-horizon Consistency are identified as core bottlenecks for next-generation video deep research agents.
2 Related Work
Related benchmarks separately study deep research from textual queries and video reasoning within closed evidence. VideoDR combines these capabilities by conditioning open-web research on video.
- Deep Research Benchmarks: Deep-research benchmarks evaluate search, reasoning, and tool use through multi-step query planning, information localization, and reliability-oriented settings.Most still start from textual queries.
- Video Reasoning Benchmarks: Video reasoning benchmarks typically require answering from the video alone while stressing long-video temporal understanding and long-context reasoning.Recent agentic video efforts also explore interaction and tool calling within video-understanding loops.
3 Video Deep Research
VideoDR is an open-domain factoid question-answering benchmark conditioned on video, requiring multi-frame visual reasoning, interactive web search, and multi-hop evidence integration. It contains 500 quality-checked samples spanning diverse domains, question lengths, durations, and human difficulty levels.
- Benchmark design: VideoDR evaluates factual question answering from a given video while models iteratively search the open web for supporting evidence.The benchmark requires reasoning anchored in video cues while leveraging browser-based search.
- Benchmark design: Annotators construct questions around multiple visual cues and enforce both multi-frame and multi-hop reasoning requirements.Questions must depend on cues spanning multiple frames and admit a decomposable multi-step reasoning path.
- Quality control: Two ablation tests remove samples answerable through web search alone or video alone, preserving dependence on both evidence sources.The web-only test filters questions solvable without visual anchors, while the video-only test filters questions solvable directly from the video.
- Data statistics: 500 samples span six semantic domains, with Daily Life largest at 33.6% and Geography smallest at 7.6%.Technology accounts for 22.6%, Culture 13.0%, History 11.8%, and Economics 11.4%.
- Data statistics: The average question length is 25.54 tokens, with the 95th percentile at 54 tokens.Question lengths are concentrated, keeping phrasing concise while preserving necessary constraints.
- Data statistics: Video durations are distributed across short, medium, and long groups at 32.4%, 30.2%, and 37.4%, respectively.The long-tailed duration distribution supports testing both rapid cue capture and cross-segment association.
4 Experiments
Experiments compare Workflow and Agentic paradigms across models, difficulty, duration, domains, and error types. Agentic is not uniformly superior; performance depends on preserving video anchors over long search and reasoning chains.
- Experimental Setting: Workflow extracts cross-frame cues into structured text before web search, whereas Agentic retains direct access to the raw video during end-to-end retrieval and reasoning.
- Experimental Setting: LLM-as-judge assigns binary semantic-equivalence labels used to compute overall Accuracy for open-domain factual answers.
- Main Results: 51.80% under Workflow and 53.60% under Agentic make Gemini-3.1-pro-preview the leading model, while only Gemini-3.1-pro-preview and GPT-5.4 surpass the 40.22% human average.
- Difficulty Stratification: Performance declines consistently from Low to Mid to High difficulty, while Agentic's gains vary by model and difficulty level.Gemini improves on Low and High but drops on Mid; GPT-5.4 improves on Low/Mid but drops on High.
- Duration and Domain Stratification: Agentic produces heterogeneous results across video durations and domains rather than uniformly benefiting from longer videos or richer inputs.Longer videos increase demands on retaining and reusing initial cues, while domain effects vary across models.
- Error Analysis: Goal drift and long-horizon consistency constrain Agentic, whereas Workflow's intermediate text supplies more stable anchors for downstream search and reasoning.Once early visual anchors drift, web noise can amplify later deviations, especially on high-difficulty questions.
- Error Analysis: Categorical errors remain prominent across representative models, while numerical errors are especially notable even for stronger models.Gemini records 84 numerical errors under Workflow and 95 under Agentic despite the lowest overall error counts.
5 Conclusion
VideoDR is introduced as a benchmark for video-conditioned open-web factual question answering. It evaluates visual-anchor extraction, web evidence retrieval, and multi-hop reasoning across stratified video deep-search settings.
- 5 Conclusion: VideoDR requires models to extract multi-frame visual anchors, convert them into searchable queries, retrieve open-web evidence, and perform multi-hop reasoning for verifiable factual answers.
- 5 Conclusion: The benchmark compares Workflow and Agentic paradigms and stratifies analysis by difficulty, video duration, and semantic domain.
- 5 Conclusion: VideoDR characterizes capability boundaries for video deep search through systematic performance and error-pattern analyses across these settings.
A Retrieval-and-Reasoning Analysis
Retrieval-and-reasoning analysis shows that search volume and execution time alone do not determine accuracy. Stronger models convert retrieval and reflection into more reliable evidence integration, whereas longer runs can still yield low accuracy.
- A Retrieval-and-Reasoning Analysis: Search calls and reasoning steps are not monotonically correlated with accuracy.
- A Retrieval-and-Reasoning Analysis: 53.60% accuracy accompanies Gemini-3.1-pro-preview's Agentic average of 4.55/3.16 reasoning/search calls and 345.73s runtime.
- A Retrieval-and-Reasoning Analysis: 511.98s in Workflow and 502.96s in Agentic still produce only 17.80% / 21.80% accuracy for GLM-4.6V-Flash.
B Token Usage Analysis
Agentic generally consumes more tokens than Workflow, mainly because it maintains larger video-conditioned retrieval and reasoning contexts.
- Agentic generally increases total token consumption, especially prompt tokens, compared with Workflow.The main efficiency burden comes from maintaining larger contextual states rather than generating longer responses.
C Efficiency Analysis
Agentic efficiency varies substantially by model: accuracy gains can require more runtime and tokens, while some models remain stable or improve efficiency. Constructing VideoDR also requires substantial manual effort.
- 53.60% Agentic accuracy for Gemini-3.1-pro-preview comes with runtime rising from 188.20s to 345.73s and token efficiency falling from 0.65 to 0.27.
- GPT-5.4 slightly improves accuracy from 47.00% to 47.40% while reducing runtime from 178.84s to 166.32s.Its Agentic setting also improves both accuracy per 1K tokens and accuracy per minute.
- VideoDR construction is time-consuming because every sample must satisfy both video dependency and web dependency.Annotators inspect videos, identify multi-frame anchors, design multi-hop questions, archive evidence, and filter shortcut-answerable samples.
E Data
The release provides benchmark annotations and reconstruction metadata, while accompanying tables document prior-version performance, tool use, and error analyses.
- The open-source release includes question-answer pairs, video references, evidence references, and sample-level metadata.
- Archived videos and evidence webpages support reproducibility and long-term auditing, while redistribution follows licensing or terms-of-use constraints.
- The v1 tables compare Workflow and Agentic performance across difficulty levels, video durations, and semantic domains.
- The v1 analysis also reports tool-use statistics and error-type distributions for diagnosing retrieval behavior and failure modes.
F.5 Error Type Distribution
The v1 error analysis categorizes failures across eight error types to identify dominant failure modes and compare their changes between Workflow and Agentic runs.
- Table 12 breaks errors into categorical, incomplete, not-found, numerical, context, semantic, reasoning, and other types.The breakdown diagnoses which failure modes dominated and how they changed between paradigms.