Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoning

Motivation

• VideoDR targets open-domain questions whose answers are distributed across the web while relevant clues must be associated across video frames, a setting absent from text-query and closed-video benchmarks.Existing multimodal evaluations often use closed evidence, while deep-research benchmarks mostly start from text queries or treat visuals as static auxiliary information (Wei et al., 2025; Wu et al., 2025; Li et al., 2025b; Jiang et al., 2024).Source: §1 · p. 2

Video Deep Research Task

• VideoDR requires models to compose visual anchors across frames, search the open web interactively, and reason over joint video-web evidence to produce a unique, verifiable answer.Its annotation and quality-control process removes examples answerable from the video alone or webpages alone, keeping combined video understanding, web search, and evidence-based reasoning central to evaluation.Source: §1 · p. 2, Figure 2 · p. 3

Agent Capability Boundaries

• The paper evaluates closed- and open-source models under Workflow and Agentic paradigms and analyzes performance and errors by difficulty, video duration, and semantic domain.The evaluated models include Gemini-3.1pro-preview, GPT-5.4, Qwen3.5-35B-A3B, InternVL3.5-14B, Gemma-4-31B-it, and GLM-4.6V-Flash.Source: §1 · pp. 2–3

VideoDR Benchmark

• The benchmark uses rigorous human annotation and quality control to ensure multi-step evidence gathering remains dependent on multi-frame visual cues.VideoDR spans diverse real-world scenarios and is designed to prevent answering from video-only or webpage-only evidence.Source: §1 · p. 2

Agent Capability Boundaries

• The analysis identifies Goal Drift and Long-horizon Consistency as core bottlenecks for next-generation video deep research agents.The paper examines performance and error patterns across paradigms and VideoDR’s distributions of semantic domains, question lengths, and video durations.Source: §1 · p. 3

Deep Research Benchmarks

• Most deep research benchmarks begin with textual queries, treating visual content as auxiliary rather than first-class evidence in retrieval and verification.They assess search, reasoning, and tool use together, including live-web query planning and more controllable settings for reliability and stability (Wei et al., 2025; Wu et al., 2025; Xue et al., 2025; Chen et al., 2025b; Jin et al., 2025; Li et al., 2025a).Source: §2 · p. 3

Video Reasoning Benchmarks

• Video reasoning benchmarks use closed evidence, testing questions answerable from the video and emphasizing long-video temporal understanding and long-context reasoning.(Fu et al., 2025; Li et al., 2024a; Wu et al., 2024; Zhou et al., 2025; Wang et al., 2025a; Yang et al., 2025a; Li et al., 2024b; Nagrani et al., 2024; Yu et al., 2025).Source: §2 · p. 3
• Although recent agentic video work explores multi-round interaction and tool calling, systematic evaluation of open-web evidence gathering and reasoning anchored to video cues remains scarce.(Zhang et al., 2025; Yang et al., 2025b; Chen et al., 2025a).Source: §2 · p. 3

3.1 Task Definition

• VideoDR asks models to use multi-frame cues from a video to search the open web iteratively and derive a unique factual answer through multi-hop reasoning over both sources.After receiving the video and question, models may use browser search but cannot repeatedly re-watch the video.Source: §3.1 · p. 4

3.2 Data Annotation Process

• Three annotators built questions around multi-frame visual anchors, using stratified video selection and filtering to target complex, verifiable video-web reasoning.Questions had to require cues across multiple frames and at least one exchange between video perception and external search; annotators archived web pages supporting each answer.Source: §3.2 · p. 4

3.3 Quality Control

• VideoDR uses a two-stage quality-verification procedure to ensure sample rigor and unambiguous answers.The subsequent checks test video-and-web dependence and use blind human evaluation to validate solvability and annotation correctness.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 5

3.3.1 Video & Web Dependency Testing

• A sample is retained only if it cannot be answered uniquely from web search alone or directly from the video alone.Web-only answers indicate insufficient dependence on visual anchors; video-only answers indicate the question is merely a video-understanding task.Source: §3.3.1 · pp. 4–5

3.3.2 Human Testing

• 0.4022 was the mean sample-level human success rate across 500 samples, with a standard deviation of 0.298.Five blind participants attempted each question using the video, question, and autonomous web search. Difficulty was High for 0–1 correct participants, Mid for 2–3, and Low for 4–5; mean success rates were 0.817 ± 0.105, 0.496 ± 0.091, and 0.114 ± 0.117, respectively. Ambiguous samples were discarded after re-examination.Source: §3.3.2 · pp. 5–6, Figure 3 · p. 5

3.4 Data Statistics

• 500 samples span six domains, led by Daily Life (33.6%) and Technology (22.6%), with Culture (13.0%), History (11.8%), Economics (11.4%), and Geography (7.6%).The category distribution is described as relatively balanced and is presented in Figure 4(a).Source: §3.4 · p. 6, Figure 4 · p. 6
• 25.54 tokens was the average question length, and the 95th percentile was 54 tokens.Question lengths are concentrated, keeping questions compact while retaining necessary constraints.Source: §3.4 · p. 6, Figure 4 · p. 6
• 37.4% of videos were Long, 32.4% Short, and 30.2% Medium, while durations followed a long-tailed distribution.This mix covers short-clip cue capture and cross-segment association in longer videos.Source: §3.4 · p. 6, Figure 4 · p. 6

4.1 Setting

• Workflow first extracts video cues into structured text for later retrieval and reasoning, whereas Agentic handles video understanding, query generation, retrieval, and evidence integration in one loop.The study tests six closed- and open-source multimodal models under both paradigms and uses a DeepSeek-V3-0324 judge to assign binary semantic-equivalence correctness labels for Accuracy.Source: §4.1 · pp. 6–7

4.2 Main Results

• As difficulty rose, all models generally declined, while Agentic’s effects varied by model and difficulty rather than reliably improving harder cases.Gemini improved on Low (78.57%→84.82%) and High (28.57%→30.41%) but fell on Mid (63.74%→62.57%); the authors link drift risk to Agentic’s inability to re-watch video, unlike Workflow’s reusable text anchors.Source: Table 1 · p. 7, §4.2 · p. 8
• 58.29% versus 54.01% on Long videos was Gemini-3.1-pro-preview’s largest reported duration-group Agentic gain, but results across models and durations were heterogeneous.The authors attribute the polarization to longer videos dispersing cues and increasing demands on retaining initial anchors; Workflow offers controllable textual anchors, whereas Agentic relies on long-term state consistency.Source: Table 2 · p. 7, §4.2 · pp. 8–9
• 53.60% under Agentic and 51.80% under Workflow made Gemini-3.1-pro-preview the top model; GPT-5.4 ranked second at 47.40% and 47.00%, respectively.Only these two models exceeded the 40.22% human average under this evaluation setting.Source: §4.2 · pp. 7–8
• 59.32% versus 52.54% in History, 54.87% versus 50.44% in Technology, and 60.12% versus 57.74% in Daily Life were Gemini’s Agentic gains, while effects varied across models and domains.The authors associate these trade-offs with video-memory stability and domain-specific ambiguity in downstream search.Source: Table 3 · p. 8, §4.2 · p. 9

4.3 Error Analysis

• Categorical errors remained high for several models, while GLM-4.6V-Flash had 101 Workflow and 96 Agentic reasoning errors, indicating different failure stages across models.The authors connect error propagation to first-pass visual anchors that cannot be corrected by revisiting the video during research.Source: Table 4 · p. 9, §4.3 · pp. 9–10
• 84 numerical errors for Gemini-3.1-pro-preview under Workflow and 95 under Agentic show that numerical errors persisted even for the model with the lowest overall error count.Across models, these errors remained substantial, particularly for precise counting, comparison, or numerical verification.Source: Table 4 · p. 9, §4.3 · p. 10

• VideoDR requires models to extract multi-frame visual anchors, turn them into searchable queries, retrieve open-web evidence, and use multi-hop reasoning to produce verifiable factual answers.Source: §5 · p. 10
• The benchmark evaluates mainstream multimodal models under both Workflow and Agentic paradigms.Source: §5 · p. 10
• Stratified analyses by difficulty, video duration, and semantic domain characterize models’ capability boundaries on video deep search.Source: §5 · p. 10

A Retrieval-and-Reasoning Analysis

• 53.60% overall, Gemini-3.1-pro-preview’s best score, came under Agentic with 4.55/3.16 reasoning/search calls and 345.73s runtime.The passage interprets this as effective conversion of retrieval and reflection into reliable evidence integration.Source: A Retrieval-and-Reasoning Analysis · p. 13
• GLM-4.6V-Flash had the longest runtime under both paradigms—511.98s in Workflow and 502.96s in Agentic—yet scored 17.80% and 21.80% accuracy, respectively.Longer execution time did not necessarily yield better evidence acquisition or answer correctness.Source: A Retrieval-and-Reasoning Analysis · p. 13
• Qwen3.5-35B-A3B used fewer reasoning/search calls and less runtime in Agentic than Workflow, but its accuracy slightly fell from 35.00% to 34.60%.The respective reasoning/search calls were 1.59/3.02 versus 2.97/3.80, and runtime was 374.47s versus 485.57s.Source: A Retrieval-and-Reasoning Analysis · p. 13
• Gemma-4-31B-it’s Agentic accuracy rose from 31.60% to 33.60% as reasoning calls and runtime increased while search calls decreased.Reasoning calls changed from 3.69 to 3.84, search calls from 2.00 to 1.70, and runtime from 232.28s to 285.67s.Source: A Retrieval-and-Reasoning Analysis · p. 13

B Token Usage Analysis

• Agentic increases average total tokens for Qwen3.5-35B-A3B (168,483 to 301,187) and Gemini-3.1-pro-preview (80,271 to 199,008), while GPT-5.4 decreases (109,236 to 101,936).GLM-4.6V-Flash also increases from 56,839 to 96,329 average total tokens.Source: B Token Usage Analysis · p. 13
• Prompt tokens dominate Agentic’s overall token increase, while completion tokens remain relatively stable or decrease in several settings.The passage attributes this burden to maintaining larger contextual states rather than generating longer responses.Source: B Token Usage Analysis · p. 13
• High P95 values, especially for Qwen3.5-35B-A3B under Agentic, indicate occasional long search-and-reasoning trajectories and less predictable token costs.Source: B Token Usage Analysis · p. 13

C Efficiency Analysis

• Gemini-3.1-pro-preview reaches 53.60% Agentic accuracy versus 51.80% under Workflow, but runtime rises from 188.20s to 345.73s and accuracy per 1K tokens falls from 0.65 to 0.27.The Agentic setting yields the best accuracy, but with substantially higher runtime and lower token efficiency.Source: C Efficiency Analysis · p. 13
• Gemma-4-31B-it improves accuracy from 31.60% to 33.60% under Agentic, while increasing runtime and tokens per correct sample.Source: C Efficiency Analysis · p. 13
• Agentic is not uniformly more efficient than Workflow; its practical value depends on converting autonomous reasoning into accuracy gains without excessive token or latency overhead.Efficiency varies substantially across models, and higher accuracy does not necessarily imply higher efficiency.Source: C Efficiency Analysis · pp. 13–14, Table 7 · p. 14
• GPT-5.4 has the most favorable Agentic efficiency profile: accuracy rises from 47.00% to 47.40%, while runtime falls from 178.84s to 166.32s.Accuracy per 1K tokens and accuracy per minute also improve.Source: C Efficiency Analysis · p. 14
• For weaker models, Agentic may improve speed without improving cost-effectiveness; Qwen3.5-35B-A3B’s tokens per correct sample rise from 481,379 to 870,483.Source: C Efficiency Analysis · p. 14

D Annotation Effort

• VideoDR annotation requires repeated video inspection, multi-frame anchor identification, multi-hop web-question design, evidence archiving, and filtering out video-only or web-only shortcuts.Source: D Annotation Effort · p. 14
• Although time-consuming and limiting rapid scaling, this process is necessary to retain questions with unique, verifiable answers that test video-grounded deep research.The filtering is intended to exclude questions answerable through simple video understanding or text-based web search alone.Source: D Annotation Effort · p. 14

E Data

• The release includes benchmark annotations and structured fields for reconstructing samples, including question-answer pairs, video references, evidence references, sample metadata, source URLs, and evaluation scripts.Third-party raw videos or webpages are redistributed only when their licenses or terms of use explicitly permit it.Source: E Data · pp. 14–15
• Local copies of annotated videos and evidence webpages are preserved for reproducibility and long-term auditing, with source references and archived evidence records associated with each benchmark item.The public release focuses on annotations, metadata, source URLs, evidence references, and evaluation scripts.Source: E Data · pp. 14–15

F Previous Version Experimental Results

• The section presents experimental tables from the manuscript’s previous version (v1).Source: F Previous Version Experimental Results · p. 15

F.1 Overall Performance by Difficulty

• Table 8 groups v1 Workflow and Agentic accuracy by Low, Mid, and High difficulty, defined by human success rate, to examine performance as evidence chains grow longer and more fragile.Source: §F.1 · p. 15

F.2 Performance by Video Duration

• Table 9 groups v1 results by video duration to study whether longer videos hinder preservation and reuse of initial visual anchors during web search and reasoning.Source: §F.2 · p. 15

F.3 Performance by Domain

• Table 10 compares v1 Workflow and Agentic performance across six domains to examine whether semantic domain relates to different search-drift patterns.Source: §F.3 · p. 15, Table 10 · p. 16
• The domain comparison covers History, Geography, Culture, Economy, Technology, and Daily Life questions.Source: §F.3 · p. 15

F.4 Tool-use Statistics

• Table 11 reports v1 average think calls, search calls, and runtime for both paradigms, probing whether performance reflects tool-use frequency or more effective evidence acquisition and filtering.Source: §F.4 · p. 16, Table 11 · p. 16

F.5 Error Type Distribution

• Table 12 reports v1 errors across categorical, incomplete, not-found, numerical, context, semantic, reasoning, and other types for different models under Workflow and Agentic.Source: §F.5 · p. 16, Table 12 · p. 17
• The breakdown was used to diagnose which failure modes dominated in the previous experimental setting.Source: §F.5 · p. 16
• The analysis examined how error types changed between Workflow and Agentic runs.Source: §F.5 · p. 16