PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports

Jiahao Meng, Qize Yang, Yuxuan Wang, Yunfei Chu, Wensi Dai, Jin Xu, Yunhai Tong

arXiv:2610.31158v1cs.CVcs.CL

TL;DR

Existing benchmarks do not evaluate complete research reports organized around a source video, despite the video-specific context users may need to inspect and verify. The paper defines this task and presents a skill-based workflow, Omni-ARTs, ARC, and OmniDR-Bench. Across 18 configurations, the best system scores 63.3 out of 100, while presentation scores exceed content and citation scores.

  • Problem

    Existing benchmarks do not evaluate complete research reports organized around one source video and enriched by external multimodal evidence, although video context can preserve details useful for verification and practical decisions.

    Source: §1 · p. 1

  • Method

    The paper defines video-centered report generation and presents a skill-based workflow, Omni-ARTs, and ARC for constructing an audited multimodal benchmark.

    Source: §1 · p. 2

  • Results

    Across 18 configurations, the best system scores 63.3 out of 100, while presentation averages 81.8 points versus 45.4 for content and 47.8 for citation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark exposes gaps in coverage, cross-modal reasoning, and claim support, with future work targeting audiovisual use, efficiency, reasoning, and citations.

    Source: §6 · p. 9

    OmniDR-Bench is not population representative, and its reports should not replace professional advice.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Deep research agents synthesize open-web evidence into long reports, but current benchmarks do not evaluate research centered on a source video. We introduce Video-Centered Deep Research Report Generation. Given a video and user request, an agent identifies research questions, retrieves external multimodal evidence, and produces an interactive report linked to relevant video moments. This task has direct practical value because it helps users locate relevant moments, verify claims, reproduce demonstrated procedures, and make informed decisions from video content. We present a skill-based workflow and OmniDR-Bench for this task. For evaluation, we propose Omni Atomic Rubric Trees (Omni-ARTs). Each tree contains three dimensions, 23 shared directions, and task-specific atomic binary criteria. The directions test audiovisual grounding, multimodal evidence use, and interactive presentation. Audited Rubric Construction (ARC) builds these trees through independent evidence collection, automated checks, behavioral calibration, and iterative human review. OmniDR-Bench contains 50 Chinese and English tasks across 19 realworld subcategories. It includes 6,414 criteria. Across 18 agent configurations, the best system scores 63.3 out of 100. Current agents still need broader content coverage, stronger causal and logical analysis across modalities, and more accurate citation support. We further analyze models, harnesses, skill configurations, video categories, and research trajectories. The code and data are released at https://github.com/marinero4972/OmniDR-Bench.

1 INTRODUCTION

Video’s audiovisual and temporal context can raise research questions that broad text topics may not expose, yet existing benchmarks do not evaluate complete reports organized around a source video. The paper introduces a video-centered report task, evaluation framework, benchmark, and reference workflow; experiments show current agents perform more consistently on presentation than on research substance.

1 INTRODUCTION

  • Video preserves situational context, visible entities, spatial relations, and fine-grained actions that can prompt questions about mechanisms, credibility, alternatives, and practical use.These needs motivate research that remains centered on the source video, rather than treating video as incidental context.Source: §1 · p. 1
  • Existing settings cover text-centered reports, short-answer video research, or text-image reports, but none evaluates a complete report organized around one source video and enriched by external multimodal evidence.The cited works are presented as examples of these settings: text-centered agents (Java et al., 2025; Li et al., 2025; Qiao et al., 2025), video deep research (Liang et al., 2025; Liu et al., 2026a; Gao et al., 2026; Fang et al., 2026; Liu et al., 2026b; Zhang et al., 2026b), and text-image reports (Shi et al., 2026; Zhang et al., 2026a; Ma et al., 2026).Source: §1 · pp. 1–2
  • The proposed task connects user-requested research to source-video moments and external webpages, documents, images, and related videos in an interactive HTML report.Its reference workflow uses specialized media readers, anchor-conditioned research, evidence verification, and interactive rendering.Source: §1 · p. 2
  • Omni-ARTs and ARC provide fine-grained report evaluation and audited benchmark construction, alongside OmniDR-Bench and a reference agent workflow.The paper also analyzes models, agent harnesses, skill configurations, video categories, and research trajectories.Source: §1 · p. 2
  • 63.3 out of 100 is the strongest system score across 18 configurations; presentation averages 81.8 points, versus 45.4 for content and 47.8 for citation.Under the shared report template, agents satisfy presentation requirements more consistently than audited coverage, reasoning, and citation-support criteria.Source: §1 · p. 2

2 VIDEO-CENTERED DEEP RESEARCH

Video-Centered Deep Research Report Generation takes a source video, a user request, and an open tool environment as inputs, then produces an interactive report. Its workflow connects request-relevant video moments to external evidence and validates and repairs the resulting report.

2.1 TASK DEFINITION

  • The task takes a source video V, user request Q, and open tool environment E, and produces an interactive HTML report H.The report must preserve what the video shows and says while adding external research that addresses the user’s need.Source: §2.1 · p. 3
  • Research anchors turn request-relevant claims, procedures, parameters, entities, and open questions into links between source-video moments and external evidence.The workflow retrieves evidence from webpages, images, videos, and documents, then records how it supports, contradicts, updates, explains, compares, or applies video content.Source: §2.1 · p. 3, Figure 2 · p. 3
  • A valid report keeps the given video central, adds research beyond a summary, and connects important findings to external sources and relevant video moments.Text, media, citations, and interactions must let readers inspect evidence; accepted tasks require both the video and external evidence and sufficient public evidence within a practical budget.Source: §2.1 · p. 3

2.2 AGENT WORKFLOW

  • The agent aligns visual observations with speech on a timeline, gathers structured evidence through parallel media branches, and connects it to the source video.It renders an interactive report with video-time navigation, runs programmatic, citation, and required-process checks, and returns failed checks for targeted repair.Source: §2.2 · pp. 3–4

3 OMNIDR-BENCH

OmniDR-Bench combines video-centered research tasks with a rubric designed to assess grounded content, citation support, and interactive presentation. Its construction independently collects and validates evidence, then applies calibrated scoring to reports.

3.1 VIDEO COLLECTION AND QUERY CONSTRUCTION

  • OmniDR-Bench covers four video-research families with 50 Chinese and English tasks across 19 subcategories, requiring both the source video and external evidence.Queries combine a general report request with video-focused research directions and user needs such as decisions, diagnosis, reproduction, or application; each task averages 128 criteria.Source: §3.1 · pp. 4–5

3.2 OMNI ATOMIC RUBRIC TREES

  • Omni-ART scores reports across content, citation, and presentation using 23 shared directions and task-specific binary criteria.Its leaves are tied to independent fact records or executable checks, supporting auditable decisions without prescribing a report structure or identical source set.Source: §3 · p. 4, §3.2 · p. 5
  • ARC builds each rubric from separately collected video and web evidence, links their support and conflicts, and screens criteria through static checks, behavioral calibration, and human review.Calibration compares video-only, full-tool research, and fact-library-assisted reports; retained tasks must satisfy R1 < R2 < R3 and R3 − R1 > δcal, while repeated R2 scoring has 0.89% non-unanimous decisions on average.Source: §3.2 · pp. 5–6

3.3 EVALUATION METHODOLOGY

  • Scoring aggregates applicable binary-leaf outcomes into direction, dimension, and report scores, with gates and capped weights for fact coverage, causal analysis, and comparison.The three dimensions are content, citation, and presentation; missing or unreadable reports receive zero in the all-task average, while evaluator errors are excluded from the relevant denominator.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

Across 18 agent configurations, the strongest overall score is 63.3, but task-level uncertainty and recurring gaps in research substance qualify the rankings. Further analyses find that orchestration can improve citation quality at higher resource cost, while category and trajectory differences require careful interpretation.

4.2 MAIN RESULTS

  • 63.3 is Claude Opus 5’s highest overall point estimate, but its 1.3-point lead over GLM-5.3 has a 95% interval of [−4.0, 5.6]; the order is uncertain.GLM-5.3 completes every task and scores 62.0; Qwen3.8-Max’s 60.1 successful-report score corresponds to 52.9 overall at 88.0% completion.Source: §4.2 · pp. 7–8
  • 81.8 presentation points contrast with 45.4 for content and 47.8 for citation, while cross-source verification and multi-fact composition average 8.8% and 5.0%.Video faithfulness averages 92.4%, but logical consistency and claim support reach only 21.2% and 15.9%; omitted audited targets dominate explicit contradictions.Source: §4.2 · p. 8
  • Systems show distinct trade-offs: Claude Opus 5 leads all three dimensions, GLM-5.3 combines second place overall with complete delivery, and Qwen3.8-Max has the highest single-report score but lower completion reliability.GPT-5.6 Sol and Gemini-3.8-Flash complete every task but remain content-limited; Qwen3.8-Omni-Flash is stronger in content and presentation than citation.Source: §4.2 · p. 8
  • Source collection does not ensure claim support: reports often place links in a final Sources section instead of citing them beside the claims they support.Under Claude Code, GPT-5.6 Sol passes 98.0% of source-count criteria but only 23.9% of claim-support criteria.Source: §4.2 · p. 8
  • Successful-report means span 52.8–54.8 across four major video families, whereas subcategories range from 46.0 to 59.1; query-form differences are descriptive, not causal.Subcategories contain one to five tasks, and query forms are not randomly assigned.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 FURTHER ANALYSIS

  • Adding text web research raises the all-task mean from 31.8 to 45.5 and success from 73.3% to 100.0%; full orchestration adds 7.3 points over atomic skills and improves 12 of 15 tasks.Orchestration’s largest gain is 20.7 citation points versus 0.8 content points, while median steps rise from 103 to 167 and time from 0.7 to 1.6 hours.Source: §4.3 · pp. 8–9
  • Claude Code and Codex each average 48.4 on three matched models, but Codex completes 99.3% of runs versus 96.0% for Claude Code; OpenClaw averages 39.4 with 84.7% completion.Harness effects depend on the model and resource use.Source: §4.3 · p. 9
  • Research trajectories do not predict quality by length alone: drafting can trigger retrieval, but evidence transfer matters more than trajectory length, and short or long traces can fail in different ways.Eight configurations meet the interleaving rule; fragmented traces can still recover, so steps, tokens, and tool errors alone do not predict quality.Source: §4.3 · p. 9

5 RELATED WORK

Prior work addresses text-centered research, multimodal reporting, video deep research, and several benchmark dimensions, but not their combination in complete reports organized around a supplied video.

5 RELATED WORK

  • Text research agents produce cited reports, multimodal report agents add visual retrieval and interleaved output, and video deep research links source-video grounding with web search but mainly targets short-answer QA.The proposed task instead uses the audiovisual timeline as its research backbone and returns a cited interactive report enriched with external multimodal evidence.Source: §5 · p. 9
  • Existing benchmarks assess long-video understanding, open-web question answering, text-report quality, visual retrieval, tool use, report logic, or citation support, while OmniDR-Bench jointly evaluates video grounding, multimodal evidence, claim support, and interaction in complete reports.The cited work includes (Meng et al., 2026; Liang et al., 2025; Liu et al., 2026a) and (Huang et al., 2026; Ye et al., 2026; Wang et al., 2026; Wan et al., 2026; Leung et al., 2026).Source: §5 · p. 9

6 CONCLUSION

The conclusion presents Video-Centered Deep Research Report Generation and OmniDR-Bench, with fine-grained evaluation grounded in audited multimodal evidence. Results across 18 configurations reveal gaps in coverage, cross-modal reasoning, and claim support.

6 CONCLUSION

  • Across 18 configurations, the best score is 63.3 out of 100, exposing gaps in coverage, cross-modal reasoning, and claim support.The authors identify better audiovisual use, efficiency, reasoning, and citations as future-work targets.Source: §6 · p. 9

AI USE STATEMENT

Generative AI assisted with language polishing and writing revision, and AI models were used as evaluated systems and as components of the benchmark evaluator.

AI USE STATEMENT

  • Generative AI assisted with language polishing and writing revision; AI models also served as evaluated systems and benchmark-evaluator components.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The benchmark uses public videos and web sources but is not population representative, and its reports should not replace professional advice. The appendix documents evaluation settings, scoring, and human verification.

ETHICS STATEMENT

  • The benchmark is not population representative, and its reports should not replace professional advice.Human verification must follow applicable review, consent, and data-protection requirements.Source: ETHICS STATEMENT · p. 10

APPENDIX

  • The appendix provides the evaluation specification, task taxonomy, construction protocol, experimental details, prompts, examples, and limitations.Source: APPENDIX · p. 13

A.2 OMNI-ART DIRECTION INVENTORY

  • Omni-ART evaluates fixed direction definitions with instance-specific leaves, using model judgments for semantic properties and deterministic checks for observable conditions.Failed gates zero their direction; high-volume directions use saturation, while other core directions aggregate continuously.Source: §A.2 · p. 13, Figure 5 · p. 14

A.3 MODEL-JUDGE PROTOCOL

  • The fixed judge returns binary decisions and rationales; gates receive three judgments with majority voting, while machine checks test directly observable properties.Fixing the judge improves comparability but does not remove model-specific bias; scores are tied to the released judge, prompts, and evaluator version.Source: §A.3 · p. 13

A.4 SCORING AND FAILURE HANDLING

  • Scoring normalizes fact coverage, causal analysis, and comparison to evidence-budget saturation, while capping their within-dimension weights.Failed gates zero only their direction, and guarded citation N/A assignments require linked coverage failures and a direction-level marginal comparison.Source: §A.4 · pp. 14–15

A.5 HUMAN VERIFICATION AND AGREEMENT

  • Reviewers inspect complete fact libraries and criterion sets through two iterative validation rounds, repairing or removing packages before final sampling.The candidate pool shrinks from 82 to 60 to 50 tasks; more than 94.0% of 1,000 sampled final items are marked correct.Source: §A.5 · p. 15, Table 7 · p. 15

A.6 HUMAN ALIGNMENT ON GENERATED REPORTS

Automatic rubric decisions agree with human majority labels on sampled reports, and their scores correlate with holistic ratings. The accompanying taxonomy and selection criteria define the benchmark’s user goals and video scope.

A.6 HUMAN ALIGNMENT ON GENERATED REPORTS

  • Human raters agree with each other on 93.0% of decisions, and automatic decisions match the human majority on 91.0%.Across 30 reports, Omni-ART scores correlate with mean holistic ratings at Spearman ρ = 0.64.Source: §A.6 · pp. 15–16, Table 8 · p. 16

B.1 SCENARIO TAXONOMY

  • The scenario taxonomy starts from viewer research needs and covers efficient understanding, verification and extension, and practical application.These goals may coexist within one task.Source: §B.1 · p. 16

B.2 VIDEO INCLUSION AND EXCLUSION

  • Included videos need researchable content, at least one element requiring visual, audiovisual, or temporal evidence, and enough public evidence for further research.The selection rejects direct lookups, transcript restatements, video-independent queries, insufficient public evidence, and unsafe or unstable-source tasks.Source: §B.2 · p. 16

B.3 FIXED DATA SNAPSHOT

The fixed benchmark spans Chinese and English videos and three query forms. Its construction mines distinct research views, applies manual acceptance rules, and uses a staged workflow to connect video moments with external evidence in an interactive report.

B.3 FIXED DATA SNAPSHOT

  • The benchmark contains 50 tasks and 17.1 hours of source-video content, including 23 Chinese and 27 English tasks, with 7 general, 17 video-view, and 26 user-demand queries.Subcategory findings remain descriptive because some scenarios have few tasks.Source: §B.3 · p. 16

C.1 THREE QUERY FORMS

  • General queries cover a whole video, video-view queries investigate a selected video element, and user-demand queries start from a concrete viewer objective.The two focused query forms may share a video anchor but differ in who sets the research boundary.Source: §C.1 · p. 17, Table 11 · p. 18

C.2 CANDIDATE MINING PIPELINE

  • Parallel video readers merge timestamped speech, visible text, events, objects, and uncertain regions into the sole shared input for two independent query miners.One miner proposes video-view questions; the other uses six user roles to propose user-demand questions.Source: §C.2 · pp. 17–18

C.3 MANUAL ACCEPTANCE PROTOCOL

  • Candidates are deduplicated, scored, ranked, and schema-checked; annotators retain queries only when they are video-centered, researchable, and tied to a concrete user task where applicable.The strip-the-persona test detects user-demand queries that merely attach a role label to generic research.Source: §C.2 · p. 18, §C.3 · p. 18

D.1 ROLES AND SHARED STATE

  • The main agent manages goals, shared evidence, report assembly, and repairs, while specialized readers return structured video and document records.Records preserve modality, source, date, excerpt, anchor, and verification state.Source: §D.1 · p. 19

D.2 SEVEN EXECUTION STAGES

  • The workflow inspects the video, builds a timestamped audiovisual timeline, selects request-relevant research anchors, and searches external multimodal sources in anchor-specific branches.Branches return provenance, excerpts, media, dates, and confidence.Source: §D.2 · p. 19
  • The workflow relates external evidence to video claims, renders a dual-pane report with clickable timestamps, then validates and repairs citations, assets, and browser interactions.Fresh-context checks reopen cited sources to verify their support for report claims.Source: §D.2 · p. 19

D.3 RELEASED BUDGETS

  • The reference configuration sets minimum external-source, related-video, document, and per-anchor source counts through released budgets.Source: §D.3 · p. 19

E BENCHMARK CONSTRUCTION AND ANNOTATION

The benchmark is built through evidence collection, rubric screening, and iterative validation. Its reported analyses include a controlled workflow comparison, while uncertainty and category findings qualify how system scores should be interpreted.

E.1 SEVEN-STAGE CONSTRUCTION AND HUMAN CURATION

  • Human quality control reduces candidate packages from 82 to 60 and then to the final 50-task benchmark.Reviewers identify artifact errors, repair viable packages, and exclude packages with high or persistent errors.Source: §E.1 · pp. 19–20, Table 13 · p. 20

E.2 INDEPENDENT EVIDENCE ACQUISITION

  • ARC independently collects video evidence without web access and web evidence without source-video access, then reconciles claims while preserving factual conflicts.Records are verified with two independent domains or a suitable primary source; conflicting records remain disputed.Source: §E.2 · p. 20

E.3 RUBRIC TREE LEAF GENERATION AND SCREENING

  • Rubric generation maps evidence to binary criteria, screens ambiguous or unsupported candidates, and assigns difficulty from evidence structure.The snapshot contains 6,414 atoms across content, citation, and presentation, referencing 3,102 task-local facts.Source: §E.3 · pp. 20–21, Table 14 · p. 21

E.4 VALIDATION AT THREE LEVELS AND HUMAN REVIEW

  • All 50 retained tasks pass behavioral calibration, with R1 < R2 < R3 and R3 − R1 > δcal.Human review records 64.0% item correctness and 77.0% unanimity in round one, then 87.0% and 81.0% in round two before final repair and filtering.Source: §E.4 · p. 21

F.1 SYSTEM CONFIGURATIONS AND BUDGETS

  • The 18-system comparison uses matched report contracts and budgets; the Claude Opus 5–GLM-5.3 difference is 1.3 points with a 95% interval of [−4.0, 5.6].Alternative scoring parameters retain the same top three order, but absolute scores vary most with dimension weights and coverage saturation.Source: §F.2 · p. 22, §F.1 · p. 22, Table 15 · p. 22

F.3 DIRECTION-LEVEL ERROR ANALYSIS

  • Omissions of facts or details dominate explicit factual contradictions, while broad video-family means differ by only 2.0 points and subcategory means span 13.1.Category interpretations are descriptive because subcategories contain only one to five tasks.Source: §F.3 · p. 22, §F.4 · p. 23, Table 16 · p. 23

F.5 CONTROLLED SKILL ABLATIONS

  • The full workflow exceeds the all-atomic-skills variant by 7.3 points on average and wins 12 of 15 paired tasks.The paired bootstrap 95% confidence interval is [3.4, 11.1]; this estimates deployed quality and cost trade-off, not compute-matched performance.Source: §F.5 · p. 24

F.6 CONTROLLED HARNESS COMPARISON

  • The harness comparison uses only shared models, and its process measurements are runner-native rather than hardware-normalized costs.Source: §F.6 · p. 24

F.7 TRAJECTORY AND EFFICIENCY STATISTICS

This appendix describes trajectory patterns, benchmark protocols, and report examples. It explains how reports are evaluated and how execution traces and common report failures should be interpreted.

F.7 TRAJECTORY AND EFFICIENCY STATISTICS

  • Trajectory tables report medians and event-order statistics; interleaving is defined by at least 30% of searches after first writing and at least four write blocks.Eight of 18 configurations meet this threshold, which summarizes a continuous pattern; runner-reported tokens are not comparable enough for a hardware-normalized efficiency ranking.Source: §F.7 · pp. 24–25
  • Trace length alone predicts neither report strength nor weakness; stronger traces tend to acquire evidence and transfer it into the report.Examples span fragmented but recoverable work (586 steps, score 72.1), long research with weak evidence transfer (549 steps, score 63.1), organized delegation (226 main-agent steps, score 71.5), shallow research (83 steps, score 31.2), and repeated failures (126 tool errors, score 38.6).Source: §F.7 · p. 25

G.1 EVALUATED AGENT INSTRUCTION

  • Each evaluated agent receives the same language-matched request specifying the video, source URL, user focus, and output directory, but not evaluation evidence or decisions.The high-level skill supplies the seven-stage workflow, while the fixed interface template is separate from the skill.Source: §G.1 · p. 26

G.2 QUERY MINING PROMPTS

  • Query mining starts with a timestamped audiovisual record, then uses isolated roles to generate candidate research questions under schema and manual boundary checks.The schema requires five to ten candidates per view, scores from 0 to 1, descending order, and language consistency.Source: §G.2 · pp. 26–27

G.3 INDEPENDENT EVIDENCE PROMPTS

  • Independent evidence collection separates video-only fact extraction from six web-search routes, preserving timestamps, source details, and unavailable or conflicting evidence.The video collector has no web access; web collectors cannot access the source video or one another’s workspaces.Source: §G.3 · p. 27

G.4 RUBRIC AND JUDGE PROMPTS

  • Rubric criteria are generated from task specifications and evidence records, screened for validity, then judged against relevant report material without credit from outside knowledge.Judges return Yes, No, or N/A with a report-grounded rationale; model-judged gates use three independent votes.Source: §G.4 · p. 28

H.1 REPORT ORGANIZATION

  • A valid report separates video observations from external findings and links timeline moments, claims, media, and citations so readers can inspect their relationships.The report interface supports navigation from a note to its video time and from a claim to its cited source.Source: §H.1 · p. 29, Figure 6 · p. 29

H.2 COMMON REPORT FAILURES

  • Recurring failures include polished reports that omit evidence-map facts, citation lists without claim support, transcript-only research, and findings that never reach the final report.These patterns motivate separate evaluation of content, presentation, citation support, audiovisual grounding, and evidence integration.Source: §H.2 · p. 29

I LIMITATIONS AND FUTURE WORK

The benchmark and report interface have defined scope boundaries, and the paper identifies directions for expanding coverage, generalizing tools, and enriching presentation.

I LIMITATIONS AND FUTURE WORK

  • OmniDR-Bench currently covers 50 tasks across 19 subcategories; future versions can broaden task, scenario, and language coverage with more user requests.Some evaluation directions reflect capabilities emphasized by the reference skills, although scoring uses only final reports and the paper compares several skill subsets.Source: I LIMITATIONS AND FUTURE WORK · p. 29
  • The current report interface uses a relatively fixed two-pane layout, while future work could add richer structures and interactions.The paper also calls for more general and efficient skills, tools, and subagents that transfer across agent frameworks.Source: I LIMITATIONS AND FUTURE WORK · pp. 29–30
  • Because the task can require hundreds of agent steps, training and releasing capable open-source models remains important for the research community.Source: I LIMITATIONS AND FUTURE WORK · p. 30

Related papers

Loading 2610.31158v1…