PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

Zhen Fang, Yu Zeng, Wenxuan Huang, Yiming Zhao, Shiting Huang, Tianfei Ren, Qi Lu, Qingnan Ren, Qisheng Su, Lionel Z. Wang, Qingyu Yin, Shuang Chen, Zehui Chen, Lin Chen, Zhenfei Yin, Yao Hu, Shaohui Lin, Wanli Ouyang, Shaosheng Cao, Feng Zhao

arXiv:2608.03979v1cs.CVcs.AI

TL;DR

Video-DeepResearch addresses the challenge of grounding autonomous agents in continuous video while integrating visual exploration with web retrieval. It introduces a staged perception-exploration and SFT–GRPO framework, achieving 64.0% accuracy and outperforming Claude-4.5-Sonnet by 5.0 points.

  • Problem

    Video-DeepResearch requires dense visual tracking, text synthesis, and evaluation beyond standard visual question-answering metrics.

  • Method

    The framework uses sequential keyframe and crop-based visual grounding before text retrieval, trained with supervised fine-tuning followed by GRPO.

  • Results

    64.0% accuracy establishes state-of-the-art performance, surpassing Claude-4.5-Sonnet at 59.0% by 5.0 points.

  • Takeaways & Limitations

    The 200-instance Video-DR-Bench provides an evaluation setting requiring both visual search and external knowledge reasoning.

  • Takeaways & Limitations

    The framework requires substantial computational resources, while human annotation limits rapid benchmark scalability.

Abstract

from arXiv · show

We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in current models: (1) modality bias, where agents bypass visual tools in favor of textual search, and (2) parametric knowledge leakage, where models rely on internal memory rather than genuine tool-augmented execution. To address these challenges, we propose Video-DR, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web retrieval. Our framework adopts a two-stage training recipe: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), enabling autonomous exploration that breaks the imitation-learning ceiling. Furthermore, we curate Video-DR-Bench, a human-AI collaborative benchmark comprising 200 complex, multi-hop VQA instances. Empirical results demonstrate that our Video-DeepResearch-35B-A3B establishes a new state-of-the-art of 64.0% average accuracy, surpassing proprietary Claude-4.5-Sonnet (59.0%) by 5.0 points and significantly outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The 30B-A3B variant achieves 59.3%, competitive with Claude-4.5-Sonnet and demonstrating the effectiveness of our training paradigm even at compact scale. Code: https://github.com/Osilly/Vision-DeepResearch.

1 Introduction

Video-DeepResearch addresses the challenge of autonomous exploration over video, where dense visual tracking and text-based synthesis are intertwined. The framework combines empirical diagnosis, decoupled perception-exploration with stage-wise tool unlocking, two-stage training, and a 200-instance benchmark to improve evaluation and performance.

  • Motivation: Video-DeepResearch shifts autonomous research from isolated modalities to environments intertwining text synthesis with dense visual tracking.This setting requires extracting critical insights from noisy, heavily redundant multimedia web layouts.
  • Failure modes: Current agents exhibit modality bias, averaging 0.10 visual tool invocations and 1.27 text-search calls per task.The finding indicates systematic avoidance of active visual exploration when agents are applied directly to Video-DeepResearch.
  • Contributions: The data engine produces 30K video-grounded QA pairs and 7K curated trajectories using decoupled perception-exploration with stage-wise tool unlocking.This design directly counters the modality bias identified in the preliminary study.
  • Training and results: 64.0% makes Video-DeepResearch-35B-A3B state of the art, surpassing Claude-4.5-Sonnet at 59.0% by 5.0 points; the 30B variant reaches 59.3%.The two-stage recipe combines supervised fine-tuning with GRPO and enables compact models to outperform far larger proprietary systems.
  • Evaluation: VideoDR-BENCH contains 200 multi-hop VQA instances created through human–AI collaborative annotation, with every question requiring visual search and external knowledge reasoning.Standard VQA metrics are insufficient for measuring multi-step strategic execution, long-term reasoning consistency, and error recovery under continuous temporal dynamics.

2 A Naive Attempt: From Image · DeepResearch to Video DeepResearch

The paper formulates Video-DeepResearch as sequential spatiotemporal grounding followed by visual and textual search, then shows that naive agents avoid visual tools and can exploit parametric knowledge instead of executing this process.

  • DeepResearch to Video DeepResearch: Video-DeepResearch models video as temporal compositions of key entity trajectories and uses sequential grounding to identify frames, localize entities, and synthesize findings through text retrieval.The workflow bridges image-centric research with Select_Keyframe and Crop_Search.
  • DeepResearch to Video DeepResearch: Given a research query Q and visual input V, the agent sequentially selects actions from its history to synthesize a comprehensive response R.The visual input may be a continuous video stream or pre-sampled keyframes.
  • DeepResearch to Video DeepResearch: The action space combines Select_Keyframe, Crop_Search, Text_Search, and other web operations for temporal selection, spatial cropping, and downstream search.Select_Keyframe isolates frame v_t, while Crop_Search applies bounding box B to produce localized visual context c_vis.
  • DeepResearch to Video DeepResearch: The naive formulation was evaluated with three representative models on the VideoDR benchmark by tracking each tool’s execution frequency.The study compares accuracy and average tool invocation counts.
  • DeepResearch to Video DeepResearch: 0.10 visual operations versus 1.27 text operations per task were executed by Qwen3.5-397B-A17B, revealing severe modality bias and visual tool aversion.The model overwhelmingly favored text tools and bypassed intended active visual exploration.
  • DeepResearch to Video DeepResearch: 57 accuracy was achieved by GPT-5 with 0.00 vision calls and 0.12 text calls, indicating susceptibility to parametric knowledge leakage.The model effectively bypassed multi-step grounding by relying on internal memory to guess or hallucinate answers.

3 VIDEO-DEEPRESEARCH

Video-DeepResearch builds a complete data-to-agent pipeline: it synthesizes grounded Video QA, constructs decoupled perception-exploration trajectories, trains agents with SFT and GRPO, and curates an answerable benchmark through human-in-the-loop verification.

  • Data synthesis: The pipeline transforms raw videos into explicit Video QA pairs with intermediate evidence annotations to ensure visual groundability and answerability.This synthesized data supports trajectory construction and later benchmark curation.
  • Data synthesis: Videos undergo rule-based duration filtering and agentic complexity assessment before partitioning into training and evaluation sets.Qwen3.5-35B-A3B removes uninformative or overly simplistic videos during agentic filtering.
  • Data synthesis: Keyframes are selected from CLIP-based candidates, entities are localized and visually searched, and retrieved results are verified for semantic alignment.Qwen3.5-397B-A17B finalizes keyframes and predicts entity boxes, while Qwen3.5-35B-A3B verifies crop-result alignment.
  • Trajectory construction: The agent-training pipeline separates visual perception from web exploration to counter agents’ tendency to bypass visual tools for text-only search.Trajectories are generated with Qwen3.5-397B-A17B and filtered through rejection sampling.
  • Training: Both model variants receive SFT followed by GRPO, using 7K synthesized trajectories for SFT and a 2K moderate-difficulty dataset for reinforcement learning.GRPO uses four rollouts per trajectory, sparse binary correctness rewards, and 20% negative-gradient application for formatting violations or repetitive loops.
  • Benchmark construction: The benchmark uses human-in-the-loop annotation, visual evidence verification, tool-dependency filtering, and recursive ranking to produce increasingly complex multi-hop questions.Questions answerable without tools are removed to prevent parametric knowledge leakage.

4 Experiments

Experiments show that Video-DeepResearch achieves leading Video-DR performance across model scales while specialized training promotes more extensive, balanced visual-text tool use. The 35B-A3B variant reaches 64.0% average accuracy, and the 30B-A3B variant reaches 59.3%, with gains over proprietary and foundation-model baselines.

  • Main Results: 64.0% average accuracy establishes Video-DeepResearch-35B-A3B as the state-of-the-art, surpassing Claude-4.5-Sonnet at 59.0% by 5.0 percentage points.The model is evaluated alongside multiple proprietary and open VLMs under Direct and Agentic settings.
  • Main Results: 65.4% on VIDEODR-BENCH is the highest score among evaluated models, while Video-DeepResearch-35B-A3B improves +21.2% over Qwen3.5-35B-A3B.Category gains include KNL at +20.3% and ENT at +15.9%.
  • Main Results: 59.3% average accuracy makes Video-DeepResearch-30B-A3B competitive with Claude-4.5-Sonnet at 59.0% and ahead of GPT-5 at 52.5% and Gemini 2.5 Pro at 57.5%.Against Qwen3-VL-30B-A3B-Instruct, it gains +24.0% on VideoDR and +13.5% on VIDEODR-BENCH.
  • Scaling Insight: +21.2% over base for 35B-A3B exceeds the 30B-A3B margin of +18.8%, suggesting stronger synergy between the training pipeline and model capacity.News gains are smaller for 35B-A3B than 30B-A3B, at 8.4% versus 16.6%, indicating brittleness on temporally dynamic content.
  • Tool-Use Analysis: 2.33 visual and 4.24 text tool invocations show Video-DeepResearch-30B using more tools than Qwen3.5-397B’s 0.10 visual and 1.27 text operations.On VIDEODR-BENCH, GPT-5’s visual calls rise from 0.00 to 0.31 and text calls from 0.12 to 1.43 relative to VideoDR.

5 Related Work

Prior work progresses from uniform-sampling Video-LLMs and closed-world video agents to text-only and image-centric deep-research systems, but these approaches do not address open-world, continuous-video reasoning. Existing benchmarks likewise evaluate static multimodal or image-search settings and remain insufficient for video streams.

  • Video understanding agents: Early Video-LLMs use uniform frame sampling without dynamic visual querying, making them prone to error accumulation and hallucinations.Recent agentic frameworks add interactive tools for fine-grained perception but remain confined to closed-world video contexts.
  • Multimodal deep research: Text-only deep-research agents have evolved toward image-centric Vision-DR frameworks using reverse image search, GRPO optimization, and entity-level cropping.This progression nevertheless bypasses the continuous video modality.
  • Multimodal deep research: Continuous-video research requires decoupled spatiotemporal dynamics and multi-step verification across noisy frames, unlike static-image reasoning.The passage characterizes this as a distinctly more demanding setting than static visual contexts.
  • Evaluation: Current benchmarks assess static multimodal factuality, external knowledge grounding, and image-based search workflows, leaving them insufficient for video streams.Dedicated Video-DR benchmark efforts remain scarce and face a critical bottleneck.

6 Conclusion

VIDEO-DEEPRESEARCH unifies data synthesis, agent training, and evaluation for video deep research, addressing visual tool aversion and parametric knowledge leakage. It introduces a decoupled perception-exploration pipeline, 30K video-grounded QA pairs, and VIDEODR-BENCH, a 200-instance multi-hop VQA benchmark requiring visual search and external knowledge reasoning.

  • Conclusion: VIDEO-DEEPRESEARCH unifies scalable data synthesis, agent training, and rigorous evaluation in a framework for video deep research.The framework is presented as a foundation for next-generation video deep research agents.
  • Conclusion: Existing agents exhibit systematic visual tool aversion and parametric knowledge leakage, undermining faithful assessment of Video-DR capabilities.These are identified as two critical failure modes in the preliminary study.
  • Conclusion: The decoupled perception-exploration pipeline with stage-wise tool unlocking produces 30K video-grounded QA pairs.This design directly addresses the identified failure modes while supporting scalable data synthesis.
  • Conclusion: 200 instances comprise VIDEODR-BENCH, a multi-hop VQA benchmark whose questions require both visual search and external knowledge reasoning.The benchmark is designed so that every question provably demands both capabilities.

Author Information

The paper lists 20 authors affiliated with institutions including USTC, Xiaohongshu Inc., CUHK, and several international universities; four first authors contributed equally, with two project leaders and three corresponding authors.

  • Author Information: The affiliations include USTC, Xiaohongshu Inc., CUHK, The Hong Kong Polytechnic University, ZJU, UCLA, Oxford, ECNU, and THU.These institutions correspond to affiliation numbers 1–9.
  • Author Information: Four first authors—Zhen Fang, Yu Zeng, Wenxuan Huang, and Yiming Zhao—are identified as contributing equally.Yu Zeng and Wenxuan Huang serve as project leaders.
  • Author Information: The corresponding authors are Wenxuan Huang, Shaosheng Cao, and Feng Zhao.The passage separately identifies the project leaders as Yu Zeng and Wenxuan Huang.

Limitation · A Appendix

The framework integrates data construction and model training for the complex VideoDR task, but this rigorous approach incurs substantial computational overhead. Its resource demands arise from high-quality data synthesis, robust model training, large-scale model deployment, and dynamic web search operations.

  • Limitation: VIDEO-DEEPRESEARCH pioneers a comprehensive pipeline for the complex VideoDR task.
  • Limitation: The pipeline integrates data construction and model training.
  • Limitation: This rigorous approach introduces certain trade-offs.
  • Limitation: The current performance level incurs considerable computational overhead.
  • Limitation: High-quality data synthesis demands substantial GPU resources.
  • Limitation: Robust model training contributes to the framework’s substantial GPU requirements.
  • Limitation: Large-scale model deployment and dynamic web search operations are required concurrently.

B Training Details

Training used a 32-GPU Megatron-LM setup with mixed parallelism to support an 80, 000-token context, alongside scheduled learning-rate decay and MoE-specific efficiency optimizations.

  • Training Setup and Hyperparameters: SFT ran on 4 nodes with 8 × 80 GB GPUs each, totaling 32 GPUs, using Megatron-LM and mixed parallelism for an 80, 000-token context.The setup avoided data packing for the ultra-long context length.
  • Training Setup and Hyperparameters: The learning rate warmed up over the first 5% of training steps to 1 × 10−5 before decaying to 5 × 10−7.This schedule used linear warmup and decay.
  • Optimization and Efficiency Enhancements: MoE training used a 1 × 10−6 auxiliary load-balancing loss coefficient and a 2.0 expert capacity factor to mitigate token dropping.The implementation also included permute fusion, Grouped GEMM, overlapping shared-expert computation with communication, and strict memory optimization.

C Data Details · D Annotation Details

The data pipeline uses uniform CLIP-based keyframe deduplication, while VIDEOHUNT annotation combines experienced annotators, qualification-based quality assurance, staged prompting, and vision-first research tooling.

  • C Data Details: Keyframe extraction uses CLIP-ViT-L/14@336px, removes consecutive frames above 0.8 similarity and monochromatic frames, and caps each video at 20 keyframes.This configuration applies uniformly to data synthesis and evaluation, including VIDEODR.
  • D Annotation Details: VIDEOHUNT annotation spans approximately three weeks and is conducted by 8 annotators with professional multimodal large language model annotation experience.The annotators are also familiar with video understanding, visual search, and multi-hop reasoning.
  • D Annotation Details: Annotators receive structured training on task definitions, tool usage, and parametric knowledge leakage before qualifying on a held-out pilot set.Only annotators meeting a predefined accuracy threshold enter the main annotation phase.
  • D Annotation Details: Video task-suitability evaluation determines whether understanding, contextualizing, or verifying core content requires external web search and domain-specific knowledge.The evaluation targets semantic depth and an information gap sufficient to require external search.
  • D Annotation Details: Multi-frame entity extraction selects diverse representative keyframes and grounds exactly one mutually exclusive salient entity per selected frame.The structured prompt specifies selecting 3 to 5 keyframes.
  • D Annotation Details: Knowledge-grounded question generation combines localized video entities with retrieved external knowledge to select a questioning strategy and synthesize one complex reasoning question.The prompt enforces visual-referencing constraints to prevent information leakage.
  • D Annotation Details: The deep-research workflow includes report-answer verification and a research agent that conducts multi-source investigations, returns answers in <answer> tags, and follows vision-first tool use.The agent grounds visual entities through select_crop_search before textual web exploration; search and visit are added only during answering.
Loading 2608.03979v1…