PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaVDR: Adaptive Tool Use and Reflection for Video Deep Research

Xintong Zhang, Xiaomeng Fan, Shilin Yan, Ekko He, Zicheng Liu, Zijian Zou, Guannan Zhang, Yuwei Wu, Zhi Gao, Hongwei Xue

arXiv:2608.25559v1cs.CVcs.AI

TL;DR

Video deep research needs different tool-use strategies across diverse questions and videos, while uncertain tools make redundant interactions costly and error-prone. AdaVDR adaptively selects tools and reflects on unreliable evidence, achieving the best results among evaluated open-source models on VDR-EE and substantially improving its base models on VideoDR.

  • Problem

    Diverse video questions require different tool strategies, while uncertain grounding and retrieval make unnecessary calls costly and potentially error-prone.

  • Method

    AdaVDR combines task- and capability-specific tool invocation with selective reflection, model-conditioned tool filtering, and redundancy-aware reinforcement learning.

  • Results

    AdaVDR performs best among evaluated open-source models on VDR-EE and improves its Qwen3-VL-8B-Instruct base model by 10.00 percentage points overall.

  • Takeaways & Limitations

    The approach reduces average tool calls while maintaining or improving overall agent performance across diverse video deep research tasks.

Abstract

from arXiv · show

Video deep research answers complex questions by jointly understanding video content and retrieving external knowledge from the open Web. However, diverse questions and videos require different tool-use strategies, and inappropriate tool calls can produce incorrect results. Uncertain grounding and retrieval also make unnecessary interactions costly and error-prone, increasing latency and reasoning errors. To address these challenges, we propose AdaVDR, an adaptive video deep research agent with adaptive tool invocation and reflection. AdaVDR selects tools according to the task and its capabilities, and backtracks only when unreliable intermediate results require correction. To enable these capabilities, we develop a video deep research data construction pipeline. We first discover retrieval-relevant events and entities in diverse videos and acquire detailed information through grounding and external retrieval to construct high-quality QA pairs. For each QA, task-specific prompts organize the information acquisition process into a tool-use trajectory, allowing different question and video types to follow different grounding and retrieval strategies. We further introduce model-conditioned tool necessity filtering, which evaluates tool calls against the target model's video understanding and internal knowledge, removing tools or tool chains the model can bypass. This yields trajectories tailored to the target model's video understanding capability and knowledge. Using this pipeline, we construct training data and VDR-EE, a benchmark covering entity-centric and event-centric questions. We perform supervised fine-tuning followed by reinforcement learning with a redundancy-aware reward to strengthen adaptive tool invocation and reflection. Experiments show that our method performs best among the evaluated open-source models on VDR-EE and substantially improves over its base models on VideoDR.

1 Introduction

AdaVDR addresses the diversity and uncertainty of video deep research with adaptive tool invocation and reflection, supported by capability-tailored trajectory construction, model-conditioned tool filtering, and redundancy-aware reinforcement learning.

  • Motivation: Video deep research is challenging because diverse questions and videos require different tool-use strategies, while grounding and external retrieval remain uncertain.Unsuitable strategies can miss critical evidence or produce incorrect reasoning, and uncertain interactions increase cost and error risk.
  • AdaVDR: AdaVDR adaptively selects task- and capability-appropriate tools and reflects only when newly acquired evidence is unreliable or insufficient.Reflection diagnoses grounding versus retrieval failures and responds by re-localizing video evidence or revising and repeating the search.
  • Data construction: The data construction pipeline generates task-specific tool-use trajectories by identifying retrieval-relevant events and entities and acquiring their details through tailored grounding and retrieval.It also applies model-conditioned tool necessity filtering so trajectories match the target model’s video understanding capability and internal knowledge.
  • Training and results: SFT teaches task-appropriate tool selection and multi-turn reasoning, while redundancy-aware RL discourages unnecessary grounding, retrieval, and reflection without sacrificing correctness.The resulting method performs best among evaluated open-source models on VDR-EE and substantially improves over its base models on VideoDR.

2 Related Work

Prior work advances agentic video evidence acquisition and multimodal deep-search tool use, but primarily treats video grounding and external retrieval separately. AdaVDR addresses this gap by adaptively coordinating both processes and revising unreliable intermediate results.

  • Video understanding agents iteratively locate question-relevant segments, frames, actions, or entities through agentic exploration, with recent work adding uncertainty-aware planning and dynamic temporal grounding [Liu et al., 2025b; Chen et al., 2025; Yang et al., 2025b; Zhang et al., 2025; Zhi et al., 2025; Pang and Wang, 2025].
  • Unlike these video-focused systems, Video Deep Research requires external information linked to grounded entities or events, motivating adaptive coordination of grounding, retrieval, and multi-turn revision.
  • Deep-search agents iteratively reason and invoke external tools, while multimodal extensions add image search, visual grounding, cropping, and learned tool-use policies through supervised fine-tuning or reinforcement learning [Yao et al., 2022; Jin et al., 2025; Li et al., 2025; Wu et al., 2025; Hong et al., 2026; Huang et al., 2026; Chen et al., 2026; Zhang et al., 2026].

3 Adaptive Video Deep Research Agent

AdaVDR formulates video deep research as iterative reasoning over video evidence and external tools, where temporal events and fine-grained entities create task-specific information needs. Its agent adaptively selects or skips tools and invokes reflection only when acquired evidence is unreliable or insufficient.

  • Task Formulation: Video deep research iteratively combines video grounding, external retrieval, and answer termination to generate an answer from a video and user question.The toolset includes temporal, timestamp, and spatial grounding; image search, web search, and page visits; and final answer generation.
  • Challenges: Because question types and videos demand different information, fixed or inappropriate tool trajectories can miss required evidence, add irrelevant operations, or produce incorrect reasoning.Grounding and retrieval outputs may be inaccurate, uninformative, or irrelevant, while unnecessary calls expose reasoning to additional uncertainty when internal capabilities already suffice.
  • Question Types: Event-centric questions primarily require temporal localization, whereas entity-centric questions may require frame or region grounding plus OCR, image search, or web search.Event questions can proceed to text-based web search after the video is understood, while entity questions resolve fine-grained identities and related information.
  • Adaptive Tool Invocation: Adaptive tool invocation selects grounding and retrieval tools according to the query, video content, available evidence, and the agent’s capabilities, skipping steps when information is already sufficient.This constructs each reasoning trajectory on demand rather than following a fixed sequence.
  • Adaptive Reflection: Adaptive reflection is triggered only when newly acquired evidence is irrelevant, inconsistent, or insufficient, prompting the agent to inspect the prior trajectory and diagnose the failure source.When evidence is reliable, the agent proceeds without reflection; otherwise, it can examine errors in queries, grounding, timestamp selection, or temporal localization.

4 Data Collection

The data pipeline builds jointly video- and Web-grounded QA pairs, then generates task-specific trajectories refined through on-demand reflection and model-conditioned tool filtering. It also introduces VDR-EE, a manually verified benchmark spanning entity-centric and event-centric questions.

  • Pipeline Overview: The pipeline constructs verified QA pairs and tool-use trajectories through two stages: QA generation and trajectory generation, removing unnecessary interactions.Questions require both video information and external evidence, while trajectories are task-specific and refined before execution.
  • QA Generation and Evidence Acquisition: Video-to-Web acquisition progressively grounds events or entities using temporal, timestamp, spatial, image, OCR, and ASR evidence before authoritative external retrieval.Retrieved facts are organized into traceable evidence paths, and generated questions combine video-grounded temporal conditions with source-backed external answers.
  • QA Generation and Filtering: Generated questions are discarded when Qwen3-VL-235B-A22B-Instruct answers them without video or without external tools, enforcing dependence on both sources.The two restricted settings test video-free answering and video-only answering, respectively.
  • Trajectory Generation and Reflection: Trajectories trigger reflection only after unreliable intermediate results, retaining failed attempts, diagnoses, and targeted corrections while leaving reliable steps unchanged.Task-specific prompts and trajectory refinement remove irrelevant tools, check dependencies and parameters, and preserve adaptive recovery behavior.
  • Tool Necessity Filtering: Model-conditioned necessity filtering removes a tool or consecutive tool chain when the target result can already be obtained from the agent’s known information.The filter constructs a verification query for the chain’s target result and retains the chain only when the predicted result does not match it.
  • Benchmark Construction: VDR-EE is a manually verified benchmark covering entity-centric and event-centric questions, with overlapping training candidates discarded and remaining items deduplicated and reviewed.Benchmark construction follows the QA generation process and prevents overlap with SFT and RL training data.

5 Training

AdaVDR is initialized with supervised fine-tuning on verified adaptive tool-use trajectories, then optimized with GRPO using correctness, formatting, and redundancy-aware adaptive rewards. The training objective conditions on video and tool observations while updating only assistant-generated tokens.

  • Supervised fine-tuning: SFT uses approximately 2.9K verified multi-turn trajectories combining task-specific grounding and retrieval actions, including intermediate failures followed by valid corrections.The resulting checkpoint initializes the subsequent RL policy.
  • Training objective: Both SFT and RL objectives update only assistant-generated tokens, using video inputs, user messages, and tool observations as conditioning context.The RL objective also keeps the SFT policy frozen as a reference policy.
  • Reinforcement learning: RL uses approximately 1K training instances, sampling groups of complete research trajectories through the grounding and retrieval environment.Rewards are normalized within each rollout group under the GRPO procedure.
  • Reinforcement learning: GRPO further trains the SFT-initialized agent with correctness, serialization-format, and adaptive rewards that discourage unnecessary tool calls without rewarding incorrect shortcuts.The adaptive reward applies only to correct trajectories, penalizing extra calls relative to the most efficient correct trajectory; reward weights are λacc = 0.9, λfmt = 0.1, and λada = 0.1.

6 Experiment

AdaVDR improves over its base models on both VDR-EE and VideoDR, with agentic interaction consistently raising performance across evaluated models. Its adaptive trajectories vary grounding and retrieval behavior by question characteristics while reflection enables recovery from unreliable intermediate evidence.

  • Benchmark Results: AdaVDR improves overall accuracy on both benchmarks, reaching 38.40% on VDR-EE and 51.00% on VideoDR with the Qwen3-VL-8B-Instruct base.These correspond to gains of 10.00 and 21.00 percentage points over the respective base-model results.
  • VDR-EE Results: Agentic interaction improves every evaluated baseline on VDR-EE, with gains ranging from 16.00 to 30.40 percentage points.Qwen3.5-35B-A3B improves from 16.00% to 46.40%, while Qwen3.5-9B improves from 12.80% to 40.00%.
  • VideoDR Results: On VideoDR, AdaVDR maintains gains across difficulty levels, with Qwen3-VL-8B-Instruct-based results of 65.62%, 44.44%, and 43.75% from low to high difficulty.Using Qwen3.5-9B as the base raises the corresponding accuracies to 75.00%, 50.00%, and 43.75%.
  • Qualitative Analysis: AdaVDR skips unnecessary grounding when video context suffices and reflects to recover from mismatched retrieval evidence.Qualitative examples show direct progression to spatial grounding in one case and revised timestamp selection followed by web verification in another.
  • Tool-Use Analysis: External retrieval is frequently used, with Web Search averaging 2.50 calls per VideoDR question and 2.77 calls per VDR-EE question.On VDR-EE, event questions use Web Search more often than entity questions, at 3.13 versus 2.54 calls per question.
  • Tool-Use Analysis: AdaVDR adapts tool use to question structure: multi-entity questions require more temporal grounding, timestamp grounding, web search, and interaction turns than single-entity questions.The average trajectory increases from 7.30 to 8.73 turns, while temporal grounding rises from 1.05 to 1.43 calls per question.
  • Ablation Study: Ablation results improve accuracy to 51.00% while reducing average tool calls from 7.84 to 6.80, increasing efficiency without sacrificing performance.This result is reported for AdaVDR-8B on VideoDR.

7 Conclusion

AdaVDR is an adaptive video deep research agent that selects tools according to task requirements and model capabilities, while correcting unreliable intermediate results only when needed.

  • AdaVDR addresses diverse tool-use requirements and unreliable intermediate results in Video Deep Research.
  • Adaptive tool invocation selects necessary tools according to task requirements and model capabilities.
  • Adaptive reflection corrects unreliable intermediate results only when needed, while the data construction pipeline generates task-specific tool-use information.

A Training Details

AdaVDR is trained in two stages: supervised fine-tuning followed by reinforcement learning. The resulting SFT checkpoint initializes GRPO-based RL with bounded tool-use rollouts and group-wise reward scaling.

  • Training proceeds in two stages: supervised fine-tuning followed by reinforcement learning.
  • Both AdaVDR-8B and AdaVDR-9B use five SFT epochs, maximum sequence length 131,072, and LoRA rank 8, with model-specific learning rates and parallelism settings.AdaVDR-8B uses 2 × 10−6, sequence parallelism 8, and gradient accumulation 16; AdaVDR-9B uses 1 × 10−5, sequence parallelism 4, and gradient accumulation 8.
  • The SFT checkpoint initializes GRPO reinforcement learning across 16 × 80GB GPUs for one epoch, with at most 20 tool turns per rollout and group-wise reward scaling.RL uses AdamW at 1 × 10−6, generation batch size 64, eight rollouts per prompt, and two GRPO iterations.

B LLM Judge Prompt

The evaluation uses GPT-5.4 as a binary semantic-consistency judge, prioritizing factual correctness over exact lexical matching.

  • B LLM Judge Prompt: GPT-5.4 judges whether each model prediction is semantically consistent with the reference answer using a binary decision focused on factual correctness rather than exact lexical matching.

Prompt for LLM-Based Answer Evaluation

The evaluator judges whether a model answer is correct against a question and standard answer, prioritizing semantic equivalence and essential content over exact wording. It returns only JSON containing a correctness label and brief reasoning.

  • The evaluator compares the question, standard answer, and model answer to determine whether the response is correct.
  • It accepts equivalent wording, synonyms, abbreviations, alternative names, formatting differences, and reasonable numerical precision or unit variations.
  • The model answer must include the essential information required by the question, while additional information is acceptable unless it contradicts the correct answer.
  • The evaluator marks answers incorrect for factual errors, different entities or concepts, omitted critical information, or failure to answer the question.
  • The output is restricted to valid JSON with an is_correct boolean and concise reasoning; explanations are judged by core conclusions and required facts rather than identical detail.

C Visual Anchor Extraction Prompt … Prompt for Initial Trajectory Refinement

The pipeline extracts visually grounded anchors, generates multi-hop video–Web QA records, and constructs executable trajectories whose tool dependencies and evidence flow are explicitly refined. Prompts enforce visual grounding, source-backed answers, causally ordered operations, and avoidance of unsupported or redundant steps.

  • C Visual Anchor Extraction Prompt: The anchor prompt uses ASR only to name or disambiguate visible targets, while rejecting loose transcript mentions and unsupported exact identities or event names.Unknown or partially resolved identities remain unresolved rather than being inferred from visual familiarity, and weak, unlocalizable, duplicate, or clutter targets are dropped.
  • Prompt for Visual Anchor Extraction: Visual anchor extraction scans the entire video for searchable entities and representative events, requiring concrete visual, temporal, or spatial locators for downstream retrieval and QA construction.Anchors include products, signs, screen items, places, public entities, actions, interactions, state changes, procedures, outcomes, and causal moments; event descriptions must rely on visible evidence.
  • Prompt for QA Generation: QA generation creates exactly the requested number of natural multi-hop records whose answers combine video evidence with explicit external source facts.Questions indirectly describe visual or temporal targets, avoid revealing decisive facts or tool operations, and distinguish entity identification from event-based temporal, causal, procedural, or interaction reasoning.
  • D QA Generation Prompt: QA records exclude dialogue, captions, transcripts, commentary, music, and lyrics, ensuring questions depend on visual evidence and source-backed facts rather than speech.Entity questions require identifying visual targets before external retrieval, and final answers must be explicit source facts or direct computations from them.
  • Prompt for Initial Trajectory Generation: Initial trajectory generation maps each QA record to executable temporal, spatial, image-search, Web-retrieval, page-visit, and reasoning operations.Visible identities require spatio_grounding followed by image_search; reasoning combines prior evidence or records OCR from a grounded region without introducing new observations or facts.
  • E Initial Trajectory Generation Prompt: Initial trajectories must be dependency-complete: every step depends on the question or earlier outputs, Web queries cannot contain their intended answers, and redundant grounding is avoided.Entity records explicitly preserve the grounding chains needed to identify their targets before retrieval and reasoning.
  • F Trajectory Refinement Prompt: Trajectory refinement preserves consistent QA content while rewriting reasoning steps to make dependencies complete, ordered, traceable, and executable before downstream grounding.It temporally localizes distinct moments before spatial grounding, matches image-search queries to preceding spatial targets, and inserts earlier retrieval when disambiguating context is needed.
  • Prompt for Initial Trajectory Refinement: Refinement adds Web search before new external claims, restricts reasoning to earlier evidence, omits unnecessary tools, shortens queries, and renumbers steps logically.OCR remains a reasoning result from a spatially grounded region rather than a standalone tool action, and outputs follow the specified JSON schema.

G Model-Conditioned Tool Necessity Filtering Prompts … Prompt for Web Query Rewrite

The pipeline filters unnecessary tools by testing whether the base model can recover downstream targets without them, while targeted retry prompts repair unreliable grounding or web-search evidence. Its prompts cover temporal, spatial, image, web, page-visit, consistency, grounding-retry, and query-rewrite operations.

  • G Model-Conditioned Tool Necessity Filtering Prompts: Tool outputs are withheld, and a tool or chain is removed only when the base model directly recovers a semantically consistent downstream result.This model-conditioned test evaluates necessity against information available before execution.
  • Prompts for Model-Conditioned Tool Necessity Filtering: Filtering prompts test temporal timestamps, spatial boxes, image entities, internal-knowledge web answers, search-result answers, and semantic consistency in JSON-only formats.Each prompt asks the model to solve the downstream task while omitting the corresponding tool.
  • H Reflection and Retry Prompts: When execution yields insufficient or unreliable evidence, the pipeline invokes a targeted retry matched to the failed operation.The principal recovery behaviors revise video localization or rewrite an external search query.
  • Prompt for Grounding Retry: Grounding retry re-examines video evidence, revising temporal or spatial descriptions while forbidding unsupported entity identities.The retry returns a JSON action, revised grounding target, and brief reason.
  • Prompt for Web Query Rewrite: Web-query rewriting seeks a concise query more likely to retrieve the expected information after prior attempts fail.The prompt supplies the original query, expected information, available context, and previous attempts.
  • Prompt for Web Query Rewrite: Rewritten web queries include the key entity or topic and missing attribute, but exclude the expected answer itself.These constraints focus retrieval on verifying missing information rather than leaking the target answer.
  • Prompt for Web Query Rewrite: Query rewrites use only available context, the original query, or previous attempts, and return JSON containing the rewritten query and reason.The prompt prohibits adding facts that the search step is intended to discover.
Loading 2608.25559v1…