Source-linked AI summary
Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding
Xiaoyi Zhang, Zhaoyang Jia, Zongyu Guo, Jiahao Li, Bin Li, Houqiang Li, Yan Lu
TL;DR
Long-form video understanding remains difficult because hour-long videos require extensive temporal-spatial reasoning and information retrieval. DVD addresses this with an adaptive agentic search framework over a multi-granular video database, achieving state-of-the-art results including 74.2% accuracy on LVBench and 76.0% with transcripts.
Problem
Long-form video understanding requires reasoning over extensive temporal-spatial complexity and retrieving information from information-dense, hour-long videos.
Method
DVD uses a multi-granular video database and search-centric tools, allowing an LLM agent to adaptively plan and combine searches for different queries.
Results
74.2% accuracy on LVBench improves to 76.0% with auxiliary transcripts, with evaluations reporting strong performance across long-video benchmarks.
Takeaways & Limitations
DVD offers a scalable and flexible framework for comprehensive long-video analysis through adaptive integration of global browsing, clip search, and frame inspection.
Takeaways & Limitations
Iterative reasoning introduces higher computational overhead, motivating more effective database construction and searching to reduce costs.
Abstract
from arXiv · showhide
Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. To overcome such limitations, we propose the Deep Video Discovery (DVD) agent to leverage an agentic search strategy over segmented video clips. Unlike previous video agents that rely on predefined workflows applied uniformly across different queries, our approach emphasizes the autonomous and adaptive nature of agents. By providing a set of search-centric tools on multi-granular video database, our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information. We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates our advantage. Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of 74.2%, which substantially surpasses all prior works, and further improves to 76.0% with transcripts. The code has been released at https://github.com/microsoft/DeepVideoDiscovery.
1 Introduction
Deep Video Discovery reframes long-form video understanding as adaptive information search over segmented clips, using multi-granular tools to gather evidence. It achieves 74.2% accuracy on LVBench and 76.0% with auxiliary transcripts.
- Long-form videos require integrating fine-grained spatiotemporal details with semantic relations distributed across extended temporal intervals.
- Deep Video Discovery treats long-video understanding as a multi-step information search problem over shorter video clips.
- Existing video agents often use manually designed search workflows, which can be inefficient for fine-grained queries and non-temporally local entities.
- DVD provides Global Browse, Clip Search, and Frame Inspect tools operating at global, clip, and pixel-level granularities.
- The agent autonomously combines complementary tools through adaptive reasoning, strategy orchestration, and iterative decision-making tailored to each query.
- 74.2% accuracy on LVBench rises to 76.0% with auxiliary transcripts, while ablations support the effectiveness of the tool design.
2 Related Work
Related work addresses long-video understanding through visual-token and context-length strategies, while agent research develops tool-using systems for information gathering and planning. DVD extends these directions to long-video understanding.
- Long-video understanding research tackles extended temporal and spatial reasoning by extending model context or reducing visual-token redundancy.
- Agent and tool-use research enables LLMs to acquire information, plan, and execute actions in complex environments.
3 Deep Video Discovery
DVD constructs a multi-granular video database and equips an LLM with search-centric tools for iterative agentic search and answering. The agent selects and combines actions adaptively to locate and inspect relevant evidence.
- Overview: DVD comprises a multi-grained video database, search-centric toolset, and LLM orchestrator that iteratively chooses tool actions or ANSWER for a query.
- Overview: The pipeline has database construction followed by Agentic Search and Answer with Tool Use.
- Multi-granular Video Database Construction: The database uses global summaries, clip captions, and indexed original frames to support retrieval while preserving detailed visual content.
- Multi-granular Video Database Construction: The input video is uniformly partitioned into non-overlapping short clips, whose frames are decoded for subsequent processing.
- Multi-granular Video Database Construction: A progressive subject registry guides VLM captioning across clips to produce subject-centric global information.
- Multi-granular Video Database Construction: The resulting structured database stores frames, captions, and embeddings, linking searchable representations to source pixels.
- Agentic Search and Answer with Tool Use: The agentic stage provides tools for global understanding, semantic clip retrieval, and detailed exploration of original video content.
- Search-centric Tool Preparation: Global Browse returns subject-centric and event-centric summaries that provide high-level contextual information.
4 Experiment
DVD is evaluated across long-video benchmarks, model configurations, search tools, efficiency settings, and reasoning behaviors. Results show strong benchmark performance, benefits from integrated tools and adaptive workflows, and behavior-dependent accuracy.
- Model Ablations: 68.5% accuracy is achieved with DeepSeek-R1, while DVD reaches 57.3% with the 32B Qwen3-32B model on LVBench.The 32B configuration outperforms existing VLM approaches, including GPT-4o and OpenAI o3.
- Tool Ablations: Removing Clip Search causes a 12.3% decline, compared with 8.4% without Frame Inspect and 2.9% without Global Browse.Clip Search supports iterative refinement, Frame Inspect provides fine-grained VQA, and Global Browse supports global summarization and long-range event linking.
- Efficiency Analysis: The adaptive workflow outperforms a hand-crafted VideoAgent workflow by 4.0% even when the latter uses 52% more reasoning steps.The hand-crafted workflow reaches 70.2% accuracy at 11.1 average steps, while replacing the adaptive workflow causes a 7.5% absolute accuracy drop.
- Reasoning Behavior: Simple Action is the most common strategy and achieves high accuracy, whereas Iterative Search uses 8.0 versus 5.2 steps for OpenAI o3 and yields slightly lower accuracy.Clip Search Trap is a primary OpenAI o3 failure mode when repeated searches fail to locate information in the video database.
- Reasoning Behavior: Longer reasoning trajectories can signal uncertainty within a model but correlate with superior performance across models when reasoning is more thorough.GPT-4o collapses into Simple Action on 91.4% of queries, averaging 4.6 reasoning steps and frequently concluding prematurely.
5 Conclusion
Deep Video Discovery combines multi-granular search tools with iterative reasoning to analyze long-form videos. The framework reports state-of-the-art performance, while acknowledging computational overhead from iterative reasoning.
- Deep Video Discovery uses multi-granular search tools on a constructed database for iterative search and reasoning over extensive video content.
- The approach adaptively integrates global browsing, clip search, and frame inspection for long-form video understanding.
- The framework achieves state-of-the-art results on multiple long video understanding benchmarks, with ablations supporting its tool design.
- Iterative reasoning introduces higher computational overhead, motivating future work on database construction and searching to reduce reasoning difficulty and cost.
Change Log
The change log records revisions from the initial submission through the camera-ready version, including an evaluation-code fix and repository update.
- The initial submission was recorded on 2025-05-23.
- On 2025-05-28, the evaluation code was fixed to account for parenthesized answers, improving reported accuracy consistently.
- On 2025-07-24, the GitHub repository link was updated.
- The 2025-11-03 camera-ready version updated Tables 6 and 7 to discuss DVD efficiency.
A Evaluation Details
The evaluation details describe the prompts used for DVD’s model components and report API cost and service limitations affecting evaluation.
- The appendix documents prompts for captioning and subject extraction, reasoning, frame inspection, and event-centric global information gathering.
- DVD costs an average of $0.213 and uses 0.15 million tokens per LVBench question with OpenAI API pricing at 2025-11-03.
- Azure OpenAI Service evaluation is affected by a maximum input-image restriction and a safety content-filtering mechanism.
A.3.1 Maximum Image Count Limitation
Azure OpenAI Service limits each request to 50 images, so DVD combines frames into composite images to provide richer visual context under this constraint.
- Azure OpenAI Service imposes a 50-image limit per request, below the token limits of the large VLMs used.
- DVD horizontally splices up to five frames per image, allowing up to 250 frames for Global Browse.
- DVD uses up to 50 frames in Frame Inspect, while the 250-frame setting is reserved for event-centric Global Browse information gathering.
- OpenAI o3 is evaluated with 256 frames arranged as 32 images using a 2×4 splicing format, which performs better than DVD’s 1×5 format.
A.3.2 API Content Filtering
Azure OpenAI safety filtering occasionally blocks benign inputs, affecting both OpenAI o3 and DVD. DVD remains operational through adaptive handling, though blocked captioning queries leave database entries incomplete.
- Filtering impact: Safety filtering occasionally misclassifies benign inputs as offensive and blocks requests.This affects the performance of both OpenAI o3 and the DVD agent.
- Filtering impact: 10% to 25% of queries are filtered across benchmarks.The filtering rate is reported in Table 8.
- DVD mitigation: DVD lets the agent decide how to handle content-filtering failures without additional error handling.The agent may invoke alternative tools or answer using available information.
- DVD mitigation: Blocked captioning queries leave corresponding database entries blank, reducing database completeness while preserving agent operation.The adaptive behavior allows DVD to remain operational despite incomplete caption data.
B More Results
Repeated evaluation on LVBench with auxiliary transcriptions produced a consistent DVD score, supporting the reported robustness of the method.
- Statistical significance: 74.0 average score with 0.125 variance was achieved across three LVBench evaluations using auxiliary transcriptions.The repeated runs were used to assess statistical significance and consistency.
B.2 Case Study
Case studies show DVD adapting its tool sequence to different long-video questions, while also exposing failure modes in Frame Inspect and Clip Search. The examples illustrate both successful evidence refinement and errors caused by misleading or sparse retrieval results.
- Global Browse Only: Global Browse can classify a video by combining subject-level and event-level summaries of its global context.In the example, this process correctly identifies a technology documentary.
- Simple Action: When Global Browse is wrong, Clip Search locates the relevant event and Frame Inspect provides finer-grained evidence.This sequence corrects the initial claim that mirror fragments are thrown away.
- Iterative Search: Iterative Search issues additional Clip Search queries when initial tool results leave the pursued subject ambiguous.The case involves multiple subjects and compares searches for a man chasing a dog versus a bird.
- Frame Inspect Trap: Frame Inspect can return misleading interpretations, causing the agent to answer incorrectly when it cannot distinguish reliable information.The failure occurs despite three consecutive questions around a target timestamp.
- Clip Search Trap: Clip Search fails when the database contains few directly relevant captions or subjects, even after query rephrasing and refinement.The resulting inability to identify the correct time range leads to an incorrect answer.
- Broader utility: DVD’s adaptive search behavior supports natural-language interaction with long-form video and can reduce manual review for summarization and analysis.The paper also identifies accessibility benefits for users with visual or cognitive impairments.
C.2 Negative Impacts
DVD’s use of large models and agentic search introduces potential bias and substantial computational demands. These concerns create sustainability and accessibility challenges for deploying the system broadly.
- Model-related risks: Large-model captioning and reasoning can inherit training-data bias, producing inaccurate or unfair interpretations of video content.The concern applies to the LLMs and VLMs used by DVD.
- Resource demands: The agentic search process requires substantial computational resources.This resource demand is identified as a deployment concern for DVD.
- Resource demands: Computational demands raise sustainability and accessibility concerns.The paper presents these as consequences of the resource requirements of agentic search.