PaperlayerView exact revision on arXiv

Source-linked AI summary

VCA: Video Curious Agent for Long Video Understanding

Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, Chuang Gan

arXiv:2412.10471v2cs.CVcs.AI

TL;DR

Long-video understanding is challenged by uneven information density and the costs of dense sampling or auxiliary processing. VCA explores video segments using VLM-generated intrinsic rewards, and reports state-of-the-art performance and efficiency without auxiliary tools.

  • Problem

    Long-video QA must handle uneven information density, while dense sampling can waste computation and auxiliary captioning approaches remain costly.

    Source: §1 · p. 1

  • Method

    VCA uses tree-search self-exploration, guided by VLM-generated intrinsic rewards, to navigate video segments and gather relevant information.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 4

  • Results

    Across long-video benchmarks, VCA achieves state-of-the-art performance and efficiency; on VideoMME, it outperforms VideoTree by 4.9% with less than 20% of observed frames.

    Source: §6 · p. 9, §9.1 · p. 15

  • Takeaways & Limitations

    The authors identify more accurate reward guidance, potentially developed with synthetic data and a specialized reward model, as a valuable future direction.

    Source: §6 · p. 9

    Reward-model errors can direct the agent to the wrong segments and produce an incorrect answer.

    Source: §10.2 · p. 18

Abstract

from arXiv · show

Long video understanding poses unique challenges due to their temporal complexity and low information density. Recent works address this task by sampling numerous frames or incorporating auxiliary tools using LLMs, both of which result in high computational costs. In this work, we introduce a curiosity-driven video agent with self-exploration capability, dubbed as VCA. Built upon VLMs, VCA autonomously navigates video segments and efficiently builds a comprehensive understanding of complex video sequences. Instead of directly sampling frames, VCA employs a tree-search structure to explore video segments and collect frames. Rather than relying on external feedback or reward, VCA leverages VLM's self-generated intrinsic reward to guide its exploration, enabling it to capture the most crucial information for reasoning. Experimental results on multiple long video benchmarks demonstrate our approach's superior effectiveness and efficiency.

1. Introduction

Long videos combine extended visual narratives with uneven information density, making efficient extraction and reasoning over relevant content difficult. VCA addresses this with curiosity-driven segment exploration, intrinsic rewards, and memory management, reporting stronger performance while observing fewer frames at comparable or lower computational cost.

1. Introduction

  • Long-form video understanding is needed for extended visual narratives, including surveillance footage, documentaries, and instructional videos.Such videos may range from minutes to hours and require multimodal processing and reasoning over long sequences.Source: §1 · p. 1
  • VCA outperforms baselines while observing fewer frames and incurring comparable or lower computational cost on EgoSchema with GPT-4o and Qwen2-VL-72B.The comparison is shown in Figure 1.Source: Figure 1 · p. 1, §1 · p. 2
  • Dense, uniform frame sampling can spend computation on redundant content while overlooking frames with essential information.Video segments differ in information density, with some containing critical details and others mostly trivial content.Source: §1 · pp. 1–2
  • Existing approaches can remain costly: captioning an hour-long video with Qwen2-VL-72B at 1 fps takes over 3 hours on one H100.The cited approaches may also select key frames by image similarity rather than holistic video content, risking missed features.Source: §1 · p. 2
  • VCA explores video segments using tree search, intrinsic rewards, and a fixed memory buffer that retains key information and discards irrelevant frames.The framework is designed to work without auxiliary tools.Source: §1 · p. 2

2. Related Work

Prior work has developed multimodal agents and video-processing methods, but efficient extraction of relevant information from long videos remains a challenge. VCA’s related-work context spans agent capabilities, video architectures, sampling, and auxiliary processing.

2.1. Long-context Multi-modal Agents

  • Long-context multimodal agents can make decisions in environments, yet VLM agents still struggle to extract crucial information and reason effectively over lengthy inputs.VCA aims to address this by enabling agents to self-explore and gather relevant information.Source: §2.1 · p. 2

2.2. Long Video Understanding

  • Video understanding has used two-stream networks, 3D CNNs, Vision Transformers, and self-supervised pretraining for spatial-temporal modeling and transferability.The passage places these approaches in the progression of video-understanding methods.Source: §2.2 · pp. 2–3
  • Frame sampling and feature extraction reduce processing costs, but sampling can miss essential context in lengthy videos, while processing full sequences as tokens is costly.Transitions and subtle cues are among the context that sampling may miss.Source: §2.2 · p. 3
  • Auxiliary captions, multimodal summaries, and tools can add computation and memory costs; VideoTree requires captioning every frame, whereas VCA uses a single VLM without supplementary tools.Some auxiliary-tool approaches also require traversing the entire video.Source: §2.2 · p. 3

3. Method

VCA iteratively explores video segments within a limited context window, using query-relevance rewards and a fixed-size memory buffer to guide and bound its search. Its tree-search agent can use rewards as guidance while retaining the option to choose or revisit segments.

3. Method

  • VCA iteratively analyzes videos within a limited context window, extracting essential information in a coarse-to-fine process rather than relying on dense sampling or captioning.The approach is described as self-exploration of video environments.Source: §3 · p. 3

3.1. Reward Model

  • The reward model scores segments by relevance to the query to guide exploration; the same VLM-based agent can serve as both reward model and explorer.The model can explain each sub-segment and assign a relevance score, with previous reward history used alongside new segments for consistency.Source: §3.1 · p. 4

3.2. Memory Management

  • VCA’s fixed-size memory buffer retains the most relevant frames and removes low-scoring ones when the buffer limit is exceeded.Descriptions of discarded frames are encoded in exploration history, allowing reasoning within a fixed resource limit.Source: §3.2 · p. 4

3.3. Tree-Search Exploration

  • The video is an exploration tree of segments; the agent expands a selected segment or answers once it has gathered sufficient information.The agent receives candidate segments and a memory buffer containing frames.Source: §3.3 · pp. 4–5
  • Because reward scores can be inaccurate, the agent uses them as guidance rather than selecting greedily, and can backtrack to earlier coarse or fine segments.The method is intended to balance exploitation and exploration; empirical results are said to validate tree search over naive frame selection.Source: §3.3 · p. 5

4. Experiments

Across EgoSchema and LVBench, VCA improves over uniformly sampled GPT-4o while using fewer frames; ablations indicate that reward guidance, tree search, and buffer size contribute to performance or efficiency.

4.1. Experimental Settings

  • VCA uses EgoSchema’s 500 validation QA pairs and LVBench’s roughly 1,500 evaluation samples; experiments use August 2024 GPT-4o and buffers of 8 and 16 frames, respectively.Video-model baselines use uniform frame sampling, while agent baselines follow their official implementations with GPT-4o.Source: §4.1 · p. 5

4.2. Experimental Results

  • With less than 30% of frames, VCA gains 3.2% on the EgoSchema subset and 6.6% on LVBench over uniformly sampled GPT-4o.The experiments use the EgoSchema and LVBench result tables.Source: §4.2 · p. 6, Table 2 · p. 6, Table 1 · p. 6
  • VCA outperforms the compared video-agent baselines by over 5% on EgoSchema while processing fewer frames, with similar insights reported for LVBench.The comparison re-implements VideoAgent and VideoTree with GPT-4o for reasoning, but uses open-source VLMs for captioning because GPT-4o captioning is prohibitively expensive.Source: §4.2 · p. 6, Table 2 · p. 6

4.3. Ablation Study

  • Removing the reward model reduces performance on both benchmarks, indicating that the agent’s reasoning alone does not reliably identify informative segments.The ablation table covers EgoSchema and LVBench; detailed LVBench results are in Appendix 9.Source: §4.3 · p. 6, Table 3 · p. 7
  • Without segment-based tree search, performance drops and acquired frames increase, especially on LVBench, producing a longer, more redundant trajectory.The ablation replaces tree search with direct frame-index selection.Source: §4.3 · p. 7
  • Increasing the LVBench memory buffer size consistently improves performance, although tests were limited to 32 frames by computation resources.The reported trend suggests potential gains with additional frames.Source: §4.3 · p. 7, Table 4 · p. 7

5. Analysis

VCA generalizes to an open-source model while using fewer frames, adapts exploration beyond reward scores alone, and benefits from stronger reward guidance. Its reward model’s segment choices align with ground truth, and ground-truth reward scores further improve LVBench results.

5.1. Can VCA generalize to Open-source Models?

  • On EgoSchema, VCA with Qwen2-VL beats uniformly sampled Qwen2-VL using about 60% of the frames, outperforms VideoTree with far fewer frames, and exceeds VideoAgent.VideoTree’s caption-generation process requires over 60 GPU hours on a single H100.Source: Table 5 · p. 7, §5.1 · p. 7

5.2. What’s the Exploration Behavior of VCA?

  • VCA redirects exploration after visual inspection rather than following reward scores blindly; segment-level semantic exploration also outperforms selection based only on start- and end-image relevance on EgoSchema.In the described trajectory, it leaves an irrelevant high-reward segment, then selects a segment with a lower reward score than an alternative.Source: §5.2 · pp. 7–8

5.3. How Reliable is the Reward Model?

  • Reward-model top-scoring segments have a mean ground-truth distance of 1.28, while agent selections have an average distance gap of −0.60 versus greedy selection and align closer to ground truth.The agent’s segment-distance distribution is sharper than the greedy approach’s.Source: §5.3 · pp. 8–9, Figure 5 · p. 9

5.4. Can VCA Benefit from Better Reward?

  • Ground-truth reward scores yield a consistent 3% improvement on LVBench with both GPT-4 and LongVA, indicating potential gains from more accurate reward guidance.The comparison substitutes ground-truth time references for the reward scores.Source: §5.4 · p. 9, Table 6 · p. 9

6. Conclusion

VCA is a self-exploring agent for long-video understanding that achieves state-of-the-art performance and efficiency without auxiliary tools. The authors identify more accurate reward guidance as a promising direction for future work.

6. Conclusion

  • VCA achieves state-of-the-art performance and efficiency on long-video understanding without auxiliary tools, with more accurate reward guidance identified as a future-work opportunity.The authors suggest collecting synthetic data and training a specialized reward model.Source: §6 · p. 9

7. Prompt Design

VCA’s prompts guide reward scoring and iterative exploration. They carry reward history across rounds to keep relevance scores consistent, while the agent uses stored frames and dialogue history to choose whether to answer or explore further.

7. Prompt Design

  • The first-round reward prompt asks the model to assess each video segment’s relevance to the question.The prompt templates for the reward model and exploration agent are shown in Figures 6–8.Source: §7 · p. 13
  • For later rounds, the reward prompt includes prior reward history because the model sees only sampled frames from the selected local segment, helping maintain a consistent relevance-score scale.Source: §7 · p. 13
  • The exploration agent uses relevance scores, stored frames, and dialogue history to decide whether it has enough information to answer or should continue exploring.Source: §7 · p. 13

8. Implementation Details

The implementation details cover four benchmarks, dataset and prompt examples, and baseline configurations. The authors use GPT-4o for evaluation and baseline reasoning, while noting a potential training-data overlap concern for movie-based benchmarks.

8.1. Dataset

  • The four evaluated benchmarks include minute-level EgoSchema and MMBench-Video and hour-level LVBench and long-subset VideoMME; GPT-4o is used to evaluate VideoMME.Source: Table 7 · p. 13, §8.1 · p. 13
  • The LVBench ablation table reports that both tree-search exploration and the reward model contribute to performance and efficiency improvements.The agent’s memory buffer is set to 16 frames in this ablation.Source: Table 9 · p. 14
  • MovieChat and MovieQA are excluded because publicly available movie videos may have appeared in recent closed-source VLM training data, risking data leakage.Source: §8.1 · pp. 13–14
  • Dataset prompt examples follow each benchmark’s paper or official implementation, and the appendix shows sample instances with their instruction templates.The reward-model prompts show initial-round scoring and later-round exploration prompts with historical segment information and reward annotations.Source: Table 8 · p. 13, §8.1 · p. 14, Figure 6 · p. 14

8.2. Baselines

  • VideoAgent and VideoTree are reimplemented with GPT-4o reasoning and Qwen2-VL captions at 0.2 FPS; GPT-4o captioning is described as computationally prohibitive.Captioning a single 30-minute video with GPT-4o would require over 4M tokens.Source: §8.2 · p. 14
  • VideoAgent’s initial-frame count is varied for EgoSchema, while LVBench and Video-MME use 15 initial frames; VideoTree’s clustering hyperparameters are also specified.The VideoTree settings are max breadth 32, max depth 3, branch width 4, and relevance-number threshold 4.Source: §8.2 · pp. 14–15

9. Experiments

VCA outperforms comparison methods on VideoMME’s long split while observing fewer frames; ablations show that reward guidance and tree-search exploration contribute differently across task domains.

9.1. Experimental Results

  • VCA outperforms VideoTree by 4.9% with less than 20% of observed frames and VideoAgent by 12.1% with about 75% on VideoMME’s long split.The split contains videos averaging over 2,000 seconds; similar trends are reported on MMBench-Video.Source: Table 10 · p. 15, §9.1 · p. 15, Table 11 · p. 16
  • The exploration prompt presents the target question and task, then segment candidates, allowing the agent to explore a segment or answer when it has enough information.Figure 8 describes the prompt template and its decision options.Source: Figure 8 · p. 16

9.2. Ablation Study

  • Without the reward model or tree-search exploration, performance improves in Key Information Retrieval and Summarization as the agent explores more broadly or nearly twice as many frames.The ablation reports that both components enhance performance overall, despite these domain-specific exceptions.Source: §9.2 · p. 16
  • Reward guidance and tree-search exploration contribute to performance gains in Event Recognition and Temporal Grounding, where precise, focused exploration is essential.The ablation describes the two components as addressing distinct aspects of exploration.Source: §9.2 · p. 16

10. Case Study

The case studies identify failures in finding subtle visual details, following inaccurate reward guidance, and reasoning from correctly identified segments. The discussion distinguishes the especially difficult visual-detail cases from errors that more capable multimodal foundation models could mitigate.

10. Case Study

  • The case studies examine common failure cases to provide data points and insights for future research.Source: §10 · p. 17

10.1. Failure Mode: Inability to Detect Subtle Visual Details

  • Subtle visual details can remain undiscovered after prolonged exploration, leading to a guessed answer even when the agent observes about three times the average number of frames.One example involves briefly visible kitchen-wall decorations during rapid camera movements, a task described as challenging even for humans.Source: Figure 9 · p. 17, §10.1 · p. 18

10.2. Failure Mode: Guidance Errors from the Reward Model

  • Incorrect reward scores can steer the agent away from the needed segment: segments 4 and 6 receive 100% relevance, while the answer is in segment 5.The resulting focus on the wrong segments produces an incorrect answer.Source: Figure 10 · p. 18, §10.2 · p. 18

10.3. Failure Mode: Limited Multi-modal Reasoning Abilities

  • The agent can identify crucial segments yet answer incorrectly because of limited spatial reasoning or inadequate alignment of multimodal knowledge.In the example, it finds the frame naming the Best Lead Actress winner but fails to give the correct response.Source: §10.3 · p. 18

10.4. Discussion

  • More robust multimodal foundation models could mitigate reward-guidance and reasoning failures, while subtle visual-detail detection remains challenging even for humans.Figure 11 illustrates a case where the agent identifies the ground-truth visual information but gives an incorrect response.Source: §10.4 · pp. 18–19, Figure 11 · p. 19

Related papers

Loading 2412.10471v2…