PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoAgent: Long-form Video Understanding with Large Language Model as Agent

Xiaohan Wang, Yuhui Zhang, Orr Zohar, Serena Yeung-Levy

arXiv:2403.10517v1cs.CVcs.AIcs.CLcs.IR

TL;DR

Long-form video understanding requires reasoning over lengthy multimodal sequences, yet humans address this challenge through interactive, question-guided information gathering. VideoAgent models this process with an LLM agent and vision-language tools, achieving strong zero-shot accuracy on EgoSchema and NExT-QA while using few frames.

  • Problem

    Long-form video understanding requires models to process multimodal information, handle exceedingly long sequences, and reason over them effectively.

  • Method

    VideoAgent uses an LLM agent to iteratively assess information needs, rewrite retrieval queries, select relevant frames, and use CLIP and VLM tools to gather observations.

  • Results

    VideoAgent achieves 54.1% and 71.3% accuracy on EgoSchema and NExT-QA, respectively, outperforming LLoVi by 3.8% and 3.6% while using 8.4 and 8.2 frames on average.

  • Takeaways & Limitations

    The framework emphasizes interactive reasoning and planning over directly processing lengthy visual inputs and is presented as a significant step toward agent-based long-form video understanding.

Abstract

from arXiv · show

Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding.

1 Introduction

Long-form video understanding requires processing multimodal, lengthy sequences while reasoning effectively, capabilities existing models struggle to combine. VideoAgent instead uses an LLM-controlled, iterative search process that mirrors human understanding and achieves strong benchmark accuracy with few frames.

  • Long-form video understanding requires multimodal processing, long-sequence handling, and effective reasoning, which existing models struggle to achieve simultaneously.
  • Humans first establish video context, then iteratively select question-relevant frames until sufficient information is available, making interactive reasoning central to the process.
  • VideoAgent models understanding as iterative states, actions, and observations controlled by an LLM, using CLIP for frame retrieval and VLMs for visual descriptions.
  • VideoAgent differs from prior approaches by selecting frames across multiple rounds and rewriting retrieval queries for more accurate, fine-grained information gathering.
  • 54.1% and 71.3% accuracy on EgoSchema and NExT-QA exceed LLoVi by 3.8% and 3.6%, respectively.VideoAgent uses 8.4 frames on average, 20x fewer than LLoVi.
  • The framework emphasizes agent-based reasoning over directly modeling lengthy visual inputs and is presented as a direction for long-form video understanding.

2 Related Work

Long-form video methods balance performance against the computational burden of high-dimensional spatio-temporal inputs through compressive or selective sparsity. VideoAgent extends selective retrieval by letting an LLM direct frame sampling rather than relying only on fixed or question-driven selection.

  • 2.1 Long-form Video Understanding: Long-form video understanding is computationally demanding because its spatio-temporal inputs are complex and high-dimensional.
  • 2.1 Long-form Video Understanding: Compressive sparsity methods reduce long videos into lower-dimensional embeddings or representations to limit token redundancy.
  • 2.1 Long-form Video Understanding: Selective-compressive methods sub-sample frames using the input question or text as a guide to retrieve content relevant to the task.
  • 2.1 Long-form Video Understanding: VideoAgent lets an LLM direct which video frames should be sampled by the captioner, unlike prior approaches that use fixed or more limited selection strategies.
  • 2.2 LLM Agents: LLM agents make decisions and take actions in dynamic environments toward specific goals, drawing on reasoning, planning, tool use, and self-reflection capabilities.
  • 2.2 LLM Agents: In long-form video understanding, VideoAgent reformulates the task as decision-making rather than merely adding tools or functionalities to existing systems.

3 Method

VideoAgent represents long-form video understanding as an iterative state-action-observation process controlled by an LLM. It begins with uniformly sampled context, predicts and self-reflects, retrieves missing information segment-wise, and updates the state with newly captioned frames.

  • 3 Method: VideoAgent follows a human-inspired process of glancing at context frames, searching additional frames, and aggregating information before answering.
  • 3 Method: The method represents the process as states, actions, and observations, where states store seen-frame information, actions answer or search, and observations are newly retrieved frames.
  • 3 Method: GPT-4 serves as the agent controlling the process through memory, reasoning, planning, and tool-use capabilities.
  • 3 Method: The initial state is formed by uniformly sampling N frames, captioning them with a VLM, and supplying the resulting descriptions to the LLM.
  • 3 Method: The LLM predicts an answer, self-reflects with a three-level confidence score, and either exits with an answer or searches for missing information.
  • 3.3 Gathering a New Observation: Segment-level retrieval avoids confusing repeated content across time by restricting searches to temporally relevant video segments.
  • 3.3 Gathering a New Observation: CLIP selects the frame with highest text-image cosine similarity within each segment, and VLM captions are merged into the state for the next prediction round.
  • 3.3 Gathering a New Observation: Segment-level CLIP retrieval is computationally efficient through single-pass features, cached image representations, and computation limited to selected segments.

4 Experiments

VideoAgent is evaluated on established long-form video benchmarks and through analyses of its iterative frame-selection process and foundation-model components. It achieves strong zero-shot performance with few frames, while ablations show that adaptive iteration, self-evaluation, and segment selection materially affect efficiency and accuracy.

  • Benchmark Results: VideoAgent achieves 54.1% accuracy on EgoSchema and 71.3% on NExT-QA, surpassing LLoVi by 3.8% and 3.6%, respectively.It uses an average of 8.4 frames on EgoSchema and 8.2 frames on NExT-QA.
  • Frame Efficiency: 8.4 frames achieve 60.2% accuracy on the EgoSchema 500-question subset, exceeding uniform sampling of 180 frames at 59.6%.The comparison indicates that selecting informative frames can outperform using many uniformly sampled frames.
  • Number of Rounds: 53.8%, 58.6%, 60.2%, and 59.8% accuracy result from 1, 2, 3, and 4 rounds using 5, 7.5, 8.4, and 9.9 frames, respectively.Performance saturates at three rounds on the EgoSchema 500-question subset.
  • Different Question Types: Temporal questions require more selected frames than causal and descriptive NExT-QA questions.The analysis links dynamic frame allocation to differing information requirements across question types.
  • Initial Number of Frames: 60.2% accuracy with 8.4 average frames is obtained when five frames are initially sampled, compared with 58.4% and 57.4% using three and eight initial frames.The corresponding average frame counts are 6.4, 8.4, and 11.0.
  • Self-evaluation: Removing self-evaluation increases average frames from 8.4 to 11.8 and decreases accuracy from 60.2% to 59.6%.Self-evaluation determines whether sufficient information has been gathered before continuing iteration.
  • Segment Selection: Disabling segment selection causes a 3.6% accuracy degradation.Segment-specific queries improve temporal reasoning and reduce conflation of information from disparate segments.
  • Foundation-Model Ablations: GPT-4 significantly outperforms the compared LLMs, with structured JSON prediction identified as a major factor in its performance.The iterative process depends on correctly formatted JSON outputs for parsing.

5 Conclusion

VideoAgent uses a large language model as an agent to mirror human long-form video understanding and iteratively search for and aggregate information. Quantitative and qualitative studies on various datasets demonstrate its effectiveness and efficiency.

  • VideoAgent employs a large language model as an agent to mirror the human cognitive process for understanding long-form videos.The system searches for and aggregates information through a multi-round iterative process.
  • VideoAgent demonstrates exceptional effectiveness and efficiency in long-form video understanding across quantitative and qualitative studies on various datasets.

A Run-time Analysis of CLIP

VideoAgent’s runtime analysis attributes most computation to VLM captioning and LLM operations, while CLIP retrieval remains a small fraction of total effort. Segment-level retrieval further improves efficiency beyond the upper-bound estimate.

  • CLIP feature computation is computationally efficient because it uses a single feed-forward process and caches reusable image features through late interaction.
  • CLIP computes features for all N video frames and selected n frames, while VLM captioning processes n frames and LLM operations run across t rounds.The analysis assigns these costs to Nx, nx, ny, and tz terms, respectively.
  • 1.9% of total computational effort is attributed to CLIP feature computation under the stated runtime assumptions.The calculation uses 180 frames, 8.4 selected frames, three rounds, and the listed per-operation costs.
  • Segment-level retrieval computes features only within specified segments in practice, making the runtime estimate an upper bound.

B Additional Implementation Details

The implementation uses distinct captioning models for the two benchmarks and EVA-CLIP-8B-plus for frame retrieval. The captioners and CLIP model differ in architecture, input resolution, parameterization, and output representation.

  • CogAgent is used for NExT-QA as a frame-based captioner with 18B parameters and 1120 × 1120 input resolution.
  • LaViLa is used for EgoSchema as a clip-based captioner with 4 × 336 × 336 clip input resolution.The zero-shot setup uses a LaViLa model retrained on ego4D data while filtering overlapping EgoSchema videos.
  • EVA-CLIP-8B-plus retrieves frames using a 7.5B-parameter vision encoder and 0.7B-parameter text encoder at 448 × 448 image resolution.It produces 1280-dimensional output features.

C Prompts for GPT-4

The GPT-4 prompting process supplies sampled-frame descriptions, evaluates whether the available information is sufficient, and requests additional relevant frames when needed. It then uses the gathered evidence to select an answer, as illustrated by a dough-processing example.

  • The prompts are used for GPT-4 answer prediction, self-reflection, and finding missing information during iterative video understanding.
  • Prediction prompt: The prediction prompt asks GPT-4 to select the best answer index from a question, sampled-frame descriptions, and multiple-choice options.The example requests the result in JSON format as a final answer index.
  • Prompt example: The dough example selects frames around picking up dough, using the dough roller, and placing dough on a tray to reconstruct the process order.The resulting answer is option 4, describing rolling the dough on the table, rolling it on the flour, and placing it in the tray.
  • Self-reflection prompt: The self-reflection prompt evaluates whether the initial uniformly sampled frames provide sufficient information to answer the question.In the example, GPT-4 concludes that the initial five frames are insufficient.
  • Frame-selection prompt: When information is insufficient, the prompt divides the video into four segments and identifies relevant frames containing visual elements that support the query.The requested descriptions focus on objects, humans, interactions, actions, and scenes, with multiple frames allowed from one segment when necessary.
Loading 2403.10517v1…