PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding

Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, Qing Li

arXiv:2403.11481v2cs.CV

TL;DR

Long-form video understanding is difficult because lengthy videos impose computation and memory costs and contain long-range spatial-temporal relations. VideoAgent addresses this with unified temporal and object memories plus interactive LLM-driven tool use, achieving comparable or better results across challenging benchmarks, including reported gains of 6.6% on NExT-QA and 26.0% on EgoSchema over baselines.

  • Problem

    Long-form video models face increasing computation and memory costs and can struggle to capture long-range spatial-temporal relations.

  • Method

    VideoAgent builds temporal memory for segment descriptions and object memory for tracked object states, then interactively invokes tools to answer task queries.

  • Results

    VideoAgent offers comparable or better empirical results than end-to-end multimodal LLMs and tool-use agents across challenging long-form benchmarks, with average increases of 6.6% on NExT-QA and 26.0% on EgoSchema over baselines.

  • Takeaways & Limitations

    A minimalist memory-centered tool-use pipeline can support strong video understanding without expensive training.

Abstract

from arXiv · show

We explore how reconciling several foundation models (large language models and vision-language models) with a novel unified memory mechanism could tackle the challenging video understanding problem, especially capturing the long-term temporal relations in lengthy videos. In particular, the proposed multimodal agent VideoAgent: 1) constructs a structured memory to store both the generic temporal event descriptions and object-centric tracking states of the video; 2) given an input task query, it employs tools including video segment localization and object memory querying along with other visual foundation models to interactively solve the task, utilizing the zero-shot tool-use ability of LLMs. VideoAgent demonstrates impressive performances on several long-horizon video understanding benchmarks, an average increase of 6.6% on NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between open-sourced models and private counterparts including Gemini 1.5 Pro.

1 Introduction

VideoAgent addresses the difficulty of understanding lengthy videos by combining a unified structured memory with interactive tool use. Its temporal and object memories support spatial-temporal reasoning, and evaluations report gains over multimodal-model and agent baselines.

  • Long-form video understanding remains challenging because computation and memory costs grow with video length, while self-attention can struggle with long-range relations.
  • VideoAgent represents videos with a unified memory containing segment-level event descriptions and temporally tracked object states.
  • The agent decomposes task queries and interactively invokes caption retrieval, segment localization, visual question answering, and object-memory querying tools.
  • VideoAgent is evaluated on localization, generic question answering, and egocentric question answering across multiple long-form video benchmarks.
  • 6.6% average improvement is reported on NExT-QA and 26.0% on EgoSchema over baselines.

2 VideoAgent

VideoAgent first converts a video into temporal and object memories, then uses an LLM to query those representations and selected video segments through a minimal interactive tool set.

  • 2.1 Overview: VideoAgent constructs temporal and object memories before solving incoming tasks through interactive tool invocation.
  • 2.2 Temporal Memory MT: Temporal memory stores captions, video features, and caption embeddings for short video segments, typically sliced into approximately 2-second units.
  • 2.2 Temporal Memory MT: LaViLa generates captions from four frames of each 2-second segment, using markers to distinguish the camera wearer from other people.
  • 2.3 Object Memory MO: Object memory tracks object and person occurrences, links visual features to object identifiers, and stores occurrence information in a SQL database.
  • 2.3 Object Memory MO: CLIP and DINOv2 feature similarities are ensembled to group tracking identifiers into consistent object identities across temporally discontinuous appearances.
  • 2.4 Tools and Inference: The tool set includes caption retrieval, segment localization, visual question answering, and object-memory querying, with segment localization combining video-text and query-caption similarities.
  • 2.4 Tools and Inference: During inference, the LLM maintains a history buffer, selects tools and arguments iteratively, appends results, and produces a response when it stops or reaches the step limit.

3 Capabilities and Analysis

VideoAgent is evaluated across long-form video understanding benchmarks and tool configurations, showing strong performance and evidence that unified memory supports reasoning and flexible tool use.

  • EgoSchema: VideoAgent significantly outperforms state-of-the-art models on EgoSchema, reaching 62.8 accuracy on the 500-question subset and 60.2 on the full test set.The 500-question result approaches a nearly 30-percentage-point advantage over comparison models, while the full-test result closes to Gemini 1.5 Pro.
  • Analysis: Unified memory provides structured video representations that support stronger spatial-temporal reasoning than implicit end-to-end reasoning or limited tool access.The analysis attributes the advantage to representing videos with a unified memory that supports reasoning and tool use.
  • WorldQA: On WorldQA, VideoAgent surpasses existing open-source multimodal LLMs on both open-ended and multiple-choice question answering.It exceeds GPT-4V on multiple-choice QA but trails GPT-4V on open-ended QA, where video frames provide visual conditions for response generation.
  • NExT-QA: VideoAgent achieves the strongest performance among comparative methods on NExT-QA, outperforming SeViLA by nearly 10 percent on causal questions.The evaluation uses a 600-question subset sampled across temporal, causal, and descriptive question types.
  • Ablation analysis: Caption retrieval establishes baseline context, while object memory substantially improves temporal and causal question performance by preserving temporally consistent object information.Caption retrieval alone reaches an average result of 40.7, compared with 43.2 for mPLUG-Owl on the 600-question subset.

4 Related Work

Research has extended LLMs to multimodal agents that plan tasks and invoke visual tools interactively, but applying these agents to long-form video remains non-trivial. Existing approaches face video generalization, computation, attention, and pipeline-complexity challenges.

  • Multimodal agents prompt LLMs to plan multimodal tasks and interactively invoke foundation-model tools such as captioning and visual question answering.
  • VisProg demonstrated visual reasoning by equipping a GPT-3 planner with a large collection of visual tools.
  • Applying multimodal agents to video requires careful design because many available tools do not guarantee generalization to videos.
  • LifeLongMemory uses natural-language video narrations as text-based episodic memory, while DoraemonGPT introduces a sophisticated prompting strategy.

5 Conclusions

VideoAgent reconciles foundation models through a unified memory mechanism and a minimalist tool-use pipeline for long-form video understanding. It avoids expensive training while achieving comparable or better empirical results across challenging benchmarks, with future applications mentioned for robotics, manufacturing, and augmented reality.

  • VideoAgent combines several foundation models with a novel unified memory mechanism for video understanding.
  • Its minimalist tool-use pipeline does not require expensive training.
  • VideoAgent achieves comparable or better empirical results on EgoSchema, Ego4D NLQ, WorldQA, and NExT-QA.
  • The paper identifies further exploration of real-world applications in robotics, manufacturing, and augmented reality as a possible future direction.
  • Object re-identification groups tracking IDs belonging to the same object into unique object IDs for the object-memory database.

B Tasks and Models

VideoAgent uses task-specific models whose extracted information has different granularity levels. The task descriptions illustrate how segment captioning captures character actions and the primary objects they interact with.

  • Segment captioning typically describes characters’ actions and the primary objects with which they interact.
  • Table 7 lists the methods and information-granularity levels used for different VideoAgent tasks.

C Experiment Settings

VideoAgent’s experimental setup uses an LLM prompted for iterative tool use over segmented videos, temporal captions, and object memory. Its tools retrieve captions, localize segments, answer questions from segments, and query object information through SQL-oriented processing.

  • The agent prompt represents videos as chronological 2-second segments with captions and an object memory recording appearing objects.
  • The interaction format repeats thought, action, action input, and observation steps before producing a final answer.
  • Caption retrieval returns captions between specified segment IDs, with at most 15 captions.
  • Segment localization returns the top five candidate segments whose captions have the highest similarity to a textual description.
  • Visual question answering examines the segment before, at, and after a specified segment ID to answer a question.
  • Object memory querying answers object-related questions from the object memory, but the tool is not totally accurate.
  • A specialized GPT-4 memory agent retrieves object IDs, translates queries into SQL, executes them, and returns a natural-language answer.

Experiment Settings of VideoAgent

Ego4D NLQ evaluation uses tuned ensemble proportions of video-text and text-text similarities for two compared model combinations.

  • LaViLa+VICLIP uses an 18:11 ensemble proportion of video-text to text-text similarities for Ego4D NLQ.The proportion was selected by grid search on the Ego4D NLQ training set.
  • Ego4D+ViCLIP uses a 7:8 ensemble proportion of video-text to text-text similarities.This proportion was also selected by maximizing overall training-set performance across four recall metrics.
  • Grid search maximizes overall performance on R1@0.3, R1@0.5, R5@0.3, and R5@0.5.

C.2 Settings of Comparative Methods

The comparative evaluation tests multiple video-language methods under their stated or reproduced evaluation settings, while VideoAgent answers through iterative LLM-directed tool use.

  • Settings of Comparative Methods: Experiments evaluate the performance of several comparative methods using detailed method-specific settings.
  • Settings of Comparative Methods: SeViLA is evaluated with 32 video frames and 4 key frames.
  • Settings of Comparative Methods: Video-LLaVA is evaluated with 8 input frames.
  • Settings of Comparative Methods: mPLUG-Owl follows the EgoSchema evaluation procedure by judging whether each proposed answer is correct given the question and video frames.
  • Settings of Comparative Methods: VideoAgent iterates between LLM reasoning, tool selection, tool execution, and textual results until producing a final answer.

D.1 Case 1

In the first case, VideoAgent combines localization, caption retrieval, and visual question answering so the LLM can reconcile imperfect visual-model outputs and select the answer.

  • Case 1: VideoAgent uses segment localization, caption retrieval, and visual question answering to solve the question about the man in red.
  • Case 1: Video-LLaVA misses the small man in red and hallucinates an answer, whereas VideoAgent synthesizes the available information correctly.
  • Case 1: The LLM searches for a segment matching the man in red looking near the plane, considering drone-related captions as potentially relevant.
  • Case 1: Caption retrieval examines segments 37–42 to provide surrounding context about a man adjusting and staring at a drone.
  • Case 1: The visual question answering result says the man walks toward the plane and opens its door, leading the LLM to choose the closest listed option, “moves away.”

D.2 Case 2

In the second case, VideoAgent uses segment localization and visual question answering, with the LLM selecting an answer despite incorrect details from the visual model.

  • Case 2: VideoAgent uses segment localization and visual question answering to answer the question about how the boy keeps the bat stable.
  • Case 2: Although Video-LLaVA provides incorrect details, the LLM combines captions and visual question answering results to select the correct answer.
  • Case 2: The LLM identifies segment 7 as relevant because it depicts a person holding a baseball bat, then queries that segment for more detail.
  • Case 2: The selected answer is option 0: the boy keeps the bat stable by holding it firmly in his hand and using body movements.

D.3 Case 3

The case studies show VideoAgent combining retrieval and visual tools to answer questions about people, objects, and ingredients across video segments. Its tools can identify relevant evidence, but visual question answering may produce incorrect or hallucinated interpretations.

  • The visual_question_answering tool identifies a wedding ring as evidence that the woman is married, selecting choice 2.
  • Object memory querying retrieves the elephant count, but the final answer conflicts with the tool result.The query returns two elephants, while the final answer is 4.
  • Caption retrieval gathers ingredient events across segments before visual question answering investigates how ingredients contribute to the dish.The agent retrieves captions from segments 0–14, then queries the visual tool about ingredient roles.
  • The agent concludes that sauce, spice, scrambled eggs, tomato paste, and macaroni and cheese enhance the dish’s flavor or texture.The cited tool outputs describe specific flavor, aroma, creamy, tender, savory, and comforting contributions.
  • Visual question answering can hallucinate unsupported content, such as claiming tomato sauce and milk were poured over ice cream absent from the captions.The agent itself flags the ice-cream interpretation as hallucinated and seeks further tool-based evidence.
Loading 2403.11481v2…