PaperlayerView exact revision on arXiv

Source-linked AI summary

LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agents

Boyu Chen, Zhengrong Yue, Siran Chen, Zikang Wang, Yang Liu, Peng Li, Yali Wang

arXiv:2503.10200v5cs.CV

TL;DR

Long-video MLLMs struggle with long-term temporal context, while tool-assisted agent methods still rely on a single model. LVAgent coordinates multiple MLLM agents through selection, retrieval, reasoning exchange, and reflection, achieving over 80% accuracy across four tasks and up to 13.3% improvement on LongVideoBench.

  • Problem

    MLLMs struggle with long-term temporal context in long videos, and tool-assisted agent methods still rely on a single MLLM with limited understanding.

  • Method

    LVAgent dynamically collaborates selected MLLM agents through Selection, Perception, Action, and Reflection for iterative long-video question answering.

  • Results

    LVAgent exceeds 80% accuracy across four mainstream long-video tasks and improves LongVideoBench accuracy by up to 13.3%.

  • Takeaways & Limitations

    LVAgent is reported to outperform closed-source and open-source models in long-video understanding tasks.

Abstract

from arXiv · show

Existing MLLMs encounter significant challenges in modeling the temporal context within long videos. Currently, mainstream Agent-based methods use external tools to assist a single MLLM in answering long video questions. Despite such tool-based support, a solitary MLLM still offers only a partial understanding of long videos, resulting in limited performance. In order to better address long video tasks, we introduce LVAgent, the first framework enabling multi-round dynamic collaboration of MLLM agents in long video understanding. Our method consists of four key steps: 1) Selection: We pre-select appropriate agents from the model library to form optimal agent teams based on different tasks. 2) Perception: We design an effective retrieval scheme for long videos to improve the coverage of critical temporal segments while maintaining computational efficiency. 3) Action: Agents answer long video questions and exchange reasons. 4) Reflection: We evaluate each agent's performance in each round of discussion and optimize the agent team for dynamic collaboration. The agents iteratively refine their answers by multi-round dynamical collaboration of MLLM agents. LVAgent is the first agent system method that outperforms all closed-source models (like GPT-4o) and open-source models (like InternVL-2.5 and Qwen2-VL) in the long video understanding tasks. Our LVAgent achieves an accuracy of 80\% on four mainstream long video understanding tasks. Notably, LVAgent improves accuracy by 13.3\% on LongVideoBench. Code is available at https://github.com/64327069/LVAgent.

1. Introduction

LVAgent addresses long-video understanding challenges by coordinating selected MLLM agents through perception, action, and reflection. Its reported results exceed 80% accuracy across four tasks and improve LongVideoBench accuracy by up to 13.3%.

  • Motivation: Long videos are important across healthcare, education, and entertainment, but MLLMs struggle to model their long-term temporal context efficiently.Feeding dense frame sequences into a single MLLM introduces computational costs and redundant information.
  • Motivation: Existing agent-based methods use retrieval or external tools, yet single-MLLM systems still provide limited long-video understanding.CLIP-based retrieval struggles with long-term temporal information and domain mismatch.
  • LVAgent: LVAgent forms task-specific agent teams and coordinates them through Selection, Perception, Action, and Reflection.Agents are pre-selected, retrieve query-related video content, exchange answers and reasons, and are evaluated for dynamic collaboration.
  • LVAgent: LVAgent uses adaptive retrieval to increase coverage of crucial temporal chunks without sacrificing computational efficiency.The perception process retrieves video clips related to the question before agents answer and justify their responses.

2. Related Work

Long-video research addresses temporal redundancy through context extension, token compression, and agent-based decomposition. LVAgent is positioned as a multi-round dynamical collaboration approach for this setting.

  • Long Video MLLMs: Long-video MLLMs face critical challenges because redundant temporal sequences create overwhelming numbers of visual tokens.Prior approaches primarily extend context length or compress video tokens.
  • Long Video MLLMs: Token-merging approaches improve efficiency but can sacrifice fine-grained details in complex scenarios.The related-work discussion uses MovieChat as an example of this trade-off.
  • Long Video Agent: Agent-based long-video methods decompose complex tasks, but CLIP retrieval can suffer from domain gaps when identifying relevant video regions.Related approaches also use external tools such as memory banks and search engines for information extraction.
  • Long Video Agent: LVAgent extends agent-based video understanding with multi-round dynamical collaboration among MLLM agents.The method is presented as inspired by multi-round collaboration pipelines in NLP.

3. Method

LVAgent organizes long-video understanding into selection, perception, action, and reflection, with agents repeatedly updating retrieval and reasoning from discussion history. Its perception pipeline targets relevant temporal chunks efficiently, while reflection scores agents and removes weaker contributors.

  • Overview: LVAgent introduces four sequential processes: Selection, Perception, Action, and Reflection, forming a multi-round collaboration pipeline.The framework uses an agent library, temporal retrieval, answer-and-reason exchange, and iterative performance-based refinement.
  • 3.1. Selection: Pseudo-label voting evaluates agents on 150 unlabeled videos, and the three highest-scoring agents form the task-specific team.The most frequent answer among agents serves as each video's pseudo label, and accuracy against these labels ranks agents.
  • 3.2. Perception: The three-stage perception pipeline lets agents decide whether to watch the entire video, summarize key information, and retrieve relevant chunks.Agents may globally sample 16 frames, or use progressively refined key information for targeted retrieval.
  • 3.4. Reflection: Reflection-derived key information initiates subsequent retrieval, enabling agents to iteratively refine their approach to each long-video question.Each collaboration round incorporates answer, reason, score, and removal history.
  • 3.2. Perception: ASP-CLIP scores six equal video chunks using sampled frames, selecting chunks above threshold 0.8 or the highest-scoring chunk when none qualifies.This converts long-video retrieval into six-category classification and uses LongVR-finetuned ASP-CLIP for temporal retrieval.
  • 3.3–3.4. Action and Reflection: Agents produce answers and reasons, trigger early stopping when a majority agrees, then score one another and remove the lowest-scoring agent.Remaining agents summarize prior answers and history to regenerate key information for the next retrieval round.

4. Experiment

Experiments evaluate LVAgent on four long-video benchmarks and compare accuracy, efficiency, and collaboration choices. Results show strong benchmark performance, reduced inference inputs and latency, and gains from perception, reflection, diverse agents, and repeated discussion.

  • 4.1. Benchmarks: LVAgent is evaluated on EgoSchema, VideoMME, MLVU, and LongVideoBench, covering videos from short clips to hour-long content.The benchmarks include curated question-answer pairs, long-video categories, and varying video durations.
  • 4.2. Main Results: Over 80% accuracy is achieved on each of the four long-video understanding datasets, exceeding the reported agent-based benchmark threshold.Table 1 compares closed-source models, open-source MLLMs, and agent-based systems; Video-MME values are reported with and without subtitles.
  • 4.3. Comparison with SOTA: 13.3% higher accuracy than GPT-4o is achieved on LongVideoBench, while MLVU gains over GPT-4o and InternVL-2.5 are 19.3% and 8.2%.These comparisons are reported in the benchmark evaluation against closed-source and open-source models.
  • 4.3. Efficiency: 71.2 frames and 33.6s of inference time are used by LVAgent on VideoMME, with fewer frames and less inference time than compared MLLMs.The efficiency analysis jointly compares inference latency and performance.
  • 4.4. Ablation Study: 82.9% on EgoSchema and 80.0% on LongVideoBench are attained when Perception and Reflection are combined, outperforming either component alone.The ablation removes Perception by replacing it with global sampling of 128 frames, while testing Reflection and Perception separately.
  • 4.4. Ablation Study: 80.0% on LVBench and 83.9% on MLVU are achieved by combining three agents, compared with individual-agent scores of 67.8%–69.0% on LVBench.The tested agents are LLaVA-Video-72B, InternVL-2.5-8B, and InternVL-2.5-78B.
  • 4.4. Ablation Study: LongVideoBench performance rises from 69.0% to 80.0% as maximum reflection rounds increase from one to three.The paper attributes this improvement to iterative refinement and consensus that progressively eliminate noisy knowledge.

5. Conclusion

LVAgent is presented as a multi-agent framework for long-video understanding that combines selection, efficient temporal retrieval, answer-and-reason exchange, and iterative reflection. Across four benchmarks, it achieves over 80% accuracy and up to 13.3% improvement on LongVideoBench.

  • Conclusion: LVAgent addresses insufficient reasoning in single-model methods through multi-round dynamic collaboration among MLLM agents.Its four processes are Selection, Perception, Action, and Reflection.
  • Conclusion: Over 80% accuracy is achieved across VideoMME, EgoSchema, MLVU, and LongVideoBench, with up to 13.3% improvement on LongVideoBench.The conclusion reports superiority over closed-source and open-source models, including GPT-4o, InternVL-2.5, and Qwen2-VL.

A. Details of LongVR Dataset.

LongVR is assembled as a retrieval dataset from multiple video sources, with dataset-specific instance counts, caption lengths, and video durations reported for training ASP-CLIP. The appendix provides its distribution and statistics.

  • LongVR Dataset: LongVR combines video data from ActivityNet-Caption, OpenVid-1M, ViTT, MovieChat-Caption, and Youcook2 for long-video retrieval training.The dataset statistics are summarized in Table 8, which reports instance counts and average caption or video properties.
  • LongVR Dataset: ActivityNet-Caption contributes 37,421 instances, OpenVid-1M contributes 30,000, and ViTT contributes 5,086.Reported averages include caption lengths of 15.80, 143.16, and 24.98 words, respectively.

B.1. Model Selection for Different Tasks

LVAgent selects different groups of MLLM agents for each dataset, using task-specific pseudo labels and dataset features to tailor collaboration.

  • VideoMME uses InternVL2.5 and Qwen2VL agents, while EgoSchema uses Qwen2VL and LLaVAVideo agents.
  • MLVU and LongVideoBench use LLaVAVideo and InternVL2.5 agent groups.
  • The selection mechanism chooses agent groups according to pseudo labels and dataset-specific features.

B.2. Efficiency Analysis

LVAgent’s evaluation requires less time per video than the cited Qwen2VL baseline, despite additional preselection and fine-tuning stages.

  • Qwen2VL (72B) requires 350.5 hours in total, averaging 90.5 seconds per video.
  • 139.71 hours total evaluation time, averaging 36.07 seconds per video, is 2.5× faster than Qwen2VL.This total includes preselection and evaluation.
  • 130.13 hours are spent in evaluation, averaging 33.6 seconds per video.
  • 6.58 hours of preselection on 750 videos averages 31.6 seconds per video.

B.3. The Result of Using More Than Three Agents

Using four agents yields slightly higher benchmark performance than three agents but increases per-video processing time, so LVAgent adopts the three-agent configuration.

  • The 4-agent setup is slightly superior to the 3-agent configuration across the reported benchmarks.Its scores are EgoSchema: 83.0, LongVideoBench: 80.1, MLVU: 84.1, and VideoMME: 81.8 / 86.5.
  • 38.7 seconds per video makes the 4-agent setup 5.1 seconds slower than the 3-agent setup.
  • LVAgent opts for three agents to balance accuracy and efficiency, while allowing more agents to be added if needed.

C. Prompts of LVAgent.

LVAgent uses stage-specific prompts to coordinate visual-temporal reasoning, whole-video viewing decisions, and key-information extraction.

  • The prompts are designed to guide task performance, interactions, and accurate responses across LVAgent stages.
  • One prompt asks agents to combine textual and visual information before deciding whether watching the entire video would improve the answer.
  • Another prompt asks agents to identify key information from four sampled frames, subtitles, questions, and answer options.
  • Table 9 presents prompting templates for LVAgent’s different key steps.
Loading 2503.10200v5…