Source-linked AI summary
LongVideoAgent: Multi-Agent Reasoning with Long Videos
Runtao Liu, Ziyi Liu, Jiaqi Tang, Yue Ma, Renjie Pi, Jipeng Zhang, Qifeng Chen
TL;DR
Long-video QA is challenged by lossy compression and limited toolsets that can constrain temporal grounding and fine-grained cues. LONGVIDEOAGENT uses a master agent to coordinate grounding and vision agents, with reinforcement learning for open-source masters. It outperforms non-agent baselines on LongTVQA, and RL adds gains for several open-source models.
Problem
Long-video systems often compress content into static inputs or use limited tools, while the field lacks a solution jointly achieving efficiency, multimodal completeness, and fine-grained temporal reasoning.
Source: §1 · pp. 1–2Method
Results
LONGVIDEOAGENT significantly outperforms existing LongTVQA baselines, and reinforcement learning further improves several open-source models.
Source: §1 · p. 2, §4.4 · p. 7Takeaways & Limitations
The reported results support an agentic approach to long-video understanding, with interpretable coordination that selects relevant segments and extracts fine-grained visual information.
Source: §1 · p. 2The system relies on provided subtitles as its primary textual channel and does not process raw audio; its vision and grounding modules remain fixed during reinforcement learning.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Recent advances in multimodal LLMs and systems that use tools for long-video QA point to the promise of reasoning over hour-long episodes. However, many methods still compress content into lossy summaries or rely on limited toolsets, weakening temporal grounding and missing fine-grained cues. We propose a multi-agent framework in which a master LLM coordinates a grounding agent to localize question-relevant segments and a vision agent to extract targeted textual observations. The master agent plans with a step limit, and is trained with reinforcement learning to encourage concise, correct, and efficient multi-agent cooperation. This design helps the master agent focus on relevant clips via grounding, complements subtitles with visual detail, and yields interpretable trajectories. On our proposed LongTVQA and LongTVQA+ which are episode-level datasets aggregated from TVQA/TVQA+, our multi-agent system significantly outperforms strong non-agent baselines. Experiments also show reinforcement learning further strengthens reasoning and planning for the trained agent. Code and data will be shared at https://longvideoagent.github.io/.
1 Introduction
Long-video QA remains difficult because evidence is sparse across hours of content, while compressed inputs and limited tools constrain fine-grained temporal reasoning. LONGVIDEOAGENT coordinates specialized agents and reward-guided planning; it is reported to outperform existing LongTVQA baselines.
1 Introduction
- Long-video systems still struggle to jointly achieve efficiency, multimodal completeness, and fine-grained temporal reasoning because many rely on static compressed inputs and limited toolsets.Prior agentic systems are described as using generic captioning and image-retrieval tools that may miss fine-grained semantics, precise references, and subtle temporal cues.Source: §1 · pp. 1–2
- The framework pairs a master agent that reasons and answers with grounding and vision agents that locate relevant segments and extract detailed visual information.The master iteratively gathers agent outputs and uses reward-driven training to encourage structured, multi-step reasoning and penalize irrelevant tool use and incoherent reasoning.Source: §1 · p. 2
- LongTVQA and LongTVQA+ extend TVQA and TVQA+ to episode-level video question answering.The paper presents them as a testbed for its agent.Source: §1 · p. 2
- LONGVIDEOAGENT outperforms existing LongTVQA baselines by a significant margin, with ablations attributing gains to both its multi-agent architecture and reward-guided training.The system also produces interpretable decisions by coordinating agents to select relevant segments and extract fine-grained visual information.Source: §1 · p. 2
2 Related Work
Related work moves from video QA models and efficient frame sampling toward retrieval, multimodal reasoning, and agentic evidence gathering. Long-form video QA remains constrained by context and retrieval challenges.
2.1 Video Question Answering
- Video QA evolved from appearance–motion memory and attention toward multimodal transformers for efficient frame sampling, retrieval-aware reasoning, and temporal grounding.These developments motivate selectively retrieving segments under limited context budgets.Source: §2.1 · p. 2
2.2 LLM Agents
- LLM agents combine reasoning with planning, tool calls, and iterative evidence gathering; long-video agents use memory, targeted retrieval, and temporal grounding under strict context budgets.Related approaches also include video-RAG extraction and persistent-memory or structural-planning agents.Source: §2.2 · pp. 2–3
2.3 Multi-Modal LLMs
- Multimodal LLMs combine vision encoders and instruction-tuned language models, while video models explore efficient tokenization, temporal coverage, and long-context processing.Despite these advances, most models still face long-video context-length and retrieval constraints.Source: §2.3 · p. 3
3 Method
LONGVIDEOAGENT treats long-video QA as iterative multi-agent reasoning: a master coordinates temporal grounding and targeted visual reads, then answers when it judges the evidence sufficient. For open-source masters, GRPO trains structured actions and answer correctness while specialist agents remain fixed.
3 Method
- The master coordinates grounding and vision agents, accumulates subtitles, segment tags, and visual observations, and answers when it judges the evidence sufficient.The resulting traces are step-by-step; open-source masters receive reinforcement learning while the other agents remain frozen.Source: §3 · p. 3
3.1 Multi-agent System Framework
- The bounded master-agent loop emits one action per turn: request grounding, query a visual read, or answer; agent outputs are appended to its context.The grounding agent returns a symbolic clip tag and may be re-queried, while the vision agent extracts prompted observations such as entities, attributes, actions, OCR, or scene cues.Source: §3.1 · p. 4
3.2 Reinforcement Learning for LONGVIDEOAGENT
- GRPO trains open-source master agents using structural-validity rewards for actions and terminal rewards for multiple-choice answer correctness.The objective combines per-step structural shaping with the final answer reward; sampled rollouts supply sequence-level advantages, while grounding and vision agents remain fixed.Source: §3.2 · p. 5
4 Experiments
Experiments evaluate episode-aggregated LongTVQA and LongTVQA+ under matched agentic and non-agentic settings. The reported validation results favor multi-agent systems over non-agent counterparts, with further gains from reinforcement learning for several open-source models.
4.1 Datasets
- LongTVQA and LongTVQA+ aggregate clips from each TV episode into hour-scale sequences, merging video, subtitles, and questions while preserving TVQA+ frame boxes.Results use the original validation splits after episode-level aggregation unless otherwise noted.Source: §4.1 · pp. 5–6
4.2 Baselines
- The baseline comparison uses open- and closed-source models under unified settings, matching agent and non-agent runs on the same backbone; closed-source models are evaluated without additional training.Non-agent runs consume full subtitles without grounding or vision agents.Source: §4.2 · p. 6
4.3 Experimental Setup
- The default setup uses Grok-4-fast-reasoning for localization, GPT-4o for vision, a one-clip context window, and a five-step limit; the master receives text rather than raw images.Validation performance is measured primarily by answer accuracy, with grounding accuracy also reported for clip-grounding experiments.Source: §4.3 · p. 6
4.4 Performance
- LONGVIDEOAGENT yields significant validation-accuracy gains over non-agent settings, and reinforcement learning further improves several open-source models, with Qwen2.5-7B reaching accuracy comparable to GPT-5-mini.Table 2 also reports that using frames generally outperforms subtitle-only inputs and that agentic designs narrow the gap to closed-source models.Source: Table 2 · p. 6, §4.4 · p. 7
- Adding grounding improves over the non-agent baseline, and adding a vision agent yields the best results; increasing the master’s step budget generally improves grounding and overall accuracy until saturation.These are reported as ablation trends, without specific values in the supplied passages.Source: §4.4 · p. 7
K Grounding Accuracy (%) Accuracy (%)
The ablations examine how agentic components, context-window size, step limits, and vision-backbone strength affect long-video QA. They show gains from grounding and targeted visual inspection, alongside trade-offs in context and action budgets.
4.5 Ablation Studies and Analysis
- Increasing the action limit from K=2 to K=5 raises temporal localization accuracy from 67.0 to 71.0 (+4.0) and answer accuracy from 68.30 to 73.67 (+5.37).At K=10, answer accuracy remains 73.67 while localization reaches 72.0, so K=5 is the default.Source: §4.5 · p. 7
- With GPT-4o, expanding the evidence window from 1 to 2 clips raises localization from 71.67 to 78.67 (+7.00) and answer accuracy from 70.33 to 75.00 (+4.67).Three clips raise localization to 81.94 and answer accuracy to 77.26, but larger windows require more visual queries and latency; the default remains Window=1.Source: §4.5 · p. 7, K Grounding Accuracy (%) Accuracy (%) · p. 7
- The example trace repeatedly invokes grounding and vision agents to gather evidence before producing a correct answer.The master first localizes the referenced segment, then requests a more precise visual read when the first description lacks the needed relation.Source: Table 5 · p. 8, §4.5 · p. 8
- GPT-4o reaches 73.30 localization and 78.00 answer accuracy, exceeding Qwen3-VL-235B-a22b at 71.00 and 73.67 by +2.30 and +4.33, respectively.The authors connect stronger recognition of small objects, OCR, and fine attributes with better end-task accuracy.Source: §4.5 · p. 8
- Answer accuracy rises from 64.3 to 74.8 (+10.5) when temporal grounding and vision are added to the same backbone.Grounding raises accuracy to 69.0 (+4.7), and adding vision after grounding adds 5.8 points.Source: §4.5 · p. 8
5 Conclusion
The conclusion presents LONGVIDEOAGENT as a modular framework for long-form video QA, combining temporal grounding and targeted perception with reinforcement learning for open-source master agents. It reports gains over non-agent baselines and identifies richer modalities, finer grounding, and larger-scale RL as future directions.
5 Conclusion
- LONGVIDEOAGENT coordinates a master agent with grounding for temporal localization and vision for targeted perception.The framework is model-agnostic; open-source master agents are fine-tuned with GRPO while the other agents remain frozen.Source: §5 · p. 8
- The system reports strong gains on LongTVQA and LongTVQA+ over non-agent baselines, with transparent step-by-step traces.Ablations identify grounding plus vision as essential, modest step limits as sufficient, adjacent-window context as helpful, and stronger perception as beneficial to accuracy.Source: §5 · p. 8
- Future work includes adding audio and knowledge background, finer grounding, and larger-scale reinforcement-learning training.Source: §5 · p. 8
Limitations
The paper identifies three practical limitations: reliance on provided subtitles without raw-audio processing, fixed vision and grounding modules during RL, and a simple reward design.
Limitations
- The datasets rely on provided subtitles as the primary textual channel, and the system does not process raw audio.The authors plan to add an audio-to-subtitles module for raw speech.Source: Limitations · p. 9
- Vision and grounding modules remain fixed during RL, although jointly optimizing them could improve robustness and accuracy.Source: Limitations · p. 9
- The reward uses format and answer correctness, a deliberately simple design that may be improved.Source: Limitations · p. 9