PaperlayerView exact revision on arXiv

Source-linked AI summary

LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding

Jihao Qiu, Lingxi Xie, Xinyue Huo, Qi Tian, Qixiang Ye

arXiv:2602.20913v2cs.CV

TL;DR

Long-video understanding remains difficult under limited context windows and computational budgets. LongVideo-R1 addresses this with hierarchical, reasoning-guided clip navigation supported by captioning and QA tools, trained through supervised fine-tuning and reinforcement learning. It achieves competitive QA accuracy with a favorable accuracy-efficiency tradeoff across long-video benchmarks.

  • Problem

    Finite MLLM context and exhaustive processing make long-form video understanding computationally expensive, motivating accuracy-efficiency optimization.

  • Method

    LongVideo-R1 uses hierarchical video navigation with reasoning and captioning/QA tools, trained on curated trajectories through SFT followed by RL.

  • Results

    LongVideo-R1 achieves competitive QA accuracy with an average of 10.5 reasoning and navigation/answering rounds, yielding lower computational cost than linear-scan methods.

  • Takeaways & Limitations

    The framework shows a favorable accuracy-efficiency tradeoff and performs information retrieval and grounding tasks across several long-video benchmarks.

  • Takeaways & Limitations

    The framework uses only video captioning and video QA tools, and its reasoning performance heavily relies on video-caption quality.

Abstract

from arXiv · show

This paper addresses the critical and underexplored challenge of long video understanding with low computational budgets. We propose LongVideo-R1, an active, reasoning-equipped multimodal large language model (MLLM) agent designed for efficient video context navigation, avoiding the redundancy of exhaustive search. At the core of LongVideo-R1 lies a reasoning module that leverages high-level visual cues to infer the most informative video clip for subsequent processing. During inference, the agent initiates traversal from top-level visual summaries and iteratively refines its focus, immediately halting the exploration process upon acquiring sufficient knowledge to answer the query. To facilitate training, we first extract hierarchical video captions from CGBench, a video corpus with grounding annotations, and guide GPT-5 to generate 33K high-quality chain-of-thought-with-tool trajectories. The LongVideo-R1 agent is fine-tuned upon the Qwen-3-8B model through a two-stage paradigm: supervised fine-tuning (SFT) followed by reinforcement learning (RL), where RL employs a specifically designed reward function to maximize selective and efficient clip navigation. Experiments on multiple long video benchmarks validate the effectiveness of name, which enjoys superior tradeoff between QA accuracy and efficiency. All curated data and source code are provided in the supplementary material and will be made publicly available. Code and data are available at: https://github.com/qiujihao19/LongVideo-R1

1. Introduction

LongVideo-R1 targets long-video understanding under tight computational budgets by replacing exhaustive clip processing with active, reasoning-guided navigation. It combines hierarchical exploration, tool use, and two-stage training to preserve competitive QA accuracy while reducing computation.

  • Motivation: Long-form video QA is constrained by finite MLLM context, while exhaustive clip processing incurs computational cost and latency that hinder deployment.The limitation is especially relevant to low-latency embodied agents and high-throughput video-chat systems.
  • Method: LongVideo-R1 navigates a hierarchical video structure by captioning local context, reasoning about answer sufficiency, and selecting child, sibling, or upper-level clips.The process stops when the answer is derived or the maximum iteration limit is reached.
  • Training: The model is trained with 33K reasoning episodes derived from CGBench grounding annotations and synthesized GPT-5 reasoning trajectories.Training uses supervised fine-tuning followed by reinforcement learning with a reward mechanism targeting efficient navigation and accurate grounding.
  • Results: Across LVBench, VideoMME, and MLVU, LongVideo-R1 maintains competitive QA accuracy with an average of 10.5 reasoning and navigation/answering rounds.The reported result is accompanied by lower computational cost than linear-scan methods and includes ultra-long TV-drama understanding under strict budgets.

2. Related Work

Long-video understanding has emerged as a major challenge for multimodal models, while computational efficiency remains comparatively underexplored. LongVideo-R1 frames this gap as pursuing an accuracy-efficiency Pareto optimum through an agent-based solution.

  • Long-form video benchmarks with hour-long videos and complex QA tasks challenge multimodal large language models.
  • Research on long-video understanding has addressed context-length limitations through two main lines of work.
  • Computational-budget reduction has received relatively little attention despite rapid progress in long-video QA accuracy.
  • Recent agentic architectures may require exhaustive processing of all or a substantial proportion of a video.
  • LongVideo-R1 defines its objective as pursuing the accuracy-efficiency Pareto optimum with a competitive agent-based solution.

3. On Efficient Long Video Understanding

LongVideo-R1 performs active hierarchical video exploration with reasoning and termination control, progressively narrowing from broad clips to finer regions. Its inference uses captioning and QA tools within a natural-language chain-of-thought-with-tool procedure.

  • LongVideo-R1 uses contextual exploration and reasoning-based termination control to navigate video context and stop when sufficient information is collected.The framework is described as a self-regulating closed-loop mechanism.
  • The agent represents a long video as a hierarchical tree whose non-leaf nodes split into K equal-length, non-overlapping sub-clips.The root is the entire video, and the tree has D levels.
  • The default hierarchy uses D = 3 and selects K so leaf clips are approximately 16 seconds long.
  • The hierarchy lets the agent inspect long clips first and zoom into finer-scale visual content when necessary.The paper notes that uniform partitioning is easy to implement but not optimal for semantic localization.
  • LongVideo-R1’s CoTwT framework combines video captioning and video QA tools for clip description, question answering, and iterative reasoning.The video QA tool can return an answer or “I don’t know.”
  • Inference begins with a top-level caption, updates reasoning after each tool call, and continues until the reasoning output contains the answer.The procedure is expressed through a natural-language episode involving external multimodal functions.

4. Data Curation

The authors curate CoTwT training data from clue-grounded CG-Bench QA pairs by generating hierarchical captions and GPT-5 trajectories. The resulting collection contains 5.6K trajectories averaging 5.8 steps, yielding approximately 33K samples for supervised fine-tuning.

  • CG-Bench is selected because its clue-grounded QA pairs support supervision for localizing key sub-clips before answering.
  • The curation uses 800 CG-Bench videos and 5.6K corresponding QA pairs from a corpus containing 1.2K long-form videos.
  • Hierarchical captions are extracted with Qwen2.5-VL-72B using level-specific frame counts and description lengths, with absolute timestamps inserted to guide localization.
  • GPT-5 generates CoTwT trajectories by starting from top-level clips and continuing until it is confident enough to produce a final answer.Tool-use rules constrain captioning to traversed nodes and QA to the lowest-level nodes.
  • GPT-5 produces incorrect answers or fails verification in about 30% of zero-shot scenarios, motivating fixes to improve data quality and stability.One fix traverses all first-level sub-clips instead of beginning at the root.
  • 5.6K CoTwT trajectories average 5.8 steps and yield approximately 33K high-quality supervised fine-tuning samples.
  • The authors report that SFT data helps train a powerful agent and that performance is positively related to the amount of SFT data.They identify expanding clue-grounded QA pairs and generating more trajectories as a path for enhancement.

5. Training LongVideo-R1 Agent

LongVideo-R1 is trained in two stages: supervised fine-tuning establishes structured tool-using reasoning, while reinforcement learning optimizes interactive video exploration for accuracy and efficiency. Its composite reward balances correct answers, relevant temporal coverage, precision, and avoidance of repeated exploration.

  • Supervised fine-tuning: SFT first trains LongVideo-R1 on curated data to generate structured reasoning trajectories and coherent tool interactions.This cold-start model provides the foundation for subsequent reinforcement learning.
  • Reinforcement learning: The reinforcement-learning formulation treats the model as an agent, video tools as actions, and hierarchical video as the environment state.GRPO is used to optimize the policy for reasoning efficiency and accuracy.
  • Reward design: The composite reward combines answer correctness, location quality, and a repeat penalty.The answer reward is binary, while location quality encourages efficient identification of the correct segment and the repeat penalty reduces wasted computation.
  • Reward design: Location quality uses an F1-like coverage-and-precision criterion to reward relevant content while penalizing unnecessary exploration.The predicted temporal set is the union of non-overlapping segments corresponding to requested nodes.
  • Reinforcement learning: RL rollouts continue until a final answer or a maximum reasoning-step limit, after which GRPO updates the policy.The resulting model minimizes redundant exploration while maintaining high answer accuracy and achieves a superior performance–computational-efficiency tradeoff against conventional MLLMs.

6. Experiments

LongVideo-R1 achieves competitive long-video QA accuracy while reducing navigation and inference cost through selective exploration. Experiments show strong temporal localization, efficient ultra-long-video handling, and clear sensitivity to tool quality, training data, and reward design.

  • Benchmark results: 50.0% accuracy on LVBench, outperforming other agent-based methods by at least 5.6%.It also exceeds GPT-4o by 1.1% and GLM-4V-plus by 1.3%.
  • Benchmark results: 56.4% TG accuracy on LVBench surpasses all other models by 10.9%, indicating strong temporal-segment localization.The KIR and TG sub-categories are highlighted as particular strengths.
  • Benchmark results: On MLVU and Video-MME, LongVideo-R1 performs well but does not excel among open-source MLLMs because these benchmarks favor short-video or global-question methods.The paper attributes this pattern to benchmark composition rather than to the long-video navigation setting alone.
  • Ablations and analysis: Stronger multimodal tools improve overall accuracy, while KIR and TG advantages persist with Qwen3-VL-32B-Instruct captioning.The experiments also examine accuracy versus SFT data size, location reward, tool-model scale, and maximum tool-use rounds.
  • Limitations: LongVideo-R1 can be distracted by semantically related segments and may remain there instead of shifting to the correct segment.The paper identifies this failure mode in Appendix D; textual hints can redirect attention to the correct segment.
  • Ultra-long videos: 10–20 rounds suffice to navigate tens-of-hours-long TV dramas, whereas competing agent systems require samples that grow linearly with video duration.The paper presents this as evidence of efficient ultra-long-video QA under strict budgets.

7. Conclusion

LongVideo-R1 targets efficient long-video understanding through hierarchical navigation, reasoning-based termination, and adaptive visual processing. It achieves competitive QA accuracy with a favorable accuracy-efficiency tradeoff while supporting multi-hour video retrieval.

  • 7. Conclusion: LongVideo-R1 explores long videos from top-level sections, reasons about answer sufficiency, and navigates to finer content only when needed.Its agentic process combines video descriptions, reasoning, search actions, and answer generation.
  • 7. Conclusion: LongVideo-R1 achieves competitive QA accuracy and a favorable accuracy-efficiency tradeoff across long-video benchmarks.The paper also reports information-retrieval and grounding strengths and qualitative success on multi-hour narratives.
  • 7. Conclusion: The hierarchical representation uses three levels, adaptive width, and 16-second leaf segments for efficient temporal localization.Width is chosen according to video length, while leaf-level segments remain fixed at 16 seconds.
  • 7. Conclusion: Visual detail increases down the hierarchy while keeping the approximate visual-token count and caption-call computation cost consistent.Frame sampling and spatial resolution are adjusted across levels to provide finer signals during downward traversal.
  • 7. Conclusion: The captioning and video QA components can be replaced by other models, supporting flexible deployment configurations.The reported implementation uses Qwen-2.5-VL-72B for captioning and Qwen-2.5-VL-32B for video QA.

B.2. Data Generation

CoTWT supervision is generated from CGBench using timestamp grounding annotations and GPT-5, with separate data allocations for supervised fine-tuning and reinforcement learning. Providing multiple high-level captions as initial context improves generation stability and accuracy.

  • B.2. Data Generation: CGBench supplies timestamp annotations for constructing CoTWT supervision, with approximately 1,200 videos and 12,000 question-answer pairs.The dataset is used as the primary source because its questions include timestamp annotations.
  • B.2. Data Generation: 800 videos and around 8,000 question-answer pairs are used for SFT data, yielding 5,600 high-quality CoTWT trajectories after filtering.GPT-5 generates the supervision signals before filtering produces the final trajectories.
  • B.2. Data Generation: The remaining 400 videos and approximately 4,200 question-answer pairs are reserved for RL training.This separates the source data used for supervised trajectory imitation from the reinforcement-learning split.
  • B.2. Data Generation: Initializing GPT-5 with only the root caption produced unstable behavior and around 30% accuracy, whereas adding the W highest-level captions substantially improved stability and accuracy.The generation process therefore uses multiple high-level captions as initial information.
  • B.2. Data Generation: The tree width W is set between 4 and 8 for CGBench, LVBench, and VideoMME-Long, and between 3 and 8 for EgoSchema.EgoSchema uses a different range because it contains many short two-minute videos.

B.3. Training hyper-parameters

Training uses Qwen3-8B as the reasoning model in a two-phase process: supervised imitation of tool trajectories followed by reinforcement learning. Captions are pre-extracted during RL to accelerate training, while video QA remains real time.

  • B.3. Training hyper-parameters: Qwen3-8B receives the W highest-level captions initially and interacts with the video tree through sequential tool calls.The model serves as the central reasoning model for both SFT and RL.
  • B.3. Training hyper-parameters: Training consists of supervised fine-tuning followed by reinforcement learning.The two phases are explicitly separated in the training procedure.
  • B.3. Training hyper-parameters: SFT teaches the model to imitate reasoning processes, search actions, and answers from CoTWT trajectories.This stage targets hierarchical search behavior and structured video reasoning skills.
  • B.3. Training hyper-parameters: During RL, hierarchical captions are pre-extracted for faster training, while the video QA tool is invoked in real time.Qwen-2.5-VL-32B serves as the video QA module during this phase.
  • B.3. Training hyper-parameters: The detailed training hyper-parameters are provided in Table 8.The supplied section identifies the hyper-parameter table but does not enumerate its values.

B.4. Time consumption calculation

Inference time is modeled as the sum of reasoning, captioning, and video-QA call costs. On VideoMME-Long, the reported averages produce an end-to-end estimate of approximately 135 seconds, including hierarchical search and occasional fine-grained QA.

  • B.4. Time consumption calculation: Inference cost is decomposed into reasoning-model, captioning, and video-QA runtimes weighted by their respective average call counts.The total is expressed as T = C1T1 + C2T2 + C3T3.
  • B.4. Time consumption calculation: 10.5 reasoning rounds per question and 0.36 video-QA calls per question are reported for VideoMME-Long.These values define C1 and C3 in the time calculation.
  • B.4. Time consumption calculation: 14.14 caption calls per question are estimated from tree width W = 5, reasoning rounds, and video-QA calls.The calculation is C2 = W + C1 − 1 − C3 = 14.14.
  • B.4. Time consumption calculation: T1 ≈ 2.5s, T2 ≈ 7.0s, and T3 ≈ 2.7s are the empirical average runtimes used in the estimate.The calculation assumes Qwen-2.5-VL-32B is used for both captioning and video QA.
  • B.4. Time consumption calculation: 135s is the estimated end-to-end time for one VideoMME-Long question under the reported call counts and runtimes.The estimate includes caption retrieval and occasional fine-grained Video QA queries.
  • B.4. Time consumption calculation: Qualitative examples show hierarchical search, cross-hour scene disambiguation, and joint use of high-level and fine-grained information.Examples from Downton Abbey include repeatedly locating characters, objects, or events across visually similar episodes.

D. Failure Examples

LongVideo-R1 can be misled by visually similar but irrelevant objects, causing it to follow the wrong branch and miss the correct segment. Simple textual hints can often redirect the model and enable the correct answer.

  • Visually similar but irrelevant objects can cause LongVideo-R1 to commit to the wrong branch and fail to return to the correct segment.
  • Simple textual hints can often guide LongVideo-R1 back to the correct segment and enable it to produce the correct answer.
  • The failure examples include questions about identifying a dog and answering a detailed dialogue question in a TV series exceeding 10 hours.
Loading 2602.20913v2…