LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding

1. Introduction

• Finite context limits prevent MLLMs from comprehensively ingesting long videos, while exhaustive clip processing is costly; Ego-R1 [45] and Videotree [53] scale linearly with video length.These costs and latency constrain low-latency embodied agents and high-throughput video-chat services.Source: §1 · pp. 1–2
• The proposed setting evaluates long-video understanding by accuracy-efficiency tradeoff, seeking Pareto-optimal accuracy with minimal estimated operation cost through reasoning that selects the next clip from partial context.The motivation is to replace exhaustive search with dynamic, goal-oriented sampling of the event relevant to the question.Source: §1 · p. 2
• LongVideo-R1 organizes video hierarchically and iteratively uses captioning and thinking tools to decide whether to answer, sample a child or sibling clip, or backtrack, stopping at an iteration limit.When the model determines that the answer can be derived, a video QA tool generates it.Source: §1 · p. 2
• Training uses 33K reasoning episodes grounded in CGBench [3], GPT-5-generated trajectories, and Qwen3-8B [55] fine-tuned with SFT followed by RL using a reward for efficient navigation and accurate grounding.Training uses pre-extracted captions and is reported to remain stable over a few epochs.Source: §1 · p. 2
• On LVBench, VideoMME, and MLVU, LongVideo-R1 achieves competitive QA accuracy with an average of 10.5 reasoning, navigation, and answering rounds, at lower computational cost than linear-scan methods.The paper also demonstrates ultra-long video understanding on complex TV dramas under strict budget constraints.Source: §1 · p. 2

Related Work

• MLLMs [7] [17] [48] extend LLM reasoning [1] [25] [43] [47] to visual inputs, enabling open-world video question answering beyond bounded recognition tasks.They encode visual inputs as discrete tokens integrated into textual context [33] [46].Source: §2 · p. 2
• Long-video research addresses context limits through information-dense video representations [18] [21] [37] or segmenting, separately processing, and integrating video components [40] [42] [61].The latter direction has developed into reasoning-based agent systems [9] [51] [53], which use tools strategically and dominate many leading benchmarks [11] [41].Source: §2 · p. 2
• Video-SALMONN 2 [41] and Ego-R1 [45] process all or many video segments, incurring substantial MLLM-call overhead; this paper targets the accuracy-efficiency Pareto optimum with an agent-based solution.The contribution is described as a competitive agent-based solution for pursuing that optimum.Source: §2 · pp. 2–3
• Policy-optimization research extends PPO [35] to GRPO [36], which removes the need for an explicit critic; subsequent LLM and MLLM methods often tailor reward functions to desired agent behavior.The cited later work includes LLM methods [57] [64] [65] and MLLM methods [11] [16] [23] [59].Source: §2 · p. 3

3.1. What Makes Efficient Video Understanding?

• LongVideo-R1 uses contextual exploration and reasoning-based termination to gather enough context for QA, reducing computation versus exhaustive search while preserving competitive accuracy.The paper defines task cost as the aggregate computational overhead across the operational pipeline and targets a Pareto-optimal accuracy-efficiency tradeoff.Source: §3.1 · p. 3

3.2. LongVideo-R1 Framework

• The framework organizes each video as a multi-level tree, letting the agent inspect broad clips first and zoom into finer clips when needed.By default, the tree has three levels of depth and leaf clips of approximately 16 seconds; uniform partitioning can leave semantically similar content in neighboring clips, complicating localization.Source: §3.2 · p. 3
• LongVideo-R1 uses generic clip captions to localize key content and reserves question-specific video QA for lowest-level clips.Both tools sample frames uniformly, with vanilla visual encoding and no compression.Source: §3.2 · p. 3

3.3. Chain-of-Thought-with-Tool Procedure

• Inference begins with a top-level caption, then alternates reasoning with selected captioning or QA tool calls until the reasoning yields an answer.The episode is expressed in natural language, with multimodal tools called as external functions, connecting reasoning and tool use in a transparent procedure.Source: §3.3 · p. 4

4.1. Data Preparation

• CG-Bench [3] was selected for its clue-grounded QA pairs, which supervise localization of key sub-clips before answering.The authors selected 800 of its 1.2K videos and 5.6K corresponding QA pairs for trajectory generation.Source: §4.1 · p. 4
• Video captions were generated at four hierarchy levels with Qwen2.5-VL-72B [2], incorporating absolute timestamps to support sub-clip localization.The frame counts for levels 0–3 were 256, 128, 64, and 32, with suggested description lengths of 400, 400, 400, and 200 English words, respectively.Source: §4.1 · p. 4

4.2. Generating CoTwT Trajectories

• GPT-5 generated CoTwT trajectories by starting from the top-level clip and continuing until confident in its answer, using video captioning and QA tools.Captioning was allowed at a node only after traversing its parent, while video QA was restricted to lowest-level nodes.Source: §4.2 · p. 4
• To improve zero-shot trajectory quality, the process first traversed all first-level sub-clips, then added increasingly precise clue-grounded hints when GPT-5 failed verification.Initial zero-shot runs produced incorrect answers or failed verification in about 30% of scenarios; the incremental hints were intended to ensure correctness while minimizing leakage and encourage efficient exploration rather than memorization.Source: §4.2 · p. 4
• 5.6K trajectories averaging 5.8 steps yielded approximately 33K high-quality samples for supervised fine-tuning.The authors report that released SFT data helps train a powerful agent and that agent performance is positively related to SFT data quantity, motivating additional clue-grounded QA pairs and trajectories.Source: §4.2 · pp. 4–5

5.1. Supervised Fine-tuning

• SFT on curated CoTwT data cold-starts the model to generate structured reasoning trajectories, and LongVideo-R1-SFT learns when to continue, invoke tools, or answer.Training samples encode reasoning, tool calls, and answers in separate special-token spans; tool calls are executed and their observations returned as context.Source: §5.1 · p. 5

5.2. Reinforcement Learning with GRPO

• The RL formulation treats the model as an agent, video tools as its action space, and hierarchical video as the environment state; GRPO [36] using the Ego-R1 [45] version optimizes reasoning efficiency and accuracy.The SFT policy is used for KL regularization during optimization.Source: §5.2 · p. 5

5.3. Reward Design

• The composite reward combines ground-truth answer correctness, efficient localization of the relevant segment, and a penalty for repeatedly visiting segments.The location reward uses an F1-like coverage-and-precision metric over requested video intervals, encouraging relevant coverage while penalizing unnecessary exploration.Source: §5.3 · p. 5

5.4. Rollout and Optimization

• After RL, LongVideo-R1 performs multi-tool reasoning efficiently, minimizing redundant exploration while maintaining high answer accuracy and a superior performance–computational-efficiency tradeoff versus conventional MLLMs.Rollouts continue until a final answer or a predefined maximum number of reasoning steps; collected trajectories are used to compute policy gradients and update the policy with GRPO.Source: §5.4 · p. 6

6.1. Implementation Details

• LongVideo-R1 is trained on Qwen3-8B with Qwen2.5VL-72B captioning and Qwen2.5VL-32B video QA tools, using 3 SFT epochs followed by 2 RL epochs.Evaluation uses multiple-choice QA on LVBench, Video-MME-long, and MLVU.Source: §6.1 · p. 6

6.2. Results and Analysis

• On Video-MME, LongVideo-R1 has similar accuracy to Ego-R1 while using an average of 10.5 rounds rather than 86 caption segments; it does not excel among open-source MLLMs on MLVU or Video-MME.The paper attributes those benchmark patterns to many short videos in MLVU and global questions in Video-MME, which favor uniform or adaptive frame sampling; stronger captioning tools also improve performance.Source: §6.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7
• 50.0% LVBench accuracy takes 3 minutes per QA, while reducing time to 2 minutes costs only a 0.2% accuracy drop.Results varying captioning-model scale and maximum tool-use rounds suggest adapting tool size and number to question difficulty.Source: §6.2 · p. 6, Table 6 · p. 8, Table 7 · p. 8
• 50.0% LVBench accuracy exceeds other agent-based methods by at least 5.6%, while 56.4% temporal-grounding accuracy surpasses all other models by 10.9%.With an 8B language model, it also exceeds GPT-4o by 1.1% and GLM-4V-plus by 1.3%.Source: §6.2 · p. 6, Table 1 · p. 7
• LongVideo-R1 locates the segment containing a target performer, searches finer-grained sub-segments to pinpoint the exact performance moment, then invokes video QA to obtain a precise answer.Source: §6.2 · p. 7
• Semantically related segments can distract LongVideo-R1, but simple textual hints can redirect it to the correct segment and answer.The paper contrasts this failure with humans’ ability to recognize the error and refocus.Source: §6.2 · pp. 7–8
• Full 33K-sample SFT outperforms a 10K subset after both SFT and RL, while adding the location reward improves overall, KIR, and temporal-grounding performance.The authors conclude that larger SFT data and the location reward improve video navigation and long-video QA.Source: §6.2 · p. 6, Table 4 · p. 8, Table 5 · p. 8

6.3. Extension to Ultra-long Videos

• On videos lasting tens of hours, LongVideo-R1 reaches the correct location and answer in 10–20 rounds, whereas other approaches incur high or duration-growing sampling costs.The comparison includes open-source MLLMs sampling 256 frames and agent systems Ego-R1 [45] and VideoTree [53], whose sample counts grow linearly with video duration.Source: Figure 3 · p. 7, §6.3 · p. 8

7. Conclusion

• LongVideo-R1 achieves competitive QA accuracy across several long-video benchmarks and a favorable accuracy-efficiency tradeoff over other agentic algorithms.The conclusion also reports strengths in information retrieval and grounding, and notes that richer SFT data can help.Source: §7 · p. 9

A. Hierarchical Video Definition

• Each video is represented without preprocessing as a hierarchical tree: the root covers the whole video, and each node is recursively divided into W child segments across D levels.The agent starts with top-level sections, maintains descriptions, reasons whether it can answer, and selects where to navigate next.Source: §7 · p. 9, A. Hierarchical Video Definition · p. 12
• In practice, the tree has D = 3 levels, 16-second leaf segments, and an adaptively chosen width that typically ranges from 4 to 8.The number of leaf segments is Duration/16.Source: A. Hierarchical Video Definition · p. 12
• Captioning uses 256, 128, 64, and 32 frames from levels 0 through 3, respectively, with frame sampling and resolution adjusted to preserve approximately constant visual-token counts across levels.This keeps computation per caption call consistent while providing finer temporal and spatial detail deeper in the hierarchy.Source: A. Hierarchical Video Definition · p. 12

B.1. Environment Setup

• During data generation and evaluation, Qwen-2.5-VL-72B serves as the video captioning model and Qwen-2.5-VL-32B as the video QA model.The components are replaceable; Qwen-2.5-VL-32B can also serve as an effective captioning model.Source: §B.1 · p. 12
• GPT-5 generates CoTWT data, while SFT and RL training use eight NVIDIA H800 GPUs with mixed precision and FSDP sharding.The setup uses these techniques to maximize training throughput.Source: §B.1 · p. 12

B.2. Data Generation

• From CGBench’s approximately 1,200 videos and 12,000 QA pairs, GPT-5 generated SFT supervision using 800 videos and around 8,000 QA pairs, yielding 5,600 high-quality CoTWT trajectories after filtering.The model used for generation is proprietary.Source: §B.2 · p. 12
• The remaining 400 CGBench videos and approximately 4,200 QA pairs were reserved for RL training.Source: §B.2 · p. 12
• Starting GPT-5 with only the root-level caption led to unstable behavior and around 30% accuracy, while providing the highest-level nodes’ W child captions substantially improved stability and accuracy.Source: §B.2 · p. 12
• Tree width W was set between 4 and 8 for CGBench, LVBench, and VideoMME-Long, and between 3 and 8 for EgoSchema.The EgoSchema setting reflects its large number of short 2-minute videos.Source: §B.2 · p. 12

• Qwen3-8B serves as the central model for both SFT and RL, starting from the W highest-level captions and interacting with the video tree through tool calls.Source: §B.3 · p. 12
• SFT trains the model to imitate CoTWT trajectories by predicting reasoning, search actions, and answers, building hierarchical search and structured video reasoning skills.Source: §B.3 · p. 12
• During RL, hierarchical captions are pre-extracted while the video QA tool runs in real time; Qwen-2.5-VL-32B serves as the QA module on two GPUs, with six GPUs for RL.The detailed hyperparameters are listed in Table 8.Source: §B.3 · pp. 12–13

B.4. Time consumption calculation

• The expected per-question time is T = C1T1 + C2T2 + C3T3, combining reasoning-model, video-captioning, and video-QA calls.Timing was tested on an A800.Source: §B.4 · p. 13
• VideoMME-Long averages 10.5 reasoning rounds and 0.36 video-QA calls per question, yielding 14.14 caption calls.The caption-call calculation uses average tree width W = 5 and C2 = W + C1 − 1 − C3.Source: §B.4 · p. 14
• Assuming Qwen-2.5-VL-32B is used for both video captioning and video QA, the empirical average runtimes are T1 ≈2.5s, T2 ≈7.0s, and T3 ≈2.7s.Source: §B.4 · p. 14
• ≈135s is the expected end-to-end time per VideoMME-Long question for the full hierarchical search, including caption retrieval and occasional fine-grained Video QA.The estimate is calculated as 10.5 × 2.5 + 14.14 × 7.0 + 0.36 × 2.7 seconds.Source: §B.4 · p. 14

C. More Qualitative Examples

• Examples in Figures 5–7 illustrate hierarchical search, disambiguation of similar scenes across hours, and joint use of high-level and fine-grained information.Figure 5 contains ultra-long-video examples; Figures 6 and 7 provide additional qualitative results.Source: C. More Qualitative Examples · p. 14
• In Downton Abbey examples, LongVideo-R1 navigates multi-hour narratives to locate characters, objects, or events despite substantial visual similarity across episodes.The examples show repeated successful localization across the series’ similar scenes.Source: C. More Qualitative Examples · p. 14

D. Failure Examples

• Despite performing well across long-video benchmarks, LongVideo-R1 still exhibits failure cases.The examples show that similar information can disturb the model, while human hints can bring it back on track.Source: D. Failure Examples · p. 14, Figure 8 · p. 16, Figure 9 · p. 17
• A visually similar but irrelevant object can lead LongVideo-R1 down the wrong branch, from which it fails to return to the correct segment.Figures 8 and 9 illustrate the distraction by similar information.Source: D. Failure Examples · p. 14, Figure 8 · p. 16, Figure 9 · p. 17
• Simple textual hints can often guide LongVideo-R1 back to the correct segment and enable it to answer correctly.Figures 8 and 9 describe people guiding the model back on track with a few hints.Source: D. Failure Examples · p. 14, Figure 8 · p. 16, Figure 9 · p. 17