Source-linked AI summary
Scaling RL to Long Videos
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, Sifei Liu, Hongxu Yin, Yao Lu, Song Han
TL;DR
Long-video VLM reasoning requires integrating information across extended footage, while RL over hundreds or thousands of frames is computationally demanding. LongVILA-R1 combines a 104K-pair reasoning dataset, two-stage CoT-SFT and RL, and MR-SP, achieving broad benchmark gains, 65.1% and 71.1% on VideoMME, and support for up to 8,192 frames.
Problem
Long-video reasoning requires integrating temporal, spatial, goal-oriented, and narrative information across minutes or hours, while long-video RL remains computationally challenging.
Method
LongVILA-R1 combines LongVideo-Reason, two-stage Long-CoT-SFT and RL, and MR-SP with sequence parallelism, a tailored vLLM engine, and cached video embeddings.
Results
LongVILA-R1-7B consistently outperforms LongVILA-7B across video benchmarks, achieving 65.1% and 71.1% on VideoMME without and with subtitles.
Takeaways & Limitations
LongVILA-R1-7B supports processing up to 8,192 video frames per video, while MR-SP supports hour-level RL training and provides up to 2.1× speedup.
Takeaways & Limitations
Scaling to significantly longer sequences, more modalities, or larger batch sizes requires distributed training across multiple GPUs, increasing resource demands.
Abstract
from arXiv · showhide
We introduce a full-stack framework that scales up reasoning in vision-language models (VLMs) to long videos, leveraging reinforcement learning. We address the unique challenges of long video reasoning by integrating three critical components: (1) a large-scale dataset, LongVideo-Reason, comprising 104K long video QA pairs with high-quality reasoning annotations across diverse domains such as sports, games, and vlogs; (2) a two-stage training pipeline that extends VLMs with chain-of-thought supervised fine-tuning (CoT-SFT) and reinforcement learning (RL); and (3) a training infrastructure for long video RL, named Multi-modal Reinforcement Sequence Parallelism (MR-SP), which incorporates sequence parallelism and a vLLM-based engine tailored for long video, using cached video embeddings for efficient rollout and prefilling. In our experiments, LongVILA-R1-7B achieves strong performance on video benchmarks, reaching 65.1% and 71.1% accuracy on VideoMME without and with subtitles, respectively, and consistently outperforming LongVILA-7B across multiple benchmarks. Moreover, LongVILA-R1-7B supports processing up to 8,192 video frames per video, and configurable FPS settings. Notably, our MR-SP system achieves up to 2.1x speedup on long video RL training. In addition, we release our training system for public availability that supports RL training on various modalities (video, text, and audio), various models (VILA and Qwen series), and even image and video generation models. On a single A100 node (8 GPUs), it supports RL training on hour-long videos (e.g., 3,600 frames).
1 Introduction
Long-video understanding requires reasoning over temporal, spatial, goal-oriented, and narrative information distributed across extended footage. LongVILA-R1 combines a large reasoning dataset, two-stage training, and MR-SP to improve long-video reasoning and efficiency.
- Motivation: Long-video questions require integrating clues across time, hidden goals, spatial relations, and evolving narratives.Examples include football outcomes, poker decisions, and tracking a hidden object’s final location.
- Contributions: LongVideo-Reason provides 104K question-reasoning-answer pairs, with 36K used for Long-CoT-SFT and 68K plus 102K existing samples used for RL.The dataset supports four reasoning perspectives and a separate 1K-sample evaluation benchmark.
- Results: 2.1× speedup is achieved by MR-SP for 512-frame video RL training on 7B models.The comparison uses a single node with 8 A100 GPUs.
- Contributions: MR-SP addresses massive visual embeddings, heavy rollouts, and long-context prefilling through sequence parallelism, a tailored vLLM engine, and cached video embeddings.The system is designed to reduce memory demands during long-video RL training.
- Results: 65.1% and 71.1% accuracy are achieved by LongVILA-R1-7B on VideoMME without and with subtitles, respectively.It consistently outperforms LongVILA-7B across multiple video benchmarks and supports up to 8,192 frames with configurable FPS.
2 Related Work
Prior work advances multimodal reasoning, long-context processing, and sequence parallelism, but long-video RL introduces additional scaling challenges. Existing systems provide relevant foundations without directly resolving the full long-video RL setting.
- Multimodal reasoning: Multimodal reasoning research includes stronger visual reasoning, million-token context models, and two-stage or reinforcement-learning approaches.Examples include GPT-4o, Gemini-1.5-Pro, LMM-R1, and Vision-R1.
- Sequence parallelism: Sequence parallelism distributes long-context computation using ring-based point-to-point or Ulysses-style all-to-all communication.LongVILA extends sequence parallelism to multimodal inputs through MM-SP.
- RL frameworks: Multimodal reinforcement learning adds challenges because long-video training requires extensive sampling from long sequences.This motivates specialized infrastructure beyond existing sequence-parallel approaches.
3 LongVideo-Reason Data Construction
LongVideo-Reason is constructed through large-scale video collection and automated chain-of-thought annotation focused on four types of long-video reasoning. Filtering selects examples suited to supervised fine-tuning and reinforcement learning.
- Data curation: 18K long videos are curated from Shot2Story and supplemented with 2K 4K-resolution videos from driving, games, robotics, and wildlife.The resulting dataset contains 104K question-reasoning-answer pairs.
- Data curation: The dataset categorizes questions into Temporal, Goal and Purpose, Spatial, and Plot and Narrative Reasoning.These categories are intended to cover diverse long-video reasoning tasks.
- Data filtering: Repeated inference labels questions as easy, hard, or medium, and filtering removes the easy and hard cases for training.The remaining questions are selected because GRPO requires diverse rollouts to produce meaningful advantages.
- Annotation pipeline: Videos are segmented into approximately 10-second clips, which NVILA-8B captions before questions and reasoning annotations are generated from all clip captions.Spatial examples additionally use object bounding boxes and corresponding captions.
- Annotation pipeline: A reasoning LLM generates question-reasoning-answer pairs through prompts targeting the four reasoning categories and iterative visual examination.The annotations are then reformatted for concision and alignment with video details.
4 LongVILA Training Pipeline
LongVILA-R1 uses supervised warm-up followed by reinforcement learning for long-video reasoning. MR-SP supports the RL stage by scaling frame encoding, rollout, and long-context prefilling under high computational demands.
- Long CoT-SFT: Stage 1 uses 36K high-quality chain-of-thought examples for long-video supervised fine-tuning on MM-SP.This warm-up initializes reasoning and instruction-following abilities before RL.
- Reinforcement learning: Stage 2 applies reinforcement learning with dense long-video frames after supervised warm-up.The training pipeline combines LongVILA’s base pipeline, MM-SP, and MR-SP.
- GRPO: For each question, GRPO samples a response group from the old policy and optimizes the policy using rule-based format and accuracy rewards.The sampled rewards are normalized into advantages for updating the model.
- MR-SP: MR-SP addresses long-video RL bottlenecks in visual embeddings, rollouts, and LLM prefilling through sequence parallelism and a LongVILA-tailored vLLM engine with cached embeddings.These mechanisms target memory-intensive long-context training.
- Reinforcement learning: 68K filtered examples plus 102K samples from other video datasets are used to scale RL training and encourage exploration of generalized reasoning strategies.Filtering is motivated by GRPO’s sensitivity to sampling diversity.
5 Multi-modal Reinforcement SP
MR-SP is an efficient long-video RL framework that combines sequence parallelism, balanced multimodal sharding, video-embedding reuse, and vLLM acceleration.
- MR-SP applies sequence parallelism to long-video RL rollout and prefilling, addressing the high token volume of long videos.The framework uses sequence parallelism in both rollout and pre-filling stages.
- Stage 1 - Rollout with Paralleled Encoding: Video frames are distributed across GPUs for independent encoding, then embeddings are aggregated through an all-gather operation.Each GPU processes only a subset of frames using its own vision tower.
- The workflow uses custom multimodal sharding and vLLM-based rollout acceleration while supporting dense-video policy prefilling.The design targets balanced workload distribution and compatibility with sequence-parallel communication.
- Cached video embeddings are reused across multiple rollouts, avoiding repeated encoding of the same video during each training step.Typical training steps perform 8 to 16 rollouts, making embedding reuse important for reducing redundant computation.
- Stage 2 - Parallel Prefilling: MR-SP pads gathered embeddings to a uniform length and shards the sequence across GPUs for policy and reference-model prefilling.The same sequence-parallel prefilling strategy is applied to both models.
6 Experimental Results
LongVILA-R1-7B consistently improves over LongVILA-7B across video benchmarks, while scaling reasoning with more frames and benefiting from the complete CoT-SFT-plus-RL pipeline and MR-SP efficiency.
- 6.1 Main Results: 65.1% and 71.1% are LongVILA-R1-7B’s VideoMME accuracies without and with subtitles, respectively.The model is evaluated with 512 video frames in the detailed comparison.
- 6.1 Main Results: LongVILA-R1-7B outperforms LongVILA-7B across ActivityNet-QA, LongVideoBench, PerceptionTest, NExT-QA, VNBench, and VideoMME.The performance gaps vary according to the complexity of the reasoning tasks.
- 6.1 Main Results: 72.0% is LongVILA-R1-7B’s average accuracy on LongVideo-Reason-eval, surpassing Video-R1-7B and slightly outperforming Gemini-1.5-Pro.
- 6.2 Ablation Study: 64.3% is LongVILA-R1-1.5B’s score at 512 frames, while LongVILA-1.5B reaches a performance bottleneck and degrades at that frame count.LongVILA-R1-1.5B improves steadily as the number of input frames increases.
- 6.2 Ablation Study: Adding RL after CoT-SFT improves accuracy, whereas skipping CoT-SFT or using Video-R1 datasets in both stages reduces performance.The ablation uses LongVILA-1.5B on LongVideo-Reason-eval.
- 6.2 Ablation Study: 2.1× is the maximum speedup from full MR-SP at 512 frames, while the system scales to 1024 frames without OOM.Stage 1 alone improves efficiency up to 512 frames but encounters OOM beyond that point.
7 Conclusion
LongVILA-R1 combines the LongVideo-Reason dataset, two-stage CoT-SFT and RL training, and MR-SP to scale VLM reasoning over long videos. It reports strong benchmark performance, supports long inputs, and enables hour-level RL training on one A100 node.
- LongVideo-Reason contains 104K long-video question-reasoning-answer pairs used with CoT-SFT and RL in the LongVILA-R1 framework.The framework combines the dataset with other open-source video datasets.
- 65.1% and 71.1% are LongVILA-R1-7B’s VideoMME accuracies without and with subtitles, respectively.
- LongVILA-R1 supports up to 8,192 video frames per video and configurable FPS settings.
- MR-SP provides a 2.1× speedup for long-video RL training and supports 3,600-frame, hour-level training on one node with 8 A100 GPUs.
- LongVideo-Reason data generation builds long videos from grouped short clips, captions, and LLM-generated multiple-choice Long-CoT question-answer pairs.The design emphasizes temporal and causal evidence rather than identity labels or free-form attribution.
8 Border Impacts
LongVILA-R1 is presented as enabling sophisticated temporal and compositional understanding for extended visual contexts, with potential applications across embodied and immersive systems. Its data-generation design also aims to reduce privacy and profiling risk.
- LongVILA-R1 is positioned for temporal and compositional understanding across diverse, extended visual contexts.The paper connects this capability to embodied AI, robotics, autonomous systems, and AR/VR.
- Long-video reasoning could help robots sustain environmental understanding, perform multi-stage tasks, adapt dynamically, and build richer world models.The passage also identifies possible opportunities in education, healthcare, and entertainment.
- LongVideo-Reason supervision emphasizes temporal and causal evidence while avoiding identity labels and free-form attribution.This design is described as reducing privacy and profiling risk.
- The paper frames responsible progress in long-video reasoning as requiring privacy protection and ethical principles.
NeurIPS Paper Checklist
The checklist indicates that the paper’s claims match its contributions and scope, while its limitations are acknowledged. It also reports positive assessments of reproducibility, openness, experimental detail, statistical significance, and computational-resource disclosure.
- Claims and limitations: The paper’s abstract and introduction accurately reflect its contributions and scope.
- Claims and limitations: The paper acknowledges the limitations of the work performed by the authors.
- Reproducibility and access: The paper fully discloses information needed to reproduce its main experimental results.
- Reproducibility and access: The paper provides open access to the data and code with instructions for faithfully reproducing the experiments.
- Experimental reporting: The paper specifies the training and test details necessary to understand the results.
- Experimental reporting: The checklist reports suitable statistical-significance information and sufficient computer-resource details for reproducing the experiments.
Appendix
The appendix compares Gemini-1.5-Pro, Video-R1-7B, and LongVILA-R1-7B across strategic, narrative, spatial, and temporal long-video reasoning examples. LongVILA-R1-7B generally provides more accurate, coherent reasoning by analyzing detailed video moments and spatial relationships.
- Goal and Purpose: LongVILA-R1-7B accurately analyzes player styles and marked moments to predict a 20-minute StarCraft match’s trajectory.Gemini-1.5-Pro predicted correctly but included factual inaccuracies, while Video-R1-7B neglected critical video details.
- Spatial: LongVILA-R1-7B uses dense frame analysis to infer spatial relationships across rooms and levels, producing coherent reasoning and the correct answer.Video-R1-7B exhibited localization errors that created a mismatch between its reasoning and final answer.
- Temporal: All three models correctly reason through the Lego video’s sequence of events and interactions in the temporal example.The example demonstrates proficiency in temporal reasoning despite differences observed in other reasoning categories.
Temporal 2022 FIFA Argentina v.s. Netherlands
The examples test temporal and spatial tracking in football, box-swapping, narrative, and cake-preparation videos. Model responses differ in whether they use detailed temporal evidence and maintain object or event identities across scenes.
- Temporal 2022 FIFA Argentina v.s. Netherlands: Gemini-1.5-Pro selects Argentina based on prior shootout history and nine extra-time periods, while Video-R1 selects the Netherlands based on physical condition and consistency.The two models produce different answers, B and A respectively, for the football example.
- Spatial Track - Where the ball is: Gemini-1.5-Pro tracks the swapped boxes to place the purple ball in the left box, whereas Video-R1 places it in the right box.The models’ responses illustrate disagreement when tracking the trajectory of the ball-containing box.
- Plot and Narrative: The narrative example asks how Susan recognizes that the returning man is not her husband, with the options including an absent wedding ring.The supplied model responses disagree about whether the open sleeve or missing left-hand ring is decisive.
Spatial – House Tour
The house-tour and cake examples require models to combine spatial layout, temporal progression, and visual details across multiple scenes. The responses show correct and inconsistent reasoning depending on the task and model.
- Spatial – House Tour: The house-tour question asks which room is adjacent to the staircase with the wedding photo on the upper floor.The options include a kitchen, nursery, bathroom, and dining room.
- Spatial – House Tour: Gemini-1.5-Pro identifies the nursery by linking the staircase scene to a room with a crib and blue walls.The response cites the room’s appearance and Emily’s description of it as the baby’s room.
- Spatial – House Tour: Video-R1’s reasoning considers the kitchen but ultimately answers B, while expressing uncertainty about the upper-floor location of rooms.Its analysis contains an intermediate kitchen choice before giving the nursery option as the final answer.
- Temporal – LEGO PINATA: The cake question requires identifying the preparation involving both a blue hammer and a red silicone mold.The answer options distinguish several Lego and non-Lego cake preparations.
- Temporal – LEGO PINATA: Gemini-1.5-Pro and Video-R1 both select the Lego-themed cake option involving the two specified tools.Gemini-1.5-Pro attributes the mold to shaping a decorative brick, while Video-R1 also answers D.