PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-R1: Reinforcing Video Reasoning in MLLMs

Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, Xiangyu Yue

arXiv:2503.21776v4cs.CV

TL;DR

Video reasoning with rule-based RL is challenged by insufficient temporal modeling and scarce high-quality video-reasoning data. Video-R1 addresses these gaps with T-GRPO and hybrid image-video training datasets, improving performance across video benchmarks and reaching 37.1% accuracy on VSI-Bench with Video-R1-7B, above GPT-4o.

  • Problem

    Directly applying GRPO to video reasoning lacks explicit temporal reward signals, while high-quality video-reasoning training data remains scarce.

  • Method

    Video-R1 introduces T-GRPO, which contrasts ordered and shuffled frame reasoning, and combines image-reasoning data with video data for SFT and RL.

  • Results

    Video-R1 achieves consistent improvements across six video benchmarks, with Video-R1-7B reaching 37.1% accuracy on VSI-Bench and outperforming GPT-4o.

  • Takeaways & Limitations

    The results support rule-based RL with temporal-aware training and hybrid image-video data as an approach for video reasoning in MLLMs.

  • Takeaways & Limitations

    The model is trained with 16 video frames, which may limit its ability to handle long-range temporal dependencies.

Abstract

from arXiv · show

Inspired by DeepSeek-R1's success in eliciting reasoning abilities through rule-based reinforcement learning (RL), we introduce Video-R1 as the first attempt to systematically explore the R1 paradigm for incentivizing video reasoning within multimodal large language models (MLLMs). However, directly applying RL training with the GRPO algorithm to video reasoning presents two primary challenges: (i) a lack of temporal modeling for video reasoning, and (ii) the scarcity of high-quality video-reasoning data. To address these issues, we first propose the T-GRPO algorithm, which encourages models to utilize temporal information in videos for reasoning. Additionally, instead of relying solely on video data, we incorporate high-quality image-reasoning data into the training process. We have constructed two datasets: Video-R1-CoT-165k for SFT cold start and Video-R1-260k for RL training, both comprising image and video data. Experimental results demonstrate that Video-R1 achieves significant improvements on video reasoning benchmarks such as VideoMMMU and VSI-Bench, as well as on general video benchmarks including MVBench and TempCompass, etc. Notably, Video-R1-7B attains a 37.1% accuracy on video spatial reasoning benchmark VSI-bench, surpassing the commercial proprietary model GPT-4o. All code, models, and data are released in: https://github.com/tulerfeng/Video-R1.

1 Introduction

Video-R1 extends rule-based RL to video reasoning by addressing missing temporal modeling and scarce high-quality video-reasoning data. It introduces T-GRPO and hybrid image-video training, achieving improvements across video benchmarks, including 37.1% on VSI-Bench for Video-R1-7B.

  • Motivation: Applying GRPO directly to video reasoning leaves temporal reasoning under-rewarded, encouraging shortcuts based on isolated frames.These shortcuts can produce sub-optimal reasoning patterns and poor generalization.
  • Motivation: High-quality video-reasoning data is scarce because existing datasets largely emphasize simple recognition rather than challenging reasoning.This limits exposure to diverse reasoning patterns during training.
  • Approach: T-GRPO contrasts responses to ordered and randomly shuffled frames, rewarding temporal reasoning when ordered-frame accuracy exceeds shuffled-frame accuracy.The design explicitly encourages use of temporal information rather than isolated-frame shortcuts.
  • Approach: Video-R1 combines image-based reasoning data with curated video data through Video-R1-CoT-165k for SFT and Video-R1-260k for RL.Image data supports general reasoning skills, while video data supplies temporal complexity.
  • Results: 37.1% accuracy on VSI-Bench is achieved by Video-R1-7B, outperforming GPT-4o while improvements extend across multiple video reasoning and general video benchmarks.Reported benchmarks include VideoMMMU, MMVU, MVBench, TempCompass, and VideoMME.

2 Related Works

Related work establishes video understanding as a core MLLM capability and highlights rule-based RL as a route to eliciting complex reasoning with outcome-based rewards.

  • Multimodal Large Language Models for Video: Video MLLMs improve dynamic understanding through approaches including token compression, spatial-temporal convolution, audio integration, long-context processing, and knowledge-driven segmentation.These methods target efficient long-video handling, richer multimodal cues, long-range reasoning, or complex video queries.
  • Reinforcement Learning for Reasoning: DeepSeek-R1 showed that coarse, outcome-only rewards can support complex reasoning, motivating extensions of rule-based reinforcement learning to multimodal models.This line of work differs from earlier approaches using dense step-level supervision or learned reward models.

3 Methods

Video-R1 combines mixed image-video training with T-GRPO, which explicitly rewards reasoning that depends on temporal order rather than isolated frames. Its pipeline uses curated datasets, rule-based rewards, and staged SFT followed by RL, with reported self-reflective reasoning behavior.

  • Dataset Construction: The training data emphasizes verifiable outputs for stable rule-based rewards, while also including free-form generation, OCR, and regression tasks for broader flexibility.Reward functions include exact matching for multiple-choice and numerical QA, WER for OCR, and ROUGE averages for free-form QA.
  • Temporal Group Relative Policy Optimization (T-GRPO): T-GRPO compares responses to temporally ordered and randomly shuffled frame sequences, granting positive temporal reward only when ordered inputs yield a higher correct-answer proportion.This contrastive design discourages shortcuts based on isolated frames and encourages policies that exploit temporal information.
  • Temporal Group Relative Policy Optimization (T-GRPO): Temporal reward is applied only to correct responses, so successful reliance on temporal patterns receives reinforcement while incorrect responses remain unaffected.For continuous-reward tasks, a threshold such as 0.5 can determine whether a response counts as correct.
  • Training Pipeline: Training proceeds from SFT on Video-R1-CoT-165k to RL with T-GRPO on Video-R1-260k, using image-video mixed training in both stages.The SFT stage initializes reasoning abilities; the RL stage encourages exploration beyond rigid pattern matching.
  • Aha Moment in Video Reasoning: Video-R1 exhibits self-reflective “aha moments,” revisiting video interpretations or prior outputs during ambiguous temporal cues and multi-step inference.The reported behavior is presented as evidence of internal re-examination rather than merely executing memorized patterns.

4 Experiments

Experiments evaluate Video-R1 across video reasoning and general video-understanding benchmarks, then analyze reinforcement learning, frame count, temporal rewards, and ablations. Results consistently favor the full method, with T-GRPO increasing temporally grounded responses from 60.2% to 75.0%.

  • 4.2 Main Results: Video-R1 significantly outperforms previous models across most benchmarks, especially VSI-Bench, VideoMMMU, and MMVU.The evaluation covers six benchmarks spanning video reasoning and general video understanding.
  • 4.2 Main Results: After only 1k reinforcement-learning steps, Video-R1 achieves significant performance boosts, whereas Qwen2.5-VL-7B-SFT improves inconsistently and slightly drops on VideoMME.The comparison attributes the stronger outcome to reinforcement learning rather than SFT alone.
  • 4.2 Main Results: Increasing the input frame number improves performance on almost all benchmarks, indicating benefits from richer temporal context.The result motivates models capable of reasoning over longer video inputs.
  • 4.3 Ablation Study: Removing image data, replacing T-GRPO with GRPO, or skipping SFT cold start makes performance worse across all benchmarks.The ablations support the roles of image-based reasoning data, temporal-aware training, and SFT initialization.
  • 4.5 Effect of Temporal Reward Analysis: 75.0% of Video-R1 responses incorporate temporal reasoning, compared with 60.2% for Video-R1-wo-temporal.The responses were evaluated on temporally grounded questions identified across six benchmarks.
  • 4.5 Effect of Temporal Reward Analysis: Average performance is favorable and similar at α = 0.2 and α = 0.3, while slight drops occur at α = 0.1 and α = 0.4.This indicates relative insensitivity to α within a reasonable range.

5 Conclusions

Video-R1 extends the R1 paradigm to video reasoning through temporal-aware reinforcement learning and mixed image-video reasoning datasets. Results across six benchmarks support the approach as a foundation for further video-reasoning research.

  • 5 Conclusions: Video-R1 introduces T-GRPO, a temporal-aware extension of GRPO that explicitly encourages temporal reasoning in video.The method is paired with Video-R1-CoT-165k for SFT cold start and Video-R1-260k for RL.
  • 5 Conclusions: Experimental results across six benchmarks validate the effectiveness of the Video-R1 approach.The paper presents the work as a foundation for further research in video reasoning with MLLMs.

A.1 Scaling Up RL Training

Scaling reinforcement learning from 1k to 10k steps generally improves Video-R1 performance across benchmarks, indicating benefits from additional RL training.

  • A.1 Scaling Up RL Training: Performance generally improves across various benchmarks when RL training increases from 1k to 10k steps.The results demonstrate the effectiveness of larger-scale RL training and suggest further gains from additional training.

A.2 Effect of Length Reward Analysis

Length-reward analysis shows that removing the reward shortens responses during RL training and lowers average benchmark performance. The full model maintains longer responses that stabilize at a higher level.

  • A.2 Effect of Length Reward Analysis: Removing the length reward causes a clear decline in response length during RL training, while Video-R1’s length rises and stabilizes higher.The comparison is between Video-R1 and Video-R1-wo-len.
  • A.2 Effect of Length Reward Analysis: Video-R1-wo-len achieves lower average performance across benchmarks than Video-R1.The authors guess this reflects failure to preserve deep reasoning.

C Prompt Template

This appendix presents the prompt templates used for training and inference, including a dedicated template for temporal reasoning evaluation.

  • C.1 Prompt Template for Training and Inference: Figure 11 provides the prompt template used for training and inference.
  • C.1 Prompt Template for Training and Inference: The training and inference template is presented as a figure in Section C.1.
  • C Prompt Template: Together, Figures 11 and 12 document prompt formats for general training or inference and temporal reasoning evaluation.
  • C.2 Prompt Template for Temporal Reasoning Evaluation: The paper explicitly identifies Figure 12 as illustrating the temporal reasoning evaluation prompt template.
  • C.2 Prompt Template for Temporal Reasoning Evaluation: Figure 12 provides the prompt template used for temporal reasoning evaluation.
  • C.2 Prompt Template for Temporal Reasoning Evaluation: The temporal reasoning evaluation template is presented as a figure in Section C.2.

D Additional Implementation Details

The implementation details specify the optimizer, learning rate, training durations, GRPO settings, stabilization controls, and response-length limit.

  • Adam is used with a 1e-6 learning rate, while SFT takes approximately 40 hours per epoch and RL takes around 15 hours for 1k steps.
  • The GRPO KL-divergence coefficient β is set to 0.04.
  • Training stability uses 0.01 weight decay and a maximum gradient norm of 5.
  • The maximum response length is 768 tokens.

E Limitations and Future Works

The paper frames Video-R1 as a foundation for video reasoning research and identifies limits involving temporal coverage, computational efficiency, response control, and image-to-video transfer.

  • Video-R1 is presented as a foundation for advancing video reasoning with MLLMs.
  • Increasing Frames Number: Training currently uses 16 video frames, which may limit handling of long-range temporal dependencies.
  • Better Temporal Modeling Method: T-GRPO introduces additional computational overhead through contrastive evaluation and reward calculation.
  • Adaptive Response Length Control: The current response-length control applies a fixed reward within a predefined range regardless of sample complexity.
  • Refined Image-to-Video Knowledge Transfer: Image-based reasoning data is currently incorporated by straightforwardly mixing it into the training set.
Loading 2503.21776v4…