PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO

Jinyoung Park, Jeehye Na, Jinyoung Kim, Hyunwoo J. Kim

arXiv:2506.07464v5cs.CVcs.AI

TL;DR

VideoLLM reasoning remains challenging, and GRPO's effectiveness in VideoLLMs is limited by reliance on safeguards and vanishing advantages. DeepVideo-R1 combines Reg-GRPO, which regresses group-based advantages, with difficulty-aware augmentation, improving reasoning across multiple benchmarks.

  • Problem

    GRPO's application to VideoLLMs remains underexplored, with heuristic safeguards and vanishing advantages hindering effective training.

  • Method

    DeepVideo-R1 combines Reg-GRPO, a regression objective over group-based advantages, with difficulty-aware augmentation that adjusts video-text difficulty.

  • Results

    DeepVideo-R1 consistently improves reasoning across multiple video benchmarks and outperforms GRPO-based reinforcement fine-tuning.

  • Takeaways & Limitations

    Combining a regression-based RL objective with difficulty-aware data augmentation improves video reasoning performance across in-distribution and out-of-distribution tasks.

Abstract

from arXiv · show

Recent works have demonstrated the effectiveness of reinforcement learning (RL)-based post-training for enhancing the reasoning capabilities of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) has shown impressive success using a PPO-style reinforcement learning algorithm with group-normalized rewards. However, the effectiveness of GRPO in Video Large Language Models (VideoLLMs) remains underexplored. In this paper, we explore GRPO and identify two issues that hinder effective learning: (1) reliance on safeguards, and (2) vanishing advantage. To mitigate these challenges, we propose DeepVideo-R1, a video large language model trained with Reg-GRPO (Regressive GRPO) and difficulty-aware data augmentation. Reg-GRPO reformulates the GRPO loss function as a regression task that directly predicts the advantage in GRPO, eliminating the need for safeguards such as clipping and min operations. This directly aligns the model with the advantages, providing guidance to prefer better outputs. The difficulty-aware data augmentation strategy augments input prompts/videos to target solvable difficulty levels, enabling diverse reward signals. Our experimental results show that our approach significantly improves video reasoning performance across multiple benchmarks.

1 Introduction

VideoLLMs extend language-model reasoning to dynamic video but still struggle with complex temporal, spatial, and semantic tasks. DeepVideo-R1 addresses GRPO's training limitations with Reg-GRPO and difficulty-aware augmentation, improving reasoning performance across challenging benchmarks.

  • VideoLLMs unify video understanding and language generation for temporal event reasoning, video question answering, and related capabilities.
  • Complex VideoLLM reasoning remains difficult, while supervised fine-tuning primarily fits instruction data rather than reasoning processes.The motivation passage frames reinforcement learning as a way to optimize beyond likelihood objectives using task-specific or preference-based rewards.
  • GRPO application to VideoLLMs is underexplored and is limited by heuristic safeguards that suppress gradients and vanishing advantages that remove training signals.Extremely easy or difficult samples can yield zero advantages, while minimum and clipping operations can impede convergence.
  • DeepVideo-R1 improves reasoning across multiple challenging benchmarks, with consistent gains on both in-distribution and out-of-distribution tasks.The cited results compare the model favorably with recent VideoLLMs such as Qwen2.5-VL.
  • Reg-GRPO casts GRPO as regression over group-based advantage values, eliminating clipping and min operations while directly aligning model outputs with advantages.
  • Difficulty-aware augmentation adaptively scales video-text inputs, injects video cues, and perturbs noise to generate richer reward signals across difficulty levels.For easy samples it injects uncertainty, whereas for hard samples it provides auxiliary reasoning cues.

2 Related Work

Related work establishes reinforcement-learning fine-tuning as a route to improved reasoning in LLMs and introduces GRPO as a group-normalized reward method. VideoLLMs extend these capabilities to dynamic video tasks, while related RL variants explore reward-regression objectives.

  • VideoLLMs extend LLM reasoning to dynamic video domains, including video question answering and other video understanding tasks.
  • Reinforcement-learning fine-tuning methods, including DPO and RLHF, have improved LLM reasoning, while newer variants derive direct reward-regression losses from RL objectives.
  • GRPO computes group-wise normalized rewards to stabilize training and improve efficiency.

3 Methods

The method addresses GRPO’s safeguard dependence and vanishing advantages in VideoLLM training with Reg-GRPO and difficulty-aware augmentation. Reg-GRPO regresses group-based advantages, while augmentation adjusts input difficulty to preserve informative reward variation.

  • GRPO limitations: GRPO training is limited by heuristic safeguards and vanishing advantages, especially when samples are too easy or too difficult.Safeguards can suppress gradients, while equal rewards within a group produce zero training signals.
  • Reg-GRPO: Reg-GRPO reformulates GRPO as regression over group-based advantages, removing min and clipping safeguards.The reformulation directly aligns model outputs with advantage values and removes the normalization term Z(x).
  • Reg-GRPO: Reg-GRPO minimizes the gap between target and predicted advantages while retaining KL-divergence regularization.The method defines a predictive advantage and optimizes the policy by regressing it toward the target advantage.
  • Difficulty-aware data augmentation: Difficulty-aware augmentation estimates sample difficulty from average rewards relative to a replay-buffer reference that accounts for model evolution.The input is a video-question pair, and its difficulty is computed from reward comparisons over recent replay-buffer samples.
  • Difficulty-aware data augmentation: The augmentation increases difficulty for easy samples with perturbations and decreases difficulty for hard samples with reasoning cues, adapting intervention strength to sample difficulty.Figure 2 describes noise perturbation for easy samples and reasoning-cue injection for hard samples, with guidance or noise levels chosen adaptively.

4 Experiments

DeepVideo-R1 is evaluated across general, long-video, grounded, and reasoning benchmarks, where it consistently outperforms baselines. Ablations and analyses attribute gains to Reg-GRPO and difficulty-aware augmentation, which reduce vanishing advantages and improve reward learning.

  • Benchmark results: DeepVideo-R1-3B improves over Qwen2.5-VL-3B + GRPO by +8.5, +10.1, and +8.6 points on SBR-L1, L2, and L3, respectively.It achieves the best performance across all three settings, with larger gains on cross-task and cross-environment settings.
  • Benchmark results: DeepVideo-R1 achieves an overall score of 51.1 on LongVideoBench and gains +7.4 over Qwen2.5-VL-3B + GRPO on 900–3600 s videos.The benchmark evaluates long-video reasoning and temporal compositional understanding.
  • Benchmark results: DeepVideo-R1 improves performance on 5 out of 6 datasets against Qwen2.5-VL-7B and raises TempCompass from 29.7 to 63.1.The evaluation covers diverse video reasoning and general video understanding benchmarks.
  • Benchmark results: DeepVideo-R1-7B reaches 36.8 mIoU and 72.5 Acc@QA on NExTGQA, exceeding VideoChat-R1 by +4.4 mIoU and +2.3 Acc@QA.The model uses a composite reward combining accuracy, format consistency, and IoU.
  • Ablation and augmentation analysis: Both Reg-GRPO and difficulty-aware augmentation contribute to improvement, including a +2.1-point SBR (L1) gain for augmentation and a +4.1-point gain for Reg-GRPO without augmentation.Relative difficulty measurement outperforms absolute measurement across L1–L3; combining difficulty-increasing and difficulty-decreasing augmentation yields a +7.7-point SBR (L3) gain, while adaptive scaling outperforms fixed scaling and no augmentation.
  • Method analysis: DeepVideo-R1 outperforms compared reinforcement fine-tuning methods, with direct advantage alignment performing better than reward regression.The qualitative example also reports correct next-action prediction where the GRPO-only model fails to recognize berries.
  • Training analysis: Difficulty-aware augmentation reduces the vanishing-advantage ratio, while DeepVideo-R1 obtains a higher average reward than GRPO during training.The reward curves plot training step against average reward.

5 Conclusion

The conclusion presents DeepVideo-R1 as a VideoLLM combining Reg-GRPO with difficulty-aware augmentation to address two GRPO training challenges. Experiments report consistent reasoning improvements across diverse VideoLLMs and superiority over GRPO-based reinforcement fine-tuning.

  • DeepVideo-R1 combines Reg-GRPO, which directly aligns outputs with group-normalized advantages, with difficulty-aware augmentation that modulates input difficulty.
  • Experiments show consistent reasoning improvements across diverse VideoLLMs, with DeepVideo-R1 outperforming GRPO-based reinforcement fine-tuning.

NeurIPS Paper Checklist

The supplied checklist passages state that abstract and introduction claims should match contributions, assumptions, limitations, and the expected scope of generalization.

  • The checklist requires abstract and introduction claims to clearly state contributions, important assumptions, and limitations.
  • Claims should match theoretical and experimental results and reflect how far results are expected to generalize.

2. Limitations

The paper’s review materials emphasize reproducibility, experimental detail, statistical reporting, compute disclosure, theoretical assumptions, and explicit discussion of limitations.

  • Limitations: The review materials call for the paper to discuss assumptions, robustness, claim scope, and factors that influence performance.These include limits from testing on only a few datasets or runs and possible violations of assumptions.
  • Theory: The paper states that theoretical assumptions and derivations are provided in the appendix.The review guidance emphasizes complete assumptions and proofs for theoretical results.
  • Reproducibility: The paper reports experimental reproducibility information in the experiments and appendix.The supplied review material also marks training and test details, compute resources, and open access to data and code as addressed in the appendix or paper.
  • Statistical reporting: The experiments are reported as single runs without error bars or other statistical-significance information.The justification explicitly attributes the absence of error bars to following prior single-run practice.
  • Ethics: The review materials state that the paper follows the NeurIPS Code of Ethics.The supplied justification marks ethics compliance as addressed.

10. Broader impacts

The supplied broader-impact materials address societal impacts, asset licensing, human-subject considerations, LLM-use disclosure, and responsible release of data or models.

  • Societal impacts: The paper states that both potential positive and negative societal impacts are discussed in the appendix.The supplied justification identifies the appendix as the location of this discussion.
  • Responsible release: The paper states that it does not release datasets, so safeguards for released high-risk assets are marked not applicable.The supplied justification specifically gives non-release of datasets as the reason.
  • Asset licensing: The paper states that existing assets are properly credited and their licenses and terms of use are addressed in the appendix.The supplied review item marks this requirement as satisfied.
  • Human subjects: The paper states that it does not involve crowdsourcing or research with human subjects.The same justification is provided for both human-subject and IRB-related review items.
  • LLM disclosure: The paper states that LLM usage is described in the appendix when relevant to the core methods.The supplied review item asks for disclosure when LLMs are important, original, or non-standard method components.

A.4 Reg-GRPO

Reg-GRPO casts GRPO optimization as direct regression on group-based advantages, while retaining group normalization and verifiable format, accuracy, and temporal-localization rewards.

  • Reg-GRPO: Reg-GRPO directly predicts group-based advantage values by reformulating the GRPO objective as a regression problem.The method is presented as directly learning the model to predict the advantage.
  • Reg-GRPO: Reg-GRPO directly regresses the group-normalized target, capturing how much better one response is rather than only which response is preferred.The discussion contrasts this finer-grained signal with preference-only learning.
  • Comparison: Unlike REBEL, Reg-GRPO learns group-based normalized advantages instead of unnormalized pairwise reward gaps.The group-level normalization is described as addressing scale discrepancies and high variance across batches.
  • Rewards: Training uses format rewards that verify whether outputs contain the required thought and answer tags.Regular expressions check the specified output format.
  • Rewards: Accuracy rewards compare extracted answers with ground truth, while IoU rewards evaluate temporal-segment localization.For temporal grounding, predictions and ground truth are represented as event timestamps.

C.1 Implementation Details

The implementation uses standard post-training libraries and frozen visual encoders, evaluates several preference-optimization baselines, and reports reward regression as weaker than Reg-GRPO.

  • Implementation: The implementation uses PyTorch, Hugging Face Transformers, TRL, and vLLM for VideoLLM post-training and rollout.The experiments use Qwen2.5-VL and Qwen2-VL bases, with NVIDIA A100 GPUs for 3B models and H200 GPUs for 7B models.
  • Implementation: Only the language model is fine-tuned while the visual encoder remains frozen.Qwen2.5-VL and Qwen2-VL are used as the base VideoLLMs.
  • Implementation: The SEED-Bench-R1 setup uses a 0.1 KL regularizer, groups of 8 generations, and a reward history of the most recent W = 100 samples.The default analysis model is Qwen2.5-VL-3B, and GRPO omits safeguards in this setup.
  • Baselines: The comparison includes DPO, Online DPO, REINFORCE, RLOO, and REBEL as baseline methods.DPO uses pairwise comparisons, REINFORCE uses reward-weighted likelihood, and RLOO subtracts a Monte Carlo baseline.
  • Results: Reward-Regression performs worse than Reg-GRPO because Monte Carlo estimation of Z (x) is inaccurate and direct reward regression has high variance.The passage identifies these as limitations of the baseline.

E Datasets

The paper evaluates VideoLLMs across datasets spanning temporal, spatial, general, and expert-level video understanding, using varied video durations, domains, and task formats.

  • SEED-Bench-R1 evaluates video understanding post-training methods using videos from Epic-Kitchens and Ego4D and benchmark sources from EgoPlan-Bench and EgoPlan-Bench2.
  • LongVideoBench contains 3,763 videos and 6,678 question-answer pairs covering diverse domains, tasks, and progressively longer video durations.Its duration groups range from 8–15 seconds to 900–3,600 seconds, with a 100-second overall average.
  • VSI-Bench evaluates visual-spatial intelligence with over 5,000 question-answer pairs across eight tasks and three task types.
  • Video-MMMU and MMVU assess expert-level, knowledge-intensive video understanding across disciplines, while MVBench and TempCompass target temporal comprehension and perception.Video-MMMU uses 300 expert-level videos and 900 questions; MMVU covers 1,529 videos and 27 subjects.
  • Video-MME evaluates general video understanding with 900 videos and 2,700 question-answer pairs spanning six domains, 30 subtypes, and multiple duration categories.
  • NExTGQA contains 5,417 videos, 43,043 question-answer pairs, and 10,531 timestamp labels for temporal grounding evaluation.The validation split is used for training and the test split for evaluation because temporal annotations are available only in those splits.

F.2 Limitations

The authors identify unresolved data-leakage risk from possible pretraining–benchmark overlap and substantial computational and memory requirements inherited from the underlying VideoLLM.

  • Possible overlap between pretraining content and downstream evaluation benchmarks creates a risk of implicit data leakage.
  • Because DeepVideo-R1 is fine-tuned on a pretrained VideoLLM, it may inherit high computational and memory requirements.
Loading 2506.07464v5…