Source-linked AI summary
Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency
Hongyu Li, Songhao Han, Yue Liao, Junfeng Luo, Jialin Gao, Shuicheng Yan, Si Liu
TL;DR
Real-world video understanding remains difficult because videos involve complex semantics and long temporal dependencies. Temporal-RLT uses GRPO with discrete semantic and continuous temporal rewards, alongside variance-aware selection of training examples. It outperforms SFT and Video-R1 across eight benchmarks, and the authors identify reward design and data selection as important for reasoning-centric video understanding.
Problem
Real-world video understanding remains challenging because videos combine complex semantics, diverse visual content, and long-range temporal dependencies.
Source: §1 · p. 2Method
Temporal-RLT builds on GRPO and combines task-specific semantic and temporal rewards with repeated-inference-based variance-aware sample selection.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors conclude that tailored reward design and data selection are important for post-training VideoLLMs for reasoning-centric video understanding.
Source: §6 · p. 10The work does not explore constructing high-quality, fine-grained thinking traces for SFT to enhance explicit thinking and reasoning abilities.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Understanding real-world videos with complex semantics and long temporal dependencies remains a fundamental challenge in computer vision. Recent progress in multimodal large language models (MLLMs) has demonstrated strong capabilities in vision-language tasks, while reinforcement learning tuning (RLT) has further improved their reasoning abilities. In this work, we explore RLT as a post-training strategy to enhance the video-specific reasoning capabilities of MLLMs. Built upon the Group Relative Policy Optimization (GRPO) framework, we propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals. To facilitate effective preference-based optimization, we introduce a variance-aware data selection strategy based on repeated inference to identify samples that provide informative learning signals. We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA. Our method consistently outperforms supervised fine-tuning and existing RLT baselines, achieving superior performance with significantly less training data. These results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs. Notably, The initial code release (two months ago) has now been expanded with updates, including optimized reward mechanisms and additional datasets. The latest version is available at https://github.com/appletea233/Temporal-R1 .
1 Introduction
Real-world video understanding remains challenging because videos combine complex semantics with long-range temporal dependencies. Temporal-RLT applies GRPO-based reinforcement learning tuning with task-specific rewards and variance-aware sample selection, and outperforms strong SFT baselines and Video-R1 across eight benchmarks.
1 Introduction
- Temporal-RLT combines discrete answer-correctness rewards for multi-choice VideoQA with continuous temporal-IoU rewards for grounding, extending the formulation to Grounded VideoQA.Grounded VideoQA requires answering questions and localizing supporting video segments.Source: §1 · p. 2
- Repeated inference selects samples with moderate answer consistency for QA and high within-group IoU variance for grounding to provide informative GRPO preference signals.The strategy filters out examples judged uninformative for optimization.Source: §1 · p. 2
- The authors construct Temporal-RLT-Full-490k across VideoQA, Temporal Grounding, and Grounded VideoQA, then derive the 32k Temporal-RLT-32k subset for diversity and optimization potential.The subset is intended to support efficient training with significantly less data.Source: §1 · p. 2
- Across eight video-understanding benchmarks, Temporal-RLT consistently outperforms strong SFT baselines and achieves superior performance to Video-R1 [4].The benchmarks span General VideoQA, Reasoning VideoQA, Temporal Grounding, and Grounded VideoQA.Source: §1 · p. 2
- Ablations examine the data-selection strategy and reward gains across tasks, supporting the reported effectiveness of reward design and data-efficient RLT.The framework overview shows VideoLLM outputs as either a QA answer choice or a temporal segment, optimized with task-specific rewards.Source: §1 · p. 3, Figure 2 · p. 3
2 Temporal-RLT
Temporal-RLT is a GRPO-based post-training framework for video-specific reasoning, combining semantic and temporal rewards across video tasks. It integrates structured outputs and task-specific rewards into a unified video-understanding strategy.
2 Temporal-RLT
- Temporal-RLT extends a standard VideoLLM with a GRPO-based reinforcement-learning module as a unified post-training strategy for video understanding.The pipeline comprises a base VideoLLM, dual rewards, and GRPO optimization.Source: §2 · p. 3, §2.3 · p. 5
2.1 Base Model: QwenVL-2.5 with Structured Reasoning
- The framework uses QwenVL-2.5 with a <think>/<answer> response template and a format-alignment reward to encourage reasoning before the final answer.The format reward encourages outputs to conform to the predefined structure.Source: §2.1 · p. 3
2.2 Dual Verifiable Rewards in Video Tasks
- Temporal-RLT combines discrete semantic rewards from multi-choice QA with continuous temporal IoU rewards from grounding to supervise complementary reasoning dimensions.The temporal reward provides a smooth signal for localization alongside binary semantic supervision.Source: §2.2 · p. 4
2.3 Temporal-RLT Optimization with GRPO
- GRPO compares multiple responses sampled for one input, normalizes their rewards into relative advantages, and updates the model toward higher-scoring responses.It does so without an explicit value function or learned critic.Source: §2.3 · pp. 4–5
- MC-QA, TVG, and Grounded VideoQA use task-specific rewards combining format adherence with answer correctness, temporal overlap, or both.Grounded VideoQA requires answering questions and marking relevant temporal spans with <observe> tags.Source: §2.3 · p. 4
3 Rethinking Data Efficiency for Video RLT
The section frames data efficiency for video RLT around finding samples whose varied rewards provide informative GRPO signals. It estimates difficulty through repeated inference, selects samples differently for discrete and continuous rewards, and builds a medium-difficulty training subset.
3.1 Difficulty Estimation via Multi-Sampling
- Difficulty estimation runs a base VideoLLM N ≫ K times per instance, measuring correct-response frequency for discrete tasks and reward spread for continuous tasks.The resulting output variance supports sample selection and prioritization for GRPO.Source: §3.1 · p. 5
3.2 Data Efficiency under Discrete Rewards
- For discrete QA, difficulty is the frequency of correct predictions across N inferences, and medium-difficulty samples are retained to preserve behavioral variance.GRPO benefits from groups containing both correct and incorrect answers.Source: §3.2 · p. 5
3.3 Data Efficiency under Continuous Rewards
- For continuous grounding rewards, the method estimates difficulty from within-group temporal IoU spread and excludes samples with negligible spread because they provide no preference gradients.The reward-discrepancy metric is used to avoid instability in direct variance computation with limited sampling.Source: §3.3 · p. 6
3.4 Difficulty-Aware Data Construction
- Temporal-RLT-Full-490k covers VideoQA, Temporal Video Grounding, and Grounded VideoQA, while Temporal-RLT-32k primarily selects medium-difficulty samples for stable, informative GRPO updates.The 32k subset is derived through offline difficulty estimation on the full dataset.Source: §3.4 · p. 6
- Difficulty-aware construction improves sample efficiency by reducing redundant updates on easy samples, mitigating instability from noisy hard samples, and aligning heterogeneous reward types.The unified optimization covers semantic and temporal reasoning tasks.Source: §3.4 · p. 6
4 Experiment
Temporal-RLT improves performance over SFT across video understanding tasks, with especially large gains in temporal grounding. Ablations show that data composition, output variance, and video-token count relate to task-specific performance and generalization.
4.1 Implementation Details
- Experiments use Qwen-VL-2.5-7B and evaluate general VideoQA, reasoning, temporal grounding, and grounded QA benchmarks.Training uses eight sampled responses per input; inference processes video at 2 FPS.Source: §4.1 · p. 6
4.2 Main Results
- Temporal-RLT beats SFT across evaluated tasks, with gains of +14.0 mIoU on Charades, +14.7 on ActivityNet, and +9.5 on ActivityNet-RTL.It also outperforms Video-R1 across evaluated tasks while requiring less training data.Source: §4.2 · p. 7, Table 1 · p. 7
4.3 Ablation Study
- TVG-only tuning gains +27.6 mIoU on Charades and +11.6 on ActivityNet, with no MMVU degradation and a +5.8 mIoU gain on grounded QA.Adding VideoQA improves general and reasoning-oriented QA but moderately reduces grounding precision; adding grounded VQA yields further domain-specific gains.Source: §4.3 · p. 7, Table 2 · p. 8
- Increasing video-token counts improves performance across four task types, while Video-R1 drops on MMVU and Temporal-RLT maintains gains.Temporal-RLT benefits more than SFT from additional tokens.Source: §4.3 · pp. 7–8, Table 3 · p. 8
- More medium-difficulty VideoQA samples consistently improve benchmarks, while a greater share of hard samples may help reasoning but hurt general performance.The passage also reports gains from reducing easy samples while increasing medium or hard samples.Source: §4.3 · p. 8, Table 4 · p. 9
- Increasing temporal-grounding ΔIoU from 0 to 0.3 consistently improves grounding performance, supporting selection of higher-variance samples for GRPO.The ablation identifies ΔIoU as a proxy for intra-group prediction diversity.Source: §4.3 · p. 8, Table 4 · p. 9
- Using only Charades training data, GRPO gains +4.7 mIoU in-domain, +16.0 on ActivityNet, and +8.3 on ActivityNet-RTL versus SFT.The comparison evaluates both in-domain and out-of-domain benchmarks.Source: §4.3 · p. 8, Table 5 · p. 9
- After RLT, VideoQA outputs become less diverse, whereas temporal-grounding ΔIoU shifts toward zero and mean IoU shifts rightward.The authors interpret this as reduced intra-group variance and diminishing returns for VideoQA, while grounding retains room for improvement.Source: §4.3 · pp. 8–9
5 Related Work
Prior video-language work has developed structured datasets and techniques for temporal perception, but most approaches still rely on supervised fine-tuning. Reinforcement learning remains comparatively underexplored for video, particularly for temporally sensitive rewards and spatiotemporal consistency.
Video Large Language Models
- VideoLLMs progressed from captioning and question answering toward temporally structured datasets, frame alignment, and long-range dependency modeling.Sparse sampling and temporal token compression target temporal perception and efficiency.Source: §5 · p. 9
6 Conclusion
The conclusion presents GRPO-based post-training with dual rewards, variance-aware selection, and dynamic sampling as a framework for video-specific reasoning. It also identifies fine-grained thinking-trace supervision for SFT as an unexplored direction.
6 Conclusion
- Temporal-RLT combines GRPO, semantic and temporal rewards, variance-aware selection, and dynamic sampling for video-specific reasoning.The authors report improvements over SFT and RLT baselines across video-understanding tasks.Source: §6 · pp. 9–10
- The approach combines repeated inference with dynamic sampling to progressively refine training data, and experiments show significantly improved reasoning-centric video understanding performance versus existing SFT and RLT baselines.Figure 4 reports correctness-count distributions for MC-VideoQA and max–min IoU variance for temporal grounding; post-RLT data includes only samples selected from the original dataset by the sampling strategy.Source: §6 · p. 10, Figure 4 · p. 10
- The study does not explore fine-grained thinking-trace construction for SFT, which the authors leave for future research.Source: §6 · p. 10
A Data Selection Implementation Details
The appendix describes task-specific filtering of Temporal-RLT-Full-490k to construct a balanced 32k subset for GRPO training. Selection uses response consistency for discrete rewards and ΔIoU for continuous rewards, with a lower variance threshold for grounded QA.
A Data Selection Implementation Details
- The pipeline first separates data into discrete-reward, continuous-reward, and combined-reward tasks, then runs eight-sample No-Think inference with Qwen-VL-2.5-7B.Inference uses temperature 1.0 and top-p 0.99.Source: A Data Selection Implementation Details · p. 14
- For discrete-reward tasks, thresholds τeasy = 1 and τhard = 7 define difficulty categories, and 16k medium samples are selected across data sources.The medium-category choice follows the reported multi-choice ablation result.Source: A Data Selection Implementation Details · p. 14
- For continuous-reward tasks, the pipeline selects 8k samples with ΔIoU ≥0.3, distributing them evenly across data sources.Source: A Data Selection Implementation Details · p. 14
- Grounded QA applies both selection rules but lowers the ΔIoU threshold to 0.1 because this category has limited data, selecting 8k samples.The resulting filtered data contribute to the Temporal-RLT-32k subset.Source: A Data Selection Implementation Details · p. 14
B.1 Treating VideoQA as Grounded QA
Temporal-RLT can treat VideoQA as Grounded QA by changing the prompt format. The evaluated approach shows some performance degradation but lets the model specify relevant time intervals before answering.
B.1 Treating VideoQA as Grounded QA
- Changing the prompt format lets Temporal-RLT treat VideoQA as Grounded QA and specify relevant time intervals before answering.The passage describes this as an inherent capability of the model.Source: §B.1 · p. 14
- The evaluated VideoQA-as-Grounded-QA approach shows some performance degradation.The passage points to Figure 6 for this result but gives no numerical value.Source: §B.1 · p. 14
B.2 Visualization
The paper visualizes Temporal-RLT reasoning and final answers across Temporal Video Grounding, Video QA, and Grounded QA examples.
B.2 Visualization
- The visualizations include Temporal Video Grounding examples from Charades-STA, ActivityNet, and ActivityNet-RTL.The examples appear from top to bottom in that order.Source: §B.2 · p. 14, Figure 5 · p. 15
- Video QA reasoning-output examples come from VideoMME and MVBench.The examples appear in that order.Source: §B.2 · p. 14, Figure 6 · p. 16
- The Grounded QA visualization uses examples from the NextGQA test set.The visualized outputs include the reasoning process and final answers.Source: §B.2 · p. 14, Figure 7 · p. 16