PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding

Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, Xiangnan Fang, Zewen He, Zhenbo Luo, Wenxuan Wang, Junqi Lin, Jian Luan, Qin Jin

arXiv:2503.13377v3cs.CVcs.AIcs.CL

TL;DR

Temporal Video Grounding requires LVLMs to localize queried video segments, but their generalization remains limited and SFT can rigidly penalize reasonable timestamps. The paper introduces Time-R1 and TimeRFT for reinforcement-learning post-training with verifiable rewards, alongside TVGBench for evaluation. Time-R1 achieves state-of-the-art TVG performance with 2.5K training data and improves general video understanding, while remaining limited by speed, frame sampling, and ultra-long-video handling.

  • Problem

    LVLMs can underperform smaller feature-based TVG models despite larger pretraining, while SFT disproportionately penalizes reasonable timestamp predictions and limits generalization.

  • Method

    Time-R1 applies reinforcement-learning post-training with verifiable rewards, TimeRFT adds dynamic hard-sample training on a curated dataset, and TVGBench evaluates LVLMs with balanced coverage.

  • Results

    Time-R1 achieves state-of-the-art performance across TVG benchmarks using only 2.5K training data and improves general video understanding on video QA benchmarks.

  • Takeaways & Limitations

    Reinforcement-learning post-training can improve LVLM temporal grounding and transfer gains to short- and long-form video question answering.

  • Takeaways & Limitations

    Time-R1 has slower training and inference, may lose fine-grained motion information from low frame sampling, and cannot currently handle ultra-long videos.

Abstract

from arXiv · show

Temporal Video Grounding (TVG), the task of locating specific video segments based on language queries, is a core challenge in long-form video understanding. While recent Large Vision-Language Models (LVLMs) have shown early promise in tackling TVG through supervised fine-tuning (SFT), their abilities to generalize remain limited. To address this, we propose a novel post-training framework that enhances the generalization capabilities of LVLMs via reinforcement learning (RL). Specifically, our contributions span three key directions: (1) Time-R1: we introduce a reasoning-guided post-training framework via RL with verifiable reward to enhance the capabilities of LVLMs on the TVG task. (2) TimeRFT: we explore data-efficient post-training strategies on our curated RL-friendly dataset, which trains the model to progressively comprehend difficult samples, leading to better generalization. (3) TVGBench: we carefully construct a small yet comprehensive benchmark for LVLM evaluation, assessing 11 types of queries and featuring balanced distributions across both videos and queries. Extensive experiments demonstrate that Time-R1 achieves state-of-the-art performance across multiple downstream datasets using only 2.5K training data, while improving its general video understanding capabilities.

1 Introduction

Temporal Video Grounding localizes video segments from language queries, yet large vision-language models can underperform despite extensive pretraining. The paper addresses this with reinforcement-learning post-training, a data-efficient strategy, and a compact benchmark for evaluating generalization.

  • Temporal Video Grounding localizes segments in long-form videos corresponding to natural-language queries and supports applications such as video retrieval.
  • LVLMs with 7B+ parameters can underperform smaller feature-based models despite pretraining on datasets 100× larger than domain-specific benchmarks.Feature-based methods may suffer error accumulation from imperfect pretrained features, while LVLMs process videos and queries end to end.
  • Supervised fine-tuning can disproportionately penalize reasonable timestamp predictions, such as [1.9s, 3.9s] for ground truth [2s, 4s], encouraging overfitting and poor generalization.
  • Time-R1 uses reinforcement learning with verifiable rewards, generating chain-of-thought descriptions before timestamps and combining structured-template and timestamp-aware tIoU rewards.
  • TimeRFT progressively trains on difficult low-IoU samples using dynamic hard sampling, cold-start video-caption reasoning, and a curated dataset with difficulty annotations.
  • TVGBench is a compact benchmark with balanced video and query distributions and specific query semantics for evaluating LVLMs on TVG.The overview identifies 800 data items and 11 query types.
  • Using only 2.5K training data, Time-R1 outperforms prior SFT-based methods on TVG and improves performance on short- and long-form video question answering.It also surpasses many feature-based approaches after fine-tuning on downstream benchmarks such as Charades and ActivityNet.

2 Related Works

Prior TVG work largely follows feature-based video-language pretraining or frame-based LVLM approaches. Reinforcement learning is increasingly used for language and vision-language post-training, including human-feedback and verifiable-reward paradigms.

  • Feature-based TVG methods extract video and language features with pretrained encoders, then predict timestamps from multimodally fused features.Examples include I3D, EgoVLP, CLIP, and BERT.
  • Reinforcement learning updates models by interacting with environments and maximizing rewards, with RLHF and RLVR representing two major post-training categories.RLHF primarily targets human-preference alignment, safety, and hallucination reduction, whereas the passage identifies RLVR as verifiable-reward post-training.

3 Method

Time-R1 applies GRPO-based reinforcement learning to temporal video grounding, combining timestamp-aware rewards with structured reasoning and selectively curated training data. The method also introduces TVGBench, a compact benchmark designed for balanced, fine-grained evaluation of LVLMs.

  • Reward Modeling: The composite reward combines timestamp-aware IoU with a reasoning-template reward for structured video grounding outputs.The required response format is “<think>...</think> <answer><ts to te></answer>”.
  • Reward Modeling: Timestamp-aware IoU penalizes boundary deviations relative to video duration, reducing the standard IoU example from 0.5 to 0.28.This stricter signal is intended to better reflect temporal alignment than standard IoU in cases such as full-video ground truth.
  • Time-R1 Framework: Time-R1 trains LVLMs with GRPO, sampling candidate responses, assigning rewards, and updating the language model while freezing the visual encoder.The framework uses video frames and a language query as input, then optimizes reasoning and localization capabilities through GRPO.
  • TimeRFT Training: TimeRFT reduces RL training cost by assigning difficulty scores, selecting a small subset, and progressively filtering easy samples across multiple epochs.Easy samples with IoU greater than 0.7 are excluded after becoming easy, while difficult cases receive repeated training exposure.
  • TVGBench Evaluation: TVGBench contains 800 balanced instances spanning video duration, query centers, dataset sources, and 11 manually defined semantic categories.The categories cover human, object, and environment-related query aspects and support fine-grained evaluation across dimensions.

4 Experiments

The experiments evaluate Time-R1 across temporal video grounding and video understanding benchmarks, compare post-training paradigms, and analyze component contributions. Time-R1 achieves strong zero-shot grounding results with limited data, while RL preserves or improves generalization relative to SFT.

  • Experimental Setup: The evaluation covers Charades-STA, ActivityNet, TVGBench, MVBench, TempCompass, EgoSchema, and VideoMME, spanning temporal grounding and general video understanding.Charades-STA and ActivityNet provide TVG evaluation, while the remaining benchmarks assess video understanding across temporal and varied-length video tasks.
  • Experimental Setup: Time-R1 uses Qwen2.5-VL-7B as the default base model, samples frames at 2 FPS, and trains reinforcement fine-tuning for 5 epochs.Downstream fine-tuning uses 2 epochs, with experiments conducted on 8×NVIDIA A100 GPUs.
  • State-of-the-Art Comparison: With only 2.5K RL samples, Time-R1 achieves R1@0.7 scores of 35.3 on Charades-STA, 21.4 on ActivityNet, and 16.4 on TVGBench in zero-shot evaluation.These results outperform the cited SFT-based comparisons, including VideoChat-Flash and TimeSuite on Charades-STA and VTimeLLM and TimeChat on ActivityNet.
  • State-of-the-Art Comparison: Time-R1∗ exceeds EaTR and fine-tuned TimeSuite by 3.4-7.1 percentage points across Charades-STA R1@0.3 to R1@0.7.On ActivityNet, Time-R1∗ also outperforms SSRN and TRACE according to the reported comparison.
  • Ablation Study: Adding Gaussian Filtering, Multi-Epoch training, and Sample Filtering ultimately reaches R1@0.5 of 29.4 and R1@0.7 of 16.4.Multi-Epoch training improves R1@0.7 from 13.2 to 14.2, while combining tIoU supervision with Multi-Epoch training boosts all reported metrics.
  • Generalization Analysis: RL preserves generalization across TVG, short video QA, and long video QA, whereas full-parameter SFT significantly degrades it; on ActivityNet, RL raises mIoU from 16.3 to 29.2.SFT-LoRA improves ActivityNet mIoU to 25.9 but decreases VideoMME performance to 51.7, while RL increases VideoMME from 53.0 to 54.2.
  • Ablation Study: SFT-based cold start boosts the 3B model’s performance and reduces thinking-token counts for both evaluated models.The paper attributes this effect to cold start suppressing hallucinations, which are more prevalent in weaker models.

5 Conclusion

The paper presents Time-R1, TimeRFT, and TVGBench as a reinforcement-learning framework, training strategy, and benchmark for data-efficient and generalizable video-language understanding.

  • 5 Conclusion: Time-R1 uses verifiable rewards to improve LVLM temporal reasoning for temporal video grounding with limited data.The framework targets stronger generalization than large-scale supervised fine-tuning.
  • 5 Conclusion: TimeRFT combines a curated dataset with a training strategy intended to foster temporal grounding.The conclusion identifies it as a distinct contribution alongside Time-R1 and TVGBench.
  • 5 Conclusion: TVGBench is a small yet comprehensive benchmark for evaluating LVLMs on temporal video grounding.The benchmark is presented as part of the paper’s evaluation contribution.
  • 5 Conclusion: Extensive experiments report state-of-the-art performance across TVG benchmarks in zero-shot and fine-tuned settings, alongside improved general video understanding.The reported comparisons include prior LVLMs and traditional video-language pretraining models.

A Limitations

The paper identifies limits in speed, frame sampling, and video-length coverage that constrain Time-R1’s applicability.

  • A Limitations: Time-R1 has slower training and inference because of its large model size and reliance on autoregressive text generation.This limitation concerns both the training and deployment efficiency of the approach.
  • A Limitations: A relatively low frame sampling rate may lose fine-grained motion information across frames because of GPU memory constraints.The limitation follows from the system’s memory-management strategy.
  • A Limitations: Time-R1 cannot currently handle ultra-long videos, limiting applicability to scenarios such as full-length movie understanding.The stated scope boundary concerns video duration rather than temporal grounding performance on supported inputs.

B Implementation Details

The implementation uses GRPO- and DAPO-inspired training details, curated filtering and cold start procedures, standardized prompts, and established baseline configurations.

  • Time-R1 Framework: Time-R1 adopts DAPO’s token-level loss instead of GRPO’s sample-level loss while otherwise keeping GRPO settings identical.The authors report that other DAPO techniques did not benefit TVG and were omitted.
  • Time-R1 Framework: The GRPO configuration uses a sample number G of 8 and coefficient β of 0.04.The passage reports these as implementation settings for the training objective.
  • TimeRFT Training: TimeRFT filters data with a Gaussian distribution and uses 150 LoRA cold-start samples, with LoRA rank 64 and alpha 128.The filtering variance is fixed at 0.2 while the mean controls sample selection.
  • Evaluation Prompts: TVG prompts encourage reasoning before timestamp prediction through a template-based answer format, while VideoQA uses versions with and without CoT.The prompt design differs between temporal grounding and video question answering.
  • Baseline Implementations: Baseline evaluation on TVGBench uses each method’s original best-performing setting, focusing primarily on video input and prompt design.The cited baseline implementations include methods such as TimeChat, TRACE, TimeSuite, VideoChat-Flash, and Gemini-2.5-Pro.
  • TVGBench Evaluation: TVGBench comparisons across all semantic types use mIoU as the evaluation metric.The semantic categories include environment, human action and pose, object attributes, counting, existence, and transition.
  • SFT Baselines: SFT baselines include full-parameter fine-tuning and LoRA-based fine-tuning of the LLM.The LoRA configuration uses rank 64 and alpha 128.

C Additional Analyses

Time-R1 performs strongly across TVGBench semantic categories and improves inference efficiency through vLLM. Its strongest category-level results concern human-action understanding and object features, while complex instructions remain challenging.

  • TVGBench Analysis: Time-R1 achieves the highest scores in HAC, HAS, HP, and OA among compared methods.These categories cover complex human action, simple human action, human pose, and object attributes.
  • TVGBench Analysis: 56.2 mIoU on HP exceeds TRACE’s 45.1 by 11.1 points.On HAS, Time-R1 reaches 39.3 versus Gemini-2.5-Pro’s 30.9.
  • TVGBench Analysis: Time-R1 remains competitive with TRACE on ES, EC, and OT, and is in Gemini-2.5-Pro’s tier on HAP.Complex instructions remain difficult, with Gemini leading on HAP, OC, and OEC.
  • Inference Efficiency: 502 seconds on 800 TVGBench samples with vLLM and 8 GPUs compares with 2520 seconds using transformers.The accelerated implementation was used for the related downstream benchmarks.

D Ablation Studies

Ablations show that moderately difficult samples, tIoU rewards, sample filtering, and CoT contribute to training quality. The studies also examine KL regularization, token-level loss design, and model scaling.

  • Data Filtering: Gaussian filtering with standard deviation 0.3 performs best, outperforming uniform and random filtering.Moderately challenging samples generalize better than overly easy or extremely difficult examples.
  • KL and CoT: CoT training improves performance over No-CoT training, while the No-KL+CoT setting often fails to produce a thinking process.The adopted setup includes both KL and CoT to balance performance and interpretability.
  • Reward Design: tIoU consistently outperforms standard IoU across the first 5 training epochs.tIoU steadily improves, whereas IoU declines by the fifth epoch, supporting tIoU as a more stable reward.
  • Data Filtering: Sample filtering consistently improves performance across multi-epoch training by removing simpler samples.The analysis suggests high-IoU easy samples may add noise or reduce learning effectiveness.
  • Loss Design: DAPO uses token-level loss to address unequal token contributions arising from varying CoT lengths in GRPO.The design is intended to guide CoT generation across sampled reasoning chains of different lengths.
  • Model Scaling: Larger models achieve better zero-shot performance and continue outperforming smaller models after fine-tuning.The comparison supports a relationship between model capacity and TVG generalization.

E Qualitative Result

Qualitative cases show Time-R1 localizing events more accurately than baselines and improving video-question-answering reasoning across short and long videos. The paper also documents prompts and semantic annotations used in training and evaluation.

  • Temporal Video Grounding: Time-R1 more accurately localizes complex events on Charades and ActivityNet than the compared baselines.It captures “goes back to the pink bucket” and distinguishes sitting from lying while avoiding overly broad segments.
  • Short Video QA: Time-R1 improves over the base model on positional judgment, scene storyline reasoning, and visual reasoning in VideoMME and TempCompass.One case identifies a car missing its right-front wheel, which the base model misses.
  • Long Video QA: Time-R1 selects the correct answer in the “Hanging the Dress” example by generating a more accurate CoT than the base model.The case concerns summarizing task steps in a long egocentric video.
  • Prompts: Figure 11 presents prompts for temporal video grounding and video QA at both training and inference time.The TVG prompt requests reasoning with time ranges and an answer containing start and end times.
  • TVGBench Annotation: TVGBench query annotation classifies inputs into 11 predefined semantic categories using selected positive and negative examples.The examples include queries that are easy or difficult for the model to answer.
Loading 2503.13377v3…