TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs

Jun Zhang; Teng Wang; Yuying Ge; Yixiao Ge; Xinhao Li; Ying Shan; Limin Wang

Source-linked AI summary

TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs

Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, Limin Wang

arXiv:2512.14698v2Dec 16, 2025cs.CVcs.AIcs.CLcs.MM

TL;DR

MLLMs understand what happens in videos but remain weak at determining when events occur, while VTG recipes and benchmark reliability are under-explored. TimeLens systematically improves data and algorithmic design, producing models that lead open-source VTG and surpass reported proprietary-model baselines.

Problem
MLLMs largely fail at fine-grained video temporal grounding, and reliable recipes and dataset-quality assessments for VTG remain under-explored.
Method
TimeLens combines manually refined evaluation benchmarks, automated training-data re-annotation, and systematic studies of time encoding and thinking-free RLVR training.
Results
TimeLens models achieve state-of-the-art VTG performance among open-source models and surpass proprietary models including GPT-5 and Gemini-2.5-Flash.
Takeaways & Limitations
TimeLens provides quality-assured data and practical algorithmic best practices as a foundation for developing MLLMs with stronger VTG capabilities.

Abstract

from arXiv · show

This paper does not introduce a novel method but instead establishes a straightforward, incremental, yet essential baseline for video temporal grounding (VTG), a core capability in video understanding. While multimodal large language models (MLLMs) excel at various video understanding tasks, the recipes for optimizing them for VTG remain under-explored. In this paper, we present TimeLens, a systematic investigation into building MLLMs with strong VTG ability, along two primary dimensions: data quality and algorithmic design. We first expose critical quality issues in existing VTG benchmarks and introduce TimeLens-Bench, comprising meticulously re-annotated versions of three popular benchmarks with strict quality criteria. Our analysis reveals dramatic model re-rankings compared to legacy benchmarks, confirming the unreliability of prior evaluation standards. We also address noisy training data through an automated re-annotation pipeline, yielding TimeLens-100K, a large-scale, high-quality training dataset. Building on our data foundation, we conduct in-depth explorations of algorithmic design principles, yielding a series of meaningful insights and effective yet efficient practices. These include interleaved textual encoding for time representation, a thinking-free reinforcement learning with verifiable rewards (RLVR) approach as the training paradigm, and carefully designed recipes for RLVR training. These efforts culminate in TimeLens models, a family of MLLMs with state-of-the-art VTG performance among open-source models and even surpass proprietary models such as GPT-5 and Gemini-2.5-Flash. All codes, data, and models will be released to facilitate future research.

1. Introduction

TimeLens systematically investigates how data quality and algorithmic design can improve MLLMs for video temporal grounding. It introduces corrected evaluation and training data, derives practical design principles, and reports strong final models.

TimeLens studies data quality and algorithmic design, including time encoding, training recipes, and optimization strategies for time-aware MLLMs.The investigation is organized around the two dimensions shown in the framework overview.
TimeLens exposes unreliable VTG benchmarks, creates TimeLens-Bench from three manually re-annotated datasets, and produces TimeLens-100K through automated training-data re-annotation.The benchmark refinement uses strict criteria for query uniqueness, existence, clarity, and timestamp accuracy.
Interleaved textual time encoding outperforms more complex alternatives, while thinking-free RLVR provides better efficiency and performance than other training paradigms.The design analysis treats VTG as perception-driven and favors a pure thinking-free RLVR approach.
Early stopping after reward metrics plateau and difficulty-based data sampling are identified as effective RLVR training recipes.These recipes target both final performance and training efficiency.
TimeLens models achieve state-of-the-art VTG performance among open-source models and surpass proprietary models including GPT-5 and Gemini-2.5-Flash.The cumulative exploration analysis links successive gains from data curation, RLVR, early stopping, and difficulty-based sampling to the final models.
The paper presents its data, algorithmic insights, code, and models as a foundation for future research on MLLMs with strong VTG capabilities.The authors state that these resources will be open-sourced.

2. Related Work

Prior VTG work spans diverse datasets and architectural or training strategies, but inconsistent settings and overlooked data quality hinder reliable comparison. TimeLens addresses these gaps through systematic dataset inspection, refinement, and controlled analysis of design choices.

Existing VTG research uses single-domain benchmarks or aggregates diverse datasets for training and cross-domain evaluation, but dataset reliability has received limited systematic examination.The related-work discussion contrasts earlier single-benchmark studies with newer multi-source training and multi-benchmark evaluation.
TimeLens manually inspects existing datasets, corrects errors, and produces quality-improved training and evaluation suites for more practical VTG models.The paper frames data quality as an overlooked issue in temporal grounding datasets.
Prior MLLM work explores token compression, timestamp encoding, supervised fine-tuning, and verifiable-reward reinforcement learning for temporal grounding.These approaches cover both model architecture and training strategy.
Because existing studies use inconsistent experimental settings, their design choices are difficult to compare fairly; TimeLens evaluates them with quality-assured data suites.The paper aims to establish insights and best practices for improving MLLM VTG capability.

3. Towards Reliable, High-Quality VTG Data

TimeLens audits VTG data quality with explicit criteria, a diagnose-then-refine pipeline, and refined benchmarks and training data. The audit finds widespread annotation errors that distort evaluation, while re-annotated training data improves grounding performance.

3.1–3.2. Criteria and Refinement: The auditing pipeline checks query clarity, event existence, uniqueness, leakage, boundary precision, and annotation exhaustiveness before correction.Annotators diagnose errors, revise queries or events, and annotate precise temporal segments.
3.2. Manual Auditing and Refinement: The same annotator detects and corrects errors, while cross-validation and batch rejection provide additional quality control.Batches exceeding an error threshold are rejected for re-annotation and validation.
3.3. Empirical Analysis on TimeLens-Bench: TimeLens-Bench refines Charades-STA, ActivityNet Captions, and QVHighlights into a diverse, higher-quality evaluation suite.The suite is intended to provide more reliable evaluation for developing practical VTG models.
3.2. Manual Auditing and Refinement: 20.6% of Charades-STA samples violate query uniqueness and 34.9% have annotation-accuracy issues, illustrating consistently high error rates across benchmarks.Error categories vary by dataset, but all audited datasets show high overall error rates.
3.3. Empirical Analysis on TimeLens-Bench: Low-quality benchmarks inflate open-source model scores while underestimating proprietary models, with refined benchmarks reversing this ranking pattern.The original and refined evaluations produce drastically contrasting performance trends.
3.4. Training Data Re-annotation: Models trained on TimeLens-100K perform substantially better on refined benchmarks, validating the improved quality of the automatically re-annotated training data.The training-data re-annotation process was independent of manual benchmark refinement.

4. Benchmarking Grounding MLLMs

TimeLens benchmarks grounding MLLMs on quality-assured data while comparing timestamp representations. Its models achieve strong gains over open-source and proprietary baselines, and interleaved textual encoding with raw timestamps is the most effective simple design.

4.2. Timestamp Encoding: Timestamp encoding aligns each frame’s temporal position with its visual features so models can use absolute time and frame order for localization.The compared alternatives include position embeddings, visual overlays, and textual tokenization.
4.2. Timestamp Encoding: Interleaved textual prefix with raw timestamps is the most effective timestamp encoding approach while maintaining simplicity.The ablation compares textual, visual-overlay, and position-embedding-based representations across raw timestamps and frame indices.

5. Exploring Algorithmic Designs

TimeLens systematically studies algorithmic designs for MLLM-based VTG using TimeLens-Bench and TimeLens-100K, identifying effective timestamp encoding and RLVR training practices. These choices culminate in models that achieve state-of-the-art performance among open-source models and surpass GPT-5 and Gemini-2.5-Flash.

TimeLens combines systematic algorithmic studies with TimeLens-Bench evaluation and TimeLens-100K training to isolate design choices for VTG.Ablations use Qwen2.5-VL-7B, GRPO, and a single-choice-at-a-time experimental setup.
5.1. Timestamp Encoding: Interleaved textual timestamp encoding is the most effective strategy while retaining simplicity.
5.2. Training Paradigms: Thinking-free RLVR achieves superior VTG performance and efficiency, making SFT and thinking-based RLVR unnecessary.Its simpler formulation avoids explicit reasoning generation and format-reward engineering.
5.3. Recipes for RLVR Training: Early stopping when temporal IoU reward and within-group reward standard deviation plateau saves computation and prevents degradation.On approximately 12K samples, performance peaked at reward plateaus and declined with continued training.
5.3. Recipes for RLVR Training: Performance improves as sampled training difficulty rises, then plateaus when difficulty exceeds 0.75.The result supports selecting samples sufficiently difficult relative to the model.
The resulting TimeLens models achieve state-of-the-art VTG performance among open-source models and surpass GPT-5 and Gemini-2.5-Flash.

B. Details of TimeLens-Bench

TimeLens-Bench is a refined VTG evaluation suite built from three mainstream benchmarks. Its construction revises or removes problematic annotations and yields a compact, quality-improved benchmark.

TimeLens-Bench refines Charades-STA, ActivityNet Captions, and QVHighlights into Charades-TimeLens, ActivityNet-TimeLens, and QVHighlights-TimeLens.
The refinement process revises temporal segments for high-quality queries, rewrites low-quality queries, and discards unfixable annotations.
The resulting benchmark contains 4,279 videos averaging 107.8 seconds and 9,404 annotations.

B.1. Evaluation Metrics

TimeLens-Bench evaluates VTG with temporal overlap and thresholded retrieval metrics. Results are reported both jointly and separately across its three constituent benchmarks.

Evaluation uses mIoU and Recall@1 at IoU thresholds 0.3, 0.5, and 0.7.mIoU averages temporal IoU, while R1@m measures the fraction of predictions exceeding threshold m.
Metrics are reported separately for the three constituent benchmarks to support fine-grained cross-domain analysis.

C.1. Preliminaries for RL

The RL formulation samples multiple responses for each video-query pair and optimizes relative group advantage. TimeLens contrasts thinking-based and thinking-free reward designs, while difficulty-aware sampling targets challenging examples.

GRPO samples a group of responses for each video-query pair and optimizes their relative advantage.
Thinking-based RLVR generates explicit reasoning and an answer, combining grounding accuracy with format rewards.
Thinking-free RLVR directly generates the temporal answer and uses a simplified reward based solely on grounding accuracy.The paper reports that this paradigm simplifies implementation and improves efficiency and performance.
Difficulty-aware sampling estimates each sample’s difficulty from the model’s predicted segment, with higher values indicating more challenging samples.
Gaussian sampling weights examples around a target difficulty mean, correcting for the original dataset’s empirical difficulty density.Varying the target mean produces training sets with different average difficulty levels.

C.2. Experimental Setup

The experiments standardize video sampling, token budgets, timestamp encodings, and training settings while examining RL behavior across Qwen2.5-VL and Qwen3-VL.

Model Configuration: Videos are sampled at 2 FPS, with adaptive spatial resolution controlled by minimum per-frame and total-video token budgets.Ablations use 16 minimum tokens per merged frame pair and 3584 total tokens, while final TimeLens models use a larger resolution budget.
Timestamp Encoding: Interleaved textual encoding prepends tokenized timestamps to each frame’s visual tokens while preserving the computational cost of 2 FPS temporal encoding.Timestamps retain one decimal place, and consecutive video frames are merged during vision encoding.
Training Configuration: Training freezes the vision encoder, runs for one epoch, and uses distinct batch-size, learning-rate, and difficulty-aware sampling settings for SFT and RL.RL uses Gaussian difficulty-aware sampling with µ = 0.05 and σ = 0.2, batch size 8, eight roll-outs per prompt, and learning rate 1 × 10^-6.
Qwen3-VL: Direct RL training on Qwen3-VL fails to improve performance, likely because its prior multi-task RL training already includes VTG data and prevents rollout generation.The paper treats the resulting workaround as specific to Qwen3-VL rather than generally required for base MLLMs.

D. More Experimental Results

Additional experiments show that TimeLens generalizes across model sizes and an external benchmark, improves refined-benchmark comparisons, preserves general video understanding, and motivate perception-focused training choices.

Results across different model sizes: TimeLens consistently improves performance across model sizes, with TimeLens-3B substantially surpassing the larger Qwen2.5-VL-7B model.The result is reported across base models of varying sizes.
Generalization to an external benchmark: TimeLens-7B achieves the best result among compared models on the external VUE-TR benchmark, indicating generalization beyond TimeLens-Bench.VUE-TR is described as a concurrently introduced, high-quality human-annotated VTG benchmark.
Comparison of refined and original benchmarks: On refined benchmarks, proprietary models retain a significant advantage over open-source models, while TimeLens substantially narrows that gap.Original benchmarks instead misleadingly place open-source models above proprietary systems such as Gemini-2.5-Pro.
Results on general video understanding: TimeLens-7B maintains strong general video understanding on Video-MME while improving video temporal grounding.The benchmark is presented as a comprehensive, widely adopted general video understanding evaluation.
Thinking-free RLVR: The paper characterizes VTG queries as primarily perception-driven, supporting thinking-free RLVR over thinking-based RLVR in these experiments.The authors describe existing grounding tasks as mainly requiring localization intuition rather than complex reasoning.

H. Details of Curating TimeLens-100K

TimeLens-100K is constructed by sampling diverse VTG videos and using MLLMs to re-annotate events, queries, timestamps, and quality, producing a large high-quality training set.

Dataset construction: TimeLens-100K contains approximately 20K videos and 100K VTG annotations generated through automated re-annotation of existing training datasets.The training-data construction is independent of the manual benchmark-refinement process.
Video sampling: The pipeline samples from diverse VTG datasets and approximately uniformly covers video durations from 0–240 seconds, with some longer videos included.Source datasets include CosMo-Cap, InternVid-VTime, DiDeMo, QuerYD, and HiREST.
Automated re-annotation: MLLM re-annotation identifies distinct events across different time periods, describes them as queries, assigns timestamps, and verifies annotation quality.The pipeline addresses original annotations that lack clarity, specificity, or event existence.
Re-annotation model and prompting: Gemini-2.5-Pro performs re-annotation using a concise prompt refined through prompt engineering, because overly complex prompts can degrade annotation quality.The model is reported to verify event uniqueness and temporal distribution during reasoning.
Training-data validation: The improved training data substantially increases model performance, supporting the value of the curated training set.The comparison is designed independently of manual benchmark refinement for fairness.

I. Implementation Details for Benchmarking Existing MLLMs

The benchmarking setup adapts input formatting and sampling to each MLLM family while enforcing model-specific or shared settings for fair temporal-grounding comparisons.

Prompting: TimeLens training and evaluation use a dedicated prompt, while the paper documents separate prompts for GPT, Gemini, and other open-source models.The prompt figures cover training and evaluation for TimeLens and evaluation for the external model families.
GPT-5 and GPT-4o: GPT-5 and GPT-4o receive 1-FPS frame sequences with textual timestamps, while longer videos are compressed into sampled frames or 2 × 2 frame grids.The Azure OpenAI API limit of 50 images motivates the different strategies for videos longer than 50 seconds.
Gemini models: Gemini models are evaluated without audio, using 1-FPS sampling, default media resolution, and timestamp outputs in “MM:SS” format.Removing audio aligns evaluation with the vision-only, audio-free benchmark annotations.
Annotation procedure: The annotation guidelines require watching videos, confirming event occurrence, and recording or correcting event descriptions and start and end times.The detailed instructions specify watching the entire video in mute mode before identifying event segments.