Source-linked AI summary
TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs
Yuhan Zhu, Changlian Ma, Xiangyu Zeng, Xinhao Li, Zhiqiu Zhang, Songze Li, Jun Zhang, Tianxiang Jiang, Yuandong Yang, Ziang Yan, Zikang Wang, Xinyu Chen, Haoran Chen, Shaowei Zhang, Limin Wang
TL;DR
Temporal-grounding MLLMs must identify supporting moments, not merely describe video content. TimeLens2 treats evidence as interval sets during supervision and optimization, enabling compact models to outperform much larger open-source systems across seven benchmarks.
Problem
Existing video MLLMs rarely identify supporting moments, while long-video annotation and set-valued training remain unreliable for precise, potentially disjoint evidence intervals.
Method
TimeLens2 builds verified single- and multi-span supervision and optimizes interval-set predictions with a matching-free temporal Wasserstein reward.
Results
Across seven benchmarks, TimeLens2 compact 2B, 4B, and 8B models outperform size-matched baselines and much larger open-source models across diverse grounding settings.
Takeaways & Limitations
Treating evidence consistently as an interval set makes video MLLM outputs more traceable across long-video, multi-span, question-form, and egocentric grounding.
Takeaways & Limitations
The quality of TimeLens2-93K annotations is bounded by the capabilities of the models used in its construction.
Abstract
from arXiv · showhide
Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs. We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task: long-video labels often rely on brittle one-pass annotation, while reinforcement-learning rewards either fail to distinguish non-overlapping predictions or require fragile segment matching. TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. TimeLens2-93K constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. Our temporal Wasserstein reward computes exact one-dimensional \(W_1\) between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation; temporal IoU complements it with precise-overlap feedback. Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.
1 Introduction
TimeLens2 frames temporal grounding as a generalist, set-valued video task requiring verifiable evidence intervals across diverse videos and queries. It addresses supervision and optimization mismatches with staged evidence verification and a geometry-aware temporal Wasserstein reward, achieving strong performance across seven benchmarks.
- Motivation: Temporal grounding makes video MLLM outputs traceable by locating precise, potentially disjoint evidence rather than only describing what happened.Without temporal support, users must search the full timeline and cannot verify otherwise correct descriptions.
- Task scope: TimeLens2 studies unified localization of single- or multi-interval evidence across video lengths, domains, query forms, and viewpoints.Supporting evidence may be brief, recurrent, disjoint, or embedded in first- or third-person footage, with queries expressed as descriptions or questions.
- Challenges: One-pass long-video annotation can miss repeated evidence, confuse similar occurrences, and produce imprecise boundaries because relevant moments are sparse amid distractors.This reflects a supervision mismatch: temporal evidence is an interval set, but labels are often made through a single global decision.
- Optimization: The temporal Wasserstein reward provides graded, fragmentation-invariant, matching-free feedback for disjoint and unequal-cardinality predictions using exact one-dimensional W1 over merged interval support.It complements temporal IoU, which provides precise-overlap feedback but cannot distinguish how disjoint predictions should move.
- Supervision: TimeLens2-93K constructs reliable single- and multi-span supervision through hierarchical captions, independent relocalization, temporal consensus, semantic verification, and local boundary refinement.The pipeline replaces one brittle global decision with increasingly focused evidence checks.
- Results: 44.5, 47.7, and 48.0 average mIoU are achieved by TimeLens2-2B, TimeLens2-4B, and TimeLens2-8B across seven benchmarks, respectively.The variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 points, respectively.
2 Related Work
Related work has progressed from dedicated temporal-localization architectures and single-moment datasets toward generative MLLMs, broader evidence settings, and instruction-following recipes. Optimization remains misaligned because timestamp token likelihood and existing overlap rewards do not fully capture localized evidence quality.
- Temporal Grounding Models: Temporal grounding models have evolved from proposal matching and boundary regression to generative MLLMs with temporal instruction tuning, boundary-aware training, and timestamp-aware representations.The cited systems include TimeChat, VTimeLLM, LITA, and Grounded-VideoLLM.
- Temporal Grounding Data: Temporal grounding datasets have expanded from short, domain-specific videos with single moments to subtitle-aware, repeated-evidence, long-form, and egocentric search settings.MLLM recipes convert timestamped annotations into instruction-following conversations, often scaling through dense captions.
- Temporal Grounding Optimization: Supervised instruction tuning teaches timestamp syntax through token likelihood rather than localized-evidence quality, while GRPO/RLVR adds verifiable format and temporal-overlap rewards.Subsequent work explores reward design, data selection, refusal, curricula, and multi-segment reasoning.
3 TimeLens2
TimeLens2 formulates temporal grounding as prediction of variable-cardinality interval sets and constructs reliable long-video supervision through staged proposal, verification, consensus, and boundary-refinement procedures. It then separates long-context evidence-search training from interval-level geometric optimization.
- Interval-set formulation: TimeLens2 represents each example as a variable-cardinality set of supporting intervals, unifying single- and multi-span grounding while preserving search context.The representation is (v, q, Y), where Y contains one or more supporting intervals.
- Supervision pipeline: TimeLens2-93K separates candidate construction from label determination, using hierarchical captions, segment-aware query synthesis, independent local grounding, consensus, semantic verification, and boundary refinement.Caption-derived proposals are relocalized from video clips, unstable or mismatched instances are rejected, and surviving boundaries are locally refined.
- Corpus construction: 34,867 YouTube videos are stratified across five duration ranges and diverse visual domains to make search horizon and distractor load explicit design axes.The duration ranges span under one minute to over one hour.
- Supervision pipeline: IoUset(ŶQ, ŶT) > 0.9 and normalized text–video cosine similarity of at least 0.5 determine whether independently localized interval sets are retained.Qwen annotation is kept as canonical, while consensus tests reproducibility and embedding verification tests query–evidence relevance.
- Training strategy: Long-context supervised training teaches evidence search and output conventions, while reinforcement learning optimizes decoded interval sets against non-differentiable temporal objectives.Supervised examples primarily come from TimeLens2-93K and TimeLens-100K, with Ego4D-NLQ added for first-person video.
- Optimization motivation: MUSEG’s one-to-one matching is fragile: identical merged support can receive NGIoU scores of 1.0 and 0.32 solely because interval correspondence differs.Fragmentation can send a predicted interval to the wrong target, motivating matching-free treatment of multi-span geometry.
4 Experiments
Experiments across seven temporal-grounding benchmarks show that TimeLens2 improves performance across model scales, video lengths, query forms, and viewpoints. Ablations attribute these gains to curated supervision, long-context training, diverse interaction formats, and temporal Wasserstein rewards.
- Evaluation: TimeLens2 is evaluated on seven benchmarks using mIoU and Recall@1 at tIoU thresholds 0.3, 0.5, and 0.7.The main paper reports mIoU and R1@0.5, while R1@0.3 and R1@0.7 appear in supplementary Table 11.
- Temporal grounding: 44.5 average mIoU: TimeLens2-2B beats all three size-matched open-source baselines on all seven benchmarks and exceeds Qwen3-VL-2B by 14.2 points.It also exceeds TimeLens-8B by 2.4 points.
- Effect of TimeLens2-93K: The first 5% of TimeLens2-93K raises average mIoU from 34.7 to 42.8, while the full corpus reaches 45.8 and improves harder settings.On question-form MomentSeeker, mIoU rises from 15.3 to 25.8, supporting evidence-search learning beyond query-template imitation.
- Progressive label curation: 45.8 mIoU: refining the same 93.2K curated labels provides the largest curation-stage gain, adding 1.7 points without additional data.The full cascade improves over raw Qwen labels by 3.8 points overall, including 6.8 on ActivityNet and 8.7 on QVHighlights.
- Training ablations: 46.4 average mIoU: increasing packed context from 16K to 100K helps unevenly, with VUE-TR-V2, MomentSeeker, and Ego4D-NLQ gaining 4.0, 1.5, and 3.3 points.Rendering and request diversity together reach 45.8 (+0.9), performing best on six of seven benchmarks.
- Temporal reward ablations: 47.7 average mIoU: temporal Wasserstein outperforms matched NGIoU at 47.1 and improves zero-overlap learning by ranking otherwise silent training groups.RTW recovers positive overlap for 21.9% of near misses and reduces constant-reward groups from 13.8% to 3.6%.
5 Conclusion
TimeLens2 makes video MLLM answers traceable to supporting moments by improving evidence labels and the objective guiding imperfect interval predictions. It combines multi-stage evidence supervision with a dense, matching-free temporal reward.
- Conclusion: TimeLens2-93K replaces brittle global annotation with evidence proposal, independent localization, consensus, semantic verification, and boundary refinement.These stages target trustworthy evidence labels for traceable video answers.
- Conclusion: The temporal Wasserstein reward provides dense, matching-free geometry for imperfect interval predictions.It is designed to guide the objective underlying temporal evidence prediction.
A Supplementary Material · A.1 Instruction and Response-Format Diversity
This section motivates interface diversity for temporal grounding and describes factorized supervision that varies localization requests and response formats while preserving the video, query, and target interval set. Independent sampling creates 756 possible request–response pairings rendered deterministically from the same target intervals.
- A.1 Instruction and Response-Format Diversity: Temporal grounding requires both locating relevant evidence and expressing intervals through a specified interface.
- A.1 Instruction and Response-Format Diversity: Training with a single prompt–response convention can reduce robustness when request phrasing or output format changes.
- A.1 Instruction and Response-Format Diversity: Each QA turn keeps the video, synthesized query q, and target interval set Y fixed during interface sampling.
- A.1 Instruction and Response-Format Diversity: The method independently and uniformly samples one of 27 grounding requests and one of 28 response specifications.
- A.1 Instruction and Response-Format Diversity: The request pool varies how localization is posed, while the response pool controls answer syntax, timestamp encoding, and multi-interval composition.
- A.1 Instruction and Response-Format Diversity: 756 possible pairings result from independently sampling 27 grounding requests and 28 response specifications.A deterministic renderer expresses the same target interval set under each sampled specification.
A.2 Full Temporal Grounding Results
The full evaluation reports temporal grounding performance across multiple IoU thresholds, distinguishing coarse evidence retrieval from boundary precision. TimeLens2’s 4B and 8B variants achieve the highest reported R1@0.3 and R1@0.7 on all seven benchmarks.
- A.2 Full Temporal Grounding Results: TimeLens2’s 4B and 8B variants achieve the highest reported R1@0.3 and R1@0.7 on all seven benchmarks.The 2B variant also extends evaluation to a smaller scale, with average R1@0.3, R1@0.5, and R1@0.7 scores reported.
- A.2 Full Temporal Grounding Results: Table 11 reports full temporal grounding results across IoU thresholds, including R1@0.3 and R1@0.7 alongside the main paper’s R1@0.5 and mIoU.The thresholds assess progressively stricter overlap requirements, separating event retrieval from boundary precision.
A.3 Additional Qualitative Examples
Additional qualitative examples show that TimeLens2 consistently recovers intended evidence across seven benchmarks, while baselines select distractors, miss events, overextend boundaries, or fragment predictions. The examples test complementary temporal-reasoning challenges, including precise boundaries, conjunctive actions, role-conditioned binding, recurring evidence, sparse long-video search, and underspecified anomaly queries.
- Qualitative results: TimeLens2 variants consistently recover intended evidence across all seven benchmarks, whereas baselines select distractors, miss events, overextend boundaries, or fragment predictions.Each qualitative row pairs target-evidence frames and queries with full-video timelines showing ground truth and four model predictions.
- Temporal-reasoning challenges: The examples probe precise state-transition boundaries, conjunctive action coverage, role-conditioned entity binding, complete recurring-evidence retrieval, sparse long-video search, and underspecified anomaly mapping.Charades-TL, ActivityNet-TL, QVHighlights-TL, VUE-TR, VUE-TR-V2, and MomentSeeker provide these complementary challenges; VUE-TR-V2 includes a two-second relation nearly 20 minutes into a film.
A.4 Temporal Grounding Evaluation Protocol
The evaluation protocol covers seven benchmark subsets using VLMEvalKit and visual frames only, with standardized per-benchmark video inputs, documented model-specific deviations, and dataset-side prompts. Configurations are fixed by benchmark rather than tuned per model.
- Benchmarks and evaluation subsets: Seven benchmark subsets are evaluated with VLMEvalKit using visual frames only, without audio or subtitles.The subsets cover re-annotated Charades-STA, ActivityNet Captions, QVHighlights, VUE-TR, VUE-TR-V2, MomentSeeker, and Ego4D-NLQ-v2.
- Standard video input configuration: Per-benchmark input configurations are fixed rather than tuned per model.Video processing uses per-frame and aggregate pixel-area bounds, specified frame rates, and uniform frame capping when the frame limit is exceeded.
- Model-specific input settings: Model-specific deviations are documented for cases where models cannot accommodate the standard context or frame budget.All evaluated Qwen3-VL and Qwen3.5 variants, along with the listed baseline and TimeLens models, use standard per-benchmark settings unless Table 13 specifies otherwise.
- Prompt templates: Evaluation uses dataset-side prompt templates defined in Table 14.The evaluation entry point applies the prompts supplied by each dataset.
A.5 Train–Test Overlap Audit · A.6 Limitations and Future Directions
The appendix audits train–test identifier overlap and specifies evaluation configurations, then outlines limitations of the model-dependent data pipeline and future extensions toward long-context spatiotemporal grounding. The authors frame TimeLens2-93K as an effective first demonstration with substantial room for improvement and scaling.
- A.5 Train–Test Overlap Audit: Training-source YouTube IDs were canonicalized and compared against identifiers across all seven evaluation subsets.The audit removed ActivityNet prefixes, reduced QVHighlights clip names to source IDs, and used native IDs for VUE-TR, VUE-TR-V2, and identified MomentSeeker videos.
- A.5 Train–Test Overlap Audit: Table 12 defines the standard input configuration and exact arguments for all seven temporal grounding benchmarks.The passage identifies this table as the standard configuration reference for evaluation.
- A.5 Train–Test Overlap Audit: Table 13 records model-specific input overrides, including configurations that deviate from the standard settings.TL denotes the three TimeLens-Bench datasets.
- A.5 Train–Test Overlap Audit: Table 14 provides dataset-side prompt templates used in standard temporal-grounding evaluation, with {query} replaced by each benchmark query.The table specifies the exact templates used by the evaluation procedure.
- A.6 Limitations and Future Directions: Finite financial and computational budgets led TimeLens2-93K to rely on predominantly open-weight models for captioning, proposal generation, temporal annotation, and refinement.The pipeline used Qwen3-VL-235B-A22B-Instruct, Kimi-K2.5, Qwen3-VL-30B-A3B, and TimeLens-8B across these stages.
- A.6 Limitations and Future Directions: Stronger proprietary multimodal models, such as Gemini 3.1 Pro, could improve annotation richness, diversity, and temporal precision beyond the current dataset.The authors describe TimeLens2-93K as a first demonstration rather than the endpoint of the construction recipe.
- A.6 Limitations and Future Directions: The construction framework may extend to long-context spatiotemporal grounding that tracks when events occur, where entities appear, and how they evolve over time.The proposed direction is especially relevant to embodied intelligence and long-horizon agents connecting past observations with current scenes.