PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards One-to-Many Temporal Grounding

Qi Xu, Yue Tan, Shihao Chen, Jiahao Meng, Anna Wang, Shunping Ji, Hao Fei, Jason Li

arXiv:2606.06294v2cs.CVcs.AI

TL;DR

Most temporal grounding methods retrieve one segment, although queries may refer to multiple disjoint occurrences. This paper introduces OMTG benchmarks, data, and rewards, achieving state-of-the-art EtF1 of 43.65% on OMTG Bench.

  • Problem

    One-to-Many Temporal Grounding requires retrieving all disjoint video segments matching a query, beyond conventional one-to-one grounding.

  • Method

    The paper establishes OMTG evaluation metrics, curates 56k training samples, and trains with temporal and caption rewards.

  • Results

    43.65% EtF1 on OMTG Bench is state of the art, exceeding Gemini 2.5 Pro and Seed-1.8 by 15.85% and 15.61%, respectively.

  • Takeaways & Limitations

    The study establishes a strong baseline for the OMTG setting and supports further research on multi-segment temporal grounding.

  • Takeaways & Limitations

    The approach incurs high training costs and faces scalability challenges with extremely long videos.

Abstract

from arXiv · show

Temporal Grounding (TG) aims to localize video segments corresponding to a textual query. Prior research predominantly focuses on single-segment retrieval. Real-world scenarios, however, often require localizing multiple disjoint segments for a single query -- a setting we term One-to-Many Temporal Grounding (OMTG). Previous state-of-the-art MLLMs, optimized for one-to-one settings, struggle in this context, often yielding near-zero scores due to a lack of event cardinality perception. To bridge this gap, we present a systematic solution with three key contributions. First, we establish the first comprehensive OMTG benchmark, introducing Count Accuracy (C-Acc) and Effective Temporal F1 (EtF1) as evaluation metrics. Second, we curate a high-quality OMTG dataset comprising 56k samples through a sophisticated construction pipeline. Third, we develop novel temporal and caption reward functions specifically designed for OMTG. In particular, the caption reward leverages Chain-of-Thought reasoning over dense video captions to explicitly guide policy optimization toward both preciseness and completeness. Extensive experiments show our model achieves a new state-of-the-art EtF1 of 43.65\% on OMTG Bench, outperforming Gemini 2.5 Pro and Seed-1.8 by 15.85\% and 15.61\%, respectively. Project Page: https://insomniaaac.github.io/OMTG/

1. Introduction

The paper frames One-to-Many Temporal Grounding (OMTG) as retrieving all disjoint video segments matching a query, addressing limitations of one-to-one grounding with new benchmarks, data, training rewards, and a state-of-the-art model.

  • Problem: OMTG requires identifying all disjoint temporal segments corresponding to a single query, unlike conventional one-to-one temporal grounding.The problem arises because semantic actions can recur at multiple distinct intervals in dynamic, repetitive videos.
  • Evaluation: The proposed evaluation suite introduces Temporal F1-Score (tF1), Count Accuracy (C-Acc), and Effective Time F1 (EtF1) for precision-recall balance, event cardinality, and incomplete retrieval or hallucination penalties.These metrics address the limitations of one-to-one measures such as tIoU and R@1.
  • Benchmark: The authors establish the first comprehensive OMTG benchmark, where existing open-source models and traditional temporal-grounding experts often achieve near-zero EtF1 scores.Advanced proprietary models, including Gemini-series models and Seed-1.8, also show weak OMTG capability.
  • Method: The method constructs 56k high-quality training samples and combines Supervised Fine-Tuning with Reinforcement Learning using caption and temporal rewards.Caption rewards use dense video captions and Chain-of-Thought reasoning, while temporal rewards supervise temporal boundaries.
  • Results: 43.65% EtF1 on OMTG Bench establishes a new state of the art, surpassing Gemini 2.5 Pro by 15.85% and Seed-1.8 by 15.61%.The result exceeds both leading open-source and proprietary models.

2. Related Work

Related work spans MLLMs for video temporal grounding, shortcomings in existing benchmarks and datasets, and reinforcement learning for improving video MLLM reasoning. Existing resources predominantly use one-to-one formulations, motivating methods that better address recurring or overlapping events.

  • MLLMs for Video Temporal Grounding: MLLMs unify language, image, and video understanding within a single cross-modal reasoning framework, reshaping video temporal grounding.Earlier temporal grounding methods relied on visual encoders with task-specific heads.
  • Video Temporal Grounding Benchmarks and Datasets: Existing temporal grounding benchmarks contain annotation noise and rigidly impose one-to-one formulations that miss recurring or overlapping events.This limitation prevents benchmarks from capturing the broader event structures relevant to one-to-many grounding.
  • Video Temporal Grounding Benchmarks and Datasets: Current temporal grounding datasets rarely provide one-to-many supervision despite scaling through MLLM generation and large-scale data collection.The passage identifies insufficient one-to-many supervision as a limitation of existing training data.
  • Reinforcement Learning for Video MLLMs: Reinforcement learning improves MLLMs’ visual and cross-modal reasoning through verifiable or preference-based rewards.Recent work also applies reinforcement learning to video MLLMs to model spatio-temporal structure and long-range dependencies.

3. One-to-Many Temporal Grounding

One-to-Many Temporal Grounding localizes multiple disjoint video segments for repeated query occurrences and requires predictions to match ground-truth cardinality and temporal boundaries. The section introduces cardinality-aware evaluation through Count Accuracy and Effective Temporal F1, supported by bipartite matching and precision–recall measures.

  • Task Formulation: OMTG models a query as multiple semantic occurrences, generating structured temporal intervals that are deterministically parsed into predicted segments.The objective requires matching both the number of predicted and ground-truth segments and their temporal boundaries.
  • Evaluation Challenge: One-to-many evaluation must distinguish instance coverage from prediction correctness because high recall or tIoU can overlook redundant, hallucinated, or incorrectly merged segments.Merging two distinct events can yield deceptively high tIoU, such as 0.9, despite incorrect event cardinality.
  • Evaluation Metrics: The framework uses Hungarian bipartite matching at IoU threshold ξ to define instance-level Temporal Precision, Temporal Recall, and Temporal F1-Score.Temporal Precision penalizes redundant or hallucinated predictions, while Temporal Recall measures successful coverage of ground-truth instances.
  • Evaluation Metrics: Count Accuracy (C-Acc) measures the percentage of samples whose predicted segment count exactly matches the ground-truth count.It directly evaluates whether the model perceives the correct number of event occurrences.
  • Evaluation Metrics: Effective Temporal F1-Score (EtF1) gates Temporal F1 by count consistency, assigning zero to cardinality mismatches while coupling precision–recall with localization across IoU thresholds Ξ = {0.3, 0.5, 0.7}.This provides a holistic evaluation of temporal localization and event-count correctness.

4. Method

The method builds a 56k-sample OMTG dataset through automated discovery, grounding, verification, recall checking, query refinement, and dense captioning, then trains with SFT and GRPO. Its composite rewards jointly target temporal precision, event-count completeness, caption quality, and response conciseness.

  • Dataset Construction: The OMTG Dataset contains approximately 56k instruction-tuning samples sourced from diverse public video datasets.Sources include Cosmos-Cap, Moment-10M, and VTimeLLM.
  • Dataset Construction: The construction pipeline discovers repetitive events, grounds all occurrences, visually verifies segments, checks recall, refines queries, and generates query-guided dense captions.Visual verification uses all-or-nothing filtering, while refined queries guide captions that contextualize repetitive events within the full activity stream.
  • Training: The resulting dataset provides dense, verified annotations, split into 46k samples for SFT and 10k samples for RL.SFT integrates fine-grained temporal localization into chain-of-thought reasoning with dense video captions before final grounding predictions.
  • Reward Optimization: SFT initializes chain-of-thought grounding, while GRPO optimizes a composite reward to address the trade-off between retrieval completeness and localization precision.The reward balances temporal localization, counting completeness, caption quality, and response conciseness using λ1 = λ2 = λ3 = 0.5 and λ4 = −0.3.
  • Reward Optimization: The temporal reward combines RtIoU with RC-Acc, which activates only when predicted and ground-truth segment counts exactly match, correcting event-cardinality errors.RtIoU refines temporal boundaries, while RC-Acc imposes strict count matching.
  • Reward Optimization: The final reward combines temporal, counting, caption, and length objectives, achieving holistic alignment for accurate event counting and precise temporal localization.Caption rewards assess coverage, boundary precision, discriminability, and caption-guided timestamp localization; the length penalty suppresses distracting excess detail.

5. Experiment

Experiments evaluate OMTG and OOTG performance, showing that OMTG-4B achieves state-of-the-art one-to-many grounding while also improving conventional single-segment grounding. Ablations show SFT establishes OMTG ability, RL further improves it, and reward design benefits localization, counting, and EtF1.

  • Datasets and Metrics: OMTG evaluation uses OMTG Bench with C-Acc, tF1@0.3/0.5/0.7, EtF1, and tIoU, while OOTG evaluation uses refined TimeLens with R@1 and tIoU.The experiments cover both one-to-many and one-to-one temporal grounding settings.
  • Results on OMTG Task: 43.65 EtF1 is achieved by OMTG-4B, outperforming the best proprietary baselines by over 15.61%.Existing open-source models, traditional temporal-grounding experts, and advanced proprietary models show weak OMTG capability, with many near-zero EtF1 scores.
  • Results on OMTG Task: OMTG-4B accurately identifies all event occurrences with precise temporal boundaries, whereas existing models exhibit under-segmentation and over-segmentation.Qualitative comparisons show baselines retrieving a single segment or merging distinct segments into one continuous span.
  • Results on One-to-One Temporal Grounding: OMTG-4B consistently improves upon the base model and domain-specific baselines across all three TimeLens datasets.RL trained exclusively on OMTG data further improves over SFT across all OOTG benchmarks without one-to-one supervision.
  • Ablation on Training Strategy: EtF1 rises from 0.21 for the base model to 34.81 after SFT and 43.65 after RL, with RL adding +8.84 over SFT.The ablation indicates that SFT establishes foundational OMTG ability, while RL further strengthens it.
  • Ablation on RL Reward Design: RCaption improves C-Acc by 11.57 and EtF1 by 8.84 over the SFT baseline, achieving the best performance when combined with RtIoU and RC-Acc.RtIoU improves localization metrics, while RC-Acc improves counting accuracy and EtF1.

6. Conclusion

The paper formalizes One-to-Many Temporal Grounding (OMTG), showing that existing MLLMs struggle with event cardinality and disjoint-segment localization, and introduces data and reward-based training that establishes a strong baseline. The approach still faces high training costs and scalability challenges on extremely long videos.

  • The paper formalizes One-to-Many Temporal Grounding (OMTG) to address the gap between one-to-one paradigms and dynamic real-world scenarios.
  • Existing state-of-the-art MLLMs struggle to perceive event cardinality and localize disjoint segments.This difficulty persists despite their success in standard settings.
  • 56k high-quality one-to-many training samples support SFT+RL training with temporal and caption rewards, achieving state-of-the-art results.The samples are curated through a sophisticated data pipeline.
  • The study establishes a strong baseline for the novel OMTG setting and facilitates future research.
  • Limitations and Future Works: The current approach incurs high training costs and faces scalability challenges with extremely long videos.Future work will explore OMTG with memory in long-video settings.

Impact Statement · Appendix Overview

The paper highlights potential benefits and privacy risks of precise One-to-Many Temporal Grounding, while the appendix documents training, rewards, benchmarks, evaluations, statistics, failures, and annotation procedures.

  • Impact Statement: The framework could enhance video search efficiency, automate video editing workflows, and improve content accessibility, but precise localization may be misused for surveillance or privacy intrusion.These societal impacts and risks are identified in the paper’s impact statement.
  • Appendix Overview: Section A provides additional details about the training-data pipeline construction process.
  • Appendix Overview: Section B elaborates on the designs of the reward functions.
  • Appendix Overview: Sections C and D report results on Video MME and across different model sizes, respectively.
  • Appendix Overview: Section E details the OMTG benchmarking process, while Section F presents zero-shot OOD evaluation on longer in-the-wild videos.
  • Appendix Overview: Section G analyzes statistical details, and Section H examines failure cases.
  • Appendix Overview: Section I describes the OMTG annotation process and the human-check process.

A. More Details of Training Data Pipeline … B.2. Length Penalty

The paper details a staged OMTG data pipeline with strict verification and adaptive refinement, then specifies caption rewards and soft penalties to promote informative, concise outputs. These mechanisms target semantic consistency, grounding quality, and resistance to verbosity.

  • A. More Details of Training Data Pipeline: The training-data appendix provides prompt templates for five pipeline stages: repetitive-event discovery, initial grounding, visual verification, recall checking with query refinement, and query-guided dense captioning.The templates are listed in Tables 5–9.
  • A.2. Quality Control: Stage 2 outputs are filtered by Qwen3-VL-235B, which verifies every predicted segment against the query and discards the entire sample if any segment is irrelevant.This strict visual check follows initial grounding and uses a one-vote veto principle.
  • Theoretical Proof of Quality Gain: The quality-gain analysis models verifier filtering with mismatch prior θ and error rate p, showing lift L(N) = β^N grows exponentially with segment count N.Under the stated conditions, β > 1, so visual checking becomes more effective for complex samples.
  • Theoretical Proof of Quality Gain: Samples with N ≥4 are accepted directly, whereas N = 2, 3 samples undergo recall checking and query refinement to further improve data quality.The paper states that experiments in Tab. 10 validate this strategy.
  • B. More Details of Reward Functions Design: The caption reward uses Qwen3-30B-A3B in parallel LLM-as-Judge evaluations combining caption quality with caption-guided grounding.The two components assess complementary aspects of generated captions.
  • B.1. Caption Reward Prompts: Caption Quality Score combines coverage, precision, and discriminability as Scq = µ1 · Scov + µ2 · Sprec + µ3 · Sdisc, with µ1 = 0.5, µ2 = 0.3, and µ3 = 0.2.The weighting emphasizes coverage completeness.
  • B.1. Caption Reward Prompts: Caption-guided grounding evaluates text-only event localization from queries and captions, comparing predicted intervals with ground truth using tF1 at IoU thresholds 0.3 and 0.5.This ensures captions are semantically informative rather than merely temporally co-occurring.
  • B.2. Length Penalty: The length penalty softly penalizes responses beyond thresholds because irrelevant details can dilute temporal cues and degrade localization; it separately penalizes thinking content and caption length.The final penalty combines the thinking-content and average-caption components, using character counts and soft and hard thresholds.

B.3. Temporal Rewards Design Choices

The temporal-reward ablation compares RtIoU, RtF1, and RC-Acc against the SFT baseline. Results show cardinality supervision is essential, while adding redundant dense temporal rewards can reduce performance.

  • Temporal Rewards Design Choices: The ablation evaluates combinations of RtIoU, RtF1, and RC-Acc using performance gains over the SFT baseline.Reward definitions follow the metrics in Section 3.2.
  • Critical Role of Cardinality Supervision: +0.31 C-Acc is the negligible Count Accuracy improvement from using only the boundary-aware RtIoU reward.RtIoU refines local boundaries but does not correct the number of predicted segments, such as by preventing event merging or splitting.
  • Critical Role of Cardinality Supervision: Adding cardinality-aware RC-Acc to RtIoU produces a substantial performance leap over using RtIoU alone.The RtIoU + RC-Acc combination improves event-count supervision beyond boundary refinement alone.
  • Redundancy in Dense Temporal Rewards: The full RtIoU + RtF1 + RC-Acc combination degrades performance relative to the simpler RtIoU + RC-Acc setting.RtIoU and RtF1 may provide overlapping localization supervision, diluting the gradient signal from sparse cardinality rewards.

C. More Results on Video MME · D. Performances across Different Model Sizes

The VideoMME evaluation shows that CoT and caption-reward reinforcement learning mitigate the general-video-understanding costs of OMTG specialization. The paper also examines how model capacity affects OMTG Bench performance.

  • C. More Results on Video MME: 62.1 versus 66.7 Overall: naive SFT without CoT underperforms the Qwen3-VL backbone on VideoMME.Evaluation uses the no-subtitle setting with 128 sampled frames per video.
  • C. More Results on Video MME: +2.3% Overall: adding Chain-of-Thought data during SFT recovers general performance, matching the backbone at 77.6 on Short videos.The passage attributes this recovery to enhanced reasoning capabilities.
  • C. More Results on Video MME: 65.1 Overall: reinforcement learning with Caption Reward further improves VideoMME performance and nearly closes the gap with the backbone.The passage identifies Caption Reward as crucial for preserving semantic representations during grounding optimization.
  • C. More Results on Video MME: The final OMTG-4B model is characterized as a temporal-grounding specialist that remains a robust generalist in video understanding.This summarizes the reported outcome of the specialized training strategy.
  • C. More Results on Video MME: RtIoU + RC-Acc provides the best balance between localization and cardinality among the evaluated temporal-reward combinations.The ablation results are reported as absolute improvements over the SFT baseline in Row 1.
  • D. Performances across Different Model Sizes: The paper analyzes how model capacity affects performance on OMTG Bench across different model sizes.The analysis is presented with results in Table 15.

E. Implementation Details for OMTG Benchmarking · F. In-the-Wild Generalization

The paper standardizes OMTG benchmarking across proprietary and open-source MLLMs, then evaluates zero-shot generalization on a diverse, out-of-domain video set. The proposed model reportedly outperforms baselines across all metrics on this challenging in-the-wild evaluation.

  • E. Implementation Details for OMTG Benchmarking: The evaluation suite specifies implementation details for comparing existing MLLMs on OMTG, with results reported in Table 1.
  • E. Implementation Details for OMTG Benchmarking: Gemini 2.5 Pro and Gemini 3 Pro used the official Video Understanding API with visual and audio inputs, while Seed-1.8 used Volcano Engine with complete videos sampled at 2.0 FPS.
  • E. Implementation Details for OMTG Benchmarking: OMTG-4B and Qwen models used sglang, other open-source baselines used transformers, and all open-source models shared fps=2, min_pixels=2048, and total_pixels=8388608.
  • E. Implementation Details for OMTG Benchmarking: Evaluation prompts were standardized for reproducibility, using a unified template for open-source models and Seed-1.8 and a specific template for the Gemini series.
  • F. In-the-Wild Generalization: The OOD test set contains 60 human-verified samples across 52 recent Bilibili and YouTube videos spanning travel, gaming, sports, news, and anime.
  • F. In-the-Wild Generalization: The videos are entirely outside training sources, average 422.87s in duration, reach 1419.93s maximum length, and support challenging zero-shot evaluation despite limited sample size.
  • F. In-the-Wild Generalization: The proposed model demonstrates strong generalization to longer, truly in-the-wild videos and significantly outperforms baselines across all reported metrics.

G. Statistics Details of OMTG Benchmark · H. Failure Cases Study · I. Annotation Interface and Manual Check For OMTG Benchmark

The benchmark statistics characterize segment-count and video-duration distributions, while failure analysis identifies temporal misalignment under ambiguous action semantics. A custom annotation workflow and strict quality criteria target complete, precise, human-clear, and high-cardinality OMTG ground truth.

  • G. Statistics Details of OMTG Benchmark: Benchmark statistics summarize the distribution of ground-truth temporal-segment counts and video durations.
  • H. Failure Cases Study: On ambiguous or visually subtle actions, models can exhibit complete temporal misalignment by confusing transitional motions with sustained states.For “moves the fridge door,” predictions may default to coarse segmentation spanning the interaction.
  • H. Failure Cases Study: Models may prioritize scene context over fine-grained motion semantics, diluting precise boundaries or drifting from the actual motion event.
  • I. Annotation Interface and Manual Check For OMTG Benchmark: The benchmark uses a lightweight local Python web application that lets annotators label multiple non-contiguous segments for one textual query.The tool supports precise labeling of disjoint temporal segments in multi-event video retrieval.
  • I. Annotation Interface and Manual Check For OMTG Benchmark: Annotators review each video, mark every described-event occurrence, revise inaccurate queries, and add queries for distinct uncovered events.Annotations are automatically saved locally and collected for quality verification after batch completion.
  • I. Annotation Interface and Manual Check For OMTG Benchmark: Quality control requires complete coverage, tight temporal boundaries, and conservative windows that exclude ambiguous transition frames.Missing segments are critical errors, and boundaries should include only clearly visible action frames.
  • I. Annotation Interface and Manual Check For OMTG Benchmark: Videos with low resolution, heavy occlusion, or unclear events are discarded so ground truth rests on clear human perception.
  • I. Annotation Interface and Manual Check For OMTG Benchmark: Annotators preferentially retain high-cardinality queries corresponding to multiple time segments rather than single-occurrence events.This selection addresses existing datasets’ dominance of single-segment events.
Loading 2606.06294v2…