PaperlayerView exact revision on arXiv

Source-linked AI summary

Factorized Learning for Temporally Grounded Video-Language Models

Wenzheng Zeng, Difei Gao, Mike Zheng Shou, Hwee Tou Ng

arXiv:2512.24097v1cs.CVcs.AIcs.CLcs.MM

TL;DR

Video LLMs still struggle with accurate temporal grounding, which is also important for text answers based on grounded evidence. The paper proposes D2VLM and factorized preference optimization with synthetic preference data. Experiments report that its 3.8B model outperforms existing SOTA methods across various tasks.

  • Problem

    Video LLMs still struggle with accurate temporal grounding for event perception and localization, a capability relevant to grounding tasks and textual answering.

    Source: §1 · p. 2

  • Method

    D2VLM grounds temporal evidence before generating evidence-referencing answers, while FPO and synthetic preference data support learning both grounding and textual responses.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The 3.8B model outperforms existing SOTA methods, typically 3.8B–13B, across various tasks, with at least 7.0% average F1 score improvement on E.T. Bench grounding.

    Source: §1 · p. 2, §6.2 · p. 7

  • Takeaways & Limitations

    Conditioning textual generation on correctly grounded evidence fits teacher-forcing autoregressive training and provides a learning shortcut for more stable training.

    Source: §1 · p. 2

    The factorized data generation focuses only on negative samples; generating diverse positive samples to enrich preference data remains future work.

    Source: §7 · p. 8

Abstract

from arXiv · show

Recent video-language models have shown great potential for video understanding, but still struggle with accurate temporal grounding for event-level perception. We observe that two main factors in video understanding (i.e., temporal grounding and textual response) form a logical hierarchy: accurate temporal evidence grounding lays the foundation for reliable textual response. However, existing works typically handle these two tasks in a coupled manner without a clear logical structure, leading to sub-optimal objectives. We address this from a factorized learning perspective. We first propose D$^2$VLM, a framework that decouples the learning of these two tasks while also emphasizing their inherent dependency. We adopt a "grounding then answering with evidence referencing" paradigm and introduce evidence tokens for evidence grounding, which emphasize event-level visual semantic capture beyond the focus on timestamp representation in existing works. To further facilitate the learning of these two tasks, we introduce a novel factorized preference optimization (FPO) algorithm. Unlike standard preference optimization, FPO explicitly incorporates probabilistic temporal grounding modeling into the optimization objective, enabling preference learning for both temporal grounding and textual response. We also construct a synthetic dataset to address the lack of suitable datasets for factorized preference learning with explicit temporal grounding. Experiments on various tasks demonstrate the clear advantage of our approach. Our source code is available at https://github.com/nusnlp/d2vlm.

1. Introduction

Video-language models struggle with temporal grounding, while existing approaches couple grounding and answering and underrepresent grounded-event semantics. The paper proposes a factorized framework and preference-learning approach, reporting advantages across tasks.

  • Existing methods couple temporal grounding with textual response, while timestamp-focused tokens lack explicit capture of grounded events’ visual semantics.Accurate grounding matters both for grounding tasks and for textual answers based on grounded evidence.Source: §1 · p. 2
  • D2VLM separates pure temporal evidence grounding from subsequent interleaved text-evidence answer generation, retaining their dependency through evidence referencing.The answer stage produces text and temporal information linked to previously grounded evidence.Source: §1 · p. 2
  • Evidence tokens represent event locations while capturing event-level visual semantics as context for subsequent answer generation.In the answer stage, evidence tokens align with tokens from the grounding stage to maintain consistency and logical coherence.Source: §1 · p. 2
  • Conditioning textual response on correctly grounded evidence fits teacher-forcing autoregressive training and provides a learning shortcut for more stable training.Experiments identify both the sequence-generation objective and event-level visual semantic capture as essential for performance improvement.Source: §1 · p. 2
  • FPO incorporates probabilistic temporal grounding into preference optimization, and a synthetic dataset uses factorized sub-video-event perturbations to create preference data without manual annotation.The perturbations address temporal grounding and textual response, with known causes and noise types.Source: §1 · p. 2
  • The 3.8B model outperforms existing SOTA methods, typically ranging from 3.8B to 13B in model size, across various tasks, as shown by an intuitive comparison in Fig. 1(a).Source: §1 · p. 2

2. Related Work

The related work identifies limitations in temporal grounding by video LLMs and notes that explicit temporal grounding remains underexplored in video preference optimization. The paper responds with factorized learning and preference optimization.

2.1. Video LLMs

  • Video LLMs still struggle with temporal event perception and localization [31] [39]; timestamp tokens couple grounding with answering and lack grounded-event visual semantics.The related-work discussion also notes targeted data and special-token approaches to temporal representation [20] [31] [37] [43] [29] [36].Source: §2.1 · pp. 2–3
  • The paper addresses these issues by factorizing temporal evidence grounding and textual response while strengthening their dependency.It also proposes factorized preference learning for the two tasks.Source: §2.1 · p. 3

2.2. Preference Optimization

  • Explicit temporal grounding remains underexplored in video LLM preference optimization; FPO models grounding probabilities to learn preferences for grounding and textual responses.The paper also constructs a synthetic dataset for preference learning with explicit temporal grounding.Source: §2.2 · p. 3

3. Method

D2VLM separates temporal evidence grounding from textual answering while retaining their dependency. It grounds events first, then generates answers that reference evidence through <evi> tokens.

3.1. Preliminary

  • The preliminary formulation encodes video and question inputs as tokens that feed a video-aware LLM decoder for autoregressive response generation.For illustration, the formulation sets one frame-level visual token per frame.Source: §3.1 · p. 3

3.2. D2VLM

  • The method targets coupled grounding and answering objectives and timestamp-focused tokens that do not explicitly capture grounded-event visual semantics.The paper identifies event-level visual semantics as useful context for subsequent answer generation.Source: §3.2 · p. 3
  • D2VLM first performs pure temporal grounding, then generates the answer in an interleaved text-evidence manner.Grounded events are intended to serve as evidence for the subsequent response.Source: §3.2 · pp. 3–4
  • The <evi> token identifies an event’s temporal location and captures its visual semantics by pooling salient frame-token features into the token.Salient tokens are selected by similarity to the <evi> token.Source: §3.2 · p. 4
  • During answer generation, <evi> tokens reference temporal evidence, with timestamps obtained from salient frame indices; </evi> marks the transition from grounding to answering.The response combines text with temporal grounding information.Source: §3.2 · p. 4
  • A consistency constraint aligns corresponding <evi> tokens across the grounding and response stages, whose tokens are generated in temporal order for distinct ground-truth events.The paper says this constraint preserves logical coherence and strengthens alignment between grounding and answering.Source: §3.2 · p. 4
  • Training combines standard token-classification loss with average grounding loss for generated <evi> tokens across both stages.The grounding loss uses frame-level foreground/background labels and binary cross-entropy on normalized similarities.Source: §3.2 · p. 4

4. Factorized Preference Optimization

Factorized Preference Optimization (FPO) extends preference learning to temporal grounding as well as textual responses. It models grounding probability explicitly from frame-level similarities, addressing the challenge of incorporating grounding into preference objectives.

4. Factorized Preference Optimization

  • FPO incorporates probabilistic temporal grounding modeling into preference optimization, enabling explicit preference learning for grounding alongside textual responses.Its log-probability combines standard textual-token prediction with explicit temporal event-grounding terms.Source: §4 · pp. 4–5
  • The preference objective favors a preferred response over a dispreferred one using their joint generation probabilities.Under autoregressive generation, each response probability is expressed through its token probabilities.Source: §4 · p. 4
  • The approach addresses two obstacles: missing factorized grounding-aware preference data and difficulty quantifying grounding likelihood for preference optimization.The latter is challenging because many grounding methods depart from discrete text-token prediction.Source: §4 · p. 5
  • For each evidence token, frame-level feature similarity supplies grounding probability, and those probabilities are multiplied to obtain interval-level probability.The formulation uses similarity within the interval and one minus similarity outside it.Source: §4 · p. 5

5. Factorized Preference Data Synthesis

The paper synthesizes factorized preference pairs by perturbing selected sub-video events in either temporal grounding or textual response. This gives the dispreferred responses specific, controllable noise rather than arbitrary degradation.

5. Factorized Preference Data Synthesis

  • Existing video preference data often lacks controlled factorization, and its dispreferred responses may be as good as or better than preferred responses.Such data also largely uses pure-text responses, leaving explicit temporal grounding in preference responses underexplored.Source: §5 · pp. 5–6
  • The synthesis framework selects temporal-grounding or textual-response perturbations and applies them to selected sub-video events.Grounding perturbations include shifting, adding or deleting events, and merging events; textual perturbations include distorting key information or repetition.Source: §5 · p. 6
  • Event-level perturbations provide localized, fine-grained modifications that respect temporal and semantic coherence rather than applying coarse noise to an entire response.The paper states that the resulting dispreferred responses contain specific, well-factorized noise whose source can be identified without manual costs.Source: §5 · p. 6
  • Preference pairs are formed by combining a newly perturbed noisy response with the original response, using E.T. Instruct 164K as the synthesis source.Perturbations can shift or delete timestamps, or use an LLM to distort event-level text.Source: §5 · p. 6

6. Experiments

Experiments evaluate D2VLM on E.T. Bench, Charades-STA, and YouCook2, then ablate its generation objective, evidence-token design, and FPO. Results show gains across grounding and textual description tasks.

6.1. Experimental Setup

  • Evaluation covers E.T. Bench grounding and dense captioning, Charades-STA, and YouCook2; implementation uses a Phi-3-Mini-3.8B base and LoRA fine-tuning on E.T. Instruct 164K.Training used four NVIDIA H100 GPUs for one day.Source: §6.1 · p. 6

6.2. Main Benchmarking Results

  • D2VLM improves E.T. Bench grounding by at least 7.0% average F1, and dense-captioning grounding and text scores by at least 5.3% average F1 and 3.6% average Sim.The results span five grounding subtasks and two dense-captioning subtasks.Source: Table 1 · p. 7, §6.2 · p. 7
  • On Charades-STA, D2VLM gains 4.4% R@1(IoU=0.5) over the same-base counterpart and at least 1.6% over 7B methods; on YouCook2, gains are at least 4% F1, 2.5% CIDEr, and 1.0% SODA c.Source: §6.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7

6.3. Ablation Studies

  • Decomposing generation into grounding then answering improves grounding and textual description, while interleaved text-evidence generation and a consistency constraint add further gains.Reported gains include 7.7% and 8.8% average F1 on grounding and dense-captioning tasks, respectively, and at least 6.7% grounding and 3.8% textual-description improvement from interleaving.Source: §6.3 · p. 8, Table 4 · p. 8
  • Event-level modeling and explicit visual-semantic capture in evidence tokens both contribute to performance, with semantic capture especially beneficial for dense captioning.Event-level modeling represents an interval rather than an individual frame; visual semantics provide context for answer generation.Source: §6.3 · p. 8, Table 6 · p. 8
  • FPO consistently improves temporal grounding and captioning-related tasks, with larger improvements in temporal grounding.Source: §6.3 · p. 8, Table 5 · p. 8

7. Conclusion and Limitations

The conclusion presents D2VLM’s factorized grounding-and-answering approach, evidence tokens, and FPO, alongside experiments showing broad superiority. The authors note remaining weak task scores and a limitation in their preference-data synthesis.

  • D2VLM separates grounding from answer generation while referencing evidence, and uses evidence tokens for event-level visual semantics alongside factorized preference learning and synthetic data.Source: §7 · p. 8
  • Extensive experiments show the approach is superior across various scenarios.Source: §7 · p. 8
  • The approach scores only 14.4% F1 on episodic memory and 26.4% on YouCook2 dense video captioning; its synthetic preference data generates negative, but not diverse positive, samples.Source: §7 · p. 8

A. More Implementation Details

Implementation uses sampled video frames and evidence-token similarity for grounding and visual-semantic capture. The annotation formats distinguish task types and specify a two-stage response with shared evidence supervision.

A. More Implementation Details

  • Frames are sampled at 1 FPS and resized to 224 × 224; inference includes frame tokens whose similarity to an evidence token exceeds 60% of its maximum frame similarity.Ground-truth interval frames are treated as salient during training.Source: A. More Implementation Details · p. 12
  • A projected evidence token serves both as an autoregressive generation token and as a query for similarity-based grounding and visual-semantic aggregation.Source: A. More Implementation Details · p. 12

B. Data Annotation Formats

  • The annotation formats distinguish grounding-focused tasks, dense captioning, and temporally grounded question answering, which pairs answers with temporal evidence.Source: B. Data Annotation Formats · p. 12
  • Responses contain pure evidence grounding followed by interleaved text-evidence generation, separated by </evi>; the same ground-truth event positions supervise both stages.The “time gt” key supplies the temporal event position used to supervise evidence-token and frame-token similarity.Source: B. Data Annotation Formats · p. 12

C.1. Performance on E.T. Bench Complex Dataset

The appendix reports results on E.T. Bench Complex and general video QA, and measures the cost of frame-wise similarity computation. D2VLM is reported to outperform existing methods on the complex benchmark and its grounding-focused QA counterpart.

C.1. Performance on E.T. Bench Complex Dataset

  • On E.T. Bench Complex, covering temporal event matching and grounded video question answering, D2VLM outperforms existing state-of-the-art methods by large margins.Source: §C.1 · pp. 12–13

C.2. Extension to General QA Tasks

  • For general video QA, D2VLM outperforms grounding-focused E.T. Chat and is comparable to some broadly trained video-understanding models; the authors attribute remaining gaps to absent large-scale generic pretraining and smaller model size.Source: §C.2 · p. 13

C.3. Cost of Frame-Wise Similarity Calculation

  • Frame-wise similarity computation takes less than 0.4 ms per generated token on one 3090 GPU, or 1.4% of the 29 ms network-forwarding time.Source: §C.3 · p. 13

C.4. Sensitivity Analysis on Similarity Threshold

Performance is robust across salient-frame thresholds, with 0.6 performing best; thresholds that are too high lose information, while thresholds that are too low add less relevant context.

  • 0.6 achieves the best performance among the tested salient-frame identification thresholds.Source: §C.4 · p. 13
  • Performance is relatively robust across thresholds, and 0.5 already gives acceptable results.Source: §C.4 · p. 13
  • Overly high thresholds cause information loss, whereas overly low thresholds introduce less relevant context.Source: §C.4 · p. 13

D. More about the Factorized Data Synthesis

The factorized data synthesis perturbs both temporal grounding and textual response to create preference examples that contrast correct and distracted responses.

E. Visualization Results

The visual examples compare D2VLM with a recent SOTA method across dense captioning, grounding-focused tasks, and temporally grounded video question answering. They report stronger event localization and descriptions, finer boundary discrimination, and reliable evidence alongside correct answers.

E. Visualization Results

  • For dense captioning, D2VLM localizes individual events better and generates more coherent descriptions than the compared method, which often repeats similar content.Source: E. Visualization Results · pp. 13–14
  • For grounding-focused tasks, D2VLM more accurately localizes desired events and better distinguishes boundaries between events.Source: E. Visualization Results · pp. 13–14
  • In grounded video question answering, D2VLM answers correctly and provides more reliable temporal evidence, while the compared method fails to answer the question.Source: E. Visualization Results · pp. 13–14

E.T. Chat

  • E.T. Chat illustrates single- and multiple-event grounding, plus a glassware-location question with four answer choices.Source: E.T. Chat · p. 17
Loading 2512.24097v1…