PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Preference Optimization for Long-Form Video Understanding

Rui Li, Xiaohan Wang, Yuhui Zhang, Orr Zohar, Zeyu Wang, Serena Yeung-Levy

arXiv:2501.13919v3cs.CVcs.AIcs.CLcs.LGcs.RO

TL;DR

Video-LMMs still struggle with precise temporal grounding, while existing explicit-supervision approaches require costly annotations. TPO generates and filters temporal preference pairs for DPO training, improving performance across three benchmarks and producing the strongest 7B result on Video-MME.

  • Problem

    Existing video-LMM training provides weak temporal alignment, while explicit temporal supervision requires costly annotations.

    Source: §1 · p. 2

  • Method

    TPO creates preferred and dispreferred responses from relevant, irrelevant, or incomplete frame sets, filters noisy pairs, and optimizes them with DPO.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    TPO improves LongVA-7B by 2.9% on LongVideoBench, 3.1% on MLVU, and 2.5% on Video-MME, while LLaVA-Video-TPO becomes the strongest 7B model on Video-MME.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    TPO provides a scalable post-training route for improving temporal grounding across long-form video understanding benchmarks.

    Source: §6 · p. 12

    The preference distinction can fail because base video-LMM errors produce noisy or misaligned preference pairs.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Despite significant advancements in video large multimodal models (video-LMMs), achieving effective temporal grounding in long-form videos remains a challenge for existing models. To address this limitation, we propose Temporal Preference Optimization (TPO), a novel post-training framework designed to enhance the temporal grounding capabilities of video-LMMs through preference learning. TPO adopts a self-training approach that enables models to differentiate between well-grounded and less accurate temporal responses by leveraging curated preference datasets at two granularities: localized temporal grounding, which focuses on specific video segments, and comprehensive temporal grounding, which captures extended temporal dependencies across entire video sequences. By optimizing on these preference datasets, TPO significantly enhances temporal understanding while reducing reliance on manually annotated data. Extensive experiments on three long-form video understanding benchmarks--LongVideoBench, MLVU, and Video-MME--demonstrate the effectiveness of TPO across two state-of-the-art video-LMMs. Notably, LLaVA-Video-TPO establishes itself as the leading 7B model on the Video-MME benchmark, underscoring the potential of TPO as a scalable and efficient solution for advancing temporal reasoning in long-form video understanding. Project page: https://ruili33.github.io/tpo_website.

1. Introduction

Video-LMMs must model temporal dependencies, but existing training often provides only weak temporal alignment or requires costly annotations. TPO addresses this gap through preference-based post-training and improves performance across three long-form video benchmarks.

  • Existing video-LMM training learns temporal grounding implicitly from weak video–text correspondences, limiting precise temporal alignment.Source: §1 · p. 2
  • Explicit temporal-supervision methods improve alignment but depend on costly annotations that are difficult to scale.Source: §1 · p. 2
  • TPO post-trains video-LMMs to distinguish temporally relevant responses from responses generated using manipulated video inputs.The framework uses contrastive responses from original and corrupted frame sets, then optimizes the resulting preference data with DPO.Source: §1 · p. 2
  • TPO improves temporal grounding across LongVideoBench, MLVU, and Video-MME, with gains of 2.9%, 3.1%, and 2.5% on LongVA-7B.Source: §1 · p. 2
  • With LLaVA-Video, TPO yields a 2.3% improvement and establishes LLaVA-Video-TPO as the best-performing 7B model on Video-MME.Source: §1 · p. 2

2. Preliminaries

Preference learning trains models to favor preferred outputs over dispreferred ones, while DPO incorporates such preferences directly without explicit reward models or complex reinforcement learning.

  • Preference learning models which of two outputs is preferred for a given input to align model behavior with user expectations.Source: §2 · p. 2
  • A video-LMM preference dataset contains a video, query, preferred temporally grounded response, and dispreferred response.Source: §2 · p. 2
  • DPO directly integrates preference data into model optimization without explicit reward models or complex reinforcement learning algorithms.Source: §2 · p. 3

3. Temporal Preference Optimization

TPO creates temporally grounded preference pairs by manipulating frame inputs, filters noisy examples, and optimizes the model with DPO plus supervised fine-tuning. Its design covers question generation, contrastive temporal modeling, data curation, and preference-based training.

3. Temporal Preference Optimization

  • TPO aligns video-LMM outputs with intrinsic temporal preferences by contrasting responses from well-grounded and manipulated video inputs.The preference dataset is optimized with DPO to improve temporal reasoning without compromising pretrained capabilities.Source: §3 · p. 3

3.1. Temporal Preference Modeling

  • Questions are generated from sampled video frames and frame captions to provide contextually relevant queries for response generation.CogVLM2 captions sampled frames, and GPT-4o-mini produces candidate questions from those captions.Source: §3.1 · p. 3
  • Preferred responses use questions with their corresponding relevant frames, while dispreferred responses use irrelevant or incomplete frame sets.The two dispreferred settings exclude all relevant frames or randomly retain only a subset, creating contrastive temporal signals.Source: Figure 2 · p. 3, §3.1 · p. 4
  • Manipulated inputs increase ambiguity and noise, making temporal reasoning errors and hallucinations more likely and sharpening the preference signal.Source: §3.1 · p. 4

3.2. LLM-based Post-Filtering

  • Post-filtering removes pairs with inferior dispreferred responses, incorrect preferred responses, or ambiguous queries to reduce training noise.GPT-4o-mini evaluates frame captions, queries, and preference pairs against predefined filtering criteria.Source: §3.2 · p. 4

3.3. Training Objective

  • DPO trains the model to assign higher probabilities to preferred outputs while limiting deviation from its pretrained distribution.The training framework also incorporates a supervised fine-tuning objective controlled by hyperparameter α.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate TPO on three long-form video benchmarks using LongVA-7B and LLaVA-Video-7B, with results covering overall performance, scalability, robustness, and qualitative temporal grounding.

4.2. Results

  • 2.9%, 3.1%, and 2.5% gains on LongVideoBench, MLVU, and Video-MME, respectively, show that LongVA-TPO consistently outperforms baseline methods.The comparison is reported in Table 2 across the three benchmarks.Source: §4.2 · p. 7, Table 2 · p. 7
  • LongVA-TPO gains 1.2% to 2.8% over SFTSelf, while outperforming SFTLLM and Hound-DPO on the reported comparisons.The gains are attributed to contrasting preferred responses with temporally dis-preferred responses and injecting temporal preference priors.Source: §4.2 · p. 7
  • LLaVA-Video-TPO outperforms all 16 listed baselines, establishing the strongest 7B result on Video-MME.TPO also improves LongVA and LLaVA-Video relative to their corresponding baselines by 2.5% and 2.3% on Video-MME.Source: Table 1 · p. 6, §4.2 · p. 7

4.3.1. Performance with Different Input Frame Count

  • LongVA-TPO benefits from longer inputs through 128 frames, whereas LongVA degrades at 128 frames compared with 64 frames.This result supports robustness on extended inputs and localization of relevant information within long sequences.Source: §4.3.1 · p. 9, Figure 5 · p. 11

4.3.2. Effect of Dataset Sizes

  • TPO achieves consistent performance gains as the preference dataset grows from 2k to 5k and 10k examples across all three benchmarks.Table 3 evaluates the Video-MME results without subtitles.Source: §4.3.2 · p. 9, Table 3 · p. 9

4.3.3. Effect of Post-Filtering

  • Post-filtering consistently improves LongVA-TPO performance across multiple benchmarks.The comparison evaluates TPO with and without the post-filtering pipeline.Source: §4.3.3 · p. 10, Table 4 · p. 10

4.3.4. Effect of Different Data Mix Ratio

  • A 5:5 mix of negative responses from incomplete and irrelevant videos produces the best general video-understanding performance.The ablation compares ratios from 10:0 through 0:10 at a fixed overall dataset size.Source: §4.3.4 · p. 10, Table 5 · p. 10

4.3.5. Needle-in-a-Haystack

  • LongVA-TPO outperforms LongVA on the needle-in-a-haystack task, demonstrating stronger long-context temporal localization.The task embeds image questions within a 3-hour video and evaluates localization across temporal depths and input lengths.Source: Figure 4 · p. 9, §4.3.5 · pp. 10–11

5. Related Work

Prior video-LMM work improves video understanding, temporal grounding, preference learning, and self-training, but TPO explicitly brings temporal priors into post-training.

Video Large Multimodal Models

  • Existing video-LMMs often rely on synthetic video-text data or weak temporal supervision, limiting their ability to capture visual-temporal dynamics and precise alignment.Source: §5 · pp. 11–12
  • TPO pioneers post-training strategies that explicitly incorporate temporal priors into video-LMM optimization.Source: §5 · pp. 11–12

Temporal Grounding

  • Temporal grounding research includes dense captioning, highlight detection, temporal video grounding, temporal-aware architectures, and agentic systems.Source: §5 · p. 12

Preference Learning

  • Preference-learning methods use PPO, DPO, feedback, distillation, or manipulated inputs, while prior video approaches do not fully exploit temporal information inherent in videos.Source: §5 · pp. 11–12

Self-Training in Foundation Models

  • Self-training methods improve foundation models through generated rationales, visual priors, weak supervision, or iterative preference optimization across language, image, and video domains.Source: §5 · p. 12

6. Conclusion

TPO is presented as a scalable post-training framework that improves temporal grounding through preferences contrasting well-grounded and manipulated video responses. Experiments across three benchmarks show robust improvements, while future work targets broader data and model coverage.

  • TPO achieves robust improvements across LongVideoBench, MLVU, and Video-MME by contrasting preference responses from well-grounded and manipulated video clips.Source: §6 · p. 12
  • Multi-granularity temporal preferences provide a robust and efficient solution for advancing temporal reasoning in multimodal tasks.Source: §6 · p. 12
  • Future work includes scaling preference data for greater coverage and diversity and applying TPO to broader and larger video-LMM families.Source: §6 · p. 12

A. Reproducibility Statement

The appendix provides reproducibility resources and organizes additional implementation, dataset, and qualitative material for examining the TPO pipeline.

A. Reproducibility Statement

B. Appendix overview

  • The appendix covers implementation details, preference-dataset curation and examples, and qualitative examples.Source: B. Appendix overview · p. 18

C. Implementation Details

TPO is applied to LongVA and LLaVA-Video under specified training, optimization, and evaluation settings across three benchmarks.

  • TPO is trained on LongVA and LLaVA-Video, using 8 Nvidia A100 80GB GPUs, batch size 64, and one epoch on 10k curated samples.Source: C. Implementation Details · p. 18
  • LongVA-TPO uses β=0.3 and α=0.5, whereas LLaVA-Video-TPO uses β=0.2 and α=1 for KL-divergence and supervised fine-tuning losses.Source: C. Implementation Details · p. 18
  • Both models fully fine-tune the multimodal projector and language model while freezing the visual encoder, with model-specific learning rates and approximately four-hour training times.Source: C. Implementation Details · p. 18
  • Evaluation follows the LongVA and LLaVA-Video protocols through the official lmms-eval repository, using model-specific maximum frame settings.Source: C. Implementation Details · p. 18

D. Preference Dataset Curation

The preference dataset was built from 8,000 internet videos and 10,000 queries, covering diverse video lengths and reasoning tasks.

D. Preference Dataset Curation

  • 8,000 internet videos and 10,000 queries form the curated preference dataset for broad task coverage.For LLaVA-Video, a 10k-pair subset from LLaVA-Video-178k uses negatives generated from incomplete videos.Source: D. Preference Dataset Curation · p. 18
  • The dataset spans Temporal, Action, Causal, and Spatial Reasoning tasks, comprising 8.7%, 12.4%, 14.9%, and 13.5%, respectively.Source: D. Preference Dataset Curation · pp. 18–19
  • The collected videos have a documented length distribution, supporting preference data construction across varied temporal scales.Source: Figure 6 · p. 19
  • Question types are distributed across a 10K curated preference dataset for LongVA-TPO.Source: Figure 7 · p. 19

E. Preference Dataset Examples

The preference examples contrast temporally grounded responses with errors caused by wrong segments, irrelevant clips, or insufficient frame coverage.

E. Preference Dataset Examples

F. Qualitative Analysis Examples

Qualitative comparisons show TPO improving temporal localization, central-theme identification, OCR, and symbolic interpretation over LongVA.

F. Qualitative Analysis Examples

Related papers

Loading 2501.13919v3…