PaperlayerView exact revision on arXiv

Source-linked AI summary

The Shape of Time: Video-Token Contrast for Temporal Understanding in VideoLMs

Yumeng Shi, Quanyu Long, Yin Wu, Wenya Wang

arXiv:2609.04110v1cs.CV

TL;DR

VideoLMs are expected to reason about event progression, but response-level supervision does not directly require video-token representations to encode it. VT-Contrast contrasts order-preserving views with same-video reordered counterfactuals on selected late-layer last-frame tokens. It yields consistent gains across temporal benchmarks, including a 5.73% TOMATO improvement for the 2B model under the 8-frame setting over corresponding Qwen3.5 base models.

  • Problem

    VideoLMs are expected to reason about event progression, but response-level language supervision does not directly require video-token representations to encode it.

    Source: §1 · p. 1

  • Method

    VT-Contrast contrasts order-preserving views with same-video reordered counterfactuals on selected late-layer last-frame video tokens, grading temporal disruption with Kendall tau distance.

    Source: §1 · p. 2

  • Results

    Across temporal benchmarks VT-Contrast shows consistent overall gains; for the 2B model under the 8-frame setting, it improves TOMATO by 5.73% over corresponding Qwen3.5 base models.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Shaping video-token representations is a promising direction for improving temporal reasoning in VideoLMs.

    Source: §5 · p. 9

    Validation is limited to moderate-scale supervised finetuning on existing VideoLMs, leaving effectiveness on larger and more diverse video-text data for further exploration.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Seeing frames in order does not mean representing time. Modern VideoLMs receive ordered video streams, yet their main supervision acts on generated text rather than video-token representations where event dynamics should first emerge. This mismatch allows models to learn temporal answers from shortcuts such as objects, scenes, and language priors, without requiring internal video representations to capture event progression. To address this, we propose VT-Contrast, a representation-level temporal counterfactual objective for VideoLMs. Its design asks where temporal supervision should act and what temporal differences it should expose. VT-Contrast supervises selected late-layer last-frame video tokens, where temporal information is expected to be integrated before language generation, and contrasts order-preserving views with same-video reordered counterfactuals graded by Kendall tau distance. It requires no architectural changes, is compatible with diverse VideoLM training tasks, and improves overall performance across temporal understanding benchmarks. Our code is available at https://github.com/ANDgate99/VT-Contrast.

1 Introduction

VT-Contrast directly supervises video-token representations to capture temporal order, contrasting order-preserving views with same-video reordered counterfactuals. The paper reports consistent gains across temporal benchmarks, especially on order-sensitive tasks.

1 Introduction

  • VideoLMs are expected to distinguish event progressions such as opening versus closing, but response-level supervision does not require video representations to capture them.Similar visual evidence or answer priors can support correct responses while leaving representations underconstrained for event progression.Source: §1 · p. 1
  • VT-Contrast contrasts order-preserving views with same-video reordered counterfactuals in video-token space, making temporal order rather than static appearance the learning signal.The reordered views retain the same visual evidence while changing event progression.Source: §1 · p. 2
  • Selected late-layer last-frame video tokens receive supervision, and Kendall tau distance grades the temporal disruption of counterfactuals.These tokens are expected to aggregate information from preceding frames before language generation.Source: §1 · p. 2
  • Experiments across temporal benchmarks show consistent overall gains, especially on order-sensitive tasks.VT-Contrast is jointly optimized with the standard language modeling objective and requires no architectural modification.Source: §1 · p. 2
  • The paper frames temporal supervision as a representation-level signal, rather than supervision applied only through textual responses.The objective is designed to complement language modeling while leaving the response format unchanged.Source: §1 · p. 2

2 Related Works

Related work describes VideoLM training as primarily supervising generated text, while temporal benchmarks expose difficulty with fine-grained event progression. Prior work explores temporal modeling and contrastive learning, but directly shaping VideoLM video-token representations for temporal sensitivity remains less explored.

2.1 Video Language Models

  • VideoLMs encode video frames or clips as visual tokens for LLM feature space and generate responses autoregressively, while training signals primarily target generated text tokens.Language loss updates visual components but does not directly constrain video tokens to distinguish temporal order changes.Source: §2.1 · p. 2

2.2 Temporal Understanding

  • VT-Contrast constructs anchor, order-preserving positive, and temporal-counterfactual negative views, then contrasts their selected video-token representations alongside language modeling.A shared VideoLM processes the views.Source: Figure 2 · p. 3
  • Temporal benchmarks show that VideoLMs struggle with fine-grained semantics when similar visual content differs in event order or progression.Prior approaches include temporal-aware architectures, reasoning transfer, and auxiliary ordering tasks.Source: §2.2 · pp. 2–3

2.3 Contrastive Learning

  • Prior contrastive video methods use temporal perturbations, but mainly target standalone video encoders rather than VideoLM video-token representations.VT-Contrast directly separates temporally reordered videos in the VideoLM's internal representation space.Source: §2.3 · p. 3

3 Method

VT-Contrast adds representation-level temporal counterfactual learning to VideoLMs, complementing response supervision with targeted video-token constraints. It contrasts same-video order-preserving and reordered views, grades order disruption, and jointly optimizes the auxiliary signal with language modeling.

3.1 Problem Setup

  • Standard response-level loss aligns answers but only indirectly supervises video tokens, leaving fine-grained order distinctions underconstrained and allowing reliance on static cues or language priors.Source: §3.1 · pp. 3–4

3.2 Temporal Counterfactual Learning

  • Three views control visual content while contrasting chronology: the anchor and frame-dropped positive preserve order, whereas a same-video negative reorders the sampled frames.Source: §3.2 · p. 4
  • An InfoNCE objective trains each anchor representation to approach its order-consistent positive and move away from counterfactual negatives using similarity scores.The similarity uses cosine similarity and contrastive temperature τc.Source: §3.2 · p. 4

3.3 Emergence-aware Temporal Supervision

  • Contrastive supervision uses pooled last-frame video-token features from selected layers, focusing constraints on compact features expected to carry temporal information.The selected layer set depends on the VideoLM backbone.Source: §3.3 · p. 5
  • Kendall tau distance grades order disruption: smaller normalized distances produce harder negatives closer to the original order, while larger distances indicate stronger disruption.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Optimization Objective

  • The final loss adds direct temporal supervision to standard language modeling, with λ = 0.1 used for stable joint optimization.Source: §3.4 · pp. 5–6
  • The 4B model outperforms recent 4B MLLMs on most benchmarks, and VT-Contrast improves corresponding Qwen3.5 models in most settings.Source: Table 1 · p. 6

4 Experiments

Experiments assess VT-Contrast on temporal benchmarks, compare it with baselines and Qwen3.5 scales, and test its design choices and representation effects.

4.1 Experimental Settings

  • Comparisons include 4B InternVL3.5 and LLaVA-OneVision1.5, 7B Visual Jigsaw, and Qwen3.5 base models at 0.8B, 2B, and 4B.Source: §4.1 · p. 6
  • Training uses the SSv2 training split, with TOMATO, TempCompass, and Vinoground evaluating temporal reasoning, instructions, and fine-grained video-caption alignment.The three evaluation benchmarks contain 1,484, 7,540, and 2,000 questions, respectively.Source: §4.1 · pp. 6–7

4.3 Ablation Studies

  • Compared with LM-only training, VT-Contrast improves every reported metric across three benchmarks, with larger gains on TOMATO and Vinoground Text.The default loss weight is λ = 0.1.Source: Table 2 · p. 7, §4.3 · p. 8
  • Video-token supervision outperforms answer-generation query-token supervision on TempCompass and Vinoground Group, while last-frame tokens outperform full-frame tokens overall.Source: Table 3 · p. 7, §4.3 · p. 8

4.1 Experimental Settings

  • Qwen3.5 models are fine-tuned for video question answering with joint language-modeling and VT-Contrast losses, using 16 sampled negatives and late-layer last-frame tokens.The main-text loss weight is λ = 0.1; models use 8 frames and train for 2,250 steps.Source: §4.1 · p. 7

4.2 Main Results

  • Across most scales and frame settings, VT-Contrast improves over Qwen3.5; 2B gains 5.73% on TOMATO and 3.95% on TempCompass Y/N at 8 frames.The 4B model also outperforms recent open-source VideoLMs on most benchmarks, particularly TOMATO; frame budgets are upper bounds, so some videos may use fewer frames.Source: §4.2 · p. 7

4.3 Ablation Studies

  • Hard negatives with Kendall-tau distances in (0, 0.5] perform best overall, outperforming both full and simple negative sets.The study compares the three distance ranges under the 2B model with 8 frames.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 Representation Analysis

  • After training, reordered variants separate more from original videos; negative similarity falls more clearly as Kendall-tau distance increases, and PCA shows more distinct representations.The analysis uses 300 SSv2 validation samples.Source: Figure 4 · p. 8, §4.4 · p. 8

5 Conclusion

VT-Contrast shifts temporal supervision from generated responses to video-token representations by contrasting order-preserving views with same-video reordered counterfactuals. Experiments show consistent gains across temporal benchmarks.

5 Conclusion

  • VT-Contrast shifts temporal learning from response space to video-token space by contrasting order-preserving views with same-video reordered counterfactuals.Source: §5 · p. 9
  • The method targets last-frame representations and grades counterfactual temporal disruption with Kendall tau distance.Source: §5 · p. 9
  • Experiments across temporal benchmarks show consistent overall gains, making video-token representation shaping a promising direction for temporal reasoning in VideoLMs.Source: §5 · p. 9

Limitations

Validation is limited to moderate-scale supervised finetuning on existing VideoLMs. Effectiveness on broader data and tasks remains to be explored, and gains may depend on the base model’s task-specific capabilities.

Limitations

  • Computational resources limited validation to moderate-scale supervised finetuning on existing VideoLMs.The authors identify larger-scale pretraining as a direction for further work.Source: Limitations · p. 9
  • Effectiveness on longer videos, multi-event scenarios, and broader instruction-style tasks remains to be explored.The authors specifically identify larger and more diverse video-text data as a setting for further exploration.Source: Limitations · p. 9
  • VT-Contrast gains may depend on the base model’s temporal modeling and generation ability, with larger models not always performing better on TempCompass Caption.The authors attribute this qualification to task-specific capability differences.Source: Limitations · p. 9

A Additional Analysis

The appendix examines loss weight, layer selection, and negative-view count, finding that these choices affect results and identifying defaults for the 2B model.

A Additional Analysis

  • Across its additional analyses, the appendix reports that VT-Contrast is relatively robust to common hyperparameter choices.Unless otherwise specified, analyses use the 2B model with 8 input frames.Source: A Additional Analysis · p. 12

A.1 Contrastive Loss Weight

  • λ = 0.1 gives the best overall trade-off: adding VT-Contrast improves all three reported benchmarks, but larger weights can hurt some tasks.The authors suggest that excessive contrastive weight may interfere with general downstream performance.Source: §A.1 · p. 12, Figure 5 · p. 12

A.2 Targeted Layer Selection

  • Middle-to-late layers generally perform better than early layers, and layers 21–24 are selected for the 2B model.The authors note that the best range may vary across model scales; later layers are a reliable default.Source: §A.2 · p. 12, Figure 6 · p. 12

A.3 Number of Counterfactual Negatives

  • Using 16 counterfactual negatives gives the most stable results, while 8 is already competitive on several tasks and 4 provides a weaker signal.The number of negatives determines how many temporally perturbed same-video views are compared with each anchor.Source: §A.3 · p. 12

A.4 Contrastive Temperature

The temperature analysis finds comparable performance across most tested settings and uses τc = 0.10 as a strong overall trade-off.

A.4 Contrastive Temperature

  • Different temperatures yield comparable performance across most tasks, with τc = 0.10 providing a strong overall trade-off.The analysis identifies τc = 0.10 as the default setting.Source: §A.4 · pp. 12–13, Table 6 · p. 13
  • Smaller τc sharpens the similarity distribution and can help some tasks, but may produce less stable results.The temperature controls the sharpness of the similarity distribution in the InfoNCE objective.Source: §A.4 · pp. 12–13
  • Larger τc smooths the distribution and weakens the separation signal, leading to weaker overall performance.Source: §A.4 · p. 13

A.5 Qualitative Analysis

VT-Contrast improves most metrics across frame settings on InternVL3.5, and qualitative examples show more reliable adaptation to reversed temporal semantics than Qwen3.5.

A.5 Qualitative Analysis

  • On reversed-video examples, Qwen3.5 misses fine-grained transitions, while the method adapts more reliably to reversed temporal semantics.Examples include turning direction, paper deformation, and an apple’s visual state change.Source: §A.5 · p. 13

B Additional Experiments

B.1 Evaluation with InternVL3.5

  • VT-Contrast improves most metrics across 8-, 16-, and 32-frame inference settings on InternVL3.5-4B.Training used 8-frame inputs, λ = 0.12, layers 29–32, and 12 negatives.Source: §B.1 · p. 13

B.2 Evaluation on General Video QA

On Video-MME, VT-Contrast improves overall accuracy, extending its evaluation beyond temporally focused tasks. The appendix also reports its training overhead as the number of counterfactual negatives increases.

B.2 Evaluation on General Video QA

  • On reversed TempCompass Y/N video pairs, VT-Contrast better adapts predictions to changed temporal order than Qwen3.5.Each pair shares similar visual content but requires different answers.Source: Figure 7 · p. 14
  • VT-Contrast improves most metrics over the InternVL3.5-4B baseline across 8, 16, and 32 frames.The table caption describes these as changes over the baseline.Source: Table 7 · p. 14
  • VT-Contrast improves overall Video-MME accuracy over the Qwen3.5-2B baseline, extending its evaluation to broader video QA.The evaluation trains on a subset of LLaVA-Video-178K open-ended QA data for 1,500 steps; Table 8 reports overall accuracy.Source: §B.2 · pp. 13–14, Table 8 · p. 14

B.3 Training Overhead

  • Training overhead increases with the number of counterfactual negatives; the current implementation is not efficiency-optimized, so further optimization may reduce costs.The overhead profile uses M = 0, 4, 8, and 16 negatives; Table 9 reports wall-clock time per step and peak allocated and reserved CUDA memory.Source: §B.3 · p. 14

C Prompt Details

The appendix specifies evaluation prompts and answer constraints for TOMATO, TempCompass, and Vinoground, using consistent prompt formats within each benchmark to reduce prompt-induced variation.

C Prompt Details

  • Across reported models and settings, each benchmark uses a consistent prompt format to reduce prompt-induced variation.The appendix lists example prompts for each task.Source: C Prompt Details · p. 14

C.1 TOMATO

  • TOMATO is multiple-choice, with responses constrained to one uppercase option letter to reduce open-ended answers and simplify evaluation.The prompt also instructs models not to answer “not possible to determine.”Source: §C.1 · p. 15
  • TOMATO prompts provide uniformly sampled chronological frames and instruct models to rely on those frames, while guessing if they are insufficient.An example asks which direction a person’s hand moved.Source: §C.1 · p. 15

C.2 TempCompass

  • TempCompass uses task-specific prompts for Yes/No, multiple-choice, captioning, and caption matching, with answer formats specified when applicable.Captioning follows the lmms-eval default and uses GPT-3.5-Turbo grading.Source: §C.2 · p. 15
  • TempCompass examples include a yes/no question about dunking and a multiple-choice question distinguishing dunking, dribbling, and passing.Source: §C.2 · p. 15
  • TempCompass captioning selects consistent subject-action information and generates a caption, while caption matching chooses the better-fitting description.The examples distinguish dribbling from dunking.Source: §C.2 · p. 15

C.3 Vinoground

  • Vinoground uses separate text-to-video and video-to-text prompts, with the model selecting an option letter in either direction.The text-to-video example contrasts two captions with different event order.Source: §C.3 · pp. 15–16
  • The Vinoground video-to-text example matches a caption to one of two segments separated by a two-second black frame.The response is a single option letter.Source: §C.3 · p. 16

Related papers

Loading 2609.04110v1…