PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving

Shiva Sreeram, Erfan Aasi, Tsun-Hsuan Wang, Alaa Maalouf, Daniela Rus

arXiv:2610.12575v1cs.CVcs.RO

TL;DR

It remains unclear whether video-language models reason over temporal dynamics, a consequential question for driving, where ego-motion must be inferred from evolving cues. The paper evaluates controlled ego-motion tasks using prompting, lightweight finetuning, and intermediate-feature probes. Prompting remains unreliable and biased, while finetuning and probes recover motion-discriminative information that final predictions do not reliably express.

  • Problem

    It remains unclear whether video-language models genuinely reason about temporal dynamics rather than relying primarily on frame-based or shortcut cues, especially for driving ego-motion.

    Source: §I · p. 1

  • Method

    The paper evaluates controlled driving ego-motion tasks with prompting interventions, lightweight finetuning, and probes trained on intermediate visual features.

    Source: A. Our Contributions · p. 1, §III · p. 4

  • Results

    Prompting remains unreliable and class-biased across models and tasks, while LoRA finetuning and feature probes recover strong motion-discriminative performance despite failures in prompted predictions.

    Source: §III · p. 4, §IV · p. 7

  • Takeaways & Limitations

    Off-the-shelf video-language models should not be treated as dependable temporal reasoners for driving; their motion cues may be present without stable temporal predictions.

    Source: A. Our Contributions · p. 2, §IV · p. 7

    Without single-frame or frame-shuffling controls, high LoRA and probe accuracy cannot establish that recovered cues are genuinely temporal, since CARLA renders may contain per-frame appearance cues.

    Source: §IV · p. 7

Abstract

from arXiv · show

Modern video-language models are increasingly treated as general-purpose video reasoners, yet it remains unclear whether they robustly capture temporal dynamics in safety-critical settings such as driving. We study this question through controlled ego-motion tasks that require identifying actions from driving videos. Across video-language models, we find that standard prompting, visual prompting, few-shot prompting, and chain-of-thought prompting remain unreliable and highly task- and model-dependent, with persistent class biases, although newer models show substantial task-specific few-shot gains. In contrast, lightweight LoRA finetuning and simple probes trained on intermediate visual features recover strong motion-discriminative performance, indicating that motion-relevant cues are often present inside the models but are not reliably used by the final video-to-language pathway. These results suggest that, in driving, current video-language models remain unreliable temporal reasoners and often behave more like frame-level, bias-driven recognizers than dependable temporal world models.

I. INTRODUCTION

The paper asks whether video-language models genuinely reason about temporal dynamics, focusing on ego-motion in driving as a consequential test. It contributes a controlled evaluation and finds prompting unreliable, while lightweight adaptation and intermediate-feature probes reveal motion information that final predictions do not reliably use.

I. INTRODUCTION

  • The study asks whether video-language models capture temporal dynamics or rely primarily on frame-based cues, a question with practical consequences for driving.Driving depends on inferring ego-motion and acceleration or deceleration from temporally evolving cues.Source: §I · p. 1

A. Our Contributions

  • The paper designs controlled ego-dynamics evaluations and compares prompting, lightweight adaptation, and representation-level diagnostics.The targeted motion primitives include forward/backward, left/right, and acceleration/deceleration from third-person car video.Source: A. Our Contributions · pp. 1–2
  • Prompted ego-motion predictions are highly model- and task-dependent, with persistent class biases and poor forward/backward discrimination.More elaborate prompting can improve fluency without consistently improving temporal correctness or transferring gains across motion categories.Source: A. Our Contributions · p. 2
  • Lightweight LoRA finetuning can unlock motion reasoning, while layer-wise probes find motion information decodable before the language model but not reliably reflected in final predictions.The authors identify a gap between visual representation and language-based reasoning; they suggest supervision and training procedure are relevant to adaptation.Source: A. Our Contributions · p. 2
  • The authors caution against treating off-the-shelf video-language models as reliable driving world models without explicit temporal grounding and dynamics-aware training.Source: A. Our Contributions · p. 2

II. PROBING VIA DIFFERENT LEVELS.

The paper uses a multi-level probing framework to locate where video-language models’ ego-motion reasoning succeeds or breaks down. It moves from prompting through targeted inference interventions to lightweight learning-based adaptation.

  • Comparing intervention levels is intended to isolate where ego-motion reasoning breaks down and characterize models’ latent strengths and gaps.Source: §II · p. 2
  • Standard prompting measures baseline ability to identify and describe motion patterns in driving videos.Source: §II · p. 2
  • Targeted prompt techniques test whether temporal consistency and causal grounding can be elicited through inference procedures rather than representation changes.The highest intervention level uses lightweight learning-based adaptations to refine predictions.Source: §II · p. 2

A. Prompt-Level Probing Strategies

The prompt-level strategies vary how motion evidence is presented to a video-language model. They test direct classification, spatial grounding, demonstrations, and explicit reasoning about changes in a static reference object.

  • Standard prompting directly asks the model to classify the vehicle’s motion from the video.Source: A. Prompt-Level Probing Strategies · p. 2
  • Visual prompting places a green box around a static object to test whether a stable spatial reference helps the model infer motion.The reference object is selected using open-set detection, tracking, and following.Source: A. Prompt-Level Probing Strategies · p. 3
  • Few-shot visual-text fusion provides examples of forward and backward motion before the query video to test generalization to unseen videos.Source: A. Prompt-Level Probing Strategies · p. 3
  • Chain-of-thought prompting explains how changes in a static object’s position or size indicate ego-motion, with a visual-box variant combining this reasoning with spatial prompting.The approach tests whether explicit step-by-step reasoning connects background changes to motion.Source: A. Prompt-Level Probing Strategies · p. 3

B. Probing via Learning Techniques

The learning-based methods test whether lightweight supervision or classifiers on frozen visual features can recover ego-motion information. Feature extraction compares representations before and after architecture-specific visual compression or projection.

  • LoRA finetuning adds trainable low-rank adapters to attention and projection layers while freezing pretrained weights, enabling efficient specialization on driving-motion labels.Source: B. Probing via Learning Techniques · p. 3
  • Simple classifiers trained on frozen features from before and after visual compression test whether motion cues are encoded in the backbone and survive the interface to the language model.The probes include linear layers and one-layer CNNs, with and without pooling.Source: B. Probing via Learning Techniques · pp. 3–4
  • Pre-compression features retain higher-resolution visual-token structure, but the authors do not assume they lack temporal information.The precise extraction point differs by architecture, including vision-tower outputs before a merger, connector, downsampling pathway, or resampler.Source: B. Probing via Learning Techniques · pp. 3–4
  • Comparing pre- and post-compression probes distinguishes possible loss during visual compression from motion cues that remain decodable but are not reliably used in prompted predictions.Source: B. Probing via Learning Techniques · p. 4

III. EXPERIMENTAL RESULTS

The evaluation combines prompt-level interventions with learning-based tests of ego-motion understanding in simulated and real-world driving videos. Prompted models often fail despite motion information being available in their internal representations.

Experimental design

  • Prompt-level probing, lightweight finetuning, and representation-level probing test whether models can infer ego-motion from driving videos.The pipeline separates training-free prompt interventions from methods that extract or adapt internal representations.Source: §III · p. 4

Results reporting

  • Prompted inference is evaluated alongside learning-based strategies, with accuracy results reported across ego-motion tasks and video-language models.The tables cover standard prompting on simulated and real-world videos; other Table I results use simulated data.Source: Table I · p. 5, Table II · p. 5

Data and protocol

  • The CARLA dataset contains 1,200 videos, while the real-world set contains 60; real-world videos are used only for standard prompting.The smaller real-world set is not used for supervised LoRA or feature-probing experiments.Source: §III · pp. 4–5

A. Standard Prompting

Standard prompting produces strong class biases rather than reliable discrimination of driving motion. Similar class-level biases appear in simulated and real-world videos, making aggregate accuracy an incomplete picture.

Standard Prompting

  • All models achieve 100% accuracy on forward motion and 0% on backward motion, collapsing to the forward class.Other motion pairs also show model-dependent class preferences.Source: A. Standard Prompting · p. 5, Table III · p. 5
  • Under a basic binary prompt describing the static reference vehicle, InternVL 3.5 achieves 67% accuracy on left/right, while aggregate accuracy can obscure severe class imbalance and it favors rightward motion.Source: A. Standard Prompting · p. 5
  • The class-level correlation between simulated and real-world standard-prompting accuracy is r = 0.957, but forward/backward collapse inflates it.The authors treat the result as evidence that similar biases appear outside CARLA, not as an independent measure of sim-to-real generalization.Source: A. Standard Prompting · p. 5, Figure 2 · p. 5

B. Visual Prompting

Visual grounding alone provides little benefit, while few-shot prompting improves selected models and motion pairs without delivering consistent performance across categories. Gains across model generations are concentrated in some motion tasks, not forward/backward discrimination.

Visual Prompting

  • Visual prompting provides little benefit; the only notable improvement is Qwen2-VL on forward/backward, from 50.00% to 65.00%.Source: B. Visual Prompting · p. 5

Fusion of Visual and Text

  • Few-shot results are model- and task-dependent: Qwen3-VL reaches 72.61% on left/right and 77.14% on accelerate/decelerate, while forward/backward remains near chance.Qwen2-VL improves only on forward/backward, from 50.00% to 58.80%; Qwen2.5-VL remains near chance.Source: B. Visual Prompting · p. 6
  • VideoLLaMA2's few-shot results are omitted from Table II because the approximate concatenated-video setup causes severe class collapse.Source: B. Visual Prompting · p. 6

Model generations

  • Few-shot gains across model generations are task-dependent, with forward/backward accuracy near chance and most improvements in left/right and accelerate/decelerate.The few-shot trend excludes VideoLLaMA2 because its protocol is not comparable.Source: B. Visual Prompting · p. 6, Figure 3 · p. 6

D. Chain-of-Thoughts

Chain-of-thought prompting does not reliably improve temporal reasoning correctness. In some model-task pairs, it performs worse than plain few-shot prompting, and combining it with visual prompting offers no consistent gains.

Chain-of-Thoughts

  • Chain-of-thought prompting provides no consistent benefit over few-shot prompting, and combining it with visual prompting also yields no consistent gains.Table IV compares few-shot strategies across models, including visual-prompting and chain-of-thought variants.Source: D. Chain-of-Thoughts · p. 6, Table IV · p. 6
  • Qwen3-VL performs worse with CoT than plain few-shot on left/right and accelerate/decelerate; InternVL 3.5 also drops on left/right.Source: D. Chain-of-Thoughts · p. 6
  • Qwen2.5-VL stays near chance across all three motion pairs under CoT, while Qwen2-VL remains strongly class-biased.Source: D. Chain-of-Thoughts · p. 6

E. LoRA Finetuning

Lightweight LoRA finetuning enables near-perfect ego-motion classification in almost all tested models, while simple probes on intermediate visual features substantially outperform prompted inference. The probe results vary across architectures and processing stages.

E. LoRA Finetuning

  • Near-perfect LoRA accuracy in all but one model shows that lightweight supervision can teach ego-motion distinctions despite earlier prompting biases.VideoLLaMA2 is the exception for accelerate/decelerate; LoRA uses 400 examples per action pair and an 80/20 train/held-out split.Source: E. LoRA Finetuning · p. 6
  • LoRA turns ego-motion recognition into supervised binary classification, rather than preserving the open-ended language reasoning produced by prompting.The authors use feature extraction to investigate where motion information is encoded and why it does not transfer reliably to prompted reasoning.Source: E. LoRA Finetuning · p. 6

F. Feature Extraction

  • Simple feature-level probes substantially outperform prompted inference, showing that intermediate visual features can support stronger motion classification.The probe evaluation uses one-layer CNNs, with and without pooling, on features extracted before or after model-specific visual processing stages.Source: F. Feature Extraction · p. 6, Table V · p. 7

FOR FEATURES EXTRACTED BEFORE AND AFTER THE VISUAL FEATURE

Motion cues are decodable within intermediate visual representations, but their evolution through the visual pipeline depends on architecture. Prompted predictions remain unreliable despite this internal information.

FOR FEATURES EXTRACTED BEFORE AND AFTER THE VISUAL FEATURE

  • 97.50% forward/backward accuracy follows visual merging in both Qwen2.5-VL and Qwen3-VL, up from 50.00% and 55.00%, respectively.Both models reach 100% on left/right after merging.Source: FOR FEATURES EXTRACTED BEFORE AND… · p. 7
  • MiniCPM-V 4.5 improves from 67.50% to 100% forward/backward accuracy after visual processing, while InternVL 3.5 is already highly informative pre-merge.InternVL 3.5 reaches 100% on forward/backward and left/right and 98.75% on accelerate/decelerate before merging.Source: FOR FEATURES EXTRACTED BEFORE AND… · p. 7
  • Strongly decodable internal motion representations coexist with a gap in the motion information reliably used by the final prompted prediction pathway.Across architectures, later processing may preserve, strengthen, or partially degrade the information.Source: FOR FEATURES EXTRACTED BEFORE AND… · p. 7
  • Prompting remains model- and task-dependent, with class biases and near-chance performance common, while few-shot gains are selective.Newer models show substantial gains on selected motion categories, but those improvements do not transfer consistently across categories.Source: FOR FEATURES EXTRACTED BEFORE AND… · p. 7
  • Lightweight LoRA finetuning achieves near-perfect performance in most cases, demonstrating that the models can learn ego-motion distinctions under supervision.The reported exception is VideoLLaMA2 on accelerate/decelerate.Source: FOR FEATURES EXTRACTED BEFORE AND… · p. 7

IV. CONCLUSION

Across six video-language models, prompting remains biased and unreliable for driving ego-motion tasks, while LoRA and feature probes recover motion-sensitive performance. The authors caution that these results do not establish that the recovered cues are genuinely temporal.

IV. CONCLUSION

  • Standard prompting shows strong, systematic class biases across six models, including complete forward/backward collapse; few-shot gains do not transfer consistently across motion categories.In a smaller real-world evaluation, standard-prompting class-level performance is strongly correlated with CARLA results (r = 0.957).Source: §IV · p. 7
  • LoRA often achieves near-perfect accuracy and simple probes recover motion cues, but absent single-frame and frame-shuffling controls, these results cannot establish that the cues are genuinely temporal.Per-frame appearance cues in CARLA renders may account for some performance; the authors leave this question to future work.Source: §IV · p. 7
  • The authors caution against treating fluent video-language outputs as evidence of dependable temporal reasoning in driving and call for tests on larger real-world datasets.They identify nuScenes and the Waymo Open Dataset as future evaluation targets, alongside architectures or objectives that preserve and use temporal information.Source: §IV · p. 7

Related papers

Loading 2610.12575v1…