Source-linked AI summary
TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility
Saman Motamed, Minghao Chen, Luc Van Gool, Iro Laina
TL;DR
Video-language models struggle to judge physical plausibility in videos, while quantitative assessment remains challenging. The paper introduces TRAVL, a motion-aware fine-tuning recipe, and ImplausiBench, a benchmark designed to isolate visual-temporal reasoning. TRAVL yields consistent gains across two VLM backbones, and ImplausiBench is more resistant to shortcut exploitation than prior benchmarks.
Problem
Quantitatively assessing physical realism in video remains challenging, and VLMs struggle with temporal and physical reasoning.
Source: §1 · p. 1, §2.1 · p. 3Method
TRAVL fine-tunes spatial and trajectory-aware attention using balanced plausible and implausible video data, while ImplausiBench evaluates paired real and generated videos with linguistic shortcuts controlled.
Source: §1 · p. 2Results
Takeaways & Limitations
ImplausiBench provides a more shortcut-resistant signal of grounded physical reasoning than existing benchmarks such as Impossible Videos.
Source: §7 · p. 8The fine-tuning dataset is modest and limited in diversity, and externally generated patch trajectories add computational overhead and sensitivity to occlusion or blur.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Despite impressive visual fidelity, modern video generative models frequently produce sequences that violate intuitive physical laws, such as objects floating, teleporting, or morphing in ways that defy causality. While humans can easily detect such implausibilities, there remains no robust method for quantitatively assessing physical realism in video. In this work, we explore whether Video-Language Models (VLMs) can be trained to serve as reliable judges of physical plausibility. We find that existing VLMs struggle to identify physics violations, exposing fundamental limitations in their temporal and causal reasoning. To address this, we introduce TRAVL, a fine-tuning recipe that combines a balanced training dataset with a trajectory-aware attention module to improve motion encoding and discrimination in VLMs. To evaluate physical reasoning more rigorously, we propose ImplausiBench, a benchmark of 300 videos (150 real, 150 generated) that removes linguistic biases and isolates visual-temporal understanding. Performance is reported both with gold-standard human judgments and stricter LLM-as-judge metrics. Together, TRAVL and ImplausiBench offer a unified framework for probing and improving physical plausibility in multimodal models, shedding light on a challenging and underexplored aspect of visual-temporal understanding.
1 Introduction
Video generators can produce physically implausible events, but quantitative assessment remains difficult and current VLMs have temporal and causal reasoning limitations. The paper introduces TRAVL and ImplausiBench to improve and rigorously evaluate VLM judgments of physical plausibility.
Motivation
- Existing video metrics emphasize perceptual similarity, while VLMs struggle with motion and physical plausibility and may rely on language or positional shortcuts in evaluation.The introduction also notes that narrow physics-domain fine-tuning may fail to generalize.Source: §1 · pp. 1–2
Contributions
- TRAVL combines a balanced plausible-and-implausible video dataset with spatial and trajectory-aware temporal attention, while leaving the vision encoder and language model unchanged.Its motion paths are computed with CoTracker, and only attention and projection layers are fine-tuned.Source: §1 · p. 2
- ImplausiBench contains 300 paired real and generated videos and uses adversarially checked multiple-choice questions to isolate visual-temporal reasoning across diverse implausibility types.The benchmark covers events including teleportation, levitation, deformation, duplication, and state changes.Source: §1 · p. 2
- The paper frames TRAVL and ImplausiBench as a combined approach to improving and testing physical plausibility judgments in VLMs.Performance is evaluated using human judgments and LLM-judge metrics.Source: §1 · p. 2
2 Related Work
Video-language models have limitations in temporal and physical reasoning, motivating motion-aware modeling and evaluation that requires grounded video understanding. Prior physical-reasoning benchmarks vary in interaction, realism, and whether they target implausible events or resist language-only shortcuts.
2.1 Advancements and Limitations in Video-Language Models
- Many video-language models encode frames independently, weakening motion continuity and temporal context needed to recognize physical violations.The related work also describes benchmarks that emphasize static understanding or treat frames independently.Source: §2.1 · pp. 2–3
2.2 Incorporating Trajectory-Based Temporal Modeling
- Trajectory-aware methods improve motion modeling in other vision tasks, but the cited approaches were not designed to improve physical-plausibility reasoning in VLMs.TRAVL applies trajectory-guided attention to this gap.Source: §2.2 · p. 3, Figure 2 · p. 3
2.3 Evaluating Physical Reasoning in VLMs
- Physical-reasoning benchmarks range from interactive simulations and symbolic puzzles to diagnostic video tests, but some do not target implausible or counterfactual scenarios.InfLevel is diagnostic and prohibits training, whereas ImplausiBench uses multiple-choice questions with adversarial distractors.Source: §2.3 · pp. 3–4
- ImplausiBench evaluates paired plausible and implausible videos, resists linguistic shortcuts, and uses LLM-as-a-judge scoring validated against human evaluation.It targets complex natural videos, extending benchmarks that often use simplified synthetic data.Source: §2.3 · pp. 3–4
3 Method
TRAVL adds spatial and trajectory-guided temporal attention to pretrained video-language models to represent both scene structure and motion continuity. The method integrates with different VLM architectures and is fine-tuned on balanced plausible and implausible video data with temporally grounded supervision.
3.1 TRAVL
- TRAVL combines intra-frame spatial attention with trajectory-guided temporal attention to model scene geometry and object continuity without changing the vision or language backbones.Sparse patch trajectories extracted with CoTracker guide temporal connections.Source: §3.1 · p. 4, §3 · p. 4
- The temporal mask links patches that share motion trajectories, restricting attention to valid links and supporting detection of persistence violations such as teleportation or sudden morphing.Spatial attention is intended to help detect anomalies including duplication and deformation.Source: §3.1 · p. 4
3.2 Model Integration: TRAVL Across Architectures
- TRAVL replaces pooling or spatial pooling with spatial and trajectory-guided attention in Video-ChatGPT and LLaVA-NeXT while preserving their input-output formats and keeping the backbones frozen.LLaVA-NeXT uses chunked temporal attention, while Video-ChatGPT applies attention over sparse CoTracker masks.Source: §3.2 · p. 5
3.3 Fine-tuning Dataset
- The fine-tuning data combines naturally occurring diffusion-model failures with human temporal descriptions and GPT-4o-generated captions and balanced question-answer pairs.The training pipeline also includes plausible-video captioning, feedback-augmented captions for implausible videos, and QA generation.Source: §3.3 · p. 5, Figure 3 · p. 5
- The final dataset contains 3,482 videos and 19,708 QA pairs, with question types mirrored across plausible and implausible clips to limit class-pattern shortcuts.It includes 535 Impossible Videos clips, 92 additional generated clips, and diverse real-world QA data.Source: §3.3 · pp. 5–6
4 ImplausiBench
ImplausiBench pairs plausible real videos with generated implausible counterparts and uses shared multiple-choice questions to test visual-temporal understanding. Its balanced options and blind-test validation are designed to limit language-only shortcuts.
Benchmark Construction
- ImplausiBench pairs each real-world plausible video with a generated counterpart and manually checks generations to ensure a clear physical violation.The videos cover motion anomalies, continuity violations, structural changes, unnatural interactions, appearance shifts, and implausible state changes.Source: §4 · p. 6
Multiple Choice Format
- Each video pair shares one manually curated seven-option question, with three plausible answers, three implausible answers, and a “None of the above” option.Blind-test validation revises answer sets when language or positional patterns permit above-chance performance without the video.Source: §4 · p. 6
Comparison to Existing Benchmarks
- Without video input, GPT-4o and Qwen2.5 score well above chance on Impossible Videos, whereas blind-test accuracy on ImplausiBench is closer to chance.The respective random-chance rates are 20% for Impossible Videos and 14.3% for ImplausiBench.Source: §4 · p. 7, Table 1 · p. 7
Why We Do Not Use Impossible Videos for Evaluation
- The authors use Impossible Videos for training rather than evaluation because its questions permit language and positional shortcuts, while ImplausiBench requires success on both versions of each scenario.This construction is intended to provide a more rigorous assessment of physical reasoning and visual grounding.Source: §4 · p. 7
5 Results
TRAVL improves physical-implausibility detection over same-data supervised fine-tuning, and the authors evaluate it with human and stricter LLM-judge metrics. Ablations indicate that spatial and temporal attention contribute complementary benefits.
5 Results
- Human annotators provide gold-standard judgments, while GPT-4o maps open-ended answers to multiple-choice options using a stricter LLM-judge metric.The LLM judge often withholds credit for partial answers; its scores are lower, but relative model trends are preserved.Source: §5 · p. 7
- 18.7% on implausible videos and 2.0% points on real ones: LLaVA-NeXT with TRAVL outperforms same-data SFT under the Human metric.The paper reports similar improvements for Video-ChatGPT and gains over SFT on both subsets under Human and LLM-judge metrics.Source: §5 · p. 7
5.1 Ablation Studies
- Spatial-only and temporal-only ablations both improve over supervised fine-tuning, but neither matches full TRAVL.Spatial attention supports detection of implausible structures, while temporal attention supports motion-continuity tracking.Source: §5.1 · p. 8
- In binary plausibility classification, TRAVL improves implausibility detection while maintaining plausible-video accuracy, with ablations between SFT and full TRAVL.This classification metric provides a complementary view of discrimination ability, not reasoning quality.Source: §5.1 · p. 8
6 Limitations and Future Work
The authors identify limits in dataset coverage, trajectory generation, long-range temporal reasoning, and mediated evaluation. They suggest broader data, integrated tracking, memory-efficient attention, and note the dependence on GPT-4o judgments.
6 Limitations and Future Work
- The fine-tuning dataset is modest and less diverse than real-world video, while externally generated patch trajectories add computational overhead and sensitivity to occlusion or blur.The authors suggest broader scenarios and possibly automated generation or simulation, and propose learned or differentiable tracking as a robustness direction.Source: §6 · p. 8
- Temporal attention over 4–16-frame chunks limits long-range reasoning, and mediated evaluation depends on GPT-4o's interpretation of model outputs.The authors suggest memory-efficient attention for full-sequence modeling.Source: §6 · p. 8
7 Conclusion
The paper presents TRAVL as a minimally modifying, trajectory-aware fine-tuning framework and reports consistent physical-plausibility gains on two VLM backbones. It also introduces ImplausiBench to reduce linguistic shortcuts and isolate visual-temporal understanding.
7 Conclusion
- TRAVL combines spatial and trajectory-aware temporal attention with plausibility supervision, improving physical reasoning with minimal changes to vision or language backbones.The authors demonstrate consistent physical-plausibility judgment gains on Video-ChatGPT and LLaVA-NeXT.Source: §7 · p. 8
- ImplausiBench is designed to eliminate linguistic shortcuts and isolate visual-temporal understanding, offering a clearer signal of grounded physical reasoning.The paper reports that its blind test protocol is more robust to shortcut exploitation than Impossible Videos.Source: §7 · p. 8
TRAVL: Supplementary
The supplementary material expands the paper's evaluation, dataset, and model details, including qualitative examples, quantitative tradeoffs, and construction and training procedures.
TRAVL: Supplementary
- The supplement details GPT-4o scoring of open-ended VLM responses against multiple-choice ground truth and provides qualitative visualizations from both benchmarks.The visualizations cover model successes and failures across training stages.Source: TRAVL: Supplementary · p. 12
- It further breaks down quantitative results to examine plausibility-sensitivity and implausibility-detection tradeoffs.It also outlines the fine-tuning dataset and prompts for generating temporal and physics-based QA pairs.Source: TRAVL: Supplementary · p. 12
- The supplement documents ImplausiBench construction, TRAVL architecture and training, practical observations, and comparisons with prior physical-reasoning benchmarks.The referenced example videos are available in the supplementary result viewer.Source: TRAVL: Supplementary · p. 12
A Results in Detail
TRAVL fine-tuning improves implausible-video accuracy but reduces accuracy on plausible videos. The evaluation uses a GPT-4o judge to map open-ended explanations to multiple-choice answers, alongside qualitative examples of model outputs.
LLM-as-a-Judge Evaluation
- GPT-4o maps each VLM’s open-ended explanation to a benchmark option without receiving the ground truth.Blind probes checked whether the judge could exploit language bias, and a “None of the above” option prevents partial credit.Source: A Results in Detail · p. 12
Visualizations of VLM Outputs
- Qualitative examples compare Video-ChatGPT and LLaVA-NeXT backbones with supervised and TRAVL fine-tuned variants across successes and failures.Examples cover four implausible Impossible Videos clips and four plausible–implausible pairs from ImplausiBench.Source: A Results in Detail · p. 12
Understanding the Plausibility–Implausibility Tradeoff
- Video-ChatGPT improves from 0.0% to 12.0% and LLaVA-NeXT from 3.3% to 52.7% on implausible videos with TRAVL, but plausible-video accuracy falls.TRAVL still performs better on both plausible and implausible videos than the same backbone trained via SFT; the authors suggest the training set’s limited plausible examples may contribute to the tradeoff.Source: A Results in Detail · p. 12
B Fine-Tuning Dataset
The fine-tuning data combines real and generated plausible and implausible videos with QA pairs designed to avoid question-based shortcuts. ImplausiBench pairs real videos with generated implausible counterparts and uses distractors intended to limit language-only solutions.
B Fine-Tuning Dataset
- The fine-tuning set contains 3,482 videos and 19,708 QA pairs drawn from four sources, combining real and generated content across motion contexts and implausibility types.Sources include Video-ChatGPT, Physics-IQ with generated variants, 535 Impossible Videos clips, and 92 clips generated with Pika 1.5.Source: B Fine-Tuning Dataset · p. 12
- The source subsets add regenerated, plausibility-oriented QA for 1,763 Video-ChatGPT videos, 894 generated Physics-IQ variants, 535 Impossible Videos clips, and 92 additional Pika 1.5 clips.Physics-IQ variants were reviewed and labeled by violation type; the additional Pika clips were manually checked for clear implausibility.Source: B Fine-Tuning Dataset · pp. 12–13
B.1 QA Generation Prompts
- QA prompts keep questions neutral for both plausible and implausible videos, so plausibility cues appear in answers rather than question wording.For implausible clips, answers explain why the video is unrealistic; for real clips, answers explain why events follow physical principles.Source: §B.1 · p. 13
C ImplausiBench Construction
- ImplausiBench pairs 150 diverse real-world videos with generated implausible continuations and uses multiple-choice questions refined to resist linguistic shortcuts.Gemini 2.5-pro, the best-performing model in the reported experiments, achieved 41% on implausible videos.Source: C ImplausiBench Construction · p. 13
D TRAVL Details and Model Specifications
TRAVL adds spatial and trajectory-aware temporal attention to two VLM backbones, using tracked patch trajectories to constrain temporal attention. The integrations aggregate and project the resulting features before passing them to each language model.
Video-ChatGPT Integration
- Video-ChatGPT uses spatial attention over 256 tokens per frame and temporal attention over 100 tracked-patch tokens, forming a 356-token sequence for the language model.The resulting features are aggregated and projected through a 2-layer MLP.Source: D TRAVL Details and Model Specifications · p. 14
LLaVA-NeXT Integration
- LLaVA-NeXT applies spatial attention to 729 SigLIP patch tokens per frame, then temporal attention over patch-aligned trajectories using sparse flow masks.Temporal attention is chunked into overlapping 4–16-frame windows; the authors note that shorter chunks may hurt longer-video reasoning.Source: D TRAVL Details and Model Specifications · p. 14
Trajectory-Guided Sparse Attention Masking
- CoTracker-generated masks connect patch pairs sharing a trajectory, constraining temporal attention to motion-consistent features while reducing computation.Tracking points are reinitialized every k frames to account for new objects or major scene changes.Source: D TRAVL Details and Model Specifications · p. 14
Training Specifications
- Training uses a 1 × 10−4 learning rate for attention modules, 5 × 10−5 for the projector, batch sizes of 8 and 2, and four NVIDIA H200 GPUs.The batch sizes are for Video-ChatGPT and LLaVA-NeXT, respectively.Source: D TRAVL Details and Model Specifications · p. 14
E Observations
The observations identify dataset coverage, token format, and trajectory tracking as practical constraints on TRAVL. They also describe a scope boundary: the benchmark targets detecting implausibility, while other kinds of physical reasoning remain future directions.
Fine-tuning Frame Rate
- Training retains each video’s original frame rate; re-encoding at varied FPS values is proposed as a way to expose the model to more temporal resolutions.The authors suggest this may improve temporal-attention robustness and downstream performance.Source: E Observations · p. 14
Impact of Token Count
- Increasing tokens beyond Video-ChatGPT’s original 356-token setup consistently underperformed, which the authors attribute to mismatch with its pretrained projector.Without reinitializing or retraining that projector, deviating from its specialized token format appears to hinder alignment and performance.Source: E Observations · p. 14
Scaling the Dataset
- Limited plausible-content diversity is associated with TRAVL’s lower plausible-video performance relative to a pretrained model, despite improved implausibility detection.The authors call for broader coverage of implausibility types and more varied, complex plausible videos.Source: E Observations · p. 14
- The lack of timestamped training captions makes long-video segmentation difficult, so training videos are limited to under 800 frames to preserve coherent trajectories.Improved timestamp alignment or synthetic supervision is proposed as a route to larger, more balanced corpora.Source: E Observations · pp. 14–15
Trajectory Masks
- On-the-fly CoTracker masks add computational overhead, so tracking uses one center point per patch; denser tracking is left for future work.The authors suggest efficient or precomputed trajectory pipelines as directions to explore.Source: E Observations · p. 15
Beyond Implausibility
- The benchmark focuses on detecting physical violations; grounded captions, affordance predictions, and causal reasoning are proposed as future evaluation targets.Source: E Observations · p. 15