PaperlayerView exact revision on arXiv

Source-linked AI summary

LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference

Jianhao Yuan, Fabio Pizzati, Francesco Pinto, Lars Kunze, Ivan Laptev, Paul Newman, Philip Torr, Daniele De Martini

arXiv:2510.11512v3cs.CVcs.AI

TL;DR

Evaluating video diffusion models’ intuitive physics understanding is difficult when physical correctness is entangled with visual appearance. LikePhys compares denoising-based likelihood preferences for paired valid and invalid videos. Its metric aligns more strongly with human annotations than the evaluator baselines considered, while model results vary across architectures.

  • Problem

    Evaluating video generators’ intuitive physics is difficult because existing approaches can conflate physical correctness with visual appearance or introduce subjective judgments.

    Source: §1 · p. 1

  • Method

    LikePhys uses denoising loss as an ELBO-based likelihood surrogate to compare visually matched valid–invalid video pairs and calculate PPE.

    Source: §1 · p. 1, §3.2 · p. 5

  • Results

    PPE correlates more strongly overall with human annotations than the evaluator baselines, with τ = 0.44; newer DiT-based models have lower errors than early UNet-based models.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    LikePhys provides a means to assess physics understanding in video diffusion models and analyze their limitations and potential for progress.

    Source: §1 · p. 2, §5 · p. 10

    LikePhys requires access to a model’s noise-prediction error, making closed-source models difficult to evaluate.

    Source: §5 · p. 10

Abstract

from arXiv · show

Intuitive physics understanding in video diffusion models plays an essential role in building general-purpose physically plausible world simulators, yet accurately evaluating such capacity remains a challenging task due to the difficulty in disentangling physics correctness from visual appearance in generation. To the end, we introduce LikePhys, a training-free method that evaluates intuitive physics in video diffusion models by distinguishing physically valid and impossible videos using the denoising objective as an ELBO-based likelihood surrogate on a curated dataset of valid-invalid pairs. By testing on our constructed benchmark of twelve scenarios spanning over four physics domains, we show that our evaluation metric, Plausibility Preference Error (PPE), demonstrates strong alignment with human preference, outperforming state-of-the-art evaluator baselines. We then systematically benchmark intuitive physics understanding in current video diffusion models. Our study further analyses how model design and inference settings affect intuitive physics understanding and highlights domain-specific capacity variations across physical laws. Empirical results show that, despite current models struggling with complex and chaotic dynamics, there is a clear trend of improvement in physics understanding as model capacity and inference settings scale.

1 INTRODUCTION

Video diffusion models can generate physically implausible videos, while evaluating their physics understanding is difficult because existing approaches may conflate physical correctness with visual appearance. LikePhys evaluates paired valid and invalid videos through diffusion-model likelihood preferences.

1 INTRODUCTION

  • Evaluating physics understanding in video generators is challenging because visual appearance can confound judgments of physical correctness.VLM judgments can also introduce subjective bias and interpretive variance.Source: §1 · p. 1
  • LikePhys uses video diffusion models’ density estimation to compare the likelihoods of visually matched, physically valid and invalid videos.The method corrupts both videos with noise and uses denoising losses as likelihood proxies to compute Plausibility Preference Error (PPE).Source: §1 · pp. 1–2
  • The benchmark comprises twelve scenarios across four physics domains, designed to isolate specific violations under matched visual conditions.The domains are rigid-body mechanics, continuum mechanics, fluid mechanics, and optical effects.Source: §1 · p. 2
  • The paper presents LikePhys as a training-free evaluation method verified to align with human preference and uses it to analyze model and domain differences.Its stated analyses cover architecture, inference settings, and variation across physics domains.Source: §1 · p. 2

2 RELATED WORK

Prior work has advanced video diffusion architectures and evaluated intuitive physics in general vision and video generation, but adapting those evaluations to text-to-video models remains difficult. LikePhys instead compares denoising-based likelihood preferences on paired videos.

2.1 VIDEO DIFFUSION MODELS

  • Early video diffusion models used 3D UNets, while newer systems adopt DiT-style Transformer backbones; whether they learn physical principles remains unclear.The newer systems advance long-sequence generation, visual quality, and inference efficiency.Source: §2.1 · pp. 2–3

2.2 INTUITIVE PHYSICS UNDERSTANDING

  • Violation-of-expectation studies use plausible and implausible events to assess physics understanding, often with tightly controlled synthetic pairs that isolate a single violation.Adapting this approach to text-conditioned video diffusion is difficult because such models lack image conditioning and are not naturally discriminative.Source: §2.2 · p. 3

2.3 PHYSICS EVALUATION IN VIDEO GENERATIVE MODELS

  • VLM-based evaluators can be affected by visual-appearance bias and prompt interpretation, while other methods rely on image-conditioned generation; LikePhys compares denoising-based preferences on paired videos.Its benchmark depicts valid and invalid examples across four physics domains.Source: §2.3 · pp. 3–4, Figure 2 · p. 3, Figure 3 · p. 4

3 METHODOLOGY

LikePhys operationalizes intuitive physics as a model’s preference for physically valid videos over visually matched invalid ones, using denoising loss as a likelihood surrogate. The method evaluates this preference on a controlled synthetic benchmark spanning 12 scenarios across four physics domains.

3.1 PRELIMINARIES ON VIDEO DIFFUSION MODELS

  • Denoising loss, trained to reconstruct injected noise, serves as an ELBO-based surrogate for negative log-likelihood.The loss is computed by comparing the injected noise with the denoising network’s prediction.Source: §3.1 · p. 4

3.2 PHYSICS UNDERSTANDING AS LIKELIHOOD PREFERENCE

  • A model has ideal intuitive physics when it assigns higher likelihood to every physically valid video than to its invalid counterpart.Because lower denoising loss corresponds to higher likelihood, valid samples should have lower denoising loss than visually matched invalid samples.Source: §3.2 · p. 5
  • PPE measures the rate of pairwise misrankings, aggregating across valid–invalid pairs within each variation and then across variations; lower PPE indicates stronger physics preference.A misranking occurs when the valid sample receives no greater likelihood than the invalid one.Source: §3.2 · p. 5

3.3 EVALUATION BENCHMARK CONSTRUCTION

  • The synthetic benchmark contains 12 scenarios across rigid-body, continuum, fluid, and optical physics, with controlled valid videos and single-violation variants.Videos are rendered at 512 × 512 resolution for 60 frames; each scenario has 10 variations, with visual factors held constant within each variation.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments benchmark pre-trained video diffusion models with LikePhys and examine human alignment, visual-quality disentanglement, model and inference factors, domain differences, and protocol robustness. Recent models perform better, but results vary across scenarios and physical laws.

4.1 MAIN RESULTS

  • Hunyuan T2V (43.6%), Wan2.1–T2V–14B (43.8%), and CogVideoX1.5–5B (43.8%) have the lowest reported PPE among the twelve benchmarked models.Only a handful of models achieve PPE below the 50% random-guess threshold in more than a few scenarios.Source: §4.1 · p. 7
  • PPE correlates more strongly overall with human annotations than the compared automatic evaluators, with Kendall’s τ=0.44.Models with lower PPE tend to be judged by humans as more physically consistent.Source: §4.1 · p. 7, Table 2 · p. 7
  • PPE is largely orthogonal to visual-quality measures: aesthetic quality has no significant correlation (r = -0.05), while motion smoothness and temporal flickering show moderate positive correlations.Subject consistency and background consistency each show very weak correlation (r = -0.01).Source: Table 3 · p. 7, §4.1 · pp. 7–8

4.2 ANALYSIS ON INFLUENCING FACTORS

  • PPE generally declines with larger models and longer context windows, while training-data-size trends are weaker and CFG strength has only a marginal effect.Larger models consistently outperform smaller variants; most models show decreasing PPE as frame count increases, and CFG can be tuned for visual quality without compromising physics understanding.Source: Figure 4 · p. 8, §4.2 · p. 8

4.3 ANALYSIS ACROSS PHYSICS DOMAINS AND LAWS

  • Across physics domains, Fluid Mechanics has the highest average error and greater variability across models; faucet-flow and droplet-fall scenes yield 30–40% PPE, while complex river flows often exceed 70%.Source: Figure 5 · p. 9, §4.3 · p. 9
  • Temporal continuity has the largest variance across models; geometric invariance and optical consistency are better satisfied than laws requiring global coupling through time and space.Spatial continuity and conservation principles such as momentum and mass also yield high errors, while material response remains moderately challenging.Source: Figure 5 · p. 9, §4.3 · p. 9

4.4 ABLATION ON EVALUATION PROTOCOL DESIGN

  • Valid–invalid preference trends are consistent across most sampled timesteps, but the peak separation varies by model and scenario, motivating uniform sampling of ten timesteps.The uniform strategy is presented as a balance between computational efficiency and stable estimation.Source: §4.4 · p. 9, Figure 6 · p. 10
  • Discriminative performance shows no significant change across eight representative prompt variants.The authors attribute this robustness to controlled data generation that isolates physical-law compliance as the error source.Source: §4.4 · pp. 9–10, Figure 6 · p. 10

5 DISCUSSION AND CONCLUSION

LikePhys evaluates video diffusion models’ physics understanding by comparing denoising losses for controlled valid–invalid video pairs. The paper reports limitations in obtaining controlled violations and accessing noise-prediction errors, especially for closed-source models.

5 DISCUSSION AND CONCLUSION

  • LikePhys empirically assesses whether video diffusion models assign higher likelihood to physically valid than invalid videos, without assuming what causes their capacity.The method uses denoising loss on controlled video pairs; the paper does not attribute observed capacity to training data, optimisation, or objectives.Source: §5 · p. 10
  • Controlled physics violations cannot be obtained in real-world data, so broader scenario coverage requires additional simulator or video-editing curation.The authors note this is more costly than text-conditioned generation evaluation, while allowing precise control of valid–invalid pairs.Source: §5 · p. 10
  • Access to a model’s noise-prediction error is required, making LikePhys difficult to apply to closed-source models.The authors describe this requirement as less restrictive in the open-source community and suggest using LikePhys during development to monitor training and select checkpoints.Source: §5 · p. 10

LLM USAGE

The authors report using an LLM for grammar polishing, with author verification of all text, code, and results.

LLM USAGE

  • An LLM was used for grammar polishing, and no text, code, or results were accepted without author verification.The authors state that all technical content, claims, and conclusions are their own.Source: LLM USAGE · p. 15

A DETAILS ON EVALUATION PROTOCOL

The evaluation protocol computes denoising losses for valid and invalid videos, then counts pairwise cases in which the model prefers the invalid video to calculate PPE.

A DETAILS ON EVALUATION PROTOCOL

  • The protocol takes physics-scenario subgroups of valid and invalid videos, a pretrained diffusion model, diffusion timesteps, and noise samples, and outputs PPE.The algorithm specifies M videos in each valid and invalid set.Source: A DETAILS ON EVALUATION PROTOCOL · p. 15
  • For each video, the algorithm accumulates squared noise-prediction error over sampled timesteps and noises, then normalizes the denoising loss.The noised input is formed from the video and sampled noise using the diffusion schedule.Source: A DETAILS ON EVALUATION PROTOCOL · p. 15
  • PPE is the fraction of valid–invalid comparisons in which the invalid video has lower denoising loss, counted as an error.The algorithm totals these errors and divides by the number of comparisons.Source: A DETAILS ON EVALUATION PROTOCOL · p. 15

B DETAILS ON BENCHMARK DATASET GENERATION

The appendix describes benchmark-scenario construction, inference settings and loss comparisons, and a table classifying specific physics laws.

B DETAILS ON BENCHMARK DATASET GENERATION

  • Benchmark scenarios use controlled invalid variants to probe physics violations; examples include altered collision energy for Ball Collision and broken material continuity for Cloth Drape.The appendix describes scenario settings and how invalid cases are generated.Source: B DETAILS ON BENCHMARK DATASET… · pp. 15–16

C ADDITIONAL DETAILS OF INFERENCE SETTINGS

  • Inference uses each model’s recommended resolutions, an Euler discrete scheduler with T = 10, and averaged noise-prediction errors to proxy negative log likelihood.Valid and invalid variants are processed identically, their losses compared pairwise, and PPE counts cases where the invalid sample has lower loss; randomness is fixed for reproducibility.Source: C ADDITIONAL DETAILS OF INFERENCE… · p. 17, Table 4 · p. 17

D.1 CLASSIFICATION OF PHYSICS LAWS

  • Tab. 5 classifies and summarizes specific physics laws.The appendix points to the table for details on the classification.Source: §D.1 · p. 18

D.2 HUMAN STUDY EXPERIMENT DETAILS

The human study rates generated videos for physical consistency, while baseline evaluators use thresholded physics and semantic-adherence scores or a binary judge prompt.

D.2 HUMAN STUDY EXPERIMENT DETAILS

  • Human annotators rate 10 samples per physics scenario per model on a 1–5 physical-consistency scale.The videos use the same prompts as Sec. C, enhanced with GPT5; the annotation protocol follows VideoPhy2.Source: §D.2 · p. 18
  • VideoPhy1 and VideoPhy2 evaluation retains samples only when both physics consistency and semantic adherence exceed their respective thresholds.The cutoffs are >0.5 for VideoPhy1 and >4 for VideoPhy2, after which the passing proportion is computed.Source: §D.2 · p. 18
  • Qwen2.5-VL judges each video with a strict binary prompt asking whether its dynamics align with the physical world.The response is 0 for no or for physics-law violations, and 1 for yes.Source: §D.2 · p. 18

D.3 INFLUENCING FACTOR EXPERIMENT DETAILS

The ablations test how context-window length and classifier-free guidance strength affect discrimination between valid and invalid physics cases across selected scenarios and models.

D.3 INFLUENCING FACTOR EXPERIMENT DETAILS

  • The protocol ablations vary input-frame count and classifier-free guidance (CFG) strength to assess valid-versus-invalid physics discrimination.The experiments use five representative scenarios and five exemplar models.Source: §D.3 · p. 18
  • Input-frame experiments test 8 to 64 frames in eight-frame increments, with AnimateDiff limited to 32 frames.The selected scenarios are Ball Drop, Block Slide, Fluid, Cloth, and Shadow; models include AnimateDiff, ModelScope, CogVideoX, Hunyuan T2V, and LTX.Source: §D.3 · p. 18
  • CFG experiments evaluate guidance scales from 1.0 to 8.0 for their effect on valid-versus-invalid physics discrimination.Source: §D.3 · p. 18

E APPLICABILITY TO OTHER INTUITIVE PHYSICS BENCHMARKS

LikePhys is applied unchanged to the IntPhys Development split, where block-level PPE profiles vary across principles and reflect domain-specific trends observed in other evaluations.

E APPLICABILITY TO OTHER INTUITIVE PHYSICS BENCHMARKS

  • On IntPhys Development, LikePhys reports PPE for object permanence (O1), shape constancy (O2), and spatio-temporal continuity (O3), plus their average.The table reports PPE in percent, with lower values better.Source: E APPLICABILITY TO OTHER INTUITIVE… · p. 18, Table 6 · p. 20
  • Models show distinct principle-level profiles, with some performing better on continuity in O3 than permanence in O1.The authors say these profiles mirror domain-specific trends in their rigid-body and optics evaluations.Source: E APPLICABILITY TO OTHER INTUITIVE… · p. 18
  • The benchmark examples label impossible cases including missing or misshapen shadows, impossible reflections, cloth penetration, and hovering or irregular block motion.Source: E APPLICABILITY TO OTHER INTUITIVE… · p. 19

F VISUAL SAMPLES

The appendix provides visual samples across all twelve benchmark scenarios, showing one valid and two invalid cases per scenario.

F VISUAL SAMPLES

  • Figure 7 samples Ball Drop, Ball Collision, Pendulum Oscillation, and Block Slide.Source: Figure 7 · p. 21
  • Figure 8 samples Droplet Fall, Faucet Flow, Cloth Drape, and Cloth Waving.Source: Figure 8 · p. 22
  • Figure 9 samples River Flow, Moving Shadow, Pyramid Impact, and Orbit Shadow.Source: Figure 9 · p. 23

Related papers

Loading 2510.11512v3…