PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoPhy: Evaluating Physical Commonsense for Video Generation

Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, Aditya Grover

arXiv:2406.03520v2cs.CVcs.AIcs.LG

TL;DR

Assessing whether text-to-video models capture physical commonsense is difficult because relevant video qualities are intertwined. VideoPhy benchmarks generated videos for physical commonsense and text adherence; across evaluated models, these capabilities remain limited, with CogVideoX-5B achieving both in 39.6% of cases.

  • Problem

    Determining whether generated videos reflect physical commonsense is difficult because motion, text adherence, physical commonsense, and temporal consistency are intertwined.

    Source: §3.1 · p. 5

  • Method

    VideoPhy evaluates generated videos for text adherence and physical commonsense, and VideoCon-Physics provides an automatic evaluator for the benchmark.

    Source: §3.1 · p. 5, §3.3 · p. 6, §8 · p. 11

  • Results

    Across evaluated models, physical commonsense and semantic adherence are significantly lacking; CogVideoX-5B achieves both caption adherence and physics in 39.6% of cases.

    Source: §5.1 · p. 7, §8 · p. 11

  • Takeaways & Limitations

    The findings indicate that existing video-generation methods remain far from general-purpose world simulators.

    Source: §8 · p. 11

    The evaluation excludes Sora, Kling AI, and Genmo because API support for their videos was unavailable.

    Source: §4 · p. 6

Abstract

from arXiv · show

Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., CogVideoX) and closed models (e.g., Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVideoX-5B, generates videos that adhere to the caption and physical laws for 39.6% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we propose an auto-evaluator, VideoCon-Physics, to assess the performance reliably for the newly released models.

1 Introduction

VideoPhy assesses whether text-to-video models generate videos that follow physical commonsense, using human judgments of real-world interactions. It also introduces an evaluator intended to make that assessment scalable.

1 Introduction

  • Although T2V models can generate complex scenes and realistic motions, how well their videos adhere to physical laws remains unclear despite interest in using them for real-world agents.The paper describes intuitive physics as the ability to understand object interactions through real-world experience, without formal physics education.Source: §1 · p. 2
  • Simulation-based ground truth is difficult to establish, and benchmarks such as VBench [40] [63] assess video qualities without specifically evaluating adherence to physical laws.The paper notes challenges in recovering 3D geometry, tuning simulation parameters, and rendering results; simulations are approximations rather than ground truth.Source: §1 · p. 2
  • VideoPhy evaluates intuitive behavior and dynamics of solids and fluids [10], using 688 human-verified captions covering solid-solid, solid-fluid, and fluid-fluid interactions.Its three-stage pipeline uses an LLM to propose captions, humans to verify them, and physics-simulation-based annotations of rendering or motion complexity.Source: §1 · pp. 2–3
  • CogVideoX-5B [113] follows the text and generates physically accurate videos for only 39.6% of instances, while existing models overall struggle with caption adherence and physical commonsense.The authors also report that models perform particularly poorly on prompts requiring solid-solid interaction.Source: §1 · p. 3
  • Qualitative analysis finds that models often misidentify objects or their material properties, undermining plausible dynamics—for example, rigid objects should not deform over time.The reported weakness is especially pronounced for solid-solid prompts such as a ball bouncing on the floor or a hammer hitting a nail.Source: §1 · p. 3
  • VideoCon-Physics [3], finetuned on generated videos and human annotations, improves over Gemini-Pro-Vision-1.5 [84] by 9 points in semantic adherence and 15 points in physical commonsense on unseen prompts.The evaluator also generalizes to unseen generative models, supporting scalable assessment of future models.Source: §1 · p. 3

2 VIDEOPHY Dataset

VIDEOPHY is a curated benchmark of real-world physical interactions, built through caption generation, human verification, and difficulty annotation. It spans material types and simulation challenges, with metadata describing its captions and generated videos.

2 VIDEOPHY Dataset

  • VIDEOPHY targets diverse daily activities, material interactions, and graphics-simulation complexity, spanning rigid and deformable bodies, fluids, metals, sand, and other materials.Its guidelines include interactions across solid-solid, solid-fluid, and fluid-fluid states of matter, as well as differences in modeling difficulty such as non-Newtonian ketchup versus Newtonian water.Source: §2 · pp. 3–4
  • GPT-4 [73] generated 1,000 candidate captions across solid-solid, solid-fluid, and fluid-fluid interactions; authors human-verified them for clarity, manageable complexity, and category accuracy, retaining 688 captions (289, 291, and 108, respectively).Source: §2 · pp. 3–4
  • 688 verified captions comprise 289 solid-solid, 291 solid-fluid, and 108 fluid-fluid interactions.The authors prioritize caption quality because expert human verification is labor-intensive and limits dataset scale.Source: §2 · p. 4
  • Two graphics researchers independently labeled each caption easy or hard based on physics-engine simulation complexity, resolving disagreements for fewer than 5% of instances.Difficulty depends on material-model complexity and numerical challenges such as velocity; labels are comparable only within, not across, interaction categories.Source: §2 · pp. 4–5
  • The benchmark contains 11,330 generated videos, captions averaging 8.5 words, and 138 unique actions.Figure 3 presents the most frequent verbs and their direct nouns, illustrating the range of actions and entities represented.Source: §2 · p. 5

3 Evaluation

VideoPhy evaluates generated videos through binary judgments of semantic adherence and physical commonsense, using qualified human annotators and a fine-tuned automatic rater. The section defines these metrics and describes how human and automated judgments are obtained.

3.2 Human Evaluation

  • Human evaluation uses binary judgments focused on semantic adherence and physical commonsense, dimensions that can be intertwined with other video characteristics.Fourteen workers with high-school physics backgrounds passed a qualification test, judged caption-video pairs without model identities, and scored the two metrics independently. They also assessed the plausibility of static scenes and noisy patterns; annotators were not asked to enumerate physics violations, so advanced physics education was unnecessary.Source: §3.1 · p. 5, §3.2 · pp. 5–6

3.1 Metrics

  • Semantic adherence (SA) measures whether captioned actions, events, entities, and relationships appear in the video, while physical commonsense (PC) measures whether actions and object states follow real-world physics.Both are annotated as binary scores; the evaluation computes the fractions with SA = 1, PC = 1, and both equal to 1. PC is treated as independent of semantic adherence.Source: §3.1 · p. 5

3.3 Automatic Evaluation

  • VideoCon-Physics fine-tunes the 7B-parameter VideoCon model on human annotations to assess semantic adherence and physical commonsense, with effectiveness measured by ROC-AUC against human judgments.The zero-shot VideoCon model is not expected to perform well because it was not trained on generated videos or equipped to judge physical commonsense; GPT-4Vision and Gemini-1.5-Pro-Vision are presented as baselines in Appendix J.Source: §3.3 · p. 6

4 Setup

The setup evaluates twelve open and closed text-to-video models on VIDEOPHY, using human judgments on held-out prompts both to benchmark the videos and to train VIDEOCON-PHYSICS.

Video Generative Models

  • Twelve open and closed T2V models were tested, including ZeroScope, CogVideoX-2B and 5B [113], Lumiere [7], Dream Machine, and Pika.The testbed also included LaVIE, VideoCrafter2, OpenSora, StableVideoDiffusion, Gen-2, and other listed models.Source: §4 · p. 6

Dataset setup

  • The 688 VIDEOPHY prompts were split equally into 344-prompt training and test sets, with similar matter-state and complexity distributions.Human annotations on test-set videos were used for benchmarking, while training-set annotations trained VIDEOCON-PHYSICS.Source: §4 · p. 6

Benchmarking

  • For benchmarking, each test prompt/model pair had one video judged by three annotators; majority-voted SA and PC achieved 75% and 70% inter-annotator agreement, respectively.The lower agreement for physical-commonsense judgments indicates that annotators found them more subjective; the benchmark collected 24,500 annotations.Source: §4 · pp. 6–7
  • Sora [64], Kling AI, and Genmo were excluded because lack of API support prevented access to their videos.Source: §4 · p. 6

Training set for VIDEOCON-PHYSICS

  • VIDEOCON-PHYSICS training used two videos per prompt from nine models, with one annotator per video and 12,000 labels split evenly between semantic adherence and physical commonsense.The model was fine-tuned on Yes/No judgments using a multimodal template; separate classifiers offered no added benefit over the multitask classifier, and annotation collection cost $3,500.Source: §4 · pp. 6–7

5 Results

VideoPhy results show that current text-to-video models often fail to combine caption adherence with physical correctness, and the analysis identifies recurring weaknesses across scene types and physical behaviors. VideoCon-Physics outperforms diverse automatic-evaluation baselines on semantic-adherence and physical-commonsense judgments.

5.1 Performance on VIDEOPHY Dataset

  • CogVideoX-5B satisfies both caption adherence and physics laws in 39.6% of cases, and its advantage over other models is statistically significant (p<0.0001).Its physical-commonsense performance is 53%.Source: §5.1 · p. 7
  • Human annotations vary because evaluators differ in their tolerance for commonsense violations in imperfect videos.The paper expects annotations to align more closely as generative models improve.Source: §5.1 · p. 7
  • An open model scores 34.1% on VIDEOPHY; OpenSora performs worst there, and Dream Machine scores 61.9% for semantic adherence versus 21.8% for physical commonsense.The passage also says increasing network capacity improves capture of underlying physical constraints. Pika achieves positive judgments for semantic adherence and physical commonsense in 19.7% of cases.Source: §5.1 · p. 8
  • All evaluated models perform worst on solid-solid interactions, with CogVideoX-5B reaching 24.4%, while Pika performs best on fluid-fluid interactions.The results indicate that performance depends on the states of matter involved and identify solid-solid scenes as a target for improvement.Source: §5.1 · p. 8
  • Semantic adherence and physical commonsense decrease as caption complexity rises, and both correlate positively with video quality but negatively with motion.The analysis suggests that more-motion scenes prompt more semantic-adherence and physical-commonsense mistakes, although absolute benchmark performance remains poor.Source: §5.1 · p. 8

5.2 Qualitative Analysis

  • Qualitative comparisons show model-specific failures in fluid dynamics, rigid geometry, and meaningful motion, while CogVideoX-5B still produces unnatural material deformations.CogVideoX-5B tends to distinguish objects and use simpler backgrounds, but can fail to represent material properties consistently.Source: Figure 4 · p. 8, §5.2 · p. 9, Figure 5 · p. 9
  • Recurring physical failures include mass-conservation violations, violations of Newton’s laws, implausible solid or fluid behavior, and objects penetrating one another.Examples include objects changing velocity without external force and rigid objects deforming over time.Source: §5.2 · p. 9

5 Results

  • VideoCon-Physics outperforms diverse baselines, including GPT-4Vision and Gemini-1.5-Pro, on ROC-AUC for semantic-adherence and physical-commonsense judgments.CogVideoX and Dream Machine were not included in the evaluator’s training set.Source: §5.1 · p. 7, Table 4 · p. 9

6 VIDEOCON-PHYSICS: Automatic Evaluator for VIDEOPHY Dataset

VideoPhy supplements its dataset with VideoCon-Physics, an automatic evaluator for semantic adherence and physical commonsense. Its tests show improved judgments on unseen prompts and generative models, rankings that largely track human evaluations, and remaining limitations.

VideoCon-Physics generalizes to unseen prompts

  • On unseen prompts, VideoCon-Physics exceeds zero-shot VideoCon by 17 points for semantic adherence and 19 points for physical commonsense judgment.The evaluator is trained on videos from the same models used for its training, and its agreement is higher for semantic adherence than physical commonsense.Source: §6 · p. 9

Judgments by multimodal foundation models

  • GPT-4-Vision judgments are close to random, while Gemini-Pro-Vision-1.5 scores 73 points for semantic adherence but 54 points for physical commonsense.These results indicate that existing multimodal foundation models struggle to judge physical commonsense in this dataset.Source: §6 · p. 10

VideoCon-Physics generalizes to unseen generative models

  • On unseen generative models, VideoCon-Physics outperforms VideoCon by 15 points for semantic adherence and 15 points for physical commonsense judgment.The ablated evaluator is trained on human annotations for VideoCrafter2, ZeroScope, LaVIE, OpenSora, SVD-T2I2V, and Gen-2; Table 5 reports results averaged over three unseen video models on testing prompts.Source: §6 · p. 10, Table 5 · p. 10

Automatic leaderboard reliably tracks human leaderboard

  • Automatic and human leaderboards have similar rankings for most models, supporting VideoCon-Physics as a reliable indicator for future model development.For open models, the automatic order is CogVideoX-5B>VideoCrafter2>LaVIE>CogVideoX-2B>SVD-T2I2V>ZeroScope>OpenSora, while the human order swaps LaVIE and CogVideoX-2B; closed models show similar trends, though Pika scores relatively low automatically.Source: §6 · p. 10

Finetuning video models

  • After video-model finetuning on VideoPhy data, semantic adherence decreases significantly while physical commonsense remains unchanged.The authors attribute this likely to limited training samples, optimization challenges, and the nascent state of video finetuning.Source: §6 · p. 10

7 Related Work

Prior work advances video generation, evaluation, and physical simulation, but existing video metrics and generative approaches leave physical commonsense in generated videos insufficiently assessed. VideoPhy focuses on evaluating whether generated videos adhere to physical laws.

Video Generation Models

Evaluating Video Generation Models

  • FVD [99] [19] and CLIPScore [82] do not reliably assess physical commonsense, while broader benchmarks [40] [63] and image-text alignment methods [49] remain insufficient for VIDEOPHY.FVD requires difficult-to-obtain reference videos for novel scenes, favors video quality, and can miss unrealistic motion [16] [92]; V-Bench [40] uses hierarchical quality evaluation, while EvalCrafter [63] proposes 17 objective metrics. VideoCon-Physics is trained on diverse generated videos and human annotations for reliable judgments.Source: §7 · p. 11

Physics Modeling

  • Physics modeling spans rigid and deformable solids, material-specific models, fluids, and interactions, while physics-informed generation often relies on 3D reconstructions or limited 2D simulation [8, 90, 72, 44, 94, 15, 47, 96, 51, 54, 55, 9, 109, 69, 117, 89, 111, 67, 110, 119, 62].Prior simulated physics in video generation is constrained to 2D by limited 3D information; VideoPhy instead identifies whether generated videos adhere to physical laws.Source: §7 · p. 11

8 Conclusion … F VIDEOCON details

VideoPhy introduces a benchmark for physical commonsense in generated videos and finds that evaluated text-to-video models lack both physical commonsense and semantic adherence. It also presents VideoCon-Physics for scalable automated evaluation.

8 Conclusion; B Data Licensing

  • VideoPhy introduces a physical-commonsense benchmark, and VideoCon-Physics enables cheap, scalable evaluation; generated videos follow model owners’ licenses, while captions and annotations use the MIT License.The benchmark’s captions and human annotations are licensed under MIT.Source: §8 · p. 11, B Data Licensing · p. 20

A Limitations

  • The 688-caption dataset uses a three-stage curation pipeline, but its coverage, model set, and human evaluations have important limitations.Future work should expand coverage to physics areas such as projective geometry, include high-performance models, and assess annotator demographic effects; the mostly US- and Canada-based AMT workers may reflect Western perceptual biases. The paper also calls for quantifying societal biases in generated videos and developing safe-deployment methods [102] [6].Source: A Limitations · p. 20

C Video Generative Models

  • The benchmark evaluates open and closed models, including CogVideoX [113], Gen-2 [27], Pika, and Lumiere [7].For SVD, the study uses an image-to-video model conditioned on SD-XL-Base-1.0 images, calling the pipeline SVD-T2I2V.Source: C Video Generative Models · p. 20

E Human Annotation Screenshot

F VIDEOCON details

  • VideoCon answers Yes or No from multimodal templates for semantic adherence and physical commonsense, scoring the probability of Yes; physical-commonsense evaluation is independent of the caption.The physical-commonsense template therefore omits the conditioning caption.Source: F VIDEOCON details · pp. 22–23, Figure 11 · p. 23

G Fine-Grained Diversity Analysis · H Results for Task complexity

The section visualizes collection statistics across physical interaction categories and compares video-generation models across task complexity.

G Fine-Grained Diversity Analysis

H Results for Task complexity

I Fine-Grained Results

The section reports semantic-adherence and physical-commonsense scores across video-generation models, physical-interaction categories, and difficulty levels.

I Fine-Grained Results

  • Scores are broken down by model, interaction category (solid-solid, solid-fluid, and fluid-fluid), and difficulty level (0 or 1).The reported measures are semantic adherence and physical commonsense.Source: I Fine-Grained Results · p. 23

J Automatic Evaluation Baselines · K Inference Details

The paper evaluates automatic video assessments with a zero-shot GPT-4Vision baseline using sampled frames, and with Gemini-Pro-Vision-1.5 using the full video. It also provides inference configurations for the video-generation models in Table 9.

J Automatic Evaluation Baselines

K Inference Details

L Training Details for VIDEOCON-PHYSICS … O Correlation with video quality and motion

VideoCon-Physics is fine-tuned to evaluate semantic adherence and physical commonsense, and the authors describe its uses in model selection, data filtering, and post-training. Its scores track human rankings and relate positively to video quality but negatively to motion, although high-quality videos can still perform poorly on the benchmark.

L Training Details for VIDEOCON-PHYSICS

  • VideoCon-Physics is fine-tuned with LoRA for 5 epochs, using 2 A6000 GPUs and 32 video frames resized to 224 × 224.The setup uses LoRA r = 32 and α = 32, dropout = 0.05, Adam, a 50-step linear warmup followed by linear decay, and a peak learning rate of 1e −4; 32 segments are formed by sampling the middle frame of each.Source: L Training Details for VIDEOCON-PHYSICS · pp. 24–25

N Applications of VIDEOCON-PHYSICS

  • Model builders can use VideoCon-Physics to evaluate candidate models on VIDEOPHY at scale, then send top candidates to human workers for more accurate evaluation.The authors frame this as a way to perform downstream model verification cheaply and reliably.Source: N Applications of VIDEOCON-PHYSICS · p. 26
  • Data builders can use VideoCon-Physics to filter video-text data that lacks semantic adherence or physical commonsense.The proposed filtering use addresses low-quality internet data used to train foundation models [31].Source: N Applications of VIDEOCON-PHYSICS · p. 26
  • VideoCon-Physics can serve as a reward model in video-generation post-training, providing feedback to refine generated content.The authors situate this use within alignment with human or AI feedback [88] [83] [5] [101] [56].Source: N Applications of VIDEOCON-PHYSICS · p. 26

O Correlation with video quality and motion

  • Physical commonsense and semantic adherence correlate positively, though not strongly, with video quality and negatively with video motion.The authors measure quality with the LAION aesthetic classifier and motion with RAFT optical flow, then calculate Pearson correlations over VIDEOPHY.Source: O Correlation with video quality and… · p. 27, Table 11 · p. 28

M Automatic and Human Leaderboard

  • The automatic leaderboard reliably tracks the human leaderboard for open and closed video models.The human ranking uses the joint SA = 1, PC = 1 metric; the automatic ranking averages semantic adherence and physical commonsense scores.Source: M Automatic and Human Leaderboard · p. 25, Table 10 · p. 28

O Correlation with video quality and motion

  • Gen-2 has a LAION aesthetics classifier score of 5.8 but poor semantic adherence and physical commonsense performance, with a score of 7.6.The authors caution that the positive correlation between quality and physical commonsense does not mean models perform well overall on VIDEOPHY.Source: O Correlation with video quality and… · p. 28

P Finetuning video model with VideoPhy data · Q More Qualitative Examples of Poor Physical Commonsense · R Examples from diverse states of matter and complexity

Finetuning Lumiere-T2I2V on selected VideoPhy examples reduced semantic adherence without improving physical commonsense. The qualitative examples illustrate physical-law violations across generative models and show majority-scored examples from different material categories.

P Finetuning video model with VideoPhy data

Q More Qualitative Examples of Poor Physical Commonsense

  • LaVIE and Gen-2 examples show failures including solid deformation, nonphysical penetration, inconsistent mass, and motion that ignores gravity.Examples include spoons deforming or passing through liquid, changing object geometry, and water appearing static despite gravity.Source: Figure 15 · p. 29, Figure 16 · p. 30

R Examples from diverse states of matter and complexity

  • VideoCrafter2, ZeroScope, and OpenSora examples depict violations involving gravity, momentum, object deformation, and conservation of mass.Examples include a floating bottle, changing object geometry, and objects moving contrary to expected momentum.Source: Figure 17 · p. 31, Figure 18 · p. 32, Figure 19 · p. 33
  • SVD-T2I2V, Pika, and Lumiere-T2V examples include unnatural fluid behavior, deformation, and objects appearing, disappearing, or changing form.The examples also show drops floating against gravity and tea flowing through a cup.Source: Figure 20 · p. 34, Figure 21 · p. 35, Figure 22 · p. 36
  • Lumiere-T2I2V, CogVideoX-2B and 5B, and Dream Machine examples show violations of motion, mass conservation, object consistency, and nonpenetration.Examples include floating objects, spontaneous appearances, changing geometry, and liquids passing through a cup.Source: Figure 23 · p. 37, Figure 24 · p. 38, Figure 25 · p. 39
  • Figures 27–29 contrast majority physical-commonsense scores of 0 and 1 in fluid-fluid and solid-solid examples.Illustrated failures include upward-floating droplets, changing liquid volume or color, illogical clock-hand motion, and inconsistent object geometry.Source: R Examples from diverse states of… · p. 30, Figure 27 · p. 41, Figure 28 · p. 42

Related papers

Loading 2406.03520v2…