VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation

Motivation and evaluation

• Physical likelihood can often be judged by humans using real-world experience without formal physics education, while human evaluation is expensive and difficult to scale.The introduction contrasts this assessment with simulation-based comparisons that require precise pose, geometry, dynamics, air pressure, and material properties.Source: §1 · pp. 3–4

Motivation and prior work

• Prior benchmarks leave gaps: Physics-IQ [44] has unclear agreement with human judgment, PhyGenBench [36] uses 160 prompts and one-law associations, and VIDEOPHY [5] omits law violations.Comparisons to ground-truth simulations also face limits in rendering diverse materials and accurately modeling complex interactions [1] [47] [5] [36].Source: §1 · p. 3

Dataset and evaluation design

• VideoPhy-2 covers 197 diverse actions with 3,940 detailed prompts, candidate physical rules, and human ratings of semantic adherence, physical commonsense, and rule compliance.Vision-language models help compile candidate rules, such as a tennis ball following a parabolic trajectory under gravity.Source: §1 · pp. 3–4

Results

• Wan2.1-14B achieves 32.6% joint semantic-and-physical performance, falling to 22% on the hard subset; models struggle most with mass and momentum conservation.The joint score requires high semantic adherence and physical commonsense.Source: §1 · p. 4
• On unseen prompts, VideoPhy-2-AutoEval improves relative correlation over Gemini-2.0-Flash-Exp by 81% for semantic adherence and 236% for physical commonsense.The evaluator also scores physical commonsense and classifies physical-rule grounding.Source: §1 · p. 4

Seed Actions (Stage 1)

• 197 actions remain after two annotator groups select physical-commonsense-relevant activities and semantic duplicates are removed.From over 600 candidate actions, filtering yields 232 before refinement; the final set includes 54 object-interaction actions and 143 physical and sports activities.Source: §2 · p. 4

LLM-Generated Prompts (Stage 2)

• The dataset pairs actions with 3,940 prompts designed around visible interactions, diverse entities, and multiple events while excluding nonvisual details.Gemini-2.0-Flash-Exp generates 20 prompts per action; Mistral-NeMo-12B-Instruct upsamples captions with visual detail while preserving their semantic meaning [1].Source: §2 · p. 5

Candidate physical rules and laws (Stage 3)

• Physical rules are generated from captions of model-generated videos rather than directly from prompts, grounding the rules in video details.Gemini-2.0-Flash-Exp proposes three rules per video, and human annotators add violated rules not covered by the predefined set; captioning first produced higher-quality rules than prompting directly from video.Source: §2 · p. 5

Construction of the Hard Subset (Stage 4)

• 60 of 197 actions were selected as hard cases after CogVideoX-5B [58] failed to generate prompt-adherent, physically commonsense videos, yielding 1,200 hard-subset prompts.These actions include physics-rich interactions, state changes, balancing, and complex motions.Source: §2 · p. 5

Data Analysis

• 3,940 captions—5.72× VIDEOPHY—with average original and upsampled lengths of 16 and 138 tokens (1.88× and 16.2×), accompany 102K human annotations.The annotations cover semantic adherence, physical commonsense, and physical rules across video generative models; Figure 4 shows the diversity of root verbs and direct nouns.Source: §2 · p. 5

3.1 Metric

• The evaluation uses fine-grained rating feedback rather than pairwise preferences and extends prior work [5] by assessing adherence to diverse physical rules and laws.Ratings use a 5-point scale, unlike prior work [5], which did not collect dense rating feedback.Source: §3.1 · p. 6
• Semantic adherence and physical commonsense receive separate 5-point ratings, while physical-rule judgments classify each rule as followed, violated, or cannot be determined.Annotators can also note additional rule violations; the cannot-be-determined category accounts for rules that may not be visually grounded.Source: §3.1 · p. 6, §3.2 · p. 6
• Joint performance is the fraction of videos with both semantic adherence and physical commonsense ratings of at least 4; physical commonsense is assessed independently of the prompt.The study does not report PC ≥ 4 conditioned on SA ≥ 4, since a model could obtain a misleadingly high posterior by matching very few prompts.Source: §3.1 · p. 6

3.2 Human Evaluation

• Twelve qualified Amazon Mechanical Turk annotators evaluate semantic adherence separately from physical commonsense, seeing the original prompt only for the semantic task.For physical evaluation, annotators judge the video alone for overall commonsense, specific physical rules, and observable violations, avoiding prompt-related evaluation bias noted in prior work [5].Source: §3.2 · pp. 6–7

3.3 Automatic Evaluation

• Existing video-language models struggle to agree with human judgments on semantic adherence, physical commonsense, and rule classification, so the authors fine-tune a 7B-parameter evaluator on 50K human annotations.VideoPhy-2-AutoEval uses a shared backbone to predict semantic and physical-commonsense scores on a 1–5 scale and classify physical rules on a 0–2 scale.Source: §3.3 · p. 7

Video generative models

• The evaluation covered five open and two closed text-to-video models, with HunyuanVideo-13B and VideoCrafter2 limited to 77-token prompts.The models were CogVideoX-5B [58], VideoCrafter2, HunyuanVideo-13B [31], Cosmos-Diffusion-7B [1], Wan2.1-14B, OpenAI Sora, and Luma Ray2. Videos were shorter than 6 seconds; other closed models, including Veo2 and Kling, were excluded because API access was unavailable.Source: §4 · p. 7

Dataset setup

• The VIDEOPHY-2 split contains 3,350 training prompts and 590 test prompts across 197 actions, with 17 and 3 captions per action, respectively.The test set benchmarks models, while the training set supports training VideoPhy-AutoEval; the data-driven evaluator approach uses human annotations across multiple tasks, similar to [5].Source: §4 · p. 7

Benchmarking

• Three annotators assessed generated videos for semantic adherence, overall physical commonsense, and physical-rule violations, with agreement ranging from 75% to 80%.Each model generated one video for each of 590 test prompts, except Sora (60 videos) and Ray2 (394); CogVideoX-5B defined a hard subset of 60 actions where it had zero joint performance. Overall physical-commonsense judgments were intended to extend beyond predefined rules. The benchmark collected 10.2K semantic-adherence, 10.2K physical-commonsense, and 30.6K physical-rule annotations at a cost of $2600.Source: §4 · pp. 7–8

Training set for VIDEOPHY-2-AUTOEVAL

• VideoPhy-AutoEval training sampled one video per training caption from HunyuanVideo-13B, Cosmos-Diffusion-7B, or CogVideoX-5B and collected approximately 50K human annotations.The training annotations cost $3515. Evaluation compares semantic-adherence and physical-commonsense scores using Pearson correlation, joint-score prediction using accuracy and F1, and physical-rule classification using accuracy.Source: §4 · p. 8

5.1 Main Results

• Wan2.1-14B achieves only 32.6% joint performance on the full split and 21.9% on the hard split; closed models do not necessarily outperform open models.Performance on physical activities and sports is generally lower than on object interactions, and physical commonsense is not well-correlated with aesthetics or motion quality.Source: §5.1 · pp. 9–10
• Conservation of momentum and mass are among the most frequently violated laws, each with 40% violation scores, while reflection and buoyancy have scores below 20%.Violation scores measure the fraction of associated physical rules violated in human-annotated videos.Source: Figure 5 · p. 9, §5.1 · p. 10
• Qualitative examples show unnatural reverse motion, solid-object deformation, and failures even in Wan2.1-14B, including a rock accelerating uphill.Other examples include a hot-air balloon shrinking and expelling water, as well as a javelin expelling sand before impact.Source: Figure 6 · p. 9, §5.1 · p. 10, Figure 7 · p. 10

5.2 VIDEOPHY-2-AUTOEVAL

• VIDEOPHY-2-AUTOEVAL achieves relative correlation gains of 47.4% on unseen prompts and 49% on unseen video models over the best-performing baselines.It also exceeds Gemini-2.0-Flash-Exp by 81% in semantic adherence and 236% in physical-commonsense judgments.Source: Table 4 · p. 10, §5.2 · p. 11
• VIDEOPHY-2-AUTOEVAL maintains a strong balance between joint accuracy and F1 scores and gains 32.9% and 27.7% in physical-rule classification over Gemini-2.0-Flash-Exp on unseen prompts and unseen video models, respectively.The joint-score comparison is against VideoCon-Physics; the classification gains are reported for the unseen-prompt and unseen-model settings.Source: §5.2 · p. 11, Table 5 · p. 11, Table 6 · p. 11

6 Conclusion

• VIDEOPHY-2 evaluates physical commonsense in generated videos and reveals a large gap in prompt alignment and physical adherence.The work frames the benchmark as a way to assess video generative models as general-purpose world simulators.Source: §6 · p. 11
• VIDEOPHY-2 provides physical-law violations and an auto-rater for scalable evaluation.Together, these resources advance understanding of current video generative models as general-purpose world simulators.Source: §6 · p. 11

A Related Work

• Rapid video-generation progress makes robust physical-commonsense benchmarks crucial to realizing video models as “world simulators”.The related-work discussion is organized around key areas of prior research.Source: A Related Work · p. 16
• VIDEOPHY-2 extends prior work by addressing key limitations in existing benchmarks and evaluation methods.Source: A Related Work · p. 16

A.1 Video Generation Models

• Text-to-video models use diffusion approaches such as SVD [8] and Sora [38], or autoregressive approaches such as VideoPoet and CogVideo [23].Diffusion models use multi-stage training in latent space; autoregressive models predict future frames from past frames.Source: §A.1 · p. 16
• Despite impressive visual fidelity, these models’ ability to capture physical principles remains an open question, motivating rigorous benchmarks such as VIDEOPHY-2.Source: §A.1 · p. 16
• Interactive video generation, explored by Genie, expands potential applications beyond static content creation.Source: §A.1 · p. 16

• VBench [25] uses hierarchical evaluation covering motion smoothness, background consistency, and overall visual fidelity.Source: §A.2 · p. 16
• EvalCrafter [37] proposes 17 objective metrics focused on different aspects of video quality.Source: §A.2 · p. 16
• These benchmarks do not isolate or systematically evaluate physical commonsense in generated videos.Source: §A.2 · p. 16

A.3 Benchmarks for Physical Understanding in Video Generation

• VIDEOPHY-2 expands VIDEOPHY [5] from 688 prompts to 3940 prompts, focusing on real-world actions and adding 5-point human ratings and explicit physical-rule annotations.VIDEOPHY [5] evaluates semantic adherence and physical plausibility; VIDEOPHY-2 shifts from material interactions to real-world actions.Source: §A.3 · p. 16
• Physics-IQ [44] measures predicted continuations against ground-truth videos, but is designed primarily for video prediction rather than open-ended text-to-video generation.Extending it to longer or more complex multi-event scenarios presents challenges.Source: §A.3 · p. 16
• PhyGenBench [43] introduces PhyGenEval for structured physical-reasoning evaluation across 160 prompts, but its relatively small scale makes generalization difficult and maps each prompt one-to-one to physical laws despite real-world interactions among multiple principles.VIDEOPHY-2 expands VIDEOPHY’s 688 prompts to 3940, shifts focus from material interactions to real-world actions, and adds a 5-point Likert scale for human evaluation and explicit annotations of physical rules in each video.Source: §A.3 · p. 16
• WorldSimBench [47] aligns model outputs with numerical solvers, but VIDEOPHY-2 instead uses real-world videos to avoid the sim-to-real gap and reflect practical physical dynamics.Simulations may not capture the complexity and variability of real-world interactions.Source: §A.3 · p. 17

A.4 Automatic Evaluation Methods

• VideoPhy-2-AutoEval uses a larger, more diverse dataset with explicit physical-rule annotations to improve alignment with human assessments while retaining interpretability.The section contrasts it with traditional video-quality metrics and earlier VLM-based evaluation approaches, which are not designed for physical plausibility or remain unreliable at assessing physical commonsense.Source: §A.4 · p. 17

A.5 Physical Reasoning in AI

• VideoPhy-2 provides a large-scale, real-world benchmark that bridges theoretical physical reasoning and practical video generation.The paper also characterizes its contributions as an action-centric benchmark, a fine-grained evaluation framework with explicit rule annotations, and a robust, human-aligned automatic evaluator.Source: §A.5 · p. 17

B List of Actions

• The benchmark spans simple motions, object interactions, and complex physical activities, with Table 9 listing 197 actions.Examples include walking through snow, peeling fruit, and pole vault.Source: B List of Actions · p. 17

C List of Hard Actions

• The hard-action subset contains 60 actions selected using CogVideoX-5B as a baseline, including challenges involving rapid motion, intricate interactions, and structural deformation.The subset is intended to test stability, object manipulation, and real-world physics beyond simple motions.Source: C List of Hard Actions · p. 17

D Diversity of prompts for diverse actions

• Prompt diversity varies settings and verb-noun combinations within each action to test models across a broad spectrum of physical interactions.Examples include rowing with boats, paddles, or water, and knitting with yarn, needles, or fabric.Source: D Diversity of prompts for diverse… · pp. 17–18

E LLM-generated captions prompt

• Gemini-2.0-Flash-Exp was given a prompt to generate diverse captions for each action.The section points to Table 11 for the prompt; its contents are not included in the supplied passage.Source: E LLM-generated captions prompt · p. 18

F Video-specific physical rule generation prompt

• Gemini-2.0-Flash-Exp was given a prompt to generate three video-specific physical rules for each prompt.The section points to Table 12 for the prompt; its contents are not included in the supplied passage.Source: F Video-specific physical rule… · p. 18

G Upsampled Caption Examples

• Caption upsampling adds finer motion, environment, and interaction details, using cues such as lighting, camera angles, and material properties to guide video generation.The section presents examples of simple captions expanded into more vivid descriptions in Table 12.Source: G Upsampled Caption Examples · p. 18

H Training Details for VIDEOPHY-2-AUTOEVAL

• VideoPhy-2-AutoEval fine-tunes VideoCon-Physics [5] (7B) with LoRA [24] across transformer blocks for three epochs, selecting the checkpoint by validation performance.The adapter uses r and α of 32 and dropout of 0.05.Source: H Training Details for… · p. 18
• Training uses Adam [27], 50-step linear warmup followed by linear decay, a 5e-4 peak learning rate, and four Nvidia A6000 GPUs with global batch size 64.The peak learning rate was selected from {5e-5, 1e-4, 5e-4, 1e-3}.Source: H Training Details for… · p. 18

I Multimodal Prompts for Automatic Evaluation

• The evaluator generates text responses from multimodal templates for semantic adherence, physical commonsense, and physical-rule scoring.The prompt formulation conditions task-specific evaluation on multimodal inputs.Source: I Multimodal Prompts for Automatic… · p. 18
• Task inputs include caption C and generated video V for semantic adherence, video V for physical commonsense, and rule candidate R for rule scoring.The appendix presents the semantic, physical-commonsense, and rule-scoring prompts in Figures 15, 16, and 17, respectively.Source: I Multimodal Prompts for Automatic… · p. 18

J.1 VideoPhysics and VideoCon

• VideoPhysics [5] and VideoCon [3] raw 0–1 scores are mapped to the 1–5 rating scale by multiplying by 4, adding 1, and rounding to the nearest integer.The stated transformation maps 0 to 1 and 1 to 5 while preserving relative performance differences.Source: §J.1 · p. 19

J.2 VideoScore Regression

• VideoScore [20] uses Text-to-Video Alignment for semantic adherence and Factual Consistency for physical commonsense.The selected metrics measure correspondence to text and physical plausibility, respectively.Source: §J.2 · p. 19

J.3 VideoLlava

• VideoLlava [34] uses the Appendix I prompts and directly produces scores on the 1–5 scale under the prompt-provided evaluation criteria.Source: §J.3 · p. 19

J.4 Gemini

• Gemini-2.0-Flash-Exp used more detailed evaluation prompts and few-shot examples to support more granular assessments of semantic adherence and physical commonsense.The larger context window enabled the expanded prompts; Figures 13 and 14 show the assessments.Source: §J.4 · p. 19

K Human Annotation Interface

• The annotation interface combines rule scoring and physical-commonsense assessment, while allowing annotators to add rules they believe were violated.The interface supports evaluation of both semantic adherence and physical commonsense.Source: K Human Annotation Interface · p. 19

L Model Inference Details

• Inference settings vary in resolution, frame rate, guidance scale, sampling steps, precision, schedulers, and caption use across models.Models with a 77-token limit due to CLIP [48] embeddings did not use upsampled captions; Wan2.1-14B and CogVideoX-5B particularly benefit from richer descriptions. Closed models such as Ray2 and Sora used fewer captions, which influences their evaluation outcomes.Source: L Model Inference Details · p. 19, Table 8 · p. 20

N Poor physical commonsense qualitative examples by model

• The paper provides additional examples from each generative model in which one or more physical laws are violated.These qualitative examples appear in Figures 20–25.Source: N Poor physical commonsense… · p. 20

• Examples show violations of conservation of momentum and conservation of mass.Source: Figure 26 · p. 34, Figure 27 · p. 35
• Further examples depict violations of elasticity and friction.Source: Figure 28 · p. 36, Figure 29 · p. 37
• Other examples illustrate violations of gravity and buoyancy.Source: Figure 30 · p. 38, Figure 31 · p. 39
• The examples also include violations of reflection and inertia.Source: Figure 32 · p. 40, Figure 33 · p. 41