PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation

Hritik Bansal, Clark Peng, Yonatan Bitton, Roman Goldenberg, Aditya Grover, Kai-Wei Chang

arXiv:2503.06800v1cs.CV

TL;DR

VideoPhy-2 addresses the unclear physical-commonsense capabilities of video generative models by building an action-centric benchmark with human and automatic evaluation. It finds major performance gaps, including only 22% joint performance for the best model on the hard subset, while identifying conservation laws as frequent failure points.

  • Problem

    Existing evidence is limited on whether video generative models follow physical commonsense across diverse real-world actions.

  • Method

    VideoPhy-2 curates diverse actions and prompts, collects human judgments of semantic adherence, physical commonsense, and physical-rule grounding, and trains an automatic evaluator.

  • Results

    22% joint performance was achieved by the best model on the hard subset, while conservation of mass and momentum were among the most frequently violated laws.

  • Takeaways & Limitations

    VideoPhy-2 exposes substantial gaps in video models as general-purpose physical world simulators and provides scalable evaluation for studying them.

  • Takeaways & Limitations

    Other closed models were excluded because API access was unavailable.

Abstract

from arXiv · show

Large-scale video generative models, capable of creating realistic videos of diverse visual concepts, are strong candidates for general-purpose physical world simulators. However, their adherence to physical commonsense across real-world actions remains unclear (e.g., playing tennis, backflip). Existing benchmarks suffer from limitations such as limited size, lack of human evaluation, sim-to-real gaps, and absence of fine-grained physical rule analysis. To address this, we introduce VideoPhy-2, an action-centric dataset for evaluating physical commonsense in generated videos. We curate 200 diverse actions and detailed prompts for video synthesis from modern generative models. We perform human evaluation that assesses semantic adherence, physical commonsense, and grounding of physical rules in the generated videos. Our findings reveal major shortcomings, with even the best model achieving only 22% joint performance (i.e., high semantic and physical commonsense adherence) on the hard subset of VideoPhy-2. We find that the models particularly struggle with conservation laws like mass and momentum. Finally, we also train VideoPhy-AutoEval, an automatic evaluator for fast, reliable assessment on our dataset. Overall, VideoPhy-2 serves as a rigorous benchmark, exposing critical gaps in video generative models and guiding future research in physically-grounded video generation. The data and code is available at https://videophy2.github.io/.

1 Introduction

VideoPhy-2 addresses uncertainty about whether video generative models can produce physically plausible worlds across diverse real-world actions. Its evaluation reveals substantial shortcomings even in the strongest tested model.

  • Motivation: VideoPhy-2 targets the unclear ability of video models to generate physically likely worlds across diverse real-world actions.Accurate simulation requires modeling poses, dynamics, materials, and complex physical interactions.
  • Findings: 32.6% joint performance was achieved by Wan2.1-14B, measuring high semantic adherence and physical commonsense.The hard subset reduced this score to 22%.
  • Contribution: VideoPhy-2 combines semantic adherence, physical commonsense, and physical-rule compliance in its evaluation pipeline.The benchmark also supports automatic evaluation alongside human judgments.

2 VIDEOPHY-2 Dataset

VideoPhy-2 constructs a diverse action-centric benchmark through staged curation, prompt generation, physical-rule annotation, and difficulty selection. The resulting dataset is substantially larger and richer than VIDEOPHY.

  • Seed Actions: The dataset begins with over 600 candidate real-world actions spanning sports, physical activities, and object interactions.Student reviewers select actions relevant to laws including gravity, elasticity, buoyancy, reflection, mass, and momentum conservation.
  • LLM-Generated Prompts: Gemini-2.0-Flash-Exp generates 20 prompts per action emphasizing visible physical interactions and excluding non-visual or abstract details.The prompts are designed to support clear grounding in generated videos.
  • Candidate Physical Rules: Physical rules are generated from captions of model-produced videos rather than directly from conditioning prompts.This two-step process is used because direct rule generation from videos did not yield high-quality outputs.
  • Hard Subset: 60 of 197 actions form a hard subset selected where CogVideoX-5B failed to follow prompts and physical commonsense.The subset is identified with a model-based difficulty-selection strategy.
  • Data Analysis: 3940 captions and 102K human annotations make VideoPhy-2 5.72× larger in captions than VIDEOPHY and richer in evaluation data.Original and upsampled captions average 16 and 138 tokens, respectively.

3 Evaluation

The evaluation separates prompt alignment from physical plausibility and adds fine-grained physical-rule judgments. Human annotation provides the benchmark standard, while automatic evaluators support scalable assessment.

  • Metrics: Semantic adherence measures whether prompted entities, actions, and relationships are accurately depicted in the generated video.Annotators use a 5-point scale from Very Unlikely to Very Likely.
  • Metrics: Physical commonsense measures whether a generated video intuitively follows real-world physical laws independently of prompt alignment.The task evaluates the holistic physical plausibility of the video.
  • Metrics: Physical rules are labeled as violated, followed, or cannot be determined, enabling fine-grained analysis of model behavior.Annotators may also record additional violations beyond predefined candidate rules.
  • Human Evaluation: Twelve qualified AMT annotators provide human judgments, treated as the gold standard for generative-model assessment.The study gives annotators detailed instructions and analyzes annotation examples to promote quality.
  • Automatic Evaluation: Automatic evaluators score semantic adherence and physical commonsense on a normalized 5-point scale for faster, more cost-effective assessment.The study evaluates video-language foundation models for these two tasks.

4 Setup

The study benchmarks seven text-to-video models and trains its automatic evaluator from human annotations across separate training and testing prompt splits. Model coverage is constrained by API access and budget limitations.

  • Video Generative Models: Seven text-to-video models are evaluated, comprising five open models and two closed models.The models include CogVideoX-5B, VideoCrafter2, HunyuanVideo-13B, Cosmos-Diffusion-7B, Wan2.1-14B, Sora, and Ray2.
  • Dataset Setup: 3350 training prompts and 590 testing prompts are used to train and benchmark VideoPhy-2-AutoEval.The split assigns 17 captions per action to training and 3 captions per action to testing.
  • Limitation: Other closed models are excluded because API access is unavailable, limiting the study’s closed-model coverage.This is an author-stated setup limitation.
  • Benchmarking: Benchmark videos receive annotations for semantic adherence, overall physical commonsense, and physical-rule violations.The standard setup generates one video per test prompt, with reduced coverage for Sora and Ray2.
  • AutoEval Training: Approximately 50K human annotations are collected to train the automatic evaluator across three evaluation tasks.Training videos come from HunyuanVideo-13B, Cosmos-Diffusion-7B, and CogVideoX-5B.

5 Experiments

Experiments show that current video generators struggle with physical commonsense, including conservation laws, while VIDEOPHY-2-AUTOEVAL provides scalable evaluation across prompts, models, and physical rules.

  • Performance on the dataset: 32.6% and 21.9% are Wan2.1-14B’s joint scores on the full and hard VIDEOPHY-2 splits, respectively.The hard split substantially reduces performance relative to the full dataset.
  • Performance on the dataset: Optimizing aesthetics and motion quality alone cannot achieve high performance on VIDEOPHY-2; physical commonsense must also be incorporated.The benchmark therefore remains challenging for modern video models.
  • Fine-grained Analysis: 40% violation scores make conservation of momentum and conservation of mass among the most frequently violated physical laws.Reflection and buoyancy are relatively mastered, with violation scores below 20%.
  • Qualitative Analysis: Qualitative failures include reverse-moving jetskis, deforming solid objects, and rocks accelerating uphill against gravity.These examples span multiple models, including Wan2.1, the best-performing model.
  • VIDEOPHY-2-AUTOEVAL: VIDEOPHY-2-AUTOEVAL is evaluated on unseen prompts, unseen video models, joint score judgments, and physical rule classification.Table 4 reports correlation, Table 5 reports joint accuracy and F1, and Table 6 reports rule-classification accuracy.

6 Conclusion

The conclusion presents VIDEOPHY-2 as a benchmark for testing whether generated videos both follow prompts and obey physical commonsense. It positions the benchmark, physical-law violations, and automatic evaluation as tools for understanding video models as world simulators.

  • 6 Conclusion: VIDEOPHY-2 benchmarks physical commonsense in videos generated by modern models.The benchmark targets the ability to align generated videos with physical expectations.
  • 6 Conclusion: The benchmark reveals a large gap between prompt alignment and physically commonsense video generation.This conclusion is framed as evidence about current video generative models as general-purpose world simulators.
  • 6 Conclusion: VIDEOPHY-2 provides physical-law violation analysis and an auto-rater for scalable evaluation.These components extend benchmark and evaluation methodologies for physical commonsense understanding.

A.2 General Video Generation Benchmarks

General video benchmarks evaluate broad qualities such as motion smoothness and visual fidelity, but they do not specifically isolate physical reasoning.

  • A.2 General Video Generation Benchmarks: VBench evaluates motion smoothness, background consistency, and overall visual fidelity through a hierarchical approach.EvalCrafter proposes 17 objective metrics covering different aspects of video quality.
  • A.2 General Video Generation Benchmarks: General video-quality benchmarks do not specifically assess physical reasoning.Their focus is broader video quality rather than systematic physical-commonsense evaluation.

A.3 Benchmarks for Physical Understanding in Video Generation

Prior physical-understanding benchmarks assess plausibility or world simulation but remain limited by scale, evaluation complexity, and the sim-to-real gap. VIDEOPHY-2 addresses these limitations with broader action coverage and a more interpretable framework.

  • A.3 Benchmarks for Physical Understanding in Video Generation: VIDEOPHY evaluated semantic adherence and physical plausibility across 688 prompts, whereas VIDEOPHY-2 expands to 3940 prompts.VIDEOPHY-2 also shifts emphasis from material interactions to real-world actions and uses a 5-point Likert evaluation.
  • A.3 Benchmarks for Physical Understanding in Video Generation: Sequential vision-language queries can introduce inconsistencies and increase computational complexity in prior evaluation.VIDEOPHY-2 is presented as providing a more interpretable evaluation framework.
  • A.3 Benchmarks for Physical Understanding in Video Generation: WorldSimBench aligns video-model outputs with numerical solvers but faces challenges bridging simulation and real-world physical complexity.Physical simulations may not fully capture the complexity of real environments.

A.4 Automatic Evaluation Methods

VIDEOPHY-2 evaluates physical commonsense in diverse real-world actions using human-aligned automatic assessment, with prompts and rules designed around visible interactions and challenging behaviors.

  • Automatic evaluation: VIDEOPHY-2-AutoEval extends prior VLM evaluation with a larger, more diverse dataset and explicit physical-rule analysis.The broader benchmark combines semantic adherence, physical commonsense, and rule annotations.
  • Benchmark scope: VIDEOPHY-2 bridges theoretical physical reasoning and practical video generation through a large-scale, real-world benchmark with explicit rule annotations.The benchmark targets diverse movement dynamics, object interactions, and physical complexity.
  • Difficulty design: The benchmark includes hard actions involving rapid motion, intricate interactions, and structural deformations, selected using CogVideoX-5B failures.The hard subset contains 60 actions selected from 197 based on prompt adherence and physical-commonsense failures.
  • Prompt construction: The dataset curates diverse actions and generates multiple prompts emphasizing visible physical interactions while excluding non-visual or abstract details.Prompt generation covers contextual variations in actions, objects, environments, and interactions.

H Training Details for VIDEOPHY-2-AUTOEVAL

VIDEOPHY-2-AutoEval uses a fine-tuned multimodal model to score semantic adherence, physical commonsense, and candidate-rule compliance from videos and associated text.

  • Training setup: VideoCon-Physics is fine-tuned with low-rank adaptation across transformer blocks for three epochs, selecting the best checkpoint on validation performance.The reported configuration uses r and α equal to 32 with dropout 0.05.
  • Evaluation formulation: The evaluator conditions text generation on multimodal templates tailored to semantic adherence, physical commonsense, and physical-rule scoring.The inputs include generated video V, conditioning caption C, and, for rule scoring, candidate rule R.
  • Evaluation formulation: The task variable t selects whether the model evaluates caption adherence, physical commonsense, or a candidate physical rule.The same evaluation framework supports all three scoring tasks.
  • Baseline calibration: Baseline scores from VideoPhysics and VideoCon are linearly mapped from 0–1 to the 1–5 annotation scale using scorenormalized = scoreraw × 4 + 1.The normalized values are rounded to the nearest integer.
  • Baseline calibration: VideoScore uses Text-to-Video Alignment for semantic adherence and Factual Consistency for physical commonsense.These component metrics were selected to match the evaluation objectives.

J.4 Gemini

Gemini-2.0-Flash-Exp provides structured, detailed evaluation prompts, while the study reports score distributions and model-generation settings for semantic and physical assessment.

  • Gemini evaluation: Gemini-2.0-Flash-Exp uses a larger context window, granular criteria, and few-shot examples for semantic-adherence and physical-commonsense assessment.The corresponding prompt designs are shown in the cited evaluation figures.
  • Human evaluation: Human annotation interfaces combine physical-commonsense and rule scoring while allowing annotators to add rules they judge violated.Separate interfaces support semantic adherence and physical-rule or commonsense judgments.
  • Generation settings: Inference settings report resolution, frame rate, guidance scale, sampling steps, and precision across video-generation models.Most models use upsampled captions, while models with a 77-token limit are exceptions.
  • Score analysis: Figures 9 and 10 present the distributions of semantic-adherence and physical-commonsense scores for the evaluated models.The figures summarize score distributions rather than a single aggregate value.
  • Qualitative analysis: Additional figures show model-specific examples of physical-law violations across multiple generative models.The examples include violations involving momentum, mass, elasticity, friction, gravity, buoyancy, reflection, and inertia.

O Poor physical commonsense qualitative examples by law

The qualitative analysis documents physically unlikely generations across models and physical laws, using examples, action lists, prompts, and evaluation templates to frame the observed failures.

  • Action coverage: The benchmark’s action list covers physical activities and object interactions used to test alignment with physical commonsense.The full action collection is presented as a comprehensive test set.
  • Prompt context: Sample prompts pair actions with categories and potential physical principles, providing the textual context for qualitative evaluation.Upsampled captions add specificity for video-generation models.
  • Evaluation materials: The evaluation templates separately address physical-rule generation, semantic adherence, physical commonsense, and rule validation.The human interfaces support semantic adherence and combined physical-rule or commonsense judgments.
  • Model examples: Qualitative examples span physical-law violations in CogVideoX-5B, Cosmos, Ray2, Hunyuan, Wan2.1, and VideoCrafter2.The cited figures organize examples by model.
  • Violations by law: The analysis isolates violations of momentum, mass, elasticity, friction, gravity, buoyancy, reflection, and inertia.Each cited figure focuses on examples associated with one named law.
Loading 2503.06800v1…