PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation

Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, Ping Luo

arXiv:2410.05363v1cs.CV

TL;DR

The paper addresses the limited evaluation of intuitive physics in text-to-video models, whose physical commonsense understanding is relevant to their proposed role as world simulators. It introduces PhyGenBench and PhyGenEval to benchmark and automatically assess physical-law compliance, finding that current models still struggle, especially with dynamics, and that scaling or prompt engineering is insufficient.

  • Problem

    The physical commonsense capabilities of T2V models remain insufficiently explored despite intuitive physics being considered foundational to world simulation.

  • Method

    PhyGenBench provides 160 prompts covering 27 physical laws across four domains, while PhyGenEval uses a three-tier hierarchical framework to automate physical-commonsense evaluation.

  • Results

    Current T2V models struggle to generate videos aligned with physical commonsense, and PhyGenEval achieves an overall human-correlation coefficient of 0.81.

  • Takeaways & Limitations

    Scaling up models or applying prompt engineering does not resolve challenges such as dynamic physical phenomena in PhyGenBench.

Abstract

from arXiv · show

Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as a promising path towards constructing the universal world simulator. Cognitive psychologists believe that the foundation for achieving this goal is the ability to understand intuitive physics. However, the capacity of these models to accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, a comprehensive \textbf{Phy}sics \textbf{Gen}eration \textbf{Ben}chmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which could comprehensively assesses models' understanding of physical commonsense. Alongside PhyGenBench, we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utilizing appropriate advanced vision-language models and large language models to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can conduct large-scale automated assessments of T2V models' understanding of physical commonsense, which align closely with human feedback. Our evaluation results and in-depth analysis demonstrate that current models struggle to generate videos that comply with physical commonsense. Moreover, simply scaling up models or employing prompt engineering techniques is insufficient to fully address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We hope this study will inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We will release the data and codes at https://github.com/OpenGVLab/PhyGenBench

1 INTRODUCTION

T2V models are increasingly considered potential world simulators, but their ability to represent intuitive physics remains limited and insufficiently evaluated. PhyGenBench and PhyGenEval address this gap by benchmarking and automatically assessing physical commonsense in generated videos.

  • Motivation: T2V models struggle with basic physical phenomena, revealing a substantial gap between their physical understanding and human intuitive physics.Examples include failing to make a stone sink in water or a plane appear reflected in water.
  • Evaluation gap: Existing T2V benchmarks mainly assess video qualities or spatial relationships rather than adherence to fundamental physical laws.This leaves physical commonsense insufficiently measured.
  • Evaluation gap: Physical-commonsense evaluation lacks both benchmarks with clear phenomena and metrics that can detect implausible motions without reference videos.FVD requires reference videos, while video-based VLMs may not understand physical laws reliably.
  • Contributions: PhyGenBench covers mechanics, optics, thermal phenomena, and material properties through 27 physical laws and 160 validated prompts.The benchmark is designed around clear, observable physical phenomena.
  • Contributions: PhyGenEval uses GPT-4o and a three-tier hierarchical strategy to automate evaluation of physical commonsense correctness.The framework is designed to address weaknesses in generic video evaluation methods.
  • Findings: Gen-3 scores only 0.51 in the reported evaluation, and dynamics remain difficult despite scaling up models or using prompt engineering.The authors use this result to characterize current models as far from functioning as world simulators.

2 RELATED WORK

Prior video-generation benchmarks and metrics primarily evaluate general quality, motion, or spatial relationships, but they do not adequately assess physical commonsense. PhyGenBench instead constructs physics-focused prompts through a structured data pipeline.

  • Existing benchmarks: Recent benchmarks such as VBench and EvalCrafter evaluate general video quality across multiple dimensions rather than physical-law compliance.Their focus differs from the physical commonsense target of this work.
  • Existing evaluation: Traditional metrics such as FVD and IS have difficulty detecting unrealistic motions, while FVD also requires reference videos that are hard to obtain for novel scenes.These constraints limit their suitability for evaluating physical correctness.
  • Proposed benchmark: PhyGenBench selects physical laws, manually creates initial prompts, augments them with details, varies objects using GPT-4o, and applies manual quality control.This pipeline produces prompts intended to express clear physical phenomena without revealing the expected outcome.
  • Existing evaluation: Video-based VLM evaluators improve alignment with human judgments in some settings but remain weak at generalizing to physical commonsense correctness.Their limitations include inadequate understanding of physical laws and lack of physics-specific design.

3 PHYGENBENCH

PhyGenBench defines physical commonsense through four categories and constructs prompts that isolate clear, observable phenomena associated with specific physical laws. Its design combines broad coverage with one-to-one correspondence between prompts and evaluated phenomena.

  • Definitions: Physical commonsense is distinguished from physical laws and physical phenomena as intuitive behavior, universal principles, and observable law-driven events, respectively.PhyGenBench evaluates whether T2V models understand physical commonsense through designed prompts.
  • Overview: The benchmark spans mechanics, optics, thermal phenomena, and material properties, covering 27 physical phenomena with 160 prompts.These categories provide the benchmark’s overall coverage.
  • Mechanics: Mechanics contains 7 laws and 40 prompts, including gravity, buoyancy, pressure, elasticity, friction, and surface tension.A buoyancy example tests whether iron sinks because its density exceeds water’s.
  • Optics: Optics contains 6 light-related aspects and 50 prompts, including reflection, refraction, scattering, dispersion, interference, diffraction, and straight-line propagation.A kite above a smooth pond tests reflection generation.
  • Thermal: Thermal phenomena include 6 phase transitions and 30 prompts, with boiling evaluated through a water-heating timelapse.The category covers solidification, melting, liquefaction, boiling, deposition, and sublimation.
  • Material Properties: Material properties cover 8 physical or chemical properties and 40 prompts, such as hardness tested by an egg striking a rock.The expected outcome is that the egg breaks while the rock remains intact.
  • Design and construction: Prompt curation enforces one-to-one correspondence between each physical phenomenon and its physical law to reduce ambiguity for human and machine evaluation.Construction uses conceptualization, prompt engineering, augmentation, GPT-4o-based diversity enhancement, and quality control.

4 PHYGENEVAL

PhyGenEval evaluates physical commonsense through key-phenomena detection, temporal-order verification, and overall naturalness assessment. Its hierarchical VLM-based design separates semantic alignment from physical correctness and combines the three stages into a final score.

  • Framework overview: PhyGenEval decomposes physical commonsense evaluation into key physical phenomena detection, physics order verification, and overall naturalness evaluation.Each stage uses physical-law-specific questions and vision-language models, with GPT-4o generating evaluation prompts.
  • Evaluation validity: PhyGenEval correlates more closely with human judgments than previous evaluation methods, while VideoScore reaches only a 0.19 Spearman correlation on PhyGenBench.The framework addresses the difficulty of detecting embedded physical commonsense with direct video-based evaluation.
  • Key Physical Phenomena Detection: Key physical phenomena detection retrieves salient frames and checks whether expected observable phenomena occur at the correct point in the video.The evaluator uses retrieval prompts, physics-related questions, and adjacent frames to improve robustness.
  • Physics Order Verification: Physics order verification checks whether prerequisite events occur in the correct temporal sequence, such as an egg touching a stone before cracking.Three questions assess the sequence before the keyframe, after it, and across the entire video.
  • Overall Naturalness Evaluation: Overall naturalness evaluates whether the video’s dynamic progression conforms to the prompt’s physical law and a GPT-4o-generated, prompt-specific standard.The stage scores the video using the prompt, physical law, detailed standard, and video frames.
  • Overall score: The final score averages discretized scores from the three stages using floor rounding and ensembles multiple evaluator models for robustness.Physics order uses GPT-4o and LLaVA-Interleave, while naturalness uses GPT-4o and InternVideo2.

5 EXPERIMENT

Experiments evaluate PhyGenEval and physical commonsense across open-source and proprietary video-generation models. The results show low physical accuracy overall, strong relative performance in optics, and persistent failures in mechanics, thermal phenomena, and material properties.

  • Experimental setup: Experiments compare eight text-to-video models, including five open-source and three proprietary systems, against Videophy, VideoScore, and DEVIL.The study evaluates OpenSora V1.2, Lavie, CogVideoX 2B, CogVideoX 5B, Vchitect2.0, Kling, Pika, and Gen-3.
  • Human evaluation: 0.81 overall correlation coefficient shows that PhyGenEval aligns strongly with human judgments across physical commonsense categories.Human evaluation covers 512 generated videos scored by three annotators.
  • Metric comparison: Existing metrics often misclassify smooth but physically impossible videos, whereas PhyGenEval detects violations such as rubber-like egg impacts and floating rocks.VideoScore, DEVIL, and VideoPhy can rate these physically incorrect cases as correct.
  • Quantitative results: All evaluated models score very low in physical commonsense accuracy, and the best-performing Gen-3 reaches only a 0.51 PCA score.These results indicate difficulty generating videos that comply with intuitive physics.
  • Category analysis: Models perform better in optics than in mechanics, thermal phenomena, and material properties.Qualitative examples include failures to depict sinking, melting, sublimation, egg breaking, and chemical reactions.
  • Ablation study: The ablation study finds that the key design elements of PhyGenEval, including its evaluation levels and two-stage naturalness strategy, are essential.Detailed robustness results are reported in Appendix C.3.

6 DISCUSSION

Scaling, prompt engineering, and general video-quality methods do not fully resolve physical commonsense failures. Scaling helps some static phenomena, but dynamic scenarios remain difficult, while prompt engineering fixes only a few simple issues.

  • Scaling: Scaling improves some physical commonsense issues but still fails on dynamic physical phenomena.The authors suggest that these cases require extensive training on synthetic data.
  • Prompt engineering: Prompt engineering resolves only a few simple issues, such as flame color, highlighting the difficulty of PhyGenBench.The paper presents this as insufficient for the broader benchmark challenges.
  • General video quality: Methods that improve general video quality do not enhance models’ understanding of physical commonsense.The discussion distinguishes visual-quality improvements from physical reasoning improvements.

7 CONCLUSION

The paper introduces PhyGenBench and PhyGenEval to assess physical commonsense in T2V models and finds persistent difficulties with physically aligned video generation.

  • 7 CONCLUSION: PhyGenEval provides a three-tier hierarchical framework for automating physical commonsense evaluation.
  • 7 CONCLUSION: Current T2V models struggle to generate videos that align with physical commonsense, revealing a significant gap from world simulation.
  • 7 CONCLUSION: Scaling up models or applying prompt engineering does not resolve challenges in PhyGenBench, including dynamic scenarios.
  • 7 CONCLUSION: PhyGenBench evaluates physical commonsense using four physics domains and 27 representative physical laws.The benchmark provides a comprehensive and fine-grained assessment of models’ physical capabilities.

A.2 DIFFERENCE BETWEEN VIDEOPHY AND OURS

PhyGenBench broadens physical-commonsense coverage beyond Videophy and uses clearer prompts that better support semantic alignment and physical evaluation.

  • Benchmark coverage: PhyGenBench covers 160 prompts and 27 physical laws across four domains, while Videophy has narrower material-interaction coverage and lacks physical-law annotations.
  • Semantic alignment: PhyGenBench achieves an average SA score of 0.80 versus 0.63 for Videophy.
  • Prompt quality: Videophy prompts can be too underspecified for modern T2V models to generate text-aligned, high-quality videos.
  • Evaluation design: PhyGenEval reduces semantic-evaluation complexity by first extracting prompted objects and actions, then checking their presence and occurrence in generated videos.
  • Evaluation design: PhyGenEval evaluates physical phenomena through key-frame detection, sequence verification, and overall naturalness scoring.
  • Evaluation design: Its final score combines discretized scores from phenomenon detection, sequence verification, and naturalness evaluation.

C.1 EXPERIMENTS SETUP

The experiments compare several T2V models and evaluation metrics, using human evaluation and ablations to assess semantic alignment and physical commonsense evaluation.

  • Models and metrics: The evaluation compares PhyGenEval with VideoPhy, VideoScore, Grid-LLaVA, and DEVIL on semantic and physical commonsense alignment.
  • Experimental setup: Table 5 reports duration, FPS, and resolution for each evaluation model.
  • Ablation analysis: The study conducts ablation experiments to evaluate the effectiveness of PhyGenEval’s design choices.
  • Human evaluation: Human annotators score semantic alignment and physical commonsense alignment using standards specified in Figure 9.

C.2 QUANTITATIVE EVALUATION

PhyGenEval shows strong human alignment for semantic evaluation, while most tested models achieve high semantic-alignment scores on the relatively straightforward PhyGenBench scenarios.

  • Semantic evaluator correlation: PhyGenEval achieves the highest correlation scores across all semantic-alignment categories.
  • Semantic alignment results: The semantic scenarios in PhyGenBench are simple enough to clearly reflect physical phenomena.
  • Semantic alignment results: Most models achieve relatively high semantic-alignment scores under both machine and human evaluation.
  • Semantic alignment results: Kling achieves the highest human-evaluated SA score, reaching 0.89.

C.3 ABLATION STUDY

The ablations show that PhyGenEval’s two-stage strategy and complete three-stage pipeline improve agreement with human judgments, while open-source models provide a lower-cost alternative with substantial correlation.

  • Two-stage evaluation: The two-stage strategy produces evaluations more consistent with human judgments than the one-stage strategy for both InternVideo2 and GPT-4o.The authors attribute the improvement to LLM-based rewriting that reduces VLM evaluation complexity.
  • Stage ablation: Optimal performance is achieved only when all three PhyGenEval stages are used concurrently.The ablation supports the contribution of each stage to the final performance.
  • Model selection: An open-source PhyGenEval configuration achieves a correlation coefficient of 0.66, while ensembling open-source and closed-source methods performs best.The alternatives use LLaVA-Interleave and InternVideo2, whereas the closed-source method uses GPT-4o.
  • Semantic alignment: GPT-4o achieves higher semantic-alignment correlations than LLaVA, with Kendall’s τ of 0.53 versus 0.42 and Spearman’s ρ of 0.56 versus 0.44.The result motivates selecting GPT-4o as the preferred VLM for semantic-alignment evaluation.
  • Semantic alignment: PhyGenEval-Grid-LLaVA improves over Grid-LLaVA, reaching Kendall’s τ of 0.42 versus 0.35 and Spearman’s ρ of 0.44 versus 0.39.The two-stage method decomposes evaluation into object and action detection.

D DISCUSSION

Discussion experiments indicate that scaling and prompt rewriting improve some physical-commonsense behaviors but leave dynamic and complex physical processes unresolved, while visual enhancement does not improve physical understanding.

  • Scaling: Scaling from CogVideoX 2B to 5B improves some static physical phenomena but leaves simple motion dynamics, such as bouncing footballs, inaccurate.The authors suggest scaling may help individual-object phenomena but remain insufficient for dynamic physical laws.
  • Prompt rewriting: Prompt rewriting helps models generate videos aligned with some physical laws but does not resolve the broader PhyGenBench challenges.Kling reaches only a score of 0.56 despite improvement.
  • Prompt rewriting: Prompt rewriting addresses simple issues such as flame-color reactions but remains ineffective for complex processes including egg breaking and stone sinking.The qualitative comparison indicates that most problems remain unsolved.
  • Visual enhancement: VEnhancer improves visual quality without addressing models’ poor understanding of physical commonsense.Its enhancement adds spatial detail and synthetic temporal motion, but the physical deficiency remains.
  • Visual enhancement: VEnhancer produces similar PhyGenBench scores before and after enhancement, with a Spearman coefficient of 0.86 between the scores.This indicates that PhyGenEval primarily tracks physical correctness rather than general visual quality.
Loading 2410.05363v1…