Source-linked AI summary
WorldModelBench: Judging Video Generation Models As World Models
Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph E. Gonzalez, Ion Stoica, Song Han, Yao Lu
TL;DR
Existing benchmarks emphasize general video quality and inadequately evaluate whether video generation models capture real-world dynamics such as physics. WorldModelBench evaluates world modeling across application-driven domains using fine-grained instruction-following and physics-adherence measures, human annotations, and an automated judger; the judger outperforms GPT-4o in predicting human preferences, and optimizing its rewards improves world modeling capability.
Problem
Existing video benchmarks primarily measure general video quality, inadequately capturing real-world dynamics such as physics adherence needed for evaluating video world models.
Method
WorldModelBench evaluates text-to-video and image-to-video models across 350 image-text condition pairs spanning 7 application-driven domains and uses instruction-following, physics-adherence, and human-preference-based judging.
Results
The fine-tuned 2B-parameter judger achieves a 9.9% lower error rate than GPT-4o, while maximizing its rewards improves video generation models' world modeling capability.
Takeaways & Limitations
WorldModelBench provides fine-grained feedback on commonsense, instruction following, and physics adherence, and its judger supports automatic evaluation and reward-based improvement.
Takeaways & Limitations
WorldModelBench uses fewer prompts than larger video benchmarks to enable faster evaluation because contemporary models have high inference costs.
Abstract
from arXiv · showhide
Video generation models have rapidly progressed, positioning themselves as video world models capable of supporting decision-making applications like robotics and autonomous driving. However, current benchmarks fail to rigorously evaluate these claims, focusing only on general video quality, ignoring important factors to world models such as physics adherence. To bridge this gap, we propose WorldModelBench, a benchmark designed to evaluate the world modeling capabilities of video generation models in application-driven domains. WorldModelBench offers two key advantages: (1) Against to nuanced world modeling violations: By incorporating instruction-following and physics-adherence dimensions, WorldModelBench detects subtle violations, such as irregular changes in object size that breach the mass conservation law - issues overlooked by prior benchmarks. (2) Aligned with large-scale human preferences: We crowd-source 67K human labels to accurately measure 14 frontier models. Using our high-quality human labels, we further fine-tune an accurate judger to automate the evaluation procedure, achieving 8.6% higher average accuracy in predicting world modeling violations than GPT-4o with 2B parameters. In addition, we demonstrate that training to align human annotations by maximizing the rewards from the judger noticeably improve the world modeling capability. The website is available at https://worldmodelbench-team.github.io.
1. Introduction
Video generation models are increasingly viewed as video world models, but existing benchmarks mainly assess visual quality rather than real-world dynamics. WorldModelBench addresses this gap with application-driven evaluation, fine-grained world-modeling dimensions, large-scale human labels, and a trained judger.
- Motivation: Existing benchmarks emphasize temporal consistency and aesthetic coherence, but inadequately capture real-world dynamics and physics adherence.VideoPhy evaluates interactions between daily objects but overlooks broader application-driven scenarios.
- Benchmark: WorldModelBench evaluates world modeling across 350 condition pairs spanning 7 application-driven domains and 56 subdomains, supporting both T2V and I2V models.The benchmark uses text and image conditions for diverse application settings.
- Evaluation Dimensions: Instruction following is scored across four levels, while physics adherence targets five common violations, including object-size changes that violate mass conservation.These dimensions complement minimal general video-quality evaluation with fine-grained world-modeling assessment.
- Human Alignment: 67K human labels support evaluation of 14 frontier models and fine-tuning of a lightweight judger for future benchmark evaluations.The annotations are used to measure current models and train the automated evaluator.
2. Related Works
Prior video-generation benchmarks largely use general quality or text-video criteria, while reward-model research explores aligning generated outputs with human preferences. WorldModelBench builds on these evaluation and reward-alignment directions for video world modeling.
- Video Generation Evaluation: Earlier video-generation evaluations commonly use single-number metrics such as FVD and CLIPSIM.VBench broadens evaluation to general video quality and video-condition consistency, while T2VScore uses text-video and general-quality criteria.
- Reward Alignment: Reward-model approaches have been extended from text to video generation using text-image, text-video, and combined reward models.Reward gradients can incorporate multiple reward models to align model outputs with human preferences.
3. WorldModelBench
WorldModelBench evaluates video world models through instruction following, feasible future-frame generation, physics adherence, commonsense, and diverse application domains. It combines curated conditions, human annotations, automatic judging, and reward-based alignment.
- Design principle: WorldModelBench assesses whether videos follow text or image prompts and represent feasible next-world states across diverse application domains.Its grading separates instruction following from future-frame generation, including physics adherence and commonsense.
- Instruction Following: Instruction following uses four levels scored 0–3, from an absent or stationary subject to fully completing the instructed task.Intermediate levels distinguish incorrect actions from partially completed actions.
- Physics and Commonsense: Physics adherence evaluates five fundamental physical laws with binary scores totaling 0–5, while commonsense covers frame-wise and temporal quality.General video quality remains a prerequisite rather than the benchmark’s main focus.
- Diverse Domains: The benchmark spans autonomous driving, robotics, human activities, industrial, natural scenes, simulation gaming, and animation using curated text-image conditions.Each domain contains 50 samples from 5–10 subdomains; all 350 conditions undergo manual quality verification.
- Human Annotation: WorldModelBench collects 67K human labels from 8,336 complete votes and reports 70% pairwise agreement, with expert and crowd votes largely within expert variation.It also curates WorldModelBench-Hard from 45 prompts with the lowest average scores from five closed-source models.
- Automatic Judging and Alignment: A 2B-parameter VLM judger is fine-tuned on annotations to score grading criteria, and its rewards support feedback-based improvement of a pre-trained video diffusion model.The reward model evaluates instruction following, physical adherence, and commonsense; optimization maximizes its reward.
4. Experiments
Experiments evaluate current video generation models, judge reliability, reward-based improvement, and alignment with VBench. Results expose substantial world-modeling gaps, especially in instruction following, physics adherence, and difficult application domains.
- Model evaluation: 61% of kling videos correctly finish the specified task, while 12% violate mass conservation and 11% contain object penetration.These results indicate a substantial gap from an ideal video world model.
- Model evaluation: Higher frame-wise and temporal quality do not ensure stronger world modeling: Luma scores 0.81 and 0.76, but trails mochi in instruction following, 44% versus 53%.Physics adherence is similar, 4.13 versus 4.14.
- Model evaluation: I2V models underperform their T2V counterparts across all three pairs: CogVideoX 7.31 versus 6.75, OpenSoraPlan 7.62 versus 6.62, and OpenSora 6.11 versus 5.83.The reported comparison concerns total scores.
- Model evaluation: Mochi and OpenSoraPlan achieve total scores of 7.62 and 7.61, close to Luma’s 7.72, showing that top open models can be competitive with some closed models.Mochi is specifically noted for promising instruction following and physics adherence.
- Model evaluation: Most models struggle in autonomous driving, human activities, and robotics, while performing better on natural domains such as plants, animals, and water bodies.The harder categories require complex environmental interaction and accurate subject modeling.
- Reward optimization: Reward optimization with the judger shows positive qualitative signs for improving OpenSora-T2V video quality and world-modeling behavior.Examples are reported in Figures 8 and 11.
- Benchmark comparison: Physics-adherence win rates correlate only 0.28 with VBench win rates, compared with 0.69 for frame-wise quality, indicating VBench poorly distinguishes physical-law adherence.The comparison uses pairwise model win rates.
5. Discussion
The discussion identifies benchmark size as a practical limitation while arguing that the selected application domains and results remain indicative.
- Limitations: WorldModelBench uses fewer prompts than other video benchmarks to enable fast evaluation despite contemporary models’ high inference costs.Mochi, for example, takes 5 minutes on 4 A100 GPUs.
- Scope: The benchmark supports application domains such as robotics and image-to-video evaluation, unlike VideoPhy’s focus on daily objects.The authors also plan to open-source fine-grained labels.
- Discussion: Despite its smaller sample size, WorldModelBench separates the top two proprietary models, with scores of 8.82 versus 8.59.The authors describe the benchmark as indicative.
6. Conclusion
WorldModelBench evaluates video world models with fine-grained human-centered criteria and reveals shortcomings that general video-quality benchmarks miss. Its judger also provides promising feedback for improving world-modeling capability.
- Conclusion: WorldModelBench evaluates commonsense, instruction following, and physics adherence with fine-grained world-modeling feedback.The benchmark targets video generation models as world models.
- Conclusion: 67K human annotations support analysis of contemporary video generation models as world models.The annotations are used to measure model performance with the proposed dimensions.
- Conclusion: A VLM fine-tuned on human annotations performs automatic judgment on the benchmark.The judger is presented as a mechanism for scalable evaluation.
- Conclusion: Maximizing rewards from the provided judge shows promising signals for improving video generation models’ world-modeling capability.The conclusion characterizes the evidence as promising rather than definitive.
8. Appendix
Appendix analyses further separate general video quality from physics adherence and provide additional evidence for judger-based optimization. They also document evaluation details, model scores, and qualitative examples.
- Correlation analysis: Across VBench dimensions, aesthetic quality has the highest correlation with WorldModelBench physics adherence at 0.41.The analysis compares customized-video VBench dimensions with physics-adherence win rates.
- Reward optimization: The reward-feedback method shows potential to enhance instruction following, physics adherence, and aesthetics.Additional optimization examples are presented in Figure 11.
- Implementation: The appendix documents inference details for open models used in the evaluation.These details are referenced for the experiments in Section 4.
- Judger evaluation: The judge’s instruction-following predictions achieve an average error of 2.79% against human scores.This result is reported in Table 7.
- Hard subset: On the hard subset, Kling’s score regresses from 9.08 to 7.87, a 1.21-point decrease.The hard-subset evaluation is described as lightweight and difficult enough to distinguish models.
- Model results: Table 9 reports model performance across WorldModelBench, with abbreviations for Deformation, Penetration, and Gravitation.Bold and underline identify the best overall and best open-model performance, respectively.