PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts

Rishi Upadhyay, Howard Zhang, Jim Solomon, Ayush Agrawal, Yunhao Ba, Alex Wong, Celso M de Melo, Achuta Kadambi

arXiv:2601.21282v2cs.CV

TL;DR

Existing world-model evaluations rely on visual realism or coarse trajectory metrics, leaving a gap in fine-grained assessment of physical understanding. WorldBench evaluates video continuations designed around single physics concepts; across models, parameter estimates are poor and vary substantially between rollouts.

  • Problem

    Existing evaluations use visual-realism judgments or coarse trajectory metrics, leaving a gap in fine-grained assessment of concept-specific physical understanding.

    Source: §1 · p. 2, §5 · p. 9

  • Method

    WorldBench tests world models by having them continue short videos, each designed to evaluate a single physical concept or law.

    Source: §3 · p. 4

  • Results

    Across models, physical parameters are estimated poorly and with extremely high variance between rollouts, including in constrained sequences.

    Source: §4 · p. 9

  • Takeaways & Limitations

    WorldBench can distinguish visually appealing generations from physically accurate ones and diagnose concept-specific failures in current world models.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent advances in generative foundational models, often termed "world models," have propelled interest in applying them to critical tasks like robotic planning and autonomous system training. For reliable deployment, these models must exhibit high physical fidelity, accurately simulating real-world dynamics. Existing physics-based video benchmarks, however, suffer from entanglement, where a single test simultaneously evaluates multiple physical laws and concepts, fundamentally limiting their diagnostic capability. We introduce WorldBench, a novel video-based benchmark specifically designed for concept-specific, disentangled evaluation, allowing us to rigorously isolate and assess understanding of a single physical concept or law at a time. To make WorldBench comprehensive, we design benchmarks at two different levels: 1) an evaluation of intuitive physical understanding with higher level concepts such as object permanence or scale/perspective, and 2) an evaluation of low-level physical constants and material properties such as friction coefficients or fluid viscosity, allowing to measure excatly how far from reality generated videos are. When SOTA video-based world models are evaluated on WorldBench, we find specific patterns of failure in particular physics concepts, with all tested models lacking the physical consistency required to generate reliable real-world interactions. Through its concept-specific evaluation, WorldBench offers a more nuanced and scalable framework for rigorously evaluating the physical reasoning capabilities of video generation and world models, paving the way for more robust and generalizable world-model-driven learning.

1 Introduction

WorldBench evaluates world models’ physical reasoning through videos designed to isolate individual physics concepts and measurable physical parameters. Tests of state-of-the-art models reveal a gap between visually plausible scene evolution and adherence to physical parameters.

1 Introduction

  • WorldBench addresses demand for synthetic data in robotics and embodied AI, where useful generated videos must match real-world visual quality and object dynamics.Video models are of interest for generating diverse, long-tailed cases that may be uncommon in real life.Source: §1 · p. 1
  • WorldBench uses video prediction and simplified, physically rich, visually realistic scenes to evaluate physical reasoning with fine-grained, interpretable categories.It assesses plausible evolution of full visual scenes over time rather than relying on previous binary metrics.Source: §1 · p. 2
  • Its intuitive-understanding subset targets motion physics, object permanence, support relations, and scale/perspective, while parameter estimation tests adherence to gravity, viscosity, and friction.The parameter subset provides quantifiable measures of specific physical laws, including constants such as gravitational acceleration.Source: §1 · p. 2
  • WorldBench tests of state-of-the-art world models, including Cosmos [2], find visually realistic scene evolution but failures to adhere to physical parameters.Video outputs let the benchmark quantitatively distinguish visual plausibility from physical accuracy.Source: §1 · p. 2
  • By isolating a single physics concept or constant per sample, WorldBench enables concept-specific diagnosis and measures dynamics including velocity, acceleration, rotation, deformation, and occlusion.Its intuitive subset covers object permanence and scale/perspective, which the paper identifies as absent from prior benchmarks such as Physion.Source: §1 · p. 2

2 Related Work

WorldBench addresses limited evaluation of physical fidelity in world and image-to-video models by offering concept-specific physics assessment. Prior benchmarks vary in realism, task design, scope, and objectivity.

2.1 World Foundation Models

  • World foundation models aim to generate physically accurate videos or interactive environments, but their physical-fidelity claims have received little evaluation.Examples include Cosmos [2] [3], which targets downstream AI training, and Genie [13], which aims to generate physically accurate interactive environments.Source: §2.1 · p. 3

2.2 Image-to-Video Models

  • Image-to-video models prioritize visual fidelity and prompted motion over physical plausibility, while typical evaluation metrics do not test adherence to physical laws.I2V evaluation commonly uses FID, IS, and temporal coherence measures; this gap motivates benchmarks that test whether generated motion reflects physical parameters needed for real-world simulation.Source: §2.2 · pp. 3–4

2.3 Physics Datasets and Benchmarks

  • Prior physics benchmarks trade off visual realism, objective measurements, task breadth, or direct video-based evaluation, motivating WorldBench’s objective video-based metrics.PHYRE [6] and CLEVRER [64] use simplistic scenes; Physion [9] and IntPhys2 [12] use binary predictions, while Morpheus [66] and report scores rather than exact constants. PISA-Bench [31] is limited to falling objects, and VideoPhy and related work [7] [8] [36] use VLM judges, yielding metrics described as non-objective and difficult to compare.Source: §2.3 · p. 4

3 Benchmark

WorldBench evaluates world foundation models by asking them to continue short videos designed to isolate one physical concept or law. Its two complementary subsets assess intuitive physics and estimation of physical parameters.

3.1 Intuitive Physics Understanding

  • The intuitive-physics subset tests motion, support relations, object permanence, and scale/perspective across common real-world scenarios.Motion scenarios probe gravity and friction; the other concepts test continued existence when hidden, physical support and stability, and changes in apparent size or location with viewpoint.Source: §3.1 · p. 5
  • The intuitive-physics subset contains 469 videos across four concepts: 425 simulated and 44 real, each 132 frames long with ground-truth object segmentations.Scenarios are hand-designed and vary object type, location, and material; synthetic videos also include ground-truth depth, normals, and optical flow.Source: §3.1 · p. 5

3.1.1 Evaluation Methodology

  • Intuitive-physics evaluation tracks objects from first-frame ground-truth boxes with SAM2, then compares predicted and ground-truth masks using mIoU and constant-background RMSE.The background RMSE measures how well generated videos maintain the unchanged background.Source: §3.1.1 · p. 6

3.1.2 User Study

  • IoU correlated with human ratings of realism (r = 0.768), motion quality (r = 0.846), consistency (r = 0.662), and overall quality (r = 0.713).Fifty participants rated 60 videos sampled across models on a 1–10 scale; the authors report that the correlations support using their metrics as quick predictors of human judgments.Source: §3.1.2 · p. 6

3.2 Physical Parameter Estimation

  • The parameter-estimation subset contains 279 videos testing gravity, friction, and viscosity, with setups designed to reduce confounds such as depth ambiguity and uncertain object motion.Gravity has 51 videos, friction 103, and viscosity 80; an additional 30 gravity and 15 friction videos are synthetic. Viscosity videos use glycerine, corn syrup, or honey and were collected at 75°F in one session.Source: §3.2 · pp. 6–7

3.2.1 Evaluation Methodology

  • Parameter estimation derives 3D object positions from calibrated camera parameters, tracked 2D locations, and setups with constant, measurable depth.The camera is calibrated with a checkerboard, objects are tracked with SAM2, and they move parallel to the camera plane; acceleration is fitted from positions over time for gravity and friction, while viscosity uses fitted terminal velocity.Source: §3.2.1 · pp. 7–8

3.2.2 Relative Ranking Evaluation

  • Across models, viscosity rankings were near chance, while friction rankings showed more separation; models had similar strengths and uncertainties across the same material pairs.Most models performed well on far-apart friction pairs such as Rubber vs Plastic, but were unsure on Rubber vs SP80; viscosity comparisons were mostly near chance. The authors suggest shared training data may contribute to the similar performance.Source: §3.2.2 · p. 8

4 Discussion

Across the tested models, plausible-looking motion does not ensure accurate physical parameters or robust physical understanding. Performance varies with temporal input, interaction duration, training priors, and material properties, while the benchmark remains limited in scope and visual-input accessibility.

Discussion

  • The evaluation covers Cosmos-family and image-to-video models across both WorldBench subsets, including several closed-source image-to-video systems.The evaluated image-to-video models include Kling and Runway, which the passage identifies as closed source.Source: §4 · p. 8
  • Across models, physical-parameter estimates are poor and highly variable despite realistic-looking motion, including in gravity, viscosity, and friction tests.The constrained sequences still produce high variance between rollouts; gravity trajectories may look realistic without following the correct acceleration.Source: §4 · p. 9
  • Image-to-video models underperform on gravity estimation, where missing temporal input can yield severely low or even negative gravity estimates.The passage attributes this especially poor performance to image-to-video models’ lack of temporal information in their input.Source: §4 · p. 9
  • All tested models perform similarly on synthetic and real benchmark scenarios, pointing away from a distribution gap and toward poor physics understanding.This pattern holds across both benchmark subsets.Source: §4 · p. 9
  • Models simulate long-tail materials such as honey and plastic closer to average properties, and do so consistently despite being incorrect.The benchmark finds little meaningful increase in prediction variance for these materials.Source: §4 · p. 9
  • Models perform better with longer object interactions, such as ramps, tables, and walls, than with shorter interactions such as dominoes or two-object motion.Within intuitive physics, slowly sliding friction tests perform better than viscosity or gravity tests.Source: §4 · p. 9
  • Models rely on training priors: they handle familiar balls rolling down ramps better than less common interactions such as a ball hitting a block.The authors suggest this reflects reliance on training examples rather than understanding physical parameters or laws.Source: §4 · p. 9

Limitations

  • WorldBench aims to expand its tested concepts and tasks, but its visual-input design limits the number of models it can evaluate.The authors name collision mechanics and optics as planned additions and note that careful design and validation make expansion ongoing.Source: §4 · p. 9

5 Conclusion

WorldBench is a video-based, modular benchmark for measuring world-foundation models’ adherence to physical concepts, constants, and material properties. It aims to distinguish visual appeal from physical accuracy and diagnose concept-specific failures to help guide future model development.

5 Conclusion

  • WorldBench evaluates concept-specific physical understanding, directly measuring adherence to physical concepts, constants, and material properties.Its video-based, modular framework contrasts with prior assessments based on human judgments of visual realism or coarse-grained metrics such as object contact prediction.Source: §5 · pp. 9–10
  • WorldBench aims to disambiguate visually appealing generations from physically accurate ones and diagnose modern WFMs’ core concept-specific failures.Source: §5 · p. 10
  • The authors hope WorldBench will highlight current methods’ distinct challenges and guide the development and understanding of new ones.Source: §5 · p. 10

A Additional Benchmark Details … B.1 Previous Work on Multi-modal VLMs

The appendix details benchmark prompts, scenes, and experimental setups, then describes a language-based extension for VLM evaluation and its context in prior multi-modal VLM work.

A.1 Prompt Choice

  • Final prompts were selected based on benchmark performance, used consistently across tested models, and will be released with the benchmark.The authors tested prompts with varying amounts of scene and setup detail.Source: §A.1 · p. 14

A.2 Intuitive Physics Subset

  • The intuitive subset spans free-fall, object collisions, parabolic motion, and domino toppling, randomizing initial conditions or object properties to diversify scenes.The bouncing-ball clips include post-bounce frames; domino outcomes depend on the thrown object's initial velocity.Source: §A.2 · pp. 14–15
  • Occlusion scenarios test tracking through walls, columns, or blocks, including objects that disappear and reappear and spheres that bounce or roll behind obstacles.The scenarios randomize object type, velocity, mass, restitution, material, or friction coefficient, depending on the scene.Source: §A.2 · p. 14
  • Other intuitive tests probe support and balance on ramps or tables and perspective-driven size changes as objects move toward or away from the camera.Scene geometry, object properties, and starting positions vary across these tests.Source: §A.2 · p. 15

A.3 Physical Parameter Estimation Subset

  • Physical-parameter videos use calibrated iPhone 13 slow-motion captures; segmentation and fitted 3D trajectories support acceleration estimates, while linear regression estimates terminal velocity.Checkerboard calibration and manual measurements validate depth, and the evaluation pipelines are checked against collected videos and accepted ground-truth values.Source: §A.3 · p. 15
  • Gravity, viscosity, and friction scenes isolate falling motion, terminal-velocity flow through liquids, and sliding on varied ramp materials, with rulers or beaker markings providing scale.Video-to-video models receive enough input frames to estimate the relevant physical quantity; gravity objects are chosen to minimize air-resistance effects.Source: §A.3 · pp. 15–16

B Text-Enhanced Subset

  • The text-enhanced subset supports VLM evaluation of visual interpretation and physical prediction using 181 videos, each paired with one True/False or up-to-four-choice question.Example questions and answers are shown in Figure 3.Source: B Text-Enhanced Subset · p. 16

B.1 Previous Work on Multi-modal VLMs

  • Prior multi-modal VLM work motivates the language-based visual-reasoning extension, with Gemini [48] [49] and Qwen2.5-VL [54] [5] described as examples of models capable of video understanding and reasoning.The passage notes Qwen2.5-VL's dynamic-resolution processing and absolute time encoding, and Gemini's flexible handling of visual, textual, or audio input.Source: §B.1 · p. 16

B.2 Results

On WorldBench’s language-based benchmark, Gemini 2.5 Pro performs best overall, while all five tested models achieve accuracy only slightly above chance. Qwen2.5 32B leads the open-source models, outperforming the larger 72B model.

B.2 Results

  • The evaluation tested Qwen2.5 at 7B, 32B, and 72B parameter sizes, plus Gemini 2.5 Flash and Gemini 2.5 Pro.Qwen2.5 handles vision inputs natively; the models received a system prompt describing the tasks, expected output format, and data format.Source: §B.2 · p. 16
  • Across four scenarios, Gemini 2.5 Pro leads overall at 49.72% accuracy, Qwen2.5 32B leads the open-source models and outperforms 72B, and all five models score only slightly above chance.Qwen2.5 32B’s advantage over 72B is largely due to strong motion-physics performance.Source: §B.2 · p. 17, Table 6 · p. 17
  • Model outputs were evaluated by direct comparison with the answers, and the total evaluation cost was approximately $25.Qwen models ran on one H100 GPU using vLLM, while Gemini models were evaluated through the provided API [29].Source: §B.2 · p. 17

B.3 Additional Quantitative VLM Metrics · C Additional Results · C.1 Additional Qualitative Results

Additional VLM metrics reveal a sharp Walls-category gap and shared strengths and weaknesses across scene types; supplementary results also show qualitative generations while warning that individual rollouts are not representative of broad trends.

B.3 Additional Quantitative VLM Metrics

  • In Walls scenes, Qwen models score near 0.0 accuracy, while both Gemini models exceed 0.6.Source: §B.3 · p. 18
  • All five models perform similarly in most categories, struggle most with Object Permanence, and perform best overall on Motion Physics.Source: §B.3 · p. 18
  • The authors suggest models may do better on Motion Physics because their training data likely contains more examples of it than of Object Permanence.Source: §B.3 · p. 18

C Additional Results

  • Additional video-generation results include scene rollouts, background RMSE results, and scene evolution over time.Source: C Additional Results · p. 18

C.1 Additional Qualitative Results

  • Qualitative generations are shown for both the intuitive-physics subset (Figs. 4–7) and physical-parameter-estimation subset (Figs. 8–9).Source: §C.1 · p. 18
  • Results varied greatly between generations, so a single illustrated rollout should not be treated as indicative of broad benchmark trends.Source: §C.1 · p. 18

C.2 Background RMSE Results

For the intuitive physical understanding subset, background RMSE is provided as an additional metric alongside foreground mIoU.

C.2 Background RMSE Results

  • Table 7 reports background RMSE for Cosmos models, computed only over the ground-truth background segmentation; lower values are better.The section also references Table 5 for results.Source: §C.2 · p. 18, Table 7 · p. 19

C.3 Results Over Time · D User Study Details

The long-rollout analysis finds lower foreground mIoU farther into the future, indicating compounding physical-accuracy errors; the appendix also reports user-study procedures and participant details.

C.3 Results Over Time

  • Foreground mIoU is inversely related to prediction horizon when comparing 30-frame and 120-frame rollouts, suggesting compounding error in physical accuracy.The passage notes that intuitive-physics rollouts can be especially long compared with the short experiments in the second subset.Source: §C.3 · p. 19
  • Compounding errors can be especially detrimental when world models are used as synthetic data generators, underscoring the need for accurate physical understanding.Source: §C.3 · p. 19

D User Study Details

  • The Prolific user study had 50 participants, each of whom reviewed 60 videos and rated each video on a 1–10 scale.Source: D User Study Details · p. 19
  • Prolific participants were recruited globally through its matching service, took an average of 30 minutes, and were paid $12 per hour.Source: D User Study Details · p. 19
  • In a Scale/Perspective example, a metallic sphere rolls away from the camera, and all models perform well on that sample.Source: Figure 7 · p. 20
  • A Google Forms study had 54 people begin; three did not finish, and one submission was rejected for low quality because all videos were ranked identically.The passage says the results were provided in the main paper and reproduced in Table 9.Source: D User Study Details · p. 20

E SAM Validation

WorldBench uses SAM2 for segmentation and tracking across both benchmark subsets; validation compares its segmentations with hand annotations and finds high overlap on the intuitive-physics subset.

E SAM Validation

  • SAM2 segmentations achieve an overall mIoU of 0.9445 against hand annotations for the real portion of the intuitive-physics subset.The authors use SAM2 for object segmentation and tracking in both benchmark subsets.Source: E SAM Validation · pp. 20–21
  • In the object-permanence scenario, SAM2 continues tracking an object well while it is occluded for more than 20 frames, aided by prompting and maintained state.Source: E SAM Validation · p. 21

F Common Language Effect Size Results

Common Language Effect Size results show that nearly all models are near chance on viscosity comparisons, while friction results distinguish distant material pairs more reliably than close ones. The section reports per-model and per-scenario results across both settings.

F Common Language Effect Size Results

  • Tables 9–24 report Common Language Effect Size results for all models across friction and viscosity settings, with per-model and per-scenario results.Source: F Common Language Effect Size Results · p. 21
  • Almost all models are near 50% chance on viscosity pairs, while most distinguish distant friction pairs better than closer pairs.The section reports Common Language Effect Size results for both settings.Source: F Common Language Effect Size Results · p. 21

Related papers

Loading 2601.21282v2…