Source-linked AI summary
WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts
Rishi Upadhyay, Howard Zhang, Jim Solomon, Ayush Agrawal, Yunhao Ba, Alex Wong, Celso M de Melo, Achuta Kadambi
TL;DR
World models intended for robotics and synthetic data must reproduce real-world dynamics, yet existing benchmarks entangle multiple concepts and provide limited diagnostic evidence. WorldBench introduces disentangled video evaluations for intuitive physics and measurable physical parameters, and tests show current models often produce visually plausible but physically inconsistent outputs with highly variable parameter estimates. The benchmark therefore supports more detailed diagnosis of concept-specific physical failures.
Problem
Existing physics-based video benchmarks entangle multiple physical concepts, limiting objective diagnosis of whether world models reproduce real-world dynamics accurately.
Method
WorldBench evaluates video predictions through separate intuitive-physics scenarios and physical-parameter tests covering quantities such as gravity, viscosity, and friction.
Results
Current world models estimate physical parameters poorly with extremely high variance across rollouts, while often generating visually realistic scene evolutions.
Takeaways & Limitations
WorldBench provides a concept-specific framework for distinguishing visual realism from physical accuracy and diagnosing failures in world models.
Abstract
from arXiv · showhide
Recent advances in generative foundational models, often termed "world models," have propelled interest in applying them to critical tasks like robotic planning and autonomous system training. For reliable deployment, these models must exhibit high physical fidelity, accurately simulating real-world dynamics. Existing physics-based video benchmarks, however, suffer from entanglement, where a single test simultaneously evaluates multiple physical laws and concepts, fundamentally limiting their diagnostic capability. We introduce WorldBench, a novel video-based benchmark specifically designed for concept-specific, disentangled evaluation, allowing us to rigorously isolate and assess understanding of a single physical concept or law at a time. To make WorldBench comprehensive, we design benchmarks at two different levels: 1) an evaluation of intuitive physical understanding with higher level concepts such as object permanence or scale/perspective, and 2) an evaluation of low-level physical constants and material properties such as friction coefficients or fluid viscosity, allowing to measure excatly how far from reality generated videos are. When SOTA video-based world models are evaluated on WorldBench, we find specific patterns of failure in particular physics concepts, with all tested models lacking the physical consistency required to generate reliable real-world interactions. Through its concept-specific evaluation, WorldBench offers a more nuanced and scalable framework for rigorously evaluating the physical reasoning capabilities of video generation and world models, paving the way for more robust and generalizable world-model-driven learning.
1 Introduction
WorldBench addresses the need for objective, concept-specific evaluation of world models’ physical fidelity by separating intuitive physics from measurable parameters. Tests of current models reveal visually plausible but physically inconsistent generations and substantial parameter-estimation variance.
- Benchmark motivation and design: The benchmark is motivated by the need for synthetic data to match real-world visual quality, object dynamics, and governing physical constants.The paper identifies concept-specific and objective evaluation as necessary for judging such data generators.
- Benchmark motivation and design: WorldBench evaluates physical reasoning through video prediction using simplified, physically rich, visually realistic scenes and fine-grained evaluation categories.The framework is designed to produce repeatable and interpretable outcomes rather than relying only on binary metrics.
- Benchmark subsets: The intuitive subset tests motion physics, object permanence, support relations, and scale/perspective as separate principles.Examples include rolling behind pillars and objects moving toward the camera.
- Benchmark subsets: The physical parameter subset measures adherence to known quantities such as gravitational acceleration, fluid viscosity, and friction coefficients.These measurable parameters provide a direct metric of how accurately generated dynamics follow physical laws.
- Benchmark significance: The benchmark provides finer-grained diagnosis of concept failures and directly measures object dynamics and physical accuracy for potential synthetic-data use.Evaluated dynamics include velocity, acceleration, rotation, deformation, and occlusion.
- Findings: State-of-the-art world models often generate visually realistic evolutions but fail to maintain the corresponding physical parameters and show substantial gaps in physical consistency and generalization.For example, a parabolic trajectory may look plausible even when gravitational acceleration is incorrect.
2 Related Work
Prior physics benchmarks span simplified scenarios, visual question answering, binary judgments, and coarse scores, while WorldBench targets video-based physical parameters and modular evaluation. Its parameter-estimation pipeline extracts 3D motion and fits curves to estimate quantities such as acceleration and terminal velocity.
- Image-to-video evaluation: Image-to-video models prioritize aesthetic realism and user-specified motion, while common metrics assess visual quality and temporal smoothness rather than physical plausibility.The paper identifies this as a gap addressed by benchmarks testing physical parameters for real-world simulation.
- Physical parameter estimation: Its physical-parameter pipeline combines checkerboard detection and SAM2 to extract 3D object positions, then fits curves to estimate properties such as acceleration or terminal velocity.Post-processing converts fitted motion quantities into the relevant physical parameters when needed.
- Physical parameter estimation: Validation on the PerfectPhysics subset reports all estimated parameters within an acceptably small margin of error, with gravity measured in m/s2 and viscosity in Pa.s.SP denotes Sandpaper in the validation table.
- Existing physics benchmarks: Prior datasets differ in task and realism: PHYRE uses simple 2D objects, CLEVRER uses 3D VQA, and Physion uses visually realistic videos with binary object-contact prediction.The paper describes these choices as limiting their usefulness for fine-grained development guidance or visual realism.
3 Benchmark
WorldBench evaluates physical understanding through separate intuitive-physics and physical-parameter-estimation subsets, using concept-specific video continuations and controlled evaluation pipelines. It covers high-level principles and measurable constants or material properties, with assessments based on object segmentation, parameter estimates, and relative rankings.
- Benchmark structure: WorldBench asks models to generate video continuations designed to evaluate a single physics concept or law.The benchmark targets concept-specific physical prediction rather than entangled tests.
- Intuitive Physics Understanding: The intuitive-physics subset probes motion physics, object permanence, support relations, and perspective or scale relations.Its scenarios include bouncing and falling objects, hidden objects, support configurations, and objects moving toward or away from the camera.
- Evaluation methodology: The intuitive subset evaluates generated videos against ground-truth object segmentations using foreground mIoU, with a user study measuring realism, motion quality, consistency, and overall quality.The user study recruited 50 participants, each rating 60 videos on four 1–10 scales.
- Physical Parameter Estimation: The physical parameter-estimation subset tests adherence to known gravity, viscosity, and friction values through controlled real and simulated videos.Gravity uses straight drops and parabolic motion; viscosity uses terminal velocity for steel balls in glycerine, corn syrup, and honey.
- Relative Ranking Evaluation: Parameter evaluation compares estimates with ground truth and uses CLES to rank model estimates across materials, revealing poorer viscosity discrimination than friction discrimination.Most viscosity comparisons are near chance, while widely separated friction pairs show much stronger separation.
4 Discussion
WorldBench evaluates both Cosmos and image-to-video models across intuitive and physical-parameter subsets, revealing poor parameter estimation, high rollout variance, and weak physical consistency despite realistic trajectories.
- 4 Discussion: WorldBench evaluates the Cosmos family and multiple image-to-video models, including both open- and closed-source systems.Evaluated systems include Cosmos-1, Cosmos-2, Cosmos-2.5, Wan 2.2, Hunyuan Video, CogVideoX, Kling 3.0, Runway Gen-4.5, and Lightricks 2.0.
- 4 Discussion: All evaluated Cosmos-family and image-to-video models estimate physical parameters poorly and show extremely high variance across rollouts.The parameter-estimation sequences were tightly constrained to isolate individual physical parameters, yet rollout variability remained substantial.
- 4 Discussion: Models often generate visually realistic motion paths, including parabolic trajectories and straight drops, without matching realistic physical parameters.In gravity experiments, models failed to reproduce the proper 9.8 m s^2 acceleration, with similar trends for viscosity and friction.
- 4 Discussion: Image-to-video models underperform on gravitational acceleration, with limited temporal input associated with severely low or even negative gravity estimates.The problem was especially pronounced in the gravitational-acceleration subtask.
- 4 Discussion: Models perform similarly on synthetic and real versions of both benchmark subsets, suggesting poor physics understanding rather than a real-versus-synthetic distribution gap.This pattern was observed across both intuitive and physical-parameter evaluations.
5 Conclusion
WorldBench is introduced as a modular, video-based benchmark for measuring concept-specific physical understanding and distinguishing visual appeal from physical accuracy in world-foundation models.
- 5 Conclusion: WorldBench evaluates concept-specific physical understanding rather than relying only on visual realism or coarse-grained dynamic metrics.Its modular video framework measures adherence to physical concepts, constants, and material properties.
- 5 Conclusion: The benchmark helps distinguish visually appealing generations from physically accurate ones and diagnose concept-specific failures in modern world-foundation models.The authors present these diagnostic goals as uses of the benchmark.
A Additional Benchmark Details
The additional benchmark details describe randomized physical scenes, calibrated video capture, parameter-estimation pipelines, and a language-based subset for visual physics reasoning.
- A. Additional Benchmark Details: The intuitive-physics scenes include collisions, support and balance, perspective changes, object permanence, occlusion, bouncing, and wall interactions.Examples include dominoes, ramp-block, table-drop, camera-motion, behind-wall, raised-block, and bouncing-ball scenarios.
- A. Additional Benchmark Details: The benchmark also includes two-object bounce scenes that randomize both spheres’ masses, restitution, and materials.One sphere periodically occludes the other as their vertical positions diverge.
- A. Additional Benchmark Details: WorldBench uses randomized scenarios that vary object types, locations, velocities, shapes, masses, materials, and other physical conditions.The scenarios cover intuitive concepts including support, perspective, object permanence, collisions, occlusion, bouncing, and parabolic motion.
- A. Additional Benchmark Details: Scenes were captured with an iPhone 13 slow-motion camera using checkerboard calibration, manually adjusted camera poses, and manually validated depth.The calibration process supports consistent camera geometry and depth measurements across captures.
- A. Additional Benchmark Details: Acceleration is estimated from quadratic fits and terminal velocity from linear regression, with evaluation pipelines validated against collected videos and accepted ground-truth values.Videos are trimmed and frame rates verified before estimating physical parameters.
- A. Additional Benchmark Details: Rulers and marked beakers provide scale cues in gravity, parabolic-motion, friction, and viscosity experiments.Video-to-video models receive enough input frames to estimate gravity, viscosity, or friction coefficients.
- A. Additional Benchmark Details: The language-based subset evaluates visual interpretation and physical-outcome prediction using True/False or multiple-choice questions over benchmark videos.Figure 3 presents qualitative examples of the 9-frame video-question format.
B.2 Results
The language-based WorldBench evaluation tests closed- and open-source vision-language models on video-based physics questions, with Gemini 2.5 Pro performing best overall but all models remaining near chance.
- B.2 Results: The language-based subset uses one natural-language question per video, with binary or up-to-four-choice answers.The figure caption describes the nine-frame video input and prediction-based questioning format.
- B.2 Results: Gemini 2.5 Pro achieves 49.72% accuracy across all four language-based scenarios, the best overall result.The table caption identifies Gemini 2.5 Pro as the best-performing closed model.
- B.2 Results: All five evaluated language models perform relatively poorly, achieving accuracy only slightly better than chance.Models answer True/False or multiple-choice questions using nine-frame videos and future predictions.
B.3 Additional Quantitative VLM Metrics
The benchmark reveals category-specific differences across models, with the strongest contrast in Walls and the greatest shared difficulty in Object Permanence. Motion Physics is the category where models perform best overall.
- Walls shows the largest model differences: Qwen models are near 0.0 accuracy, while both Gemini models exceed 0.6.
- All five tested models perform similarly in most scene categories.
- Object Permanence is the most difficult category for all models.
- Models perform best overall on Motion Physics scenes.
- Qualitative results are provided for both intuitive physics and physical parameter estimation subsets, but individual rollouts vary greatly and are not broadly representative.The reported figures include intuitive-physics scenes and physical-parameter scenarios.
C.2 Background RMSE Results
The intuitive-physics subset is evaluated with foreground mIoU, while background RMSE provides an additional measure of generated background quality. Background RMSE is computed only over ground-truth background regions, with lower values preferred.
- Foreground mIoU is the primary metric for intuitive-physics video generation performance.
- Background RMSE is an additional metric computed only over the ground-truth segmentation for background regions.
- Lower background RMSE indicates better performance across the reported columns.
C.3 Results Over Time
Longer intuitive-physics rollouts reduce foreground mIoU as prediction extends further into the future, indicating compounding error in physical accuracy. The accompanying evaluation includes qualitative scale/perspective examples and user-study materials.
- Results Over Time: Foreground mIoU is inversely related to prediction horizon when comparing 30-frame and 120-frame rollouts.The analysis interprets this relationship as evidence of compounding error effects in physical accuracy.
- User Study: The user study used 50 Prolific participants, each rating 60 videos on a 1–10 scale.Participants took an average of 30 minutes and were paid $12 per hour.
- Qualitative Examples: Qualitative scale/perspective examples show a metallic sphere rolling away from the camera, with all models performing well on that sample.
- User Study: A separate Google Forms study had 54 starters, with three non-completions and one low-quality submission rejected.
- User Study: Table 9 reports the correlation coefficient between user-study ratings and evaluation mIoU values.
E SAM Validation
SAM2 validation compares automatic segmentations with hand annotations and evaluates tracking through prolonged occlusion. The validation reports high segmentation overlap and continued tracking during an object’s disappearance.
- Validation Scope: The benchmark relies on SAM2 for object segmentation and tracking in both subsets.
- Segmentation Validation: SAM2 segmentations achieve an overall mIoU of 0.9445 against hand-annotated masks.The comparison uses automatic SAM2 segmentations generated from the paper’s prompting methods.
- Occlusion Tracking: SAM2 continues tracking an object through more than 20 frames of disappearance behind another object.The reported tracking behavior is attributed to prompting and maintained state.
F Common Language Effect Size Results
Common Language Effect Size results show widespread difficulty with viscosity and more differentiated performance on friction. Qualitative and temporal results also reveal model-specific failures in motion, scale, object permanence, and future-frame prediction.
- Viscosity: Almost all models perform poorly on viscosity, hovering around 50% chance across nearly all evaluated pairs.The results are reported across per-model and per-scenario settings.
- Friction: Friction results separate models more clearly: distant material pairs are usually correct, while closer pairs remain difficult.The cited example contrasts plastic versus 80-grit sandpaper with closer material comparisons.
- Qualitative scenarios: In motion-physics examples, the autoregressive model distorts object shapes, while the diffusion model changes the vase into a tank and adds a hand.Both models perform well on the cited scale/perspective example of a sphere rolling away from the camera.
- Temporal results: Foreground mIoU decreases as prediction extends farther into the future, with sharp drops after frame 5 or frame 9 depending on the model.The shaded region represents one standard deviation; background RMSE is also reported over time.
- Object permanence: SAM2 tracks an object through its disappearance for more than 20 frames and when it reappears in the object-permanence scenario.The figure documents tracking behavior rather than a Common Language Effect Size value.