Source-linked AI summary
World Models' Last Exam in Physics
Mingju Gao, Qingle Liu, Yuzhao Peng, Xinjie Lin, Ziming Qin, Zheng Jiang, Wenyi Li, Calvin Xiao, Youjie Zheng, Kaisen Yang, Qinhuai Na
TL;DR
Video world models can generate physically implausible events, raising concerns about their reliability for prediction and planning. This paper introduces a measurement-based benchmark of controlled tasks and evaluates eight models; the results show persistent physical difficulties, with the best model scoring 57.76 out of 100.
Problem
Video models can generate physically implausible motions and interactions, raising concerns about their reliability for prediction and planning.
Source: §1 · p. 1Method
The benchmark pairs controlled image–prompt tasks with predefined physical criteria and combines screening with task-specific measurements of observable relationships.
Source: §1 · pp. 2–3Results
Across eight models, evaluation reveals persistent difficulties in reproducing quantitative physical relationships and completing required events; the best model achieves 57.76 out of 100 overall.
Source: §1 · p. 3, §5 · p. 12Takeaways & Limitations
Explicit measurements provide evidence for interpreting failures and assessing generated videos across physical domains.
Source: §5 · p. 12The tests cover only observable relationships under stated experimental assumptions, leaving unmeasured physical properties unverified.
Source: §2.2 · p. 5
Abstract
from arXiv · showhide
Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and planning in embodied AI systems. Existing evaluations often rely on model-based judgments or reference videos, while direct physical tests largely focus on mechanics. We introduce World Models' Last Exam in Physics, a measurement-based benchmark for evaluating physical consistency in video world models. The benchmark comprises 40 controlled tasks spanning mechanics, optics, fluids, thermal and phase-change phenomena, electromagnetism, and surface tension. Each task pairs an initial image and a generation prompt with predefined physical criteria, enabling interpretable tests of observable physical relationships without requiring reference videos. Its evaluator combines task-observability screening with task-specific quantitative physical measurements. Experiments on eight video generation models across 1,280 videos reveal persistent physical inconsistencies and substantial variation across tasks, with the best model achieving an overall score of 57.76 out of 100. Evaluation on synthetic videos with known physical relationships provides evidence for the validity of the measurement module under controlled conditions. The evaluator also achieves higher agreement with human judgments than a direct vision-language model baseline in both within-task rankings and pairwise comparisons. By combining coverage across physical domains with scores grounded in measurable evidence and explicit measurement limitations, the benchmark provides an interpretable basis for diagnosing physical inconsistencies and tracking progress toward physically consistent video world models.
1 Introduction
The introduction presents physical consistency as a requirement for using video models in prediction and planning, while noting that existing evaluation approaches leave gaps. It introduces a measurement-based benchmark and reports persistent physical inconsistencies across evaluated models.
1 Introduction
- Physically implausible generated events raise concerns about the reliability of video models as world simulators for prediction and planning.Systematic evaluation can characterize which phenomena models reproduce reliably and where they fail.Source: §1 · p. 1
- Existing evaluations use VLM judgments, reference comparisons, or direct physical tests, but direct tests mainly cover mechanics and some measurement approaches require real recordings or simulator ground truth.The benchmark targets observable relationships without requiring reference videos or simulator ground truth.Source: §1 · pp. 1–2
- 57.76 out of 100 was the best overall score among eight video generation models, with substantial variation across tasks and physical domains.The evaluation covered 40 tasks and 1,280 generated videos.Source: §1 · p. 3
- The benchmark evaluates 40 controlled tasks spanning mechanics, optics, fluids, thermal and phase-change phenomena, electromagnetism, and surface tension.Each task pairs an image–prompt input with observable physical relationships and predefined criteria.Source: §1 · pp. 2–3
- Its evaluator combines temporal-consistency and task-observability screening with task-specific measurements, distinguishing physical test outcomes from insufficient measurement evidence.Passing the screen neither establishes physical correctness nor guarantees recovery of every required quantity.Source: §1 · p. 3
2 Method
The method defines controlled tasks through assumptions, image–prompt inputs, and measurable physical criteria, then evaluates generated videos against those criteria. Task design and scoring emphasize observable relationships while retaining limits imposed by the input setup and available evidence.
2.1 Overview and Task Formulation
- Each task specifies physical assumptions, an image–prompt instance, and measurable criteria; for example, free fall tests whether falling distance is proportional to elapsed time squared.The assumptions can include release from rest and negligible air resistance.Source: §2.1 · p. 3
2.2 Physics-Grounded Task Design
- The 40 tasks organize physical predictions as spatial, temporal, or coupled spatiotemporal relationships, designed to be tested without absolute spatial or temporal calibration where possible.Matched comparative setups can cancel shared unknown factors; the projectile height-to-range ratio still requires appropriate side-view geometry because it does not remove arbitrary perspective distortion.Source: §2.2 · pp. 4–5
2.3 First-Frame Construction and Video Generation
- Task-specific first frames and video prompts are iteratively refined through pilot generation and human inspection to make initial configurations and required measurements observable.The finalized image–prompt pairs were submitted to eight models, producing 1,280 videos across 40 tasks.Source: §2.3 · p. 5
2.4 Physical Evaluation Protocol
- The evaluator screens temporal consistency, attempts physical measurements independently, and gates the physical-score contribution when consistency is below 80 or no physics score is recorded.Recovered quantities are compared with task-specific predictions; indicators are aggregated into a physics score, with intermediate measurements and visual evidence retained for diagnosis.Source: §2.4 · p. 6
3 Experiments
Across 40 tasks, eight video generation models show substantial variation in physical performance, and the strongest model averages about 57.76 out of 100. The evaluator also agrees with human judgments more often than a direct VLM baseline on both reported agreement measures.
3.1 Settings
- Across 40 tasks, the models received four samples per task, and task evaluators extracted measurements and compared them with task-specific physical constraints.Measurements included trajectories, periods, angles, and event timings; the physical score averages task-defined indicators with normalization parameters fixed across models.Source: §3.1 · pp. 6–7
3.2 Main Results
- On P18, Seedance and VBVR scored 96.15 and 95.75 while other models scored at most 49.82; on P31, Seedance and MiniMax scored 98.34 and 97.24 while others scored at most 15.00.Figure 2 shows four temporally ordered frames per model for each task, including missing reflected rays and additional spheres in some outputs.Source: §3.2 · pp. 7, 9, Figure 2 · p. 7
- Seedance averaged approximately 57.76; on P26, the model-averaged score was 3.45 (maximum 15.00), while all models scored 93.09–99.01 on P21.The Table 2 composite score applies a per-video consistency gate, so physical scores contribute only when C ≥80.Source: §3.2 · pp. 7, 9, Table 2 · p. 8
- Hard tasks passed the consistency gate on 56.9% of videos versus 89.4% for Easy tasks, and Seedance led Hard tasks with 37.76 versus MiniMax’s 19.87.All available free-fall videos passed the gate, yet the cross-model average composite score was 26.61; the Hard group includes interaction and successive-event tasks.Source: §3.2 · p. 9
3.3 Agreement with Human Judgments
- The evaluator agreed with human orderings on 473 pairs (51.58%) versus 398 (43.40%) for Direct VLM, and confirmed 85 pairwise agreements (53.12%) versus 73 (45.62%).Human–Human references were 519 pairs (56.60%) for ranking agreement and 58.54% for confirmed pairwise agreement.Source: Figure 4 · p. 10, §3.3 · p. 11
3.2 Main Results
- Synthetic reference videos received mean physical scores of 97.93, 96.64, and 98.25 for Easy, Medium, and Hard tasks, with an overall mean of 97.53.The 480 references were 2D animations across 40 tasks; the consistency gate was bypassed to isolate physical measurement.Source: Figure 3 · p. 9, §3.2 · p. 10
3.3 Agreement with Human Judgments
- Human comparisons used ratings on a five-point scale and 917 assessable within-task pairs, alongside 160 predefined pairs for confirmed pairwise agreement.Ten annotators supplied judgments for 320 videos across 40 tasks; ties were retained in the within-task ranking comparison.Source: §3.3 · pp. 10–11
4 Related Work
Prior video-generation evaluations use quality metrics, evaluator judgments, reference observations, or direct physical measurements, each with different evidential limits. The paper positions its controlled, quantitative tests across physical domains as a complement to existing measurement-based work focused mainly on selected mechanical dynamics.
Physics-Aware Video Generation
- Physics-aware generation methods incorporate physical structure through simulation, reasoning, representation learning, or post-training, motivating tests of which relationships generated videos preserve.The paper names approaches including trajectory-guided synthesis, physical representations, and rewards built from measurable proxies.Source: §4 · p. 11
Video Generation Evaluation
- Existing video-generation benchmarks assess general quality, evaluator judgments, reference observations, or direct physical measurements, but the first three do not directly establish specific physical correctness.Judgment scores reflect evaluator assessments; reference similarity requires suitable references and does not isolate specific physical-law violations.Source: §4 · pp. 11–12
- Prior measurement-based evaluations target selected mechanical dynamics, while this benchmark tests spatial, temporal, and spatiotemporal relationships across mechanics, optics, and other physical domains.It uses controlled experiments, task-specific measurements, and scale-canceling comparisons where applicable, without VLM judgments of physical correctness.Source: §4 · p. 12
5 Conclusion
The conclusion presents a 40-task benchmark grounded in explicit physical measurements and reports persistent inconsistencies across eight video-generation models. A case study shows how extracted measurements support task-specific assessment and expose potential tracking or boundary-detection errors.
5 Conclusion
- The benchmark uses 40 controlled tasks and explicit task-specific measurements, separating temporal consistency and task observability from physical correctness.Scores are grounded in measurable relationships, with evidence retained to interpret failures.Source: §5 · p. 12
- Eight video-generation models showed persistent difficulty reproducing quantitative physical relationships and completing required events; synthetic videos supported measurement validity, and human agreement exceeded direct VLM scoring.Human agreement was higher for both within-task rankings and pairwise comparisons; the evaluator remains constrained by visibility and reliable extraction of task-relevant evidence.Source: §5 · p. 12
A Case Study on Evaluator
- For freezing expansion, the evaluator estimates sample height from the material surface and container bottom; for pendulums, it tracks both bobs to recover motion trajectories.These intermediate measurements connect video content to physical assessment while leaving boundary-detection and tracking errors available for inspection.Source: A Case Study on Evaluator · p. 17
B Detailed Task Catalog
The task catalog specifies the benchmark’s tasks, measured quantities, normalization, and scoring rules. Difficulty labels summarize results for the evaluated models, while physical scores remain limited to defined metrics and observable relationships.
B Detailed Task Catalog
- The benchmark covers 40 tasks in nine physical families with 65 metrics: one primary metric per task and a secondary metric for 25 tasks.Tasks are grouped by physical family and difficulty; catalog entries describe scenes, measured visible quantities, and score normalization.Source: B Detailed Task Catalog · p. 17, Table 3 · p. 18
- Difficulty labels comprise 15 Easy, 15 Medium, and 10 Hard tasks, based on scores averaged across available seeds and equally across eight models.Missing videos are excluded, and the labels describe empirical difficulty for the evaluated models.Source: B Detailed Task Catalog · p. 17
- Metric residuals are denoted e_j and normalized scores s_j in [0, 1], with higher scores indicating better agreement; residuals are dimensionless unless units are specified.The scale a gives q = 0.5 at |e| = a and is not a pass threshold; image y increases downward, with physical-height-change signs specified by task.Source: B Detailed Task Catalog · p. 18
- A task’s physical score averages its defined metrics, with P = 100s1 when no secondary metric is defined; the composite score combines P and C.C is the automatic consistency and observability score and contributes once per video; metrics undefined for a task are excluded from its averaging denominator.Source: B Detailed Task Catalog · p. 18
- An unresolved required measurement receives operational score zero while its raw measurement remains undefined, so that zero is neither a successful measurement nor evidence of a physical violation.Input or execution errors yield no valid physical score, and each score evaluates only its stated observable relationship.Source: B Detailed Task Catalog · p. 19
B.1 Translational Motion and Collisions
The catalog defines mechanics tasks through controlled motions and task-specific measurements, with normalization rules for comparing observed behavior against physical relationships. Several tasks include conditions that determine whether measurements are resolvable or how partial evidence is scored.
B.1 Translational Motion and Collisions
- P1 tests a dropped ball with at least four visible rebounds, scoring restitution-related timing and height behavior, including height increase and measurable restitution variability.Trackable single motion without repeated rebounds receives s1 = s2 = 0.1.Source: §B.1 · p. 19
- P2 tests free fall from rest before ground contact, measuring acceleration consistency from equal-interval velocity increments and applying its defined normalization.The task is specified in a fixed side view.Source: §B.1 · p. 19
- The free-fall acceleration test requires first displacement to exceed localization noise; resolved constant-speed descent fails that test..Source: §B.1 · p. 19
- P3 launches identical balls simultaneously at 30° and 60° with equal initial speed, measuring range agreement and checks of trajectory fit and speed matching.Both balls follow separate lanes and land at their launch heights.Source: §B.1 · p. 19
- P4 launches a ball at 45° and measures the relation between apex height and range, with velocity-variation and trajectory-fit checks.When return to launch height is absent but launch, apex, and descent are resolved, the primary metric uses a partial-arc estimate.Source: §B.1 · p. 19
- P5 tests a head-on elastic collision between equal-mass balls, measuring horizontal momentum agreement; elasticity is not separately scored.The moving red ball strikes an initially stationary blue ball on a level surface.Source: §B.1 · p. 20, §B.2 · p. 20
B.2 Rolling, Friction, and Rigid-Body Statics
This section defines controlled mechanics tasks and measurements for testing whether generated motion follows observable physical relationships. The tasks span sliding, chain shape, rolling, pivoted rotation, and friction on an incline.
Sliding onset independent of block mass (P6)
- Two different-mass blocks are tested for matching sliding onset; onset-frame and onset-angle differences are measured, with both-stationary videos receiving scores of 0.7 for each metric.The task compares onset frames and inclinations, then normalizes both differences; the special stationary case assigns s1 = s2 = 0.7.Source: §B.2 · p. 20
Static shape of a hanging chain (P7)
- A hanging chain released from a nonequilibrium shape is evaluated by catenary-fit error and support-height difference, normalized against sag and support span.The two scores use thresholds of 0.10 for normalized contour-fit RMSE and 0.02 for support-height difference.Source: §B.2 · p. 20
Pure rolling of a solid sphere (P8)
- A marked sphere rolling down an incline is assessed by fitting center displacement to unwrapped rotation angle and measuring fit residual over valid moving frames.The evaluation also uses sphere radius in the displacement–rotation fit.Source: §B.2 · p. 20
Rolling solid sphere versus thin ring (P9)
- A solid sphere and equal-radius thin ring rolling from rest are compared using their median travel-time ratio over shared distances.Only the first metric is defined for this task.Source: §B.2 · p. 20
Forced rotation about a support edge (P10)
- For forced block rotation, the evaluator combines pivot drift, ground gap, and shape error, and scores this measurement only when observed rotation reaches 5°.Pivot drift and ground gap use 95th-percentile values adjusted for pixel noise; rotation is measured over continuous intervals of at least four frames and 0.3 seconds.Source: §B.2 · pp. 20–21
Sliding up and down a rough incline (P11)
- A block sliding up and then down a rough incline is evaluated by comparing ascent and descent acceleration magnitudes; a confirmed ascent-only event receives s1 = 0.1.The task specifies an approximately 30° incline and does not define a second metric.Source: §B.2 · p. 21
B.3 Pendulum Motion and Oscillations
This section operationalizes pendulum tests through measured periods, trajectories, lengths, and cycle variability. The tasks examine mass, length, amplitude, and small-angle relationships.
Pendulum period independent of mass (P12)
- Pendulum period independence from mass is also measured using the large-to-small period ratio, compared with an elliptic-integral reference ratio.A second score measures the ratio’s increase above 1.Source: §B.3 · p. 21
Pendulum period independent of mass
- For pendulums with different bob masses, the measurements compare periods and first-frame lengths recovered from swing trajectories.The period metric compares heavy and light bobs; the length metric compares their recovered lengths.Source: §B.3 · p. 21
Finite-amplitude pendulum periods (P13)
- Equal-length pendulums released at 15° and 30° test whether the larger-amplitude pendulum has a longer period.The stated physical relationship is that the larger-amplitude pendulum should have a longer period.Source: §B.3 · p. 21
Pendulum period versus length (P14)
- Identical pendulums with a 1:2 string-length ratio are evaluated by period difference and cycle-to-cycle period variability.Variability is measured separately for the shorter- and longer-length pendulums.Source: §B.3 · pp. 21–22
Small-angle pendulum isochronism (P15)
- Equal-length pendulums released simultaneously from different small amplitudes are assessed by period comparison and sinusoidal-fit residuals.The residuals are measured in radians for each angular trajectory.Source: §B.3 · p. 22
B.4 Optics and Projective Geometry
This section tests optical and projective relationships using visible rays, shadows, interfaces, and geometric markers. Its measurements include both relationship errors and explicit limits on what the observations establish.
Cross-ratio of four points on a rigid rod (P16)
- A rod with four fixed collinear markers is evaluated by variation in projected cross-ratio and framewise collinearity error.Collinearity error is normalized by marker span.Source: §B.4 · p. 22, §B.5 · p. 23
Refraction at an air–water interface (P17)
- At a stationary air–water interface, the evaluator measures Snell’s-law residual and the mismatch between incident- and refracted-ray interface intersections.The intersection mismatch is normalized by calibrated container width.Source: §B.4 · p. 22
Specular reflection (P18)
- Specular reflection is assessed by the difference between incidence and reflection angles and by reflected-ray contact error at the mirror.Contact error is normalized by frame diagonal.Source: §B.4 · p. 22
Concurrent shadows from a point source (P19)
- Four rods illuminated by a stationary point source test whether backward extensions of their shadow axes meet at a common point.The metrics measure normalized RMS concurrence error and variation among pairwise intersection estimates.Source: §B.4 · p. 22
Refraction and total internal reflection (P20)
- Four beams at a water interface test consistency of independently inferred refractive indices during refraction or total internal reflection.Total internal reflection provides a bound rather than an exact critical-angle measurement, and no second metric is defined.Source: §B.4 · p. 23
B.5 Hydrostatics and Buoyancy
This section defines controlled, image-based tests of observable physical relationships across hydrostatics, phase changes, electromagnetism, granular flow, and surface tension or viscous flow. One task tests volume conservation when two unequal free water droplets merge into a near-spherical droplet, using two normalized measurement scores.
B.5 Hydrostatics and Buoyancy
- The benchmark tests communicating-vessel levels, floating-ice submergence, and liquid-surface orientation using equilibrium levels, residual motion, submerged fraction, and angle or trajectory measurements.The surface-orientation task uses a freely falling steel ball as a visible gravity reference.Source: §B.5 · p. 23
B.6 Phase Transitions and Melting
- Phase-change tasks measure expansion during freezing and test whether floating freshwater ice melting in freshwater preserves the initial water level.They quantify height expansion and vessel-width stability for freezing, and level change and vessel-width change for freshwater melting.Source: §B.6 · pp. 23–24
- Other melting tasks test the level change after an embedded stone sinks, rising level and no residual ice in saltwater, and whether crushed ice leads an intact block in liquid height.The crushed-ice comparison requires a lead beyond localization uncertainty; the stone task is scored only when the level change exceeds measurement uncertainty.Source: §B.6 · p. 24
B.7 Electrostatics, Magnetism, and Electromagnetic Induction
- Electromagnetic tasks test whether a magnet descends more slowly than a matched nonmagnetic object on copper and whether coil entry coincides with indicator illumination.The braking task compares arrival time and speed; the induction task combines evidence of magnet entry and illumination, allowing a lower-bound score when the indicator is unreadable.Source: §B.7 · pp. 24–25
- Additional tests measure symmetric repulsion of charged balls and opposing settled directions of two compass needles around an energized wire.They quantify string-angle and displacement asymmetry for charged balls and angular separation between compass needles.Source: §B.7 · p. 25
- Ring and plate tasks compare closed versus split aluminum-ring rise and the damping of matched solid versus slotted plates in magnetic fields.The ring score uses normalized peak-rise ratios, while the plate score compares complete swing cycles in a common observation window.Source: §B.7 · p. 25
B.8 Granular Media and Discharge Flow
- Granular-flow tasks assess whether unequal sand piles maintain comparable repose angles and compare water and sand discharge from matched conical funnels.The measurements compare pile-angle ratios and fit discharge as Q ∝ h^β for each material.Source: §B.8 · pp. 25–26
B.9 Surface Tension and Viscous Flow
- Surface-tension and viscous-flow tasks measure capillary rise in narrower tubes, terminal settling speeds of unequal spheres, and the partition curvature of joined unequal soap bubbles.The sphere comparison requires a radius ratio of at least 1.1, and the bubble measurement uses identifiable partition intervals lasting at least three frames and 0.08 s.Source: §B.9 · p. 26
- The task measures volume-conservation error from fitted radii and a second error based on the final and mean initial roundness; both scores use q(e; 0.10).Roundness is defined as 1 − RMSradial/rfitted; the passage gives s1 = q(e1; 0.10) and s2 = q(e2; 0.10).Source: §B.9 · p. 27
C Detailed Evaluation
Tables 4 and 5 report consistency, physical, and combined scores for eight models across 40 tasks. The combined score gates the physical score using the consistency score, while retaining consistency and independently evaluated physical scores.
C Detailed Evaluation
- S = 0.15C + 0.85P 1[C ≥80], with the gate applied before averaging across available seeds for each model–task pair.C, P, and S are scored on a 0–100 scale; the gate affects only P’s contribution to S.Source: C Detailed Evaluation · p. 27
- Physical scores are evaluated independently for all available videos, including those with C < 80, and are not set to zero solely because the consistency gate rejects a video.The original C and independently evaluated P are retained.Source: C Detailed Evaluation · p. 27
- Tables 4 and 5 report C, P, and S for eight video generation models across 40 tasks, with the all-model mean repeated in both tables.Each table covers four models and half of the task-level evaluation.Source: Table 4 · p. 28, Table 5 · p. 29