Source-linked AI summary
Do generative video models understand physical principles?
Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, Robert Geirhos
TL;DR
The paper asks whether realistic-looking video generation reflects understanding of physical principles, a question existing appearance-focused metrics cannot answer. It introduces Physics-IQ to evaluate physical understanding and finds that current models perform poorly, with physical understanding distinct from visual realism.
Problem
Appearance-focused video metrics assess similarity, smoothness, or feature distributions but are not equipped to evaluate whether models understand physical laws.
Method
The authors introduce Physics-IQ, a comprehensive benchmark with physical-understanding metrics including Spatial IoU, using captions and conditioning frames to evaluate video-model predictions.
Results
VideoPoet (multiframe) scores 29.5 versus 100.0 for the physical variance baseline, and visual realism is not statistically significantly correlated with physical understanding.
Takeaways & Limitations
Visual realism does not imply physical understanding, although some scenarios can be solved and multiframe conditioning performs better than image-to-video variants.
Takeaways & Limitations
The benchmark’s physical-variance baseline reflects natural variability in real videos, so scientific benchmarks should err on the side of caution.
Abstract
from arXiv · showhide
AI video generation is undergoing a revolution, with quality and realism advancing rapidly. These advances have led to a passionate scientific debate: Do video models learn "world models" that discover laws of physics -- or, alternatively, are they merely sophisticated pixel predictors that achieve visual realism without understanding the physical principles of reality? We address this question by developing Physics-IQ, a comprehensive benchmark dataset that can only be solved by acquiring a deep understanding of various physical principles, like fluid dynamics, optics, solid mechanics, magnetism and thermodynamics. We find that across a range of current models (Sora, Runway, Pika, Lumiere, Stable Video Diffusion, and VideoPoet), physical understanding is severely limited, and unrelated to visual realism. At the same time, some test cases can already be successfully solved. This indicates that acquiring certain physical principles from observation alone may be possible, but significant challenges remain. While we expect rapid advances ahead, our work demonstrates that visual realism does not imply physical understanding. Our project page is at https://physics-iq.github.io; code at https://github.com/google-deepmind/physics-IQ-benchmark.
Physics-IQ benchmark
Physics-IQ is a real-world benchmark designed to test whether video generative models understand physical events across diverse laws and challenging continuations. It evaluates generated predictions against ground truth using metrics targeting spatial, temporal, action-magnitude, and pixel-level agreement.
- Dataset: Physics-IQ contains 396 real-world videos spanning 66 scenarios involving solid mechanics, fluid dynamics, optics, thermodynamics, and magnetism.Each scenario focuses on a specific physical law and physical event.
- Dataset: Each scenario is recorded twice from three perspectives, controlling and measuring natural physical variability for evaluation.The paired takes capture variability from chaotic motion, friction, and force trajectories.
- Evaluation protocol: The benchmark targets out-of-distribution physical events that cannot be solved reliably by reproducing memorized training patterns.An example is a domino chain interrupted by a rubber duck, where the continuation depends on the unusual interaction.
- Evaluation protocol: The protocol conditions models on one or more frames, optionally with text describing the observed scene, and evaluates a 5-second continuation against ground truth.Stable Video Diffusion is the only evaluated model that does not accept text conditioning.
- Metrics: Standard realism metrics such as PSNR, SSIM, FVD, and LPIPS are not equipped to assess whether generated videos obey physical laws.They measure appearance, smoothness, feature distributions, or perceptual similarity rather than physically plausible motion and interactions.
- Metrics: Its four metrics measure where action occurs, when it occurs, how much action occurs, and pixel-level fidelity.They are Spatial IoU, Spatiotemporal IoU, Weighted spatial IoU, and MSE.
Results
Physics-IQ reveals a large gap between current video models and real-world physical variability, while performance differs across metrics and categories. Visual realism does not reliably indicate physical understanding.
- Overall physical understanding: 29.5% was the best aggregated Physics-IQ score, versus 100.0% for the physical variance baseline.Physical variance estimates variability between real videos differing only by physical randomness.
- Overall physical understanding: VideoPoet (multiframe) ranked best overall, and multiframe variants outperformed corresponding image2video variants.The benchmark requires predicting future video content, so access to temporal information can help.
- Category and metric results: Performance varied across solid mechanics, fluid dynamics, optics, thermodynamics, and magnetism, but no category was solved.Figure 6 compares these categories across multiple metrics and model outputs.
- Qualitative examples: VideoPoet generated a plausible paint-smearing sequence and Runway Gen 3 generated plausible liquid pouring, but both failed on other physical scenarios.The failures involved a ball falling into a crate and cutting a tangerine with a knife.
- Category and metric results: Spatial-IoU scores were higher than stricter metrics because they assess where an action occurred without requiring the correct timing or amount.Spatiotemporal-IoU additionally evaluates timing, while Weighted-spatial-IoU evaluates the amount of action.
- Visual realism: Sora achieved the best visual-realism MLLM score at 55.6%, while visual realism was uncorrelated with physical understanding.The reported relationship was Pearson’s r = -0.46 with p=.249, not significant.
Discussion
Physics-IQ reveals a substantial gap between visual realism and physical understanding in current video models, while some scenarios are already handled successfully. The benchmark’s metrics offer broad but proxy-based evidence, and its findings leave open whether scaling prediction alone will suffice.
- Physics-IQ provides a comprehensive real-world benchmark and metrics for evaluating physical understanding across diverse settings.
- Current models often fail physically, but VideoPoet successfully simulates some cases such as paint smearing on glass.
- Visual realism and physical understanding are not statistically significantly correlated, so realistic videos can contain physically impossible continuations.
- Models hallucinated objects, although Runway Gen 3 and Sora often produced hallucinations consistent with the scenario.
- The benchmark metrics are proxies rather than direct measures of physical phenomena, and their combined score is intended as a holistic assessment.
- The study intentionally uses conservative scoring that strongly penalizes hallucinations, camera movement, and shot changes.
- Whether scaling next-video-frame prediction alone can produce better physical understanding, or whether more interactive training is needed, remains open.
Supplementary Material
The supplementary material documents the benchmark’s recording setup, scenario switch frames, model specifications, and preprocessing algorithms. It also describes procedures for adjusting frame rates, generating motion masks, and evaluating generated videos with an MLLM.
- Physics-IQ scenarios were recorded from left, center, and right perspectives, with switch frames marking the last conditioning frame before five-second prediction.
- Table 2 summarizes evaluated models by input conditioning, frame rate, and resolution.
- Linear interpolation adjusts video frame rate and optionally resolution to adapt Physics-IQ videos to different model requirements.
- The interpolation procedure creates intermediate frames from neighboring frames before resizing and reconstructing the output video.
- The resulting binary masks highlight moving objects for spatial and temporal masking in Physics-IQ evaluations.
- The MLLM evaluation uses a two-alternative forced-choice prompt asking which of two videos is computer-generated and requiring an explanation.
- Figure 10 uses applied scene distortions to provide intuition for how different MSE values affect images; benchmark videos themselves are undistorted.