PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-IQ Verified

Tim Rädsch, Yuki M Asano, Hilde Kuehne, Stefan Bauer, Priyank Jaini, Robert Geirhos, Carsten T. Lüth

arXiv:2606.18943v1cs.CV

TL;DR

Video generative models need evaluation that distinguishes physical reasoning from plausible-looking motion. This paper audits Physics-IQ, improves prompts, ground truth, and scoring, and finds that Physics-IQ Verified changes model rankings with moderate agreement (Kendall's τ = 0.46).

  • Problem

    Physics-IQ must distinguish whether video models reason about physical reality or merely produce plausible-looking motion.

  • Method

    The paper audits Physics-IQ and introduces clearer prompts, artifact removal, and sample-level aggregation that weights samples and metrics equally.

  • Results

    Kendall's τ = 0.46 between original and verified model rankings, indicating moderate but meaningful ranking changes across six video models.

  • Takeaways & Limitations

    Physics-IQ Verified more faithfully reflects video models' ability to model physical phenomena and supports fine-grained sample-level failure analysis.

  • Takeaways & Limitations

    Sora 2's measured performance was notably worse in April 2026 than in October 2025.

Abstract

from arXiv · show

Video generative models ( VGMs) have become a new frontier that can be used not just for video generation but for a multitude of downstream tasks, including world modeling. To advance these tasks, a good video model must understand the physical reality of the world. Evaluating this understanding is an emerging field and has led to the Physics-IQ benchmark, which quantifies this explicitly by comparing model-generated videos to real-world videos of physical experiments. In this work, we present a systematic audit of the Physics-IQ benchmark, expose shortcomings and propose three solutions that sharpen how we can measure physical understanding of VGMs. Specifically, we improve prompt and ground-truth quality to reduce the influence of confounding factors and further introduce a sample-level scoring system that weights each sample and metric equally. Our resulting benchmark, Physics-IQ Verified, refines 57.6\% of all samples and improves over 34.8\% of prompts. In a comparison study using six image-to-video generative models, we observe moderate but meaningful ranking changes (Kendall's $τ= 0.46$). We hope Physics-IQ Verified advances the community by providing a more reliable signal toward physically accurate VGMs. The code for the benchmark can be accessed at https://github.com/google-deepmind/physics-iq-benchmark

1 Introduction

Physics-IQ Verified audits a benchmark designed to measure whether video generative models understand physical phenomena rather than merely produce plausible motion. It improves prompts, score aggregation, and artifact handling to reduce measurement errors and enable sample-level analysis.

  • Motivation: VGMs are increasingly used as world models for robotics and visual task solving, motivating evaluation of whether they encode physical reality.This use assumes next-frame prediction implicitly teaches causal structure in the physical world.
  • Original benchmark: Physics-IQ compares generated videos with ground-truth recordings from controlled physical experiments using four metrics covering action location, timing, strength, and pixel similarity.Unlike earlier distributional metrics comparing unmatched generated and real-video sets, it directly targets physical understanding.
  • Benchmark refinements: Physics-IQ Verified improves prompt quality, metric aggregation, and artifact handling to reduce evaluation-protocol measurement errors.Prompt templating addresses ambiguity and model-specific prompting practices; artifact cleaning removes unrelated events from reference videos.
  • Benchmark refinements: The sample-level score traces failure modes to individual samples and weighs all samples and metrics equally.This replaces the original dataset-level score, where samples could have unequal influence on the final result.
  • Results: 57.6% of all samples had measurement errors removed, while 34.8% of prompts were corrected for high ambiguity.The refinement also influenced 29.8% of videos and supplied more accurate template-based descriptions for all videos.
  • Results: Evaluating six image-to-video VGMs with the original and verified protocols substantially changed the overall model ranking.The models reacted differently to the evaluation improvements, underscoring the importance of testing the intended physical effect under suitable prompting.

2 Background: The original Physics-IQ benchmark

The original Physics-IQ benchmark evaluates physical understanding by comparing generated continuations with real-world experiment videos across a diverse dataset. It uses defined physical phenomena, multiple input modalities, four metrics, and a normalized aggregate score.

  • Dataset construction: 66 experiments span solid dynamics, fluid dynamics, thermodynamics, optics, and magnetism, producing 396 videos from three views and two takes.Each video is 8 seconds, split into a 3-second conditioning segment and a 5-second ground-truth continuation.
  • Evaluation task: Models predict post-switch-frame physical phenomena from text-only, image-and-text, or video-and-text prompts, depending on the generation task.The benchmark supports text-to-video, image-to-video, and video-to-video models.
  • Evaluation metrics: Performance uses Spatial IoU, Spatiotemporal IoU, Weighted spatial IoU, and Mean Squared Error (MSE) to measure where, when, how much, and how action occurs.The first three metrics are activation-based IoU measures, while MSE is pixel-based.
  • Scoring: The final Physics-IQ score averages metrics, normalizes them by physical variation, and applies a weighted sum with a negative sign for MSE.Physical variation is estimated from the corresponding metrics on the first and second takes of each experiment.
  • Benchmark positioning: Unlike plausibility-judgment and synthetic-simulation benchmarks, Physics-IQ compares generated continuations against real recordings of the same physical setup.This reference-based design supplies a concrete physical target rather than a categorical plausibility judgment.

3 Physics-IQ Verified: Sharpening How Physical Understanding is Assessed

Physics-IQ Verified sharpens physical-understanding assessment by correcting prompt ambiguity, adapting prompts to model input conventions, and replacing dataset-wide score aggregation with sample-level scoring. It also addresses ground-truth activation artifacts that can bias metrics or add irreducible measurement noise.

  • Prompt quality: A well-designed prompt specifies the complete experimental setup and phenomenon catalyst without revealing the outcome, enabling prediction while preserving generation difficulty.Unresolved ambiguity introduces output freedoms unrelated to physical understanding, inflating metric variance or irreducibly reducing performance.
  • Prompt quality: Four prompt defects—factual incorrectness, temporal imprecision, omitted key information, and vague language—either make accurate generation impossible or increase output variance.These defects can cause scores to reflect prompt clarity rather than model capability, motivating minimally invasive corrections.
  • Prompt structure: Six structured fields and model-specific templaters improve conditioning: SETUP, SCENE, and ACTION vary by scenario, while CAM, STYLE, and SCOPE remain consistent across 66 scenarios.STYLE requests a realistic scientific demonstration, SCOPE excludes undescribed actors or interactions, and CAM uses descriptive cinematographic language.
  • Scoring: The sample-level Physics-IQ Verified score uses arithmetic means across metrics and samples, while normalizing physical variation per experiment so each sample’s score has an appropriate upper bound.The original dataset-wide denominator could downweight low-variation experiments and upweight high-variation experiments, allowing individual scores to exceed 1.
  • Ground-truth quality: Ground-truth activation quality is essential because deterministic artifacts measure apparatus behavior, whereas non-deterministic artifacts are neither prompt-specified nor experimentally controlled and add irreducible noise.Both artifact types hinder physical-understanding assessment, but through distinct mechanisms.

4 Experiments

Experiments evaluate six image-to-video generative models under original and best-practice prompts, comparing the original Physics-IQ evaluation with Physics-IQ Verified. Verified evaluation generally raises scores, while prompt improvements, artifact removal, and sample-level scoring have distinct effects on performance and rankings.

  • Experimental Setup: Six I2V VGMs each generate four 198-video sets using original and best-practice prompts on Physics-IQ.The models are Wan 2.2, HunyuanV-1.5, Cosmos3-N, Sora 2, P-Video, and Grok Imagine Video.
  • Method of Analysis: Rankings are evaluated with Kendall’s-τ and Spearman’s-ρ, supplemented by bootstrap confidence intervals, effect sizes, and Wilcoxon significance tests.The bootstrap constructs 500 complete 198-video sets by sampling corresponding videos from the four original sets.
  • Overall Comparison: Physics-IQ Verified increases scores for most models, with Sora 2 and Cosmos-3N showing the highest increases and both outperforming P-Video.The comparison uses artifact-removed ground truth and best-practice prompts.
  • Influence of Prompts: Best-practice prompts significantly improve all primary-metric subscores, with Cohen’s d ≥0.55 for every score; Wan 2.2 is the only model that declines.For Sora 2, reduced unwanted camera motion drives large gains across all metrics.
  • Influence of Artifacts: Artifact removal significantly reduces all IoU-based metric scores and the original Physics-IQ score, with Cohen’s d ≤−1 for all scores.The analysis attributes these reductions to changes in score components, while physical variance is nearly identical across protocols for most metrics.
  • Influence or Benefit of Proposed Score on the Ranking: The proposed sample-level score raises Physics-IQ scores for every model without changing rankings, with bootstrap alignment values ≈1 for both ¯ρ and ¯τ.Its main benefit is therefore score calibration rather than rank separation.

5 Conclusion … B.1 Qualitative Artifact Examples

The paper audits Physics-IQ and introduces Physics-IQ Verified by improving prompts, removing artifacts, and aggregating scores sample-wise. Qualitative examples illustrate how verified prompts and artifact cleaning reduce confounding visual events unrelated to the physical phenomena.

  • 5 Conclusion: The audit identifies three measurement-error sources and proposes text-prompt improvements, artifact removal, and sample-wise score aggregation.Experiments with six VGMs show that these refinements change absolute scores and final model rankings.
  • A.1 Qualitative Prompt Examples: Verified prompts eliminate unintended actions, camera zoom, and underspecified motion information in representative video generations.Examples cover a rubber duck, rotating teapot, and tennis ball interacting with a rubber duck.
  • A.3 Prompt Template Design: Prompts should let humans predict the experimental outcome while avoiding ambiguity that adds non-physical degrees of freedom and increases metric variance.The rewrite also expresses instructions positively because video models may systematically mishandle negations.
  • A.2 Prompt Template Design: Prompt templates add scenario-specific and shared fields, including two novel fields absent from the original prompts.The template covers all 66 scenarios.
  • A.4 Camera Guidance: More thorough cinematographic specifications are needed because evaluation metrics penalize camera-pose and motion deviations, while models remain vulnerable to camera drift.Original prompts only specify “Static shot with no camera movement”.
  • B Artifact Cleaning and Dataset Modification: Artifact cleaning manually annotates artifact extent and freezes affected pixels from a chosen timestamp onward, avoiding new visual information and artificial boundaries.Post-effect removal freezes frames after the physical effect, while mid-effect removal targets spatially disjoint artifact regions.
  • B.1 Qualitative Artifact Examples: Qualitative artifact examples show that grabber motion, recording errors, and rotator movement can activate metrics despite being unrelated to the observed physical phenomena.Verified activation maps remove these unrelated events, including post-effect and mid-effect artifacts.

B.2 Dataset-Wide Modification Overview · C Detailed Metric Definition

The modification overview visualizes changes across the 198-video evaluation set, distinguishing removed, retained, and continuously ongoing physical activity while marking original-version errors.

  • B.2 Dataset-Wide Modification Overview: 198 videos comprise the evaluation set represented by the modification-overview tiles.Each tile corresponds to one take-1 video.
  • B.2 Dataset-Wide Modification Overview: Red marks activity removed after the annotated effect end.This color identifies post-effect activity excluded from the verified evaluation.
  • B.2 Dataset-Wide Modification Overview: Blue marks activity retained in the verified evaluation.The retained activity is distinguished visually from removed activity.
  • B.2 Dataset-Wide Modification Overview: Grey indicates videos whose physical effect continues throughout the full duration.These videos are identified separately from those with activity ending earlier.
  • B.2 Dataset-Wide Modification Overview: Error icons identify videos containing the corresponding error in the original version.The icons indicate the presence of a specific original-version error.
  • B.2 Dataset-Wide Modification Overview: The figure organizes removed, retained, and continuously ongoing activity into a dataset-wide visual overview.The overview combines color-coded activity status with error indicators across the evaluation set.
  • C Detailed Metric Definition: The supplied passage does not provide a detailed metric definition for section C.No metric formula, label, or numerical result is stated in the provided evidence.

C.1 Key improvements from the original to the verified Physics-IQ evaluation. … C.6 Drawbacks of the Original Score

The Physics-IQ evaluation compares generated video continuations with repeated ground-truth takes using activation- and pixel-based metrics, then aggregates them into a composite score. The verified approach addresses dataset-level interpretability, variation weighting, and metric-score domination by enforcing per-metric ceilings before aggregation.

  • C.1 Key improvements from the original to the verified Physics-IQ evaluation.: The evaluation pipeline compares generated video continuations with ground truth using three activation-based metrics and one pixel-based metric before aggregating a final score.Each benchmark sample includes a conditioning image and prompt, one generated continuation, and two repeated ground-truth videos.
  • C.2 Variables and Derived Maps: Each experiment is captured in C takes across S viewing angles, producing C × N = E · S videos represented as H × W × T tensors.H × W denotes spatial resolution and T denotes the number of frames.
  • C.3 Basic Metric Definitions: Three IoU metrics operate on activation maps, while MSE measures pixel-level reconstruction over raw videos normalized to [0, 1].The derived representations include spatial activation, spatiotemporal activation, and weighted spatial activation maps.
  • C.4 Original Physics-IQ Score Aggregation: Each metric M ∈ {SP, ST, WS, MSE} is aggregated across N evaluation videos into a mean score and a physical variation ceiling.The ceiling compares the two ground-truth takes of each experiment to quantify irreducible trial-to-trial variability.
  • C.5 Stable Physics-IQ Score: The original composite averaged the three IoU subscores and subtracted the MSE penalty, but unbounded IoU subscores allowed one exceptional metric to dominate.The stable composite clips each subscore individually at the physical-variation ceiling, limiting each metric’s intended contribution while retaining sensitivity below the ceiling.
  • C.6 Drawbacks of the Original Score: The original score was defined only over the full dataset, downweighted low-variation samples through mean aggregation, and could be dominated by a single overflowing subscore.These drawbacks obscure sample-level performance and allow values above 1, or below 0 for MSE, to distort the composite.

C.7 Sample-Level Physics-IQ Verified Score

Physics-IQ Verified defines a sample-level score aggregated by arithmetic means across the dataset, with a revised MSE interpretation and identical sample weighting for metric subscores. Its variance analysis indicates that mean physical variance may underweight low-variance samples.

  • Sample-level aggregation: The proposed Physics-IQ score operates at the sample level across the entire dataset and uses arithmetic means to attribute improvements to individual metrics.This aggregation makes each sample’s metric contribution explicit in the per-sample score.
  • Metric interpretation: The revised MSE measures how many times the generated MSE exceeds the MSE associated with physical variation.This changes the interpretation of MSE within the verified scoring framework.
  • Variance analysis: The physical-variance distribution is non-Gaussian, suggesting that mean physical variance can downweight samples with low physical variance.The distribution supports concern that averaging physical variance may reduce the influence of low-variance samples.
  • Final score: The final Physics-IQ Verified score is the arithmetic mean across all samples.The benchmark therefore aggregates sample-level scores into a single dataset-level score by averaging them.
  • Metric subscores: Each verified metric subscore is obtained over the entire dataset by summing across all samples identically.This applies the same sample-level aggregation procedure to every metric.

D Experimental Setup · D.1 Evaluated Models

The evaluated models are documented through generation settings in Table 2, covering conditioning, seed control, pricing, and undisclosed values. For Cosmos3-Nano, prompts are passed directly without language- or vision-model preprocessing, and Opus 4.8 upsampling improves bpp performance by approximately one Physics-IQ verified score.

  • D.1 Evaluated Models: Table 2 provides the generation settings for the evaluated image-to-video models.
  • D.1 Evaluated Models: All evaluated models use text conditioning and a single conditioning frame.
  • D.1 Evaluated Models: Seed control records whether each model allows its seed to be configured.
  • D.1 Evaluated Models: Model prices come from leading API providers or estimated GPU market rates as of May 2026.
  • D.1 Evaluated Models: For Cosmos3-Nano, op and bpp prompts are passed directly to the video models without LLM or VLM preprocessing.This preserves the prompts' direct influence on model behavior.
  • D.1 Evaluated Models: Cosmos3-Nano bpp with Opus 4.8 upsampling improves over bpp without upsampling by ~1 Physics-IQ verified scor.The supplied passage truncates the final word of the reported score comparison.

E Additional Results

The section provides full quantitative results for both the original and verified Physics-IQ scores, along with Sora 2 sanity checks and bootstrap analyses of ranking stability.

  • Tables 3 and 4 report the full quantitative results for the Physics-IQ Original Score and Physics-IQ Verified Score.
  • Tables 5 and 6 provide additional Sora 2 sanity checks across evaluation dates and generation settings.
  • Figures 16, 17, and 18 present bootstrap ranking analyses assessing ranking stability.

E.1 Main Results Overview · E.2 Sora-2 Temporal Comparison

The main-results and Sora-2 temporal-comparison sections evaluate video models under original and templated prompts using both original and verified Physics-IQ evaluations. They also compare Sora 2 assessments across October 2025, the reported release-era results, and an April 2026 sanity check.

  • E.1 Main Results Overview: Each evaluated video model generates four video sets combining original prompts (op) or best-practice templated prompts (bpp) with original or verified evaluation.The verified evaluation removes artifacts; scores are multiplied by 100 and reported as points, with µ ± σSTD over four runs.
  • E.1 Main Results Overview: The original Physics-IQ overview reports model results across both prompt types and both evaluation procedures.This overview is based on four-run visualized scores reported as points.
  • E.1 Main Results Overview: The verified Physics-IQ overview reports the corresponding results after artifact removal while retaining the same prompt and evaluation design.Scores remain multiplied by 100 and reported as points, with µ ± σSTD over four runs.
  • E.2 Sora-2 Temporal Comparison: Sora 2 is assessed using one October 2025 run, values from near its original release, and an additional April 2026 sanity check.The comparison tests whether the generations validly assess Sora 2 performance over time.
  • E.2 Sora-2 Temporal Comparison: The October 2025 Sora 2 run is described as close to the original release and as showing the highest scores.This temporal comparison is reported for both the original and verified Physics-IQ evaluations.
  • E.2 Sora-2 Temporal Comparison: The Sora 2 tables include the values reported in the main paper alongside the temporal comparison and sanity-check results.All scores are multiplied by 100 and reported as points.
  • E.2 Sora-2 Temporal Comparison: Standard deviations cannot be computed for the single October Sora 2 run.This limitation applies to the October run in both the original and verified comparison tables.

E.3 Bootstrap Ranking Analysis · F Related Works

Bootstrap analyses show that Physics-IQ Verified raises scores without materially changing model rankings, while Related Works situates the benchmark among perceptual, synthetic, judgment-based, proxy-based, and law-based evaluation approaches. Physics-IQ is presented as the direct predecessor whose adoption motivates auditing its measurement reliability.

  • E.3 Bootstrap Ranking Analysis: Original and verified evaluations exhibit stable rankings, with both mean Spearman-ρ and Kendall-τ showing meaningful ranking differences in the broader comparison.Bootstrap scatter plots encode mean ranks and rank frequencies, while correlation distributions assess agreement across and within evaluations.
  • E.3 Bootstrap Ranking Analysis: Verified Physics-IQ scores are higher for every model, yet the ranking bump plot shows no ranking differences.The comparison includes standard deviations across four runs and a bootstrap ranking scatter plot.
  • E.3 Bootstrap Ranking Analysis: Bootstrap correlation analyses indicate almost perfect agreement between original and verified Physics-IQ rankings.The distributional assessment compares correlations across evaluations and within evaluations.
  • F Related Works: Video-generation evaluation has expanded beyond perceptual realism toward fine-grained dimensions including motion smoothness, temporal flickering, spatial consistency, and subject identity.Earlier work used metrics such as Fréchet Video Distance and broad evaluation suites, while VBench and VBench++ decompose video quality into more specific axes.
  • F Related Works: Synthetic benchmarks study physical reasoning through simulated interaction and prediction of physical-scene evolution, including PHYRE, Physion, and Physion++.These benchmarks address classical mechanics puzzles, future scene evolution, and latent-physics inference in simulated environments.
  • F Related Works: VGM physical-reasoning benchmarks use human or vision-language-model judgments to assess commonsense interactions and prompts spanning multiple physical laws.VideoPhy and PhyGenBench provide scalable coverage of physical concepts through judgment-based evaluation.
  • F Related Works: Motion-, mask-, and trajectory-based methods evaluate plausibility through quantities such as acceleration and velocity variance, but remain tied to synthetic settings or proxy measurements.VAMP proposes visual-appearance and motion-plausibility metrics, while Kang et al. study scaling for classical-mechanics modeling in synthetic environments.
  • F Related Works: Morpheus evaluates generated trajectories against governing equations and conserved invariants from real laboratory experiments, complementing Physics-IQ while remaining limited to object-centric phenomena.Its physics-informed neural networks assess quantities including total energy and angular momentum when reliable trajectories can be extracted.

NeurIPS Paper Checklist

The checklist finds that the paper’s claims are supported, its experimental methodology is reproducibly documented, and it has no theoretical results requiring formal proofs. It also discusses both positive and potential negative societal impacts.

  • Claims: The abstract and introduction accurately present the audit, three benchmark refinements, and experiments on six image-to-video models.The stated refinements are prompt improvements, artifact cleaning, and sample-level score aggregation.
  • Theory assumptions and proofs: The paper introduces no theoretical results or formal theorems, providing metric and score definitions in the main text and Appendix C.Consequently, the checklist does not require theorem assumptions or complete proofs.
  • Experimental result reproducibility: The paper discloses the dataset, evaluated models, prompts, generation protocol, evaluation variants, and statistical analysis needed to reproduce its main claims.Additional metric definitions and result tables are included in the appendix.
  • Broader impacts: The paper discusses positive impacts from more reliable physical-understanding benchmarks and negative risks from strengthening video-generation systems that could be misused.The work does not release a new generative model.
Loading 2606.18943v1…