PaperlayerView exact revision on arXiv

Source-linked AI summary

IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments

Florian Bordes, Quentin Garrido, Justine T Kao, Adina Williams, Michael Rabbat, Emmanuel Dupoux

arXiv:2506.09849v1cs.CV

TL;DR

IntPhys 2 addresses the limits of a simple, saturated benchmark with a more diverse video evaluation of four intuitive-physics principles. On this benchmark, most models perform near chance, while human performance remains strong.

  • Problem

    The original IntPhys benchmark used simple environments with limited variation and had become saturated, despite the importance of assessing intuitive physics in AI models.

    Source: §1 · p. 1

  • Method

    IntPhys 2 evaluates four intuitive-physics principles using diverse, photorealistic video scenes with fixed and dynamic camera perspectives.

    Source: §1 · pp. 1–2

  • Results

    Predictive models perform close to chance (50%) across IntPhys 2 subsets and the held-out set, while human performance remains consistently strong across conditions.

    Source: §1 · p. 2, §4.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    IntPhys 2 highlights the need for novel model architectures and training methodologies to bridge the gap between artificial and human cognition.

    Source: §6 · p. 9

    The benchmark uses synthetic environments, covers a limited set of physical principles, and restricts model video input to sampled frames unlike human perception.

    Source: §6 · p. 9

Abstract

from arXiv · show

We present IntPhys 2, a video benchmark designed to evaluate the intuitive physics understanding of deep learning models. Building on the original IntPhys benchmark, IntPhys 2 focuses on four core principles related to macroscopic objects: Permanence, Immutability, Spatio-Temporal Continuity, and Solidity. These conditions are inspired by research into intuitive physical understanding emerging during early childhood. IntPhys 2 offers a comprehensive suite of tests, based on the violation of expectation framework, that challenge models to differentiate between possible and impossible events within controlled and diverse virtual environments. Alongside the benchmark, we provide performance evaluations of several state-of-the-art models. Our findings indicate that while these models demonstrate basic visual understanding, they face significant challenges in grasping intuitive physics across the four principles in complex scenes, with most models performing at chance levels (50%), in stark contrast to human performance, which achieves near-perfect accuracy. This underscores the gap between current models and human-like intuitive physics understanding, highlighting the need for advancements in model architectures and training methodologies.

1 Introduction

IntPhys 2 extends an increasingly saturated, simple benchmark with more complex scenes and tests of four intuitive-physics principles. Evaluations find that current models still struggle, especially on permanence and immutability, while human performance remains strong.

1 Introduction

  • The original IntPhys benchmark used simple environments with limited real-world variation and had become saturated, motivating a more challenging assessment.Predictive models such as V-JEPA had achieved high performance on the original benchmark.Source: §1 · p. 1
  • IntPhys 2 evaluates four cognition-inspired physical principles in a more comprehensive benchmark, with diverse scenarios and sophisticated visual elements.The benchmark includes photorealistic scenes, complex lighting, shadows, occlusions, textures, and fixed and dynamic camera perspectives.Source: §1 · pp. 1–2
  • In the illustrated scene, a ball passing through a brick without changing trajectory is impossible, while its trajectory should remain unchanged without the obstacle.The scene uses obstacle presence to distinguish possible from impossible outcomes across video pairs.Source: Figure 1 · p. 2
  • Fixed and moving cameras create more realistic occlusions and more complex scenes than the original benchmark’s simple environments.Movement-based occlusions can capture an observer moving their head away and back to the original viewpoint.Source: §1 · pp. 1–2
  • Models still struggle with permanence and immutability on IntPhys 2, whereas human performance remains consistently strong across conditions.The evaluation covers state-of-the-art predictive models and multimodal large language models.Source: §1 · p. 2

2 Benchmark Design

IntPhys 2 evaluates intuitive physics through occlusions, combining four physical principles with more realistic scenes, camera movement, and diverse tasks. Its split design separates debugging and graded main-set evaluation from a metadata-free held-out test.

2 Benchmark Design

  • The benchmark targets permanence, immutability, spatio-temporal continuity, and solidity as core intuitive-physics principles.These principles concern object persistence, shape and structure, smooth movement, and objects occupying space without passing through one another.Source: §2 · p. 3
  • Each scenario uses quadruplets with two possible and two impossible videos, arranged to reduce low-level perceptual biases.The possible video in one scenario serves as the impossible video in another, requiring attention to temporal dependencies and physical principles.Source: §2 · p. 3
  • IntPhys 2 focuses exclusively on occlusions and uses static and dynamic cameras to create more realistic viewpoint changes.The scenes also add realism and task diversity, with multiple tasks within each condition rather than a single scene type.Source: §2 · p. 3
  • The benchmark provides Debug, Main, and held-out splits; Main has Easy, Medium, and Hard sub-splits, while held-out metadata is not released.Debug probes sensitivity to video-generation artifacts; the Main split supports zero-shot evaluation, and the held-out split is intended to test performance without plausibility metadata.Source: §2 · pp. 3–4, Table 1 · p. 4
  • IntPhys 2 covers longer occlusion durations than other compared benchmarks, increasing the short-term-memory demand.Its density plots compare occlusion durations, while the benchmark description identifies retaining and using information across brief intervals as a key demand.Source: §2 · p. 3, Figure 2 · p. 4

3 Evaluation protocol : measuring intuitive physics understanding

The evaluation uses human judgments and separate protocols for multimodal language models and prediction-based models. The protocols accommodate differences in model inputs and assess whether impossible events elicit greater surprise than possible ones.

Human Evaluation

  • Human annotators rated 1,416 videos, with three ratings per video, using a four-point plausibility scale and majority vote.Presentation order was randomized, annotators evaluated at most 96 videos each, and they first saw ten physically plausible videos.Source: §3 · pp. 4–5

Evaluating multimodal large language models

  • MLLMs were evaluated with multiple prompts and input-frame counts, and the reported accuracy uses the best result across repeated runs.This protocol compensates for limits on video context and visual memory; humans, by contrast, were evaluated using majority vote.Source: §3 · p. 5

Evaluating prediction-based models

  • Prediction-based models are evaluated by comparing prediction-error surprise for possible and impossible videos, using pairwise or single-video protocols.Paired videos differ only by a physics-breaking event, helping control for differences in general prediction difficulty; longer-context adaptations require relevant events in context and memory for occluded objects.Source: §3 · p. 5

Protocol scope

  • The evaluation focuses on models with prior evidence of understanding or models that can be probed similarly to humans.The paper notes that out-of-distribution tests of action-recognition models had not yielded evidence of understanding in those models.Source: §3 · p. 5

4 Experiments

IntPhys 2 evaluations show a substantial gap between human performance and current models, which remain near chance on complex intuitive-physics tests. The experiments also probe how multimodal and predictive models respond to prompts, video frames, physical conditions, and camera movement.

4.1 Results: Multimodal Large Language Models

  • The MLLM evaluation included open-source and proprietary systems, with Gemini models taking MP4 input directly and other models receiving sampled image frames.The frame-count ablations used 10, 30, 60, or 120 frames.Source: §4 · p. 6, §4.1 · pp. 6–7

4 Experiments

  • 64% accuracy on the easy subset was Gemini 2.5 Flash’s reported result; V-JEPA 2 remained below 60%, and human-model gaps persisted across splits and data categories.Table 2 reports best model performance by difficulty level; Table 3 breaks accuracy down by property and camera type.Source: Table 2 · p. 6, §4 · p. 6, Table 3 · p. 6

4.1 Results: Multimodal Large Language Models

  • MLLM performance varied substantially with prompts, and most models’ performance declined when given additional frames.For the Debug-set robustness measure, a model had to answer correctly for all three videos of the same scene; prompt effects differed by model.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Results: Predictive models

  • The predictive-model evaluation covered pixel-prediction methods VideoMAEv2 and Cosmos-Predict-4B, alongside latent-prediction V-JEPA variants.Cosmos predicts future states, whereas VideoMAEv2 was trained to reconstruct masked video tokens; V-JEPA methods predict in latent space.Source: §4.2 · p. 7
  • Predictive models performed close to chance (50%) across IntPhys 2 subsets and the held-out set, with only permanence and continuity sometimes exceeding 65% accuracy.Those condition-level results come from small subsets and may be volatile; paired-video classification can be slightly above chance, while single-video classification remains closer to chance.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

5 Related Work

Related work spans benchmarks for intuitive and broader physical understanding, computational approaches to physical reasoning, and synthetic datasets built with Unreal Engine.

Intuitive physics understanding benchmarking

  • Intuitive-physics benchmarks test possible and impossible events, embodied-AI physics, simulated and real-world understanding, physics puzzles, and unintended actions in videos [27] [8] [20].IntPhys, GRASP [27], InfLevel, PHYRE [8], and OOPS [20] address these different evaluation settings.Source: §5 · p. 8

Broader physics understanding benchmarking

  • Broader physics benchmarks assess object and material interactions, motion tracking, causal video reasoning, physical commonsense, and fundamental principles [12] [56] [14] [48] [34] [50].Examples include Physion [12], CATER, CLEVRER [56], PIQA [14], Physion++ [48], and Physics IQ [34].Source: §5 · pp. 8–9

Methods tackling physical understanding

  • Approaches include world models, generative graph networks, JEPA, LLMs, hybrid simulation-neural methods, and methods using segmentation or de-rendering priors [26] [41] [31] [40].The described methods range from learning latent dynamics or predicted representations to combining neural learning with explicit physical knowledge.Source: §5 · p. 9

Datasets generated with Unreal Engine

  • Unreal Engine supports synthetic benchmarks and datasets for driving, embodied AI, computer vision, and vision-model robustness [18] [28] [38] [15].The cited examples include CARLA [18], AI2-THOR [28], and UnrealCV [38].Source: §5 · p. 9

6 Conclusion

IntPhys 2 expands intuitive-physics evaluation through varied synthetic scenes, tasks, and data splits. Current models still perform almost randomly, with only recent MLLMs achieving non-trivial performance.

6 Conclusion

  • Current models show significant intuitive-physics limitations on IntPhys 2, with almost-random performance and only recent MLLMs such as Gemini 2.5 Flash achieving non-trivial performance.The conclusion also identifies difficulties with longer sequences, higher framerates, and short-term memory.Source: §6 · p. 9

A.1 Designing the benchmark

  • IntPhys 2 replaces simplistic wall-based occlusions with more realistic tasks, including object persistence, viewpoint changes, ball dynamics, and solidity interactions.The benchmark also retains fixed-camera tasks with moving occluders, which are described as more plausible than the original wall-based occlusions.Source: §A.1 · p. 14, Figure 6 · p. 16
  • The dataset size was intentionally limited to avoid repetitive scene variations and reduce risks of training-data contamination and overfitting.The authors note that manually designed tasks permit only idiosyncratic asset changes and that they lack tools for automatically generating diverse physical tests.Source: §A.1 · p. 14

A.2 Building the benchmark

  • Tasks were built in Unreal Engine by varying asset and scene properties, then rendered as frames and compiled into videos.Rendering was parallelized across multiple V100 GPUs, and dataset generation took one hour.Source: §A.2 · p. 14

A.3 Splits

  • The Debug set supports basic checks, the Main set is intended for zero-shot diagnostic evaluation, and the small Held-out set helps detect possible contamination.The Main-set metadata is for error analysis rather than training; the Held-out set has no metadata.Source: §A.3 · pp. 14, 16

A.3.1 Debug Set

  • The Debug set contains 60 videos from five scenes, with repeated scene samples introducing subtle environmental variations that should not change predictions.Its tasks use bright, easily visible assets.Source: §A.3.1 · p. 16, Figure 7 · p. 17

A.3.2 Main Set

  • The Main set contains 1,012 videos across Easy, Medium, and Hard subsets, increasing scene and object complexity while retaining diagnostic utility.Hard scenes include realistic objects and may add occlusions such as moving leaves in front of the camera.Source: §A.3.2 · pp. 16–17

A.3.3 Held-Out Set

  • The Held-out set contains 344 videos sampled from the Main set’s Hard distribution, with metadata withheld to reduce contamination risk.Evaluation is available through a leaderboard for standardized comparison.Source: §A.3.3 · p. 17

A.4 DataSheet

The datasheet identifies dataset funding and describes three recommended splits for calibration, primary evaluation, and contamination checks.

A.4 DataSheet

  • The dataset was funded by Meta.Source: §A.4 · p. 18
  • The recommended splits are Debug for calibration, Main with Easy, Medium, and Hard subsets for evaluation, and Held-out for checking possible pretraining contamination.The Debug set is described as a simple fixed-camera set for checking prediction consistency and frame requirements.Source: §A.4 · p. 18
  • Table 5 presents the IntPhys2 datasheet following the framework introduced by Gebru et al. [22].Source: §A.4 · p. 17, Table 5 · p. 21

B Human evaluation

Human plausibility judgments were collected for videos using multiple annotators and majority-vote consensus.

B Human evaluation

C MLLMs evaluations

The MLLM evaluation varies prompt wording and video frame rates, while using model-specific access methods and a maximum-context prediction protocol.

C MLLMs evaluations

  • The evaluation compares an open-ended prompt with more specific yes/no and binary-digit response variants to test prompt robustness.Source: C MLLMs evaluations · p. 21, Table 6 · p. 23
  • Frame-rate variants were used to ablate the number of frames, accounting for common 1-fps subsampling by extending video duration.Source: C MLLMs evaluations · p. 21
  • GPT-4o and Gemini were evaluated through APIs, while Qwen-VL was run with the transformers library.Source: C MLLMs evaluations · p. 21
  • Table 4 summarizes video metadata for the Main set.Source: Table 4 · p. 22
  • The prediction protocol treats context size as a maximum so predictions cover more frames and can use any context size.With the earlier protocol, larger contexts delayed prediction start and could miss the impossible event.Source: Figure 8 · p. 24

D.1 Protocol

The protocol scores prediction surprise over overlapping video windows, aggregates it into video-level measures, and extends prediction coverage across frames. The section also compares protocol variants and reports model-specific hyperparameter choices, with no meaningful above-chance gains from tuning.

D.1 Protocol

  • Each video is split into overlapping windows, and surprise measures the distance between the predicted future and the target representation or frames.The context is used for prediction, while the remaining frames form the target; the encoder, target function, and predictor define the comparison.Source: §D.1 · p. 24
  • Average and maximal surprise aggregate window-level scores; average surprise is used by default and is suited to pairwise comparisons with similar baseline surprise.Maximal surprise is suited to single-video classification because it ignores the same baseline surprise.Source: §D.1 · pp. 24–25

D.2 Comparison with existing protocol

  • Models perform similarly under both protocols; best performance occurs with a 16-frame window at 6 fps, under three seconds, making most samples impossible.Table 7 compares the protocols.Source: Table 7 · p. 25, §D.2 · p. 25

D.1 Protocol

  • Progressively growing the context lets the protocol predict every frame except the first, avoiding missed events when a maximum-size window would start prediction late.The concern arises when the window size approaches the video length.Source: §D.1 · p. 25
  • The prior protocol reports the maximum across frame rates, context sizes, and first predicted frames, but testing more hyperparameter settings can raise scores through noise.These settings affect model memory.Source: §D.1 · p. 25

D.3 Hyperparameters

  • V-JEPA and V-JEPA 2 were tested at 16, 32, or 48 frames at 6 fps, while VideoMAEv2 used 16-frame windows at 2, 3, or 6 fps.The search axes reflect different model characteristics: RoPE for the V-JEPA models and sincos embeddings for VideoMAEv2.Source: §D.3 · pp. 25–26
  • Cosmos uses 9 context frames and 24 prediction frames without extra conditioning, while its 4B model takes about 40 times as long to run as V-JEPA or VideoMAE.Experiments varied Cosmos temperature between 0 and 1 and top_p across 0.7, 0.8, and 0.9.Source: §D.3 · p. 26
  • No model shows a meaningful performance increase above chance from better hyperparameter tuning.The comparison is constrained by model-specific parameter limits, so it is not a perfect comparison.Source: §D.3 · p. 26

D.4 Cost of experiments

Experiments ran on H100 GPUs, with runtime varying substantially by model, especially for Cosmos.

D.4 Cost of experiments

  • Evaluations ran on NVIDIA H100 GPUs; V-JEPA, V-JEPA 2, and VideoMAEv2 used a compute node with 8 GPUs each.On the main set, these experiments took 15 to 30 minutes depending on hyperparameters.Source: §D.4 · p. 26
  • Main-set experiments for V-JEPA, V-JEPA 2, and VideoMAEv2 took 15 to 30 minutes, depending on hyperparameters.Each model's experiments ran on an 8-GPU compute node.Source: §D.4 · p. 26
  • Cosmos experiments used 8 nodes of 8 GPUs and took around 2.5 hours on the main set.The paper attributes the larger setup to Cosmos's model size.Source: §D.4 · p. 26

E Qualitative results for prediction based models on the debug set

The debug set is used to inspect prediction-model behavior qualitatively. V-JEPA and Cosmos both struggle to represent or predict the ball's return after occlusion in the illustrated scene.

E Qualitative results for prediction based models on the debug set

E.1 V-JEPA surprise visualization

  • V-JEPA's surprise rises as the ball enters and accelerates, but not when it should reappear in impossible videos.The model had a 20-frame context, predicted the next two frames with a skip of 10, and the authors suggest that looking almost 200 frames back is too difficult.Source: §E.1 · p. 27
  • Seed-related surprise variations in shared video segments are minor relative to broader trends, suggesting some robustness to natural noise.The figure plots individual curves and their average across possible and impossible videos.Source: §E.1 · p. 27

E.2 Generations from Cosmos

  • Cosmos's context reconstruction loses the visible ball after a few frames at 2 fps; at 6 fps, the effect is weaker, but the ball starts vanishing near the occluder.The authors speculate that small objects may be forgotten by the video tokenizer, but identify this as unconfirmed.Source: Figure 10 · p. 28, Figure 11 · p. 28, §E.2 · p. 28
  • Cosmos does not predict the ball reappearing in either setting; reduced frame rate and out-of-distribution data may contribute to suboptimal performance.The 9-frame context requires a drastic frame-rate reduction, and the debug example also shows the model struggling.Source: §E.2 · p. 28, Figure 10 · p. 28, Figure 11 · p. 28
Loading 2506.09849v1…