PaperlayerView exact revision on arXiv

Source-linked AI summary

Intuitive physics understanding emerges from self-supervised pretraining on natural videos

Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun

arXiv:2502.11831v1cs.CVcs.AI

TL;DR

Whether intuitive physics can emerge in deep networks without hardwired core knowledge remains an open question. This paper tests self-supervised video predictors with violation-of-expectation probes and finds that learned-representation prediction captures intuitive physics, while pixel-space prediction and text-reasoning multimodal models remain near chance.

  • Problem

    The study asks whether deep-learning systems can understand intuitive physics without strong prior information, against the view that core knowledge is necessary.

    Source: Per property analysis of V-JEPA · p. 6

  • Method

    The authors use violation-of-expectation probes and measure prediction error between predicted and actual video representations as surprise.

    Source: Measuring intutive physics understanding · p. 3

  • Results

    Learned-representation prediction achieved average accuracies of 98% on IntPhys, 66% on GRASP, and 62% on InfLevel-lab; pixel prediction and multimodal models were only marginally above randomly initialized models.

    Source: Representation prediction learns to… · p. 4

  • Takeaways & Limitations

    The results suggest that core knowledge is not necessary for deep-learning systems to understand intuitive physics concepts.

    Source: Per property analysis of V-JEPA · p. 6

    V-JEPA excels on scene-content properties but struggles with event-context categories such as gravity and solidity and with precise interactions such as collisions.

    Source: Per property analysis of V-JEPA · p. 6

Abstract

from arXiv · show

We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regions in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge -- a set of innate systems to help understand the world -- needs to be hardwired to develop an understanding of intuitive physics.

Measuring intutive physics understanding

The section describes a violation-of-expectation test for intuitive physics and a self-supervised video-prediction approach that measures surprise from prediction error. The approach probes learned representations without additional adaptation while varying context and frame rate to control memory and motion detail.

Violation ofExpectation

  • Models are tested with matched scene pairs that differ in one physical event; greater surprise at the impossible scene indicates understanding of the violated concept (Riochet et al., 2022; Smith et al., 2019; Riochet et al., 2020).In infant studies, gaze-time interpretations are partly mitigated by habituation trials before testing (Baillargeon and DeVos, 1991; Spelke, 1985; Margoni et al., 2024).Source: Measuring intutive physics understanding · p. 3

Video Prediction for intuitive physics understanding

  • V-JEPA tests whether representations learned through high-level prediction implicitly capture the structure and dynamics of objects, without explicitly representing contours or poses (Bardes et al., 2024).Its encoder and predictor are jointly trained to predict masked video representations, learning abstract predictable information while discarding lower-level features.Source: Measuring intutive physics understanding · p. 3
  • After self-supervised training, the model predicts future-frame representations without adaptation, and prediction error against actual representations provides a time-aligned measure of surprise.Changing past-frame context controls memory, while changing frame rate controls motion fineness.Source: Measuring intutive physics understanding · p. 3

Representation prediction learns to detect violations of intuitive physics

V-JEPA is the only evaluated method that significantly outperforms untrained networks across all three benchmarks, supporting the finding that prediction in a learned representation space can develop intuitive physics understanding. Pixel-space prediction and multimodal LLMs perform only marginally above randomly initialized models.

Representation prediction learns to detect violations of intuitive physics

  • The evaluation spans IntPhys, GRASP, and InfLevel-lab, probing object permanence, continuity, constancy, gravity, support, solidity, inertia, and collision across varied visual qualities and scenes.The passages cite Riochet et al. (2022), Jassim et al. (2024), and Weihs et al. (2022) for the datasets, and cite prior work for the probed properties.Source: Representation prediction learns to… · p. 4
  • VideoMAEv2 predicts in pixel space, whereas Qwen2-VL-7B and Gemini 1.5 pro generate text and are evaluated by choosing the impossible video in a pair.VideoMAEv2 is evaluated by predicting the future and measuring surprise via prediction error; the MLLMs cannot use that quantitative surprise protocol.Source: Representation prediction learns to… · p. 4
  • Models are compared with untrained networks, and each model’s context size is selected to maximize performance for each property and evaluation setup.The procedure is intended to show the best performance achievable by each model.Source: Representation prediction learns to… · p. 4
  • V-JEPA alone significantly exceeds untrained networks across all datasets, with average accuracies of 98% (95% CI [95%,99%]) on IntPhys, 66% (95% CI [64%,68%]) on GRASP, and 62% (95% CI [60%,63%]) on InfLevel-lab.The results are achieved without predefined abstractions or benchmark knowledge during pretraining or method development.Source: Representation prediction learns to… · p. 4
  • VideoMAEv2, Qwen2-VL-7B, and Gemini 1.5 pro perform only marginally above randomly initialized models on intuitive physics benchmarks.The comparison does not establish that pixel-prediction models or LLMs cannot acquire intuitive physics understanding; the task remains difficult even for frontier models.Source: Representation prediction learns to… · p. 4

Per property analysis of V-JEPA

On per-property tests, V-JEPA trained on HowTo100M significantly outperformed untrained models on several intuitive-physics properties, though gains varied by dataset and property. The flagship model matched or exceeded human performance across IntPhys properties, while occlusion remained challenging.

Per property analysis of V-JEPA

  • On IntPhys, V-JEPA exceeded untrained models for Object Permanence (M=85.7 vs. M=51.4), Continuity (M=86.3 vs. M=51.2), and Shape Constancy (M=83.7 vs. M=51.7).The per-property analysis used a ViT-L V-JEPA trained on HowTo100M and compared it with randomly initialized, untrained models.Source: Per property analysis of V-JEPA · p. 5
  • On GRASP, V-JEPA significantly exceeded untrained models in Object Permanence, Continuity, Support, Gravity, and Inertia, but not Color Constancy, Solidity, or Collision.On InfLevel, it significantly exceeded untrained models only for Object Permanence (M=72.1 vs. M=52.5), with no significant gain for Gravity or Solidity.Source: Per property analysis of V-JEPA · pp. 5–6
  • V-JEPA excelled on scene-content properties such as object permanence but struggled with properties requiring contextual events or precise interactions, including collisions.The authors hypothesize that these limitations arise mainly from the model’s framerate constraints; they argue that intuitive-physics understanding can emerge without strong prior information or innate core knowledge.Source: Per property analysis of V-JEPA · p. 6
  • On the IntPhys private test set, flagship V-JEPA matched or exceeded human performance across all intuitive-physics properties.Both humans and V-JEPA performed worse when physics-breaking events occurred behind an occluder, and their performance was well-correlated in occluded settings; using maximum rather than average surprise improved single-video performance.Source: Per property analysis of V-JEPA · p. 6

Keys to intuitive physics understanding

Ablations show that intuitive-physics performance varies substantially by pretraining data and tends to improve with encoder size, but is relatively insensitive to masking strategy and dataset size. Even with only 128 hours of unique video, the model distinguishes violations of intuitive-physics properties above 70% pairwise accuracy.

Importance of the pretraining task

  • Random Masking reduced IntPhys performance by only around 5 points on average, while Causal Block Masking performed worse than Block Masking.Although Causal Block Masking more closely matches test-time prediction, the results suggest that a tailored objective is not required and predicting in an abstract representation space is key.Source: Keys to intuitive physics understanding · p. 7

Importance of pre-training data

  • HowTo tutorial videos yielded the best performance among individual datasets, K710 yielded above-chance understanding, and SSv2 led to almost chance-level performance.HowTo was also larger than the other individual datasets, motivating the separate subsampling experiments.Source: Keys to intuitive physics understanding · p. 7
  • Over 70% pairwise accuracy on every considered property was maintained with only 128 hours of unique HowTo100M video.Training compute was held fixed at the equivalent of 30 years of video by revisiting videos, and dataset size did not meaningfully affect performance.Source: Keys to intuitive physics understanding · p. 7

Importance of the encoder size

  • A 115-million-parameter encoder achieved over 85% accuracy, although larger models tended to perform better.The experiments examined smaller and larger encoders to identify the minimum size supporting non-trivial intuitive-physics understanding.Source: Keys to intuitive physics understanding · p. 7

Discussion … A.3 V-JEPA pretraining hyperparameters

V-JEPA acquires intuitive physics understanding by predicting missing video content in a learned representation space, without task-specific adaptation. Its performance varies by property, while the appendix describes the pretraining procedure, data, and hyperparameters.

Discussion

  • V-JEPA demonstrates intuitive physics understanding on synthetic and real videos without task-specific adaptation, supporting representation-space prediction as a general route that does not require hardwired core knowledge.Model size, pretraining data, and task influence the result, but the authors attribute its emergence to the broader representation-space prediction framework.Source: Discussion · p. 8
  • V-JEPA performs well on many object-intrinsic violations but is near chance on interaction violations such as solidity and collision, which may be less frequent in training data.Its short video clips and inability to condition on context limit complex interactions; the authors suggest hierarchical representations or action channels as possible directions.Source: Discussion · p. 8
  • Future work could test whether intuitive physics emerges from videos that better resemble infants’ visual experience.The authors conclude that latent prediction is a promising path toward neural networks that understand the physical world.Source: Discussion · p. 8

A.1 Unsupervised pretraining of V-JEPA

  • V-JEPA uses a context encoder for corrupted-video representations, an exponential-moving-average target encoder for full-video representations, and a predictor to infer masked content.Training masks random spatiotemporal blocks and predicts the corresponding target representations; using initial frames to predict later ones makes the objective future prediction.Source: §A.1 · p. 13

A.2 Pretraining Data

  • Most experiments use HowTo100M, the least curated data source in the mix, which achieves the highest performance and shows V-JEPA can leverage uncurated video.The original VideoMix2M mix combines Kinetics710, SomethingSomething-v2, and HowTo100M; the latter contains about 1.2 million videos averaging 6 minutes 30 seconds.Source: §A.2 · p. 13

A.3 V-JEPA pretraining hyperparameters

  • The context and target encoders use ViT architectures processing clips of up to 3 seconds and 16 frames at 224 × 224 resolution, while the 12-block predictor has a 384-dimensional embedding.The study uses RoPE for positional information, splitting feature dimensions across height, width, and time.Source: §A.3 · p. 13
  • Training uses AdamW, batch size 3072, and 90,000 iterations, with a learning rate rising from 2 × 10−4 to 6.25 × 10−4 before cosine decay to 1 × 10−6.The cosine schedule is stretched to 112,500 iterations to avoid end-of-training performance decay and partial encoder collapse.Source: §A.3 · pp. 13, 15
  • The default masking strategy combines eight blocks at spatial scale 0.15 and two at scale 0.7; alternatives add full masking of the last four frames or randomly mask 90% of patches.Block aspect ratios are sampled uniformly from 0.75 to 1.5.Source: §A.3 · p. 15

A.4 Evaluation Data · A.5 Properties · A.6 Baselines

Evaluation uses diverse unseen video datasets, defines the intuitive-physics properties being tested, and specifies pixel-prediction and multimodal-LLM baselines.

A.4 Evaluation Data

  • IntPhys uses pixel-aligned simulated video pairs with randomized objects, occluders, and object appearances; evaluation uses its labeled dev subset because the challenge test set is private (Riochet et al., 2022).The dataset avoids compression artifacts by storing each frame individually.Source: §A.4 · p. 15
  • GRASP covers 10 properties with more videos than IntPhys, but its single-video design can let untrained networks exploit spurious features (Jassim et al., 2024).Videos attributed to multiple properties count toward each relevant property.Source: §A.4 · p. 15
  • InfLevel-lab pairs natural videos that differ only in lighting; solidity and gravity require a contextualizing event, exceeding the tested models’ memory and adaptability.The study calls its “continuity” property “object permanence” for consistency with other datasets; its setup is closer to that property in IntPhys and GRASP.Source: §A.4 · p. 15

A.5 Properties

  • The tested properties include object permanence—objects persist when occluded—and continuity, in which objects follow continuous paths (Baillargeon and DeVos, 1991; Spelke et al., 1992).Continuity is closely linked to object permanence but uses subtler experimental setups.Source: §A.5 · p. 16
  • Other properties test constancy of shape and color, gravity, support, solidity, inertia, and collision (Wilcox, 1999; Wilcox and Chapa, 2004; Kim and Spelke, 1992; Baillargeon and Hanko-Summers, 1990; Baillargeon et al., 1992; Spelke et al., 1992; Baillargeon, 1995).Objects should not change shape or color, fall without support, become unstable when unsupported, avoid overlap, maintain motion, and move when hit.Source: §A.5 · p. 16

A.6 Baselines

  • VideoMAEv2 is the pixel-prediction baseline, trained similarly to V-JEPA but predicting normalized 16x16x2-pixel patches rather than missing video content in latent space (Wang et al., 2023).Its implementation similarities make it a comparison to V-JEPA despite the different prediction space.Source: §A.6 · p. 16
  • The multimodal-LLM baselines are Qwen2-VL and Gemini 1.5 Pro; Qwen2-VL offers control over video processing and reproducible results.The study selects Qwen2-VL for video-input capability and Gemini 1.5 Pro for video understanding.Source: §A.6 · p. 16

A.7 Evaluation Protocol

The evaluation compares prediction surprise for physically possible and impossible videos, using different scoring strategies for paired and individual-video judgments. V-JEPA’s surprise distinguished violations across IntPhys properties, while Qwen2-VL’s choice probabilities were often near chance.

A.7 Evaluation Protocol

  • Prediction surprise compares future targets with predictions, using abstract encoded future frames for V-JEPA and normalized future video frames for VideoMAEv2.The evaluation uses stride s=2, predicting from frames 1, 3, 5, and so on; context length is selected per property, with IntPhys using the minimum surprise across context lengths.Source: §A.7 · pp. 16–17
  • On IntPhys, V-JEPA’s average-surprise relative scores were positive for object permanence (M=7.8e-03), shape constancy (M=7.1e-03), and continuity (M=8.5e-03).Both average and maximum surprise produced positive relative scores across all three properties; average surprise performed better for pairwise classification.Source: Figure S1 · p. 17, §A.7 · p. 17
  • For individual-video judgments, impossible videos had higher surprise than possible videos for all three properties using both average and maximum surprise.The protocol treats this as a harder task than paired comparison; maximum surprise was preferred because it focuses on the most surprising event rather than other scene properties.Source: Figure S1 · p. 17, §A.7 · pp. 17–18
  • Average surprise better supported pairwise IntPhys comparisons, whereas maximum surprise better separated possible from impossible videos individually.Figure S1 reports the corresponding task-specific pattern across surprise measures.Source: Figure S1 · p. 17, §A.7 · pp. 17–18
  • For Qwen2-VL-72B, when comparing a pair of videos, normalized probabilities for the video choices were often around 0.5, so predictions were almost like a coin toss.The probabilities were normalized for the answer choices “1” and “2” at the end of the answer.Source: Figure S2 · p. 18, §A.7 · p. 18
  • For text-only multimodal models, the protocol prompts the model to identify the impossible video in a pair, shuffles video order, and counts refusals as errors.Gemini 1.5 Pro refused to answer around 10–15% of the time; Qwen2-VL did not.Source: §A.7 · p. 18

A.8 Evaluation hyperparameters

Evaluation used dataset-specific frame-skip, window-size, and context-length grids, with the best context size selected for each property. On IntPhys, minimum surprise across windows also filtered event-related surprise spikes.

A.8 Evaluation hyperparameters

  • IntPhys and GRASP used frame skips [2,5,10], window sizes (C +M) [16,32], and context lengths [2,4,6,8,10]×(C +M)/16.Source: §A.8 · p. 18
  • InfLevel-lab used frame skips [5,10,20] for V-JEPA and VideoMAEv2, versus [5,10,20,30] for Qwen and Gemini, with window sizes (C + M) [16,32] and context lengths [2,4,6,8,10]×(C + M)/16.Source: §A.8 · p. 19
  • Each property used its best-performing context size, and IntPhys used minimum surprise over windows for each start frame to filter surprise spikes.An object entering the scene can be hard to predict and cause a spike; the filtering removes such spikes while also eliminating one hyperparameter to optimize.Source: §A.8 · p. 19

B Choice of prediction hyperparameters and influence on 0-shot performance · C Influence of semantic and motion diversity in videos

Zero-shot prediction performance depends on context length, which varies in importance across properties and datasets; training-data diversity also matters, with video subsampling outperforming frame subsampling. Because models lack task-specific calibration and many datasets lack suitable validation splits, context choices are evaluated as part of these analyses.

B Choice of prediction hyperparameters and influence on 0-shot performance

  • Context size is optimized per property to show maximal attainable performance, since models lack task-specific calibration and validation splits can be unavailable or leak scene similarities.The IntPhys development-set results correlate with its private test-set results.Source: B Choice of prediction hyperparameters… · p. 19
  • A fixed context size causes performance drops across properties and datasets, although V-JEPA remains non-trivial; a 2-frame context predicting 14 frames has only a minor impact for V-JEPA.The authors suggest longer-horizon prediction may be easier than retaining information from earlier context frames, but note that further experiments are needed.Source: B Choice of prediction hyperparameters… · p. 19, Figure S3 · p. 20

C Influence of semantic and motion diversity in videos

  • Subsampling videos produces higher performance than subsampling frames, though both retain non-trivial performance; V-JEPA-L performs well with 128h of unique videos or 2% of frames (2579h).Video subsampling reduces scene diversity, whereas frame subsampling reduces motion diversity within scenes.Source: Figure S5 · p. 21, C Influence of semantic and motion… · p. 21

B Choice of prediction hyperparameters and influence on 0-shot performance

C Influence of semantic and motion diversity in videos

  • The subsampling protocols use middle video frames to avoid less relevant intros and outros, uniformly sample videos, and nest smaller subsets within larger ones.The two strategies distinguish reduced scene diversity from reduced motion diversity within scenes.Source: C Influence of semantic and motion… · p. 21

D Results on the IntPhys challenge

On IntPhys’s larger private test set, V-JEPA maintains robust intuitive-physics performance, with outcomes depending on task setting and model scale.

D Results on the IntPhys challenge

  • For pairwise comparisons, maximum or average surprise performs well; for single-video classification, maximum surprise performs best, while average surprise may be unreliable across experimental setups.The test-set pairwise error rates surpass reported human results, and maximum-surprise single-video error rates surpass reported human baselines.Source: Table S4 · p. 22, Table S5 · p. 22
  • On IntPhys’s private test set, V-JEPA remains highly accurate, performing similarly to or better than humans and surpassing published methods using predefined segmentation-mask abstractions.The private test set contains 3,600 videos per property, compared with 120 in the development set.Source: D Results on the IntPhys challenge · pp. 21–22, Table S4 · p. 22, Table S5 · p. 22
  • In single-video classification, V-JEPA-H matches human performance while V-JEPA-L falls far short; scaling brings no significant improvement in the pairwise setting.This indicates scale can benefit the harder single-video task.Source: D Results on the IntPhys challenge · p. 22

E Importance of contextualization events on InfLevel

Relabeling InfLevel to remove the need to remember contextualization events increases performance, but the result is qualified by possible difficulty imbalances and heuristic solutions.

E Importance of contextualization events on InfLevel

  • Relabeling assumes objects are unmodified, making InfLevel test world understanding without requiring memory of the occluder’s modifications.Without knowledge of a contextualization event, those modifications are invisible during the experiment, making the original task unsolvable.Source: E Importance of contextualization… · p. 22
  • For gravity, relabeling can make impossible videos harder than possible ones because only the impossible video requires predicting an object’s bounce on a table.This mismatch can skew performance upward.Source: E Importance of contextualization… · p. 22
  • Performance significantly increases for V-JEPA, VideoMAE, and untrained networks after relabeling InfLevel to remove contextualization events.The increase for untrained networks suggests the task may be solvable with heuristics; continuity results remain more important because their setup is more controlled.Source: E Importance of contextualization… · p. 22, Figure S6 · p. 23

F Per-property performance of methods

Per-property results show V-JEPA-L and V-JEPA-H above untrained networks on most properties, while VideoMAEv2, Qwen2-VL-72B, and Gemini 1.5 pro are generally on par or slightly higher, with model-specific exceptions.

F Per-property performance of methods

  • V-JEPA-L achieved higher accuracy than untrained networks on most properties across five seeds.Black dots represent the five seeds.Source: Figure S7 · p. 23
  • V-JEPA-H achieved higher accuracy than untrained networks on most properties.Confidence intervals were obtained via bootstrapping, and gray dots represent 20 untrained networks.Source: Figure S8 · p. 23
  • VideoMAEv2 performed on par with or slightly higher than untrained networks across properties, apart from solidity and collision.Confidence intervals were obtained via bootstrapping, and gray dots represent 20 untrained networks.Source: Figure S9 · p. 24
  • Qwen2-VL-72B performed on par with or slightly higher than untrained networks across properties, except for color constancy and support.Confidence intervals were obtained via bootstrapping, and gray dots represent 20 untrained networks.Source: Figure S10 · p. 24
  • Gemini 1.5 pro performed on par with or slightly higher than untrained networks across properties.Gray dots represent 20 untrained networks, and confidence intervals were obtained via bootstrapping.Source: Figure S11 · p. 24

Related papers

Loading 2502.11831v1…