Source-linked AI summary
Intuitive physics understanding emerges from self-supervised pretraining on natural videos
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun
TL;DR
The paper asks whether intuitive physics must be built into AI systems or can emerge from general self-supervised learning. It trains video models to predict masked content in a learned representation space and evaluates them with violation-of-expectation tests, finding strong performance across intuitive-physics benchmarks without predefined abstractions.
Problem
The paper examines whether intuitive physics requires hardwired core knowledge, given that existing AI systems often struggle with common-sense physical understanding.
Method
V-JEPA jointly learns video representations and predicts masked or future video content in that learned representation space, then uses prediction surprise to evaluate physical plausibility.
Results
V-JEPA detects intuitive-physics violations across benchmarks, achieving 98% average accuracy on IntPhys, 66% on GRASP, and 62% on InfLevel-lab.
Takeaways & Limitations
Intuitive physics can emerge from self-supervised prediction in a learned representation space without predefined abstractions or benchmark knowledge during training.
Takeaways & Limitations
V-JEPA struggles with contextualizing events such as gravity and solidity in InfLevel-lab and with precise interactions such as collisions.
Abstract
from arXiv · showhide
We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regions in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge -- a set of innate systems to help understand the world -- needs to be hardwired to develop an understanding of intuitive physics.
Measuring intutive physics understanding
The study evaluates intuitive physics by measuring how surprised a self-supervised video model is when predicting future representations, then using that surprise to distinguish plausible from implausible videos.
- Measuring intutive physics understanding: The evaluation presents plausible and physically impossible video pairs and tests whether the model assigns greater surprise to violations.This operationalizes the violation-of-expectation framework used to assess intuitive physics understanding.
- Measuring intutive physics understanding: V-JEPA combines an encoder that represents video with a predictor that reconstructs masked spatiotemporal content in representation space.The joint training encourages abstract representations that preserve predictable information while discarding lower-level features.
- Measuring intutive physics understanding: Prediction error between predicted and actual video representations provides a time-aligned measure of the model’s surprise.The model observes video pixels, predicts subsequent-frame representations, and records the distance between predictions and encoded observations.
- Measuring intutive physics understanding: Context length and video frame rate are varied to control the model’s available memory and temporal resolution.These manipulations test how evaluation conditions affect the model’s prediction-based surprise measure.
Representation prediction learns to detect violations of intuitive physics
V-JEPA is compared with pixel-prediction models, multimodal language models, and untrained networks across diverse intuitive-physics benchmarks. Representation-space prediction is the only compared approach reported as significantly better than untrained models.
- Representation prediction learns to detect violations of intuitive physics: The benchmark suite probes object permanence, continuity, shape and color constancy, gravity, support, and other physical properties across varied video conditions.The datasets include differences in visual quality, scene diversity, and the intuitive-physics properties tested.
- Representation prediction learns to detect violations of intuitive physics: V-JEPA is compared with pixel-space video prediction, multimodal large language models, and untrained neural networks.The comparisons isolate the role of the prediction objective and the representation space used for prediction.
- Representation prediction learns to detect violations of intuitive physics: V-JEPA is the only evaluated method reported to achieve significantly higher performance than untrained networks.The evaluation uses pairwise classification of which video violates intuitive physics.
- Representation prediction learns to detect violations of intuitive physics: 98% accuracy was achieved on IntPhys, 66% on GRASP, and 62% on InfLevel-lab.These are reported average accuracies with 95% confidence intervals of [95%,99%], [64%,68%], and [60%,63%], respectively.
- Representation prediction learns to detect violations of intuitive physics: VideoMAEv2, Qwen2-VL-7B, and Gemini 1.5 pro performed only marginally above randomly initialized models.The authors emphasize that this comparison does not establish that pixel-prediction models or language models cannot acquire intuitive physics.
Per property analysis of V-JEPA
V-JEPA’s performance varies across intuitive-physics properties: it is strongest for scene-content properties such as object permanence and weaker for contextual events and precise interactions. Its performance remains above chance across masking, data, and model-size variations.
- Per property analysis of V-JEPA: On IntPhys, V-JEPA significantly outperformed untrained networks on object permanence, continuity, and shape constancy.Reported means were 85.7 versus 51.4 for object permanence, 86.3 versus 51.2 for continuity, and 83.7 versus 51.7 for shape constancy.
- Per property analysis of V-JEPA: On GRASP, V-JEPA significantly outperformed untrained networks on object permanence, continuity, support, gravity, and inertia.The largest reported GRASP contrast was support at 98.1 versus 58.4.
- Per property analysis of V-JEPA: On InfLevel, V-JEPA significantly exceeded untrained networks for object permanence but not gravity or solidity.Object permanence reached 72.1 versus 52.5, while gravity and solidity showed no significant gain.
- Per property analysis of V-JEPA: V-JEPA performs better on scene-content properties than on contextualizing events or precise object interactions such as collisions.The authors hypothesize that framerate constraints contribute mainly to these limitations.
- Per property analysis of V-JEPA: Maximum surprise performs better than average surprise for single-video classification, while occluded physics-breaking events reduce performance for both humans and V-JEPA.Performance between humans and V-JEPA is well correlated in occluded settings.
Keys to intuitive physics understanding
V-JEPA’s intuitive physics understanding is robust across pretraining choices, with learned representation-space prediction more important than a precisely tailored masking objective. Data source and model size still influence performance, but above-chance understanding emerges with modest scale and limited unique video.
- Scope of the ablations: V-JEPA design ablations vary training data, model size, and prediction task to identify conditions supporting intuitive physics understanding.These components influence performance, but the study tests whether understanding depends on a specific design choice.
- Pretraining task: Random masking reduces IntPhys accuracy by only around 5 points on average, while Causal Block Masking performs worse than ordinary Block Masking.The relatively small effect of masking strategy contrasts with a 20-point average drop on video classification tasks under Random Masking.
- Pretraining task: Predicting in an abstract representation space, rather than using a tailored objective, is identified as the key aspect for intuitive physics understanding.The effectiveness of Random Masking supports this interpretation within the reported ablation results.
- Pretraining data: Tutorial videos produce the best performance among individual component datasets, whereas motion-focused SSv2 yields almost chance-level performance and K710 performs above chance.The datasets cover different slices of natural video, including activities, fine-grained motion, and tutorials.
- Pretraining data: With 128 hours of unique HowTo100M video, V-JEPA maintains pairwise accuracy above 70% across all considered intuitive physics properties.The experiments keep compute fixed by revisiting videos, so the result concerns unique-data quantity rather than total training compute.
- Model size: A 115-million-parameter model achieves accuracy over 85%, although larger models tend to perform better.This establishes above-chance intuitive physics performance at relatively small model scale.
Discussion
The discussion attributes intuitive physics understanding to representation-space prediction on natural videos rather than hardwired knowledge or a precise V-JEPA configuration. The authors also note a limitation in evaluating proprietary video-generation models and describe the training setup underlying their experiments.
- Discussion: V-JEPA acquires intuitive physics understanding from a simple prediction task in a learned representation space, without task-specific adaptation.The reported understanding applies to both synthetic and real videos.
- Discussion: The emergence of intuitive physics is attributed to representation-space prediction rather than a precise choice of model size, data source, or pretraining task.Those design choices still influence the level of understanding.
- Discussion: Multimodal LLMs and pixel-prediction methods perform around chance level on the reported intuitive physics evaluations.The comparison supports a distinction between representation-space prediction and the evaluated alternatives.
- Limitations: Most state-of-the-art video-generation models are proprietary, complicating rigorous assessment of their physics understanding because they are not open.This is the discussion’s stated boundary on evaluating those models.
- Pretraining setup: V-JEPA pretraining uses multiple natural-video sources, including Kinetics710, SomethingSomething-v2, and HowTo100M.HowTo100M contains around 1.2 million videos averaging 6 minutes 30 seconds, totaling around 15 years of unique video data.
- Pretraining setup: Most experiments use only HowTo100M because it exhibits the highest performance and demonstrates that V-JEPA can use uncurated data.The training protocol uses ViT encoders and predictor architectures with masked spatiotemporal video patches.
A.4 Evaluation Data
The evaluation spans three qualitatively different datasets, covering synthetic and natural videos, varied scenes, and multiple intuitive-physics properties. The section also defines the compared baselines and the properties being tested.
- Evaluation datasets: IntPhys, GRASP, and InfLevel-lab provide complementary evaluation data from simulated and natural videos.Together, they vary in visual quality, scene diversity, and the physical properties probed.
- Evaluation datasets: IntPhys uses pixel-aligned paired simulator videos without compression artifacts, while GRASP covers 10 properties compared with IntPhys’s 3.GRASP contains more videos but was designed for single-video evaluation, creating possible spurious-feature issues for untrained networks.
- Evaluation datasets: InfLevel-lab provides paired natural videos whose within-pair visual differences are limited to scene lighting.Solidity and gravity require contextualizing events, increasing memory demands beyond the tested models’ capabilities.
- Evaluation setting: All evaluation datasets are held out from training and used with frozen networks, making them out-of-distribution evaluations.The study uses these datasets only for evaluation, not for model training.
- Physical properties: The evaluated properties include object permanence, continuity, shape and color constancy, gravity, support, solidity, inertia, and collision.These properties describe expectations such as persistence through occlusion, continuous motion, stable support, and non-overlap.
- Baselines: VideoMAEv2 predicts normalized future pixels, whereas Qwen2-VL and Gemini 1.5 Pro are multimodal language-model baselines that process video with text-based outputs.VideoMAEv2 is designed as a comparison with V-JEPA because the implementation is similar while the prediction target differs.
A.7 Evaluation Protocol
The protocol evaluates prediction models through surprise: predicted and actual video representations are compared over time, then possible and impossible videos are contrasted. Different surprise aggregations and statistical tests are used for paired and individual-video tasks.
- Prediction-based evaluation: Prediction models encode observed frames and predict future representations, using prediction error as a temporally aligned surprise measure.For V-JEPA, the target is an abstract future representation; pixel baselines instead predict normalized future video patches.
- Implementation: Surprise is computed with stride s = 2, predicting from frames 1, 3, 5, and so on.This stride reduces computation while retaining regularly spaced prediction windows.
- Surprise measures: Average surprise is used for paired-video comparisons, while maximum surprise better separates possible and impossible individual videos.Relative surprise measures the difference between impossible and possible videos, isolating the physics-breaking event.
- Paired-video evaluation: On IntPhys, average surprise exceeded zero for object permanence, shape constancy, and continuity in one-sample tests.The reported relative-surprise means were 7.8e-03, 7.1e-03, and 8.5e-03, respectively.
- Multimodal-language evaluation: Qwen2-VL outputs similar normalized probabilities for possible and impossible videos, with probabilities often near 0.5.The model is prompted to identify which video violates physics, and the final video token is converted into a predicted choice.
- Individual-video evaluation: Impossible videos produced higher surprise than possible videos for all three IntPhys properties using both average and maximum surprise.Maximum surprise yielded stronger separations, including t(76.7) = 8.4 for object permanence and t(80.6) = 8.1 for shape constancy.
A.8 Evaluation hyperparameters
The evaluation sweeps frame skip, prediction-window size, and context length separately for each dataset. These hyperparameters define the temporal information available to prediction-based models.
- IntPhys and GRASP: IntPhys and GRASP use frame skips [2,5,10], window sizes [16,32], and context lengths [2,4,6,8,10]×(C + M)/16.These values are selected from the same evaluation grid for both datasets.
- InfLevel-lab: InfLevel-lab uses frame skips [5,10,20] for V-JEPA and VideoMAEv2, and [5,10,20,30] for Qwen and Gemini.Its window sizes and context-length grid match those used for IntPhys and GRASP.
- Selection rule: For every property, the context size is chosen to maximize performance, so different properties can use different optimal contexts.On IntPhys, minimal surprise across context lengths removes the need for a context-length sweep per starting frame.
B Choice of prediction hyperparameters and influence on 0-shot performance
Context length and pretraining-data composition affect zero-shot performance, complicating evaluation because datasets often lack suitable validation splits. Fixed contexts remain workable for V-JEPA, while video subsampling generally outperforms frame subsampling.
- Context selection: Prediction hyperparameters determine how far models look into the past and how far ahead they predict, but zero-shot evaluation provides no task-specific calibration.The study therefore examines both fixed and property-specific context choices.
- Evaluation constraints: The lack of ready-made validation and test splits in datasets such as GRASP and InfLevel-lab complicates classical hyperparameter selection.Scene similarities can also make validation and test sets too similar, causing information leakage.
- Context selection: A fixed context size causes performance drops across properties and datasets, although V-JEPA remains non-trivial.The context-size optimum depends on the property and dataset, with GRASP showing the greatest variation and IntPhys and InfLevel-lab generally less sensitivity.
- Context selection: For most properties, shorter contexts and longer prediction horizons improve performance, while a two-frame context can support 14-frame prediction.The authors hypothesize that predicting over a longer horizon may be easier than retaining relevant information from a long context.
- Limitations: Variation with context size is a limitation of existing methods, motivating future models that handle longer video sequences.This limitation affects the stability of prediction-based evaluation across properties and datasets.
- Pretraining-data composition: Video subsampling generally produces higher performance than frame subsampling, although both retain non-trivial performance.The two strategies respectively reduce scene diversity and within-scene motion diversity, so they do not represent equivalent data reductions.
D Results on the IntPhys challenge
On IntPhys, V-JEPA remains robust on the private test set, with performance comparable to or exceeding human and earlier structured-model baselines. Model scale matters more for single-video classification than pairwise classification.
- The private IntPhys test set contains 3,600 videos per property, compared with 120 in development, and preserves V-JEPA’s high accuracy.
- V-JEPA achieves performance similar to or higher than human baselines on the private IntPhys test set.
- V-JEPA surpasses previously published methods that use predefined abstractions such as segmentation masks.
- V-JEPA-H matches human performance in single-video classification, whereas V-JEPA-L remains far from it.
- Scale benefits the harder single-video task but does not significantly improve pairwise classification.
E Importance of contextualization events on InfLevel
InfLevel properties such as gravity and solidity depend on contextualization events, but relabeling the data changes task difficulty and can enable heuristic solutions. Consequently, continuity results provide the more controlled evaluation.
- In the relabeled gravity condition, the impossible video is harder because it requires predicting an object’s trajectory and bounce, whereas the possible video stops after dropping.
- Performance also increases for untrained networks after relabeling, suggesting that heuristics can exploit the changed matching between videos.
- The relabeled results should be interpreted cautiously, with continuity remaining the most important result because its setup is more controlled.
- Gravity and solidity require remembering container properties shown before the experiment, so removing the contextualization prefix increases performance for V-JEPA and VideoMAEv2.
- Relabeling InfLevel removes the need for contextualization but breaks the assumption that possible and impossible videos have equal difficulty.
F Per-property performance of methods
Across individual intuitive-physics properties, larger V-JEPA models generally outperform untrained networks, while VideoMAEv2 and multimodal language models remain near untrained performance on most properties.
- V-JEPA-L achieves accuracy higher than untrained networks on most properties across five seeds.
- V-JEPA-H achieves accuracy higher than untrained networks on most properties across twenty untrained-network comparisons.
- VideoMAEv2 performs on par with or slightly above untrained networks across properties, except for solidity and collision.
- Qwen2-VL-72B performs on par with or slightly above untrained networks, except for color constancy and support.
- Gemini 1.5 Pro performs on par with or slightly above untrained networks across properties.