Source-linked AI summary
Physion: Evaluating Physical Prediction from Vision in Humans and Machines
Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, Judith E. Fan
TL;DR
Physion asks whether vision algorithms can predict the evolution of diverse physical scenarios as reliably and human-likely as people. It introduces a realistic benchmark with human comparisons and evaluates models across architectures, objectives, input-output structures, and training regimes. Vision models remain below human performance, while graph neural networks given physical state information perform substantially better and more human-like.
Problem
It remains unclear whether vision algorithms understand real-world physical dynamics well enough to support reliable prediction and safe interaction.
Method
Physion combines realistic simulations of diverse physical phenomena, a unified human-comparison protocol, and systematic evaluation across model architectures, learning objectives, input-output structures, and training regimes.
Results
Vision algorithms remain far below human performance, whereas particle-based graph models with direct physical-state access perform substantially better and make more human-like predictions.
Takeaways & Limitations
Extracting physical representations from visual scenes is the key bottleneck to human-level and human-like physical understanding in vision algorithms.
Takeaways & Limitations
Finer comparisons are still needed to determine whether pretraining captured more physically relevant information than object-centric learning on TDW.
Abstract
from arXiv · showhide
While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.
1 Introduction
Physion addresses the lack of common standards for evaluating physical understanding by combining varied visual scenarios with event-based predictions and direct human comparisons. It benchmarks whether different model designs can achieve robust, human-like physical understanding.
- Benchmark design: The benchmark requires models to predict key events from varied, unstructured visual inputs rather than reconstructing every detail of scene dynamics.This combines elements of fine-grained physical prediction and qualitative event-based evaluation.
- Benchmark design: Physion combines visually realistic examples spanning collisions, support, containment, attachment, projectile motion, rolling, sliding, and soft materials.The dataset covers eight scenario types involving diverse physical phenomena.
- Evaluation protocol: Physion uses a model-agnostic evaluation protocol and directly compares model predictions with human judgments on the same physical scenarios.The protocol is intended to support comparisons across architectures and training regimes.
- Findings: No current vision algorithm reaches human-level performance; object-centric approaches generally outperform non-object-centric ones but remain far below humans.The benchmark therefore distinguishes improvements in visual representation from human-level physical understanding.
- Findings: Particle-based models with direct physical-state access perform substantially better and produce predictions more similar to human judgments.These models use information such as object positions, poses, trajectories, and fine-scale shapes that vision systems cannot directly observe in the real world.
2 Methods
Physion uses simulated scenes spanning diverse rigid- and soft-body phenomena, with agent–patient contact as a common event-prediction task. The benchmark combines human testing with models that differ in architecture, supervision, representations, and access to physical state.
- Stimulus generation: Eight ThreeDWorld scenarios cover diverse physical phenomena using simple objects and rigid- or soft-body interactions.The scenarios include Dominoes, Support, Collide, Contain, Drop, and Link, among others.
- Task design: Agent–patient contact operationalizes higher-order outcomes such as tower collapse, containment, or attachment across scenarios.The shared task asks whether the designated agent and patient objects will contact.
- Stimulus generation: Each stimulus is a 5-10 second movie rendered at 30 frames per second, with visual, physical-state, and metadata outputs supplied for model development and evaluation.The simulator output includes images, depth, segmentation, optical flow, object states, collisions, labels, camera parameters, forces, and scenario-specific parameters.
- Human evaluation: Human testing used 150 stimuli per scenario, with familiarization trials followed by testing trials without feedback.Participants viewed an initial cue and the first 1500ms of each stimulus before responding.
- Model benchmarking: Models span unsupervised encoder-dynamics systems, supervised object-data models, ImageNet-pretrained visual encoders with RNN dynamics, and particle-graph models using simulator state.This comparison varies architecture, supervision, input representation, and access to physical dynamics.
- Benchmark pipeline: The benchmark separates testing, readout-fitting, and training sets so frozen pretrained models can receive task-specific adaptors without direct optimization on OCP labels.Readout-fitting stimuli train generalized linear adaptors, while training stimuli support learning physical-dynamics representations without visual agent–patient cues or ground-truth OCP labels.
3 Results and Discussion
Particle-based models with direct physical-state access approached human accuracy, while vision models remained below human performance despite benefits from object-centric representations and pretrained encoders. Model–human error consistency tracked task performance, but visual models did not appear to learn useful scene simulation.
- Particle-based models: Particle-based models matched human accuracy on many scenarios, and object-centric DPI reached across-scenario human performance levels.A single graph neural network learned human-level physical predictions across diverse scenarios when trained with broad scenario protocols.
- Vision-based models: Vision models remained far below human performance, although object-centric models generally outperformed non-object-centric models.CSWM outperformed SVG and OP3, while RPIN exceeded CSWM only under the all-but and only protocols.
- Vision-based models: ImageNet-pretrained Transformer models outperformed pretrained ConvNet models, which outperformed the best fully TDW-trained models.The reported differences were b=0.067, t=16.5, p<10−15 for Transformers over ConvNets and b=0.015, t=2.9, p<0.01 for ConvNets over TDW-trained models.
- Human-model comparison: Error-pattern consistency strongly correlated with absolute performance, but no model reached the human zone of statistically indistinguishable outputs.This gap remained even for particle-based models, including their ability to reproduce human errors.
- Readout protocols: Observed+simulated readout did not outperform observed readout for vision models, with p=0.53.The results imply that visual dynamics models did not learn scene simulations that improved the OCP task, although end-to-end dynamics training may still have shaped encoders.
- Generalization: Performance assessments were robust to scenario-selection protocols, but models trained on a single scenario generalized poorly to most others.Training with only versus all scenarios produced b=0.21, t=4.4, p<10−4, whereas all-but training was not significantly worse than all training: b=0.009, t=1.9, p=0.057.
Broader Impact
Physion connects physical prediction to safe interaction with real-world environments, while revealing important differences between human and model behavior. Its analyses show that generalization, observation limits, and representation choices shape performance.
- Broader impact: Physion could help assess whether AI systems predict physical consequences in ways relevant to safe interaction with people and environments.The benchmark is intended to measure physical prediction and diagnose failures in perceive-predict-act systems.
- Broader impact: Human performance was significantly below chance on some stimuli, often because of occlusion or outcomes near a physical boundary.These trials can be adversarial for human prediction when small initial changes reverse the outcome.
- Broader impact: DPI can exceed human accuracy because it receives ground-truth physical states and avoids visual occlusion, but its deterministic binary outputs limit similarity to average human decisions.The authors suggest probabilistic learned dynamics or decisions could better match human responses.
- Broader impact: No single training scenario generalized as well as training on all scenarios, although Drape and Support appeared to transfer best for DPI.Drape and Support may generalize better because they include varied soft and rigid object-object interactions.
- Model performance: The benchmark’s scenario-level results are heterogeneous: models struggle on some human-easy scenarios such as Roll but approach or exceed humans on some human-difficult scenarios such as Link.This variation motivates comparing models across multiple physical scenarios rather than relying on one task.
- Model details: Object-like models use latent vectors intended to represent individual objects and recurrent graph networks to pass messages during future prediction.CSWM and OP3 encourage object-like structure through inductive biases and unsupervised learning rather than explicitly constraining each latent to a physical object.
A.5 Experimental Details
The experiments used large-scale cloud training across 80 GPUs over multiple days.
- Experimental setup: Experiments ran on 80 NVIDIA T4 and V100 GPUs for two days on Google Cloud Platform.DPI-Nets and GNS were trained for 1.5M–2M iterations with Adam and an initial learning rate of 1e-4.
A.6 Links to access the dataset and its metadata.
The project shared its code and data under the MIT license and documented third-party software licenses.
- Access and licensing: Project code and data were released under the MIT license, with the license uploaded to the project repository.
- Access and licensing: Third-party software packages retained their own licensing provisions, documented in Table S2.
A.9 Datasheets for dataset
The datasheet documents Physion’s simulated videos, metadata, human judgments, public distribution, intended uses, and maintenance constraints.
- Purpose and uses: The dataset was created to measure adult short-term physical prediction and compare human judgments with AI-model predictions.Its intended scope concerns short-term forward predictions in simple physical scenarios.
- Composition: Physion instances pair simulated physical-scene videos with metadata such as depth, normals, object maps, and collision information.The dataset includes 2,000 training examples and 1,000 readout-fitting examples per scenario, plus 150 test examples per scenario.
- Data acquisition: Videos were generated in TDW, visually checked for scenario correctness, and human judgments were collected through online crowdsourcing.Model inputs and human responses were compared without additional preprocessing or cleaning.
- Distribution: The dataset and code are publicly distributed through GitHub and linked storage under the MIT license, with no export restrictions.The CogTools lab’s institutional GitHub account hosts the dataset-generation code.
- Maintenance: Future updates may expand the dataset and correct errors, while older versions will be maintained indefinitely.Third parties cannot contribute directly, although the open-source generation and collection code can be forked.
- Metadata: Structured metadata and a standard repository or DOI had not yet been created because availability details were expected to change during review.The project instead provided data links and project metadata through GitHub.
A.11 Dataset identifier
The project provides dataset and analysis-generation code through GitHub, while the data are accessed through Amazon S3 links listed there.
- Dataset identifier: Physion had no standard data-repository deposit or DOI at the time described.The authors expected the data-availability details to evolve during the paper review process.
- Dataset identifier: The GitHub repository serves as the access point for code and links to the dataset.The project distinguishes its code resource from its externally hosted data resource.
B Human experimental study preregistration
The preregistered human study investigated which visual-scene features make short-term physical predictions difficult and compared human judgments with computational models.
- Goals: The study aimed to identify scene features that hinder correct human physical prediction and collect data for comparison with computational models.The motivation frames prediction of physical outcomes as central to intelligent behavior in physical environments.
- Hypotheses: The preregistered hypotheses predicted greater difficulty for scenes with more elements, distractors, or occluders.The study also predicted longer reaction times for scenes producing more incorrect responses.
- Study structure: The researchers planned eight experiments covering different categories of physical scenarios.Each experiment tested physical judgments for a distinct scenario category.
- Stimulus generation: Scenes were generated by sampling physical parameters and selecting more than 150 candidate examples for each stimulus set.The planned sample balanced positive and negative trials when possible and manually excluded unusable scenes or physics-engine problems.
- Design: Participants were not assigned to experimental conditions; manipulations were embedded in the stimulus parameters and sampling process.The preregistration describes the design as stimulus-based rather than condition-assignment based.
B.2.2 Study design: evaluation protocol
The evaluation protocol showed participants brief physical-scene videos, elicited binary predictions before the outcome, and recorded response times and participant feedback.
- Trial procedure: Each trial identified agent and patient objects with flashing red and yellow overlays, played 1,500 ms of video, then queried a prediction.A fixation period and a 2,000-ms first-frame display preceded the video segment.
- Session structure: Participants completed 10 familiarization trials followed by 150 randomized test trials.Familiarization included feedback and full-stimulus outcomes, whereas test trials withheld accuracy feedback.
- Measures: The primary trial measures were a yes/no prediction and the time taken to make that prediction.These measures captured both physical-outcome judgments and response latency.
- Post-study measures: After testing, participants reported demographics, a five-point task-difficulty rating, and free-form feedback.They were also shown their overall accuracy and percentile after the study.
B.3.1 Data collection procedure
Participants are recruited through Prolific, compensated at approximately $12 per hour, and restricted to one participation per task version. Data collection ends after 100 participants complete the experiment.
- Participants are recruited from Prolific and compensated $4, corresponding to roughly $12 per hour.
- Participants may take each task only once but can complete an experiment version involving another scenario.
- Data collection stops when 100 participants have completed the experiment.
B.4.1 Data exclusion criteria
The protocol excludes incomplete or low-quality sessions from the main analyses, while retaining flagged sessions for sensitivity analyses. Human performance and human–model similarity are assessed with complementary accuracy, consistency, correlation, and reliability measures.
- Data exclusion criteria: Sessions are excluded for patterned responses, poor familiarization performance, or participant accuracy more than three standard deviations below the overall median.
- Data exclusion criteria: Flagged sessions are omitted from primary analyses but included in planned models estimating how flagging affects accuracy.
- Data exclusion criteria: Only complete sessions with responses for every trial are included in the main analyses.
- Human performance analyses: Human performance is summarized using stimulus-level and participant-level accuracy, human–human consistency, stimulus-attribute analyses, and scenario-level mixed-effects logistic regression.
- Human–model comparisons: Human–model comparisons evaluate both absolute accuracy against bootstrapped human confidence intervals and similarity of response patterns using correlation and Cohen’s kappa.