PaperlayerView exact revision on arXiv

Source-linked AI summary

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, Judith E. Fan

arXiv:2106.08261v3cs.AIcs.CV

TL;DR

How well vision algorithms predict real-world physical dynamics remains unclear, despite the importance of anticipating physical events. Physion benchmarks models and human predictions on diverse physical scenarios, finding that object-centric vision models still lag people while models given physical-state information perform better and make more human-like predictions.

  • Problem

    How well vision algorithms predict real-world physical dynamics remains unclear, despite the importance of anticipating physical events for safe and effective autonomous interaction.

    Source: §1 · p. 1, §1.5 · p. 4

  • Method

    Physion benchmarks models and human predictions on diverse simulated physical scenarios, supplying visual data and physical-state information for model evaluation.

    Source: §1.5 · p. 4, §2.2 · p. 5, §3 · p. 8

  • Results

    Object-centric vision models generally outperform non-object-centric models but remain far below humans, while particle-based graph neural networks with physical-state access perform substantially better and make more human-like predictions.

    Source: §1.5 · p. 4, §3 · pp. 8–9

  • Takeaways & Limitations

    The benchmark is intended to support development of algorithms with more general, human-like physical prediction.

    Source: §3 · p. 10

    Untangling the effects of encoder architecture, training datasets, and supervised versus self-supervised losses requires finer-scale comparisons.

    Source: §3 · p. 10

Abstract

from arXiv · show

While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.

1 Introduction

Physion provides a benchmark for predicting physical events from varied visual scenarios and comparing model predictions with human judgments. The introduction motivates this evaluation as a step toward safer AI that can anticipate physical consequences.

1 Introduction

  • Predicting physical consequences is important for AI and autonomous systems to interact safely and effectively with their environments.Source: §1 · p. 1

1.1 Establishing Common Standards for Evaluating Physical Understanding

  • The proposed standard requires event-based predictions from varied, unstructured visual inputs and enables direct comparison of model predictions with human judgments.Source: §1.1 · p. 2

1.2 Desiderata for a Generalized Physical Understanding Benchmark

  • A generalized benchmark should pair visually realistic, varied physical examples with an evaluation protocol agnostic to model architecture and training regime.Source: §1.2 · p. 2

1.3 Physion: A Dataset and Benchmark for Physical Understanding

  • Physion covers eight realistic scenario types and tests physical understanding with object contact prediction, asking whether two cued objects will touch as a scene unfolds.Source: §1.3 · p. 3

1.4 Using Physion to Benchmark Human and Model Physical Understanding

  • Physion compares human behavior with diverse state-of-the-art models on the same scenarios, spanning architectures, learning objectives, input-output structures, and training regimes.Source: §1.4 · p. 3

1.5 Summary of Key Findings

  • No current vision algorithm reaches human-level Physion prediction; object-centric vision models generally outperform others, while particle-based models with physical-state access perform better and align more with humans.Source: §1.5 · p. 4

1.6 Our Vision for Physion

  • Physion’s primary intended use is to test pretrained models’ transfer to physical understanding tasks, with shared code supporting reproducible benchmarking of progress.Source: §1.6 · p. 4

2 Methods

Physion uses eight simulated physical scenarios and a shared contact-prediction task to compare human and machine predictions. Its benchmark pairs varied stimuli and rich simulation data with standardized human testing and model training, readout, and evaluation protocols.

2.1 Benchmark Design

  • The eight scenarios probe collisions, support, containment, gravity, attachment, rolling, and cloth dynamics; agent–patient contact indicates higher-order physical outcomes across rigid- and soft-body interactions.The scenarios include Dominoes, Support, Collide, Contain, Drop, Link, Roll, and Drape; fluids, squishy objects, and jointed objects are proposed as future additions.Source: §2.1 · p. 4

2.2 Stimulus Generation and Task Design

  • Toy-block scenes vary physical initial conditions and visually irrelevant properties, while all scenarios share a task asking whether designated agent and patient objects will touch by the time motion stops.Movies are 5–10 seconds at 30 fps; simulation outputs include visual maps, physical states, collision data, labels, and metadata. Human test sets contain 150 balanced contact/no-contact stimuli per scenario.Source: §2.2 · p. 5

2.3 Testing Humans on the Physics Prediction Benchmark

  • Eight hundred participants were recruited, 100 per scenario, and 112 were excluded for failing preregistered attention-check criteria.Trials cued agent and patient identities, showed only the first 1.5 seconds before YES/NO predictions, and followed 10 familiarization trials with full-video feedback.Source: §2.3 · pp. 5–6

2.4 Benchmarking Computer Vision and Physical Dynamics Models

  • Testing trials use RGB movies with agent and patient permanently colored, include ground-truth contact labels and responses from >100 human participants; each scenario’s readout-fitting set has 1,000 matched stimuli.Training sets contain 2,000 movies without agent/patient visual cues or ground-truth OCP labels, supporting representation learning or fine-tuning agnostic to a particular readout task.Source: §2.4 · p. 7
  • The evaluation pipeline freezes pretrained models and fits a logistic-regression task adaptor; models with joint visual and dynamics components train end-to-end, while encoder-only models gain an RNN dynamics predictor.For encoder-only models, the RNN predicts future encoder outputs using mean squared error, with encoder parameters either fine-tuned or frozen.Source: §2.4 · p. 7
  • The benchmark compares unsupervised joint encoder-dynamics models, object-supervised RPIN, ImageNet-pretrained encoders with RNN predictors, and state-input graph neural networks.The listed model families are SVG, OP3, CSWM; RPIN; pVGG and pDeIT variants; and GNS, GNS-RANSAC, and DPI.Source: §2.4 · p. 7
  • Training protocols vary across all scenarios, all-but-the-test scenario, or only the test scenario; readouts use observed, observed-plus-simulated, or full-movie features and assess accuracy, Pearson correlation, and Cohen’s κ against human variability.The full protocol is a detection rather than prediction task and is not directly comparable to human data; the all-scenarios training protocol is intended as the test of general physical understanding.Source: §2.4 · pp. 7–8

3 Results and Discussion

Physion results show that particle-input graph networks can approach human accuracy across diverse physical scenarios, while vision-based models remain below human performance. Better visual representations and scenario diversity improve prediction, but model-human differences persist in error patterns and evidence of learned simulation.

Results and Discussion

  • Humans were reliably above chance across all eight scenarios, averaging 0.71 correct, though accuracy varied and some stimuli elicited below-chance responses.The tight human zones across raw performance, correlation-to-average, and Cohen’s κ indicate reliable response patterns; some reliable errors arose on unlikely events after the observed movie segment.Source: §3 · p. 8
  • Particle-input graph networks matched human accuracy on many scenarios, and object-centric DPI reached across-scenario human performance levels; vision models remained far below human accuracy.The particle models received ground-truth object particles, including physical information unavailable from images alone.Source: §3 · pp. 8–9, Figure 5 · p. 10
  • Among fully unsupervised vision models, CSWM outperformed SVG and OP3, while RPIN exceeded CSWM only under the all-but and only protocols, not the all protocol.OP3 performed marginally better than SVG, and further experiments were needed to isolate the effect of ground-truth supervision from architectural differences.Source: §3 · p. 9
  • ImageNet-pretrained ConvNet models outperformed the best fully TDW-trained models, and ImageNet-pretrained Transformer models in turn outperformed the ConvNet models.The authors note that ImageNet pretraining may capture physically relevant information, but finer-scale comparisons of architectures, datasets, and losses are needed.Source: §3 · pp. 9–10
  • Model-human error-pattern consistency rose with model accuracy, but no model entered the human zone, and particle-model predictions were uncorrelated with human predictions on adversarial stimuli.Adding noise to forward dynamics may better mimic human judgments about probable outcomes than precise simulation of rare flukes.Source: §3 · p. 9
  • Vision models’ observed+simulated readout was no better than their observed readout (p=0.53), although the full readout protocol improved accuracy (b=0.094).The results suggest that visual features may support object-contact detection, while above-chance observed-protocol performance may reflect initial-scene cues rather than learned scene simulation.Source: §3 · p. 9
  • Vision models generalized only slightly better under the only protocol and were not significantly worse under all-but training, but models trained on one scenario generalized poorly to most others.The findings support the importance of diverse observations for learning general physical predictors; the authors call for tests covering additional phenomena, including fluids, squishy materials, jointed objects, and wider physical-parameter ranges.Source: §3 · p. 10

Broader Impact … A.4 Model Details

Physion is presented as a way to assess and diagnose human-like physical prediction. Supplementary analyses show model-human limits, scenario-dependent generalization, and how visual representations and physical dynamics are tested separately.

Broader Impact

  • Physion could screen algorithms and diagnose failures in safety-relevant perceive-predict-act systems; its results suggest explicit physical representations may help prediction and generalization.The authors raise the possibility that semantic and physical understanding may benefit from different architectures and learning principles.Source: Broader Impact · p. 11

A.4 Model Details

  • Unsupervised visual dynamics span image-like SVG and object-like CSWM/OP3 latents, whereas supervised RPIN uses annotated boxes and interaction networks for object-position prediction.CSWM and OP3 encourage object-like factors without explicitly constraining them to represent physical objects; RPIN requires human-annotated bounding boxes.Source: §A.4 · pp. 16, 19, 20

A.1 Adversarial Stimuli and Model-Human Disagreement

  • Some stimuli put people below chance because of occlusion or near-boundary outcomes; DPI avoids these visual limits but its deterministic binary predictions restrict human-response matching.The authors suggest probabilistic learned dynamics or decisions could better approximate average human responses.Source: §A.1 · p. 16, Figure S1 · p. 17

A.2 Across Scenario Generalization

  • Training on all eight scenarios generalizes better overall than training on any single scenario, though some single-scenario training sets transfer well to particular phenomena.For DPI, Drape and Support came closest to broad generalization, while Collide, Drop, Link, and Roll were often learned well from other scenarios.Source: §A.2 · p. 16, Figure S5 · p. 20

A.4 Model Details

  • Frozen ImageNet-pretrained VGG-19 and DeIT encoders were paired with trained MLP or LSTM dynamics to test whether their features contain useful physical information.The dynamics models use an unsupervised forward-prediction L2 loss on benchmark training data.Source: §A.4 · pp. 20–21

A.3 Model Performance Per Scenario

  • Roll is easy for people but no model approaches human performance, while models approach or exceed humans on Link, which people find difficult.The table reports mean accuracy with bootstrapped 95% confidence intervals across the eight scenarios.Source: §A.3 · p. 16, Table S1 · p. 21

A.4 Model Details

  • Ground-truth-state models isolate physical dynamics from visual representation; DPI uses hierarchical particle graphs, while GNS is flat and GNS-RANSAC aims to prevent rigid-object deformation.DPI and GNS predict particle positions and velocities through message passing; GNS-RANSAC applies RANSAC to update rigid-object particle positions.Source: §A.4 · pp. 21–22

A.5 Experimental Details

The experiments used large-scale GCP compute and trained DPI-Nets and GNS to convergence with Adam at an initial learning rate of 1e-4.

A.5 Experimental Details

  • DPI-Nets and GNS were trained to convergence for 1.5M and 2M iterations using Adam with an initial learning rate of 1e-4.Source: §A.5 · p. 22
  • Experiments ran on 80 GCP GPUs (NVIDIA T4s and V100s) for two days, while training runs took about 2–5 days.Source: §A.5 · p. 22

A.6 Links to access the dataset and its metadata. · A.7 Long-term preservation plan · A.8 License Information

The project shares its code and data under the MIT license, with the license uploaded to the code repository. Third-party software packages may have separate licensing provisions.

A.8 License Information

  • The project’s products, including code and data, are shared under the MIT license, which is uploaded to the code repository where the data is referenced.Source: §A.8 · p. 22
  • Third-party software packages typically have their own licensing provisions, and Table S2 lists licenses for many packages used.Source: §A.8 · p. 22

A.9 Datasheets for dataset … A.10 Structured metadata

The appendix documents the Physion dataset using Datasheets for Datasets [25] and notes that structured metadata in formats such as schema.org or DCAT had not yet been created.

A.9 Datasheets for dataset

  • The appendix states that its responses follow Datasheets for Datasets [25] standards.Source: §A.9 · p. 23

A.9.1 Motivation

  • The dataset was created to measure adults’ short-term physical-future predictions and compare them with AI-model predictions.The authors include researchers from Stanford, UCSD, and MIT; funding included NSF, NIH, DARPA, and the McDonnell Foundation.Source: §A.9.1 · p. 23

A.9.2 Composition

  • Physion instances are simulated physical-scene videos with simulator metadata; training data are unlabeled, while readout-fitting and human-test examples have binary red-object/yellow-zone collision labels.There are 2,000 training examples and 1,000 readout-fitting examples per scenario across eight scenarios, plus 150 human-test examples per scenario; the released data are the examples actually generated, a tiny subset of what the simulator could generate.Source: §A.9.2 · p. 23

A.9.3 Collection Process

  • TDW generated the videos, crowdsourcing collected human judgments, researchers visually checked scenarios and tested the experiment websites, and human-data collection received Stanford and UCSD IRB approval.Participants were recruited via Prolific and compensated $4; videos were generated in early May 2021 and human data were collected over approximately one week that May.Source: §A.9.3 · p. 24

A.9.4 Preprocessing, clearning and labelling.

  • No preprocessing or cleaning was performed: simulator-generated inputs had known labels, and model comparisons used raw human judgments.The authors state that no further preprocessing was applied to the collected human responses.Source: §A.9.4 · p. 24

A.9.5 Uses.

  • The dataset is intended only for evaluating human or model short-term forward predictions about simple physical scenarios.Participants had used it to characterize human judgments for that purpose; the authors reported no other papers using it yet and identified no other tasks for it.Source: §A.9.5 · p. 24

A.9.6 Distribution.

  • The dataset and associated code were to be publicly available immediately via GitHub and linked permanent Amazon S3 resources under the MIT license.The authors reported no third-party IP restrictions or export controls.Source: §A.9.6 · pp. 24–25

A.9.7 Maintenance

  • Dataset-generation code is hosted in a public GitHub repository associated with UCSD’s CogTools lab, and the authors expect ongoing expansion and error correction.Old versions will be maintained indefinitely; third-party contributions are not contemplated, though the open-source generation and crowdsourcing code can be forked to create separate benchmarks.Source: §A.9.7 · p. 25

A.11 Dataset identifier

Physion provides code through GitHub and data through S3 links listed in the repository; the data had not yet been deposited in a standard repository or assigned a DOI while its release details awaited paper review.

A.11 Dataset identifier

  • Code is available through the GitHub repository, which also provides links to the dataset hosted on Amazon S3.Source: §A.11 · p. 25
  • The dataset had not yet been placed in a standard repository or assigned a DOI because its release details were expected to change during paper review.The authors planned to deposit it in a standardized repository and generate a DOI after review.Source: §A.11 · p. 25

B Human experimental study preregistration … B.2.1 Study design

The preregistered human study investigates which physical-scene features hinder correct prediction and collects human prediction data for comparison with computational models. It tests predictions across eight experiments using sampled and manually checked scenes.

B Human experimental study preregistration; B.1 Study information

B.1.1 Research questions

  • The study asks which physical-scene features make human prediction difficult and collects scene-level difficulty data for comparison with computational models.Source: §B.1.1 · pp. 25–26

B.1.2 Hypotheses

  • The preregistration predicts that more elements, distractors, and occluders increase prediction difficulty, and that harder scenes take longer to answer.Source: §B.1.2 · p. 26

B.2.1 Study design

  • The study conducts eight experiments, each testing physical judgments in a different category of physical scenarios.Source: §B.2.1 · p. 26
  • Scenes are generated by sampling values of physical parameters, including element count, occluder count, and positional jitter, to create >150 example scenes, from which 150 are randomly sampled.Source: §B.2.1 · p. 26
  • The sampled set targets equal numbers of positive and negative trials and roughly balanced distributions of other dimensions, where possible.Source: §B.2.1 · p. 26
  • Stimuli are manually checked for usability, including off-screen elements, physics-engine bugs, and object clipping.Source: §B.2.1 · p. 26
  • Participants are not assigned to conditions; manipulations arise from stimulus parameters and stimulus sampling.Source: §B.2.1 · p. 26

B.2.2 Study design: evaluation protocol · B.2.3 Measured variables · B.3 Sampling Plan

Participants completed 10 familiarization trials followed by 150 test trials, and researchers recorded yes/no predictions and response times. During testing, trials were randomized and participants received no accuracy feedback or view of the subsequent video frames.

B.2.2 Study design: evaluation protocol

  • Each trial began with a fixation cross for 500–1500 ms, followed by a 2000 ms first-frame preview and 1500 ms of video before a YES/NO prediction.The fixation duration was randomly sampled; response buttons were enabled after the stimulus was removed.Source: §B.2.2 · p. 26
  • During the preview, agent and patient objects were identified by red and yellow overlays that flashed at 2 Hz.Source: §B.2.2 · p. 26
  • Participants completed 150 test trials in randomized order without accuracy feedback or seeing the scenario’s subsequent video frames.They first completed 10 familiarization trials with feedback and the unabridged stimulus, including the trial outcome.Source: §B.2.2 · p. 26
  • Familiarization comprised 10 trials with feedback on prediction accuracy and presentation of the full stimulus, including its outcome.Source: §B.2.2 · p. 26

B.2.3 Measured variables

  • Researchers measured each participant’s yes/no prediction and the time taken to make it.Source: §B.2.3 · p. 26
  • After the trials, participants provided age, gender, education level, a five-point task-difficulty rating, and free-form feedback; afterward, they were shown overall accuracy and their percentile relative to other participants.Source: §B.2.3 · p. 27

B.3.1 Data collection procedure · B.3.2 Sampling procedure

The procedure recruited participants through Prolific under fixed compensation and participation rules, and sampling ended after 100 participants completed the experiment.

B.3.1 Data collection procedure

B.3.2 Sampling procedure

  • Data collection stopped after 100 participants completed the experiment.Source: §B.3.2 · p. 27

B.4 Analysis Plan … Cohen’s kappa

The analysis plan specifies session-quality filters and human accuracy and consistency measures, then compares model performance and response patterns with human behavior. It also outlines exploratory analyses of stimulus attributes, demographics, and response biases.

B.4.1 Data exclusion criteria

  • Sessions are excluded for extreme response sequences, poor familiarization accuracy, unusually low accuracy, or unusually slow response times.Thresholds are more than 12 consecutive identical answers, at least 24 alternating answers, fewer than 4/10 familiarization trials correct, or accuracy below or log-response time above the scenario median by 3 standard deviations; flagged sessions are excluded from main analyses and tested in a sensitivity analysis.Source: §B.4.1 · p. 27

B.4.2 Missing data

  • Main analyses include only complete sessions with responses collected for every trial.Source: §B.4.2 · p. 27

B.4.3 Planned analyses

  • Planned measures include stimulus-level and participant-level accuracy, stimulus-level modal-response consistency, and within-scenario correlations between participants’ binary response vectors.Source: §B.4.3 · p. 27
  • Exploratory analyses examine how stimulus attributes and their linear combinations relate to accuracy, and model accuracy across scenarios with mixed-effects logistic regression.The regression includes scenario as a fixed effect and participants and individual stimuli as random effects.Source: §B.4.3 · p. 28

Other exploratory human behavioral analyses

  • Exploratory behavioral analyses relate age, gender, education, and one-trial spatial reasoning to accuracy, and assess left/right and yes/no response biases.Source: Other exploratory human behavioral… · p. 28
  • Models will be compared with human accuracy by scenario and with human response patterns using correlation, response deviation, and Cohen’s kappa.Human accuracy estimates will use 95% bootstrap confidence intervals from 1,000 resamples, compared with each model’s point estimate; response-pattern similarity is assessed by how close mean model-human kappa is to mean human-human kappa.Source: Other exploratory human behavioral… · p. 28, Cohen’s kappa · p. 28
  • For each scenario, model accuracy is compared with a 95% confidence interval for mean human accuracy, estimated by bootstrapping participants 1,000 times.The model’s proportion correct is a point estimate compared with the human confidence interval.Source: Other exploratory human behavioral… · p. 28
  • For each scenario, model hit probabilities are compared with human hit proportions using root-mean-squared deviation and bootstrap-resampled correlations.The bootstrap resamples individual participants and recomputes the correlation between the model probability-hit vector and the human proportion-hit vector.Source: Other exploratory human behavioral… · p. 28

Related papers

Loading 2106.08261v3…