IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning

Motivation

• Current vision systems still lag human common-sense reasoning about macroscopic objects, despite strong performance on recognition, segmentation, and related tasks.Infants develop representations of object permanence, stability, causality, gravity, inertia, and momentum during the first year of life.Source: §1 · p. 1
• Future prediction is attractive because it needs no human annotations, but pixel-wise error is difficult to interpret, compare across datasets, and align with physical understanding.Macroscopic outcomes are also practically stochastic, so interactions produce distributions of possible object positions rather than a single deterministic outcome.Source: §1 · pp. 1–2

Proposed evaluation

• The proposed evaluation converts prediction error into a forced choice between matched possible and impossible events, requiring a plausibility score over an entire video.A model trained only on possible events should assign lower plausibility to clips containing physical violations.Source: §1 · p. 2
• Inspired by infants’ violation-of-expectation paradigm, the test uses scalar plausibility scores, with lower scores expected for clips violating physical principles.Varying the violation probes distinct laws, including conservation of object properties and object movement.Source: §1 · p. 2

Advantages and scope

• Well-matched, varied clips reduce dataset biases and simple heuristics while enabling interpretable results, out-of-distribution generalization tests, and human-machine comparisons.The benchmark is presented as a task- and model-agnostic diagnostic rather than a direct measure of an entire end-to-end system.Source: §1 · p. 2, Figure 1 · p. 2
• The tests measure physical understanding through prediction errors but do not establish whether a system can use that understanding for downstream tasks such as visual question answering.The paper frames the benchmark as a unit test that should be combined with other measures for complete system evaluation.Source: §1 · p. 2

2. Structure of the IntPhys 2019 benchmark

• IntPhys 2019 tests object permanence, shape constancy, and spatio-temporal continuity using unlabeled possible and impossible clips that require one plausibility score per video.The benchmark includes physically possible training videos and evaluates submitted scores on controlled test clips.Source: §2 · p. 3

2.1. Three basic concepts of intuitive physics

• Object permanence requires tracking objects through occlusion, shape constancy concerns preserving rigid-object shape despite appearance changes, and spatio-temporal continuity requires smooth trajectories.The benchmark’s three blocks target distinct principles and future releases are intended to add more complex interactions and agent motion.Source: §2.1 · p. 3

2.2. Pixels matched quadruplets

• Matched quadruplets reuse exactly the same frames across two possible and two impossible movies, isolating temporal coherence rather than low-level pixel differences.For object permanence, the possible transitions are 1→1 and 2→2, whereas impossible transitions are 1→2 and 2→1.Source: §2.2 · pp. 3–4, Figure 3 · p. 4

2.3. Parametric task complexity

• Each block varies visibility, motion complexity, and object count, producing 18 scenarios that expose where tracking and attentional demands exceed system capacity.Motion levels are static, dynamic 1, and dynamic 2; the design explicitly targets progressively harder conditions.Source: §2.3 · p. 4

2.4. Procedurally generated variability

• Each scenario has 200 procedural renderings with randomized shapes, textures, distances, trajectories, occluders, and camera positions, while dynamic 2 places violations at unpredictable times.Dynamic 2 contains two inverse violations whose beginning and ending segments can appear physically compatible.Source: §2.4 · p. 4, Figure 4 · p. 4, Figure 5 · p. 5

2.5. The possible versus impossible discrimination metric

• The relative error rate checks whether positives outrank negatives within each matched set, whereas the absolute error rate checks whether positives score higher globally; AUC supports the latter evaluation.The metrics operate on scalar plausibility scores assigned to matched movie sets.Source: §2.5 · p. 5

2.6.1 The training set

• The training set contains 15K videos of possible events, while dev and test sets use 360 and 3600 renderings per block, respectively, with test sets intended for fine-grained evaluation.Training videos are about seven seconds long at 15 fps; dev data support score and architecture selection but not parameter training.Source: §2.6.1 · p. 5, §2.6.2 · p. 5

2.6.3 Metadata

• Test metadata may include depth and shuffled instance masks, but training-only metadata additionally provides 3D object positions, camera position, and object-ID links.Shuffling masks prevents direct temporal object-ID linking, leaving object permanence as part of the task.Source: §2.6.3 · p. 5

3. Two baseline learning models

• Two baseline systems learn intuitive physics from observational video using future semantic-mask prediction, with identical training data enabling model comparison.The setup uses no interaction with objects and is intended to test how far simplified, abundant video inputs can support learning.Source: §3 · p. 6

3.1.1 CNN encoder-decoder

• The CNN encoder-decoder extracts features with a ResNet-18 and deconvolution network, then predicts the future semantic mask using an L2 loss.Semantic masks distinguish background, occluders, and objects after pixel-level prediction proved inadequate for convincing object motion.Source: §3 · p. 6, §3.1.1 · p. 6

3.1. Models

• Both architectures predict a future frame from two input frames, with spans of 5 frames for short-term or 35 frames for long-term prediction.The long-term setting is intended to capture longer dependencies and handle extended occlusions, whereas short-term prediction is expected to be more robust.Source: §3.1 · p. 6, §3 · p. 6

3.1.2 Generative Adversarial Network

• The conditional GAN predicts future masks from past masks and uses a conditioned discriminator score as the test-time plausibility signal.Its physical-event interpretation assumes the generator assigns positive support to impossible events; without that support, the discriminator cannot reliably reject them.Source: §3.1.2 · p. 7

3.1.3 Training Procedure

• Training uses a held-out 10% split for overfitting control and Adam optimization, with one CNN epoch and one or five GAN epochs depending on prediction span.The GAN uses different generator and discriminator learning rates and trains longer for the long-term task.Source: §3.1.3 · p. 7

3.2. Video Plausibility Score

• The system computes a video plausibility score as the minimum plausibility among all conditioned frame triplets in the video.This reflects the assumption that an impossible event may appear in one or more frames, even though its temporal location is unknown.Source: §3.2 · p. 7

3.3.1 Block O1

• In O1, long-term models perform better than short-term models on occluded impossible events, while occlusion produces higher relative-classification error than visibility for CNN (0.52 vs 0.07) and GAN (0.48 vs 0.17).Source: §3.3.1 · p. 8
• In O1 aggregated evaluation, the CNN encoder-decoder reaches relative error rates of 0.07 for visible and 0.52 for occluded impossible events.Its absolute-task LA is 0.37 visible and 0.50 occluded.Source: §3.3.1 · p. 8

3.3.2 Block O2

• For O2, the aggregated CNN encoder-decoder has relative error rates of 0.11 visible and 0.52 occluded, with absolute-task LA of 0.43 and 0.50.The short-term CNN relative errors are 0.16 visible and 0.49 occluded, while the long-term values are 0.11 and 0.52.Source: §3.3.2 · p. 8

3.3.3 Block O3

• For O3, the aggregated CNN encoder-decoder has relative error rates of 0.32 visible and 0.51 occluded, with absolute-task LA of 0.46 and 0.50.The long-term GAN has relative errors of 0.34 visible and 0.52 occluded, while its absolute LA is 0.44 and 0.50.Source: §3.3.3 · p. 9
• Performance is highest for presence or absence of an object and lowest for temporal continuity of trajectories, while occlusion generally reduces performance.The authors relate the occlusion effect to the difficulty of long-term prediction.Source: §3.3.3 · p. 9

Human evaluation

• 540 participants judged 3600 videos per block on a 1–6 plausibility scale, with each video seen by two participants.Participants received eight possible-scene examples, judged 40 randomly selected videos, and completed the task in about seven minutes.Source: §4 · p. 10

Visibility effects

• Human error increased under occlusion: O1 rose from 18% to 30%, O2 from 22% to 30%, and O3 from 28% to 47%.The averages were computed across condition, object count, and visibility.Source: §4 · p. 10

Event complexity

• For visible O3 scenes, human performance was close to chance, indicating difficulty detecting velocity changes associated with occlusion.Difficulty patterns also varied between one and two occlusions, depending on visibility.Source: §4 · p. 10

Heuristic errors

• When discontinuities were occluded, participants sometimes relied on object-count heuristics and missed an intermediate disappearance.For visible scenes, two discontinuities could be easier than one because they create two local discontinuities.Source: §4 · p. 10

Interpretation

• Human observers did not consistently follow the physical gold standard, suggesting attentional limitations even below the theoretical four-object tracking limit.Their performance provides a human reference for systems intended to model human perception.Source: §4 · p. 10

Model–human comparison

• Models and humans both showed increasing error from O1 to O3 and higher error when impossible events were occluded.The shared pattern links model difficulty to the same benchmark conditions that challenged people.Source: §4 · p. 10

Future prediction

• Most intuitive-physics systems have used future prediction, including models for block-tower stability and dynamics.The surveyed approaches include supervised CNNs, simulation comparisons, generative prediction, and feature-learning strategies.Source: §5 · pp. 10–11

Human results

• The supplied passages report human average error rates separately for IntPhys 2019 test-set blocks O1 and O2, using plausibility judgments collected through MTurk.Table 5 covers Block O1, while Table 6 covers Block O2.Source: Table 5 · p. 11, Table 6 · p. 11

Structured representations

• Structured approaches learn object dynamics through pairwise interactions or factored object representations decoded into future frames.These models represent states such as position, velocity, and intrinsic object properties.Source: §5 · p. 11

Benchmarks

• Among evaluation benchmarks, the Visual Newtonian Dynamics (VIND) dataset includes more than 6000 videos with bounding boxes, while frame-prediction datasets are not strictly about intuitive physics.Source: §5 · p. 11

Benchmark comparison

• A contemporaneous developmental benchmark covers object permanence, shape constancy, continuity, solidity, and containment, but its size and characteristics were not known.IntPhys differs through pixel-level quadruplet matching and factorial manipulation of scene and movement complexity.Source: §5 · p. 11

Benchmark contribution

• IntPhys evaluates intuitive physics by requiring plausibility scores that discriminate possible from impossible videos, with naive humans providing a performance reference.The benchmark is inspired by conceptual development in infants.Source: §6 · p. 11

Human findings

• Human error increased with occlusion but not with object count, consistent with the reported ability to track up to three objects simultaneously.The benchmark therefore exposes visibility-related difficulty without a corresponding object-count effect in these results.Source: §6 · p. 12

Model setup

• The two unsupervised models learned semantic-mask prediction from physically plausible clips and were evaluated on the same benchmark block as humans.Their training data contained only physically plausible clips.Source: §6 · p. 12

Model results

• The computational system performed below humans but above chance, with a strong negative effect of occlusion.Semantic-mask prediction was more successful than the authors’ earlier pixel-based systems.Source: §6 · p. 12

Scope and limitations

• The benchmark is challenging and currently tests prediction probabilities for object positions rather than how systems use them for decisions or trajectory planning.Future extensions could add object interactions, animated-agent trajectories, and decision or planning tests.Source: §6 · p. 12

Block O1

• O1 results are tabulated in Tables 8–13 for CNN and GAN models on short-term tasks, and CNN models on long-term tasks, using relative or absolute classification.Source: Table 8 · p. 16, Table 9 · p. 16, Table 10 · p. 16

Block O2

• O2 results are tabulated in Tables 18–23 for CNN and GAN models on short-term and CNN on long-term prediction tasks, using relative and absolute classification.Source: Table 18 · p. 18, Table 19 · p. 18, Table 20 · p. 18

Block O3

• O3 results are tabulated for CNN and GAN models in short-term prediction tasks, and for CNN in long-term prediction tasks, using relative and absolute classification (LR and LA).These supplied passages correspond to Tables 28–33.Source: Table 28 · p. 20, Table 29 · p. 20, Table 30 · p. 20

• Relative classification with human evaluation is reported for block O1.Source: Table 38 · p. 23
• Relative classification with human evaluation is reported for block O2.Source: Table 39 · p. 23
• Relative classification with human evaluation is reported for block O3.Source: Table 40 · p. 23

• The mask predictor has 9,747,011 parameters, with BN denoting batch-normalization.Source: Table 41 · p. 24
• Example outputs are arranged as input image, ground-truth semantic mask, and predicted semantic mask.Source: Figure 9 · p. 24