PaperlayerView exact revision on arXiv

Source-linked AI summary

CoPhy: Counterfactual Learning of Physical Dynamics

Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, Christian Wolf

arXiv:1909.12000v2cs.CV

TL;DR

Counterfactual prediction asks how physical outcomes change after an intervention, a causal-learning problem that remains unexplored for high-dimensional video. The paper introduces CoPhy and a model that estimates latent confounders from an observed experiment to predict outcomes under altered initial conditions. The counterfactual setting outperforms conventional forecasting, and the learned representation is related to ground-truth confounders.

  • Problem

    Predicting an unrealized intervention’s effect from observations requires modeling causal relations, a challenge that has been unexplored for high-dimensional video.

    Source: §1 · p. 1

  • Method

    CoPhy pairs original physical sequences with intervened alternatives, and its model estimates latent confounders from the original experience without supervising their values.

    Source: §1 · pp. 1–2, §4.1 · p. 7

  • Results

    The counterfactual setting outperforms conventional forecasting, and the latent representation is related to ground-truth confounder quantities.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that counterfactual reasoning in high-dimensional spaces is a key component of AI and propose CoPhy as a benchmark for further work.

    Source: §6 · p. 10

    Confounders such as mass, friction, and gravity are not available during training or inference, although ground-truth values are provided for evaluation.

    Source: §3 · p. 4

Abstract

from arXiv · show

Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input. We develop the CoPhy benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting. Having observed a mechanical experiment that involves, for example, a falling tower of blocks, a set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions, such as displacing one of the objects in the scene. The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision. We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment, which results in significantly more accurate predictions at the level of super human performance.

1 INTRODUCTION

The paper frames counterfactual reasoning as predicting how physical outcomes change under interventions, a challenging causal-learning problem for high-dimensional video. It introduces CoPhy and a model that uses the observed outcome to infer latent confounders and predict an alternative future.

1 INTRODUCTION

  • Counterfactual reasoning predicts the effect of an unrealized intervention from observations, requiring a model of causal relationships between the intervened variable and the alternative outcome.The paper identifies high-dimensional video as an unexplored setting for this reasoning.Source: §1 · p. 1
  • CoPhy studies falling block towers, bouncing balls, and collisions by predicting object positions and poses after changing the initial frame.The observed sequence spans six seconds, and interventions include changes such as object displacement.Source: §1 · pp. 1–2
  • Unlike feedforward forecasting from past frames, counterfactual prediction uses the original outcome to infer hidden properties such as mass, velocity, or friction before predicting from the altered past.The inferred confounder representation supplies information unavailable from the altered initial frame alone.Source: §1 · p. 2
  • The CoPhy benchmark and proposed model target counterfactual physical prediction, with the model outperforming feedforward video prediction and generalizing to unseen initial states without confounder supervision.The paper also reports human comparisons and ablations.Source: §1 · p. 2

2 RELATED WORK

The related work connects visual reasoning, intuitive physics, and perceived causality, positioning counterfactual prediction as a new problem for high-dimensional visual input. Existing benchmarks and models address adjacent tasks, but differ in their goals, supervision, or evaluation format.

2 RELATED WORK

  • Prior visual-reasoning work spans question answering, embodied AI, and intuitive physics, while this work extends visual reasoning to counterfactual settings.Related approaches include relational networks, recurrent memory, and graph-based reasoning.Source: §2 · p. 3
  • Intuitive-physics studies include stability prediction, interpretable or reconstructed physics models, and learned physical engines, with several graph-convolution approaches requiring observable physical properties.The paper contrasts that observability assumption with its own approach.Source: §2 · p. 3
  • Existing physics benchmarks differ from CoPhy’s goals of evaluating counterfactual reasoning with balanced outcomes, unbiased parameter distributions, and latent physical confounders.The paper notes that, to its knowledge, existing intuitive-physics benchmarks do not combine these properties.Source: §2 · p. 3
  • Phyre does not explicitly evaluate counterfactual reasoning, whereas CLEVRER treats counterfactual questions as answer selection rather than prediction of each object’s dynamics.Adapting other benchmarks would require changes to their data sampling and task logic.Source: §2 · p. 4
  • Cognitive studies of perceived causality report impressions that can diverge from physical principles, including a bias linking an object’s velocity with its mass.The paper situates this work at the intersection of perceived causality, visual reasoning, and intuitive physics.Source: §2 · p. 4

3 COPHY: COUNTERFACTUAL PHYSICS BENCHMARK SUITE

CoPhy is a three-scenario benchmark for predicting alternative object positions after interventions, using paired original and counterfactual sequences. It balances outcomes so that latent confounders matter, while withholding their values from model training and inference.

3 COPHY: COUNTERFACTUAL PHYSICS BENCHMARK SUITE

  • CoPhy comprises BlocktowerCF, BallsCF, and CollisionCF, evaluating alternative 3D object positions after do-interventions on visual scenes.The scenarios adapt established intuitive-physics settings to counterfactual prediction.Source: §3 · p. 4
  • Each experiment pairs an original physical sequence with a counterfactual sequence whose initial state reflects an intervention such as object displacement or removal.The alternative future is the outcome after intervention; confounders are shared across original and modified sequences.Source: §3 · p. 4
  • Confounders such as mass, friction, and gravity are visually unobservable and are not available during training or inference, although the benchmark stores ground truth for evaluation.The benchmark frames these quantities as properties that cannot be uniquely estimated from one time step.Source: §3 · p. 4
  • BlocktowerCF includes 146k experiments over 73k geometric configurations, with block displacement or removal and varying mass, friction, and gravity.It uses towers of three or four stacked cubes.Source: §3 · p. 5
  • The benchmark balances confounder statistics and possible outcomes, ensuring that at least two confounder configurations can yield significantly different trajectories for an experiment.This construction makes confounder estimation necessary for reasoning on the dataset.Source: §3 · p. 5
  • The benchmark renders RGB, depth, and instance-segmentation images at 448 × 448 pixels, but this work uses RGB only.Physical properties are stored at 20 fps, and each scenario uses a 0.7:0.2:0.1 training, validation, and test split.Source: §3 · p. 5

4 COUNTERFACTUAL LEARNING OF PHYSICS

The model combines object-centric representations, graph-based interaction modeling, and unsupervised latent confounder estimation to predict counterfactual physical dynamics. Its trajectory predictor uses stability estimates to gate predictions of object positions.

4 COUNTERFACTUAL LEARNING OF PHYSICS

  • The end-to-end model estimates physical properties, models object interactions, learns latent confounder representations without supervision, and uses them to predict object positions.Source: §4 · p. 6

4.1 UNSUPERVISED ESTIMATION OF THE CONFOUNDERS

  • The model detects object positions, applies fully connected graph convolutions to model interactions, and uses a shared-parameter per-object GRU whose final hidden states form unsupervised latent confounders.The shared recurrent parameters make the model invariant to the number of objects, allowing removal interventions that change scene size.Source: §4.1 · pp. 6–7

4.2 TRAJECTORY PREDICTION GATED BY STABILITY

  • The trajectory predictor estimates per-step residual motion and uses a binary stability estimate to gate object-position prediction while modeling inter-object relationships with a graph network.Source: §4.2 · p. 7
  • For each altered scene, stability is inferred by combining object features with latent confounders and graph-updating them; training uses stability cross-entropy and position mean squared error.Source: §4.2 · p. 7

5 EXPERIMENTS

Experiments compare CoPhyNet with human and copying baselines, test generalization to unseen settings, and examine model components and latent confounder estimates.

Human performance

  • Humans perform slightly better with the original sequence and altered frame than with the altered frame alone, but remain below copying baselines.Their predictions may follow stability biases: prediction variance decreases after stable sequences and increases after falling sequences.Source: §5 · p. 8

Performance and comparisons

  • CoPhyNet significantly outperforms human subjects and copying baselines on BlockTowerCF.The comparison uses CoPhyNet’s estimated 3D positions projected into 2D image space.Source: §5 · p. 8
  • CoPhyNet generalizes to unseen confounder combinations and block counts, unseen numbers of balls, and sphere-to-cylinder object changes.In CollisionCF, it also estimates object properties when objects are moving or initially stable.Source: §5 · p. 9

Impact of the confounder estimate

  • Confounder supervision changes performance marginally (0.004 for K=3; 0.020 for K=4), while directly supplying confounder quantities performs better but is not comparable.Table 6 also reports stability accuracy per block and distinguishes soft upper bounds using ground-truth confounders.Source: §5 · p. 9, Table 6 · p. 10

Model architecture

  • For K=4, replacing the GCN with an MLP increases MSE by 0.286, while predicting stability per object at each time step outperforms sequence-level stability prediction.Autoregressive, step-by-step prediction is more effective than predicting the whole sequence at once.Source: §5 · p. 9

Confounder estimation

  • A linear classifier on frozen latent representations predicts mass and friction better than random, and stability prediction using confounder estimates plus frame C beats the stated input-only baselines.The stability baselines use either C alone or the sequence (A, B).Source: §5 · p. 10

6 CONCLUSION

The paper formulates counterfactual physical reasoning from images, introduces a benchmark and neural approach, and reports performance gains over conventional forecasting. It also relates learned latent representations to ground-truth confounders and reports human performance on the task.

6 CONCLUSION

  • The task predicts alternative physical outcomes from the original past and outcome plus an alternative past created by intervention.The benchmark is described as challenging because the alternative future depends on confounders unobservable from a single image of the alternative past.Source: §6 · p. 10
  • The neural model is trained with supervision for the do-operator but without confounder supervision.Source: §6 · p. 10
  • Experiments show that counterfactual prediction outperforms conventional forecasting and that latent representations relate to ground-truth confounder quantities.The paper also reports human performance and describes the task as challenging.Source: §6 · p. 10
  • The authors propose counterfactual reasoning as a possible component of model-based reinforcement learning and conjecture it may help disentangle representations and infer causal relationships.Source: §6 · p. 10

A.1 NEURAL DE-RENDERING

The de-rendering module detects objects and regresses their 3D positions from images, with a separate output head for each object.

A.1 NEURAL DE-RENDERING

  • A CNN takes a 224×224 RGB image and detects K blocks, estimating each block’s 3D position as its object representation.Source: §A.1 · p. 13
  • The CNN uses three double-convolution modules with 64, 128, and 256 channels, separated by 2 × 2 max pooling.Each module repeats a convolution, batch normalization, and ReLU stack twice; the resulting feature maps are 256×56×56.Source: §A.1 · p. 13
  • The final feature maps are split into K object-specific heads, each producing a 1×56×56 map that is transformed into a position vector.Source: §A.1 · p. 13
  • The de-rendering module is pretrained on images sampled from A, B, C, and D using supervised mean squared error against ground-truth 3D positions.The remaining model is trained end-to-end.Source: §A.1 · pp. 13–14

A.2 OTHER ARCHITECTURES

The other networks comprise MLPs, GRUs, and a fully connected layer for predicting object confounders.

A.2 OTHER ARCHITECTURES

  • The functions f and g are implemented as MLPs with 4 and 2 layers, respectively, and hidden layers of size 32.Source: §A.2 · p. 14
  • The functions φ and ψ are implemented as two-layer GRUs with hidden state dimension 32.Source: §A.2 · p. 14
  • Mass and friction coefficients are predicted by a single fully connected layer on each object’s confounder representation u_k.Source: §A.2 · p. 14

A.3 FEEDFORWARD BASELINES

IN and NPE are feedforward baselines that predict each object's next position from ground-truth object positions using a Graph Convolution Network.

A.3 FEEDFORWARD BASELINES

  • Both IN and NPE use ground-truth object positions and a Graph Convolution Network to predict each object's next position.Source: §A.3 · p. 14
  • IN models pairwise interactions between all objects in the scene.Source: §A.3 · p. 14
  • NPE estimates object interactions using only neighboring objects.Source: §A.3 · p. 14

A.4 TRAINING FROM ESTIMATED POSITIONS

Training CoPhyNet with ground-truth positions gives slightly better performance than training with estimated positions, while testing uses de-rendered position estimates in both versions.

A.4 TRAINING FROM ESTIMATED POSITIONS

  • Both CoPhyNet versions use de-rendered position estimates at test time, even though one is trained with ground-truth positions.Source: §A.4 · p. 14
  • Training with ground-truth positions gives slightly better performance than training from estimated positions.Both versions are tested using positions estimated by the de-rendering module.Source: §A.4 · p. 14, Table 7 · p. 14
  • Table 7 reports BlocktowerCF 3D-pose MSE averaged over time for ground-truth versus estimated-position training, including a 50/50 split of unseen test confounder configurations.Source: Table 7 · p. 14, §A.4 · p. 14

Related papers

Loading 1909.12000v2…