PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Discovery in Physical Systems from Videos

Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, Animesh Garg

arXiv:2007.00631v3cs.LGcs.CVstat.ML

TL;DR

Images can reveal objects while leaving their causal interactions and governing mechanisms latent. The paper learns keypoint representations, infers interaction graphs, and predicts dynamics from video without graph supervision. It reports stronger multi-body results than a no-inference baseline and fabric-specific graphs that reflect connectivity.

  • Problem

    Images may show object identity and position while leaving interaction structure and edge mechanisms latent, motivating causal discovery from visual data.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    V-CDN extracts keypoints, infers hidden interaction variables between them, and conditions dynamics prediction on the inferred graph in an unsupervised approach.

    Source: §2.1 · p. 4, §3.2 · p. 6, §5 · p. 9

  • Results

    The model significantly outperforms a no-inference baseline on multi-body interactions and infers fabric-specific graphs that reflect underlying connectivity.

    Source: §3.2 · p. 8

  • Takeaways & Limitations

    The authors aim to use short image sequences to recover causal summary graphs and make counterfactual predictions into the future.

    Source: §2 · p. 3

    The work focuses on dynamical-system modeling and leaves sophisticated causal dependencies and practical sampling-rate issues unaddressed.

    Source: Broader Impact · p. 10

Abstract

from arXiv · show

Causal discovery is at the core of human cognition. It enables us to reason about the environment and make counterfactual predictions about unseen scenarios that can vastly differ from our previous experiences. We consider the task of causal discovery from videos in an end-to-end fashion without supervision on the ground-truth graph structure. In particular, our goal is to discover the structural dependencies among environmental and object variables: inferring the type and strength of interactions that have a causal effect on the behavior of the dynamical system. Our model consists of (a) a perception module that extracts a semantically meaningful and temporally consistent keypoint representation from images, (b) an inference module for determining the graph distribution induced by the detected keypoints, and (c) a dynamics module that can predict the future by conditioning on the inferred graph. We assume access to different configurations and environmental conditions, i.e., data from unknown interventions on the underlying system; thus, we can hope to discover the correct underlying causal graph without explicit interventions. We evaluate our method in a planar multi-body interaction environment and scenarios involving fabrics of different shapes like shirts and pants. Experiments demonstrate that our model can correctly identify the interactions from a short sequence of images and make long-term future predictions. The causal structure assumed by the model also allows it to make counterfactual predictions and extrapolate to systems of unseen interaction graphs or graphs of various sizes.

1 Introduction

The introduction frames visual causal discovery as recovering latent interaction structure from images when variables, edge mechanisms, and configurations are hidden. It presents an unsupervised keypoint-to-graph-to-dynamics approach and reports generalization claims across interactions and fabrics.

1 Introduction

  • Images may reveal ball identity and position while leaving edge types, parameters, and the causal structure governing interactions latent across configurations.The paper notes that implicit image measurements may obscure the underlying structure and governing mechanism [4].Source: §1 · p. 1
  • Image-to-future associative models for collision-only, disconnected entities are insufficient for latent edge mechanisms, novel generative structures, and counterfactual prediction.The introduction contrasts these models with more complex systems involving different edge types and mechanisms.Source: §1 · pp. 1–2
  • The task is to recover an SCM from images without known node variables, graph structure, hidden confounders, or interaction dynamics, using episodes with different graphs but shared dynamics.Source: §1 · p. 2
  • The work targets one-shot discovery of unseen causal mechanisms in new environments from partially observed visual data, jointly estimating model class and parameters for prediction.The authors frame this as a meta-learning approach for new latent structures at test time.Source: §1 · p. 2
  • V-CDN uses keypoints for node discovery, infers pairwise interaction variables with graph neural networks, and predicts future keypoint motion conditioned on the inferred graph.Different configurations and scenarios provide data from unknown interventions, which the authors use to seek graph recovery without explicit interventions.Source: §1 · p. 2
  • The model is reported to handle noisy, varying-complexity multi-body interactions, counterfactuals, unseen interaction graphs, variable object counts, and fabrics of different types and shapes.In fabric settings, it identifies dependency structure and models dynamics from reduced-order keypoints.Source: §1 · p. 2

2 Visual Causal Discovery in Physical Systems: V-CDN

V-CDN seeks to recover causal structure from image sequences without ground-truth graphs, using keypoints, graph inference, and graph-conditioned dynamics. Its model represents interactions with directed edges and hidden confounders, then uses the inferred graph to predict future states.

2 Visual Causal Discovery in Physical Systems: V-CDN

  • The causal summary graph has time-invariant relations with discrete interaction types and continuous parameters; it may contain cycles, while the full time graph is acyclic.The model assumes no instantaneous edges or edges that go back in time.Source: §2 · p. 3

2.1 Unsupervised keypoint detection from videos

  • Unsupervised keypoint detection uses image reconstruction to produce temporally consistent foreground points that represent system components.The detector is trained with a pixel-wise L2 reconstruction loss.Source: §2.1 · p. 4

2.2 Graph neural networks as the spatial encoder

  • A graph neural network encodes keypoint nodes and directed relations into object- and relation-centric embeddings for inference and dynamics.Edges point from keypoint j to keypoint i and carry associated attributes.Source: §2.2 · p. 4

2.3 Inferring the directed edge set of the Causal Summary Graph

  • The inference module uses keypoint sequences to estimate edge-type distributions and continuous edge parameters, forming a posterior over causal summary graphs.The first edge type denotes a null edge; Gumbel-Softmax supports end-to-end training through discrete edge sampling.Source: §2.3 · pp. 4–5

2.4 Future prediction using the forward dynamics module

  • The dynamics module predicts future keypoint positions conditioned on the current state and inferred graph, representing each next-state position with a multivariate Gaussian.Repeated application of the dynamics module yields long-term predictions.Source: §2 · p. 3, §2.4 · p. 5

2.5 Optimizing the model

  • The perception module is trained independently, while inference and dynamics are trained jointly end-to-end to maximize an ELBO-analogous objective.The graph prior factorizes across edges, and the objective can include a regularizer such as one encouraging sparsity.Source: §2 · p. 4, §2.5 · p. 5

3 Experiments

Across multi-body and fabric experiments, V-CDN discovers interaction structure from keypoints and uses it for dynamics prediction. Results include improved edge inference with more observed frames, generalization across graph sizes and fabric structures, and counterfactual prediction beyond training parameter ranges.

3.1 Results on unsupervised keypoint detection

  • The perception module spreads keypoints over the image foreground and consistently tracks objects.The experiments use multi-body systems with invisible physical constraints and fabrics of varying types and shapes.Source: §3.1 · p. 6, §3 · p. 6

3 Experiments

  • The inference module uses short keypoint sequences to identify physical connections and hidden edge variables without supervision on edge attributes.The experiments assess one-shot graph and confounder discovery, extrapolation to graph sizes absent from training, and counterfactual prediction.Source: §3.2 · p. 6, §3 · p. 6

3.2 Discovery of the Causal Summary Graph and the hidden confounders

  • The model’s edge-type accuracy rises and uncertainty falls as it observes more frames, and it significantly outperforms a baseline without the inference module.The inferred continuous variable also correlates strongly with the ground-truth hidden confounder.Source: Figure 4 · p. 7, §3.2 · p. 8
  • For different fabric types, the inference module produces different graphs that reflect their underlying connectivity patterns.Fabric keypoints serve as a reduced-order representation, although the ground-truth causal summary graph is unknown.Source: §3.2 · p. 8

3.3 Extrapolation to unseen causal graphs of different sizes

  • The inference and dynamics modules generalize from five-body training to systems with 3, 4, 6, or 7 bodies without retraining.Performance is better in the untrained three- and four-ball environments; separate perception modules are trained to reflect the number of moving components.Source: §3.3 · p. 8, Figure 6 · p. 8

3.4 Counterfactual prediction and extrapolation on parameter change

  • Counterfactual predictions match simulator futures under the same intervention at T + 30 and extrapolate to parameter ranges outside training.The evaluation intervenes on estimated hidden confounders and applies the corresponding intervention to the ground-truth simulator.Source: §3.4 · pp. 8–9, Figure 7 · p. 9

4 Related Work

The related work surveys causal discovery, relational models of dynamics, and video prediction. It positions this paper as combining image-based causal structure inference with discrete and continuous hidden-confounder modeling.

Causal Discovery

  • Observational causal discovery includes constraint-based, score-based, and asymmetry-based approaches; methods using observational and interventional data may assume known interventions, restrict model classes, or face scalability limits.The cited examples include PC and FCI, GES, and asymmetry-based methods [20] [21].Source: §4 · p. 9

Relational Neural Models

  • Graph and attention models of multi-body dynamics often assume a stationary latent causal graph, while recent relation-inference methods still differ from this work’s image input and continuous confounder modeling.The passage identifies and as variational or meta-learning approaches to entity relationships.Source: §4 · p. 9

Dynamics from Videos

  • Video prediction methods use latent embeddings or keypoints, with unsupervised keypoint learning offering an alternative to tedious domain-specific labels.The paper characterizes latent embeddings as potentially uninterpretable and poorly generalizing, and points to reconstruction or view-consistency losses for keypoint learning [55].Source: §4 · p. 9

5 Conclusion

The paper presents end-to-end unsupervised visual causal discovery: keypoints represent videos, causal relationships are inferred, and future dynamics are predicted without ground-truth graphs, confounders, or dynamics.

5 Conclusion

  • The method learns dependency structures and causal mechanisms from images without ground-truth graphs, hidden confounders, or dynamics.It extracts structured keypoints, identifies relationships among components, and predicts into the future.Source: §5 · p. 9
  • The approach is framed as a step toward more generalizable visual reasoning systems.Source: §5 · p. 9
  • The pipeline combines structured keypoint representations, causal relationship identification, and future prediction.Source: §5 · p. 9

Broader Impact

Learning only observational associations can produce dataset-biased models that generalize poorly beyond training data. The authors motivate causal modeling using varied configurations under unknown interventions, while noting that their work leaves sophisticated dependencies and sampling-rate issues unaddressed.

Broader Impact

  • Observation-only learning can encode dataset bias, limiting out-of-distribution generalization and risking catastrophic outcomes in real-world deployment.Source: Broader Impact · p. 10
  • Because randomized trials may be impractical, impossible, or unethical, the paper studies causal discovery from visual data under unknown interventions.Source: Broader Impact · p. 10
  • The authors report that causal modeling improved generalization outside the training distribution and toward high-likelihood counterfactual data augmentation.Source: Broader Impact · p. 10
  • The work focuses on dynamical-system modeling and leaves sophisticated causal dependencies and practical sampling-rate issues unaddressed.Source: Broader Impact · p. 10

A.1 Unsupervised keypoint detection from videos

The appendix describes unsupervised keypoint detection trained through image reconstruction and a graph neural network that encodes spatial relations among keypoints. The graph network produces object- and relation-level embeddings for use in inference and dynamics modules.

A.1 Unsupervised keypoint detection from videos

  • Unsupervised keypoint detection builds on, using source-to-target feature transport and pixel-wise reconstruction loss to spread keypoints across the foreground.The detector and refiner are optimized to reconstruct the target image from transported features, with heatmaps placing fixed-variance Gaussians around detected points.Source: §A.1 · p. 14

A.2 Graph neural networks as the spatial encoder

  • The graph neural network represents keypoints as directed nodes and relations as attributed edges, then propagates information to produce node and edge embeddings.Its object and relation encoders form embeddings from each node and incoming neighbors; repeated propagation can improve expressiveness.Source: §A.2 · p. 14

B.1 Multi-Body Interaction

The experiments use simulated multi-body interactions with varied pairwise relations and fabric manipulation across several shapes. The setups reserve episodes for testing and vary relational or geometric conditions.

B.1 Multi-Body Interaction

  • The multi-body dataset contains 5,000 simulated 500-frame episodes, with 250 held out for testing and each ball pair randomly assigned no relation, a rigid rod, or a spring.Spring stiffness is 20 with rest lengths sampled from [20, 120]; rigid links allow a small distance window so the model must infer relation length.Source: §B.1 · pp. 14–15

B.2 Fabric Manipulation

  • The fabric dataset contains 2,000 300-frame episodes, with 200 held out for testing, covering shirts, pants, and towels with varied dimensions.Forces applied along fabric contours deform and move the fabrics.Source: §B.2 · p. 15

C Implementation details

The implementation uses PyTorch and a single TITAN Xp GPU per model instance. The appendix specifies keypoint-network training, temporal encoders, and joint optimization settings.

C Implementation details

C.1 Unsupervised keypoint detection

  • The keypoint detector outputs N confidence maps and uses spatial softmax for locations, while a transpose-convolution refiner reconstructs target images.The reconstruction loss is optimized with Adam at learning rate 0.001 for about 240k iterations.Source: §C.1 · p. 15

C.2 Predicting the directed edge set using the inference module

  • Object and relation encoders use MLPs, while three-block temporal CNNs and max pooling produce fixed-dimensional features for variable-length sequences.Source: §C.2 · p. 15

C.3 Joint optimization of the inference module and the dynamics module

  • The inference module is trained by optimizing the Section 2.5 loss with Adam for about 300k iterations, at learning rates 0.0001 in Multi-Body and 0.0005 in Fabrics.The paper refers readers to released code for the exact network architecture and individual-module training procedures.Source: §C.3 · p. 15

D.1 Unsupervised keypoint detection

In the Fabric environment, temporally consistent keypoints track deformable objects, and the causal-discovery model makes more accurate long-term predictions than a baseline without causal inference.

D.1 Unsupervised keypoint detection

  • The keypoint bottleneck and downstream reconstruction yield temporally consistent foreground keypoints that track object movement and handle deformable objects.Qualitative examples cover both Multi-Body and Fabric environments.Source: §D.1 · p. 15
  • Figure 8 presents input images, keypoint overlays, keypoint-spanned heatmaps, and reconstructed target images in successive rows.The first row shows inputs, the second overlays, and the third and fourth intermediate results.Source: Figure 8 · p. 16

D.2 Future prediction in the Fabric environment

  • The causal-discovery model makes more accurate future predictions in the Fabric environment than the otherwise-matched baseline without an inference module.Figure 9 compares the two models; the baseline differs by omitting causal discovery.Source: Figure 9 · p. 17, §D.2 · p. 17
Loading 2007.00631v3…