PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly supervised causal representation learning

Johann Brehmer, Pim de Haan, Phillip Lippe, Taco Cohen

arXiv:2203.16437v3stat.MLcs.LG

TL;DR

Causal variables and their structure cannot generally be recovered from pixels using observational data alone. The paper proves identifiability from paired samples around random unknown interventions, introduces implicit latent causal models, and demonstrates reliable disentanglement and structure recovery on synthetic image data, while relying on restrictive assumptions.

  • Problem

    Observational low-level data such as pixels do not uniquely identify high-level causal variables and their causal structure.

  • Method

    The paper uses paired pre- and post-intervention observations and introduces implicit latent causal models that represent causal variables and structure without an explicit discrete graph.

  • Results

    Under stated assumptions, causal variables and structure are identifiable, and implicit latent causal models reliably disentangle causal variables and recover causal structure from pixels.

  • Takeaways & Limitations

    Weakly supervised paired interventions can support causal representation learning from unstructured pixel data, including in a simulated robotic manipulation dataset.

  • Takeaways & Limitations

    The result assumes stochastic perfect interventions, observation of all atomic interventions, real-valued causal variables, and an exact pre/post-intervention setting that may not match temporal data.

Abstract

from arXiv · show

Learning high-level causal representations together with a causal model from unstructured low-level data such as pixels is impossible from observational data alone. We prove under mild assumptions that this representation is however identifiable in a weakly supervised setting. This involves a dataset with paired samples before and after random, unknown interventions, but no further labels. We then introduce implicit latent causal models, variational autoencoders that represent causal variables and causal structure without having to optimize an explicit discrete graph structure. On simple image data, including a novel dataset of simulated robotic manipulation, we demonstrate that such models can reliably identify the causal structure and disentangle causal variables.

1 Introduction

The paper addresses the impossibility of identifying causal variables and structure from observational pixels alone by using paired pre- and post-intervention samples. It proves identifiability under assumptions and introduces implicit latent causal models for practical learning from pixels.

  • Observational low-level data cannot uniquely identify causal variables and their causal structure without prior assumptions or supervision.
  • Paired samples before and after random, unknown interventions provide weak supervision without intervention-target labels or active intervention control.
  • Under assumptions including stochastic perfect interventions and observed interventions, latent causal models are identifiable up to relabeling and elementwise reparameterizations.
  • Explicit latent causal models use a VAE with an SCM-based prior, but joint graph-and-variable learning creates a difficult chicken-and-egg optimization problem.
  • Implicit latent causal models avoid explicit discrete graph optimization and reliably recover causal variables and structure on synthetic image datasets, including CausalCircuit.

2 Related work

Prior work studies weakly supervised disentanglement, changing environments, known graphs, labelled interventions, or SCM-prior VAEs. This paper targets identifiability for arbitrary unknown causal graphs in the same weakly supervised setting.

  • Locatello et al. study weak supervision for disentangled representations with independent factors and a trivial causal graph.
  • Other approaches use multiple environments, known causal graphs, labelled interventions, temporally resolved effects, or labelled causal variables.
  • The paper claims the first identifiability guarantees for arbitrary, unknown causal graphs under this weakly supervised setting.

3 Identifiability of latent causal models from weak supervision

The paper defines latent causal models and paired pre/post-intervention generation, then proves that their causal variables and mechanisms are identifiable from the resulting pixel distribution under stated assumptions.

  • Setup: The weakly supervised process preserves non-target noise between paired samples while replacing targeted mechanisms and decoding both latent states into observations.
  • Setup: A latent causal model combines an SCM, observation space, decoder, intervention set, and intervention distribution.
  • Setup: LCM equivalence permits a permutation of causal variables and elementwise diffeomorphic reparameterizations, while preserving compatible causal and intervention structure.
  • Identifiability result: The identifiability theorem applies to real-valued variables with all atomic perfect interventions and full-support intervention distributions.
  • Identifiability result: Equal weakly supervised pixel distributions are equivalent to LCM equivalence under the theorem’s assumptions.
  • Identifiability result: The proof identifies intervention targets from mixture components, then uses independence under perfect interventions to establish elementwise transformations and causal graph compatibility.

4 Practical latent causal models

The paper develops two neural implementations of latent causal models: explicit models that parameterize the graph directly and implicit models that encode causal structure through neural solution functions. ILCMs avoid explicit graph optimization while supporting causal representation learning, intervention inference, and causal discovery.

  • Practical latent causal models: Maximum-likelihood training can recover the ground-truth latent causal model up to the identifiability guarantees when data and optimization are sufficient.The guarantee applies to paired pre- and post-intervention observations under the paper’s weakly supervised setting.
  • Explicit latent causal models: ELCMs use a VAE whose latent variables are causal variables and whose prior contains a learnable causal graph and causal mechanisms.The graph can be learned through exhaustive DAG search or differentiable DAG parameterization.
  • Explicit latent causal models: Optimally trained ELCMs identify causal structure and disentangle causal variables on simple datasets, but joint graph-and-representation learning has local minima corresponding to incorrect graphs.This optimization difficulty motivates the implicit model class.
  • Implicit latent causal models: ILCMs represent causal structure through neural solution functions that map noise variables to causal variables, eliminating the need for an explicit graph parameterization.Under diffeomorphic causal mechanisms, the solution function contains the same information as the causal graph and mechanisms.
  • Implicit latent causal models: In ILCMs, unchanged noise components identify non-target variables, while changed components identify intervention targets between paired observations.The encoder enforces equality for non-intervention components, and intervention probabilities increase with component-wise noise changes.
  • Implicit latent causal models: ILCMs use neural encoders, decoders, intervention-target inference, and solution functions to support causal representation learning, intervention inference, and causal graph recovery without explicitly modeling a graph.The learned transformations implicitly describe both the causal graph and causal mechanisms.

5 Experiments

Experiments evaluate implicit latent causal models on toy, pixel-level, robotic-circuit, and scaling datasets. Across these settings, ILCMs disentangle causal variables, infer interventions, and recover causal graphs, with performance degrading beyond roughly 10 causal variables and for discrete causal states.

  • Experimental setup: ILCMs are evaluated with ENCO and a heuristic algorithm for extracting learned causal graphs, against disentanglement, unstructured β-VAE, and slot-attention baselines.The disentanglement baseline assumes independent factors rather than non-trivial causal structure.
  • 2D toy experiment: In the 2D toy experiment, ILCMs disentangle causal factors almost perfectly, infer intervention targets with high accuracy, and recover the correct graph with both graph-extraction methods.The learned representation is identifiable up to elementwise reparameterizations, while the disentanglement baseline induces a wrong graph.
  • Causal3DIdent: On Causal3DIdent, ILCMs reliably disentangle pixel-level causal variables, infer interventions with almost perfect accuracy, and learn causal graphs accurately across six graph variants.The acausal dVAE-E baseline usually fails to recover the correct causal graphs.
  • CausalCircuit: On the CausalCircuit robot-arm dataset, ILCMs outperform acausal baselines in disentanglement, identify interventions with high accuracy, and correctly learn the causal graph.The dataset models a robot arm interacting with touch-sensitive lights connected by a stochastic circuit.
  • Scaling and limitations: ILCMs disentangle causal variables robustly in simple systems with up to approximately 10 causal variables, while larger systems show worse disentanglement and graph accuracy.The method also has difficulty disentangling discrete causal states because they violate the identifiability assumptions; it works reliably when variables are continuous.

6 Discussion

The paper argues that causal variables and their structure can be identified from low-level observations paired before and after random, unlabeled interventions. It introduces ILCMs as a practical alternative to explicit graph parameterization, while noting substantial assumptions and limited experimental scope.

  • Identifiability: Weak supervision can identify causal variables and causal structure from low-level representations when observations precede and follow random, unlabeled interventions.The result extends disentangled representation learning from trivial causal graphs to arbitrary causal graphs.
  • Experiments: The experiments compare ILCMs with disentanglement VAE, β-VAE, and slot attention using DCI disentanglement, intervention-inference accuracy, and graph structural Hamming distance.Table 1 reports these metrics for ILCM-E and ILCM-H alongside the baselines.
  • Methods: ILCMs parameterize causal structure without an explicit discrete graph, making gradient-based optimization easier than jointly learning variables and a graph.The learned graph and causal mechanisms can be extracted after training.
  • Experiments: ILCMs reliably disentangled causal factors, identified causal graphs, and inferred interventions from unstructured pixel data in initial experiments.The experiments included the CausalCircuit dataset of a robot arm interacting with connected switches and lights.
  • Limitations: The theory assumes stochastic and perfect interventions, observation of all atomic interventions, and real-valued causal variables, while the implementation remains restricted to simplified datasets with few continuous variables.Performance decreased quickly when the practical limitations were relaxed, and realistic temporal data may not exactly match the assumed before-and-after setting.

Checklist

The checklist records that the authors reported theoretical assumptions, proofs, training details, and limitations, while code and data were not yet available and asset licensing was not fully documented.

  • Transparency: The authors state that they described their work’s limitations and potential negative societal impacts.They refer to the theory, experiment, conclusions, Appendix B, and Appendix F for these discussions.
  • Theory: The authors report that the full assumptions of theoretical results were stated and that the identifiability theorem was proven in Appendix A.The checklist points to Section 5 and Appendix A and B for the assumptions.
  • Reproducibility: Training details were provided in Appendix D, but reproducibility materials were not yet available.The authors said they had not included the code, data, and instructions needed to reproduce the main experiments.
  • Assets: The paper uses the Causal3DIdent dataset and reports citing its creators, but does not report the asset license.The new CausalCircuit dataset was intended for later publication.
  • Data ethics: The data are synthetic simulated objects, so the checklist marks human consent, participant risks, and compensation as not applicable.The checklist also marks personally identifiable or offensive content as not applicable.

Appendices

The appendices formalize SCMs, interventions, LCM equivalence, generative distributions, and the solution function, then define the objects needed for the identifiability proof.

  • SCM definitions: An SCM combines causal and noise variables, a directed acyclic graph, causal mechanisms, and a factorized full-support noise distribution.Each mechanism is assumed to be a diffeomorphism from its noise variable to the corresponding causal variable when parents are fixed.
  • Formal equivalence: A ψ-diagonal map permutes variables while each output component depends elementwise on one corresponding input component.This notion formalizes equality up to permutation and elementwise reparameterization.
  • Interventions: An ISCM augments an acyclic SCM with interventions that replace selected mechanisms and a probability distribution over intervention choices.Atomic interventions target one variable or none.
  • Equivalence: LCM equivalence requires a graph isomorphism and elementwise diffeomorphisms that preserve causal mechanisms, noise distributions, decoders, intervention sets, and intervention distributions.The decoder must respect the induced ISCM isomorphism.
  • Generative process: The solution function maps noise variables to causal variables by repeatedly applying the causal mechanisms, and interventions produce modified solution functions.The LCM generative process maps noise to causal variables and then through a decoder to observations, with analogous intervened samples.

A.2 Identifiability proof

The identifiability proof shows that equality of weakly supervised pixel distributions is equivalent to LCM equivalence under real-valued variables and complete atomic interventions. Its steps recover intervention correspondence, coordinatewise reparameterization, graph ancestry, and compatible mechanisms.

  • Auxiliary lemmas: A differentiable measure-preserving map on [0,1] is either the identity or reflection, providing the key one-dimensional rigidity lemma.The proof derives this by showing the map is surjective, has nonzero derivative, and is monotone.
  • Theorem: Under real-valued variables and full-support empty and atomic interventions, equal weakly supervised distributions are equivalent to equivalent LCMs.The theorem assumes both models have n real-valued endogenous variables and intervention sets containing the empty and all atomic interventions.
  • Proof steps: Equality of intervention-conditioned distributions induces a permutation matching interventions between the two models.The proof uses the non-overlapping supports of distributions associated with distinct atomic interventions.
  • Proof steps: The commuting stochastic-map diagrams force the latent-variable diffeomorphism to be ψ-diagonal.The argument shows each transformed coordinate is constant in the other latent variables.
  • Proof steps: The recovered permutation preserves ancestry and therefore parenthood, yielding a graph isomorphism and compatible causal mechanisms.The final construction also preserves intervention distributions, completing the ISCM and LCM isomorphisms.

B Limitations & generalization

The identifiability result depends on strong assumptions about variable domains, intervention coverage and type, decodability, and observed causal influence. Counterexamples show that relaxing some assumptions can preserve weakly supervised observables while changing causal structure.

  • All interventions observed: Identifiability requires interventions with support for every atomic intervention, although partial identifiability may remain when some variables cannot be intervened upon.Interventions on child variables can, for example, guarantee identifiability of their parents.
  • Perfect interventions: The proof requires perfect interventions whose mechanisms do not depend on causal variables; allowing parent-dependent intervention mechanisms destroys identifiability.A transformation φ2(z1, z2) = z1 + z2 can preserve the weakly supervised distribution without being an ISCM isomorphism.
  • Diffeomorphic decoder: The theorem assumes a deterministic diffeomorphic decoder, whereas the practical VAE uses a stochastic decoder and noisy data, supported empirically rather than theoretically.An extension of the theorem to stochastic decoders is left for future work.
  • Observed causal influence: All causal variables must influence the observed low-level data; variables absent from those observations cannot be detected, and latent confounders remain unresolved.The requirement applies regardless of whether the decoder is deterministic or stochastic.
  • Real-valued causal variables: Identifiability assumes real-valued causal and noise variables; extensions to arbitrary continuous spaces or multidimensional domains are not straightforward.Circle-valued variables yield non-identifiable latent causal models with non-isomorphic graphs and identical weakly supervised observable distributions.
  • Proof mechanism: For real-valued variables, the key ISCM property is that unintervened noise encodings remain equal before and after intervention, enabling causal-variable identification.Lemma 3 states that for every non-intervened variable, ei = ˜ei almost surely.

C.2 Model specification

The ILCM represents causal variables and structure implicitly through neural components, avoiding an explicit discrete graph during optimization. Its staged training and post-hoc graph extraction address optimization difficulty while preserving causal-model identifiability.

  • Model components: ILCMs combine neural noise encoders and decoders, intervention inference, invertible solution functions, and Gaussian noise and intervention priors.The model also uses a uniform intervention-target prior.
  • Training procedure: Training encodes paired observations, infers intervention targets, and mixes pre- and post-intervention noise representations so non-intervened components remain equal.This construction enforces consistency between the two members of each weakly supervised pair.
  • Training procedure: Four training phases progressively pretrain the autoencoder, simplify intervention modeling, activate solution functions, and fine-tune the model.Separating simplified-prior training from solution-function training substantially improves success, while pretraining and fine-tuning provide smaller efficiency gains.
  • Structural refinement: ILCM solution functions can be restricted to ancestors after topological ordering, adding inductive bias for cleaner disentangled representations.The restriction is implemented with a masking layer in the neural networks.
  • Identifiability: ILCMs are theoretically equivalent to ELCMs because the mapping preserves the weakly supervised distribution, extending the identifiability result to implicit models.The correspondence relates explicit graph edges to dependencies induced by the ILCM solution function.
  • Graph inference: The heuristic graph-inference method is computationally cheaper than likelihood-based ENCO, but ENCO found the correct graph on all eight datasets versus seven for the heuristic.The heuristic determines order from functional sensitivity, extracts mechanisms in topological order, and tests parent dependence.

D.3 Causal3DIdent experiments

The Causal3DIdent experiments evaluate ILCMs across multiple synthetic causal graphs and rendered image datasets. They assess representation quality, intervention inference, graph recovery, and disentanglement using several baselines and visual analyses.

  • Dataset: The benchmark contains six datasets generated from different causal graphs, structural causal models, and decoders.The considered structures include trivial, collider, full, single-edge, chain, and fork graphs.
  • Evaluation: Evaluation reports DCI disentanglement, completeness, informativeness, intervention-inference accuracy, learned graphs, and structural Hamming distance.The comparison includes ILCM-E, ILCM-H, dVAE-E, an unstructured β-VAE, and slot attention.
  • Dataset: The datasets map three causal variables to object hue, spotlight hue, and spotlight position before rendering 64 × 64 Blender images.Each variation uses nonlinear SCMs with heteroskedastic noise and different graph structures.
  • Evaluation: The experiments additionally visualize intervention inference and the disentanglement properties of learned representations.The intervention analysis combines inferred targets with pre- and post-intervention noise encodings before decoding manipulated samples.

D.4 CausalCircuit experiments

The CausalCircuit experiments test ILCMs on a simulated robotic manipulation environment and examine disentanglement, graph scaling, and comparisons with explicit and baseline models. ILCMs separate causal factors successfully on the robot-arm dataset and remain reliable for systems of roughly ten variables.

  • CausalCircuit dataset: CausalCircuit is a MuJoCo dataset with four causal variables: red, green, and blue lights plus a robot-arm position over three buttons.Button presses depend on the arm’s distance from button centers and influence the light variables.
  • Experimental setup: The CausalCircuit images are compressed to a 16-dimensional representation before training most models, while slot attention uses 64 × 64 inputs instead of the original 512 × 512 resolution.These preprocessing choices were made to facilitate training and fast experimentation.
  • Disentanglement: ILCM successfully disentangles the causal factors in CausalCircuit, whereas the dVAE baseline fails and slot attention does not assign causal variables to separate slots.The authors attribute the slot-attention failure to the lights blending together.
  • Graph scaling: For systems with up to around 10 causal variables, ILCMs reliably disentangle variables and produce more accurate graphs than the baseline.Scaling ILCMs to larger systems requires additional research.
  • ELCM comparison: Explicit latent causal models use causal latent variables, a decoder, and a prior encoding graph structure, with graph learning implemented through explicit parameterizations.The ELCM setup can infer intervention targets or marginalize over them.

E.2 Experiments

The ELCM experiments show mixed performance and strong sensitivity to initialization, making ELCM training less robust than the ILCM experiments. Common failures preserve parts of the graph while misorienting causal effects.

  • Caveat: The ELCM metrics are not directly comparable to the main ILCM results because the experiments use slightly different Causal3DIdent datasets.This scope boundary applies to the table’s reported comparisons.
  • Results: ELCM results vary across datasets: some recover the graph and disentangle variables, while others fail at both tasks.The reported table includes learned graphs, SHD, and DCI disentanglement scores.
  • Results: ELCM performance strongly depends on random initialization, with successful and failed disentanglement often appearing among different seeds.The reported median over three runs can conceal this seed-level variability.
  • Comparison: ELCM training is substantially less robust across random seeds than the ILCM experiments.The comparison is based on the stability of disentanglement results across seeds.
  • Failure mode: A common ELCM failure retains a partly disentangled representation and the correct graph skeleton but reverses some causal directions.These incorrectly oriented graphs can remain in the same Markov equivalence class as the correct graph.
Loading 2203.16437v3…