PaperlayerView exact revision on arXiv

Source-linked AI summary

CITRIS: Causal Identifiability from Temporal Intervened Sequences

Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, Efstratios Gavves

arXiv:2202.03169v3cs.LGcs.AIstat.ME

TL;DR

Causal representation learning seeks latent variables from visual observations, but complex systems may require multidimensional factors and interventions that affect only some components. CITRIS uses temporal intervened image sequences, a VAE, and normalizing flows to identify and disentangle these representations. It reliably recovered causal factors in rendered images and generalized to unseen instantiations, within the stated intervention and identifiability limits.

  • Problem

    Causal representation learning must recover scalar and multidimensional latent factors from high-dimensional observations, despite cases where interventions leave some factor components unaffected.

  • Method

    CITRIS uses temporal sequences with observed intervention targets, a VAE for causal-factor assignment, and normalizing flows to disentangle pretrained autoencoder representations.

  • Results

    CITRIS reliably recovered causal factors in 3D rendered images and generalized its causal representation to unseen instantiations of causal factors.

  • Takeaways & Limitations

    The results support using pretrained autoencoders and intervention-guided temporal learning for causal representation learning, including potential simulation-to-real generalization.

  • Takeaways & Limitations

    Identifiability is limited when interventions jointly target variables or leave dimensions of a multidimensional factor unaffected.

Abstract

from arXiv · show

Understanding the latent causal factors of a dynamical system from visual observations is considered a crucial step towards agents reasoning in complex environments. In this paper, we propose CITRIS, a variational autoencoder framework that learns causal representations from temporal sequences of images in which underlying causal factors have possibly been intervened upon. In contrast to the recent literature, CITRIS exploits temporality and observing intervention targets to identify scalar and multidimensional causal factors, such as 3D rotation angles. Furthermore, by introducing a normalizing flow, CITRIS can be easily extended to leverage and disentangle representations obtained by already pretrained autoencoders. Extending previous results on scalar causal factors, we prove identifiability in a more general setting, in which only some components of a causal factor are affected by interventions. In experiments on 3D rendered image sequences, CITRIS outperforms previous methods on recovering the underlying causal variables. Moreover, using pretrained autoencoders, CITRIS can even generalize to unseen instantiations of causal factors, opening future research areas in sim-to-real generalization for causal representation learning.

1. Introduction

CITRIS addresses causal representation learning for scalar and multidimensional factors by using temporal image sequences with observed intervention targets. It combines a VAE with a normalizing-flow extension for disentangling pretrained autoencoder representations.

  • Multidimensional factors such as position and rotation can be more natural than separately estimating every scalar variable in complex environments.
  • TRIS consists of temporal observation sequences in which intervention targets are observed, as in observation and action trajectories.
  • Under assumptions including invertible observations and a stationary causal process, CITRIS identifies minimal causal variables containing information strictly affected by interventions.
  • CITRIS is a VAE that assigns latent variables to causal factors and conditions each latent prior on its respective intervention target to promote disentanglement.
  • The framework identifies multidimensional causal factors up to their minimal causal variables and extends to pretrained autoencoders through normalizing flows.

2. Preliminaries and Causal Assumptions

The paper models latent dynamics as a stationary, first-order Markov dynamic Bayesian network with scalar or vector-valued causal variables. Interventions are represented by target indicators and may be soft, perfect, multiple, or observationally absent.

  • The latent process is a stationary, first-order Markov dynamic Bayesian network whose nodes are scalar or vector-valued causal variables.
  • Each causal factor at time t has parents only among causal factors at time t−1, with a time-invariant graph structure and mutually independent noise terms.
  • The intervention vector I_t is binary and indicates which causal variables are intervened upon at each time step.
  • Intervention components may be confounded by an unobserved regime variable, represented in an augmented directed acyclic graph.
  • The augmented graph factors the joint distribution according to parent sets and supports arbitrary intervention targets, including observational data, soft interventions, and perfect interventions.

3. Identifiability of Minimal Causal Variables

TRIS uses temporal observation sequences with known intervention targets to identify the intervention-affected portions of scalar or multidimensional causal factors. CITRIS formalizes these minimal causal variables and proves their recovery under stated assumptions, while showing when disentanglement is impossible.

  • TempoRal Intervened Sequences (TRIS): Causal factors may be multidimensional, allowing representations such as a single factor for a two-dimensional position rather than separate scalar factors.The framework permits continuous, discrete, or mixed component domains.
  • Identifiability Result: Under the stated assumptions and sufficient latent dimensions, the learning objective identifies the intervention-relevant latent structure while assigning remaining information to a separate latent group.The result assumes an invertible observation function and a stationary causal process, among the setup assumptions.
  • TempoRal Intervened Sequences (TRIS): TRIS provides temporal sequences of high-dimensional observations together with intervention targets, but not intervention values.Observations are noisy, entangled views of the latent causal factors.
  • Necessary Condition for Disentanglement in TRIS: If two factors are always jointly intervened on or never intervened on, they may not be uniquely identified from observations and intervention targets.The ball example shows that rotationally equivalent axes can remain indistinguishable without separately informative interventions.
  • Necessary Condition for Disentanglement in TRIS: When interventions leave some dimensions of a multidimensional factor unaffected, complete reconstruction of that factor is impossible.The identifiable target is therefore the intervention-affected portion rather than necessarily the full factor.
  • Minimal Causal Variables: A minimal causal split separates each factor into intervention-variant and intervention-invariant parts, retaining only information that depends on the intervention in the minimal causal variable.For the two-box example, the box identifier is variant while the within-box relative position is invariant.

4. Causal Identifiability from Temporal Intervened Sequences

CITRIS learns causal representations from temporal observations with known intervention targets, using a VAE architecture and a normalizing-flow extension for pretrained autoencoders. Its design groups latent variables by causal factor, promotes target-specific disentanglement, and supports complex multidimensional latent distributions.

  • 4.1. Variational Autoencoder Setup: CITRIS uses a VAE to identify causal factors from high-dimensional temporal observations with interventions.The encoder and decoder approximate an invertible observation-to-latent mapping.
  • 4.1. Variational Autoencoder Setup: A target classifier predicts intervention targets from latent variables and guides disentanglement using target-specific independence relations.It encourages information about all causal variables to remain encoded while assigning latent variables to causal factors.
  • 4.2. Using Pretrained Autoencoders: CITRIS extends to pretrained autoencoders by freezing the autoencoder and learning an invertible normalizing flow from entangled to disentangled latents.The pretrained decoder can reconstruct observations without fine-tuning.
  • 4.1. Variational Autoencoder Setup: The transition prior conditions each latent variable on one intervention target and enforces independence across causal-variable blocks.An autoregressive prior models complex distributions across multiple dimensions within a causal factor.
  • 4.2. Using Pretrained Autoencoders: The pretrained-autoencoder variant can reduce interventional-data needs, support generalization across datasets, and simplify optimization.The autoencoder can learn from observational data, while the flow focuses on disentanglement.

5. Related Work

Related work spans independent-factor learning and causal representation learning. Prior methods use auxiliary variables or observation pairs, while some causal VAE approaches require true causal-variable labels.

  • Independent Factor Learning: Independent Component Analysis seeks independent latent variables under an invertible transformation, but nonlinear recovery is generally impossible without additional structure.Auxiliary variables can make nonlinear latents conditionally mutually independent.
  • Causal Representation Learning: Causal representation learning aims to discover causal structures and variables from data.Existing approaches include identifying factors from observation pairs and incorporating structural causal models into VAE priors.
  • Causal Representation Learning: Some prior causal VAE methods require the true causal variables as training labels.This requirement distinguishes them from approaches that infer causal representations from observational or intervened sequences.
  • Benchmark Context: Figure 5 represents a temporal causal graph over seven dimensions of variation, including confounders, chains, and variables with up to four parents.The graph is part of the benchmark setting used to evaluate temporal causal representation learning.

6. Experiments

CITRIS is evaluated on temporal 3D-rendered images and Interventional Pong against disentanglement baselines, using correlation and latent-factor recombination tests. It performs well across complex causal factors, pretrained-autoencoder settings, unseen shapes, and imperfect interventions.

  • Experimental setup: CITRIS is evaluated on Temporal-Causal3DIdent and Interventional Pong, with comparisons against SlowVAE and iVAE*.The evaluation uses correlation metrics and parameter-free triplet evaluations to assess disentanglement and decoding.
  • Experimental setup: Temporal-Causal3DIdent contains multidimensional position and rotation factors, while Interventional Pong includes ball, paddle, velocity, and score factors.Pong interventions are imperfect and correlated, and the score is an unintervened causal variable assigned to zψ0.
  • Teapot experiments: CITRIS-VAE and CITRIS-NF achieved average R2 and Spearman correlations above 0.9 on the teapot benchmark while keeping factor correlations low.CITRIS-NF also achieved near-optimal triplet scores and especially improved rotation modeling over VAE-based approaches.
  • Generalization: CITRIS-NF retained strong disentanglement on two unseen object shapes, although position and rotation declined because their representations depend on object shape.The authors report performance well above random despite these generalization limitations.
  • Interventional Pong: CITRIS disentangled Interventional Pong factors and assigned the unintervened score to zψ0 despite imperfect, correlated interventions.This result demonstrates handling of intervention settings that do not directly manipulate every causal factor.

7. Conclusion

The paper introduces CITRIS for identifying minimal causal variables from temporal intervened sequences and extends it with normalizing flows for pretrained autoencoders. Experiments recover causal factors in rendered images and show generalization to unseen factor instantiations.

  • Conclusion: CITRIS identifies minimal causal variables from temporal sequences containing interventions.The framework is presented as a VAE for learning causal representations.
  • Conclusion: Normalizing flows extend CITRIS to disentangle representations learned by pretrained autoencoders.This extension targets causal representation learning from already learned image representations.
  • Conclusion: Experiments reliably recovered causal factors of 3D-rendered images and generalized to unseen instantiations of those factors.The authors identify simulation-to-real generalization as a potential future direction.

A. Reproducibility Statement

The reproducibility statement documents released code, datasets, implementation details, repeated-seed experiments, and computational requirements, alongside the formal modeling assumptions used by the paper.

  • Resources: The authors release code for all models, Interventional Pong generation, and Temporal-Causal3DIdent causal-factor sequences with dataset links and licenses.The repository also includes dataset overviews and visual examples.
  • Implementation details: Experiments use documented hyperparameters, three random seeds, reported standard deviations, and additional results in the appendix.These materials support replication of the reported evaluation.
  • Computational requirements: All experiments ran on one NVIDIA TitanRTX GPU with a 6-core CPU and required approximately 80 GPU days, excluding search and research trials.The statement reports total computation across the paper’s experiments.
  • Model assumptions: The theoretical setup assumes a first-order Markov, stationary dynamic Bayesian network without instantaneous effects, with scalar or vector-valued causal variables.Known binary intervention targets are associated with causal factors at each timestep.
  • Model assumptions: The minimal causal split separates intervention-dependent manipulable information from the remaining invariable information.This split defines which component of each causal factor is included in the minimal causal variable.
  • Entropy formalism: The proof uses limiting density of discrete points because its entropy is invariant under invertible changes of variables.The paper omits an additive log N constant because it compares entropy values rather than requiring an absolute value.

B.2. Proof Outline

The proof outline establishes that the true disentanglement is a global likelihood optimum, characterizes alternative optima, and uses entropy maximization to recover minimal intervention-dependent causal variables under temporal-intervention assumptions.

  • Step 1: The proof first shows that the true disentanglement and assignment function form a global, though not necessarily unique, optimum of the conditional likelihood.This relies on the true causal graph, invertible observation mapping, and change-of-variables arguments.
  • Step 2: Alternative optimal representations may place intervention-independent information in different latent factor sets without changing likelihood.The proof therefore analyzes the full class of disentanglement functions achieving the same optimum.
  • Step 3: The final proof step selects among likelihood-equivalent representations by maximizing the entropy of the latent group assigned to information independent of interventions.This criterion is intended to isolate the minimal causal mechanisms while collecting remaining information separately.
  • Assumptions: The argument assumes a stationary first-order temporal causal process with scalar or vector-valued factors, known intervention targets, and an invertible observation function.The proof also allows sufficient latent dimensions and uses the temporal causal graph to derive the likelihood factorization.
  • Step 2: The model separates each causal factor into intervention-variant and intervention-invariant parts, whose possible split depends on the available interventions.Only the affected part is guaranteed to be identifiable as the minimal causal mechanism.

B.4.2. INDEPENDENT INTERVENTIONS

Under independent interventions, the likelihood identifies the intervention-dependent information of each causal factor, while intervention-independent information remains freely assignable unless an entropy criterion resolves the ambiguity.

  • Setup: The analysis assumes intervention targets are conditionally independent given the previous causal state.This assumption enables the entropy comparisons used to constrain valid latent assignments.
  • No invariant parts: For factors with no intervention-independent part, maximum likelihood requires assigning all factor information to its corresponding latent set.Splitting such information across other causal sets would violate the entropy inequality while preserving the true conditional distributions.
  • Invariant parts: Intervention-independent information can be assigned to any latent factor set without changing the optimal likelihood.This applies both to observational noise and to invariant parts of causal factors.
  • Identifiable content: The identifiable content is the intervention-dependent part of each causal factor, provided no alternative split yields additional intervention-independent information.Noise and invariant parts from other factors may still be assigned arbitrarily.

B.4.3. CONFOUNDED INTERVENTIONS

With confounded intervention targets, the intervention-dependent part of a factor remains identifiable when intervention patterns prevent one target from being a deterministic function of another, but jointly intervened factors can remain entangled.

  • Setup: The confounded-intervention analysis allows intervention targets to be dependent through an unobserved variable while considering single-target and grouped interventions.The proof examines several intervention-pattern cases for pairs of causal variables.
  • Always-joint interventions: If two variables are always intervened on together, their intervention-dependent information cannot generally be disentangled.Multidimensional factors may even distribute information across the two targets without losing likelihood.
  • Boundary: The excluded case is the absence of observed interventions for both variables, because then the intervention-dependent part has no influence on the modeled solution.Thus, identifiability depends on intervention variation rather than only on the causal dynamics.
  • Separating interventions: When intervention patterns include separating cases, the maximum-likelihood solution assigns each factor’s intervention-dependent part to its own latent set.This holds for the analyzed cases involving observational, joint, and single interventions.
  • Conclusion: The resulting condition is that intervention-dependent parts are disentangled when no deterministic function maps one target to the other across all time steps.This is the stated summary condition for confounded interventions.

B.5. Step 3: Deriving the Final Theorem

The final theorem resolves likelihood-equivalent representations by maximizing intervention-independent latent entropy, assigning intervention-dependent parts to their causal factors and collecting remaining information separately.

  • Target representation: The ideal representation assigns each latent group zψi the intervention-dependent part of causal factor Ci and assigns intervention-independent information to zψ0.This separates causal factors while consolidating information unaffected by interventions.
  • Disentanglement goal: The construction is designed to ensure that zψi contains information from Ci rather than information from another causal factor Cj.This addresses the ambiguity left by likelihood alone.
  • Selection criterion: Among likelihood-optimal disentanglements, maximizing the entropy of zψ0 selects the split that collects all intervention-independent information there.Any omitted invariant information would permit another optimal solution with greater entropy in zψ0.
  • Theorem: The theorem states that, with sufficient latent dimensions, the optimized model learns the intervention-dependent structure of the causal factors and places remaining intervention-independent information in zψ0.The result is stated under the likelihood and entropy constraints of the theorem.

B.6. Non-Identifiability without Interventions

Without sufficiently diverse interventions, temporal dependencies do not guarantee unique identification: rotated source representations can preserve the same density. The dataset setup therefore uses multidimensional causal factors, temporal sequences, and intervention targets to study identifiability.

  • Non-identifiability result: Jointly intervening on two causal factors, or never intervening on them separately, can make their identities non-unique from observations and targets.This parallels non-linear independent component analysis, where orthogonal mixing of Gaussian sources preserves the observed density.
  • Non-identifiability result: In the temporal Gaussian construction, an orthogonal mixing preserves the density because the rotation matrix has A^T A = I and determinant one.The rotated and original signals therefore have identical density functions.
  • Non-identifiability result: A valid observation function can implement this orthogonal mixing, so two causal variables cannot always be uniquely recovered from observational data alone.The proof maps the mixed sources to causal variables and the mixing matrix to the observation function.
  • Dataset setting: Sequences begin from random causal-factor values and evolve through conditional dynamics, with selected factors replaced by intervention samples at subsequent time steps.Targets are sampled independently, and intervened angles or hues use U(0, 2π), positions use U(-2, 2), and shapes use uniform categorical sampling.
  • Dataset setting: The dataset includes 150,000-image teapot sequences and 250,000-image seven-shape sequences to reduce sampling bias in identifiability experiments.The seven-shape version includes Cow, Head, Dragon, Hare, Armadillo, Horse, and Teapot.

C.3.3. TARGET CLASSIFIER IMPLEMENTATION

CITRIS uses a target classifier to predict intervention targets from temporally structured latent subsets, encouraging each subset to retain information about its assigned causal factor and not others. Its implementation combines masked latent inputs, a Gumbel-Softmax assignment, and a classifier loss added to the ELBO.

  • Target classifier: The target classifier predicts intervention targets from latent variables across consecutive time steps, using distribution changes that are often detectable even when targets are not perfectly identifiable.Its purpose is to preserve information about all causal variables while guiding disentanglement.
  • Target classifier: For causal factor C_i, the classifier receives z_t and the z_t+1 dimensions assigned to C_i by a sampled Gumbel-Softmax mask.The mask selects the latent subset associated with the factor being classified.
  • Target classifier: If intervention targets are independent and disentanglement is successful, latents assigned to C_i should predict only I_t+1^i rather than other targets.Gradients update the latent variables and assignment ψ so the selected representation becomes informative for its own target and uninformative for unrelated targets.
  • Optimization: The latent assignment ψ models the probability that latent dimension j belongs to causal factor i, exposing correlations between latent dimensions and intervention targets during training.The classifier parameters are optimized to recover as many intervention targets as possible from its inputs.
  • Optimization: CITRIS trains the classifier with cross-entropy against the intervention-target marginal and adds the resulting term to the ELBO with weight β_class > 0.The loss is L = L_ELBO + β_class · classifier loss, with gradient stopping used for the classifier-specific objective.
  • Implementation: The reported implementation uses a single-layer target-classifier MLP with hidden dimensionality 128, Layer Normalization, and SiLU activation.The broader models use convolutional VAE architectures, autoregressive priors, and affine autoregressive normalizing flows for CITRIS-NF.

D.1.1. DETAILS OF THE MAIN RESULTS

Across the main experiments, CITRIS-NF is especially effective when reconstruction entangles object shape and rotation, while CITRIS remains stable across seeds and under restricted intervention patterns. Generalization to unseen shapes is above random but is harder for position and rotation.

  • Main results: CITRIS-VAE and CITRIS-NF remain stable across three seeds and reach similar performance across seeds on the reported Temporal Causal3DIdent results.The standard deviations are reported in the supplementary tables.
  • Main results: CITRIS-NF avoids the rotation difficulty observed in CITRIS-VAE because its disentanglement is performed independently of reconstruction error through a pretrained autoencoder.CITRIS-VAE often produces blurry predictions when multiple shapes increase modeling complexity, making rotation harder to determine.
  • Generalization: On unseen Cow and Head shapes, generalization is stable and remains above random, although position and rotation are the most difficult factors.These factors are heavily dependent on object shape and therefore entangled in the autoencoder representation.
  • Main results: CITRIS-NF disentangles all causal factors well on the seven-shapes dataset, whereas other models especially struggle with rotation.The main entanglement in the seven-shape setting is between object shape and rotation angle.
  • Intervention settings: CITRIS performs similarly with six limited intervention sets and independently sampled intervention targets, showing robustness to restricted intervention diversity.The limited setting includes observational, single-target, and selected joint interventions.
  • Learned causal graph: The learned teapot graph matches ground truth, while the seven-shape graph contains one false-positive edge from object shape to object rotation.The extra edge is consistent with the observed shape-rotation entanglement in the seven-shape experiments.

D.1.5. INTERVENTIONS ON A SUBSET OF VARIABLES

CITRIS can disentangle variables with intervention targets provided for only a subset of factors, assigning untargeted information to z_ψ0. Experiments on rendered scenes and Ball-in-Boxes support this minimal-causal-variable behavior.

  • Subset interventions: When interventions are provided for only some variables, CITRIS assigns the remaining information to the separate latent group z_ψ0.The targeted variables remain disentangled while variables without intervention targets are mapped to the no-variable slot.
  • Subset interventions: Omitting background-hue interventions preserves disentanglement of the other seven-shape factors while assigning background hue to z_ψ0.Performance is similar to the full-intervention experiment.
  • Subset interventions: Excluding intervention targets for spotlight rotation or hue leaves the remaining variables well disentangled, with only slightly worse performance from additional entanglement in z_ψ0.Both models assign the untargeted information to the no-variable slot.
  • Latent assignments: The latent-assignment probabilities become sharply peaked after training, indicating that CITRIS-NF assigns latent dimensions strongly to particular causal variables.The assignment map is learned with a Gumbel-Softmax on the seven-shape dataset without background-hue interventions.
  • Interventional Pong: In Interventional Pong, only CITRIS consistently recovers the player’s score, while CITRIS-NF mainly improves modeling of ball velocity direction over CITRIS-VAE.SlowVAE struggles with correlated factors, and both iVAE* and SlowVAE often entangle ball y with velocity direction.
  • Ball-in-Boxes: In Ball-in-Boxes, z_ψ0 correlates almost perfectly with internal ball position and nearly zero with box identity, validating separation of intervention-dependent and independent information.The experiment evaluates box identifier, internal x position, and y position without changing CITRIS training.
Loading 2202.03169v3…