PaperlayerView exact revision on arXiv

Source-linked AI summary

Interventional Causal Representation Learning

Kartik Ahuja, Divyat Mahajan, Yixin Wang, Yoshua Bengio

arXiv:2209.11924v4stat.MLcs.LG

TL;DR

Causal representation learning needs provable latent-factor identification, while existing methods often rely on observational data and structural assumptions. This paper uses geometric support signatures induced by interventional data to establish identification guarantees, including permutation-and-scaling identification for perfect do interventions and block affine identification for suitable imperfect interventions. The results provide a theoretical basis for using interventions without assumptions on latent distributions or graphical structure.

  • Problem

    Existing causal representation-learning methods commonly use observational data and latent-dependency assumptions, motivating whether prevalent interventional data can enable identification instead.

  • Method

    The paper uses geometric signatures of latent supports, especially support independence induced by interventions, to derive representation-identification guarantees.

  • Results

    Perfect do interventions identify intervened latents up to shift and scaling, while suitable imperfect interventions provide block affine identification.

  • Takeaways & Limitations

    Interventional data can support provable causal representation identification without strong assumptions on latent distributions or dependency structure.

  • Takeaways & Limitations

    For non-polynomial decoders, approximate identification requires sufficiently many do-interventional datasets per latent, unlike the polynomial case.

Abstract

from arXiv · show

Causal representation learning seeks to extract high-level latent factors from low-level sensory data. Most existing methods rely on observational data and structural assumptions (e.g., conditional independence) to identify the latent factors. However, interventional data is prevalent across applications. Can interventional data facilitate causal representation learning? We explore this question in this paper. The key observation is that interventional data often carries geometric signatures of the latent factors' support (i.e. what values each latent can possibly take). For example, when the latent factors are causally connected, interventions can break the dependency between the intervened latents' support and their ancestors'. Leveraging this fact, we prove that the latent causal factors can be identified up to permutation and scaling given data from perfect $do$ interventions. Moreover, we can achieve block affine identification, namely the estimated latent factors are only entangled with a few other latents if we have access to data from imperfect interventions. These results highlight the unique power of interventional data in causal representation learning; they can enable provable identification of latent factors without any assumptions about their distributions or dependency structure.

1. Introduction

Causal representation learning seeks provable identification of latent factors, but existing approaches commonly rely on observational data and structural assumptions. This work shows that interventional data exposes geometric support signatures that enable identification guarantees without strong distributional assumptions.

  • Interventional data is prevalent in applications including robotics, genomics, genetic perturbations, and electrical stimulations.
  • Interventions can make an intervened latent’s support independent of its ancestors’ support, providing a geometric signature for identification.
  • Hard do interventions identify an intervened latent up to shift and scaling under a finite-degree polynomial observation map.
  • Imperfect interventions achieve block affine identification when the intervened latent’s support is independent of its ancestors, including all perfect interventions as a special case.
  • With observational data whose latent supports are independent, latent factors are identifiable up to permutation, shift, and scaling without interventional data.
  • Experiments across polynomial data-generation mechanisms and rendered-image generation empirically show that interventional data helps identification.

2. Related Work

Prior representation-identification work uses time-series structure, augmentations, observational data, or counterfactual data. This work instead studies interventional data and avoids assumptions on the latent graphical model, including beyond linear mixing and perfect interventions.

  • Existing approaches exploit time-series structure or other weak supervision such as data augmentations for representation identification.
  • Observational-data methods commonly impose assumptions on latent dependency or graphical structure, whereas this work operates with interventional data.
  • A concurrent interventional approach studies linear mixing and perfect interventions, while this work considers nonlinear mixing and imperfect interventions.

3. Setup: Causal Representation Learning

The setup models high-dimensional observations as generated by an injective decoder from latent variables and combines observational with interventional datasets. The goal is to learn an encoder whose representation matches the latents up to specified transformations.

  • Observations x in R^n are generated from latent variables z in R^d through an injective decoder g.
  • Interventional datasets use distributions of z under interventions on individual latent variables.
  • Causal representation learning seeks an encoder that maps observations to their underlying latent variables, typically through an encoder-decoder reconstruction identity.
  • The reconstruction identity alone is highly underspecified and generally cannot identify the true latents.
  • Because exact latent recovery is unnecessary in practice, the analysis permits transformations such as affine maps and coordinate permutations.

4. Stepping Stone: Affine Representation Identification with Polynomial Decoders

The paper first studies affine identification with polynomial decoders and non-collapsing encoders. Under support, injectivity, and finite-degree polynomial assumptions, reconstruction identifies the latent representation up to an invertible affine transformation.

  • Assumptions: The latent support must have non-empty interior, and the learned encoder’s image must satisfy a non-collapsing condition.
  • Assumptions: The decoder is assumed to be a finite-degree polynomial with a full-column-rank coefficient matrix, ensuring injectivity.
  • Constraints: The reconstruction identity is solved under polynomial-decoder and non-collapsing-encoder constraints.
  • Affine identification: Theorem 4.4 identifies the learned representation as ˆz = Az + c, where A is invertible and c is a shift.
  • Affine identification: This result reduces ambiguity from arbitrary invertible transformations to invertible affine transformations without assumptions on latent dependency structure.
  • Proof intuition: The polynomial argument shows that the learned representation must be degree one in the latent when the true and learned decoders have degree two.
  • Beyond polynomial maps: Theorem A.8 extends the result to maps that are approximately representable by polynomials.

5. Provable Representation Identification with Interventional Data

The section uses geometric signatures induced by interventions to identify latent factors from observational and interventional image data. Perfect interventions yield affine identification of intervened latents, while imperfect interventions provide block affine guarantees under support-independence conditions.

  • Motivation: Interventional image data encodes latent support geometry that can guide an autoencoder toward causal representation identification.The motivating example compares observational dependencies with a do intervention on one ball coordinate.
  • Do interventions: A do intervention fixes one latent coordinate while the remaining variables are sampled from an intervention-specific distribution.The intervention affects descendants, while the remaining latent variables follow the specified distribution over z_-i.
  • Do interventions: Constraint 5.1 requires one encoder component to take a fixed value across all observations from an interventional support.The learner need not know which component was intervened or which fixed value was used.
  • Do interventions: Under the stated assumptions, the intervened latent is identified up to shift and scaling, and multiple hard interventions identify the intervened latents up to permutation, shift, and scaling.The guarantee combines affine identification with the geometric support constraint and does not require distributional or graphical-model assumptions on the latent variables.
  • Beyond polynomial maps: For nonpolynomial decoders, sufficiently many do-interventional datasets can yield approximate identification, whereas one intervention per latent no longer suffices.The extension assumes a general diffeomorphism and additional support regularity conditions.
  • Perfect and imperfect interventions: Imperfect interventions achieve block affine identification when the intervened latent’s support is independent of its non-descendants.When the relevant set has size d−1, one learned component identifies a latent up to scale and shift while the remaining components exclude it.

6. Extensions to Identification with Observational Data & Independent Support

Independent latent supports can themselves provide the geometric structure needed for representation identification using observational data alone. Under pairwise support independence, the learned representation is identified up to permutation, shift, and scaling even when latent variables are dependent.

  • Independent support: The observational-data extension assumes pairwise support independence between every pair of latent variables.The condition concerns the Cartesian structure of latent supports rather than statistical independence of the variables.
  • Identification guarantee: Under this condition, an autoencoder satisfying the support-independence constraint achieves permutation, shift, and scaling identification.The encoder output has the form ˆz = ΛΠz + c, with Π a permutation matrix and Λ an invertible diagonal matrix.
  • Implications: The guarantee applies even when latent variables are dependent, provided their supports are bounded and pairwise support-independent.This extends the identification setting beyond the independent non-Gaussian latent assumption in linear ICA.
  • Implications: The section presents independent support as a theoretical justification for recent unsupervised disentanglement proposals.The cited proposals use independent-support structure in observational data.

7. Learning Representations from Geometric Signatures: Practical Considerations

The practical algorithms first learn a reconstructive autoencoder and then apply an invertible linear transformation to enforce intervention-specific or observational support constraints. These procedures implement the geometric-signature conditions used in the identification theorems.

  • Algorithmic setup: The practical section develops algorithms for the constrained representation-learning problems involving interventional and observational support geometry.The procedures target the constraints from Sections 5 and 6.
  • Do-intervention algorithm: For do-intervention data, the first step minimizes reconstruction over observational and interventional samples using an encoder and decoder.With a polynomial decoder, the learned encoder obtains the affine-identification guarantee used by the theory.
  • Do-intervention algorithm: The second do-intervention step learns a distinct linear map for each intervention so a projected representation component is constant on that intervention’s support.The final representation is formed by applying the learned transformation to the initial encoder output.
  • Implementation remarks: The fixed intervention value is arbitrary, and the learner can relax the assumption that it knows the intervened index by adding a permutation matrix.The practical objective therefore does not require the true do value and can accommodate unknown component ordering.
  • Independent-support algorithm: For independent-support constraints, the method measures support dependence with Hausdorff distance and learns an invertible map that transforms the representations.The transformed joint and marginal supports are used to formulate the optimization objective.

8. Empirical Findings

Experiments evaluate the proposed identification procedures on polynomial-decoder and image-rendering settings using observational and interventional data. The reported MCC results support the theoretical predictions, including near-one values with interventional loss and improvement with more do-interventional distributions.

  • Experimental settings: Experiments cover polynomial-decoder data and 64 × 64 × 3 images generated with PyGame, using do interventions on each latent dimension.Polynomial settings vary latent distributions, SCM connectivity, latent dimension, and decoder degree; image settings use two balls.
  • Evaluation procedure: The evaluation uses a two-step training procedure, with MLPs for non-image experiments, ResNet-18 for images, and MCC to assess representation identification.Polynomial-decoder experiments also evaluate reconstruction with R2 and support-independence identification with MCC (IOS).
  • Polynomial decoder: High R2 and MCC (IOS) values occur as predicted for polynomial-decoder experiments, with high MCC (IOS) specifically for the uniform distribution satisfying support independence.The first step minimizes reconstruction loss; the second enforces independence of support through Hausdorff-distance minimization.
  • Polynomial decoder: MCC (IL) values are close to one after learning Γ with interventional loss from one do intervention per latent dimension, as predicted by Theorem 5.3.These experiments combine observational and do-intervention data and report MCC after the second training step.
  • Image dataset: MCCs increase as the number of do-interventional distributions per latent dimension grows in the image-based experiments, matching Theorem A.12.The image procedure learns a nonlinear MLP map in the second step to minimize interventional loss.

9. Conclusions

The paper establishes theoretical foundations for causal representation learning with interventional data. It shows that intervention-induced geometric signatures, such as support independence, support provable representation identification and motivates future study of real interventional data.

  • The work establishes theoretical foundations for learning causal representations in the presence of interventional data.
  • Geometric signatures such as support independence induced by many interventions are useful for provable representation identification.
  • Exploring representation learning with real interventional data is identified as a fruitful direction for future work.

Appendices

The appendices establish injectivity and bijectivity properties underlying affine representation identification, then extend the results to approximate and sparse polynomial decoders. They also show that interventions can identify an intervened latent up to shift and scaling.

  • Proof ingredients: The appendices prove that polynomial multiplication preserves total degree, supporting the injectivity arguments used in later identification results.The proof separates highest-degree terms and shows their product cannot be cancelled by lower-degree terms.
  • Affine identification: The encoder output is related to the true latent by a bijective mapping under observational and interventional data with reconstruction.Injectivity follows because equal encoder outputs imply equal reconstructions and therefore equal observations and latents.
  • Approximate identification: Approximate reconstruction yields an approximately linear representation when higher-order weights are sufficiently small and decay with degree.The stated decay rate for a degree-k term is proportional to 1/z_k−1.
  • Interventions and extensions: Under the intervention constraints, the intervened latent is identified up to shift and scaling, while sparse polynomial decoders reduce the required observation dimensionality.The sparse-polynomial extension allows the data dimension to scale with the number of nonzero basis terms rather than all degree-p terms.

A.2.1. EXTENSION OF do INTERVENTIONS BEYOND POLYNOMIALS

With multiple do-interventional distributions on a target latent, the paper extends approximate identification beyond polynomial decoders. The number of interventions controls the approximation guarantee, while a single intervention is insufficient without additional structure.

  • Extension beyond polynomial decoders: Multiple do-interventional distributions enable approximate identification of a target latent up to an invertible transform beyond polynomial decoders.The result extends to all latents when multiple do-interventional distributions are available for each latent.
  • Assumptions: The guarantee assumes nonempty latent-support interiors, unchanged ancestor support under interventions, and intervention targets covering the observational support with density at least ϱ.These support conditions underpin the covering argument used to obtain the approximation bound.
  • Identification guarantee: The intervention count must scale with the target accuracy, confidence, support-density lower bound, and derivative bound.The sufficient condition includes log terms involving δ, ϵ, L, ϱ, βisup, and βi inf.
  • Identification guarantee: If the number of interventions is sufficiently large, the estimated component’s dependence on other latents has gradient bounded by ϵ.The theorem gives the condition ∥∇z−iak(z)∥∞≤ϵ across the latent support.
  • Why multiple interventions are needed: A single intervention imposes only local constraints on the learned bijection when the decoder and latent-support structure are otherwise unrestricted.An arbitrary bijection with bounded second derivative can satisfy the representation-learning constraints under one intervention.
  • Perfect and imperfect interventions: Under support-independence constraints, the related general-intervention result yields block affine identification with disjoint row supports in the invertible affine map.Each learned component can involve a block of latent coordinates, while distinct rows cannot share a nonzero column.

A.4. Representation identification with observational data under independent support

Under independent-support assumptions, the paper uses an autoencoder followed by a transformation constrained by reconstruction and support geometry. The resulting representation is identified up to permutation, shift, and scaling.

  • Identification result: Theorem 6.3 establishes permutation, shift, and scaling identification under observational data with independent support.The learned representation has the form ˆz = ΛΠz + c, with Π a permutation matrix and Λ invertible.
  • Identification result: The identification proof derives componentwise disentanglement by applying the block-affine result to every pair of learned components.If a row mixed multiple latent coordinates, invertibility would fail under the pairwise support constraints.
  • Two-step procedure: The proposed procedure first trains an autoencoder on observational and interventional data, then learns an invertible transformation using an independence-of-support objective.The second stage combines reconstruction loss with Hausdorff distance or uses intervention-specific mappings.
  • Two-step procedure: For interventional-data training, the transformation is learned separately for each latent using data from interventions on that latent.Multiple intervention distributions provide new targets for the corresponding mapping.

B.2. Experiment setup details: Polynomial decoder (g)

The polynomial-decoder experiments use synthetic latent distributions ranging from independent uniforms to correlated, Gaussian-mixture, and linear-Gaussian-DAG settings. Models are trained with specified neural architectures, optimization choices, and intervention data splits.

  • Latent distributions: The experiments include Uniform, Uniform-Correlated, Gaussian-Mixture, and two linear-Gaussian DAG settings with expected edge densities 0.5 and 1.0.The correlated construction uses confounders while satisfying independence of support.
  • Latent distributions: The Uniform distribution samples each latent independently from Uniform(-5, 5).This provides an independent, identically distributed baseline latent distribution.
  • Data and evaluation: The intervention experiments combine observational data with data from interventions on each latent, while observational-only experiments use D alone.Evaluation metrics are computed on the observational test split without interventional data.
  • Architectures: The encoder uses linear layers with LeakyReLU activations and hidden width h = 200, while the polynomial decoder matches the true decoder degree.The polynomial decoder represents monomials through repeated tensor products of the latent vector.
  • Optimization: Training uses Adam, early stopping after 10 unimproved validation epochs, and a learning rate selected from {10^-3, 5 × 10^-4, 10^-4}.For IOS experiments, the Hausdorff-distance weight is λ = 10.

B.3. Additional results: Polynomial decoder (g)

Additional experiments show that independence-of-support training improves representation quality in observational settings, while interventional training improves MCC for all tested latent distributions. Image reconstructions remain accurate across uniform, linear SCM, and nonlinear SCM cases.

  • Observational data: Training with the independence-of-support objective improves MCC on distributions satisfying independence of support.This includes the Uniform-C latent case, whose latent variables are dependent but have independent support.
  • Interventional data: Interventional training in Step 2 improves MCC across all tested latent distributions.The extended polynomial-decoder results report this consistent pattern for the interventional case.
  • Interventional data: MCC (IL) is high for interventional data with a polynomial decoder.The table caption identifies this result as predicted by Theorem 5.3.
  • Observational data: MCC (IOS) achieves high values for the Uniform and Uniform-C latent cases.These results are reported as predicted by Theorems 4.4 and 6.3.
  • Image experiments: The autoencoder accurately reconstructs ball positions and colors for uniform, linear-SCM, and nonlinear-SCM image data.The experiments use 64 × 64 × 3 images generated from two-ball latent distributions.
Loading 2209.11924v4…