Source-linked AI summary
Learning Linear Causal Representations from Interventions under General Nonlinear Mixing
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, Pradeep Ravikumar
TL;DR
The paper asks whether causal representations can be identified from interventional data when latent variables are unobserved and the mixing is nonlinear. It analyzes the geometry of transformed interventional densities and proposes a contrastive algorithm. It reports identifiability of the full latent model from non-paired interventions with unknown targets, subject to stated assumptions.
Problem
Causal representation learning is generally non-identifiable without added information, and prior settings often rely on known intervention targets or paired counterfactual data.
Source: §1 · pp. 1–2Method
Results
The authors show that non-paired interventional data with unknown targets identify the full latent model, including its mixing, targets, and latent causal structure.
Source: §1 · p. 2Takeaways & Limitations
The results extend identifiability to general nonlinear mixing and the authors’ experiments suggest contrastive learning is promising for interventional causal representation learning.
Source: §1 · p. 2, §7 · p. 9The guarantees depend on intervention conditions: all-pure-noise interventions are non-identifiable, and non-stochastic hard interventions must be excluded for identifiability up to linear maps.
Source: §4 · p. 6
Abstract
from arXiv · showhide
We study the problem of learning causal representations from unknown, latent interventions in a general setting, where the latent distribution is Gaussian but the mixing function is completely general. We prove strong identifiability results given unknown single-node interventions, i.e., without having access to the intervention targets. This generalizes prior works which have focused on weaker classes, such as linear maps or paired counterfactual data. This is also the first instance of causal identifiability from non-paired interventions for deep neural network embeddings. Our proof relies on carefully uncovering the high-dimensional geometric structure present in the data distribution after a non-linear density transformation, which we capture by analyzing quadratic forms of precision matrices of the latent distributions. Finally, we propose a contrastive algorithm to identify the latent variables in practice and evaluate its performance on various tasks.
1 Introduction
The introduction motivates causal representation learning as a route to interpretable representations with causal structure, while noting that identifiability is generally impossible without added information. This work studies Gaussian latent-variable models under unknown single-node interventions and general nonlinear mixing, and claims identifiability without paired counterfactual data or known intervention targets.
1 Introduction
- Identifiability is presented as a prerequisite for recovering a unique model up to tolerable ambiguities, despite general non-identifiability without added information or inductive biases.The authors motivate interventional data as additional information and connect causal structure to interpretability, fairness reasoning, planning, and reasoning.Source: §1 · pp. 1–2
- The contribution extends beyond linear or polynomial mixing to nonlinear functions that include representations learned by deep neural networks.The authors consider perfect and imperfect interventions and allow shift interventions.Source: §1 · p. 2
- The setting assumes Gaussian latent variables generated by an unknown structural equation model and observations X = f(Z) for arbitrary nonlinear f.Interventional observations are available only as marginals, without paired counterfactual samples or known target nodes.Source: §1 · p. 2
- The authors identify the full latent model from non-paired interventional data with unknown targets, recovering the mixing, intervention targets, and latent causal structure.The model permits general nonlinear mixing and studies perfect and imperfect interventions, including shift interventions.Source: §1 · p. 2
- The authors construct counterexamples to probe the tightness of their assumptions and suggest directions for future work.Source: §1 · p. 2
2 Related work
Prior causal representation learning work has addressed non-identifiability through auxiliary labels, sparsity, or restricted function classes, while interventional latent-structure learning has used several narrower settings. This paper distinguishes its setting by allowing general nonlinear mixing and non-paired interventions with unknown targets.
2 Related work
- Prior approaches to causal representation learning address non-identifiability using auxiliary labels, sparsity, or restrictions on function classes.Many practical methods are variants of the Variational Autoencoder framework.Source: §2 · p. 2
- Compared with prior interventional latent-structure work, this setting uses non-paired data with unknown targets and general nonlinear mixing.The cited alternatives include known targets, paired counterfactuals, linear or polynomial mixing, and settings requiring more interventions or graph restrictions.Source: §2 · pp. 2–3
- Applying contrastive learning to non-paired interventional causal representation learning is described as new to the authors’ knowledge.Earlier uses cited here involve time-series data or paired counterfactual data.Source: §2 · p. 3
3 Setting: Interventional Causal Representation Learning
The setting combines a flexible nonlinear observation map with a linear Gaussian causal model over latent variables. Single-node interventions provide the additional information needed to study identifiability, subject to coverage and other assumptions.
3 Setting: Interventional Causal Representation Learning
- Observed data are generated as X = f(Z), with injective, differentiable nonlinear f and latent Z following a linear Gaussian structural causal model.The model allows flexible mixing while restricting the latent distribution; the paper notes that this Gaussian assumption is restrictive but commonly used.Source: §3 · pp. 3–4
- The Gaussian latent assumption is restrictive, although the model retains universal approximation capacity through its flexible mixing function.The paper notes that results may extend to other latent distributions but fail when no assumption is made on the noise distribution.Source: §3 · p. 4
- Each intervention changes one latent node’s assigning equation while leaving the others unchanged, and interventions must cover every node.Without coverage, the model is non-identifiable; the theory also extends to independent observation noise by denoising first.Source: §3 · pp. 4–5
- Interventions may be perfect or imperfect and may shift the mean; they cannot add new incoming edges to the target node.Perfect interventions remove all connections to former parents, whereas imperfect interventions are those that do not.Source: §3 · p. 4
- Intervention targets may be unknown, including when multiple interventions target the same node, but the observational distribution must be identified.The authors expect this restriction on knowing the observational distribution may be removable.Source: §3 · p. 5
4 Main Results
The main results establish identifiability under perfect and imperfect single-node interventions, with guarantees depending on intervention type. The proof identifies nonlinear mixing up to a linear transformation by exploiting geometric structure in likelihoods.
4 Main Results
- Under perfect interventions, the model is identifiable up to permutation and scaling, with shifts recovered exactly and the causal graph recovered up to label permutation.The result identifies the mixing, intervention targets, and parameter matrices subject to the stated ambiguities.Source: §4 · p. 5
- Under imperfect interventions that are not pure-noise interventions, the intervention targets and ancestor partial order are identifiable up to permutation of labels.The guarantee is weaker than the perfect-intervention result and does not establish identifiability of the full graph.Source: §4 · p. 5
- Theorem 3 identifies the nonlinear mixing function up to an invertible linear transformation, enabling the paper to apply earlier linear-setting results.Its proof analyzes log-odds as quadratic forms of latent precision-matrix differences; single-node interventions yield rank-at-most-two structure and source-node targets rank one.Source: §4 · pp. 5–6
- The main results require at least d interventions, a tight requirement for Theorems 1 and 2 and tight to within one intervention for Theorem 3.For Theorem 3, d − 2 interventions are insufficient.Source: §4 · p. 6
- Pure-noise interventions alone leave every causal graph compatible with the observations, and non-stochastic hard interventions must be excluded for identifiability up to linear maps.The weaker guarantee for imperfect interventions cannot be improved even with linear mixing.Source: §4 · p. 6
- The results assume Gaussian latent variables; the paper notes they need not hold without assumptions on the noise distribution.The authors describe possible extension to more general latent distributions, such as exponential families.Source: §4 · p. 6
5 Experimental methodology
The experimental method trains a neural classifier to distinguish observational from interventional samples, using its output layer to model a linear Gaussian causal structure. Regularization encourages the learned graph parameters to be acyclic and sparse.
5 Experimental methodology
- A deep neural network distinguishes observational samples from interventional samples, with a designed final layer modeling the log-likelihood of a linear Gaussian structural causal model.The Bayes-optimal classifier is related to the underlying causal parameters, which the chosen parametric form is intended to learn indirectly.Source: §5 · pp. 6–7
- The method adds a NOTEARS regularizer to encourage an acyclic graph and an l1 penalty to promote sparsity in the learned matrix.The matrix is intended to learn B = D^-1/2(Id − A).Source: §5 · p. 7
- Nonzero shifts are used in the experiments because the cross-entropy loss with the quadratic log-odds expression creates a non-convex output layer that makes finding a global minimizer difficult.The authors distinguish this optimization difficulty from their theoretical identifiability result; without shifts, their algorithm often fails to find the global minimizer.Source: §5 · p. 7
6 Experiments
Experiments test recovery under linear and nonlinear mixing, including rendered images, using synthetic Erdős–Rényi graphs and latent and graph-recovery metrics. The contrastive method performs strongly under nonlinear mixing and image data, while the linear-setting methods outperform the baseline on graph recovery.
Data generation
- The synthetic and image experiments use Erdős–Rényi graphs, with each undirected edge sampled at equal probability before edges are oriented in a random node order.The experiments consider linear mixing, three-hidden-layer MLP mixing, and rendered images in which latent-coordinate pairs specify ball locations.Source: §6 · p. 8
Evaluation Metrics
- Recovery is evaluated with MCC for latent variables and SHD and AUROC for graph structure; SHD counts a wrongly oriented edge as two errors.For SHD, the edge-selection threshold is set by matching the expected number of edges.Source: §6 · p. 8
Methods
- The contrastive implementation uses an MLP for linear and nonlinear mixing and a small convolutional network for images; baselines include a VAE and linear disentanglement.A linear version of the contrastive method is also evaluated for linear mixing.Source: §6 · p. 8
Results for linear f
- For linear mixing, both contrastive variants recover the underlying graph better than the baseline, while the linear contrastive method identifies latents up to scaling.This replicates the setting from with d = 5, d′ = 10, k = 3/2, and n = 50000.Source: §6 · p. 8, Table 1 · p. 9
Results for nonlinear f
- On nonlinear synthetic data (n = 10000), the contrastive method recovered ground-truth latents and causal structure almost perfectly; linear disentanglement recovered neither graph nor latents, and vanilla VAE missed latents up to a linear map.Source: §6 · p. 9, Table 2 · p. 9
Results for image data
- On image data, the experiments report recovery of both the latent graph and latent variables, excluding samples where a ball falls outside the image.That exclusion creates slight model misspecification relative to the theory; Table 3 covers image data with ER(d, 2) graphs.Source: §6 · p. 9, Table 3 · p. 9
7 Conclusion
The paper establishes identifiability for linear latent-variable models with nonlinear mixing from interventions whose targets are unknown, and proposes a contrastive approach that recovers latent structure in experiments. The appendix develops the proof using Gaussian model notation, a rank constraint from single-node interventions, and a warm-up result under stronger assumptions.
7 Conclusion
- The paper extends identifiability results to linear latent-variable models with nonlinear mixing and unknown intervention targets, and proposes a contrastive approach that recovers latent structure in varied settings.The authors leave extension to other prior families, sample complexity, robustness, and scaling to large datasets for future work.Source: §7 · pp. 9–10
A.1 General Notation
- The proof compares two Gaussian structural causal models and their mixing functions that generate the same observed distributions.The notation represents each latent distribution through its intervention-specific precision matrix, mean, and intervention target.Source: §A.1 · p. 17
A.2 Two auxiliary lemmas
- The proof’s density comparison cancels the Jacobian term across observational and interventional distributions, yielding a relation between latent quadratic forms.The latent distributions are Gaussian, and the density identity follows from the pushforward relationship between the two models.Source: §A.2 · p. 18
- A single-node intervention makes the difference between its precision matrix and the observational precision matrix have rank at most 2.The constraint follows because all rows of the structural matrix except the target node’s row agree across the two distributions.Source: §A.2 · p. 19
A.3 Warm-up
- When interventions are perfect, target-known, shift-free, and include two distinct interventions per node, the warm-up identifies latent variables up to diagonal scaling.The result assumes 2n pairwise different interventions and known matching targets across the two representations.Source: §A.3 · p. 19
A.4 Proof of identifiability up to linear maps
The proof uses geometric properties of quadratic forms to show that corresponding latent coordinates must be linear. It proceeds by induction through a topological ordering, handling ordinary interventions and pure shifts separately.
Proof strategy
- The proof of the main result relies on two geometric lemmas and a slight extension of Lemma 2: the first handles changes to scaling matrix D, the second changes to B, and the third allows pure shift interventions.Source: §A.4 · p. 20
Geometric lemmas
- A continuous surjective function constrained by a quadratic form of rank 1 is affine; under the stated stronger relation it is linear.The proof also establishes an affine conclusion when the function multiplies a nonzero linear form and equals a quadratic-plus-linear expression.Source: §A.4 · pp. 20–21
Shift interventions
- For pure shift interventions, the proof uses a completed-square relation and a diffeomorphism contradiction to establish linearity in the remaining case.The pure-shift case is identified when the intervention and observational structural matrices agree but the shift is nonzero.Source: §A.4 · pp. 20, 25
Main theorem
- The main theorem claims identifiability up to an invertible linear map when Assumptions 1–4 hold and two models generate the same observed distributions.The models have matching latent dimensions.Source: §A.4 · p. 22
Induction proof
- The proof orders latent variables topologically and inductively establishes linearity of each coordinate map, using precision-matrix differences and the earlier coordinates.The sketch ignores shifts; the full proof addresses shifts and the special cases that arise in the induction.Source: §A.4 · pp. 22–23
B From Linear Identifiability to Full Identifiability
The appendix extends linear identifiability results to the paper’s nonlinear setting and compares the resulting guarantees with prior work. It also introduces counterexamples and proof details relevant to the scope of those guarantees.
B From Linear Identifiability to Full Identifiability
- Under Assumptions 1–4, perfect interventions identify the full latent model up to permutation and scaling; the causal graph is identifiable up to label permutation.The model includes the mixing function, intervention targets, and latent causal structure.Source: B From Linear Identifiability to Full… · p. 26
- The proof transfers the linear result through an invertible transformation, converting certain shift interventions into variance changes before applying the linear theorem.It also accounts for scaling and permutation conventions that differ from the prior result.Source: B From Linear Identifiability to Full… · pp. 25–26
- With no pure-noise interventions, intervention targets and the causal partial order remain identifiable up to permutation, but the full-model guarantee is weaker.The appendix combines the linear identifiability result with prior linear-mixing results, adjusting normalization and accommodating shifts.Source: B From Linear Identifiability to Full… · pp. 25, 27
C A comparison to related works
- Compared with [48], the results require fewer interventions—d rather than 2d—and avoid restrictions on latent and observed dimensions, mixing bijectivity, and graph edge count.The comparison is made when [48]’s additional assumptions hold.Source: C A comparison to related works · p. 28
- Unlike [2], the paper studies randomized soft and shift interventions, obtaining full identifiability with one intervention per latent under structural assumptions; the settings are not directly comparable.[2] uses deterministic hard interventions and multiple interventions per latent for approximate identifiability.Source: C A comparison to related works · p. 28
D Counterexamples and Discussions
- The appendix introduces counterexamples to relaxed assumptions and supplies missing proofs for them.Its proof of full identifiability also builds on the linear result and, rather than repeating a separate rank-based induction argument.Source: B From Linear Identifiability to Full… · p. 25, D Counterexamples and Discussions · p. 28
D.1 Main counterexamples
The counterexamples identify boundaries of the paper’s identifiability results, including intervention count, intervention type, and latent-noise assumptions. Technical constructions use distribution-preserving maps to demonstrate non-identifiability in particular settings.
D.1 Main counterexamples
- d interventions, one per node, are necessary and sufficient for the main results; with at most d−2 interventions, the mixing cannot generally be identified up to linear maps.The insufficiency is illustrated by d = 2 with no interventions and nonlinear radius-dependent rotations.Source: §D.1 · p. 28
- Theorem 1 requires perfect interventions; with imperfect interventions, only Theorem 2’s weaker identifiability guarantees can be expected.The need for interventions that are not pure-noise is also stated for Theorem 2.Source: §D.1 · p. 29
- Pure-noise interventions can leave the causal graph arbitrary, identifying noise scales but providing no information about causal variables or structure.The construction changes the latent causal matrix while preserving the observed distributions.Source: §D.1 · p. 29
- Gaussian latent noise is a key assumption; broader families such as exponential families are proposed as possible extensions, while arbitrary noise distributions do not preserve the result.The authors leave such generalizations to future work because they require different proof strategies.Source: §D.1 · p. 29
- Linear structural causal models are assumed, and extending the results beyond them is left for future work.The text notes open questions even for causal structure learning beyond additive-noise models.Source: §D.1 · p. 30
D.2 Technical constructions
- For two Gaussian environments with covariance matrices ordered in the Löewner sense, a nonlinear map can preserve both distributions, giving a non-identifiability example.The proof constructs a vector-field flow that preserves both distributions; the stated covariance-order condition is essential to this counterexample.Source: §D.2 · p. 30
- For degenerate Gaussians produced by two hard interventions, a nonlinear map can preserve the observational and interventional distributions by remaining identity near their supports.A divergence-free vector field supported away from the intervention supports yields the distribution-preserving flow.Source: §D.2 · p. 31
- A bounded-support uniform example constructs a map that agrees with a shear on the relevant rectangle and is adjusted to preserve uniform measure.The construction modifies the map outside the region needed to match the first two environments, then applies Moser’s theorem to make it measure preserving.Source: §D.2 · pp. 31–32, Figure 4 · p. 32
E Identifiability and Approximation
This section explains why identifiability within an approximating function class does not necessarily resolve ambiguity in a larger class. It also describes a possible minimum-description-length preference within the smaller class.
E Identifiability and Approximation
- Identifiability in a smaller function class G does not resolve ambiguity in a larger class F unless the ground-truth mixing is known to belong exactly to G.Spurious solutions in F can be approximated within G alongside the ground truth.Source: E Identifiability and Approximation · p. 33
- Even when the ground-truth mixing lies in G, approximate spurious solutions in G can produce arbitrarily small Wasserstein error while representing different data representations.The passage describes this as a lack of robustness of the identifiability result.Source: E Identifiability and Approximation · p. 33
- Within G, a norm or polynomial degree can favor the ground-truth mixing over approximated spurious solutions through a minimum-description-length principle.The argument says those approximations will generically have a larger norm or degree.Source: E Identifiability and Approximation · p. 33
- Extending that preference to F requires the ground truth to be better approximated in G than every spurious solution, a condition described as difficult to verify and formalize.The paper also notes that norms on F, such as derivative norms, could be used for model selection.Source: E Identifiability and Approximation · p. 33
F Additional details on experimental methodology
- The appendix outlines further experimental-methodology details, including contrastive log-odds modeling, NOTEARS regularization, limitations of the contrastive approach, and possible VAE adaptations.It describes these as topics addressed across Appendices F and G.Source: F Additional details on experimental… · p. 33
F.1 Proof of Lemma 1
The appendix derives an interventional log-odds expression and uses it to motivate a contrastive classifier whose optimal form recovers underlying parameters. It also describes graph regularization and the tractability obstacle for a VAE approach with unpaired interventions.
F.1 Proof of Lemma 1
- For perfect interventions, the appendix derives the log-odds of an interventional sample over an observational sample using the latent Gaussian density and change of variables.The expression is stated up to a constant independent of the sample.Source: §F.1 · p. 34
F.2 A theoretical justification for the log-odds model
- With a sufficiently expressive neural network and enough samples, the modeled classifier can reach the Bayes-optimal classifier, which motivates learning the inverse mixing and underlying parameters up to scaling.The optimal parameter correspondence includes h(x, θ) = f^-1(x) and intervention-specific rows corresponding to s(i).Source: §F.2 · p. 35
F.3 NOTEARS [102]
- The algorithm regularizes the learned off-diagonal parameter matrix with NOTEARS to encourage a directed acyclic graph, rather than enforcing an assumed node ordering.The paper selects the NOTEARS regularizer instead of an augmented Lagrangian formulation to avoid added training complexity.Source: §F.3 · p. 35
F.4 A Variational Autoencoder approach
- A VAE approach can encode observational data and apply intervention-specific parameters before decoding, but the authors cannot directly map observations to latent variables without knowing the latent prior.Their objective is to learn that prior.Source: §F.4 · pp. 35–36
- For unpaired interventional marginals, the VAE’s interventional likelihood term is intractable; a divergence-based substitute is proposed as a heuristic for future exploration.The passage distinguishes observed marginal distributions from paired counterfactual data.Source: §F.4 · p. 36
G Additional experimental results
- The appendix also examines how intervention shift strength, data scale, and noise distribution affect experimental results.It notes that the contrastive algorithm struggles to recover latents without shifts and points to further analysis of shift strength.Source: G Additional experimental results · p. 36
G.1 Limitations of the contrastive approach
The contrastive approach struggles to recover ground-truth latents without shifts, but recovers them almost perfectly with non-zero shifts. At zero shift, it can recover the latent only up to sign, a behavior the authors link to the quadratic log-odds objective.
G.1 Limitations of the contrastive approach
- Figure 5 examines how performance metrics depend on shift η under nonlinear mixing, ER(10, 2) graphs, and d′ = 100.Other settings match those in Table 7.Source: Figure 5 · p. 37
- The authors suspect the failure stems from non-convexity in the parametric output layer, where gradient descent may not find the global minimizer.They contrast this with neural-network settings whose final layer is often a convex optimization problem.Source: §G.1 · p. 37
- A one-dimensional illustration fixes the log-odds form to a(z − b)^2 + c, with b = 0 when there is no shift.Figure 6 plots sample conditional cross-entropy loss against estimated latent ẑ for b = 0 and b = 2.Source: §G.1 · p. 37, Figure 6 · p. 37
- At η = 0, the recovered latent is essentially |Z| rather than Z, while non-zero shifts yield almost-perfect recovery.The authors note a small offset because the estimates are centered.Source: §G.1 · pp. 37–38
- For symmetric latents, E(Z|Z|) = 0, explaining the tiny MCC scores when recovery is effectively up to sign.The authors suggest this observation could help improve the algorithm.Source: §G.1 · p. 38
G.2 Varsortability and data scale
The appendix checks whether data-scale structure or non-Gaussian noise accounts for the algorithm’s performance. Standardizing the latents leaves results roughly similar, while performance with non-Gaussian noise is slightly worse but still reasonable.
G.2 Varsortability and data scale
- Average varsortability is 0.92 ± 0.02 for d = 10 and k = 2, but R2-sortability is roughly 0.4 and thus cannot be exploited to infer causal structure in these settings.The latter holds even when Z is observed.Source: §G.2 · pp. 38–39
- With standardized latent variables, nonlinear synthetic-data results are roughly similar to the corresponding unstandardized setting.The standardized experiment is reported in Table 4 and compared with Table 2.Source: §G.2 · p. 39, Table 4 · p. 39
G.3 Effect of noise distribution
- Theoretical results cover Gaussian SCMs, and identifiability does not necessarily hold for uniformly distributed noise.The algorithm was also evaluated on linear SCMs with non-Gaussian noise.Source: §G.3 · p. 39
- For tested linear SCMs with non-Gaussian noise, algorithm performance is slightly worse but still reasonable compared with Gaussian data.The comparison uses one setting from Table 2 and reports results in Table 5.Source: §G.3 · p. 39, Table 5 · p. 39
H Hyperparameters, architectures and further additional details on experiments
This section documents experimental data generation, model and training choices, evaluation, and additional experiments. It also reports practical limits observed for image data and larger architectures.
Data generation
- Synthetic latent DAGs use ER(d, k) graphs, with non-zero edge weights sampled from U(±[0.25, 1.0]); linear mixing matrices have standard-Gaussian entries.For image data, latent coordinates determine ball positions and Pygame renders 64 × 64 × 3 RGB images.Source: H Hyperparameters, architectures and… · pp. 39–40
Models
- Synthetic experiments use a one-hidden-layer, 512-unit LeakyReLU MLP and parameterize W as D(Id − A), learning diagonal D and masked-diagonal A.Sparsity and DAGness penalties apply only to A; image models use small convolutional networks.Source: H Hyperparameters, architectures and… · p. 40
Training
- Training uses an 80–20 train and validation/test split, with observational samples copied across intervention datasets to balance sample counts for contrastive learning.Model selection uses cross-entropy validation loss, while the VAE uses its standard validation loss.Source: H Hyperparameters, architectures and… · pp. 40–41
Evaluation
- MCC measures linear correlations up to component permutation, with the transformation fitted on half the samples and evaluated on the other half.The held-out half is described as out-of-distribution samples.Source: H Hyperparameters, architectures and… · pp. 41–42
Further experimental attempts and dead ends
- Combining the contrastive algorithm with a VAE slightly worsened latent recovery and greatly worsened graph recovery; the authors suspect shared use of W made learning more error-prone.Fixing D = Id produced very similar results, while a sufficiently small shared learning rate was more robust than a larger parametric-part rate.Source: H Hyperparameters, architectures and… · p. 42
- Larger architectures were harder to train without benefit, and image experiments often overfit at smaller sample sizes, even with a frozen pretrained ResNet.The authors suggest careful regularization and model-size tuning may improve image-data performance, particularly sample complexity.Source: H Hyperparameters, architectures and… · p. 42