PaperlayerView exact revision on arXiv

Source-linked AI summary

Relating Graph Neural Networks to Structural Causal Models

Matej Zečević, Devendra Singh Dhami, Petar Veličković, Kristian Kersting

arXiv:2109.04173v3cs.LGstat.ML

TL;DR

The paper asks how GNNs can support causal inference when the underlying SCM is only partially known. It analyzes GNN-SCM connections and introduces neural causal models, establishing results on interventional expressivity and identification while leaving counterfactuals outside its scope.

  • Problem

    Causal inference often relies on only partial knowledge of the SCM, motivating a way to relate causal structure to GNNs’ structured computation.

    Source: §1 · pp. 1–2

  • Method

    The paper develops theoretical connections between GNNs and SCMs, defines GNN interventions, and introduces iVGAE with G-GNN modules for causal inference.

    Source: §4 · p. 8, §5 · p. 11, §7 · p. 12

  • Results

    For any SCM an iVGAE can be L2-consistent, and Q = pC(Vi | do(Vj)) is neurally identifiable if and only if it is identifiable from G and p(V).

    Source: §4 · p. 9, §5 · p. 11

  • Takeaways & Limitations

    The framework links SCM-induced graph structure with computation in GNN layers, as a step toward neural-graph models that are inherently causal.

    Source: §4 · p. 8

    The theoretical results do not cover counterfactuals.

    Source: §4 · p. 10

Abstract

from arXiv · show

Causality can be described in terms of a structural causal model (SCM) that carries information on the variables of interest and their mechanistic relations. For most processes of interest the underlying SCM will only be partially observable, thus causal inference tries leveraging the exposed. Graph neural networks (GNN) as universal approximators on structured input pose a viable candidate for causal learning, suggesting a tighter integration with SCM. To this effect we present a theoretical analysis from first principles that establishes a more general view on neural-causal models, revealing several novel connections between GNN and SCM. We establish a new model class for GNN-based causal inference that is necessary and sufficient for causal effect identification. Our empirical illustration on simulations and standard benchmarks validate our theoretical proofs.

1. Introduction

The paper motivates causal learning by connecting the graph structure used by GNNs to the causal structure represented by SCMs. It examines this connection and introduces a GNN-based causal model for inference.

1. Introduction

  • SCMs encode mechanisms over variables, while GNNs impose an inductive bias on graph-structured inputs, making their relationship relevant to causal learning.The paper frames causal inference as reasoning about Pearl’s Causal Hierarchy when only partial SCM knowledge, such as a causal graph or data, is available.Source: §1 · p. 2
  • The paper derives a theoretical connection between GNNs and SCMs from first principles and defines a more fine-grained neural causal model.Its contributions also include formalizing GNN interventions and establishing theoretical results on feasibility, expressivity, and identifiability.Source: §1 · p. 2
  • The authors examine their theoretical model empirically on causal identification and estimation tasks.The introduction does not report specific benchmark outcomes.Source: §1 · p. 2

2. Background and Related Work

This section reviews variational inference, graph neural networks, and structural causal models as background for the paper’s causal-learning framework.

Variational Inference

  • Variational inference approximates the latent conditional p(Z | X) within a chosen density family and uses a tractable evidence lower bound for optimization.Parameterizing the likelihood and approximate posterior with neural networks yields the variational auto-encoder model class.Source: §2 · pp. 2–3

Graph Neural Networks

  • A GNN uses graph structure as an inductive bias, with message-passing layers aggregating neighborhood information and encompassing convolutional and attentional variants.The section also describes GNN layers as permutation equivariant.Source: §2 · p. 3

Causal Inference

  • An SCM comprises structural equations over endogenous variables and a product distribution over exogenous variables, with parent relationships represented by a causal graph.Interventions replace selected equations; modularity means other mechanisms remain unchanged.Source: §2 · pp. 3–4
  • The Pearl Causal Hierarchy distinguishes associational, interventional, and counterfactual distributions, and the section notes that lower-level data alone generally cannot identify higher-level quantities.SCMs can generate observations, interventional distributions, and counterfactual distributions by simulating their structural equations.Source: §2 · p. 4

3. The GNN-SCM-NCM Connection

The section establishes a conversion between message-passing GNN computations and SCM equations, then uses the connection to define a finer-grained neural causal model. It also explains how shared message functions complicate optimization.

Interventions within GNNs

  • The paper defines GNN interventions as modified layer computations and describes them as local, with each intervened neighborhood a subset of the original neighborhood.It presents intervention as the starting point for relating GNN computations to SCMs.Source: §3 · p. 5

Scope of the Connection

  • The paper leaves counterfactuals uncovered, citing the optimization difficulty of translating shared messages into all local mechanisms.It limits the treatment to interventional queries up to rung 2 of the Pearl Causal Hierarchy.Source: §3 · p. 5

GNN-SCM Conversion

  • For any SCM, a message-passing GNN can be configured so that each node computation equals its corresponding structural equation.The construction uses a scalar decomposition of the structural equations into parent-dependency terms and a residual term.Source: §3 · p. 5
  • The conversion theorem does not address optimization: a shared GNN message function must represent the distinct dependency terms in every structural equation.The paper says this makes optimization difficult for a general causal model based on a GNN.Source: §3 · p. 6

NCM-Type 2

  • NCM-Type 2 relaxes sharedness and models structural-equation dependency terms at the edge level, rather than modeling equations only at the node level.The paper describes this as a more fine-grained formulation than the NCM of Xia et al. (2021).Source: §3 · p. 6
  • Decomposing equations into parent-specific terms can clarify their mechanisms; the paper’s example notes that a linear equation’s residual may depend only on its unmodelled noise term.The authors describe the finer-grained view as potentially more flexible and interpretable, while noting that violating sharedness means the computation is generally no longer a GNN layer.Source: §3 · p. 7

4. GNN-based Causal Inference

The section develops an SCM-based GNN approach focused on interventional queries up to rung 2, then establishes its density-approximation and causal expressivity properties. It also identifies boundaries on the queries the model can support.

4. GNN-based Causal Inference

  • The authors reverse the GNN-to-SCM direction, constructing a GNN from an SCM to target interventional queries up to rung 2.They choose this scope after noting optimization difficulties for more general GNN-based causal models.Source: §4 · p. 8
  • Intervening on a G-GNN and its corresponding SCM produces the same mutilated graph.An intervention in a GNN layer is dual to computing a regular layer on the graph mutilated by that intervention.Source: §4 · p. 8
  • A data-generative VGAE can approximate any smooth density to arbitrary nonzero-error precision, supporting the construction of an interventional VGAE.The proposed iVGAE uses interventional GNN layers for both its inference and generator models.Source: §4 · p. 9
  • For any SCM, an iVGAE exists that is L2-consistent with it, preserving expressivity for interventional distributions and allowing the model to act as a causal-inference proxy.The paper also states that the causal hierarchy is preserved across scenarios.Source: §4 · p. 9
  • An iVGAE cannot be L3-consistent with an SCM, and sufficient expressivity alone does not overcome causal-hierarchy boundaries without causal information such as structural constraints.The paper identifies counterfactuals as beyond the model's capabilities.Source: §4 · p. 10
  • Using G-GNN layers tied to the SCM-induced graph retains L2 consistency while integrating the graph structure with GNN computation more tightly.The paper presents this shared-graph construction as a step toward neural-graph models that are inherently causal.Source: §4 · pp. 8, 10

5. Identifiability and Estimation

The paper defines neural identifiability for iVGAEs and establishes when it matches identifiability from the true graph and observational distribution. It then describes estimation using a causal variational objective.

5. Identifiability and Estimation

  • A causal effect is neurally identifiable from a G-GNN iVGAE exactly when it is identifiable from the true causal graph and observational distribution.The definition requires agreement between the SCM and iVGAE on both the causal effect and observational distribution.Source: §5 · pp. 10–11
  • The iVGAE estimation procedure optimizes a causal ELBO variant that conditions the model on interventions, then estimates quantities from the modeled causal levels.The paper gives ATE as one possible target and describes marginal inference under intervention.Source: §5 · p. 11
  • The paper distinguishes identifiability, the power to infer across causal-hierarchy layers, from identification, the procedure used to obtain estimands.For an iVGAE, it calls estimation the cross-layer modeling process.Source: §5 · p. 11

6. Empirical Illustration

The empirical illustrations examine density estimation on ASIA and ATE estimation across several SCM structures. The reported results indicate good distribution fit and adequate capture of unconfounded and confounded effects.

6. Empirical Illustration

  • The empirical section presents density estimation on ASIA alongside ATE estimation across distinct SCM families.The authors point to an extensive appendix for further study and elaboration.Source: §6 · pp. 11–12
  • On the ASIA dataset, the model estimates observational and interventional distributions sufficiently well in both quality and recall.The density-estimation illustration examines intervention influence along the topological sequence.Source: §6 · pp. 11–12, Figure 4 · p. 12
  • The iVGAE adequately captures unconfounded and confounded causal effects across chain, confounder, and backdoor SCM structures.The comparison concerns ATE estimation across different structures and parameterizations.Source: §6 · p. 12, Figure 4 · p. 12

7. Conclusions and Future Work

The conclusion recaps the theoretical connection and model contributions, then identifies broader causal-hierarchy coverage and larger systems as future directions. The appendices provide proofs and further practical investigations.

7. Conclusions and Future Work

  • The paper derives a first-principles connection between GNNs and SCMs, introduces NCM-Type 2 and iVGAE, and investigates feasibility, expressivity, identifiability, and practical causal inference.Future work proposed includes supporting the complete causal hierarchy without a feasibility trade-off and testing larger causal systems.Source: §7 · p. 12

Appendix for Relating Graph Neural Networks to Structural Causal Models

Appendix A. Proofs

  • Appendix A provides proofs for the mathematical results established in the main paper.Source: Appendix A · p. 17

A.1. Proofs for Theorem 1 and Corollary 1

A general message-passing GNN can represent the structural equations of any SCM under a suitable choice of feature spaces and shared functions. Relaxing sharedness yields a more fine-grained neural causal model that operates at the edge level.

A.1. Proofs for Theorem 1 and Corollary 1

  • For any SCM, suitable feature spaces and shared functions let a message-passing GNN node computation represent each structural equation.The theorem gives an existence result; it does not address optimization or feasibility.Source: §A.1 · p. 17
  • Violating sharedness of ψ yields an NCM-Type 2 that operates on edges rather than at the node level.Its dependency terms correspond to structural-equation dependencies.Source: §A.1 · pp. 17–18
  • The NCM formulation uses uniformly distributed noise and parameterizes structural equations with learnable feedforward neural networks.The passage describes this as a special case of the proposed computation layer.Source: §A.1 · p. 18

A.2. Proof for Proposition 1

An intervention on an SCM and its corresponding G-GNN layer produces the same mutilated graph. Separately, a data-driven VGAE can approximate any smooth density to arbitrary nonzero precision.

A.2. Proof for Proposition 1

  • Intervening on an SCM and its corresponding G-GNN layer produces the same mutilated graph.The intervention removes the links from causal parents to intervened nodes in both representations.Source: §A.2 · p. 18

A.3. Proof for Theorem 2

  • A data-generative VGAE exists whose modelled density can approximate any smooth density at any nonzero error.The proof uses a Gaussian mixture model family that can be represented by a data-driven VGAE.Source: §A.3 · pp. 18–19
  • The density-approximation construction represents a Gaussian mixture through encoder outputs for its weights, means, and covariances.The proof builds the mixture as a convex combination of Gaussian densities.Source: §A.3 · p. 19

A.4. Proofs for Theorem 3 and Corollaries 2 and 3

An iVGAE can be L2-consistent with any SCM, but its compressed description prevents complete SCM reparameterization and L3 consistency. Its consistency notion is weakened, and layer-2 collapse to layer 1 occurs only on a measure-zero subset of SCMs.

A.4. Proofs for Theorem 3 and Corollaries 2 and 3

  • For every SCM, there exists an iVGAE that is L2-consistent with it.The construction models distributions from the first two causal-hierarchy levels when given the corresponding data and sufficient capacity.Source: §A.4 · pp. 19–20
  • The paper’s L_i-consistency notion is weakened: a model may agree with an SCM on only one distribution at a given hierarchy level.Within the first two levels, iVGAE can perform causal inference when supplied with the corresponding data and sufficient model capacity.Source: §A.4 · p. 20
  • The subset of SCMs where iVGAE layer 2 collapses to layer 1 has measure zero.Collapse is defined using selections of level-1 and level-2 distributions.Source: §A.4 · p. 20
  • No iVGAE is L3-consistent with any SCM because its compact description cannot represent the multiple counterfactual worlds required.An SCM specifies structural equations for each variable, whereas iVGAE uses a causal probabilistic model parameterized by two functions.Source: §A.4 · pp. 20–21

A.5. Proof for Corollary 4

An SCM’s induced graph can be used in G-GNN layers without losing L2 consistency. The paper also establishes an iff relation between neural and graph-based identifiability of causal quantities.

A.5. Proof for Corollary 4

  • For any SCM, an iVGAE using G-GNN layers based on its induced graph can remain L2-consistent.The result reduces the search space for such an iVGAE.Source: §A.5 · p. 21

A.6. Proof for Theorem 4

  • A causal quantity is neurally identifiable from an iVGAE with G-GNN modules if and only if it is identifiable from the true graph and observational distribution.The quantity considered is Q = p_C(V_i | do(V_j)).Source: §A.6 · pp. 21–22

Appendix B. Further Experimental Insights and Details

  • The appendix reports further empirical analyses of causal-effect and density estimation and makes the code publicly available.The cited passage describes the scope of the subsequent analyses but does not report their results.Source: Appendix B · p. 22

B.1. Causal Effect Estimation

The experiments evaluate binary-treatment average causal effect estimation with iVGAE across chain, confounder, and backdoor SCMs. The model adequately estimates effects in all three structures, while the analysis distinguishes effect estimation from identification.

B.1. Causal Effect Estimation

  • The target is binary-treatment ATE, defined as E[Y | do(X=1)] − E[Y | do(X=0)]; the authors note that categorical or continuous extensions are possible.They focus on binary structures in this analysis.Source: §B.1 · p. 22
  • The analysis does not address identification: iVGAE is data-dependent and cannot act as a proper SCM, unlike NCM as an SCM reparameterization.Source: §B.1 · p. 22
  • The evaluation uses chain, confounder, and backdoor SCM structures with substantially different information-flow patterns.The structures are distinguished using the d-separation criterion.Source: §B.1 · p. 22
  • Across 27 optimizations, observed ATE estimates approached zero and were bounded in [0, 0.2]; the authors report a 10% worst-case single-approximation error.The experiments cover three structures, three parameterizations per structure, and three random seeds per parameterization.Source: §B.1 · p. 23
  • iVGAE adequately estimates causal effects in the chain, confounder, and backdoor structures, including effects that can be positive, negative, or zero.The backdoor structure is described as the most difficult case; the confounder structure also tests whether the model distinguishes intervention from correlation.Source: §B.1 · p. 23
  • The numerical report compares ELBO and log-likelihood across interventions and SCM structures, and reports validation performance corresponding to test performance.The reported ELBO lower-bounds the marginal log-likelihood.Source: §B.1 · p. 24

B.2. Systematic Investigation on Density Estimation

The systematic investigation tests intervention capture, optimization variability, failure conditions, training duration, intervention count, and parameter tuning. Results show that successful fitting captures intervention propagation, while finite capacity and optimization conditions shape density-estimation quality.

B.2. Systematic Investigation on Density Estimation

  • The investigations use an iVGAE with two interventional GNNs, datasets of 10,000 samples per intervention, and an 80/10/10 train-validation-test split.The reported numerical statistics aggregate 10 random seeds; numerical instability can produce NaN values for a seed.Source: §B.2 · pp. 24, 26
  • When optimization fits the data within available capacity, iVGAE captures changes at the intervention site and their propagation along the causal sequence.The passage presents this as the general observation across evaluated settings, subject to successful optimization and sufficient capacity.Source: §B.2 · p. 25, Figure 5 · p. 27
  • Different random seeds under identical settings can yield substantially different fits because variational optimization is non-convex and guarantees only a local optimum.One seed adequately estimates the quantities of interest, while another overestimates several variables.Source: §B.2 · p. 25
  • When conditioning and intervention coincide, the intervention formulation produces identical predictions; observed density-fitting failures otherwise occur with low capacity, early stopping, or numerical instability.The authors characterize the coincident-conditioning case as a limitation of the formulation, not a failure mode.Source: §B.2 · p. 25, Figure 5 · p. 27
  • Longer training improves model fit: tub and lung are underestimated early and fit upon convergence.Source: §B.2 · p. 26, Figure 6 · p. 28
  • At fixed capacity, increasing the Earthquake training set from 2 to 4 intervention distributions degrades density estimates, with the larger set failing on the illustrated distribution.Source: §B.2 · p. 26, Figure 6 · p. 28
  • Parameter tuning improves fits for smoke, tub, and either; the authors describe tuning as essential for predictive performance.Tuned settings include pooling, layer count, learning rate, and batch size.Source: §B.2 · p. 26, Figure 6 · p. 28
Loading 2109.04173v3…