PaperlayerView exact revision on arXiv

Source-linked AI summary

The Causal-Neural Connection: Expressiveness, Learnability, and Inference

Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, Elias Bareinboim

arXiv:2107.00793v3cs.LGcs.AI

TL;DR

The paper asks whether universal neural expressiveness suffices to learn causal models from data, especially when intervention effects must be inferred from observational data. It introduces neural causal models with structural inductive biases, showing that expressiveness alone does not ensure cross-layer inference while constrained NCMs support identification and estimation.

  • Problem

    It remains largely unknown how neural generative models can perform causal identification and estimation in arbitrary settings while preserving valid inferences.

  • Method

    The paper introduces trainable neural causal models and encodes causal-diagram invariances as structural inductive biases for gradient-based causal inference.

  • Results

    NCMs are as expressive as SCMs but cannot generally infer higher causal-hierarchy layers from lower-layer data; structural constraints enable identification and estimation when supported.

  • Takeaways & Limitations

    Universal approximation does not solve causal learning: valid neural causal inference requires structural assumptions beyond observational fit.

  • Takeaways & Limitations

    Cross-layer inference remains valid only under stringent conditions, including Layer-1 consistency and the required causal-diagram constraints.

Abstract

from arXiv · show

One of the central elements of any causal inference is an object called structural causal model (SCM), which represents a collection of mechanisms and exogenous sources of random variation of the system under investigation (Pearl, 2000). An important property of many kinds of neural networks is universal approximability: the ability to approximate any function to arbitrary precision. Given this property, one may be tempted to surmise that a collection of neural nets is capable of learning any SCM by training on data generated by that SCM. In this paper, we show this is not the case by disentangling the notions of expressivity and learnability. Specifically, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020), which describes the limits of what can be learned from data, still holds for neural models. For instance, an arbitrarily complex and expressive neural net is unable to predict the effects of interventions given observational data alone. Given this result, we introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences. Building on this new class of models, we focus on solving two canonical tasks found in the literature known as causal identification and estimation. Leveraging the neural toolbox, we develop an algorithm that is both sufficient and necessary to determine whether a causal effect can be learned from data (i.e., causal identifiability); it then estimates the effect whenever identifiability holds (causal estimation). Simulations corroborate the proposed approach.

1 Introduction

Neural networks’ universal expressiveness motivates testing whether they can learn causal structure, but observational data may not determine intervention effects. The paper frames this gap through SCMs, the Pearl Causal Hierarchy, and causal identification and estimation.

  • 1 Introduction: Neural models can approximate arbitrary functions with sufficient capacity, motivating investigation of whether they can support causal reasoning.The paper contrasts this broad function-approximation ability with the requirements of causal inference.
  • 1 Introduction: A structural causal model represents system mechanisms and exogenous variation, and induces the Pearl Causal Hierarchy of seeing, doing, and imagining.These layers correspond to observational, interventional, and counterfactual queries.
  • 1 Introduction: When the true SCM is only partially observed, observational layer-1 data may be insufficient to recover higher-layer causal quantities.This is the setting in which a model must predict intervention effects from passively collected data.
  • 1 Introduction: Multiple neural models can agree on the observational distribution while producing different interventional or counterfactual distributions.Thus, expressive representation of the true SCM does not guarantee that training selects the correct higher-layer behavior.
  • 1 Introduction: The paper targets causal identification and estimation, with neural methods intended to encode causal assumptions while remaining suitable for gradient-based optimization.Identification asks whether a causal query is uniquely determined; estimation follows when identifiability holds.

2 Neural Causal Models and the Causal Hierarchy Theorem

The paper introduces neural causal models as expressive, trainable SCMs, then shows that expressiveness alone does not resolve cross-layer causal inference. Causal-diagram constraints provide the structural inductive bias needed to preserve interventional expressiveness and support valid inference.

  • 2 Neural Causal Models and the Causal Hierarchy Theorem: Neural causal models are SCMs whose mechanisms are feedforward neural networks with uniform exogenous sources, allowing gradient-based optimization and possible unobserved confounding.Their parameterization constrains mechanisms and source distributions while retaining an SCM structure.
  • 2 Neural Causal Models and the Causal Hierarchy Theorem: NCMs are as expressive as SCMs: for any SCM, an NCM exists that is consistent through the observational, interventional, and counterfactual layers.The expressiveness theorem establishes no loss of representational power from the NCM form.
  • 2 Neural Causal Models and the Causal Hierarchy Theorem: Despite this expressiveness, the subset where higher causal-hierarchy layers are determined by lower layers has measure zero for NCMs.Matching observed behavior therefore does not generally determine intervention or counterfactual behavior.
  • 2.1 A Family of Neural-Interventional Constraints (Inductive Bias): A causal diagram supplies an inductive bias by constraining the NCM’s exogenous-source structure and neural mechanisms to encode interventional invariances.The construction uses confounded components of the diagram to define shared exogenous sources and corresponding neural mechanisms.
  • 2.1 A Family of Neural-Interventional Constraints (Inductive Bias): Every G-constrained NCM is G-consistent, and such models can represent the full intervention layer of the original SCM.Constraining NCMs to the causal diagram preserves Layer 2 expressiveness while encoding the relevant structural assumptions.

3 The Neural Identification Problem

The neural identification problem asks whether an interventional effect is uniquely determined by observational data within graph-constrained neural causal models. The paper proves neural and graphical identifiability coincide and gives a necessary-and-sufficient optimization procedure for deciding and returning identifiable effects.

  • Neural-identifiability requires every graph-constrained NCM matching the observational distribution to agree on P(y | do(x)).
  • Graphical identifiability and neural identifiability are equivalent, so restricting identification to NCMs loses no generality.
  • When an effect is identifiable, mutilating a graph-constrained proxy NCM by replacing intervention equations with constants computes the correct query.
  • For Markovian NCMs, every interventional distribution is identifiable through proxy-NCM mutilation.
  • Algorithm 1 searches for NCMs that minimize and maximize the target query while remaining observationally consistent, declaring identification when their query values agree.
  • The procedure is sound and complete: it returns the true causal effect exactly when the query is identifiable, and otherwise returns FAIL.

4 The Neural Estimation Problem

The neural estimation problem adapts graph-constrained NCMs to finite samples and imperfect optimization. Training balances observational likelihood with maximization or minimization of the target interventional distribution.

  • Finite-sample estimation uses a graph-constrained NCM whose parameters are trained under observational consistency while optimizing the target intervention.
  • The construction extends beyond binary variables to categorical and continuous settings through alternative density models.
  • The NCM construction represents latent confounding and graph structure with neural mechanisms, latent component variables, and Gumbel random variables.
  • The training objective combines negative log-likelihood with a weighted term that maximizes or minimizes the estimated interventional probability.
  • Algorithm 1 compares the optimized minimum and maximum queries, using a threshold-based hypothesis test in practice to classify identification.

5 Experiments

Experiments test NCM-based identification on eight causal graphs and estimation on four identifiable graphs. NCMs distinguish identifiable from non-identifiable effects and estimate causal effects more reliably than a naïve observational model.

  • The experiments evaluate identification using 3,000 training epochs and estimation using KL divergence and ATE metrics.
  • Across eight graph cases, identifiable gaps decrease toward zero while non-identifiable gaps remain high, with non-ID classification accuracy staying high throughout training.
  • NCMs improve observational fit with more data and learn the correct ATE, whereas the naïve model fails except when P(y | do(x)) = P(y | x).
  • NCM estimation is competitive with WERM, which directly targets causal-effect estimation without generating samples.

6 Conclusions

The paper introduces neural causal models as expressive, trainable SCMs with structural inductive bias for causal inference. Its theory and algorithms establish neural solutions for identification and estimation while preserving the relevant causal constraints.

  • NCMs are trainable SCMs that retain full expressivity while encoding structural constraints over interventional distributions.
  • The framework separates expressivity from learnability, showing that unrestricted neural models cannot generally perform cross-layer causal inference.
  • The paper provides sound and complete procedures for causal identification and estimation within graph-constrained NCMs.

A.1 Proofs of Theorem 1 and Corollary 1

These proofs establish that neural causal models can represent SCMs expressively, including their counterfactual behavior, while retaining causal-hierarchy limitations on what observational data can determine.

  • Canonical construction: Canonical SCMs enumerate possible parent-to-variable mechanisms and partition exogenous assignments according to those mechanisms.This canonicalization makes the structural functions and exogenous variation suitable for neural construction.
  • Neural representation: Binary-step MLPs represent arbitrary functions on binary variables and finite numerical domains, enabling neural implementations of SCM mechanisms.The proofs use logical components such as negation, OR, AND, and one-hot encodings to construct equivalent networks.
  • Expressiveness: An NCM can be constructed to match any SCM on counterfactual distributions, establishing the model class’s expressiveness.Theorem 1 provides the existence result, while the canonical-SCM construction supplies the intermediate representation.
  • Causal-hierarchy limitation: Despite this expressiveness, the causal hierarchy theorem implies that higher-layer distributions generally do not collapse to lower layers for NCMs.The collapse subset has measure zero, so observational agreement generally does not determine interventional or counterfactual behavior.

A.2 Proof of Theorem 2

The proof shows that constraining an NCM by a causal diagram preserves the diagram exactly and therefore enforces the diagram’s interventional constraints.

  • Interventional constraints: Causal diagrams encode qualitative constraints on interventional distributions, making them useful structural information for cross-layer inference.The causal Bayesian network connection supplies the link between the diagram and layer-2 distributions.
  • Graph construction: The induced graph of a G-constrained NCM equals G because neural-function arguments reproduce directed edges and shared exogenous variables reproduce bidirected edges.Independent exogenous variables prevent additional bidirected edges beyond those represented through shared latent components.
  • Graphical consistency: Any G-constrained NCM is G-consistent, so its induced causal diagram satisfies the structural constraints encoded by G.This follows from matching vertices and directed and bidirected edges between G and the NCM-induced graph.

A.3 Proof of Theorem 3

The proof constructs a G-constrained NCM that matches any SCM on layer-2 interventional distributions by combining discrete SCM representations with neural mappings.

  • Layer-2 representation: For any SCM inducing G, there exists a G-constrained NCM that is layer-2 consistent with it.The construction first uses a finite discrete SCM with the same graph and layer-3 behavior, then realizes its mappings with MLPs.
  • Neural construction: The construction assigns one uniform exogenous variable to each C-component and neuralizes mappings from those variables to the original exogenous and endogenous variables.MLPs represent the finite-domain mappings, while shared component variables preserve the required confounding structure.
  • Interventional equivalence: The resulting NCM reproduces every interventional query of the target SCM, including the target distribution P_M*(y | do(x)).The proof identifies the NCM’s interventional event with the corresponding event in the discrete representation and then with the original SCM.

A.4 Proofs of Theorem 4 and Corollaries 2 and 4

These results establish graphical-neural equivalence: neural identifiability matches graphical identifiability, and a proxy NCM yields correct interventions whenever the query is identifiable.

  • Graphical-neural equivalence: A causal query is neural identifiable from G and P(V) if and only if it is identifiable from the graph and observational distribution.Thus, restricting the hypothesis space to G-constrained NCMs neither expands nor reduces the identifiability criterion.
  • Non-identifiability: If the query is not graphically identifiable, two G-constrained NCMs can agree observationally while disagreeing on the query.The construction transfers the non-identifiability witness from SCMs to neural causal models.
  • Operational identification: When a query is identifiable, intervening on any observationally consistent G-constrained NCM computes the correct causal effect through neural mutilation.Replacing the equation for X with the constant x and evaluating Y produces the target query.
  • Identification algorithm: The identification algorithm is sound and complete: it returns FAIL exactly when the query is not identifiable, otherwise returning the true causal effect.The corollary states that a non-FAIL result equals P_M*(y | do(x)).

A.5 Proof of Corollary 3

Under Markovianity, causal effects are identifiable through the backdoor adjustment formula, and the same guarantee extends to G-constrained neural causal models via mutilation.

  • Markovian SCMs have independent exogenous variables and no shared confounders, corresponding to causal diagrams without bidirected edges.
  • In Markovian models, every P(y | do(x)) is identifiable using the backdoor adjustment over non-descendants of X.The adjustment is given by P(Y | x, z)P(z), where Z contains variables unaffected by X.
  • The same identifiability result holds for Markovian G-constrained NCMs through mutilation of the proxy model.
  • The proof connects graph-based identification to neural identification through the equivalence between the proxy NCM and the observed distribution.

B.1 Data Generation Process

The experiments generate expressive canonical SCMs, train NCMs and comparison models on observational data, and evaluate identification and estimation across graph and sample-size settings.

  • Data Generation Process: Canonical SCMs with random parameters generate data models capable of exhibiting any behavior consistent with each graph’s constraints.
  • Identification Experiments: Identification experiments use eight graphs, 10,000 binary observations, paired minimizing and maximizing NCMs, and max-min gaps to classify queries.
  • Estimation Experiments: Estimation experiments vary sample sizes from 10^3 to 10^6 across four identifiable graphs and compare NCMs with naïve likelihood models and WERM.
  • Estimation Experiments: The experiments plot KL divergence and ATE means with 95%-confidence intervals across 25 trials for each sample-size setting.
  • Identification Hypothesis Testing: The hypothesis test declares a query identifiable when the mean max-min gap is below τ with 95% confidence; otherwise it returns non-identifiable.
  • Identification Hypothesis Testing: Across aggregated settings, most identifiable-case gaps fall below τ = 0.03, while most non-identifiable-case gaps remain above it by training’s end.

C.1 Expressiveness Examples

The expressiveness examples construct an NCM that matches a complex SCM across causal layers, while showing that observational agreement alone does not determine intervention effects.

  • Example 1: The diet–blood-pressure example defines an SCM with binary variables, exogenous confounding, and layer-specific observational, interventional, and counterfactual quantities.
  • NCM Construction: The constructed NCM uses neural implementations of logical functions and interval-based noise encoding to reproduce the SCM’s exogenous structure.
  • Expressiveness Result: The NCM reproduces the SCM distributions and answers the example’s layer-1, layer-2, and layer-3 queries, demonstrating expressiveness across all three layers.
  • Expressiveness Versus Learnability: Different NCMs can induce the same observational distribution while disagreeing on the intervention effect P(B = 1 | do(D = 1)).
  • Expressiveness Versus Learnability: Those observationally equivalent models can imply beneficial, null, or harmful conclusions about a high-vegetable diet’s effect on blood pressure.
  • Structural Constraints: Causal diagrams encode structural constraints that can eliminate observationally compatible models and support cross-layer inference.

C.3 Solving Identification through NCMs

The paper frames causal identification as agreement among structurally constrained NCMs consistent with the observed distribution and graph. It shows that expressive but misspecified model classes can falsely declare effects identifiable, while NCM-based and hybrid procedures connect identification with estimation.

  • C.3 Solving Identification through NCMs: NCM identification checks whether all G-consistent, observationally matching NCMs agree on the target intervention.This operationalizes identification through constrained proxy SCMs rather than direct recovery of the unobserved true SCM.
  • C.3 Solving Identification through NCMs: NCM structural constraints can recover an identifiable query even when the proxy is fitted only to observational data.The paper reports that a constrained NCM matches the layer-2 query despite being constructed to match layer 1.
  • C.3 Solving Identification through NCMs: Markovian models can falsely imply identifiability: they yield 0.375, whereas the true interventional value is 0.46875.The mismatch arises because Markovian models exclude unobserved confounding and are not expressive enough to represent all SCMs.
  • C.4 Symbolic versus Optimization-based Approaches for Identification: Optimization-based identification searches structural models while maximizing or minimizing the target query under data and causal constraints.This approach is linked to symbolic identification through a duality result.
  • C.4 Symbolic versus Optimization-based Approaches for Identification: Optimization performance depends on causal assumptions, accurate optimization, and potentially large amounts of training data and computation.This limitation follows from the sampling nature of the optimization procedure.
  • C.4 Symbolic versus Optimization-based Approaches for Identification: The integrated NCM pipeline performs identification and estimation together, whereas a hybrid alternative uses symbolic identification followed by NCM estimation.The hybrid approach combines deterministic identification with neural estimation for potentially complex settings.

D.1 NCMs with other Functions and Noise Distributions (Proofs)

The generalized NCM construction permits broader noise distributions and function classes while preserving the representational results needed for causal analysis. The proofs use probability transformations and function constructions to establish consistency and expressiveness.

  • D.1 NCMs with other Functions and Noise Distributions (Proofs): General NCMs replace fixed uniform noise and feedforward functions with broader distributions and function classes satisfying stated properties.The framework treats the uniform-noise, MLP version as one implementation rather than an essential limitation.
  • D.1 NCMs with other Functions and Noise Distributions (Proofs): The probability integral transform supplies uniform latent variables from arbitrary variables with well-defined densities.This fact supports constructing equivalent NCM noise variables for the generalized proofs.
  • D.1 NCMs with other Functions and Noise Distributions (Proofs): For any SCM, a generalized NCM can be constructed that is L3-consistent with the original model.The proof constructs a canonical SCM and realizes its components using the NCM architecture.
  • D.1 NCMs with other Functions and Noise Distributions (Proofs): The generalized proofs preserve the remaining NCM results with only minimal changes.This extends the framework beyond the specific implementation choices used in the initial definition.
  • D.1 NCMs with other Functions and Noise Distributions (Proofs): For any SCM inducing graph G, a G-constrained NCM exists that is L2-consistent with that SCM.The construction uses one latent variable per C2-component and neural functions with graph-specified parents.

D.2 Pearl’s Causal Hierarchy and Other Classes of Models

The causal hierarchy distinguishes models by which observational, interventional, and counterfactual distributions they can represent. Although neural and wrapper models may be expressive, higher-layer inference requires structural constraints that make practical cross-layer reasoning possible.

  • Expressiveness: NCMs are L3-expressive, matching SCMs across observational, interventional, and counterfactual distributions despite neural structural constraints.Theorem 1 states that every SCM has an NCM that is L3-consistent, so NCMs retain SCM expressive power.
  • Causal hierarchy: Generative models fitted to observational distributions can generate P(V), but they cannot generally produce interventional distributions such as P(V_x).Generative model classes that are only L1-expressive are not defined to evaluate higher-layer causal distributions.
  • Expression wrappers: An L2-expression wrapper can represent every intervention by fitting a separate generative model for each L2 distribution, but this is impractical because interventions grow exponentially or become infinite for continuous variables.The wrapper is L2-expressive, yet requires too many distributions to learn and manage in practical settings.
  • Structural constraints: Expression wrappers lack an automatic way to encode causal Bayesian-network constraints, so observational agreement may leave the corresponding interventional distribution unconstrained.Given P(V), the wrapper could produce a P(V_x) consistent with x but different from the true SCM’s P(V_x).
  • Structural constraints: Maintaining all cross-layer constraints while fitting desired distributions is infeasible for tractable optimization, whereas NCMs incorporate structural constraints directly.Even with three binary variables, enumerating the constraints is lengthy; NCMs are designed to encode them structurally.

E Frequently Asked Questions

The FAQs clarify why observational data and universal approximators do not suffice for causal inference, and explain how NCMs encode constraints while supporting identification and estimation.

  • Expressivity and learnability: Universal approximation does not identify interventions: a model can match observational distributions while differing from the true SCM on interventional distributions.This is why searching for an SCM that fits observational data generally cannot recover causal effects.
  • Neural causal models: NCMs are proxy SCMs whose explicit functions and noise distributions can be optimized, but they require G-consistency and L1-consistency to support valid interventional inferences.Without G-consistency, an NCM does not ascertain the causal constraints represented by the diagram.
  • Assumptions: The causal diagram assumption is necessary in general because cross-layer inference and even recovery of the full diagram cannot be obtained from observational data alone.The work assumes the true SCM is recursive, while the general NCM definition need not impose acyclicity a priori.
  • Neural causal models: Thm. 3 shows that G-constrained NCMs retain expressiveness for every SCM compatible with G, despite the nontriviality of enforcing graph constraints.This preserves their use as proxy models for causal inference rather than merely flexible layer-2 generators.
  • Identification and estimation: The framework uses neural models to solve the full causal pipeline: determine identifiability, then estimate the query when identification succeeds.Alg. 1 addresses identification, while optimization jointly fits the observational distribution and the query of interest.
  • Limitations: The paper makes no formal robustness claims for training error, reporting empirical accuracy when L1-consistency is nearly achieved and leaving refined analysis for future work.The stated boundary concerns error in training rather than the expressiveness of the NCM class.
Loading 2107.00793v3…