Learning Functional Causal Models with Generative Neural Networks

1. Introduction

• Predictive models may mistake correlation for causation and attribute changes in Y to Z even when Z does not cause Y.In the described example, inverse regression can explain changes in Y using Z despite the absence of a causal relation from Z to Y.Source: §1 · p. 2
• Experiments are the gold standard for causal discovery, but can be expensive, unethical, or impossible, motivating causal discovery from observations alone.The paper evaluates its observational approach by matching learned structure to known ground truth.Source: §1 · p. 2
• CGNN unifies DAG-based methods and functional causal models by searching generative neural networks for the functional causal model that best fits the data-generating process.The paper reports empirical comparisons with state-of-the-art methods on artificial and real-world benchmarks.Source: §1 · p. 3

2. Problem setting

• A functional causal model is a graph, mechanisms, and noise variables; each mechanism explains a variable from its parents and a noise term.The graph is restricted to DAGs, with intervention propagation assumed instantaneous, a setting suited to cross-sectional studies.Source: §2 · p. 3

2.1 Notations

• The notation distinguishes direct causal edges, undirected unresolved edges, and associations from external common causes; a CPDAG represents a Markov equivalence class.The section also defines conditional independence, v-structures, and graph skeletons.Source: §2.1 · p. 3, Figure 1 · p. 4

2.2 Assumptions and Properties

• Under the stated causal sufficiency, Markov, and faithfulness assumptions, an adjacent pair with a nonadjacent, conditionally dependent pair identifies a v-structure.For an FCM, the causal Markov assumption holds when the graph is a DAG and error terms are mutually independent.Source: §2.2 · p. 4

3. State of the art

• Causal-structure methods use DAG Markov properties, such as v-structures, or asymmetries in pairwise joint distributions.Source: §3 · p. 4

3.1 Learning the CPDAG

• Conditional-independence tests identify v-structures but can leave edges unoriented when multiple DAGs encode the same independences, yielding a CPDAG.In the example, the edges between X1–X2 and X1–X3 remain undirected because alternative Markov-equivalent DAGs fit the same conditional independences.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.1.1 CONSTRAINT-BASED METHODS

• PC builds a skeleton, identifies v-structures, and propagates orientations; its accuracy depends on CI tests, while propagation can introduce errors.Identifiability proofs generally assume an oracle with perfect CI knowledge, effectively infinite training data; CI testing can also require exponentially more data as variable count grows.Source: §3.1.1 · pp. 5–6

3.1.2 SCORE-BASED METHODS

• GES searches CPDAGs by adding and removing edges to optimize a global score, and under stated assumptions identifies the true CPDAG in the large-sample limit.FGES uses the same scores and search algorithm as GES but speeds it up with cached score information.Source: §3.1.2 · p. 6

3.1.3 HYBRID ALGORITHMS

• Hybrid algorithms restrict score-based search using conditional-independence tests, or combine an FGES graph sketch with FCI orientation rules for potential confounders.MMHC first builds a skeleton using CI tests, then uses Bayesian-scoring greedy search to orient edges.Source: §3.1.3 · p. 6

3.2 Exploiting asymmetry between cause and effect

• Score-based and constraint-based methods may miss distributional asymmetries induced by causal direction.Source: §3.2 · p. 6

3.2.1 THE INTUITION

• With additive independent noise and non-Gaussian variables, the forward regression residual can be independent of X while the reverse residual is not, supporting X →Y.The example illustrates this residual asymmetry for a generated joint distribution.Source: §3.2.1 · p. 6, Figure 3 · p. 7
• In linear Gaussian models, the BIC score balances residual likelihood against graph complexity, using k = 2p+1 and default penalty multiplier c = 2.Source: §3.2.1 · p. 6

3.2.2 RESTRICTION ON THE CLASS OF CAUSAL MECHANISMS CONSIDERED

• LiNGAM identifies linear structural equation models when source-node distributions are non-Gaussian, while causal identification generally requires assumptions such as non-Gaussianity or additive noise.Source: §3.2.2 · p. 7

3.2.3 PAIRWISE METHODS

• Additive noise models identify the direction except for the linear Gaussian input-and-noise case; post-nonlinear models are more general but have more non-identifiable cases.Source: §3.2.3 · p. 7
• Gaussian-process inference compares Bayesian generative models for each direction, while another approach uses independence between cause distribution and causal mechanism.Source: §3.2.3 · pp. 7–8
• Causality classifiers depend on representative training pairs, and pairwise asymmetry methods can miss edge directions that multivariate CI methods identify in linear Gaussian v-structures.Bivariate methods can orient edges independently, so an error on one edge does not propagate to the rest of the graph, but they do not use all dependence information.Source: §3.2.3 · p. 8

• Earlier hybrid methods combine constraint-based discovery with bivariate edge orientation, but require a final check for consistency between v-structures and orientations.The cited PNL approach also has super-exponential cost in the number of variables.Source: §3.3 · p. 8
• CGNN unifies graph-based and pairwise approaches to causal modeling in one framework.The goal is to combine the strengths of CPDAG and bivariate methods.Source: §3.3 · p. 8
• Unlike CAM, CGNN allows non-additive noise and non-additive contributions of causes to model flexible conditional distributions.Source: §3.3 · p. 8
• CGNN learns multivariate causal mechanisms as generative neural networks and fits the joint distribution using Maximum Mean Discrepancy.The paper contrasts this score with likelihood-based and mean-square criteria.Source: §3.3 · p. 9
• The framework is presented for learning from a known skeleton and is later extended to handle possible hidden confounding factors.Source: §3.3 · p. 9

• The setting assumes continuous variables whose joint density is continuous, strictly positive on a compact subset of R^d, and zero elsewhere.Source: §4 · p. 9
• The section first models continuous FCMs with a given graph structure, then evaluates candidate models, and finally learns a best candidate from observational data.Source: §4 · p. 9
• Model construction, candidate evaluation, and candidate learning form the three stages of the section's treatment.Source: §4 · p. 9

4.1 Modeling continuous FCMs with generative neural networks

• When the density is continuous and strictly positive on a compact convex subset of R^d and zero elsewhere, a DAG factorization admits continuous FCM mechanisms that reproduce the distribution.The mechanisms have compact support, with independent uniform noise on [0,1].Source: §4.1 · p. 9
• A CGNN pairs a DAG with neural-network causal mechanisms and mutually independent noise variables drawn from the same distribution.Source: §4.1 · pp. 9–10
• The paper states that a CGNN defines a continuous FCM.Source: §4.1 · p. 10

4.1.1 GENERATIVE MODEL AND INTERVENTIONS

• CGNN generates samples by drawing independent noise and applying each mechanism in graph topological order using its parents' values.Source: §4.1.1 · p. 10
• For an intervention, CGNN clamps the selected variable and discards its incoming causal influences, producing an interventional distribution distinct from conditioning.Source: §4.1.1 · p. 10

4.2.1 SCORING METRIC

• The candidate-model score combines MMD between observed and generated samples with a penalty proportional to the candidate graph's edge count.The edge penalty supports fair comparison across structures.Source: §4.2.1 · p. 11
• The standard MMD statistic has quadratic sample-size complexity; the experiments also use a linear approximation based on 100 random features.Source: §4.2.1 · p. 11

4.2.2 REPRESENTATIONAL POWER OF CGNN

• When the observational distribution factorizes along a DAG and meets the stated density conditions, a CGNN can approximate it with arbitrary accuracy.Source: §4.2.2 · p. 11
• With an infinite observational sample, the score can approach zero and the generated distributions converge to the observed distribution; this applies to continuous FCMs without requiring additive noise or linear mechanisms.Source: §4.2.2 · pp. 11–12
• Observational fit does not identify a unique DAG when multiple DAGs in a Markov equivalence class factorize the same distribution, including both directions for a two-variable pair.The number of hidden neurons is crucial: too few may miss data patterns, while too many may retain overly complicated mechanisms.Source: §4.2.2 · p. 12

4.3.1 PARAMETRIC (WEIGHT) OPTIMIZATION

• Neural functions for a fixed graph are trained end-to-end with backpropagation and Adam by minimizing an MMD loss, with noise resampled during training.The procedure otherwise resembles supervised continuous learning, except that it minimizes MMD rather than mean squared error.Source: §4.3.1 · p. 12

4.3.2 NON-PARAMETRIC (STRUCTURE) OPTIMIZATION

• Because the DAG search space grows super-exponentially, CGNN starts from a recovered skeleton and searches its admissible edge orientations, a problem of complexity O(2L).Here, L is the number of skeleton edges; the resulting graph must remain a DAG.Source: §4.3.2 · p. 13
• The search scores each edge direction separately, removes cycles, then uses global-score hill climbing that reverses edges and retrains CGNN until a local optimum.The authors use hill climbing as a proof of concept, citing its trade-off between computational time and accuracy.Source: §4.3.2 · p. 13

4.3.3 IDENTIFIABILITY OF CGNN UP TO MARKOV EQUIVALENCE CLASSES

• With infinite observational data and a generating distribution in the CGNN class, a DAG in the true graph’s Markov equivalence class can attain MMD zero; same-adjacency graphs outside it cannot.The proposition assumes the distribution is Markov and faithful to the generating graph, with d-connected variable pairs dependent.Source: §4.3.3 · pp. 13–14
• The result leaves identifiability within a Markov equivalence class unresolved; experiments indicate that recovering a directed graph in that class requires controlling model capacity.Source: §4.3.3 · p. 14

5.1 Experimental setting

• CGNN uses a one-hidden-layer ReLU network; experiments vary hidden-neuron count because it is a sensitive hyperparameter that controls model identifiability.The preliminary sensitivity study uses 1,500 samples from a linear additive-noise model and compares generated samples and scores across neuron counts.Source: §5.1 · p. 14, Figure 5 · p. 15, Figure 6 · p. 15

5.2 Learning bivariate causal structures

• CGNN and GPI perform well on most cause-effect datasets, including real-world CE-Tueb, at higher computational cost; simple regression underfits, and ANM-HSIC works when additive noise holds.The bivariate task compares scores for the two possible causal directions.Source: §5.2 · pp. 14, 16

5.3 Identifying v-structures

• Across three linear-Gaussian triplet datasets, CGNN scores clearly distinguish the V-structure from other structures, which are Markov equivalent and difficult to distinguish.The experiments support CGNN’s ability to detect and use conditional independence.Source: Table 1 · p. 17, §5.3 · p. 18

5.4 Multivariate causal modeling under Causal Sufficiency Assumption

• For multivariate orientation experiments, algorithms receive the true graph skeleton, separating edge orientation from skeleton discovery under causal Markov, faithfulness, and causal sufficiency assumptions.The artificial-graph experiments use polynomial mechanisms with additive or multiplicative noise.Source: §5.4 · p. 18, §5.4.1 · p. 18

5.4.1 RESULTS ON ARTIFICIAL GRAPHS WITH ADDITIVE AND MULTIPLICATIVE NOISES

• With the true skeleton, CGNN significantly outperforms other algorithms on SHD and SID; PC-HSIC ranks second among the compared methods.CGNN’s resulting graph has three mistakes, consistent with an average SHD of 2.5.Source: §5.4.1 · p. 18, Figure 9 · p. 19
• With 20% skeleton error, CGNN is more robust than PC-HSIC, while CAM achieves better SHD; on five 100-variable graphs, CGNN reaches AUPRC 85.5±4 in 30 hours.CGNN and PC-HSIC are among the most computationally expensive methods in the 20-variable experiments.Source: §5.4.1 · pp. 18–19

5.4.2 RESULT ON BIOLOGICAL DATA

• CGNN obtains the best AUPR, SHD, and SID on SynTReN graphs with 20 and 50 nodes, where few v-structures limit constraint-based methods.CAM also performs well by using distributional asymmetry and multivariate interactions.Source: §5.4.2 · p. 20, Figure 11 · p. 20

5.4.3 RESULTS ON BIOLOGICAL REAL-WORLD DATA

• On the Sachs protein network, CGNN recovers the reported raf→mek→erk pathway, while confidence scores may help control false causal discoveries.Constraint-based algorithms score poorly on this graph, where the authors find little evidence for several tested v-structures.Source: §5.4.3 · pp. 20–21

6. Towards predicting confounding effects

• For each skeleton edge, the extension considers either causal direction or a bidirected link representing two variables affected by common hidden causes.The approach retains the causal Markov and faithfulness assumptions.Source: §6 · p. 22, §6.1 · p. 22

6.1 Principle

• Pair-specific correlated noise summarizes unknown influences of hidden variables, allowing direct causal effects to persist alongside confounding.The example considers a missing cause that induces correlated noise between two observed variables.Source: §6.1 · p. 22
• The DAG G and dependency skeleton S are distinct: correlated noise can preserve S when a variable is removed from G, and the neural mechanisms remain learnable end-to-end.Graph search modifies skeleton edges by reversing, adding, or removing them and optimizes a penalized loss.Source: §6.1 · pp. 22–23

6.2 Experimental validation

• CGNN is reported robust to confounders and competitive with RFCI-HSIC; effective visible-variable causal links score highly, while latent-variable spurious links score low or are removed.Table 2 reports AUPR, SHD, and SID, but the supplied passages do not give numerical values.Source: Table 2 · p. 23, §6.2 · p. 23

7. Discussion and Perspectives

• CGNN is introduced as a framework for functional causal model learning that uses the non-parametric flexibility of generative neural networks.Source: §7 · p. 24
• Once learned, its fully parametrized models can simulate interventions on one or more variables and evaluate impacts on target variables.The paper identifies medical and sociological domains as relevant applications.Source: §7 · p. 24
• The main limitation is computational cost: the MMD learning criterion has quadratic complexity in data size, while a proposed linear approximation has comparable empirical performance.Source: §7 · p. 24
• Further work is directed toward scaling from medium- to large-sized problems and extending the approach to categorical variables.Source: §7 · p. 24

8.1 The Maximum Mean Discrepancy (MMD) statistic

• For a characteristic kernel, empirical MMD tends to zero exactly when the compared distributions are equal; its direct evaluation takes O(n^2) time, with a shift-invariant-kernel approximation evaluated in O(mn) time.The paper uses a Gaussian kernel and describes a linear-time approximation based on Bochner’s theorem.Source: §8.1 · p. 28

8.2 Proofs

• Under the stated density assumptions, a continuous FCM can generate the distribution, and CGNN networks can approximate its mechanisms to arbitrary error.The proof uses universal approximation and proceeds in topological order.Source: §8.2 · pp. 28–29
• For a faithful distribution, a candidate DAG with the same adjacencies but outside the true graph’s Markov equivalence class cannot match its distribution under the stated assumptions.The argument distinguishes graphs through differing v-structures and conditional-independence relations.Source: §8.2 · p. 30

8.3 Table of scores for the experiments on cause-effect pairs

• The appendix lists cause-effect AUPR results across five benchmarks and graph-orientation results for artificial, Syntren, and real-protein networks under specified skeleton settings.The captions describe table scope but provide no score values in the supplied passages.Source: Table 3 · p. 31, Table 4 · p. 31, Table 5 · p. 32