PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Neural Causal Models from Unknown Interventions

Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C. Mozer, Chris Pal, Yoshua Bengio

arXiv:1910.01075v2stat.MLcs.AIcs.LG

TL;DR

Observational data alone has limited causal identifiability, and incorporating interventional data—especially with unknown targets—into continuous optimization is challenging. The paper introduces SDI, a neural continuous-optimization framework for combining both data types and evaluating complete and partial graph recovery. Experiments report strong recovery across synthetic and real-world tasks, including generalization to unseen interventions and favorable comparisons with causal discovery methods.

  • Problem

    Observational data alone generally identifies causal graphs only up to a Markov equivalence class, while extending continuous-optimization methods to interventional data is not straightforward.

  • Method

    SDI jointly trains neural functional models and continuous structural parameters using alternating observational fitting, interventional graph scoring, and structural updates.

  • Results

    The method reports strong graph recovery across synthetic and real-world tasks, favorable comparisons with causal methods, generalization to unseen interventions, and scaling to approximately 50 variables for partial recovery.

  • Takeaways & Limitations

    SDI supports causal structure discovery from combined observational and interventional data when intervention targets are unknown and when part of the graph is already known.

  • Takeaways & Limitations

    The method assumes fully observed discrete variables, sparse soft interventions, and a queryable black-box SCM; unknown-target interventions lack a theoretical identifiability guarantee.

Abstract

from arXiv · show

Promising results have driven a recent surge of interest in continuous optimization methods for Bayesian network structure learning from observational data. However, there are theoretical limitations on the identifiability of underlying structures obtained from observational data alone. Interventional data provides much richer information about the underlying data-generating process. However, the extension and application of methods designed for observational data to include interventions is not straightforward and remains an open problem. In this paper we provide a general framework based on continuous optimization and neural networks to create models for the combination of observational and interventional data. The proposed method is even applicable in the challenging and realistic case that the identity of the intervened upon variable is unknown. We examine the proposed method in the setting of graph recovery both de novo and from a partially-known edge set. We establish strong benchmark results on several structure learning tasks, including structure recovery of both synthetic graphs as well as standard graphs from the Bayesian Network Repository.

1 Introduction

Observational data alone generally identifies causal graphs only up to a Markov equivalence class, while interventions provide richer structural information. The paper proposes and evaluates a model for known and unknown interventions, including complete and partial graph recovery.

  • Observational data alone generally identifies causal graphs only up to a Markov equivalence class.
  • Unknown interventions arise when the intervened variable is unclear, and existing evidence suggests recovery may still be possible without a theoretical guarantee.
  • Partial graph completion uses known edges or non-edges and is easier than learning an entire graph because fewer edges require discovery.
  • Continuous-optimization methods perform competitively on observational data, but extending them to interventional data is not straightforward.
  • The proposed model favorably compares with state-of-the-art causal methods, generalizes to unseen interventions, and scales to approximately 50 variables for partial graph recovery.

2 Preliminaries

The preliminaries formalize structural causal models, intervention types, and the identifiability limits motivating interventional structure learning. They also frame continuous optimization as a way to avoid explicit enumeration of the super-exponential DAG space.

  • A structural causal model assigns each variable through a function of its direct parents and jointly independent noise.
  • Observational data distinguishes causal graphs only up to Markov equivalence, whereas intervention data is needed to identify the true graph structure.
  • Interventions may be absent, hard, soft, or uncertain about the exact variable targeted.
  • Continuous-optimization methods penalize violations of DAG structure while avoiding explicit maintenance of all possible graph hypotheses.

3 Related Work

Prior causal discovery work spans score-based, constraint-based, asymmetry-based, invariance-based, and learning-based approaches. The paper focuses on continuous optimization for observational and interventional data when intervention targets may be unknown.

  • Existing causal structure methods include score-based, constraint-based, asymmetry-based, and continuous-optimization approaches.
  • Several methods exploit invariance across environments, but scaling can be difficult because they iterate over a super-exponential graph space.
  • Interventional methods often assume full intervention information, unlike this paper’s setting where interventions may originate from other agents or the environment and begin unknown.
  • Neural-network approaches have learned causal signals from observations, but prior adapted work was tailored to continuous variables and observational data only.

4 Structure Discovery from Interventions Method

SDI jointly learns graph structure and causal mechanisms through alternating continuous optimization, combining observational fitting with intervention-based graph scoring. It supports unknown-target interventions and prior graph information while enforcing acyclicity.

  • Problem Setting and Assumptions: SDI assumes a queryable black-box SCM that supports sampling and applying or retracting interventions, with the target and nature of the next intervention uncontrolled.
  • Problem Setting and Assumptions: The method targets discrete, fully observed systems with sparse soft interventions that affect one variable, whose identity may be unknown.
  • Variations and Prior Knowledge: SDI can incorporate known edges and non-edges, converting complete graph recovery into partial graph recovery and improving performance with prior information.
  • Method Overview: The three-phase procedure samples graphs and fits functional parameters on observational data, scores sampled graphs on interventional data, then updates structural beliefs from the rewards.
  • Phase 3: Graph configurations are sampled from Bernoulli edge beliefs, while the structural parameters use a REINFORCE-like gradient estimator for discrete sampling.
  • Phase 2: Unknown intervention targets are predicted heuristically by identifying the variable with the greatest deterioration in average log-likelihood.
  • Acyclic Constraint: An acyclicity regularizer penalizes length-2 cycles in the softened adjacency matrix, while longer-cycle suppression is omitted because its computational cost was not worthwhile.

5 Experimental Setup and Results

The experiments evaluate the proposed neural causal model on synthetic and real-world graphs, including unknown interventions, unseen interventions, and partial graph recovery. Results show strong graph recovery, generalization, and partial-recovery performance, with denser or larger graphs presenting greater difficulty.

  • Experimental design: The experiments cover synthetic SCMs, BnLearn real-world datasets, partial graph recovery, and intervention-target variations.Synthetic evaluations vary graph size and connectivity, while real-world evaluations use repository benchmarks and partially known graphs.
  • Model architecture: The model uses independent MLPs whose inputs are masked by hypothesized adjacency matrices to restrict each variable’s predictor to its parent nodes.The mask is sampled from Ber(σ(γ)), and each MLP represents one random variable.
  • Synthetic graph recovery: AUROC reaches 1.0 for synthetic graphs with 3–13 variables, although denser fullM graphs take longer to converge.For graphs from 3 to 8 variables, all AUROCs eventually reach 1.0; larger and dense graphs encounter difficulties.
  • Benchmark comparisons: The method significantly outperforms baseline methods on benchmark comparisons and correctly recovers the true graph except for Sachs and full13.On BnLearn data, Asia is recovered perfectly, while Sachs has SHD=6.
  • Unseen interventions: The causal model assigns higher log-likelihood to samples from previously unseen intervention distributions than the non-causal variant on all tested 3-variable graphs.The comparison masks the intervention variable’s contribution to log-likelihood.
  • Unknown interventions: The intervention-target heuristic predicts targets better than chance and closely tracks training with known targets on larger 7-variable graphs.Randomly guessing or failing to guess the target causes a significant performance drop.
  • Partial recovery and analysis: Partial graph recovery reaches at least 90% accuracy on Barley and Alarm, while larger and denser graphs require more samples and show higher variance.The model predicts 50 edges for Barley and 40 for Alarm; regularization sensitivity also increases with graph size.

6 Conclusion

The paper introduces SDI, a continuous-optimization method combining observational and interventional data for causal structure discovery. Experiments show recovery of true structure, generalization to unseen interventions, strong comparisons with causal discovery methods, and improved scaling for partial graph recovery.

  • SDI combines observational and interventional data for causal structure discovery using continuous optimization.
  • Experiments show that SDI recovers true causal structure and generalizes well to unseen interventions.
  • SDI compares very well with state-of-the-art causal discovery methods on real-world datasets.
  • SDI scales better when only part of the graph is known and must be completed.

7 Annexes

The annexes describe intervention assumptions, neural-network ground-truth and learner models, and the method’s use of soft interventions with unknown targets. They also state that unknown intervention targets lack a theoretical identifiability guarantee in this setting.

  • The experiments use soft interventions, including hard interventions as a limiting case, because they are more general and often easier to perform.
  • Unknown interventions randomly select and affect one variable whose identity must be inferred.
  • Synthetic interventions reinitialize the selected variable’s neural-network parameters.
  • With all variables observed and interventions known, the true causal graph is identifiable in principle; unknown targets have no theoretical identifiability guarantee.
  • The method assumes faithfulness because its gradient estimate is extracted from shifts in probability distributions.
  • The learner uses independent MLPs for variables, with masked inputs enforcing dependence on hypothesized direct ancestors.

7.5 Hyperparameters

The annexes describe synthetic experiments across representative graph structures and explain the model’s sensitivity to graph size, density, and regularization. SDI recovers all tested synthetic graphs with at most 10 variables, while larger and denser graphs are more difficult.

  • 7.5 Hyperparameters: Experiments use five runs with fixed optimizer settings for synthetic graphs of sizes 3–8 variables.
  • 7.5 Hyperparameters: Synthetic SCMs use two-layer feedforward MLPs with LeakyReLU activations and bounded parameter initialization.
  • 7.5 Hyperparameters: The evaluation covers representative graph structures including chains, full graphs, trees, jungles, colliders, and bidiagonal graphs.
  • 7.5 Hyperparameters: SDI correctly recovers all synthetic graphs with 10 variables or fewer.
  • 7.5 Hyperparameters: Graphs larger than 10 variables, particularly denser graphs such as fullM, are more challenging to recover.

7.8 Comparisons to other methods

The comparison annex describes evaluations against multiple causal discovery methods and analyzes graph density and intervention-target prediction. It reports that SDI’s intervention heuristic is important for efficient training and that denser graphs become progressively harder to learn.

  • 7.8 Comparisons to other methods: SDI is compared with five other methods across synthetic and real datasets, with results reported in baseline-comparison tables.
  • 7.8 Comparisons to other methods: DAG-GNN results are reported for outputs selected by mean squared error, negative log-likelihood, and evidence lower bound.
  • 7.8 Comparisons to other methods: For graphs larger than six variables, denser graphs become progressively more difficult to learn than sparser graphs.
  • 7.8 Comparisons to other methods: During intervention-target prediction, the intervened variable’s likelihood contribution is ignored and gradients into its functional parameters are blocked.
  • 7.8 Comparisons to other methods: The prediction heuristic makes training closely track training with ground-truth intervention identities, whereas random or absent predictions slow or prevent learning.

7.11 Sample complexity

The method’s sampling demands grow multiplicatively across iterations, batches, configurations, interventions, and training phases, reaching billions of samples in typical settings. Benchmark and ablation figures document performance and prediction behavior alongside this complexity.

  • Benchmark comparisons: The benchmark tables compare learned and ground-truth edges using Hamming distance and structured Hamming distance across Asia and synthetic graphs.Table 6 compares against DAG-GNN, while Table 8 includes comparisons with Eaton and Murphy on graphs they can scale to.
  • Synthetic recovery: Synthetic graphs with 3-13 variables learn perfectly, reaching AUROC 1.0 for edge probabilities against ground truth.Denser fullM graphs take longer to converge.
  • Intervention prediction: Training with intervention prediction closely tracks ground-truth training for both 3-variable and 7-variable graphs.The figure compares ground-truth and prediction strategies against random and no-prediction strategies for smaller graphs.
  • Sample complexity: Typical settings require 1.28e9 observational and 1.408e9 interventional samples because sampling factors multiply across the method’s phases.The paper limits samples generated per intervention to usually 500-2000 to alleviate this growth.

7.13 Effect of regularization

The sparsity and DAG regularizers mainly affect convergence behavior, with the DAG regularizer becoming more important as graph size increases. Their effects on final performance differ between small and larger graphs.

  • Sparsity regularizer: The L1 sparsity regularizer slightly accelerates convergence but does not appear to affect the model’s final converged value.It encourages a sparse representation of causal-graph edges through structure parameters γ.
  • DAG regularizer: For graphs with ≤5 variables, the acyclic regularizer speeds convergence without significantly improving final cross-entropy.The result is illustrated using 3-variable graphs.
  • DAG regularizer: For graphs larger than 5 variables, the acyclic regularizer becomes important for encouraging recovery of the correct structure.The paper contrasts this with its smaller-graph effect, where the main benefit is faster convergence.

7.14 Importance of dropout

Dropout during functional-parameter pretraining is necessary because training on complete directed graphs cannot learn the required causal functions. Related ablations examine graph-belief losses, sparsity, DAG regularization, and benchmark structure recovery.

  • Dropout: Functional-parameter training cannot use complete directed graphs and requires dropping out each edge during pretraining.In the experiments, each edge is dropped with probability equal to its current γ value.
  • Training diagnostics: Synthetic-graph training tracks edge-belief cross-entropy and AUC across chainN, jungleN, colliderN, and fullN graphs of sizes 3-13.Error bars represent ±1σ over PRNG seeds 1-5.
  • Regularization ablation: The sparsity regularizer’s effect is examined on graphs with 5, 6, and 8 variables.The corresponding figure studies the effect of l_sparse.
  • Benchmark comparisons: The benchmark comparison reports structured Hamming distance between learned and ground-truth edges across Asia and synthetic graphs.The comparison includes methods that cannot scale to larger graphs, restricting some evaluations to the largest tractable cases.
  • Dropout ablation: Without dropout, edge cross-entropy is evaluated on 3-variable graphs to show the importance of dropout during functional-parameter training.The figure specifically reports edge CE loss when no dropout is used.
  • Benchmark diagnostics: Additional edge-probability cross-entropy evaluations cover Cancer at varying temperatures and the Earthquake and Asia datasets.Earthquake has 6 variables and Asia has 8 variables.
Loading 1910.01075v2…