PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Autoregressive Flows

Ilyes Khemakhem, Ricardo Pio Monti, Robert Leech, Aapo Hyvärinen

arXiv:2011.02268v2stat.MLcs.LG

TL;DR

General structural equation models are not identifiable without constraints, and randomized trials can be infeasible for answering causal questions. This paper connects constrained autoregressive flows to causal orderings; its method outperformed alternatives on synthetic and real data.

  • Problem

    General structural equation models are not identifiable without constraints, while causal questions often cannot be studied with randomized trials.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The paper interprets constrained autoregressive flow orderings as causal orderings, proves identifiability for affine flows, and proposes a likelihood-ratio measure for bivariate causal direction.

    Source: §1 · p. 2

  • Results

    On synthetic and real data, the method outperformed alternatives in causal discovery and interventional and counterfactual predictions.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Autoregressive flow orderings can be interpreted as causal orderings in structural equation models, supporting causal discovery and interventional and counterfactual queries.

    Source: §1 · p. 2, §7 · p. 9

    Identifiability requires a constraint that limits flow flexibility; with universal approximation capability, autoregressive flow orderings are not identifiable.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Two apparently unrelated fields -- normalizing flows and causality -- have recently received considerable attention in the machine learning community. In this work, we highlight an intrinsic correspondence between a simple family of autoregressive normalizing flows and identifiable causal models. We exploit the fact that autoregressive flow architectures define an ordering over variables, analogous to a causal ordering, to show that they are well-suited to performing a range of causal inference tasks, ranging from causal discovery to making interventional and counterfactual predictions. First, we show that causal models derived from both affine and additive autoregressive flows with fixed orderings over variables are identifiable, i.e. the true direction of causal influence can be recovered. This provides a generalization of the additive noise model well-known in causal discovery. Second, we derive a bivariate measure of causal direction based on likelihood ratios, leveraging the fact that flow models can estimate normalized log-densities of data. Third, we demonstrate that flows naturally allow for direct evaluation of both interventional and counterfactual queries, the latter case being possible due to the invertible nature of flows. Finally, throughout a series of experiments on synthetic and real data, the proposed method is shown to outperform current approaches for causal discovery as well as making accurate interventional and counterfactual predictions.

1 INTRODUCTION

Causal discovery from observational data is difficult because many structural equation models can fit the same distribution, while randomized trials may be inaccessible. The paper connects autoregressive flows to causal models and uses their ordering and likelihood properties for causal discovery and inference.

1 INTRODUCTION

  • Observational data can leave causal structure unidentifiable, while randomized trials may be unethical, too expensive, or technically impossible.The paper motivates causal discovery as a data-driven alternative when trials are unavailable.Source: §1 · p. 1
  • Affine autoregressive flows with fixed orderings define an identifiable causal model, generalizing the additive noise model.The paper presents this identifiability proof as its main theoretical result.Source: §1 · p. 2
  • On synthetic and real data, the method outperformed alternatives in causal discovery and interventional and counterfactual predictions.The introduction reports favorable performance across these tasks without giving specific metrics.Source: §1 · p. 2
  • Exact flow likelihoods support a nonlinear measure of causal direction based on likelihood ratios.The paper also describes this measure as having ensuing optimality properties.Source: §1 · p. 2
  • Conditioned on the correct causal ordering, autoregressive flows can answer interventional and counterfactual queries.The introduction states that these predictions can be made accurately.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define structural equation models as generative accounts of variables and their causal relationships, then introduce flows as invertible transformations for density modeling. Autoregressive flows impose an ordering that connects their structure to causal orderings.

2.1 Structural Equation Models

  • An SEM combines structural equations with mutually independent latent disturbances and defines the observational distribution by propagating noise through the equations.Its associated causal graph is assumed to be a directed acyclic graph.Source: §2.1 · p. 2
  • A general SEM need not have an identifiable causal direction without constraints; the additive noise model is one constrained special case.In an additive noise model, the noise is added to the function of preceding variables.Source: §2.1 · p. 2

2.2 Autoregressive Normalizing Flows

  • Normalizing flows represent observations through an invertible differentiable transformation of latent variables, enabling density computation by change of variables.Transformations may be chained while preserving invertibility and differentiability.Source: §2.2 · p. 3
  • Autoregressive flows use triangular Jacobians and an ordering that restricts each output to depend on preceding input variables.The triangular structure supports simple Jacobian computation.Source: §2.2 · p. 3

3 CAUSAL AUTOREGRESSIVE FLOW MODEL

This section aligns autoregressive-flow orderings with causal orderings to define CAREFL, combining identifiable affine causal models with likelihood-based direction selection. It also extends direction discovery to pairs of multivariate variables.

3.1 From Autoregressive Flow models to SEMs

  • Composed autoregressive transformations must share a fixed ordering and use affine or additive forms, preserving SEM correspondence while restricting expressivity.Source: §3.1 · p. 3

3.2 Model Definition and Identifiability

  • Affine causal flows are identifiable with Gaussian independent noise and a nonlinear invertible cause-effect shift; the additive special case is identifiable for factorial noise.Source: §3.2 · p. 4

3.3 Choosing Causal Direction using Likelihood Ratio

  • The bivariate direction score compares held-out log-likelihoods of affine flows fit under each ordering, selecting the higher-likelihood direction.Source: §3.3 · p. 4

3.4 Extension to Multivariate Data

  • For groups of multivariate variables, the bivariate score applies directly; exhaustive full-order search grows exponentially, motivating PC plus scoring for remaining edges.Source: §3.4 · p. 5

4 CAUSAL INFERENCE USING AUTOREGRESSIVE FLOWS

Given the correct causal ordering, autoregressive flows can evaluate interventions and counterfactuals. Their invertibility makes latent-noise recovery from observations readily available for counterfactual inference.

4.1 Interventions

  • Interventions replace the target variable's structural equation with α, then propagate the remaining flow equations to sample an empirical intervention distribution.Source: §4.1 · p. 5

4.2 Counterfactuals

  • Counterfactuals condition on latent noise inferred from the observation, then fix the intervened equation and propagate forward for prediction.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

Experiments evaluate CAREFL for causal discovery, interventions, and counterfactual predictions on synthetic and real data, comparing it with alternative methods. Results cover five synthetic SEMs, benchmark cause-effect pairs, EEG time series, simulated and fMRI interventions, and counterfactuals.

5.1.1 Synthetic data

  • Across five synthetic SEMs, CAREFL consistently identifies the causal direction using the same architecture and training parameters.The simulations use Laplace noise, although the identifiability result assumes Gaussian noise; this shows Gaussianity is sufficient but not necessary in these simulations.Source: §5.1.1 · p. 6, Figure 1 · p. 7
  • With a Laplace flow prior, CAREFL’s performance stays the same when the simulated noise is changed to Laplace, Student-t, or Gaussian.Other architectural parameters were kept the same as in the preceding simulations.Source: §5.1.1 · p. 6, Figure 2 · p. 7

5.1.2 Real data

  • CAREFL performs better than alternative algorithms on the 108-pair Cause Effect Pairs benchmark.The benchmark measures the percentage of correctly identified causal variables.Source: §5.1.2 · p. 7, Table 1 · p. 7
  • For EEG arrow-of-time inference, CAREFL matches baselines at average-to-high confidence, performs better at low confidence, and improves with larger samples.Channels are ranked by confidence; the task uses data from 118 channels for one subject.Source: §5.1.2 · p. 7, Figure 3 · p. 7

5.2 Interventions

  • On simulated linear and nonlinear interventional distributions, CAREFL better infers their nature than the ANM regression baselines.The comparison uses mean squared error between predicted and true expectations under do(X1 = α).Source: §5.2 · p. 8, Figure 4 · p. 8
  • For electrical-stimulation fMRI, CAREFL outperforms alternative causal models on interventional predictions.The table reports median absolute error; the experiment uses Cingulate Gyrus and Heschl’s Gyrus data.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Counterfactuals

  • CAREFL makes accurate counterfactual predictions for x3 and x4 across a range of α values.The flow’s invertibility allows latent-variable values to be inferred for counterfactual prediction.Source: §5.3 · p. 8, Figure 5 · p. 8

6 DISCUSSION

The discussion positions affine flows as an identifiable non-additive causal model and contrasts CAREFL with existing approaches. It also notes limits of additive-noise methods and the scope trade-off relative to NonSENS.

Identifiability

  • Unlike the post-non-linear model, affine flows cannot be reduced to an additive noise model by a fixed transformation of the effect, motivating a novel non-additive identifiability result.The discussion says the estimation algorithm benefits from this result, as shown in the experiments.Source: §6 · p. 9

Scope relative to NonSENS

  • CAREFL restricts nonlinear relationships but assumes no nonstationarity, whereas NonSENS relies on access to non-stationary data.The paper presents CAREFL as applicable in more general scenarios on that basis.Source: §6 · p. 9

Limits of additive-noise methods

  • ANM and RECI depend on regression and residual-based tests, and fail when the additive-noise assumption does not hold.The discussion also notes that choosing a suitable regression model is difficult and that a very good regression can reduce test confidence.Source: §6 · p. 9

Comparison with ANM and RECI

  • CAREFL uses likelihood ratios with the aim of improving statistical efficiency, while its invertibility supports interventions and counterfactuals beyond discovery.The discussion contrasts this scope with ANM and RECI, which were designed solely for causal discovery.Source: §6 · p. 9

7 CONCLUSION

Autoregressive flows support causal inference by interpreting their variable ordering as a causal ordering in structural equation models. The paper reports favorable causal-discovery results and accurate interventional and counterfactual predictions.

7 CONCLUSION

  • The variable ordering in an autoregressive flow can be interpreted as a causal ordering in structural equation models.Source: §7 · p. 9
  • Affine flows define identifiable causal models in which the cause modulates the noise, generalizing additive noise models.Source: §7 · p. 9
  • Interventional queries can be evaluated by fixing the intervention variable while sampling from the flow, and invertibility facilitates counterfactual queries.Source: §7 · p. 9
  • The method outperformed alternatives in causal discovery and interventional and counterfactual predictions on synthetic and real data.Source: §7 · p. 9

A Identifiability of the affine causal model

The appendix characterizes cases in which affine causal models can be non-identifiable by examining when forward and backward models can both fit the data. It identifies two exceptional scenarios and rules out mixed cases in the proof.

A Identifiability of the affine causal model

B Affine flows are not universal density approximators

Affine autoregressive flows are not universal density approximators: their inverse Jacobian log-determinant depends only on x1, ruling out some distributions.

B Affine flows are not universal density approximators

  • With a standard Gaussian latent z, no affine autoregressive transformation can produce an isotropic Gumbel distribution.This is the specific non-representable distribution established by Proposition 1.Source: B Affine flows are not universal… · p. 16
  • The proof reduces the claim to two dimensions and derives a contradiction from assuming independent Gumbel-distributed x1 and x2.Setting x1 = 0 yields a condition that cannot hold for every x2.Source: B Affine flows are not universal… · p. 16
  • The inverse Jacobian log-determinant depends only on x1, which is the stated reason affine autoregressive flows are not universal density approximators.The determinant is −s1 − s2(x1).Source: B Affine flows are not universal… · p. 16

C Affine autoregressive flows are transitive

Composing autoregressive transformations with the same variable ordering preserves that ordering; for affine flows, the composition remains affine and autoregressive.

C Affine autoregressive flows are transitive

  • Composing two autoregressive transformations with ordering π yields another autoregressive transformation with the same ordering.The proof uses invertibility with respect to the first argument to preserve the autoregressive form.Source: C Affine autoregressive flows are… · pp. 16–17
  • For affine autoregressive flows sharing an ordering, their composition is also an affine autoregressive flow with that ordering.The proposition extends this closure property to k composed affine flows.Source: C Affine autoregressive flows are… · p. 17
  • The section establishes transitivity of affine autoregressive flows by showing that the composed transformation retains the required form.The proof concludes that the composition is transitive.Source: C Affine autoregressive flows are… · pp. 17–18

D Universality of the causal function

The appendix establishes approximation properties for composed affine autoregressive flows and outlines flow-based procedures for interventions and counterfactuals. It also reports task-specific architectures and an empirical benefit from stacking flows.

D Universality of the causal function

  • Under technical conditions, affine autoregressive flows with universal-approximation-capable component networks retain universal approximation capability when composed.The section notes that this can hold even when each component scale function is a one-hidden-layer network.Source: D Universality of the causal function · p. 18

E.1 Interventions

  • For an intervention, CAREFL fixes the intervened variable and samples the remaining latent variables before propagating them through the flow.For affine flows, the inverse transformation permits parallel sampling; the implementation uses v = 0 when computing the intervened latent value.Source: §E.1 · p. 19

E.2 Counterfactuals

  • Counterfactual prediction follows abduction, action, and prediction: invert the observation, adjust the latent state for the query, then propagate it forward.The abduction step obtains zobs = T−1(xobs).Source: §E.2 · p. 19

F.1 Architectures and hyperparameters

  • Experiments use task-specific compositions of two to five flow sub-transformations, with MLP architectures and training settings varying across tasks.The configurations cover causal discovery, cause-effect pairs, EEG arrow-of-time inference, and simulated and fMRI interventions.Source: §F.1 · p. 20

F.2 Exploring flow architectures

  • Across experiments varying hidden-layer width and the number of stacked flows, stacking multiple layers empirically improved performance.Each flow used a one-hidden-layer MLP with LeakyReLU activation.Source: §F.2 · p. 20

F.3 Preprocessing of EEG data

The EEG arrow-of-time experiments construct bivariate data from lagged channel time series, while the fMRI pipeline preprocesses scans and selects two regions for analysis.

F.3 Preprocessing of EEG data

  • The EEG analysis uses subject 3 recordings sampled at 1000 MHz, with 150 or 500 time points and each of 118 channels reversed with probability 0.5.Each univariate channel series is converted to bivariate data by pairing time points separated by a lag.Source: §F.3 · pp. 20–21
  • EEG arrow-of-time comparisons combine lags 1, 2, and 3 for ANM, RECI, linear LR, and CAREFL-NS, but use lags 1 and 2 for CAREFL.Source: §F.3 · p. 21

F.4 Preprocessing of functional MRI data

  • The fMRI pipeline performs anatomical and functional preprocessing, then extracts regional time series using the Harvard-Atlas and selects Cingulate and Heschl’s gyri.Those regions were selected to include as many subjects as possible; they represent higher-order heteromodal and unimodal sensory areas, respectively.Source: §F.4 · p. 21

Related papers

Loading 2011.02268v2…