PaperlayerView exact revision on arXiv

Source-linked AI summary

Invariant Causal Prediction for Nonlinear Models

Christina Heinze-Deml, Jonas Peters, Nicolai Meinshausen

arXiv:1706.08576v2stat.ME

TL;DR

The paper addresses how to identify causal parents and predict intervention effects when invariant causal prediction must handle nonlinear relationships and difficult conditional-independence testing. It develops and evaluates nonlinear ICP methods, finding pooled nonlinear residual testing robust across varied simulations, while fertility modeling reaffirms child mortality’s central causal role. The authors also note that power becomes difficult with larger parental sets and that fertility conclusions depend on causal sufficiency.

  • Problem

    The paper seeks to identify a target variable’s causal parents and estimate intervention effects despite the limitations of linear ICP and the difficulty of nonlinear conditional-independence testing.

  • Method

    The paper extends ICP to nonlinear settings by evaluating nonlinear and nonparametric conditional-independence tests and using accepted models to predict intervention effects.

  • Results

    Pooled nonlinear modeling followed by residual-distribution testing is robust across many simulation settings, while fertility modeling highlights child mortality rates as important for fertility rates.

  • Takeaways & Limitations

    Nonlinear ICP can support causal-parent learning, causal-effect bands, and intervention forecasting, including fertility-rate applications involving hypothetical mortality reductions.

  • Takeaways & Limitations

    Conditional-independence testing has limited power in some settings, especially when noise variance depends on environmental and causal variables or the parental set exceeds two variables.

Abstract

from arXiv · show

An important problem in many domains is to predict how a system will respond to interventions. This task is inherently linked to estimating the system's underlying causal structure. To this end, Invariant Causal Prediction (ICP) (Peters et al., 2016) has been proposed which learns a causal model exploiting the invariance of causal relations using data from different environments. When considering linear models, the implementation of ICP is relatively straightforward. However, the nonlinear case is more challenging due to the difficulty of performing nonparametric tests for conditional independence. In this work, we present and evaluate an array of methods for nonlinear and nonparametric versions of ICP for learning the causal parents of given target variables. We find that an approach which first fits a nonlinear model with data pooled over all environments and then tests for differences between the residual distributions across environments is quite robust across a large variety of simulation settings. We call this procedure "invariant residual distribution test". In general, we observe that the performance of all approaches is critically dependent on the true (unknown) causal structure and it becomes challenging to achieve high power if the parental set includes more than two variables. As a real-world example, we consider fertility rate modelling which is central to world population projections. We explore predicting the effect of hypothetical interventions using the accepted models from nonlinear ICP. The results reaffirm the previously observed central causal role of child mortality rates.

1. Introduction

This introduction frames nonlinear invariant causal prediction as a way to identify a target’s causal parents and estimate intervention effects when linear assumptions are inadequate. It motivates nonlinear conditional-independence testing, defining sets, causal-effect confidence bands, and fertility-rate modeling as an application.

  • The paper targets recovery of a target variable’s parental set and confidence bands for its causal effects.
  • Nonlinear causal discovery is needed because relationships may be nonlinear, causing the true parental set to lose invariance and coverage guarantees under linear ICP.
  • The paper extends ICP by evaluating nonlinear and nonparametric conditional-independence tests, although no general nonparametric test is known to maintain type-I error below the prespecified level with nontrivial power.
  • Defining sets provide information when causal parents are poorly identifiable because variables are highly correlated or the signal is weak.
  • The framework also provides nonlinear or nonparametric confidence bands for causal effects and forecasts average effects of external interventions.
  • In fertility-rate modeling, reduced child mortality is identified as a key explanatory variable for declining fertility, while the models may omit relevant variables.

2. Nonlinear Invariant Causal Prediction

The paper extends Invariant Causal Prediction to nonlinear settings, using invariance across environments to identify causal parents and estimate intervention effects. In fertility data, defining sets and nonlinear confidence bands support coherent intervention predictions, while performance depends on causal structure and conditional-independence testing.

  • Causal effects: Nonparametric confidence bands extend the method from parent identification to estimating the strength of nonlinear causal effects and predicting interventions.The paper develops confidence bands and intervention predictions alongside nonlinear ICP.
  • Nonlinear ICP: Nonlinear ICP extends invariance-based causal discovery beyond linear models using structural causal models with predictors X, target Y, and environmental variables E.The framework assumes environmental variables are neither parents nor descendants of the target.
  • Invariance approach: The true parental set is characterized by invariance of the target’s conditional distribution across environments when E has no direct effect on Y.This is the central invariance insight underlying the method.
  • Defining sets: Defining sets retain partial causal information when correlated predictors prevent resolving the exact parental set.For fertility, IMR and Q5 form a defining set: at least one is causal, but the data do not identify which or whether both are parents.
  • Fertility application: For Nigeria, jointly increasing IMR and Q5 makes all accepted models predict an increase in expected log(TFR), whereas individual changes yield inconsistent effects.The inconsistency arises because some accepted models omit IMR and others omit Q5.
  • Fertility application: Nonlinear ICP typically achieves more accurate coverage than all-covariate Random Forest regression and a cross-continent mean-change baseline.Using a noncausal regression surrogate can underestimate uncertainty and bias intervention-effect predictions, especially for interventions on target descendants.

3. Conditional Independence Tests

The paper evaluates multiple nonlinear conditional-independence tests for invariant causal prediction, covering distinct modeling strategies and assumptions. Their validity and power are limited by estimation bias, structural conditions, and the difficulty of testing conditional independence.

  • Methods: The authors evaluate several nonlinear conditional-independence tests that exploit invariance across environments, with implementations available in CondIndTests.The methods’ power and type I error rates are compared experimentally.
  • Methods: The residual prediction test uses nonlinear regression and scales residuals to test dependence on the environment and conditioning variables without knowing the noise variance.It can use an appropriate basis expansion or random features when the basis is unknown.
  • Methods: Invariant environment and target prediction compare out-of-sample performance between models with and without the environment or target included as a predictor.Under the null and optimal models, the corresponding performances should be statistically indistinguishable.
  • Methods: The invariant residual distribution test pools environments, predicts Y from X_S, and tests whether residual distributions are identical across environments.Invariant conditional quantile prediction instead tests whether quantile exceedances are independent of the environment and aggregates p-values with Bonferroni correction.
  • Limitations: Conditional-independence testing is statistically difficult: no assumption-free nontrivial test with valid level exists, and broad asymptotic power is unavailable for continuous conditioning variables.Tests may also suffer inflated rejection from estimation bias when conditioning on potential parents.
  • Limitations: The tests have method-specific limitations: invariant target prediction lacks power for environment-dependent noise variance, while residual-distribution testing assumes homoscedasticity and can miss differing pooled residual models.Quantile prediction avoids those particular issues but can still lose power when noise variance depends jointly on the environment and causal variables.

4. Simulation Study

The simulation study evaluates nonlinear conditional independence tests and baselines for recovering causal parents across varied nonlinear structural settings. Tests generally control FWER near the nominal level, but power depends strongly on targets, graph structure, and simulation conditions.

  • Simulation design: Simulations generate nonlinear additive-noise causal models while varying nonlinearities, targets, interventions, noise tails, and additive versus multiplicative parental contributions.
  • Error control: All conditional independence tests maintain FWER close to but below the nominal α = 0.05 level, whereas CAM, LiNGAM, PC(i), and PC(ii) exceed α.
  • Error-control caveat: The FWER guarantee can conceal more frequent rejection of the true-parent null because errors may cancel when accepted sets are intersected.Despite this caveat, overall FWER remains close to nominal across sample sizes, targets, nonlinearities, and other parameters.
  • Power: Average Jaccard similarity is around 0.4 for most methods and is not clearly dependent on their type-I error rates.
  • Power: Target choice is a major determinant of recovery success: targets 1 and 5 have empty parental sets and consequently achieve relatively high Jaccard similarity.Testing their true parental sets reduces to unconditional independence testing, making level control easier than for other targets.
  • Power: Faithfulness violations make target 3 difficult, while target 4’s single parent is easier to recover and target 6’s three-parent set yields low Jaccard similarity.For target 3, canceling paths allow many incorrect sets, including the empty set, to be accepted.
  • Sensitivity analyses: Increasing sample size can reduce power for target 4, sin(2πx) is the most challenging nonlinearity, and additive parental contributions often outperform multiplicative ones.The sample-size effect is attributed to estimation bias causing increasingly significant deviations from the null.
  • Method comparison: The invariant residual distribution test with GAM and Levene’s or Wilcoxon test is consistently competitive, while random-feature residual prediction is suggested for continuous environments.The invariant residual distribution test applies only to categorical environmental variables.

5. Discussion and future work

The discussion presents nonlinear ICP as a robust extension of invariant causal discovery and applies it to fertility-rate interventions. It also identifies defining sets, software resources, and assumptions that constrain interpretation.

  • Conclusions: Pooled nonlinear modeling followed by residual-distribution testing across environments provides desired coverage and high power across varied settings.
  • Fertility application: The fertility application estimates bounds for hypothetical interventions and finds better hold-out coverage for causal prediction intervals than baseline methods.
  • Fertility application: Infant and under-five mortality rates are highlighted as important fertility-rate factors, reconfirming earlier studies.
  • Defining sets: Defining sets provide information when causal parents are weakly identifiable or highly correlated by identifying variables of which at least one is causal.
  • Software: The nonlinearICP and CondIndTests R packages provide implementations of the proposed framework and discussed conditional independence tests.
  • Practical guidance: Linear ICP is recommended when linearity is strongly supported because it is expected to have greater power, with nonlinear ICP used when that assumption is uncertain.

Appendix A. Time series bootstrap procedure

The time series bootstrap generates response samples by fitting the model once, then resampling residual blocks across countries and time points. Confidence intervals are obtained by refitting the model to these bootstrap samples.

  • The procedure first fits the model using all available data points.
  • Bootstrap samples are created by resampling residuals block-wise and country-wise, using consecutive blocks of length lb = 3.
  • Residual blocks from sampled countries and time points are added to fitted values for each country until all fitted values are covered.
  • The model is fitted on each bootstrap sample to construct the confidence intervals.

Appendix B. Conditional independence tests

The generic nonlinear ICP framework tests whether candidate variable sets satisfy an invariance null hypothesis across environments. Its conditional independence tests take the target, candidate variables, environment, and significance level as inputs and return a decision about the null.

  • Invariant Causal Prediction tests whether a candidate set S satisfies the invariance hypothesis H0,S.
  • A conditional independence test returns a decision about H0,S at the selected significance level.
  • The test input is an i.i.d. sample of (Y, XS, E), where XS contains the variables in the subset S being tested, together with α.
  • Environmental variables E may be discrete or continuous, although the empirical results focus on a discrete, one-dimensional environment.

B.1 Kernel conditional independence test

The kernel conditional independence test uses environment-specific kernels and accepts multiple environmental variables as sets of random variables. For discrete environments it uses a delta kernel, and for other environments an RBF kernel.

  • The test uses a delta kernel for discrete environmental variables and an RBF kernel otherwise.
  • The kernel conditional independence test can handle more than one environmental variable because its inputs may be sets of random variables.

B.2 Residual Prediction test

The nonlinear ICP procedures test invariance through residual prediction, environment prediction, or target prediction. They use flexible models and test whether environmental information improves prediction beyond candidate variables, with assumptions and adjustments depending on the test.

  • B.2 Residual Prediction test: The residual-based test requires an assumed noise distribution Fε, and its level may be incorrect when the noise is not additive.
  • B.2 Residual Prediction test: Residual Prediction tests approximate nonlinear functions with basis expansions, regress Y on transformed candidate variables, and predict functions of scaled residuals from E and XS.
  • B.2 Residual Prediction test: Random features can automatically construct a basis expansion when the appropriate nonlinear transformation is unknown.
  • B.2 Residual Prediction test: Residual Prediction tests compute prediction-accuracy statistics and compare them with an empirical null distribution generated from B simulated samples.
  • B.2 Residual Prediction test: Invariant environment prediction compares test-set accuracy for predicting E with (Y, XS) against accuracy using XS alone.
  • B.2 Residual Prediction test: Invariant target prediction compares test-set accuracy for predicting Y with (XS, E) against accuracy using XS alone.

B.5 Invariant residual distribution test

The invariant residual distribution test pools environments to fit a nonlinear predictor, then tests whether residual distributions differ across environments. Its validity depends on additive noise and discrete environmental variables, with extensions needed for continuous or higher-dimensional environments.

  • The test is not expected to have correct level with non-additive noise and is applicable only to discrete environmental variables.
  • The procedure fits a model to predict Y from X_S using pooled data across environments.
  • It uses two-sample tests to compare residual distributions from one environment against those from the remaining environments.
  • The resulting p-values are minimized across environment-specific tests and Bonferroni-corrected for the number of performed tests.
  • Random forests or GAMs can estimate the nonlinear predictor, although GAMs implicitly assume additive components.

B.6 Invariant conditional quantile prediction

Invariant conditional quantile prediction tests whether quantile exceedances are independent of environments after estimating conditional quantiles. The procedure supports multiple quantiles and applies multiplicity corrections to its tests.

  • Continuous or higher-dimensional environmental variables require modifications to the independence-testing steps based on the structure of E.
  • The method estimates conditional quantiles of Y given X_S and tests environment independence through exceedance indicators.
  • For each quantile β, it defines whether Y exceeds the estimated 1−β conditional quantile and tests this indicator against environment membership.
  • The procedure applies a Bonferroni correction for the number of performed tests.
  • A quantile regression forest can provide the conditional quantile estimates.
  • Fisher’s exact test can assess independence using a 2×2 contingency table of exceedances across environments.

Appendix C. Experimental settings for numerical studies

The numerical studies evaluate nonlinear ICP methods across a broad randomized simulation design varying sample size, targets, noise, functional form, interventions, and intervention locations. The simulations include 27,478 runs spanning 1,240 settings for each of 22 methods.

  • 27,478 simulations from 1,240 distinct settings are evaluated for each of the 22 considered methods.
  • Sample size n is randomly selected from {100, 200, 500, 2000, 5000}.
  • Each simulation randomly selects one of six graph variables as the target variable.
  • Noise tails vary through t-distributions with degrees of freedom selected from {2, 3, 5, 10, 20, 50, 100}.
  • The design varies additive versus multiplicative parental effects and uses four nonlinear functions, with one nonlinearity shared throughout each graph.
  • Interventions vary by shift versus do type, strength, and location across observational and intervention environments.
  • Intervention locations include all eligible variables, randomly selected ancestors and descendants, or randomly selected parents and children of the target.

Appendix D. Additional experimental results

Additional experiments show that nonlinear ICP performance varies strongly with sample size, nonlinearity, intervention heterogeneity, and parental-effect structure. Larger samples can reduce power in some settings because estimation bias becomes detectable, while sinusoidal functions and multiplicative effects are especially challenging.

  • Increasing sample size from 2000 to 5000 decreases power for target variable 4.
  • The decrease occurs because bias in conditional-independence-test estimation creates deviations from the null that become significant with increasing sample size.
  • The function f4(x) = sin(2πx) is the most challenging nonlinearity considered and yields very low Jaccard similarity for targets with parents.
  • Intervening on all variables except the target does not produce the largest power, possibly because intervention settings become more similar across environments.
  • Greater heterogeneity between intervention environments, including different intervention targets, appears to improve Jaccard similarity performance.
  • Power is often higher for additive parental contributions than for multiplicative ones.
  • Figures 10–13 plot average Jaccard similarity against average FWER separately for target variables 2, 3, 4, and 6.

Appendix E. Example

The example applies the proposed nonlinear ICP approaches to a three-node causal DAG with six environments and seeks to recover X1 as the parent of X2.

  • The data follow the causal chain X1 → X2 → X3, with the target parent set S∗ = {X1}.
  • Six environments combine shift interventions on X1 and X3 with one observational environment.X1 is shifted in two environments, X3 in four, and the red environment is observational.
  • The proposed approaches are evaluated by whether they retrieve the parents of X2.
  • The accompanying results table reports which candidate sets were accepted by approaches (A)–(F), including several recoveries of {X1}.
Loading 1706.08576v2…