Invariance, Causality and Robustness

Peter Bühlmann

Source-linked AI summary

Invariance, Causality and Robustness

Peter Bühlmann

arXiv:1812.08233v1Dec 19, 2018stat.ME

TL;DR

The paper addresses causal inference and prediction under heterogeneous or perturbation data, including potential outcomes absent from observed data. It develops probabilistic invariance and causal regularization to connect causal inference with predictive robustness, while noting assumptions and identifiability limits.

Problem
Causal effects and potential outcomes often must be inferred from nonrandomized data, including interventions or treatments never observed for the relevant units.
Method
The paper uses probabilistic invariance and causal regularization, encouraging residuals to be uncorrelated with observed anchor variables, and estimates invariance across heterogeneous environments.
Results
The paper presents approaches for predictive robustness and more meaningful analysis than standard regression or curve-fitting methods, with substantial performance gains reported for Anchor Boosting except one specified setting.
Takeaways & Limitations
Causal aspects can support better predictions and causal-oriented interpretation, even when causal inference is ill-posed.
Takeaways & Limitations
The approach depends on invariance assumptions that may be violated, and causal variables are typically not identifiable without many environments or additional assumptions.

Abstract

from arXiv · show

We discuss recent work for causal inference and predictive robustness in a unifying way. The key idea relies on a notion of probabilistic invariance or stability: it opens up new insights for formulating causality as a certain risk minimization problem with a corresponding notion of robustness. The invariance itself can be estimated from general heterogeneous or perturbation data which frequently occur with nowadays data collection. The novel methodology is potentially useful in many applications, offering more robustness and better `causal-oriented' interpretation than machine learning or estimation in standard regression or classification frameworks.

1 Introduction

The paper connects causal inference and predictive robustness through invariance across heterogeneous or perturbed environments. It reviews causal-oriented methods that seek causal effects under stronger assumptions and useful robust predictions under more relaxed ones.

Causal relationships are central to interpretability, and incorporating causal aspects can improve prediction while enabling causal-structure inference from a predictive perspective.
Randomized studies support confirmatory causal-effect inference, whereas observational data require additional assumptions and methods with varying degrees of causal certainty.
The framework uses invariance extracted from heterogeneous or perturbation data to develop methods geared toward causal solutions.
Under stronger assumptions, the methods infer causal effects; under more relaxed assumptions, they provide a diluted form of causality and mathematically grounded robustness.
The manuscript reviews earlier work, broadens its perspective, and adds nonlinear anchor regression methodology with illustrative applications.

2 Predicting potential outcomes, heterogeneity and worst case risk optimization

The paper frames potential-outcome prediction as robust prediction across observed and unobserved heterogeneous environments. Its causal connection comes from restricting perturbations to preserve the X-to-Y mechanism while changing the distribution of X.

2.1 Two examples for prediction of potential outcomes: Potential-outcome prediction asks what would happen under an intervention or treatment whose corresponding outcome has not been observed.
2.1 Two examples for prediction of potential outcomes: The examples include predicting flowering time after single-gene knockouts and user behavior after exposure to a specified advertisement.
2.2 Heterogeneous data: Observed environments are experimental settings, sub-populations, or perturbations with environment-specific response vectors, covariate matrices, and sample sizes.
2.3 A prediction problem and worst case risk optimization: The target is prediction in new unseen environments from a broader space F, using data from fewer observed environments E.
2.3 A prediction problem and worst case risk optimization: Robustness means performing well in worst-case scenarios over F, linking the prediction problem to transfer learning and causal reasoning.
2.3 A prediction problem and worst case risk optimization: The causal connection restricts environments so they do not directly affect Y or change the mechanism between X and Y, while ideally changing the distribution of X.
2.3 A prediction problem and worst case risk optimization: More observed heterogeneities that change X’s distribution can improve identification of causal effects, although hidden confounding between X and Y may remain.
2.3 A prediction problem and worst case risk optimization: The causal parameter is formulated as optimizing a worst-case risk over environments satisfying the ad-hoc conditions.

3 Invariance of conditional distributions

The section defines invariance as stability of conditional distributions or regression parameters across observed environments, then uses it to connect causal variables with identifiable and robust prediction. Invariant Causal Prediction intersects invariant covariate sets to control false positive causal selections, while its power depends on heterogeneity and model specification.

Invariance assumption: Invariance requires that conditioning on a subset of covariates yields the same conditional distribution across all observed environments.In linear models, the corresponding regression parameter and error-term distribution remain unchanged across environments.
Invariance and causality: Causal variables satisfy the invariance assumption across environments under the stated structural equation model and perturbation conditions.This establishes that causal variables lead to invariance, though other variable sets may also satisfy it.
Identifiability: Because causal variables are generally not identifiable from observational data alone, additional environments, perturbations, or structural assumptions are needed.The paper frames identifiability as a central difficulty in reversing the implication from invariance to causality.
Invariant Causal Prediction: ICP intersects all covariate subsets that satisfy invariance over the observed environments, producing a conservative set of causal selections.The procedure is implemented for linear and nonlinear models and tests only environments observed in the data.
Robustness properties: Theorem 1 controls false positive causal selections with probability at least 1 − α, even when observed heterogeneities are ineffective.The guarantee does not require identifiability to be handled separately: selected variables must be identifiable as causal with controllable probability.
Power and limitations: ICP power depends on environments, sample size, and testing choices; it generally improves with more heterogeneity but can be conservative or fail under a mis-specified linear model.With sufficiently nonlinear truth and Gaussian linear invariance tests, ICP typically returns an empty selected set; nonlinear tests are proposed as an alternative.

4 Anchor regression: relaxing conditions

Anchor regression relaxes exact invariance and standard instrumental-variable assumptions by regularizing residual dependence on anchor variables, yielding a tunable connection between causal estimation and robustness to shift perturbations.

Relaxing invariance: Violations of exact invariance motivate searching for approximate invariance, which anchor regression incorporates through its methodology.Other violations include changes in regression parameters or residual distributions across environments.
Hidden confounding: With hidden confounding, environments are treated as instruments in an instrumental-variables structural equation model.The framework allows environments to act directly on hidden confounders and the response, including invalid instruments.
Anchor regression estimator: Anchor regression uses causal regularization to encourage residuals to be uncorrelated with source-node anchors.The anchor variables enter as an additional linear contribution, and the estimator can include sparsity penalties in high-dimensional settings.
Anchor regression estimator: The tuning parameter γ interpolates among ordinary least squares at γ = 1, two-stage least squares as γ →∞, and adjustment for anchors as γ →0.Under positive-definite Cov(X), the criterion yields a unique parameter.
Robustness to shifts: The estimator solves a worst-case risk problem over shift perturbations, with causal-regularized risk exactly dual to worst-case perturbed risk.The regularization parameter represents the strength of the shift; under Gaussian assumptions, larger quantiles correspond qualitatively to larger γ.
Causal interpretation: Under valid instrumental-variable assumptions, γ →∞ consistently recovers the causal parameter, while under invalid instruments large γ still yields a more meaningful stable quantity than standard regression.For large γ, coefficient magnitudes identify key drivers associated with stable prediction across perturbations.
Empirical illustration: Anchor regression improves perturbed out-of-sample prediction relative to OLS, but offers no gain and slight loss when future data contain no new perturbations.The empirical comparison is based on Figure 7’s perturbed out-sample prediction errors.

5 Nonlinear anchor regression

Nonlinear anchor regression extends the linear framework to nonlinear dependence of Y on X while allowing standard machine-learning regression technologies, with causal regularization intended to improve robustness under heterogeneity.

The methodology generalizes anchor regression to nonlinear dependence of Y on X.
The approach allows regression technologies such as Random Forests and Deep Neural Nets to serve as plug-in components.
Additional causal regularization is proposed to improve machine-learning methods in the presence of heterogeneity.
The nonlinear structural equation model can reduce to nonlinear instrumental-variables regression when M_Y and M_H are zero.

5.1 The objective function and the algorithm

The nonlinear estimator minimizes a constrained regression objective over a function class, using anchor projections when the anchors have linear effects on the modeled variables.

The nonlinear regression function f maps covariates X to responses Y and is estimated from observed training pairs.
The estimator searches over a function class C that can impose restrictions such as smoothness or sparsity.
The nonlinear anchor regression estimator is defined using the function formulation introduced for f.
The projection Π_A is appropriate when anchor variables have linear effects on X, Y, and hidden confounders H.

5.2 Anchor Boosting: a “regularized” approximation of the estimator

Anchor Boosting approximates the nonlinear anchor-regression estimator by iteratively fitting weighted negative gradients with a base learner and stopping to control overfitting; LM+RF handles partial linear structure.

The algorithm uses boosting to iteratively fit the negative gradient with a pre-specified regression base learner.
Each iteration increases m, computes a weighted pseudo-response, updates the fitted function, and repeats until m_stop.
The stopping iteration acts as a regularization parameter alongside causal regularization encoded by W = W_γ, creating a bias-variance trade-off.
Stopping rules use the weighted residual objective to guard against overfitting and avoid excessive boosting iterations.
The LM+RF learner first fits a linear model, then fits Random Forests to residuals, and combines both fitted functions.
LM+RF typically outperforms plain Random Forests when the regression function combines linear and nonlinear components.
Any machine-learning regression technique can serve as the base learner, and the stopping rule can use any reasonable regression algorithm.

5.3 Some empirical results

Simulations evaluate Anchor Boosting under moderate and strong anchor shifts using two structural models and two base learners. Performance gains are substantial in most settings, especially with LM+RF when strong shifts affect a linear component.

The simulations use training samples of n = 300 and out-of-sample samples of n_out = 2000 under models M1 and M2.
Performance is measured by empirical α-quantiles of absolute out-of-sample prediction errors across 100 independent simulation runs.
Anchor Boosting with γ = 7 is compared using Random Forests or LM+RF base learners and two stopping criteria.
Model M1 has no linear term, whereas M2 has one; both models contain two active variables.
Anchor Boosting yields substantial performance gains except for M2 with strong shifts and a Random Forest base learner.
Under strong shifts in M2, Random Forests struggle because shifts in anchors propagate through covariates and the linear component into Y.
With LM+RF, Anchor Boosting produces a massive performance gain in the strong-shift M2 setting.
LM+RF can also help under strong interventions in M1, possibly because Random Forests extrapolate poorly for large X-values.

5.4 Variable importance

The paper proposes permutation-based variable-importance measures for anchor regression and shows that Anchor Boosting with LM+RF better identifies the active variables in model (M3) than standard Random Forests.

Importance measures: Permutation-based importance measures compare prediction loss after permuting one covariate with the unpermuted model loss.The paper uses relative RSS increase and alternatively median absolute loss; permutations use training data rather than Random Forest out-of-bag observations.
Conditions: The proposed permutation measure is expected to work when the estimated anchor-regression function is reasonably regularized and does not overfit.The empirical variable-importance experiment uses n = 300; out-of-sample observations are unnecessary for variable importance.
Results: 28.5%, 18.5%, 1.5% are the reported prediction-performance gains for strong shift with LM+RF at α ∈ {0.5, 0.8, 1}.The corresponding strong-shift Random Forest gains are 9.6%, 8.4%, and 6.3%.
Simulation setting: In model (M3), X2 and X3 are the only active variables in the true function f(X).The model defines f using X2 and X3, with nonlinear indicator terms.
Results: Anchor Boosting with LM+RF substantially outperformed standard Random Forests for quantifying variable importance in model (M3).Figure 10 reports ranks across 100 independent simulations, using γ = 7 without tuning.

5.5 Some arguments why a simple linear projection ΠA is sufficient

The section explains when a linear projection on anchor variables can preserve useful robustness for nonlinear models. Under linear conditional residual means, it yields first-moment invariance, while discrete anchors provide a notable exception to the usual nonlinear case.

Linear projection: As γ →∞, population regularization enforces residuals uncorrelated with the anchor A.The argument considers functions satisfying the relevant invariance condition.
Proposition 3: Proposition 3 gives invariance of the first residual moment under do-perturbations when E[Y − f(X) | A] is linear in A.It does not establish invariance of higher moments or the full residual distribution.
Scope: For nonlinear f(·), E[Y − f(X) | A] is typically nonlinear in A, violating Proposition 3’s assumption.The proposition therefore does not generally cover nonlinear conditional residual means.
Discrete anchors: Discrete anchor variables can always be represented with dummy encoding, ensuring a linear representation of the conditional expectation under the stated positivity conditions.The encoding uses r = m − 1 dimensions when A has m discrete levels.
Approximation conditions: The linear approximation is more likely to work when A has linear effects on X, H, and Y and f(·) is not far from linear.If the conditional expectation becomes highly nonlinear, the paper says a different penalization is needed.
Simulation: The discrete-anchor simulation amplifies out-of-sample anchors threefold, and its results are consistent with Figures 8–9.The experiment uses n = 300 and nout = 2000.

6 Turning around the viewpoint

The paper reframes causality through invariance under perturbations. Rich perturbation classes can recover causal variables, whereas restricted classes yield a diluted, shift-invariance notion with interpretability value.

Invariance: The paper defines F-invariance as the invariance assumption holding for a specified class of perturbations F.The invariant set consists of covariates whose residual behavior remains stable across the perturbations.
Causal interpretation: When F is sufficiently rich and satisfies the stated conditions, F-invariance corresponds to causality in the literature.If F is not sufficiently rich, F-invariance need not coincide with the causal variables.
Diluted causality: Under restricted shift perturbations, the method retains residual invariance and yields what the paper calls F diluted causality.The paper emphasizes that this is a particular form of shift invariance.
Interpretability: Even when causal inference is ill-posed and non-identifiable, invariance or diluted causality can provide more meaningful results for interpretability.The paper connects this potential benefit to interpretable machine learning.

7 Conclusions

The paper unifies causal inference and predictive robustness through invariance estimated from heterogeneous or perturbation data. It presents this as a route to robustness and more meaningful causal-oriented analysis when standard methods or causal identification are limited.

Causality and robustness: Causality can be formulated through worst-case prediction risk, linking causal inference to predictive robustness.The paper argues that invariance can support both predictive robustness and causal analysis.
Heterogeneous data: Heterogeneous data are informative for estimating invariances and inferring causal effects.The paper discusses environments such as countries and temporal sub-populations.
Scope of contribution: The paper reviews recent invariance-based methods and contributes new developments for nonlinear problems.It references InvariantCausalPrediction, nonlinearICP, and seqICP.
Conclusion: The contribution targets statistics for perturbation or heterogeneous data and seeks predictive robustness and more meaningful results than standard regression or curve fitting.This scope is presented especially for settings where causal inference is ill-posed.