Source-linked AI summary
On Causal and Anticausal Learning
Bernhard Schoelkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, Joris Mooij
TL;DR
The paper studies function estimation when causal structure can be inferred and examines how causal direction affects learning tasks, including semi-supervised learning. It formulates the hypothesis that semi-supervised learning helps more in anticausal or confounded problems than causal problems, with preliminary literature results supporting it.
Problem
The paper asks how causal knowledge can guide function estimation and learning under changing conditions, including covariate shift, transfer learning, and semi-supervised learning.
Method
The paper analyzes causal and anticausal prediction directions and formulates a hypothesis about when semi-supervised learning should help.
Results
Preliminary meta-analysis results from the literature support the hypothesis that semi-supervised learning works better for anticausal or confounded problems than for causal problems.
Takeaways & Limitations
Knowing the causal structure can indicate which prediction and learning approaches are likely to be easier or harder.
Takeaways & Limitations
The evaluation does not cover all described methods and focuses on testing the main semi-supervised-learning prediction.
Abstract
from arXiv · showhide
We consider the problem of function estimation in the case where an underlying causal model can be inferred. This has implications for popular scenarios such as covariate shift, concept drift, transfer learning and semi-supervised learning. We argue that causal knowledge may facilitate some approaches for a given problem, and rule out others. In particular, we formulate a hypothesis for when semi-supervised learning can help, and corroborate it with empirical results.
1. Introduction
The paper asks how causal knowledge can improve machine-learning prediction beyond statistical association, especially under interventions and distribution shifts. It introduces functional causal models and assumptions that can distinguish causal direction and clarify which learning tasks are easier or harder.
- Motivation: Statistical association supports prediction under stable training conditions but does not by itself answer how interventions will affect outcomes.The stork–birth-rate example illustrates why predictive correlation does not establish intervention effects.
- Functional causal models: Functional causal models represent variables as deterministic functions of parents and independent noise, allowing predictions under interventions.Changing a function’s input can be used to compute the effect of setting variables to specific values.
- Causal direction: Additional functional assumptions can distinguish X → Y from X ← Y, unlike conditional-independence testing with only two variables.The functional perspective therefore supports causal-direction inference for input-output learning problems.
- Model assumptions: The paper assumes independent cause and effect noises and independence between the causal mechanism P(E|C) and the cause distribution P(C).This asymmetry is expected to fail in the backward direction because P(E) and P(C|E) inherit properties from both distributions.
- Model assumptions: The two-variable functional causal model is sufficiently rich that causal direction cannot generally be inferred without assumptions restricting mechanism complexity.Noise-switching constructions can implement arbitrarily different mechanisms, making the conditional distribution difficult to identify from data.
- Additive noise models: Under an additive noise model, E = φ(C) + NE, causal direction can generically be identified except in cases such as linear φ with Gaussian noise.The model can be fit by regressing effect on cause while requiring residual noise to be independent of the cause.
2. Predicting Effect from Cause
In causal prediction, the input is the cause and the output is the effect, with estimation studied under changes to the underlying noise variables. The paper distinguishes when additional output or joint-distribution data can help from settings where input-distribution data provide no information about the conditional.
- Causal prediction estimates the effect Y from the cause X and is studied for robustness to changes in underlying noise variables.
- Robustness w.r.t. input changes: Under covariate shift, independence of mechanism and input supports retaining P′(Y|X) = P(Y|X) when P(X) changes.The prediction rule may still need adaptation because finite-data estimates can be inaccurate in newly probable regions.
- Semi-supervised learning: Additional inputs from P(X) do not improve estimation of P(Y|X) because P(X) contains no information about the causal conditional.The paper therefore characterizes SSL as pointless for this causal-prediction scenario.
- Additional information about the output: Additional outputs from P(Y) can help identify the noise component in a unique convolutional decomposition, reducing conditional estimation to learning φ.The assumption is satisfied, for example, with Gaussian noise and indecomposable P(φ(C)).
- Transfer learning: With changed joint distributions and invariant causal mechanisms, Conditional ANM learns one shared function while enforcing residual independence separately across datasets.
- Concept drift: When only the causal function changes while both noises remain invariant, ANM applied to the new data yields the new conditional through the unchanged noise distribution.
3. Predicting Cause from Effect
Anticausal prediction uses the effect as input to predict the cause, a common pattern in machine learning such as inferring a digit label from its image. Because P(Y|X) depends on the cause distribution, changes in inputs or additional input data have different consequences than in causal prediction.
- Anticausal prediction uses the effect X as input to predict the cause Y, as in predicting a handwritten digit’s label from its image.The paper models the label intention as causing the image.
- P(X|Y) represents the causal mechanism and is independent of P(Y), whereas P(Y|X) is sensitive to changes in P(Y).The paper recommends modeling P(X|Y) first and recovering P(Y|X) with Bayes’ rule.
- In the additive example X = Y + N_X, E(X|Y) is linear, while E(Y|X) is nonlinear and depends heavily on P(Y).
- Robustness w.r.t. input changes: With changed input distributions, the goal is to estimate P′(Y|X), potentially using input data to determine whether P(Y) or P(X|Y) changed.The proposed route uses localizing distribution change and, when appropriate, inverting or estimating conditionals.
- Robustness w.r.t. input changes: If P(X|Y) is injective, the changed cause distribution can be recovered from the changed input distribution before constructing P′(Y|X).Injectivity means the input distribution can be uniquely computed from the output distribution.
- Transfer learning: When the joint distribution changes but the additive-noise function remains invariant, Conditional ANM or related estimation procedures target the new conditional.
4. Modules
The paper develops modules for exploiting functional causal assumptions: identifying injective conditionals, localizing distribution changes, and estimating changed causal conditionals. These modules use distributional decompositions and additive-noise structure to recover causal components or distinguish which part changed.
- Inverting conditionals: An injective conditional uniquely determines the input distribution from the output distribution.
- Inverting conditionals: For finite variables, injectivity of P(Y|X) is equivalent to the stochastic matrix having full rank, possible only when |X| ≤ |Y|.
- Inverting conditionals: In a post-nonlinear model with injective φ and ψ, the output distribution can determine the input distribution when the noise convolution is invertible.
- Localizing distribution change: Localizing distribution change compares P(E) = P(φ(C)) ∗ P(N_E) with P′(E) = P′(φ(C)) ∗ P′(N_E) when the causal function is shared.The changed component is either the cause distribution or the noise distribution.
- Localizing distribution change: Zeros in the Fourier transform of P(E) can help identify whether changes arise from P(φ(C)) or P(N_E) by checking which zeros persist in P′(E).
- Estimating causal conditionals: Under an additive-noise model with unchanged P(C) and φ, deconvolution recovers the changed noise distribution and thereby computes P′(E|C).
- Conditional ANM: Conditional ANM learns a shared φ across two datasets while enforcing independence between each cause and its corresponding noise.
5. Empirical Results
The empirical analyses test whether semi-supervised learning helps more for anticausal or confounded problems than for causal problems. Across benchmark classifications, self-training and semi-supervised regression show this predicted pattern, although the initial benchmark evidence has important design limitations.
- Semi-supervised classification: The study compares SSL methods with supervised base classifiers across categorized causal, anticausal/confounded, and unclear datasets.Causal datasets assume features cause the class without effects or common causes; anticausal/confounded datasets may have mechanisms dependent on the input distribution.
- Semi-supervised classification: SSL does not significantly improve accuracy in the one causal benchmark dataset but helps in most anticausal/confounded datasets.The benchmark used eight datasets, 100 labeled training points, and 11 SSL methods compared with 1-NN and SVM base classifiers.
- Caveats: The initial classification benchmark is difficult to interpret because it is transductive, uses heterogeneous SSL methods, and uses a different method set for SecStr.Transductive use of test inputs could improve performance independently of causal direction.
- Self-training: Self-training does not significantly decrease error on three causal datasets but improves performance on several anticausal/confounded datasets.The reanalysis covers 26 UCI datasets and six base classifiers, using inductive self-training extensions of corresponding supervised methods.
- Semi-supervised regression: Semi-supervised regression reduces RMSE on all seven anticausal datasets, whereas causal datasets usually show little performance improvement.Across the statistical tests, the anticausal p-value is 0.0156 and the causal p-value is 0.6523; only the anticausal result rejects the null at 5%.
6. Conclusion
The paper argues that knowing causal structure informs prediction direction and the suitability of learning methods. Its preliminary meta-analysis supports the hypothesis that, under mechanism–input independence, SSL helps more in anticausal or confounded problems than in causal ones.
- Conclusion: Causal structure affects which prediction tasks are easy or hard, and can guide the choice of learning approach.The paper frames prediction direction as consequential for learning under causal and anticausal relationships.
- Conclusion: The paper hypothesizes that SSL works better for anticausal or confounded problems than for causal problems when mechanism and input are independent.The conclusion presents this as the central hypothesis evaluated by the empirical analyses.
- Conclusion: A preliminary meta-analysis of results from the literature seems to support the proposed hypothesis.The conclusion characterizes the evidence as preliminary rather than comprehensive.