PaperlayerView exact revision on arXiv

Source-linked AI summary

Distinguishing cause from effect using observational data: methods and benchmarks

Joris M. Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, Bernhard Schölkopf

arXiv:1412.3773v3cs.LGcs.AIstat.MLstat.OT

TL;DR

The paper addresses whether causal direction can be recovered from joint observational data for two variables, despite the difficulty of separating cause from effect without experiments. It reviews ANM and IGCI, introduces and evaluates the CauseEffectPairs benchmark alongside simulated data, and finds that some methods—especially HSIC-based ANM—perform reasonably well, while more validation data are needed.

  • Problem

    The central problem is deciding whether X causes Y or Y causes X from joint observational data when only two variables are observed.

  • Method

    The paper reviews ANM and IGCI, introduces the 100-pair CauseEffectPairs benchmark, evaluates methods on real and simulated data, and proves consistency for the original ANM method.

  • Results

    63 ± 10% accuracy and 0.74 ± 0.05 AUC were obtained by the original ANM-pHSIC method on the CauseEffectPairs benchmark, while IGCI performance varied greatly across settings.

  • Takeaways & Limitations

    Certain statistical patterns can distinguish cause from effect in purely observational real-world data, but current methods need improvement before practical application.

  • Takeaways & Limitations

    The conclusions require more validation data for statistical significance, and extension to settings with confounding variables remains unclear.

Abstract

from arXiv · show

The discovery of causal relationships from purely observational data is a fundamental problem in science. The most elementary form of such a causal discovery problem is to decide whether X causes Y or, alternatively, Y causes X, given joint observations of two variables X, Y. An example is to decide whether altitude causes temperature, or vice versa, given only joint measurements of both variables. Even under the simplifying assumptions of no confounding, no feedback loops, and no selection bias, such bivariate causal discovery problems are challenging. Nevertheless, several approaches for addressing those problems have been proposed in recent years. We review two families of such methods: Additive Noise Methods (ANM) and Information Geometric Causal Inference (IGCI). We present the benchmark CauseEffectPairs that consists of data for 100 different cause-effect pairs selected from 37 datasets from various domains (e.g., meteorology, biology, medicine, engineering, economy, etc.) and motivate our decisions regarding the "ground truth" causal directions of all pairs. We evaluate the performance of several bivariate causal discovery methods on these real-world benchmark data and in addition on artificially simulated data. Our empirical results on real-world data indicate that certain methods are indeed able to distinguish cause from effect using only purely observational data, although more benchmark data would be needed to obtain statistically significant conclusions. One of the best performing methods overall is the additive-noise method originally proposed by Hoyer et al. (2009), which obtains an accuracy of 63+-10 % and an AUC of 0.74+-0.05 on the real-world benchmark. As the main theoretical contribution of this work we prove the consistency of that method.

1. Introduction

The paper studies whether one can distinguish X causing Y from Y causing X using only observational data in the bivariate setting, under assumptions excluding confounding, selection bias, and feedback. It reviews ANM and IGCI methods, introduces the CauseEffectPairs benchmark, and reports empirical and theoretical contributions.

  • Observational causal discovery matters because causal knowledge supports predicting intervention effects, while controlled experiments can be expensive, unethical, or impossible.
  • The paper focuses on deciding between X causing Y and Y causing X from a finite i.i.d. sample of their joint distribution.The setting assumes only two observed variables and excludes confounding, selection bias, and feedback.
  • The authors review Additive Noise Methods and Information Geometric Causal Inference for bivariate causal discovery.
  • The CauseEffectPairs benchmark contains 100 cause-effect pairs drawn from 37 datasets spanning domains including meteorology, biology, medicine, engineering, and economics.
  • The work evaluates several ANM and IGCI methods on simulated data and the CauseEffectPairs benchmark, and proves consistency for the original ANM implementation.

2. Additive Noise Models

Additive Noise Models (ANMs) exploit additive, independent noise to create asymmetries between the two possible causal directions. Under suitable assumptions, ANMs can identify the direction from observational data, and the original scoring procedure is consistent.

  • Bivariate structural causal models: A general structural model can be reduced to an effective-noise model that preserves both observational and interventional distributions.The reduction summarizes latent causes in a single effective noise variable and is formalized by Proposition 4.
  • Bivariate structural causal models: The reverse-direction model can preserve the same observational distribution while generally inducing different interventional distributions, so observational data alone cannot identify causality without extra assumptions.The forward and reverse constructions are observationally equivalent in general, but not interventionally equivalent.
  • Breaking the symmetry: Nonlinearity, non-Gaussianity, or other complexity restrictions can break the symmetry between causal directions, whereas linear-Gaussian models remain non-identifiable.Hoyer et al. showed that nonlinear relationships aid identification when noise is additive.
  • Definition and construction: ANMs restrict structural models so that the effect is a function of the cause plus additive noise independent of that cause.An ANM is a special case of the broader functional model in which the noise influence is additive.
  • Breaking the symmetry: Generic ANMs are identifiable because p(y | x) changes only through its mean, while p(x | y) typically changes more substantially with y.Only specific parameter choices yield non-identifiable ANMs; non-injective functions guarantee identifiability.
  • Consistency: Under suitable kernel and regression conditions, the original HSIC-based ANM algorithm consistently estimates the causal direction when exactly one direction satisfies an ANM.The consistency theorem applies in either data-splitting or data-recycling scenarios.

3. Information-Geometric Causal Inference

IGCI identifies causal direction through asymmetries between the cause distribution and the conditional effect mechanism, especially when variables are related by a nearly noiseless bijection. Its estimators favor X → Y when the cause is closer to its reference distribution, but finite samples, noise, repeated values, and non-injective functions constrain reliability.

  • Theory: IGCI assumes X and Y are deterministically related by a bijective function and that the cause distribution is independent of the conditional effect mechanism.This setting provides a formal asymmetry between X → Y and Y → X, but limits practical applicability to sufficiently low-noise, nonlinear causal relations.
  • Theory: IGCI infers X → Y when the cause density is uncorrelated with the slope of the causal function, because the effect density concentrates where the function is flat.Equivalently, the effect density contains information about the inverse function except when the causal function is linear.
  • Estimation methods: The basic estimator uses the average log-slope asymmetry and decides X → Y when its directional score C_X→Y is negative.The corresponding interpretation is that the variable with lower differential entropy is treated as the effect, because the function introduces irregularities rather than smoothing those of the cause.
  • Generalization to other reference measures: Generalized IGCI replaces the uniform reference with chosen reference densities and infers X → Y when the cause is closer to its reference density.For Gaussian references matched to the observed means and variances, this reduces to choosing X → Y when X has larger entropy after variance rescaling.
  • Estimation methods: IGCI implementations require continuous variables with densities, while repeated observations make the original difference quotients undefined.A cleaner treatment removes duplicate values while weighting each unique value by its original multiplicity, although repeated effect values are left untreated because they are less likely.
  • Estimation methods: For non-injective causal functions, the estimator can be asymptotically correct because cross-branch terms make the reverse score diverge, despite finite-sample errors.With sparse sampling on one branch, finite samples can instead yield C_X→Y ≈ log 2 and C_Y→X ≈ −log 2, causing an incorrect Y → X decision.

4. Experiments

The experiments evaluate causal-discovery methods on real-world and simulated data, using the CauseEffectPairs benchmark and controlled simulation scenarios with varied preprocessing and estimator choices.

  • 4.2.2 Simulated data: The study evaluates methods on both real-world benchmark data and simulated data, where the data-generating process and causal ground truth can be controlled.Simulations were included because real-world ground truths are difficult to establish, while simulated settings permit certainty about the generating process.
  • 4.2.1 Real-world benchmark data: The CauseEffectPairs benchmark contains 100 cause-effect pairs from 37 datasets, with experiments restricted to 95 pairs having one-dimensional variables.The benchmark provides real-world pairs with justified causal directions; five multidimensional pairs were excluded.
  • 4.2.2 Simulated data: Simulation data are generated from structural equation models with randomly generated noise distributions and Gaussian-process causal mechanisms, followed by Gaussian measurement noise.The framework can include a confounder Z and allows hyperparameters to control properties of the generated data.
  • 4.2.2 Simulated data: The simulations vary confounding, noise levels, and distributional structure across scenarios including SIM, SIM-c, SIM-ln, and SIM-G.SIM is the default scenario without confounders; SIM-c adds a one-dimensional confounder, SIM-ln uses low noise, and SIM-G approximates Gaussian causes and additive Gaussian noise.
  • 4.3 Preprocessing: The evaluation standardizes both variables and tests unperturbed, discretized, and undiscretized versions to assess sensitivity to discretization and small data perturbations.The discretization procedure merges values to equalize the number of unique values, while undiscretization adds uniform noise between neighboring observed values.

5. Results

Results show that ANM and IGCI performance depends strongly on estimator details, data characteristics, and perturbations. HSIC-based ANM methods are consistently competitive, whereas several entropy-based and IGCI variants are sensitive to discretization, base measures, and model mismatch.

  • 5.1.1 HSIC-based scores: 63%–85% accuracy was achieved by HSIC-based ANM methods across datasets, with robustness to small perturbations despite additive-noise model misspecification.The data can violate additivity, include confounding, and contain measurement noise.
  • 5.1.1 HSIC-based scores: ANM-pHSIC and ANM-HSIC performed similarly, and implementation differences were generally small and statistically insignificant.Data-splitting slightly improved CEP accuracy, while fixed kernels slightly reduced AUC.
  • 5.1.2 Entropy-based scores: Entropy-based ANM performance varied greatly by estimator on CEP, with discretization causing nondecisions or below-chance results for several estimators.ANM-ent-KDP and ANM-ent-PSD were the only nonparametric entropy-based methods performing well on both benchmark and simulated data; PSD appeared more robust to perturbations.
  • 5.2 Information Geometric Causal Inference: IGCI performance depended more on the base measure than on the estimation method, ranging from below chance to above 90% accuracy on simulated data.The Gaussian base measure produced substantially different outcomes across simulation settings.
  • 5.2 Information Geometric Causal Inference: IGCI can fail when causal relations are insufficiently nonlinear and the assumed base measure mismatches the actual cause distribution.Performance may fall below chance under this mismatch.
  • 5.2 Information Geometric Causal Inference: On CEP data, IGCI variants were often better than chance but showed strong sensitivity to implementation details and small perturbations.Some nonparametric estimators exhibited accuracy inversions after perturbation, while parametric estimators were more robust.
  • 5.2.2 Nonparametric entropy estimators: Entropy-estimator implementation details can produce large, poorly understood performance differences across ANM and IGCI.Discretization effects and repeated values are recurring sources of instability.

6. Discussion and Conclusion

The paper finds that some bivariate methods can distinguish cause from effect in observational data, while ANM methods are more consistently effective than IGCI methods. However, the real-world benchmark is too small for statistically significant conclusions, and further validation and methodological improvement are needed.

  • Benchmark: The CauseEffectPairs benchmark contains 100 real-world pairs and is combined with simulated datasets to evaluate bivariate causal discovery methods.The benchmark includes author-justified ground-truth directions for its pairs.
  • Main conclusions: ANM methods using HSIC perform reasonably well across all evaluated datasets, with ANM-pHSIC reaching 63 ± 10% accuracy and 0.74 ± 0.05 AUC on the CEP benchmark.ANM-ent-PSD achieves higher accuracy of 69 ± 10% but lower AUC of 0.68 ± 0.06; these differences are not statistically significant.
  • Main conclusions: IGCI performance varies greatly with implementation details, data perturbations, and data characteristics, and no IGCI implementation performs well across all evaluated datasets.The authors report that some IGCI methods appear better than chance on CEP, but this remains unexplained.
  • Theoretical contribution: The original ANM-pHSIC method was among the best overall, motivating a consistency proof for HSIC-based ANM methods.The authors expect extending the consistency result to the multivariate case to be straightforward.
  • Methodological caveats: Discretization harms several methods, especially slope-based IGCI and nonparametric differential-entropy estimators, whereas HSIC and PSD entropy estimates are more robust to small perturbations.The paper suggests that more discretization-robust entropy estimators could improve performance.
  • Limitations: The CEP benchmark is too small to support statistically significant conclusions after accounting for the many method comparisons.The authors estimate that about 120 weighted pairs would be needed for 65% accuracy to be significant, compared with 37 weighted pairs currently available.
  • Implications: The study provides evidence that causal direction can be inferred from statistical patterns in purely observational real-world data, but current methods require further improvement for practical applications.The authors also state that extending current methods to settings with confounding variables remains unclear.

Appendix A. Consistency Proof of ANM-HSIC

Appendix A establishes the theoretical basis for ANM-HSIC consistency by combining HSIC properties with asymptotic results for nonparametric regression. It also distinguishes data splitting from data recycling in the proof setup.

  • Proof strategy: The appendix proves consistency for an ANM-HSIC algorithm closely related to Hoyer et al.’s original method.The proof uses the HSIC statistic as the score, allows data splitting, and fixes the HSIC kernel.
  • Proof strategy: The appendix’s consistency analysis therefore links independence testing of residuals with the asymptotic behavior of the regression procedure used to obtain them.This connection is reflected in the appendix’s separate HSIC and regression components.
  • Proof strategy: The proof is organized around HSIC definitions and asymptotic properties, followed by asymptotic properties of nonparametric regression and their combination.These components are presented in Sections A.1–A.3.

A.1 Consistency of HSIC

The paper establishes consistency properties for empirical HSIC under bounded-kernel assumptions and develops a continuity result needed for the ANM-HSIC proof. The reported consistency result is restricted to fixed kernels.

  • HSIC consistency: With a characteristic product kernel, population HSIC equals zero if and only if the two variables are independent.This property makes HSIC suitable for detecting residual independence.
  • HSIC construction: The empirical HSIC estimator is defined from kernel Gram matrices and a centering matrix applied to paired observations.The paper uses the original biased HSIC estimator rather than later unbiased estimators.
  • HSIC consistency: Empirical HSIC converges in probability to population HSIC as the sample size tends to infinity for non-negative bounded kernels.This follows from the stated large-deviation result for i.i.d. samples.
  • Scope limitation: The appendix proves consistency only for fixed kernels because the authors do not know corresponding results for adaptively estimated kernel parameters.This is an explicit scope limitation of the theoretical result.
  • Continuity result: For real-valued Y, empirical HSIC is Lipschitz-continuous in the corresponding argument when the kernel on Y is bounded and Lipschitz-continuous.The continuity constant scales at least as N^-1/2 as N increases, and this result is identified as key to the consistency proof.

A.2 Consistency of nonparametric regression

The appendix formalizes regression consistency, residual estimation, and two training/testing schemes for ANM-HSIC. It shows when regression methods are suitable for testing residual independence.

  • Residual construction: The analysis distinguishes true residuals from predicted residuals obtained by applying an estimated regression function to test data.Predicted residuals use the fitted regression function and are central to the subsequent independence test.
  • Consistency definitions: The appendix defines weak consistency through vanishing expected risk on an independent test datum and weak universal consistency across distributions with finite second moments.Training error need not vanish merely because expected risk does, motivating separate conditions for data recycling.
  • Proof setup: The proof setup considers data splitting and data recycling as alternative ways to form training and test datasets from an i.i.d. sequence.In data recycling, the same observations are used for regression and independence testing.
  • Regression requirements: A regression method is suitable when the expected mean squared error between true and predicted residuals vanishes asymptotically.The definition averages over both training and test data.
  • Regression requirements: In data splitting, any regression method weakly consistent for the distribution is suitable, whereas data recycling additionally requires the training-error condition.The two scenarios differ because training and test data are independent only under data splitting.
  • Regression requirements: Under finite second moments, any weakly universally consistent regression method is suitable in both directions under data splitting.This applies to regressing X on Y and Y on X.

A.3 Consistency of ANM-HSIC

The paper proves that ANM-HSIC consistently identifies the causal direction when the data satisfy an additive noise model in exactly one direction and the regression and kernel conditions hold.

  • A.3 Consistency of ANM-HSIC: The proof shows that test-set HSIC on estimated residuals converges in probability to population HSIC on true residuals.
  • A.3 Consistency of ANM-HSIC: Algorithm 1 with the HSIC score is consistent for estimating the direction of an additive noise model.The result assumes suitable regression in both directions and a bounded, non-negative, Lipschitz-continuous kernel whose product is characteristic.
  • A.3 Consistency of ANM-HSIC: The procedure estimates both regression functions, forms predicted residuals, and compares their dependence with the corresponding inputs.The regressions estimate E(Y | X) and E(X | Y) from training data before residual dependence is evaluated.
  • A.3 Consistency of ANM-HSIC: The population HSIC pattern separates directions: it is zero for the true additive-noise direction and positive for the reverse direction.This follows because HSIC detects independence under the characteristic-kernel assumption.

Appendix B. Relationship between scores (11) and (12)

Appendix B derives a close relationship between the Gaussian score and the empirical-Bayes score by expressing Gaussian-process fitting through penalized likelihood and marginal likelihood.

  • Appendix B. Relationship between scores (11) and (12): The conditional score decomposes into a Gaussian input likelihood term and a complexity penalty.
  • Appendix B. Relationship between scores (11) and (12): The input term estimates Gaussian entropy, while the conditional term is related to an empirical estimator of residual entropy.
  • Appendix B. Relationship between scores (11) and (12): The empirical-Bayes score is closely related to the Gaussian score for the direction X → Y.
  • Appendix B. Relationship between scores (11) and (12): Gaussian-process penalized likelihood is maximized at the mean posterior function, with value equal to the GP marginal likelihood.The corresponding negative objective equals the negative log marginal likelihood of the data.

Appendix C. Details on the simulated data

The simulated-data procedure samples an ordered Gaussian-process input, draws a Gaussian-process function, and converts its exponentiated values into a cumulative random density.

  • Appendix C. Details on the simulated data: The procedure begins by sampling X from a standard-normal distribution and sorting it in ascending order.
  • Appendix C. Details on the simulated data: A Gaussian-process realization is sampled at the sorted inputs using a specified kernel.
  • Appendix C. Details on the simulated data: Exponentiating and integrating the interpolated function with the trapezoidal rule produces the cumulative vector G.
  • Appendix C. Details on the simulated data: The paper denotes this sequence of sampling operations as one complete sampling procedure.

C.2 Sampling cause-effect pairs

Cause-effect pairs are simulated with Gaussian processes driven by independent noise variables, with separate constructions for unconfounded and confounded settings and added measurement noise.

  • C.2 Sampling cause-effect pairs: The simulations first sample three noise variables and standardize the noise samples before generating the observed variables.
  • C.2 Sampling cause-effect pairs: In the unconfounded construction, X is generated from E_X and Y from X together with E_Y.
  • C.2 Sampling cause-effect pairs: In the confounded construction, both X and Y additionally depend on E_Z.
  • C.2 Sampling cause-effect pairs: Measurement noise is added after the generated variables are standardized.
  • C.2 Sampling cause-effect pairs: The simulations use four scenarios—without confounders, with a confounder, with low noise, and with nearly Gaussian X—and generate 100 pairs per scenario.Each pair contains N = 1000 samples.

Appendix D. Description of the CauseEffectPairs benchmark set

Version 1.0 of CauseEffectPairs contains 100 cause-effect pairs from 37 datasets, selected across domains with causal directions justified primarily by variable meanings and data-generation processes. The appendix documents selection criteria, pair coverage, and limitations of establishing ground truth without interventions.

  • Limitations: Interventional validation is often unavailable because original data-generating systems no longer exist or interventions are impractical, while undocumented variable definitions add uncertainty.
  • Selection criteria: The benchmark requires dependent variables, a significant cause-effect relationship, values in R^d, and at least a few hundred samples per pair.
  • Ground truth: Causal directions were chosen when the variable meanings and data-generating processes made the ground truth known or obvious.For example, altitude is treated as causing temperature, while the reverse intervention would not change altitude.
  • Benchmark overview: Version 1.0 contains 100 cause-effect pairs from 37 datasets spanning multiple domains.The collection extends an earlier eight-dataset benchmark and labels the pairs pair0001 through pair00100.
  • D1: DWD: The DWD subset includes six weather-station pairs involving altitude, latitude, longitude, temperature, precipitation, and sunshine duration.
  • Limitations: Some benchmark pairs retain possible confounding or selection-bias concerns, including latitude–temperature and alcohol-consumption–blood-test relationships.The alcohol dataset also leaves reverse causation possible, although the authors consider it unlikely.

D11: B. Janzing’s meteo data

The meteo-data subsection assembles weather observations into several cause-effect pairs, using temporal ordering and physical reasoning to assign directions. It covers seasonal, radiative, temporal, traffic, indoor–outdoor, and ozone-related relationships.

  • Pair coverage: The benchmark includes six meteo pair groups: day of year–temperature, solar radiation–temperature, five-day weather-state pairs, type of day–cars, outdoor–indoor temperature, and temperature–ozone.
  • Radiation and temperature: Solar radiation is treated as causing daily temperature because artificial shading could reduce air temperature, while reverse effects are expected mainly over longer timescales.
  • Temporal pairs: The weather-state vector at day t is treated as causing the same variables at day t+1 because causal influence does not propagate backward in time.The four-dimensional vector combines temperature, near-surface pressure, sea-level pressure, and relative humidity.
  • Traffic pair: The type of day is treated as causing daily traffic, since a political holiday could change traffic whereas changing traffic would not change the calendar classification.
  • Indoor and outdoor temperature: Outdoor temperature is expected to have the stronger effect on indoor temperature because the surrounding environment has much greater heat capacity.

D18: Yahoo database

This subsection describes benchmark pairs built from financial, web-server, room-temperature, experimental, medical, climate, demographic, and food-consumption data. The proposed directions are justified by ownership, physical asymmetry, experimental control, dataset construction, or temporal and demographic reasoning.

  • Financial pairs: The Yahoo database contributes three stock-return pairs linking Hang Seng Bank–HSBC, Hutchison–Cheung Kong, and Cheung Kong–Sun Hung Kai Properties.The stock returns were computed from adjusted closing prices after interpolating a few missing values.
  • Financial pairs: Ownership structures motivate the financial directions: HSBC owns 60% of Hang Seng Bank, while Cheung Kong owns about 50% of Hutchison.
  • Web-server pair: Internet connections are treated as causing bytes transferred because additional accesses increase data transfer, whereas more transferred data does not create an access.Caching makes the relationship less deterministic but does not remove the proposed direction.
  • Experimental pair: The experimental parameter is treated as causing the participant’s answer because the experimenter directly set the parameter.
  • Other benchmark pairs: Several directions rely on domain or construction logic, including symptoms causing an expert-system decision and population growth causing food-consumption growth.The latter may be confounded by food availability, agriculture, conflicts, markets, and other economic factors.

D25: Light response data

The light-response and subsequent benchmark descriptions cover ecological, demographic, biological, housing, snowfall, bone-density, and soil-process pairs. Their causal directions are assigned using known biological mechanisms, temporal ordering, physical properties, and ordinary domain knowledge.

  • D25: Light response data: The light-response data contain 721 measurements of NEP and direct, diffuse, and total PPFD, with NEP treated as driven by these light measures.NEP represents net CO2 exchange and depends on available light.
  • D26: CO2 flux: Temperature is treated as causing nighttime CO2 flux because ecosystem respiration depends largely on temperature, while the flux has no direct effect on temperature.The benchmark includes three sites: BE-Bra, DE-Har, and US-PFa.
  • Socioeconomic and biological pairs: The remaining pairs use domain-based directions such as population causing employment, measurement time causing milk protein content, and apartment size causing rent.
  • Environmental and health pairs: Temperature is treated as causing snowfall, while age is treated as causing relative bone-mineral-density change.The snowfall relationship may include a small feedback effect and confounders such as clouds.
  • Soil and decomposition pairs: Soil texture is linked directionally to moisture and organic-carbon storage, while earlier root decomposition is treated as affecting later decomposition.The soil examples use clay content as the cause and soil moisture or organic carbon as effects.

Appendix E. Computation time

The appendix reports total computation times for benchmark sets and method implementations, noting that measured times are upper bounds because implementations were not heavily optimized.

  • Reported computation times sum the processing time across individual cores on a 40-core, 125 GB RAM machine.The authors caution that limited optimization means these measurements should be interpreted as upper bounds.
  • IGCI implementations are about two orders of magnitude faster than ANM in this setting because they do not require non-parametric regression.For ANM, Gaussian Process regression accounts for most computation time, while HSIC tests and entropy estimators are relatively quick.
  • ANM-MML is a notable computational outlier because estimating the marginal distribution's MML takes substantial time.
Loading 1412.3773v3…