PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta Learning for Causal Direction

Jean-Francois Ton, Dino Sejdinovic, Kenji Fukumizu

arXiv:2007.02809v2stat.MLcs.LG

TL;DR

Randomized trials are often inaccessible, leaving causal direction to be inferred from observational data. The paper meta-trains an adaptive generative model using distributional features from datasets, and reports high performance at small dataset sizes.

  • Problem

    Randomized trials are often inaccessible, motivating the problem of distinguishing cause from effect using limited observational data.

    Source: Introduction · p. 1

  • Method

    An encoder-decoder generative model uses distributional dataset features to adapt its generator and leverage similar datasets for unseen causal pairs.

    Source: Introduction · p. 2

  • Results

    High performance on small dataset sizes can be achieved through meta learning.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The adaptive model exploits structural asymmetries without requiring retraining at test time.

    Source: Introduction · p. 2

    Training assumes the causal direction is known for every training dataset.

    Source: Functional Causal Model · p. 2

Abstract

from arXiv · show

The inaccessibility of controlled randomized trials due to inherent constraints in many fields of science has been a fundamental issue in causal inference. In this paper, we focus on distinguishing the cause from effect in the bivariate setting under limited observational data. Based on recent developments in meta learning as well as in causal inference, we introduce a novel generative model that allows distinguishing cause and effect in the small data setting. Using a learnt task variable that contains distributional information of each dataset, we propose an end-to-end algorithm that makes use of similar training datasets at test time. We demonstrate our method on various synthetic as well as real-world data and show that it is able to maintain high accuracy in detecting directions across varying dataset sizes.

Introduction

Because randomized trials are often inaccessible, the paper addresses causal direction from observational data by meta-learning across datasets. It introduces an adaptive generative model for unseen, potentially small datasets and reports high performance in small-data settings.

Introduction

  • Randomized trials are often infeasible for physical, financial, or ethical reasons, while causal-blind machine learning may generalize poorly to different data.Source: Introduction · p. 1
  • CGNN fits neural generative models separately for each direction and requires large datasets; this can hurt small-data performance and make inference slow.Source: Introduction · p. 1
  • Dataset features encode distributional information so datasets with similar causal mechanisms receive similar representations, using kernel mean embeddings or DeepSets.Source: Introduction · p. 2
  • An encoder-decoder generative model uses dataset features and FiLM to adapt its generator for direction inference on unseen, possibly small observational datasets.Source: Introduction · p. 2
  • The method leverages similar datasets for unseen causal pairs, exploits structural asymmetries without test-time retraining, and assumes no a priori causal mechanism.Source: Introduction · p. 2
  • High performance on small dataset sizes can be achieved through meta learning.Source: Introduction · p. 2

Meta Learning for Detecting Causal Direction

The section presents meta-CGNN, which learns across known-direction cause-effect datasets and transfers distributional information to unseen datasets. Its dataset features and FiLM-adapted decoder support conditional generation, with direction chosen by comparing generated-data fit.

Functional Causal Model

  • For bivariate inference, the Functional Causal Model represents the effect as a function of the cause and independent noise; CGNN fits both directions and compares their fit.The model uses neural networks to learn each proposed mechanism.Source: Functional Causal Model · p. 2

Meta Learning

  • Meta-learning shares statistical strength across known-direction datasets so the learned information can be used for causal-direction inference on unseen datasets.Unlike methods that treat datasets independently, this setting uses prior datasets to make inference on a new one.Source: Meta Learning · p. 3

Dataset features via DeepSets

  • DeepSets provides a flexible, permutation-invariant set-to-vector representation, while concatenating examples encodes joint rather than conditional distribution information.The paper therefore also considers conditional mean embeddings as dataset features.Source: Dataset features via DeepSets · p. 3

Dataset features via Conditional Mean Embeddings (CME)

  • Conditional mean embeddings represent conditional distributions and capture the mechanism in a Functional Causal Model, providing distributional information that also encodes causal direction.With finite-dimensional feature maps, their operator can be computed as vector-valued ridge regression; the paper notes efficient computation for small datasets.Source: Dataset features via Conditional Mean… · p. 3

A Feature-wise Linear Modulation (FiLM)

  • FiLM uses a dataset feature to adapt the Functional Causal Model network to the dataset without adding further model parameters.Conditioned on the feature, FiLM learns shift and scale parameters for network layers.Source: A Feature-wise Linear Modulation (FiLM) · p. 4

Proposed method

  • The method encodes each dataset’s distributional information as a feature and uses it to modulate the decoder, enabling conditional generation for an unseen task.The decoder combines a sampled task-conditioned latent variable with the input before generating the effect.Source: Proposed method · p. 4
  • Joint training shares information across datasets in one generative model, unlike CGNN’s separate model for each dataset, and avoids retraining for a new task.At test time, the model compares generated-data fit under the two candidate directions.Source: Proposed method · p. 5
  • At inference, meta-CGNN compares the MMD between generated and observed data under each direction, selecting the direction with the lower MMD.The model shares distributional information across tasks through an encoder trained jointly across datasets.Source: Proposed method · pp. 4–5

Related work

Prior causal-direction work exploits asymmetries in functional causal models or distributional complexity, while CGNN fits directions per dataset. This paper’s meta-learning approach instead shares information across datasets and supports test-time inference without retraining.

Related work

  • Functional causal model approaches exploit causal–anticausal asymmetry, but examples impose assumptions: LiNGAM is linear non-Gaussian, while Hoyer et al. (2009) use nonlinear additive-noise models.Other approaches consider invertible interactions and Gaussian-process regression.Source: Related work · p. 5
  • Information-theoretic approaches use the postulate that the causal mechanism is less complex than the anticausal mechanism, with papers approximating or proxying Kolmogorov complexity.The cited works include Janzing and Scholkopf (2010), Lemeire and Dirkx (2006), Daniusis et al. (2010), and Mitrovic, Sejdinovic, and Teh (2018).Source: Related work · p. 5
  • CGNN (Goudet et al. 2017) handles one dataset at a time, whereas meta-CGNN leverages similarity across datasets and needs only a forward pass for a new dataset.CGNN averages separately trained generative networks; meta-CGNN uses four initializations and reports similar results on larger datasets and better results on smaller ones.Source: Related work · pp. 5–6
  • Other meta-learning methods target different settings: MetaCI addresses counterfactual causality, while Bengio et al. (2020) assume perturbed test distributions and about 3,000 training examples per mechanism.Ke et al. (2019) do not focus on the bivariate setting.Source: Related work · p. 6

Experiments

The experiments evaluate meta-CGNN on synthetic cause-effect mechanisms and the Tuebingen benchmark, across dataset sizes. They report strong small-data performance and test the contributions of task embeddings and FiLM.

Synthetic Datasets

  • Synthetic tests cover CE-Net, CE-Gauss, and CE-Multi mechanisms and compare settings with 1,500 versus 100 datapoints.CE-Multi includes additive and multiplicative noise before or after the causal mechanism.Source: Synthetic Datasets · p. 6
  • AUPRC measures causal-direction discrimination while accounting for algorithm confidence, allowing a model not to predict when uncertain.Accuracy is reported in the supplementary material.Source: Synthetic Datasets · p. 6

Results

  • Across Net, Multi, Gauss, and Tuebingen, meta-CGNN is among the best in AUPRC at both 1,500 and 100 datapoints.The figure contrasts thin bars with blue dots for 1,500 datapoints against thicker bars for 100 datapoints.Source: Figure 2 · p. 7, Results · p. 8

Tuebingen Cause-Effect dataset

  • The Tuebingen evaluation uses 99 bivariate datasets, with five-fold cross-validation repeated across folds and three random splits.The experiments also assess performance as dataset sample size decreases.Source: Tuebingen Cause-Effect dataset · pp. 7–8

Results

  • ANM performs reasonably on CE-Net and CE-Gauss but fails on CE-Multi and Tuebingen, while RECI performs well on CE-Multi but not the other listed datasets.The authors attribute ANM’s pattern mainly to its strict functional causal model assumptions.Source: Results · p. 8
  • At test time, meta-CGNN requires only a forward pass rather than retraining on each dataset, making it more computationally efficient than CGNN while attaining higher performance.Meta-CGNN takes <1 minute per new dataset at inference; CGNN takes about 24 minutes per dataset when averaged over around 32–64 runs. In comparisons, the authors restricted CGNN to 12 runs.Source: Synthetic Datasets · p. 6, Results · p. 8

Ablation study

  • A decoder-only joint CGNN reaches 55–60% accuracy, near chance on Tuebingen, while adding FiLM significantly boosts performance by allowing test-time decoder adaptation.The paper reports similar FiLM trends with CME embeddings.Source: Ablation study · p. 8

Conclusion

The paper concludes that meta-CGNN performs well in small-data cause-effect inference by adapting a shared generative network to unseen datasets. It also identifies conditional mean embeddings as a way to capture distributional information for adaptation.

Conclusion

  • The proposed meta-learning algorithm performs well in the small-data regime, in contrast to existing methods.A learned dataset-feature extractor, amortization, and FiLM layers support efficient adaptation to unseen datasets.Source: Conclusion · p. 8
  • A single generative network can be adapted to new datasets with forward passes, achieving similar or better performance than existing methods while reducing CGNN’s computational burden.The comparison is with methods that train multiple models separately on each dataset.Source: Conclusion · p. 8
  • Given a relevant skeleton graph in a reinforcement-learning model, meta-CGNN could be used to infer directions of unoriented edges.The paper presents this as a possible application.Source: Conclusion · p. 8

Meta Learning

Meta-learning uses experience from prior tasks to learn new tasks more quickly or proficiently, while multi-task learning improves performance across the tasks being learned. Here, known causal directions from multiple datasets inform inference on unseen datasets.

Meta Learning

  • Meta-learning uses data or experience from previous tasks to learn a new task more quickly or proficiently (Finn, 2019).The paper contrasts this with learning all tasks more quickly or proficiently in multi-task learning.Source: Meta Learning · p. 11
  • In this setting, causal directions known for training datasets provide information for inferring directions in new unseen datasets.This contrasts with methods that treat datasets independently and retrain for each new one.Source: Meta Learning · p. 11

Ablation study

The ablations test whether task embedding and FiLM matter when datasets are trained jointly. Removing both performs near chance on the reported datasets, while FiLM improves accuracy by giving the decoder more flexibility.

Ablation study

  • Without the amortization network and FiLM layer, joint decoder training reaches only 55–60% accuracy on Tuebingen and chance-level accuracy on CE-Net, with datasets capped at 1500 points.The authors use this ablation to study the necessity of task embedding.Source: Ablation study · p. 11
  • Adding FiLM to DeepSets embeddings significantly boosts performance by letting the decoder adapt more flexibly at test time than relying on the amortization network alone.The authors report similar trends for meta-CGNN using CME embeddings.Source: Ablation study · p. 11
  • Ablating the amortization network yields similar or slightly higher performance when the network is retained in meta-CGNN.This observation is reported separately from the significant FiLM boost.Source: Ablation study · p. 11

Additional details on experiments setup

The experiments train meta-CGNN across synthetic datasets and use held-out datasets for testing, while Tuebingen uses repeated five-fold cross-validation. The appendix also specifies network architectures and embedding hyperparameters.

Additional details on experiments setup

  • For synthetic data, 100 of 300 datasets train meta-CGNN and the remaining 200 are test sets; standard methods use only those 200 test datasets.Each dataset has either 1500 or 100 datapoints.Source: Additional details on experiments setup · p. 11
  • Tuebingen contains 99 bivariate datasets and uses five-fold cross-validation, repeated three times with different splits, with datasets capped at 1500 or 100 datapoints.Each fold serves as the test fold once while the other four train.Source: Additional details on experiments setup · p. 11
  • The networks use ReLU activations, Adam with learning rate 0.01, up to 500 epochs, and mini-batches of 10 datasets.The encoder, FiLM, and amortization networks have two hidden layers; the decoder has one.Source: Additional details on experiments setup · p. 11
  • DeepSets encoder hidden sizes are [40, 10], with CMEO λ=1 and random-feature lengthscale l=1; these default embedding settings were not cross-validated.The authors state that future work will cross-validate these hyperparameters.Source: Additional details on experiments setup · p. 11
  • The latent variable Z is one-dimensional, the amortization network outputs one-dimensional µz and σz, and FiLM outputs shift and scale parameters for the decoder hidden layer.The decoder uses either 40 or 5 hidden nodes, selected by cross-validation.Source: Additional details on experiments setup · p. 11

AUPRC and Accuracy of the methods

The experiments report AUPRC and accuracy, with weighted accuracy for Tuebingen. The authors emphasize AUPRC because it reflects prediction confidence, including cases where a method should abstain rather than give a wrong answer.

AUPRC and Accuracy of the methods

  • Accuracy follows a similar trend to AUPRC, and Tuebingen results use weighted accuracy to account for similarities among datasets.The passage does not provide numerical metric values.Source: AUPRC and Accuracy of the methods · p. 11
  • AUPRC is shown in the main text because it is more indicative of prediction confidence and accommodates uncertainty, where no answer may be preferable to a wrong one.The authors also report accuracy for the proposed and competing methods.Source: AUPRC and Accuracy of the methods · pp. 11–12
  • Table 3 compares accuracy for the proposed method using DeepSet embeddings with and without FiLM.The caption specifies the comparison but supplies no cell values.Source: Table 3 · p. 12

Note on competing methods and RCC

Across the dataset-size plots, meta-CGNN is usually significantly better than standard CGNN and has the smallest performance drop when sample size falls. The paper also reports faster test-time inference and poor RCC performance outside CE-Multi.

Note on competing methods and RCC

  • The competing-method setup uses 12 CGNN models instead of 32 or 64 because of computational constraints, with a slight performance degradation on 1500-point data.The reported results remain comparable to those noted in the CGNN paper.Source: Note on competing methods and RCC · p. 12
  • CGNN requires about 24 minutes per dataset at test time, whereas meta-CGNN inference takes less than one minute while retaining similar or better performance.The authors describe this as a computational saving at test time.Source: Note on competing methods and RCC · p. 12
  • RCC performs poorly except on CE-Multi, where most other methods also do reasonably well; the authors conclude that CME alone does not solve the problem.Table 4 reports RCC results and identifies CE-Gauss as another dataset where RCC struggles.Source: Note on competing methods and RCC · p. 12, Table 4 · p. 14
  • Meta-CGNN is significantly better than standard CGNN in most cases across AUPRC and accuracy plots, and has the smallest performance drop from 1500 to 100 datapoints.The plots cover Net, Multi, Gauss, and Tuebingen; meta-CGNN is not best on every dataset but remains comparable across them.Source: Figure 3 · p. 13, Figure 4 · p. 13
Loading 2007.02809v2…