PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Reasoning from Meta-reinforcement Learning

Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, Zeb Kurth-Nelson

arXiv:1901.08162v1cs.LGcs.AIstat.ML

TL;DR

This paper asks whether agents can learn causal reasoning rather than only discover correlations. It trains a recurrent agent with model-free reinforcement learning across causal tasks, and reports causal inference from observational data, useful interventions, and counterfactual predictions in novel situations.

  • Problem

    Many machine-learning methods discover correlations, while important questions require reasoning about causes and effects and agents must learn from active as well as passive experience.

    Source: §1 · p. 1

  • Method

    The authors train a recurrent agent with model-free reinforcement learning across tasks, using held-out causal networks to assess transfer of its learning procedure.

    Source: Causal Graphs, Observations, and Actions · p. 4, §2 · p. 11

  • Results

    Across the tested settings, agents learned informative intervention policies; they also used observational data for do-calculus and interventional data to resolve confounding when the relevant graph conditions held.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results demonstrate that causal reasoning can arise from model-free reinforcement learning and suggest end-to-end learning may offer advantages for causal inference in complex settings.

    Source: §6 · p. 8

    The work tests a simple domain, and follow-up work should scale to larger environments, more complex causal structures, and more diverse tasks.

    Source: §6 · p. 8

Abstract

from arXiv · show

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.

1. Introduction

Many important questions require reasoning about causes and effects, beyond discovering correlations. This paper investigates whether meta-reinforcement learning can produce causal reasoning, including through agents’ active interactions with their environments.

1. Introduction

  • Questions about whether smoking causes cancer or discrimination caused a job denial require reasoning about causes and effects, not correlations alone.Pattern-finding has succeeded in several areas, but some problems are intrinsically causal.Source: §1 · p. 1
  • The paper asks whether meta-learning can produce causal reasoning without explicitly giving the agent a notion of causality.Formal approaches to causal reasoning already exist; meta-learning instead learns the inference procedure directly from data.Source: §1 · p. 1
  • The authors focus on reinforcement learning because agents can learn about causes through active interaction as well as passive observation.End-to-end meta-learning may also find representations suited to the required causal inferences.Source: §1 · p. 1

2. Problem Specification and Approach

The paper examines observational, interventional, and counterfactual reasoning, then frames causal learning as an inference procedure acquired through meta-learning. Its causal examples distinguish associations from effects and explain when direct interventions are needed.

2. Problem Specification and Approach

  • The three settings test passive observational inference, causal-effect estimation from interventions, and counterfactual prediction after learning causal structure.The counterfactual task asks what would have happened under a different intervention in the previous timestep.Source: §2 · p. 2

2.1. Causal Reasoning

  • In the exercise example, age can confound the association between exercise and cardiac health, so conditioning on exercise alone cannot establish its causal effect.The causal effect excludes the path through the common cause age.Source: §2.1 · p. 2
  • Do-calculus estimates an intervention’s effect from observational data by reasoning in a graph modified to remove the intervened node’s incoming edge.The other conditional distributions remain unaltered in the modified graph.Source: §2.1 · p. 2
  • With unobserved confounders, many graph structures require collecting data directly from an intervened graph to compute causal effects.The agent’s interventional setting provides such interventions by fixing a variable and observing the remaining variables.Source: §2.1 · p. 2
  • Counterfactual questions concern what would have happened in a different, unrealized situation, beyond predicting effects while accounting for causal structure and confounding.The example asks about the cardiac health a person who died of a heart attack would have had under a different exercise history.Source: §2.1 · pp. 2–3

2.2. Meta-learning

  • Meta-learning learns aspects of the learning algorithm from data, and earlier work applied it to optimizers, initial parameters, metric spaces, and external memory.Here, the authors investigate whether the learned inner-loop algorithm can become causally aware.Source: §2.2 · p. 3
  • Meta-learning parameterizes a learning algorithm as an RNN trained by model-free reinforcement learning across problems, so its recurrent dynamics can learn within new episodes.The outer loop trains the network weights; the inner-loop algorithm operates through activation dynamics even after those weights are frozen.Source: §2.2 · p. 3

3. Task Setup and Agent Architecture

Each episode presents a causal Bayesian network, an information phase, and a final quiz; held-out graphs test whether the trained agent transfers its learning procedure. The agent uses recurrent memory and reinforcement-learning training to select interventions and answer the quiz.

3. Task Setup and Agent Architecture

  • Episodes divide into an information phase for gathering evidence and a final quiz step that rewards using the acquired causal knowledge.The agent interacts with a different causal Bayesian network in each episode.Source: §3 · p. 3

Causal Graphs, Observations, and Actions

  • The agent is tested on 408 held-out graphs, including all graphs in the equivalence classes of 12 randomly selected graphs, with no equivalent graph seen during training.Training graphs were sampled from 59,049 possible acyclic graphs on five nodes with edge weights from {−1, 0, 1}.Source: Causal Graphs, Observations, and Actions · pp. 3–4
  • Information actions set an intervened node to 5; in the quiz phase, the environment randomly sets one non-hidden node to −5, and the agent is rewarded with the sampled value of its selected node.Source: Causal Graphs, Observations, and Actions · pp. 3–4
  • The active condition requires both choosing which nodes to intervene on and inferring the causal network; the random baseline selects intervention nodes randomly.The default information phase allows four interventions, the minimum generally needed to resolve the structure in the noise-free limit.Source: Causal Graphs, Observations, and Actions · p. 4

Agent Architecture and Training

  • A recurrent LSTM receives the current observation, previous action, and previous reward, and produces policy logits and a value baseline.Training uses asynchronous advantage actor-critic; after training, the agent is tested with learning rate set to zero on held-out graphs.Source: Agent Architecture and Training · p. 4

4. Experiments

The experiments test causal reasoning across observational, interventional, and counterfactual data settings, measuring reward on held-out causal Bayesian networks. Across these settings, agents show distinct advantages when they can use interventions or specific latent randomness.

4. Experiments

  • Performance was measured by quiz-phase reward across 1,632 episodes on 408 held-out test CBNs, with four possible external interventions per CBN.The experiments varied the information available during the information phase.Source: §4 · p. 4

4.1. Experiment 1: Observational Setting

  • In the observational setting, the Active-Conditional Agent outperformed the Random-Conditional Agent and its illustrated choice matched causal rather than associative predictions.The figure contrasts conditional predictions with the intervened-graph values used for causal inference.Source: Figure 2 · p. 5, Results · p. 5

Results

  • Active agents outperformed their random counterparts in conditional, interventional, and counterfactual settings, indicating tailored choices of observations or interventions.In the counterfactual setting, active selection served a counterfactual objective.Source: Figure 4 · p. 5, Results · pp. 5–7, Figure 5 · p. 6

4.2. Experiment 2: Interventional Setting

  • With unobserved confounders, the Active-Interventional Agent significantly outperformed the Active-Conditional Agent (p = 10−5), where observational data alone cannot resolve the confounder.The broader comparison was marginally better for the Active-Interventional Agent (p = 0.06).Source: Figure 3 · p. 6, Results · p. 6

4.1. Experiment 1: Observational Setting

  • Unconditional observational agents performed worse than the Active-Conditional Agent because they received less diagnostic information, though longer episodes provided more information.The conditional agents observed samples with a selected node set to an unusual value, while remaining observational rather than intervening.Source: §4.1 · p. 5, Results · p. 6

4.3. Experiment 3: Counterfactual Setting

  • The Active-Counterfactual Agent earned higher reward than the Active-Interventional Agent, with the advantage confined to cases where maximum expected values were degenerate.When maximum values were distinct, the agents received equal rewards; inferred randomness helped distinguish nodes tied in mean value.Source: Figure 6 · p. 7, Results · p. 7

5. Summary of Results

The agents used causal information across observational, interventional, and counterfactual settings, including on previously unseen causal networks. Across the settings, they learned active data-collection policies that outperformed random collection.

5. Summary of Results

  • A recurrent network trained with model-free reinforcement learning performed causal reasoning on held-out causal networks it had not previously seen.The reported capabilities included passive causal inference, active intervention, and counterfactual prediction.Source: §5 · pp. 7–8
  • Across all three data settings, agents learned non-random policies for choosing which nodes to observe or intervene on, achieving higher quiz performance than random data collection.The policy lets agents control the data they receive during the information phase.Source: §5 · p. 8
  • With observational data, agents performed better than the highest reward achievable without causal knowledge when intervention predictions differed from correlation-based predictions.This distinction occurred when the externally intervened node had a parent and the intervention changed the graph.Source: §5 · p. 8
  • With interventional data, agents resolved unobserved confounders and outperformed observational-only agents when the intervened node shared an unobserved parent with other variables.The reported advantage was limited to cases with that shared confounder.Source: §5 · p. 8
  • With counterfactual access, agents outperformed interventional-only agents when latent randomness distinguished nodes with the same maximum mean value.The additional access was to the specific randomness in the test phase.Source: §5 · p. 8

6. Discussion and Future Work

The paper presents model-free reinforcement learning as a way for agents to learn causal reasoning and active experimentation. It discusses possible advantages over formal methods while noting that the work tests the approach in a simple domain.

6. Discussion and Future Work

  • The authors describe this as the first direct demonstration that causal reasoning can arise from model-free reinforcement learning.They present it as groundwork for a meta-reinforcement-learning approach that may offer advantages in complex real-world settings.Source: §6 · p. 8
  • Unlike formal approaches that decouple causal induction from inference, the model learns both end-to-end, potentially tuning representations to the inferences required.The paper notes that induction can be expensive and often requires simplifying assumptions (Sen et al., 2017).Source: §6 · p. 8
  • Meta-learning shifts much of the compute burden from inference time to training time, which suits settings with ample training time but a need for fast runtime answers.The paper contrasts this with formal methods, whose induction and inference steps can make new queries slow.Source: §6 · p. 8
  • The active intervention policy significantly outperformed a random intervention policy, supporting end-to-end learning of experimentation alongside causal reasoning.The agents learn actions that produce useful information.Source: §6 · p. 8
  • The work uses a simple domain; future work should scale the approach to larger environments, more complex causal structures, and more diverse tasks.The authors also suggest that more advanced or compositional architectures may help with longer, more complex episodes.Source: §6 · p. 8

1. Additional Baselines

The paper compares two standard model-free reinforcement-learning baselines, which differ in whether action values use episode history. Their reward outcomes reflect different limits on learning across episodes and within an episode.

1. Additional Baselines

  • The Q-total Agent received −1.247±2.940 reward, consistent with random action selection because action-reward relationships vary between episodes.It learns one Q-value per action across all steps and episodes.Source: §1 · p. 11
  • The Q-episode agent received 0.080±2.077 reward by avoiding the intervention arm indicated by m_t and choosing randomly otherwise.It conditions action values on the current input at each time step but has no LSTM memory for prior actions and observations.Source: §1 · p. 11
  • The Q-total Agent shares action values across episodes, whereas the Q-episode agent conditions them on the current input without recurrent memory.Both are described as standard model-free RL baselines.Source: §1 · p. 11

2. Formalism for Memory-based Meta-learning

The memory-based meta-learning setup trains an RNN agent across a distribution of MDP tasks. Its recurrent policy can condition each action on the episode trajectory.

2. Formalism for Memory-based Meta-learning

  • Training samples tasks from a distribution D over Markov Decision Processes, with an RNN-based agent learning across those tasks.Each episode samples a task m ∼ D.Source: §2 · p. 11
  • At each step, the agent receives an observation and reward and takes an action, with the previous action and reward also supplied as inputs.The inputs include o_t, a_(t−1), and r_(t−1).Source: §2 · p. 11
  • Through recurrence, each action can depend on the full trajectory of observations, actions, and rewards from the episode.The trajectory-dependent function is parameterized by the neural network, with complexity limited by network size.Source: §2 · p. 11

3. Abduction-Action-Prediction Method for Counterfactual Reasoning

The paper applies Pearl et al. (2016)’s abduction-action-prediction method to ask what an individual’s cardiac health would have been under a different exercise level. It estimates the individual’s latent makeup, changes exercise, and predicts health in the counterfactual world.

3. Abduction-Action-Prediction Method for Counterfactual Reasoning

  • Action sets exercise to the counterfactual level e′.The query asks what cardiac health would have been had the individual exercised more.Source: §3 · p. 11
  • Abduction estimates individual i’s latent health-related makeup from observed age, exercise, and cardiac health.The example computes ϵ_i = h_i − w_AH a_i − w_EH e_i.Source: §3 · p. 11
  • Prediction combines the individual’s observed age and estimated latent makeup with the counterfactual exercise level to predict cardiac health.The example computes h′ = w_AH a_i + w_EH e′ + ϵ_i.Source: §3 · p. 11

4. Additional Experiments

After proof-of-concept experiments on a simple tractable system, the paper tests causal reasoning in nonlinear, non-Gaussian graphs and larger causal graphs. In both experiments, the agent’s reward comparisons indicate that it can use observational data and interventions, including selecting informative interventions.

4. Additional Experiments

  • The experiments extend the earlier proof of concept from a simple tractable system to more complex causal systems.The added experiments cover nonlinear, non-Gaussian causal graphs and graphs with six nodes.Source: §4 · p. 11, §4.1 · p. 11, §4.2 · p. 12

4.1. Experiment 4: Non-linear Causal Graphs

  • In quadratic causal graphs, Active-Interventional earns more reward than both Long-Observational and Random-Interventional agents.The Long-Observational advantage over Observational indicates learning of within-episode statistical dependencies; the active-agent comparisons indicate reasoning from and selecting informative interventions. Each node is normally distributed given its parents, but nonlinearity makes the joint distribution non-Gaussian.Source: §4.1 · pp. 11–12, Figure 2 · p. 11

4.2. Experiment 5: Larger Causal Graphs

  • For N = 6 graphs, rewards follow Observational < Active-Conditional < Active-Interventional < Active-Counterfactual, and Active-Interventional exceeds Random-Interventional.The graphs contain 1.4×10^7 unique CBNs, compared with 5.9×10^4 for N = 5; the intervention comparison indicates selection of informative interventions.Source: §4.2 · p. 12, Figure 3 · p. 12

5. Causal Bayesian Networks

This section introduces graphical and probabilistic concepts used to represent causal structure and assess statistical independence. It describes how graphs encode variables and relationships, how Bayesian networks factor a joint distribution, and how path structure determines conditional independence.

5. Causal Bayesian Networks

  • A graph consists of nodes connected by links, and a path is a sequence of linked nodes; directed paths follow the link directions.A graphical model uses nodes for random variables and links for statistical relationships.Source: §5 · p. 12
  • For the quadratic case, calculating p(X1:N\j|Xj = 5), and therefore Conditional Agents, was non-trivial.This is a computational qualification on the described case.Source: §5 · p. 12
  • A directed acyclic graph has no directed path that starts and ends at the same node; adding a link from X4 to X1 in the illustrated graph makes it cyclic.Figure 4 contrasts the acyclic graph with the graph after that link is added.Source: §5 · p. 12, Figure 4 · p. 12
  • In a Bayesian network, each node has a conditional distribution given its parents, and the joint distribution is the product of those conditional distributions.With causal semantics, the network represents both causal and statistical relationships.Source: §5 · p. 12
  • Variables X and Y are independent given Z when every path between them is blocked, including paths blocked by a non-collider in Z or a collider with neither it nor its descendants in Z.A collider can be a collider on one path and a non-collider on another.Source: §5 · p. 12
Loading 1901.08162v1…