PaperlayerView exact revision on arXiv

Source-linked AI summary

An Introduction to Causal Reinforcement Learning

Elias Bareinboim, Junzhe Zhang, Sanghack Lee

arXiv:2606.24160v1cs.AI

TL;DR

The paper addresses the limited interaction between causal inference and reinforcement learning, despite their shared focus on counterfactual decision-making. It combines structural causal models with RL, unifies existing learning modes, and introduces broader causal policy-learning settings. The resulting CRL framework supports more robust reasoning under uncertainty and expands learning to intervention choice, imitation, and counterfactual policies.

  • Problem

    Most RL systems do not explicitly represent causal models, although effective real-world decision-making relies on understanding environmental causal mechanisms.

  • Method

    The paper combines Pearl’s structural causal models with RL to encode causal knowledge, perform counterfactual reasoning, and formalize diverse learning settings.

  • Results

    The CRL framework yields algorithms resilient to unobserved confounding in offline off-policy and imitation learning and introduces new intervention and counterfactual learning regimes.

  • Takeaways & Limitations

    Causal knowledge broadens RL beyond passive observation and active intervention to include decisions about where to intervene and counterfactual randomization.

  • Takeaways & Limitations

    A complete characterization of minimality and possible-optimality for contextual mixed-policy spaces remains an open problem.

Abstract

from arXiv · show

Causal inference provides a set of principles and tools that allow one to combine data and knowledge about an environment to reason with questions of counterfactual nature, i.e., what would have happened had reality been different, even when no data of this unrealized reality is currently available. Reinforcement learning provides methods to learn a policy that optimizes a specific measure (e.g., reward, regret) when the agent is deployed in an environment and pursues an exploratory, trial-and-error approach. These two disciplines have evolved independently and with virtually no interaction between them. We note that they operate over different aspects of the same building block, counterfactual relations, which makes them umbilically connected. Based on these observations, novel learning opportunities arise when this connection is explicitly acknowledged and mathematized. To realize this potential, we note that any environment where the RL agent is deployed can be decomposed as a collection of autonomous mechanisms with different causal invariances, parsimoniously modeled as a structural causal model; any standard RL setting implicitly encodes such a model. This formalization allows us to put under a unifying treatment different modes of learning, including online, off-policy, and causal calculus learning, which appear unrelated in the literature. However, these modalities are not exhaustive: we introduce several natural and pervasive classes of learning settings that entail novel dimensions of analysis. Specifically, we introduce and discuss through causal lenses generalized policy learning, where to intervene, imitation learning, and counterfactual learning. These tasks lead to a broader view of counterfactual learning and suggest great potential for studying causal inference and reinforcement learning side by side, which we call causal reinforcement learning (CRL).

1. Introduction

The paper motivates causal reinforcement learning by arguing that robust decision-making requires explicit causal knowledge alongside reinforcement learning. It formalizes this connection and extends standard learning settings with several causal policy-learning tasks.

  • Robust decision-making in domains such as robotics, medicine, and economics depends on understanding the environment’s underlying causal mechanisms.
  • RL faces a curse of dimensionality, while real-world AI systems remain fragile, sample inefficient, opaque, and weakly generalizable.
  • CRL combines causal inference and RL to pursue more sample-efficient, transparent, and robust decision-making systems.
  • The framework represents environmental invariances with causal models and uses the Pearl Causal Hierarchy to organize associational, interventional, and counterfactual reasoning.
  • The paper formalizes policy learning through causal decision models and relates CRL tasks to standard Markov decision-process assumptions.
  • Beyond online, off-policy, and causal-identification settings, the paper introduces causal offline-to-online, mixed-policy, counterfactual, and imitation-learning tasks.

2. Foundations of Causal Inference

The section establishes structural causal models as a foundation for analyzing reinforcement-learning environments, linking environment mechanisms, agent knowledge, and the Pearl Causal Hierarchy. It distinguishes observational from interventional reasoning and shows how causal structure supports richer, more robust decision-making analysis.

  • Pearl Causal Hierarchy: Each SCM induces the Pearl Causal Hierarchy, separating associational, interventional, and counterfactual distributions into qualitatively different layers.These layers formalize the types of interactions and reasoning available to an agent, including observation, intervention, and counterfactual analysis.
  • Structural causal models: An SCM contains exogenous variables, endogenous variables, structural functions, and a probability distribution over the exogenous variables.Exogenous variables represent outside factors, while structural functions determine endogenous-variable values.
  • Structural causal models: CRL models an environment as a structural causal model whose mechanisms and exogenous factors generate the agent’s observed states and outcomes.The SCM framework can represent standard RL environments while preserving their underlying generative processes.
  • Agent and environment representations: The environment is represented by a fully specified SCM M∗, while the agent maintains a parsimonious causal model G capturing relevant environmental invariances.The two perspectives are connected through the compatible pair of SCM M∗ and causal diagram G.
  • Observational and interventional regimes: Passive observational data can support conclusions that differ from those obtained by interventions because behavior policies and agent perspectives may differ.The paper notes that naïvely cloning a nominal behavior policy may fail when observational and interventional distributions are different.
  • Observational and interventional regimes: The SCM description of an MDP specifies distributions across all Pearl Causal Hierarchy layers, unlike a standard MDP’s transition and reward functions, which abstract a single layer.The distinction provides a basis for causal analysis beyond the standard MDP representation.

3. Elements of Causal Reinforcement Learning

The section formalizes reinforcement-learning decision problems through causal decision models and structural causal models, unifying established and new learning settings. It also shows that observational or interventional data alone can leave optimal policies underdetermined in unknown environments.

  • 3.2 Causal Reinforcement Learning Tasks: CRL formalization represents canonical settings such as multi-armed bandits, MDPs, and dynamic treatment regimes within SCM semantics.Traditional off-policy learning, online learning, and causal identification are treated as variations within this framework.
  • 3.1 Causal Decision Models: Policy optimization in an SCM selects interventions over action variables to optimize reward signals through a policy space of sequential decision rules.Each policy specifies action distributions conditioned on the agent’s permitted state information at successive stages.
  • 3.1 Causal Decision Models: A causal decision model combines an environment SCM, a policy space describing agent capabilities, and a reward function measuring performance.The policy space specifies what the agent can control and observe during interaction.
  • 3.2 Causal Reinforcement Learning Tasks: The CRL task signature ⟨L, A, Π, R⟩ captures the learning regime and structural assumptions when the environment SCM is unknown.This extends the fully specified decision model by replacing the unknown environment with available learning information and assumptions.
  • 3.2 Causal Reinforcement Learning Tasks: The framework identifies additional learning dimensions, including imitation learning and tasks involving unknown rewards, beyond traditional off-policy, online, and causal-identification settings.These dimensions motivate a broader treatment of causal reinforcement learning tasks.
  • 3.3 Comparison with Markov Decision Processes: In standard MDPs, solving observational and interventional models can produce different optimal policies, while observationally equivalent environments can disagree about interventional outcomes.Thus, the Markov property and observational data may be insufficient to determine the optimal policy in the underlying causal model.

4. Reinforcement Learning through Causal Lenses

The CRL formalization connects reinforcement-learning algorithms to causal assumptions, clarifying when observational data can evaluate policies and when exploration is required. It unifies off-policy evaluation, online learning, and causal identification while exposing limitations of assumptions such as NUC.

  • Off-policy evaluation: CRL formally relates off-policy learning and causal invariances, providing justification for inverse propensity weighting and dynamic programming.The formalization connects these methods to structural causality and policy evaluation.
  • Off-policy evaluation: Under the NUC condition, observational data can be reweighted to learn a policy without deploying it online.The expected reward is identifiable from observational data, and IPW and DP provide equivalent evaluations for the same policy and data.
  • Off-policy evaluation: When NUC fails because of unobserved confounding, IPW may produce policy evaluations that differ from the true expected reward.In the cited MAB example, the estimated reward is 0 while the actual expected reward is 0.4.
  • Online learning: Online randomized trials ensure NUC but require an exploration–exploitation trade-off, with regret affected by the number of exploratory trials.The RCT regret bound separates exploration regret from exploitation regret, while UCB avoids requiring prior knowledge of the reward gap and total horizon.
  • Causal identification: Causal identification broadens policy evaluation beyond NUC: sequential backdoor assumptions can identify and compute expected rewards using IPW or DP.The sequential backdoor condition may hold even when NUC does not, and the identified effects can be recovered from observational data.

5. Causal Offline-to-Online Learning (CRL Task 1)

This section studies offline-to-online policy learning when observational data may be confounded, introducing causal bounds and algorithms that use such data to accelerate online learning while preserving performance guarantees.

  • Motivation: When the NUC assumption fails, off-policy estimates may be inaccurate because observational rewards need not equal interventional rewards.Unobserved confounding can make a suboptimal arm appear observationally more effective than it is under intervention.
  • Methods: The proposed offline-to-online strategy incorporates observational-data bounds into online learning to evaluate candidate rewards and accelerate identification of optimal actions.The approach combines observational and experimental data, while UCB+ prioritizes causal bounds when they conflict with empirical confidence bounds.
  • Extensions: The framework extends beyond MABs by deriving observational-data bounds for interventional transition probabilities and reward functions in arbitrary causal decision models.A generalized UCB algorithm achieves a near-optimal regret bound without additional observational data or structural knowledge.
  • Motivation: Observational distributions alone cannot identify arm rewards when distinct causal environments generate the same observations but different interventional rewards.This non-identifiability motivates bounds rather than point identification.
  • Causal bounds: Causal bounds use observational distributions to constrain unknown expected rewards, remaining informative and optimal in MAB models without additional assumptions or data.The bounds are estimable from observations and strictly contained in [0, 1] when each arm has positive marginal probability.
  • Results: UCB+ consistently dominates standard UCB, achieving constant regret O(1) with informative bounds versus UCB’s sublinear regret O(log(T)/∆).When bounds are uninformative, UCB+ coincides with UCB and avoids negative transfer.

6. Mixed Policy Learning: Where to Intervene (CRL Task 2)

Mixed policy learning relaxes fixed intervention scopes by allowing agents to choose whether and where to intervene, potentially reducing regret through causal structure and context. The section characterizes minimal and possibly optimal policy spaces, while showing that some contextual optimality questions remain open.

  • Motivation: Agents may choose intervention targets rather than controlling every action variable, because indiscriminate interventions can be harmful or suboptimal.The mixed policy space includes policies that intervene on subsets of variables and policies that use contexts.
  • Motivation: Partial intervention can outperform full control in non-Markovian causal systems, although searching all intervention subsets may require evaluating up to 3^k configurations.The relevant comparison is between do(X′ ←x′) for X′ ⊂ X and do(X ←x).
  • Mixed Policy with No Context: The all-at-once strategy never converges, whereas brute-force search over subsets of {X1, X2} finds the optimal policy; cumulative regret is shown in Figure 29d.Intervening on both variables incurs regret because do(X1 ←1) is optimal.
  • Structural Properties: Causal structure yields equivalence classes and partial orders among action subsets, helping agents focus on minimal arms and relate rewards across interventions.These structural relations can be used without examining collected data.
  • Structural Properties: A minimal intervention set X′ is characterized by X′ ⊆ an(Y)_{G_X′}, and directed-edge structure suffices to identify such sets without unobserved variables.The characterization includes intervening on no variables, one ancestor of Y, or suitable parent sets.
  • Contextual Mixed Policies: With contexts, selectively intervening can achieve optimal reward: intervening on X1 while using context C attains an expected reward of 1.0.The policy sets X1 equal to C, eliminating the noise affecting X1 and making X2 equivalent to U2.
  • Contextual Mixed Policies: The paper provides complete minimality and possible-optimality characterizations without contexts, but complete contextual characterizations remain an open problem.It also gives a procedure for detecting non-minimality when contexts are available.

7. Counterfactual Decision-Making (CRL Task 3)

This section introduces counterfactual decision-making and randomization, using agents’ intended actions and causal information to improve decisions beyond standard interventions. It develops criteria, randomization procedures, and an autonomy–optimality trade-off for bandit and sequential environments.

  • 7.2 Counterfactual Randomization: Counterfactual randomization interrupts an agent before execution, treats its imminent choice as an intention, and evaluates alternative actions using intention-specific randomization.The procedure is designed to let online agents search over counterfactual policies in multi-armed bandits and Markov decision processes.
  • 7.1 Counterfactual Decision Criterion: In the greedy casino, the arms are statistically indistinguishable under individual observational or interventional distributions, so algorithms based on either distribution fail to learn an optimal policy.Combining the distributions and conditioning on predilections enables the counterfactual comparison.
  • 7.1 Counterfactual Decision Criterion: Counterfactual decision-making evaluates actions using the treatment effect on the treated, incorporating the agent’s intended action or intuition.The criterion is extended to Markov decision processes with unobserved confounders.
  • 7.3 The Tradeoff between Autonomy and Optimality: The proposed framework includes an autonomy–optimality trade-off: retaining human input can improve performance, while excessive autonomy can produce suboptimal strategies.A planning algorithm and a budget-constrained task are proposed to balance these objectives.
  • 7.1 Counterfactual Decision Criterion: Counterfactual policies can outperform experimental policies because they condition decisions on information embedded in the agent’s natural predilections.The paper states that an optimal counterfactual policy is never worse than an optimal interventional policy in multi-armed bandits.
  • 7.1 Counterfactual Decision Criterion: The counterfactual policy in the casino achieves an expected reward of 0.45, compared with 0.15 for atomic intervention on either arm.The analysis also reports that the payout rate would have tripled in situations where the chosen action conflicted with natural predilections.

8. Causal Imitation Learning (CRL Task 4)

Causal imitation learning uses graphical criteria to determine when observational demonstrations can support expert-level behavioral cloning, and extends imitation through inverse reinforcement learning when reward knowledge is available.

  • Motivation: Unknown reward-function details make expected policy reward non-identifiable from observational data or a causal diagram.This blocks direct causal-identification approaches such as do-calculus learning.
  • Causal Behavioral Cloning: Under the NUC condition, a behavioral cloning policy matches expert performance by using π_i(X_i | S_i) = P(X_i | S_i).The theorem requires each action’s endogenous parents to be included in its input state.
  • Causal Behavioral Cloning: The imitation backdoor criterion determines whether a policy space contains a policy that can be learned by behavioral cloning.When the criterion holds, conditional action distributions can yield an imitating policy achieving expert performance.
  • Causal Behavioral Cloning: An imitation-admissible subspace can make expert performance achievable by mimicking P(X_i | Z_i) for each action.A polynomial-time procedure can find such a subspace when state variables contain variables preceding each action in topological order.
  • Confounding: Confounding can cause standard behavioral cloning to fail when relevant covariates are unobserved, unavailable at decision time, or incorrectly included.The manipulated diagram can nevertheless show that later actions ignore earlier mistakes caused by missing confounders.
  • Causal Inverse RL: Causal IRL uses a sequential-backdoor criterion and reward knowledge to construct policies that can dominate a suboptimal expert.The proposed formulation addresses unobserved confounders and can outperform the expert when prior reward knowledge is informative.

9. Conclusions

The conclusions frame CRL as a synthesis of reinforcement learning with structural causal models and counterfactual reasoning. This framework targets more robust learning across offline, online, and generalized interaction regimes.

  • Motivation: Most reinforcement-learning systems do not explicitly represent causal models, despite the importance of causal mechanisms for effective decision-making.The paper identifies a lack of systematic integration between causal knowledge and RL.
  • Framework: CRL combines RL agents with Pearl’s structural causal models to encode causal knowledge and perform counterfactual reasoning.The resulting framework addresses robust decision-making under uncertainty.
  • Implications: The framework relaxes assumptions for offline learning, develops resilience to unobserved confounding, and provides more efficient online algorithms with near-optimal regret.These developments include off-policy and imitation learning and use causal conclusions from biased offline data.
  • Interaction Regimes: Generalizing interaction regimes opens learning possibilities beyond passive supervised learning and active reinforcement learning.The distinction concerns whether agents only observe data or modify policies through environmental responses.

Appendix A. Comparison with Partially Observed MDPs

The appendix compares causal environments with standard POMDP representations when states are latent. It shows that observational and interventional behavior can correspond to different underlying dynamics and are generally not mutually identifiable.

  • POMDP Structure: A standard POMDP contains latent states, actions, observations, transitions, rewards, and an observation function mapping actions and resulting states to observations.Policies map observation-action histories to distributions over actions and induce joint trajectories through the decision horizon.
  • POMDP Structure: Latent states make the perceived observations non-Markov because histories can remain causally connected to future observations, actions, and rewards.The open path O1 ← S1 → S2 → O2 illustrates this long-sequence dependency.
  • Causal Hierarchy: Observational and interventional distributions from the same SCM can be represented by standard POMDPs with different detailed system dynamics.The appendix illustrates this distinction using separate finite-state automata.
  • Identifiability: Interventional quantities such as transition and reward dynamics are generally not recoverable from observational data in POMDP environments.An alternative SCM can match observational distributions while differing significantly in interventional distributions.
  • Identifiability: Conversely, observational quantities are generally not recoverable from randomized experiments in non-Markov processes.The appendix presents an alternative SCM with the same interventional distribution but a different observational one.

Appendix E. Appendix x.

The appendix notes that model-free and model-based reinforcement learning are orthogonal to causal-model-based approaches.

  • Model-free versus model-based RL is orthogonal to causal-model-based reinforcement learning.

Appendix G. Counterfactual Distributions and Axioms

Counterfactual distributions formalize joint events across alternative causal worlds within structural causal models. Their evaluation replaces mechanisms, computes unit-level potential responses, and aggregates compatible exogenous assignments.

  • An SCM induces joint distributions over counterfactual events involving variables evaluated under different interventions.
  • Different subscripts encode different counterfactual worlds, so the distribution cannot generally be represented using only observational or single-world interventional probabilities.
  • Counterfactual evaluation constructs submodels by replacing each relevant mechanism with its intervention-specific constant.
  • For each unit, the modified mechanisms are evaluated in a valid order to obtain potential responses, then compatible units’ probability masses are accumulated.
  • In the treatment example, the cross-world numerator is zero, implying treatment would not have prevented deaths among untreated non-survivors.
  • The composition, effectiveness, and reversibility properties hold for counterfactual variables in all structural causal models.

Appendix H. Proofs for Chapter 4

These proofs establish identification of policy effects and expected rewards from observational distributions under causal conditions. They connect inverse propensity weighting, dynamic programming, and do-calculus-based policy evaluation.

  • Under NUC, dynamic programming computes the expected reward of any policy from the joint observational distribution.
  • The IPW derivation uses NUC assumptions about endogenous and exogenous parents to identify interventional distributions from observational data.
  • The DP identification formula follows by recursively marginalizing states and actions through value functions.
  • Replacing action mechanisms with policy decision rules and applying backdoor adjustment yields the same policy-reward identification formula as IPW and DP.
  • Under backdoor admissibility, every policy’s expected reward is identifiable from the causal diagram and computable from observational data using IPW or DP.

Appendix I. Proofs for Chapter 5

The proofs derive regret bounds for UCB-style algorithms in causal decision models. They use performance-gap decompositions and causal bounds on action and state distributions to control exploration and suboptimal-policy selection.

  • Theorem 9 bounds UCB+ regret in a multi-armed bandit SCM with bounded rewards and K possible actions.
  • The regret proof decomposes cumulative regret into three terms and bounds them separately using concentration, policy gaps, and optimistic planning.
  • Theorem 10 bounds UCB regret for causal decision models over policy spaces, including a gap-based form for deterministic policies.
  • Lemma 4 upper-bounds counterfactual state probabilities by an observationally defined function Γ.
  • Theorem 12 provides analogous causal bounds for conditional expected rewards at the final decision stage.
  • Theorem 13 combines these ingredients to bound UCB+ regret for causal decision models with bounded rewards.

Appendix J. Proofs for Chapter 8

These results characterize causal conditions for behavioral cloning and policy evaluation, and formulate robust policy-learning programs as optimization problems. Under suitable admissibility conditions, imitation can preserve expert performance or make policy effects identifiable.

  • Under NUC, behavioral cloning perfectly imitates the expert’s performance.
  • Theorem 21 also guarantees a behavior-cloning policy whose expected reward matches expert performance.
  • If a subspace is imitation-admissible, a behavior-cloning policy is obtained by setting each decision rule to the corresponding observational conditional distribution.
  • Repeated application of the cloning identity gives equality between the cloned policy’s interventional outcome distribution and the expert’s distribution.
  • For linear reward hypotheses, the canonical program reduces to a minimax problem, while regularized hypotheses reduce to a conjugate-function optimization problem.
  • For a minimal imitation-admissible subspace, every policy’s interventional distribution is computable from the observational distribution.
Loading 2606.24160v1…