An Introduction to Causal Reinforcement Learning

1. Introduction

• Robust decisions across domains depend on understanding the environment’s often unobserved causal mechanisms.The introduction motivates this need with examples involving robots, physicians, and economists.Source: §1 · p. 2
• The introduction identifies brittleness, sample inefficiency, and poor robustness and explainability as AI limitations, alongside RL’s curse of dimensionality.It argues that an explicit language for causal mechanisms and exogenous variation is lacking, while real-world modularity can support decomposition of RL tasks.Source: §1 · pp. 2–3
• CRL represents the environment with a fully specified structural causal model and the agent’s knowledge with a parsimonious causal model, such as a diagram.These perspectives are paired through a compatibility relation and can be articulated using templates such as MABs, MDPs, and POMDPs.Source: §1 · p. 4
• The Pearl Causal Hierarchy distinguishes associational, interventional, and counterfactual queries, classifying what an agent can represent and potentially answer.Observation supports factual queries, interventions support hypothetical action-effect queries relevant to online and offline learning, and counterfactuals ask what would have happened given an observed outcome.Source: §1 · pp. 4–5

1.1 Roadmap of the Paper

• The roadmap formalizes policy learning with structural causal models, defines CRL tasks by agent interaction and available knowledge, and relates classic RL modalities to causal inference.It covers off-policy learning, online learning, and causal identification, including conditions and algorithms for policy learning.Source: §1.1 · pp. 5–6
• The roadmap extends causal offline-to-online learning to bandit and sequential settings, then studies mixed policies that choose whether and where to intervene.The transfer approach uses offline data to pre-train policy representations before online fine-tuning; mixed-policy work includes atomic and soft interventions.Source: §1.1 · p. 6
• Later sections develop counterfactual policies for biased or adversarial settings and imitation learning from observational data, including behavioral cloning and inverse reinforcement learning.They also address counterfactual randomization and the optimality–autonomy trade-off, and extend imitation methods using causal diagrams and reward knowledge.Source: §1.1 · pp. 6–7

1.2 Notations

• The notation section defines conventions for variables, distributions, indicators, DAGs, induced subgraphs, and graph operations.It also specifies abbreviations for ancestor, descendant, parent, and child sets.Source: §1.2 · p. 7

2. Foundations of Causal Inference

• The foundations section introduces causal semantics and SCMs, then distinguishes observational and interventional distributions and introduces causal diagrams for structural assumptions.These elements are presented as building blocks for causal RL analysis.Source: §2 · pp. 7–8

2.1 Structural Causal Models and the Environment

• SCMs model environments through exogenous factors, endogenous variables, causal mechanisms, and an exogenous-variable distribution, and can encode standard RL settings.Exogenous variables allow unobserved confounding, while each mechanism generates an endogenous variable from other endogenous and exogenous inputs; the agent’s perspective determines what is observed.Source: §2.1 · pp. 9, 11

2.2 Learning through Observational & Interventional Regimes (PCH’s Layers 1 and 2)

• The CRL agent’s endogenous–exogenous partition is agent-specific, and it can interact by passively seeing events or actively intervening through actuators.Under passive observation, actions are controlled by other behavior agents or natural processes rather than by the CRL agent.Source: §2.2 · p. 12, §2.2.1 · p. 12

2.2.1 OBSERVATIONAL DISTRIBUTIONS

• Passive treatment data favor X = 1, although the full SCM identifies X = 0 as preferable, so the observational comparison is not generally causally reliable.The behavior agent’s allocation may be inefficient or reflect different goals, which can make observed treatment recovery rates misleading.Source: §2.2.1 · p. 13
• Passively observing a template-like RL environment yields data from its observational distributions, including state-transition and expected-reward distributions.The inventory example represents these quantities with a probabilistic finite-state machine.Source: §2.2.1 · p. 15

2.2.2 INTERVENTIONAL DISTRIBUTIONS

• Policy interventions replace action mechanisms with rules based on observed states or history, producing submodels whose potential responses define interventional distributions; atomic interventions instead fix actions to constants.Other equations remain unchanged in the submodel.Source: §2.2.2 · pp. 15–18
• In the MAB example, intervening to assign treatment X ← 0 gives patients a higher chance of recovery than X ← 1, making X ← 0 optimal.Source: §2.2.2 · p. 18
• When α > 0, cloning the physician’s random policy yields lower recovery because observational and interventional distributions differ when the physician and agent have different perceptual capabilities.The section warns that the empirical gap between these distributions can be significant.Source: §2.2.2 · p. 19
• A standard MDP’s transition and reward functions abstract only one layer of the PCH, whereas an SCM specifies the environment completely enough to support inference across layers.The section notes that standard MDP definitions based on a transition-probability and reward-function pair describe either observational or interventional distributions, not both.Source: §2.2.2 · p. 20
• In the inventory MDP, conditional next-state transitions and rewards given the current state and action are invariant between policy and atomic interventions.The transition and reward mechanisms remain the same under both intervention types even though the interventional and observational distributions do not coincide.Source: §2.2.2 · pp. 20–21
• The CRL policy’s cumulative profit is 8.2, outperforming the behavior policy’s 1 in this inventory-management example.The authors conclude that replacing the manager with the CRL agent is more profitable in this case.Source: §2.2.2 · p. 21

2.3 Encoding Structural Assumptions through Causal Diagrams

• A causal diagram encodes possible causal inputs and shared exogenous variation, abstracting away the SCM’s specific mechanisms and exogenous distribution.A directed edge permits a causal input but does not require it; missing edges encode that an input or correlation is absent.Source: §2.3 · p. 22
• The same MAB causal diagram can fit different reward mechanisms that imply opposite optimal arms, illustrating the limits of a template’s structural detail.In the alternative mechanism, the coefficient of X is flipped to Δ, making X ← 1 preferable instead of the original X ← 0.Source: §2.3 · p. 23
• If X and Y are d-separated by Z in a causal diagram, they are conditionally independent given Z in every compatible distribution; without d-separation, dependence holds in at least one.The converse is existential: it does not establish dependence in every compatible distribution.Source: §2.3 · p. 24
• Agents can test hypothesized diagrams against distributional constraints or infer compatible diagrams from data, while combining observational and experimental distributions can further restrict the possibilities.Causal discovery commonly uses simplicity assumptions to avoid preferring saturated models.Source: §2.3 · pp. 24–25

3. Elements of Causal Reinforcement Learning

• The framework represents canonical tasks including multi-armed bandits, MDPs, and dynamic treatment regimes, while RL in SCMs addresses environments whose detailed parametrization is not fully known.Learning tasks are distinguished by the interaction regime, structural assumptions, policy space, and reward being optimized.Source: §3 · pp. 25–26
• Multiple MDPs can be compatible with observed data yet imply different optimal decisions, so observed data typically do not determine an optimal policy.Source: §3 · p. 26

3.1 Causal Decision Models

• Policy spaces define available actions and observed states at each intervention; in the DTR example, stationary policies are nested within Markov policies, which are nested within full policies.The full policy can use the history of states and actions, whereas the Markov policy uses only the current state.Source: §3.1 · pp. 26–28
• A CDM combines an environment SCM, a policy space specifying what the agent can control and observe, and a reward function measuring performance.The policy space and reward function define the decision task alongside the environment model.Source: §3 · p. 25, §3.1 · p. 29
• 1 versus 0.5: in the DTR example, the optimal full policy’s expected reward is 1, exceeding the best possible Markov policy’s 0.5.The example demonstrates that restricting the policy space can change the best achievable reward.Source: §3.1 · p. 30
• In the MAB example, because the coefficient ∆ > 0, reward is maximized when π(X = 1) = 0, yielding the deterministic optimal policy π∗ that chooses arm x∗ = 0.Source: §3.1 · p. 32
• For stationary MDP policies, invariant structural functions yield a Bellman recursion, and value iteration computes an optimal policy with expected return 8.2.The example uses a discount factor γ = 0.9.Source: §3.1 · pp. 32–33
• For the 12-month treatment problem, the optimal policy starts with behavioral therapy and follows with intensive behavioral therapy plus medication, yielding expected reward 0.6758.The reward is the patient’s days of abstinence.Source: §3.1 · p. 35

3.2 Causal Reinforcement Learning Tasks

• A CRL task is ⟨L, A, Π, R⟩: the agent uses its interaction regime and structural assumptions to estimate an optimal policy for an environment it does not fully know.The agent has access to the policy space and reward function, but not the complete environment model.Source: §3.2 · pp. 35–36
• Passive observation corresponds to off-policy learning, while active intervention corresponds to online learning; the CRL agent estimates a policy from collected data and structural assumptions.Across episodes, it observes another agent’s actions under see or intervenes on actions under do.Source: §3.2 · pp. 35, 37
• With no unmeasured confounder, an off-policy bandit agent can estimate each arm’s expected reward from observational data and select the arm with the highest estimate.The NUC assumption rules out an unobserved confounder that affects both action and reward.Source: §3.2 · p. 38
• In online bandit learning, the agent intervenes on each arm and estimates its expected reward from the resulting experimental outcomes.The example estimates expected reward using empirical means.Source: §3.2 · p. 38
• When the no-unmeasured-confounder assumption fails but a covariate and backdoor causal diagram are available, the agent can use backdoor adjustment for causal identification.The agent observes P(X, Y, Z) and uses the diagram as structural knowledge.Source: §3.2 · p. 38
• With sufficient observational data, the expected reward of every arm is computable from P(X, Y, Z), so optimizing over the action domain yields an optimal arm.The passage assumes sufficient observational data and recovery of the joint distribution P(X, Y).Source: §3.2 · p. 38
• CRL task signatures formalize popular settings and enable novel tasks; traditional online, off-policy, and causal-identification tasks optimize within the experimental policy space ΠEXP.The framework also makes critical conditions explicit, such as whether an off-policy method can recover a policy consistent with the SCM’s optimum.Source: §3.2 · pp. 38–39

3.3 Comparison with Markov Decision Processes

• Although both observational and interventional trajectory distributions can satisfy the Markov property, their MDPs can yield different optimal policies, with the observationally learned policy suboptimal in the underlying environment.Thus, the Markov property alone does not determine which policy is optimal in the causal model.Source: §3.3 · pp. 40–41
• Neither observational nor interventional data alone determines the other distribution, so passive observations may not reveal another agent’s policy or performance, and intervention data may not predict others’ behavior.Because an unknown SCM is rarely directly observable and inferences across causal hierarchy levels are unavailable, CRL agents need causal knowledge or assumptions.Source: §3.3 · pp. 43–44

4. Reinforcement Learning through Causal Lenses

• Causal formalization reveals subtle links between reinforcement learning and causal invariances, including when off-policy algorithms can be justified and applied to observational data with unobserved confounders.The section uses these classic tasks to connect causal inference and reinforcement learning.Source: §4 · p. 44

4.1 Off-Policy Learning

• Off-policy learning estimates an optimal policy from observational data generated by a different behavior policy, provided the no-unmeasured-confounder (NUC) condition holds.NUC requires action parents to be observed in past states and actions, and action noise to be independent of other system noises.Source: §4.1 · pp. 44–45

4.1.1 OFF-POLICY EVALUATION

• Under NUC, inverse propensity weighting (IPW) and dynamic programming (DP) compute policy rewards from observational data; they are equivalent, and both yield 0.6757 in the example.That estimate matches the expected reward evaluated directly in the structural causal model.Source: §4.1.1 · pp. 46, 48, 49
• When NUC fails, IPW and DP can both estimate 0 when the actual reward is 0.4, showing their validity depends on an assumption that may be fragile in practice.The passage gives electronic health records as an example where treatment assignment can be confounded by access to high-quality healthcare.Source: §4.1.1 · pp. 50–51

4.2 Online Learning

• Online learning evaluates candidate policies by deploying them through interventions in the environment and makes no structural assumptions beyond temporal ordering, so NUC need not hold.For each episode, the agent selects a policy, intervenes on actions accordingly, and receives observations under that policy.Source: §4.2 · p. 51
• Under policy interventions, the NUC condition holds, including in an example where intervention removes the unobserved confounder’s effects on treatments.The lemma formalizes NUC for any policy in the stated policy space.Source: §4.2 · pp. 51–52

4.2.1 RANDOMIZED CONTROLLED TRIALS

• RCT uniformly randomizes every action during N exploration episodes, uses the resulting NUC data to estimate candidate-policy rewards, then commits to the highest-estimated policy.It performs repeated policy interventions and receives observations from the corresponding interventional distributions.Source: §4.2.1 · p. 53
• RCT achieves sublinear regret when N is tuned, but too much exploration raises exploration regret while too little increases exploitation regret.The suggested choice of N depends on the suboptimal gap ∆ and total episodes T, which may not be known in advance.Source: §4.2.1 · pp. 56–57
• N = 500 performed best among tested RCT strategies, close to the analytical choice N ≈ 878 across 1,000 simulations.The experiment used T = 5,000 episodes and a suboptimal gap ∆ = 0.1.Source: Figure 15 · p. 57, §4.2.1 · p. 57

4.2.2 THE UPPER CONFIDENCE BOUND ALGORITHM

• UCB adaptively selects the arm with the highest confidence bound, favoring high estimated reward or insufficiently explored arms, without prior knowledge of ∆ or T.It is stated to match the theoretical guarantees of RCT tuned using prior knowledge of ∆ and T.Source: §4.2.2 · pp. 57–58
• UCB achieves sublinear regret close to RCT’s up to logarithmic terms, without requiring prior knowledge of suboptimal gaps or the total episode count while retaining the same asymptotic guarantees.The passage notes that similar regret bounds do not guarantee similar deployed performance.Source: §4.2.2 · p. 59
• In Experiment 2, UCB generally outperformed RCT when RCT’s trial count had to be chosen without prior knowledge, while optimally tuned RCT slightly lowered cumulative regret.The comparison used T = 10000 and α = 0.1.Source: §4.2.2 · p. 59, Figure 16 · p. 60

4.3 Causal Identification

• Causal identification seeks an optimal policy using observational data and a causal diagram, without assuming NUC, by checking whether the target interventional query is identifiable under the diagram’s structural assumptions.The agent collects passive observations and combines their distribution with the diagram and domain knowledge.Source: §4.3 · pp. 60–61
• Under NUC, expected rewards are identifiable from the observational distribution and computable using IPW or DP, regardless of the underlying SCM’s specific mechanisms and exogenous conditions.The section illustrates this guarantee for policies in a two-stage decision process.Source: §4.3 · p. 62

4.3.1 SEQUENTIAL BACKDOOR FOR POLICY EVALUATION

• Sequential backdoor requires observed action and state history to block relevant backdoor paths; it is broader than NUC, since some policy spaces satisfy it despite unobserved confounding that violates NUC.Conversely, NUC implies sequential backdoor admissibility.Source: §4.3.1 · pp. 63–65
• For any policy in a backdoor-admissible policy space, its expected reward is identifiable from the causal diagram and computable from observational data using IPW or DP.This generalizes standard off-policy evaluation to settings with unobserved confounders affecting actions and other variables.Source: §4.3.1 · pp. 65–66
• The sequential backdoor criterion (SBC) is a graphical condition sufficient to determine whether IPW and DP can evaluate candidate policies from observational data.Unlike the NUC assumption, it is based on the causal diagram and may be more readily evaluated using data and domain expertise.Source: §4.3.1 · p. 68
• The model families satisfy the strict containment M(3) ⊂ M(2) ⊂ M(1) ⊂ M(0), so some policy effects are identifiable even when SBC does not hold.M(0) contains environments where policy effects are identifiable; the text notes that structural constraints in the causal diagram can still recover effects outside SBC.Source: §4.3.1 · pp. 68–69

4.3.2 DO-CALCULUS LEARNING

• In the front-door example, the expected reward of each action is identifiable from P(X, Y, W), allowing the learner to choose the optimal treatment despite an unobserved confounder.The passage identifies the front-door adjustment as the means of computing the reward from the observational distribution (Pearl, 2000, Thm. 3.3.4).Source: §4.3.2 · p. 69
• Policy-effect identification reduces to atomic interventions: Pπ(Y) is identifiable if and only if the corresponding Px∗(Y, S∗) is identifiable.Here X∗ contains actions ancestral to Y in the post-interventional graph, and S∗ contains its other ancestors apart from Y and X.Source: §4.3.2 · p. 70
• Do-calculus plus probability rules is sound and complete for interventional identifiability, and algorithms determine policy-reward identifiability in polynomial time.If repeated applications of the rules and probability algebra cannot express an effect using available distributions, the effect is non-identifiable.Source: §4.3.2 · pp. 71, 73

4.4 Novel Causal Reinforcement Learning Tasks

• Causal offline-to-online learning (COOL) asks how to pre-train an online agent with imperfect policy-effect knowledge from confounded observational data while using causal-model invariances to reduce interventions.The passage frames this as a proposed learning task, not a reported solution.Source: §4.4 · p. 74
• CRL 2 asks where an agent should intervene and what to consider, choosing among candidate policies that include allowing the system to evolve naturally.For complex systems, it also considers qualitatively different actions in ΠMIX, such as drug combinations with exponentially many possible interactions.Source: §4.4 · p. 75
• CRL 3 asks whether an alternative action would have been better, using counterfactual randomization to learn over counterfactual policies ΠCTF while accounting for the agent’s natural decision-making process.The proposed Ctf-Do regime addresses settings such as adversarial environments that exploit an agent’s natural inclinations to systematically reduce its reward.Source: §4.4 · p. 76
• CRL 4 studies when imitation works, learning an effective policy from observational data and a causal diagram despite an unspecified reward and generally present unobserved confounding.This addresses demonstrations such as human driving trajectories, where defining a universal performance reward is non-trivial.Source: §4.4 · p. 76
• Together, these tasks broaden counterfactual learning and underscore the potential of studying causal inference and reinforcement learning side by side as causal reinforcement learning.Source: §4.4 · p. 76

5. Causal Offline-to-Online Learning (CRL Task 1)

• When the no-unobserved-confounding (NUC) assumption holds, UCB can pool observational and experimental data to warm-start online learning, but under confounding direct transfer can bias reward estimates and slow or hinder convergence.The reported experiment finds that UCB- favored a suboptimal arm and remained biased through T = 10,000, while standard UCB identified the optimal arm x* = 0 after a few interventions.Source: §5 · pp. 78–80
• The section’s causal offline-to-online learning strategy uses observational data to accelerate online reinforcement learning even when NUC fails and candidate-policy rewards cannot be computed from those data.Direct off-policy evaluation may introduce significant reward-estimation bias; this differs from the NUC case, where pre-trained estimates can be transferred directly.Source: §5 · pp. 78, 80

5.1 Confounding Robust Offline-to-Online Learning

• In confounded MABs, interventional rewards are not identifiable from observational data alone: distinct environments can share P(X, Y) yet assign different expected rewards to an arm.Consequently, optimizing observed conditional rewards can favor a suboptimal arm even when another arm maximizes interventional reward.Source: §5 · p. 80, §5.1 · pp. 81–82
• Natural causal bounds place each arm’s expected reward in an observationally estimable interval, informative when each arm has positive probability and unimprovable in MABs without added assumptions or data.These bounds can supply partial reward information for online learning even when the exact interventional reward is not identifiable.Source: §5.1 · p. 82
• When a suboptimal arm’s causal upper bound is below the optimal reward, UCB+ can achieve constant regret; otherwise its bound matches or improves on standard UCB without negative transfer.For two arms, an informative bound gives O(1) regret instead of UCB’s O(log(T)/∆); an uninformative bound preserves UCB’s guarantee.Source: §5.1 · p. 84
• UCB+ converges almost immediately when the suboptimal arm’s causal bound is informative, matches UCB when it is not, and outperforms direct transfer of confounded observations.The simulation uses 5,000 observational samples; the informative case has ∆=0.3 and r1=0.3<µ0, while at ∆=0.1, r1=0.5>µ0 and UCB+ and UCB virtually coincide.Source: Figure 25 · p. 85, §5.1 · p. 85

5.2 Online Learning in Sequential Decision-Making

• SMART can recover optimal policies with enough trials, but its explore-then-commit phase incurs linear regret and choosing the trial count requires prior knowledge.In the two-stage DTR simulations, UCB achieves sublinear regret while randomized RCT performs worst because of its linear exploration regret.Source: §5.2 · pp. 86, 90, Figure 26 · p. 90
• Under the NUC condition, sequential transition and reward quantities are invariant across policies and can be consistently estimated by pooling interventional data.Conditioning on past states blocks backdoor paths, allowing the needed quantities to be estimated from the corresponding conditional distributions.Source: §5.2 · p. 86
• For sequential decisions, UCB constructs plausible SCMs from interventional data, plans optimistically, and achieves sublinear regret, converging to an optimal policy as T grows.Its regret bound depends on horizon, state-action domain size, episodes, and the reward gap; in an MAB it matches O(K log(T)/∆).Source: §5.2 · pp. 87, 89

5.3 Learning from Observational Data

• When NUC holds, observational conditionals can warm-start UCB; without NUC or added causal assumptions, transition and reward quantities may be non-identifiable, motivating partial bounds from observational data.The proposed result bounds interventional transition probabilities using the observational distribution.Source: §5.3 · p. 90
• With observational probabilities positive for every state-action realization, Theorem 11’s causal bounds are generally informative—strictly inside [0, 1]—and generalize Theorem 8’s natural bounds to sequential settings with multiple actions H > 1.Source: §5.3 · p. 91
• Because these bounds depend on the identifiable observational distribution, they can generally be estimated consistently, with concentration inequalities controlling finite-sample uncertainty.The paper proposes corresponding sample-mean estimates for the transition and reward bounds.Source: §5.3 · p. 92
• UCB+ builds plausible SCMs compatible with the causal bounds and selects an optimistic model from their intersection with the confidence set to choose each episode’s policy.The true SCM is included with high probability when the bounds are sound; optimistic planning incorporates both the usual confidence constraints and constraints from the causal bounds.Source: §5.3 · pp. 93–94
• In both single-stage and sequential settings, UCB+ uses observational causal bounds and has a performance gap ∆* at least ∆, yielding regret bounds no worse than UCB’s.The bounds also partially identify candidate-policy rewards and characterize when confounded observational data accelerate online learning.Source: §5.3 · pp. 94–95
• UCB+ matches UCB’s asymptotic regret bound and can outperform it when a suboptimal policy’s causal upper bound is below the optimal policy’s expected reward.For a multi-armed bandit, its regret bound is O(K log(T)/∆x), where ∆x is the smallest gap among suboptimal arms whose causal upper bound is at least the optimal expected reward.Source: §5.3 · p. 94
• With (α1, α2) = (3, −3), r = 0.6283 < µπ∗ = 0.6757 and UCB+ significantly outperformed UCB; with (−12, −3), r = 0.9875 > µπ∗ and their performance coincided.These simulations support the claim that UCB+ has no negative impact when causal bounds do not rule out the suboptimal policy.Source: §5.3 · p. 94, Figure 27 · p. 95

6. Mixed Policy Learning: Where to Intervene (CRL Task 2)

• Mixed policy learning lets agents choose intervention scopes, including partial or no intervention, avoiding potentially harmful side effects while other variables follow their natural mechanisms.In non-Markovian systems, partial intervention can outperform full control; the mixed policy space also allows action and context subsets to vary.Source: §6 · p. 96

6.1 Mixed Policy with No Context

• In this example, intervening with do(X1 ← 1) makes Y = 1 in every subsequent round; the all-at-once agent does not converge, whereas brute-force search finds the optimal policy.Source: §6.1 · p. 99

6.1.1 STRUCTURAL PROPERTIES IN MIXED POLICY LEARNING IN A MAB SETTING

• Do-calculus identifies reward-equivalent actions, and a set is a minimal intervention set exactly when it is contained in the ancestors of Y in the graph after intervention.Possibly-optimal minimal intervention sets are those that can be optimal for some SCM, while partial orders can rule out dominated sets from consideration.Source: §6.1.1 · pp. 101–103
• MUCTs capture variables linked to the reward through unobserved confounding, while the interventional border identifies their parents outside the territory.MUCTs can be constructed by iteratively adding variables connected by bidirected edges and their descendants; intervening within a MUCT can disrupt mechanisms that yield the best result.Source: §6.1.1 · pp. 103–104
• Theorem 14 characterizes POMISes completely: an MIS X′ is a POMIS if and only if its interventional border IB(H_X′, Y) equals X′.This gives an interventional-border approach to establishing the complete collection of POMISes for a given graph, intervenable set, and reward.Source: §6.1.1 · p. 104
• Do-calculus lets agents combine compatible probability quantities from POMIS-arm data to estimate other arms’ expected rewards and improve confidence without accumulating their regret.The paper describes replacing factorization terms with quantities obtainable from other arms, then using the resulting estimator to exploit multiple data sources.Source: §6.1.1 · pp. 105–106
• At 10,000 episodes, cumulative regret was 655.96 ± 69.68 (Brute-force), 467.19 ± 66.85 (MIS), 358.86 ± 63.11 (POMIS), and 280.80 ± 60.83 (POMIS+ID).The experiment compares UCB agents using different arm-selection strategies; POMIS+ID actively infers arm rewards from other arms when possible.Source: Figure 34 · p. 106, §6.1.1 · p. 106

6.2 Mixed Policy with Context

• In the example, the best contextual-bandit policy has expected reward 0.75, while intervening only on X1 using context C yields 1.0.Setting X1 equal to C removes the noise affecting X1, making X2 equivalent to U2 under the described policy.Source: §6.2 · p. 108
• Among 15 policy spaces, only Πd and Πe satisfy both minimality and optimality; the example also identifies the contextual-bandit space as redundant.ΠCB subsumes Πe yet has equivalent optimal reward, so the contextual-bandit policy uses a redundant intervention and context.Source: §6.2 · p. 108

6.2.1 CONTEXTUAL MINIMALITY IN OPTIMAL MIXED POLICIES

• Under optimality, contexts can be fixed to desirable values relative to remaining contexts so decision rules can drop them without sacrificing reward.The paper also allows fixed intervened variables determined by their contexts, enabling smaller policy spaces.Source: §6.2.1 · pp. 109–110
• The proposed sufficient test for non-minimality first obtains an intermediate expected-reward expression for variables to fix, then checks whether fixing yields a simpler policy using retained contexts.The expression is constructed under conditions on selected exogenous and endogenous variables and their ordering; the target is a reward expression that no longer depends on the variables being fixed.Source: §6.2.1 · p. 111
• Under optimality, contexts C2 and C3 are ineffective for X1 and X2, so the policy space is not minimal.The example derives a simpler policy by fixing contexts and replacing decision rules that depend on them.Source: §6.2.1 · pp. 111–112
• 49.60, 195.03, 1134.84, and 3198.42 were the average cumulative regrets for PO-PS, MuO, BF, and naive CB, respectively; PO-PS converged fastest.MuO outperformed brute-force, while naive CB did not converge because intervening on X2 led to a suboptimal policy under unobserved confounding.Source: Figure 39 · p. 113, §6.2.1 · p. 113
• A dependency graph marks variables that can be fixed; if all redundant contexts are markable and intervened-variable ancestors meet the context constraint, a simpler policy can be obtained.The graph is built from distributions over Z and U′, with edges reflecting distribution conditions; Theorem 2 gives more detailed results.Source: §6.2.1 · pp. 112–113
• Without contexts, complete characterizations of minimality and possible-optimality are provided; with contexts, a procedure detects non-minimality, but complete characterization remains open.These characterizations do not require samples from interaction with the environment; naive intervention that ignores natural mechanisms can incur regret indefinitely.Source: §6.2.1 · pp. 113–114

7. Counterfactual Decision-Making (CRL Task 3)

• Standard bandit strategies do no better than random guessing in the Greedy Casino, while regret continues to grow.The reported experiments include ϵ-greedy, Thompson Sampling, UCB1, and EXP3.Source: Figure 42 · p. 118
• Both arms yield an observational payout of 15%, whereas randomized intervention yields 30%, meeting the legal minimum but masking the casino’s lower ordinary payouts.The two standard distributions alone leave the arms statistically indistinguishable, so algorithms based on them cannot learn an optimal policy.Source: §7 · pp. 117–118
• The section proposes counterfactual machinery to exploit layer 3 distributions and improve decisions, including a treatment-on-the-treated criterion, extensions to MDPs, and counterfactual randomization.It also outlines an autonomy–optimality planning trade-off for learning an optimal counterfactual MDP policy under a budget on human input.Source: §7 · pp. 118–119

7.1 Counterfactual Decision Criterion

• In bandits with unobserved confounding, observational rewards reflect natural predilections, while interventional rewards reflect choices imposed by deliberate policies.The section focuses on settings where unobserved confounders cannot be ruled out a priori.Source: §7.1 · pp. 119–120
• Counterfactual decision-making compares outcomes under an alternative action with outcomes under the action the agent was about to take, retaining that decision context.For example, the quantity E[Y_{X←0} | X = 1] asks whether the agent would win had it played 0 when its natural choice was 1.Source: §7.1 · pp. 120–121
• An SCM defines counterfactual distributions by evaluating modified causal mechanisms across distinct hypothetical worlds and aggregating over compatible exogenous states.This formalization allows the section to express comparisons that cannot be written using observational or interventional distributions alone.Source: §7.1 · pp. 120, 122
• The counterfactual decision criterion compares average payouts from acting with or against natural predilections, which can reveal information about payout confounders even when unknown to the agent.When the agent’s natural choice is X = 1, the criterion recommends X = 0 if its conditional counterfactual expected payout is higher, and otherwise recommends following intuition.Source: §7.1 · pp. 122–123
• A counterfactual policy maps an intended action to a realized action; its optimal performance is never worse than an optimal experimental policy, and coincides when NUC holds.The policy treats intended and realized actions as actions in observational and interventional worlds, respectively.Source: §7.1 · pp. 123, 125, 126
• An example counterfactual policy yields expected reward 0.45, exceeding the 0.15 expected reward of every atomic intervention arm.The preceding analysis also concludes that payout would have tripled by choosing against natural predilections in the specified situations.Source: §7.1 · pp. 122, 125

7.1.1 MARKOV DECISION PROCESS WITH UNOBSERVED CONFOUNDERS

• For sequential decisions, counterfactual policies extend to MDPs with unobserved confounders by choosing realized actions from both the current state and intended action.In these environments, observational and interventional transition probabilities and conditional rewards need not coincide because of unobserved confounding.Source: §7.1.1 · p. 127
• In the inventory example, counterfactual policies form a semi-autonomous system that recalibrates the manager’s intended action using the observed state rather than discarding that intention.This contrasts with the autonomous experimental policy, which bases decisions on current inventory and does not account for the manager’s intended decision.Source: §7.1.1 · pp. 128–129
• Conditional ETT transition and reward quantities remain invariant across stages, allowing counterfactual policies to be represented and optimized as a standard MDP with dynamic programming.The reduction requires detailed parametrization of the underlying MDP.Source: §7.1.1 · pp. 131–132
• Counterfactual policies can perform at least as well as experimental policies because the counterfactual policy space includes policies that reproduce experimental interventions.In the example, the optimal counterfactual policy always acts against its intended action and yields expected return 9, outperforming the best experimental policy.Source: §7.1.1 · p. 134
• Under the NUC condition, counterfactual transition and reward quantities coincide with experimental counterparts, so the intended action provides no value of information.An optimal counterfactual policy can then be simulated using an experimental policy.Source: §7.1.1 · p. 134

7.2 Counterfactual Randomization

• When intended and executed actions differ, the x-specific effect is not identifiable from passive observations and controlled experiments alone without a causal model or additional assumptions.Counterfactual randomization addresses implementation by intercepting the agent before execution, treating its final choice as its intention, then randomizing the realized action.Source: §7.2 · p. 135
• Ctf-RCT estimates x-specific effects by recording the intended action, choosing realized actions uniformly at random during exploration, then selecting the estimated best realized action for each intention during exploitation.It intervenes on the selected realized action and uses the collected intention-action samples to estimate the causal effect.Source: §7.2 · pp. 135–136
• Ctf-UCB extends counterfactual interventions to adaptive randomization by choosing the realized action with the highest confidence bound for the agent’s intended action.The bounds estimate intention-conditioned ETTs from prior samples.Source: §7.2 · p. 136
• Ctf-UCB uses a separate UCB instance for each intended action, incorporating that action through counterfactual intervention rather than treating it as ordinary context.The intended action exists under observation and is discarded under standard randomized intervention; the augmentation can also empower other bandit algorithms.Source: §7.2 · pp. 137–138
• Theorem 17 bounds Ctf-UCB regret, and its implication is that the algorithm can learn an optimal counterfactual policy that consistently improves the intended action.An optimal counterfactual policy also dominates the best possible interventional policy.Source: §7.2 · p. 137
• In some MAB environments, any online algorithm using Fisherian randomization incurs at least 0.5 regret per episode on average and cannot achieve an optimal counterfactual policy.The lower bound applies to algorithms optimizing over experimental policies, including UCB.Source: §7.2 · pp. 137–138
• Ctf-UCB’s simulations showed a significant regret difference from standard UCB, which experienced linear regret against the optimal counterfactual agent.The simulations compared cumulative regret and probability of selecting optimal arms.Source: Figure 46 · p. 138, §7.2 · p. 139

7.2.1 COUNTERFACTUAL RANDOMIZATION FOR MDPS

• Ctf-UCBVI augments UCBVI with confidence-bounded Q-value estimates based on empirical counterfactual transitions and rewards, then applies counterfactual interventions at each decision step.The analysis assumes known counterfactual reward-function parameters, though the authors say it generalizes to inaccessible reward functions.Source: §7.2.1 · p. 139
• In the evaluated MDP with unobserved confounders and horizon H = 10, standard UCBVI had linear regret, whereas augmented Ctf-UCBVI achieved sublinear regret and obtained an optimal counterfactual policy accounting for intended actions.Source: §7.2.1 · p. 142

7.3 The Tradeoff between Autonomy and Optimality

• Hybrid policies switch by state between experimental decisions that ignore human intent and counterfactual decisions that use it, subject to a budget limiting counterfactual use to a fraction δ of running time.The switch is represented by an extended action Ai ∈ {0, 1}.Source: §7.3 · pp. 143–144
• The section extends an occupation-measure linear program by adding the extended action to represent hybrid policies and optimize their MDP rewards.The supplied passage introduces the extension but does not show the completed constrained program.Source: §7.3 · p. 145
• The proposed hybrid policy imposes a δ budget on counterfactual decisions and can be obtained through polynomial optimization, for which SDP relaxation is one suggested solution method.The optimization includes constraints limiting discounted time spent using the counterfactual criterion and ensuring the experimental criterion does not take the intended action as input.Source: §7.3 · pp. 145–146
• As δ rises from 0.1 to 0.9, hybrid-policy reward moves from limited improvement over exp to performance comparable to ctf, with δ = 0.5 intermediate.The occupancy results show lower-δ policies favoring autonomous decisions and higher-δ policies more actively accounting for the human’s intended action; δ = 0.5 remains neutral.Source: Figure 49 · p. 146, §7.3 · p. 146
• Counterfactual randomization lets an agent use counterfactual reasoning to account for human intuition, which outperforms Fisherian randomization in almost all cases described.The section argues that human intuition should remain in the loop when it provides task information, even after the agent has learned and modeled the environment.Source: §7.3 · p. 147

8. Causal Imitation Learning (CRL Task 4)

• An unknown reward makes a policy’s expected reward unidentifiable from the causal diagram and observational distribution alone.The paper’s general corollary and driving example show that different reward functions can leave observational data unchanged while changing policy evaluation.Source: §8 · p. 148
• The proposed confounding-robust BC and IRL approaches use causal graphical conditions to determine when demonstrations can yield an imitating policy despite unobserved confounders.The BC criterion is described as necessary and sufficient; the IRL condition generalizes methods including GAIL and MWAL to settings where NUC does not hold.Source: §8 · pp. 149–150

8.1 Causal Behavioral Cloning

• Behavioral cloning matches expert performance when NUC holds and each action’s endogenous parents are included in its input states.Under these conditions, the cloning policy uses the expert’s conditional action distributions given those states.Source: §8.1 · p. 151

Evaluating the above equation gives decision rules

• With an unobserved confounder, cloning the action marginal yields 0.5α expected reward versus the expert’s α when α > 0.In the driving example, the missing front-car deceleration affects both the expert’s action and the outcome, so NUC fails and cloning performs worse.Source: Evaluating the above equation gives… · pp. 152–153

8.1.1 BACKDOOR CRITERION FOR IMITATION

• The imitation backdoor condition permits imitation when each action either ceases to be an outcome ancestor under future policies or its input states block all backdoor paths.Future actions can shield the outcome from an earlier action’s errors; in the example, this makes the policy space imitation-admissible despite no adjustment set for the first action.Source: §8.1.1 · pp. 153–154
• An imitation-admissible policy subspace enables behavioral cloning from P(Xi|Zi), and the criterion is necessary for a broad class of policy spaces; such a subspace can be found in polynomial time under full-history state availability.The imitation backdoor condition also provides an algorithm for deciding whether a policy compatible with a policy space is imitable.Source: §8.1.1 · pp. 155–156
• Causal behavioral cloning avoids bias in cases where standard variants fail, including when later actions can correct earlier imitation error; non-causal approaches also fail to detect non-clonability.The experiments compare causal BC with methods using observed parents or all available state variables; their reported outcomes vary across diagrams G1–G4.Source: Table 23 · p. 156, §8.1.1 · pp. 156–157

8.2 Causal Inverse RL

• Under NUC, minimax inverse RL can use observational data and informative reward-class knowledge to outperform a suboptimal expert, but unobserved confounding can make its learned policy fail to match expert performance.In the NUC multi-armed bandit example, IRL obtains reward α versus 0.9α for both the expert and behavioral cloning; without NUC, the illustrated IRL policy obtains 0.5α versus the expert’s α.Source: §8.2 · pp. 157–159

8.2.1 MINIMAL IMITATION BACKDOOR

• A minimal imitation-admissible subspace makes policy outcome distributions identifiable from observational data, whereas non-minimal admissible spaces need not be identifiable; it can be found by iteratively removing inputs in polynomial steps.Minimality requires minimal d-separating input sets, or no inputs when an action is not an ancestor of the reward in the relevant manipulated diagram.Source: §8.2.1 · pp. 160–161

8.2.2 IMITATION VIA INVERSE RL

• The canonical inverse-RL program uses a minimal imitation-admissible subspace and can be solved with established algorithms, including multiplicative weights and generative adversarial imitation learning.Minimality permits computing the imitator’s occupancy distribution from observational data, the policy, and the causal diagram.Source: §8.2.2 · p. 162
• For linear feature rewards, causal IRL reduces to a minimax matrix game; for regularized nonlinear rewards, its penalized canonical program is solvable through a conjugate optimization.The linear formulation uses feature expectations, while the nonlinear formulation penalizes reward complexity with a convex regularizer.Source: §8.2.2 · pp. 162–163
• The conjugate objective connects causal imitation to GANs: a discriminator distinguishes policy from expert occupancy measures, and alternating policy and discriminator optimization seeks a saddle point (Ho and Ermon, 2016).When the discriminator cannot distinguish the occupancy measures, the policy has matched expert performance.Source: §8.2.2 · p. 163
• Causal-IRL consistently outperforms the expert and other imitation strategies using reward knowledge; Causal-BC matches expert performance, while BC and IRL fail to do so.The simulation evaluates imitation when the imitation backdoor condition holds.Source: Figure 53 · p. 164, §8.2.2 · p. 164
• Causal-BC selects covariates under the imitation backdoor criterion, while Causal-IRL searches a minimal admissible policy subspace and uses reward augmentation for known monotonicity in Y1 and Y2.The reward augmentation encourages higher values for features Y1 and Y2.Source: §8.2.2 · p. 164
• Graphical criteria establish when expert performance can be mimicked; sequential backdoor criteria support causal IRL, whose canonical optimization addresses unobserved confounding and can use standard IRL algorithms.Behavioral cloning is applicable when the feasibility condition holds; IRL uses additional quantitative reward knowledge.Source: §8.2.2 · p. 164

• Most RL systems lack explicit causal models, and a systematic, cohesive foundation for integrating causal knowledge into RL remains lacking.Source: §9 · p. 165
• CRL relaxes assumptions, supports offline methods resilient to unobserved confounding, and enables efficient online policy learning with near-optimal regret.The framework combines RL with Pearl’s structural causal models for counterfactual reasoning and robust decision-making under uncertainty.Source: §9 · p. 165
• Generalizing passive observation and active intervention opens “where to intervene” learning, while counterfactual randomization extends Fisherian randomized experience with counterfactual reasoning.Source: §9 · p. 165

Appendix A. Comparison with Partially Observed MDPs

• Latent states make POMDP observations non-Markov: conditioning on the current observation and action does not block paths from earlier history to future observations, actions, and rewards.Standard POMDPs (Sondik (1971)) represent long-sequence dependence through latent states and dynamics; policies map observation-action histories to distributions over actions.Source: Appendix A · p. 181
• Observational data generally do not identify interventional queries in POMDPs: compatible SCMs can match observational distributions yet disagree on interventions.The constructed alternative model preserves the observational distribution while differing in interventional distributions.Source: Appendix A · pp. 182, 184
• Conversely, randomized interventional data generally do not identify observational quantities: compatible SCMs can agree on interventions yet generate different observations.The alternative model matches the interventional distribution but differs in observational distributions.Source: Appendix A · pp. 184–185
• Passive trajectories of other agents generally cannot support claims about the learning agent’s actions and performance, while intervention data cannot predict how other agents will behave in the same environment.The two failures of identification arise because observationally equivalent models can have different intervention distributions, and intervention-equivalent models can have different observational distributions.Source: Appendix A · p. 185

Appendix E. Appendix x.

• The model-free versus model-based distinction is orthogonal to causal-model-based RL.Source: Appendix E · p. 186

Appendix G. Counterfactual Distributions and Axioms

• An SCM induces joint distributions over counterfactual events by weighting compatible events by the probability of their exogenous units.Different subscripts encode different counterfactual worlds.Source: Appendix G · p. 186
• Evaluation constructs intervention-specific submodels, evaluates their mechanisms for each exogenous unit, and sums the probability mass of units consistent with the counterfactual events.For atomic interventions, P(Y_x, …, Z_w) denotes a joint distribution over variables in the respective intervened submodels.Source: Appendix G · p. 186
• In the MAB example, the analysis concludes that treating patients who died untreated would not have prevented their deaths.The joint event that a patient died untreated yet would survive under treatment has probability zero in the stated case.Source: Appendix G · pp. 186–187
• Counterfactual variables satisfy composition, effectiveness, and reversibility in all structural causal models.The section presents these as counterfactual axioms.Source: Appendix G · p. 187

Theorem 2 (Inverse Propensity Weighting, under NUC)

• Under NUC, the expected reward of any policy is computable from the observational distribution using an inverse-propensity-weighted adjustment formula.NUC requires action parents to be prior actions or states and each action’s exogenous parent to be independent noise affecting only that action.Source: Appendix H · pp. 187–189

Theorem 3 (Dynamic Programming)

• Under NUC, dynamic programming computes a policy’s expected reward from the joint observational distribution by repeatedly marginalizing over actions and states.The value function recurses backward from the terminal reward conditional on the action and state history.Source: Appendix H · pp. 189–190

Backdoor-admissible policy spaces

• For backdoor-admissible policy spaces, expected rewards are identifiable from the causal graph and computable from observational data using IPW or dynamic programming.Under intervention by a policy, the Experimental NUC lemma establishes the NUC conditions used in these identification results.Source: Appendix H · pp. 190–191

Appendix I. Proofs for Chapter 5

• Theorem 9 establishes a regret bound for UCB+ in a K-arm bandit model with rewards supported on [0, 1].Its proof bounds arm-selection counts by separating cases according to the causal upper bound on expected reward.Source: Appendix I · pp. 191–192
• Theorem 10 gives a regret bound for UCB in a causal decision-making model and a further expression for the case ϵ = 0.The policy space’s deterministic policies and performance gaps from the optimal reward are used in the stated bound.Source: Appendix I · p. 194
• Lemma 3 supplies an intermediate relation for counterfactual state probabilities in a causal decision-making model, derived by expanding the counterfactual distribution and applying the composition axiom.The proof identifies the counterfactual distribution with an expression over observed action histories.Source: Appendix I · p. 197
• Lemma 4 upper-bounds the counterfactual probability of a state history under an action history by Γ(s̄i+1, x̄i).Γ is defined from the observational distribution in Equation 284.Source: Appendix I · pp. 197–198
• Theorem 11 places the counterfactual conditional probability of the next state, given the state history, within the interval [l_x̄i(s̄i+1), r_x̄i(s̄i+1)].The theorem defines these bounds using a function of the observational distribution.Source: Appendix I · p. 198
• Theorem 12 bounds the counterfactual expected reward conditional on the state, with its upper endpoint given by 1 − (1 − E[R(Y) | s̄H, x̄H])P(s̄H, x̄H).The reward function is assumed to take values in [0, 1].Source: Appendix I · p. 199
• Theorem 13 establishes a regret bound for UCB+ in a causal decision-making model with bounded rewards, relative to the optimal policy’s expected reward.The proof separates the performance gaps and uses optimistic planning to rule out certain suboptimal policies when the true model is in the relevant sets.Source: Appendix I · pp. 200–201

Theorem 19

• Under NUC and parent-containment conditions, a policy using π_i(X_i | S_i) = P(X_i | S_i) matches the expert’s expected reward.The guarantee applies when NUC holds for the policy space and each action’s endogenous parents are contained in its covariates.Source: Appendix J · pp. 201–202

Theorem 20

• An imitation-admissible subspace yields a policy that perfectly imitates the expert in any SCM compatible with the causal diagram.The cloning policy uses π_i(X_i | Z_i) = P(X_i | Z_i).Source: Appendix J · pp. 202–203

Proposition 6

• For a linear hypothesis class R = {w·ϕ | w ∈ P^k}, the canonical-program solution is obtainable through a minimax problem.The proposition describes the solution using a game matrix.Source: §8.2.2 · p. 162, Appendix J · p. 203

Proposition 7

• For a regularized hypothesis class, the penalized canonical-program solution is obtainable by solving a problem using the conjugate function ψ*.The passage defines ψ* as the maximum over R of a⊤R − ψ(R).Source: Appendix J · p. 204

Theorem 21

• Minimal imitation admissibility makes P^π(Y) computable from P(V) for every policy in the subspace.The theorem gives the result as a formula involving action and covariate sequences up to each decision horizon.Source: Appendix J · p. 204