PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Thomas Mesnard, Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, Éric Moulines, Marcus Hutter, Lars Buesing, Rémi Munos

arXiv:2011.09464v2cs.LG

TL;DR

Credit assignment in reinforcement learning must distinguish an action’s influence on rewards from external factors, a challenge that grows with complex environments. The paper introduces model-free counterfactual credit assignment using future-conditional value functions, and reports improved performance on its challenging environments.

  • Problem

    Credit assignment in reinforcement learning must distinguish actions’ effects on outcomes from external, uncontrollable factors, a challenge that increases as environments become more complex.

    Source: §1 · p. 1

  • Method

    Counterfactual Credit Assignment uses hindsight information to form trajectory-specific counterfactual returns and future-conditional policy-gradient baselines or critics without learning an environment model.

    Source: §1 · pp. 1–2

  • Results

    The authors report improved performance of their algorithm on environments designed to study difficult credit assignment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Future-conditional value functions can separate action influence from other random events and reduce policy-gradient estimator variance.

    Source: §6 · p. 11

    Using future information requires knowing it is independent of the agent’s action; otherwise, conditioning can bias the baseline.

    Source: §2.5 · p. 4

Abstract

from arXiv · show

Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards. In particular, this requires separating skill from luck, i.e. disentangling the effect of an action on rewards from that of external factors and subsequent actions. To achieve this, we adapt the notion of counterfactuals from causality theory to a model-free RL setup. The key idea is to condition value functions on future events, by learning to extract relevant information from a trajectory. We formulate a family of policy gradient algorithms that use these future-conditional value functions as baselines or critics, and show that they are provably low variance. To avoid the potential bias from conditioning on future information, we constrain the hindsight information to not contain information about the agent's actions. We demonstrate the efficacy and validity of our algorithm on a number of illustrative and challenging problems.

1. Introduction

Credit assignment seeks to separate an agent’s contribution to rewards from external factors, but conventional model-free methods credit later rewards coarsely and with high variance. The paper proposes model-free, trajectory-specific counterfactual estimates that hold external factors constant, and presents policy-gradient algorithms with unbiasedness and lower-variance guarantees.

1. Introduction

  • Building an environment model can offer finer-grained understanding of action effects, but complex partially observed environments remain difficult to model.The proposed approach instead draws inspiration from causality theory without requiring an environment model.Source: §1 · pp. 1–2
  • Model-free methods credit rewards after an action to that action, producing high-variance estimates and limiting counterfactual reasoning to actions actually taken.This can require extensive experience to learn action–reward relations.Source: §1 · p. 1
  • As environments grow more complex, each action affects a vanishing part of the outcome, making learning with classical reinforcement learning algorithms increasingly difficult.The cited sources of complexity include partial observability, scale, long time horizons, and more agents.Source: §1 · p. 1
  • Counterfactual Credit Assignment uses hindsight to estimate returns for unchosen actions while keeping external factors constant, without learning an environment model.Its trajectory-specific estimates differ from classical Q functions, which average over possible futures.Source: §1 · pp. 1–2
  • The paper introduces policy-gradient estimators using hindsight, gives a practical instantiation with unbiasedness conditions and lower-variance guarantees, and reports improved performance on its environments.The environments examine exogenous noise, long-term effects, and task interleaving as sources of difficult credit assignment.Source: §1 · p. 2

2. Counterfactual Credit Assignment

This section develops counterfactual policy-gradient estimators that use hindsight information, independent of the credited action given the current state, to distinguish an action’s contribution from luck. It also describes CCA-PG’s practical losses, including a learned action-posterior predictor and a future-conditioned value baseline.

2.2. Policy gradient algorithms

  • REINFORCE weights the taken action by its future return, while subtracting a state-based baseline can reduce variance without bias; the all-action estimator uses a Q-function to signal every action.Baselines are typically restricted to information available before the action, because including variables causally affected by it can bias the estimate.Source: §2.2 · p. 2

2.3. Intuitive example on hindsight reasoning and skill versus luck

  • In the soccer example, a regular baseline may credit Alice for a win caused by Megan, whereas a hindsight baseline conditioned on Megan’s skill can bring Alice’s advantage close to zero.The proposed method is model-free and uses hindsight information only when it was not caused by the agent’s action.Source: §2.3 · p. 3

2.4. Future-conditional (FC-PG) and Counterfactual (CCA-PG) Policy Gradient Estimators

  • When Φt is conditionally independent of At given Xt, CCA-PG is unbiased and its hindsight advantage estimate has no higher variance than the forward estimate.The hindsight advantage can be interpreted as measuring how lucky the outcome was and what might have happened under another action.Source: §2.4 · p. 4

2.5. Learning the relevant statistics

  • Handcrafted hindsight information can use action-independent exogenous factors or other agents’ actions, but may require privileged information and prior knowledge of independence.The section also describes generative models for hindsight statistics, including models of future observations or returns.Source: §2.5 · pp. 4–5
  • A perfect conditional generative model with an exact posterior yields a single-sample hindsight variable conditionally independent of the action, making its posterior value a valid baseline.The same technique gives a posterior Q-function that is uncorrelated with the action and valid as an all-action critic.Source: §2.5 · p. 5
  • CCA learns Φt directly from the trajectory to predict outcomes while enforcing conditional independence from the action being credited.The predictive baseline captures luck, while the resulting advantage reflects skill; the method uses an action-prediction loss to enforce independence.Source: §2.5 · pp. 5–6

2.6. Practical implementation of CCA-PG

  • The practical CCA-PG agent combines a state representation, a trajectory-based hindsight network, a future-conditional value network, and an action predictor for enforcing independence.Its objectives train the hindsight baseline and penalize information about the action given the current state.Source: §2.6 · p. 6
  • The posterior P(a|Xt, Φt) is estimated by hω(At|Xt, Φt), trained with a supervised proper-scoring loss whose optimum is the true posterior.The policy probability log P(a|Xt) is simply π(a|Xt).Source: §2.6 · p. 6
  • The policy-gradient surrogate is LPG = Σt log πθ(At|Xt)(Gt − V(Xt, Φt)), with the value term treated as constant during gradient computation.Source: §2.6 · p. 6
  • The overall loss combines LPG with weighted hindsight, supervised-predictor, and imitation losses; only Lsup updates ω, while other losses treat ω as constant.The total loss is L = LPG + λhsLhs + λsupLsup + λIMLIM.Source: §2.6 · p. 6

3. Connections to causality

The section grounds CCA in structural causal models and relates its future-conditional values to counterfactual returns. Under action-independence and faithfulness assumptions, counterfactuals are identifiable from samples, while CCA avoids explicitly modeling the environment.

3. Connections to causality

  • An SCM represents future states, actions, rewards, and return as a deterministic function of Xt, At, and exogenous variables E, which encode external randomness.The passage says such a representation is always possible, though not necessarily unique.Source: §3 · p. 6
  • Because E is independent of At, V(Xt, E) can be an unbiased future-conditional baseline with potential for lower variance; CCA estimates E implicitly from hindsight information.Future actions are treated as chance when assigning credit to At.Source: §3 · pp. 6–7
  • The counterfactual construction begins by inferring exogenous noise under the observed trajectory, then intervenes by fixing an alternative action A′t.These are the abduction and intervention steps described in the procedure.Source: §3 · p. 7
  • Under faithfulness and Φt ⟂ At | Xt, the counterfactual distribution given Φt is identifiable from samples of (Xt, Φt, At).CCA interprets these counterfactuals through the future-conditional state-action value function.Source: §3 · p. 7
  • CCA discards trajectory information that would require an environment model, retaining Φt as sufficient hindsight information for model-free counterfactuals, baselines, and critics.The section frames this as a way to compute counterfactuals without modeling the environment.Source: §3 · p. 7

4. Numerical experiments

Across bandit feedback, Key-to-Door, and interleaved-task experiments, CCA-PG uses hindsight information to address credit assignment under external variance. It performs strongly on difficult credit-assignment tasks while retaining performance when external variance is low.

4.1. Bandit with Feedback

  • The bandit experiment adds feedback containing information about context, action, and exogenous reward noise, which can support a hindsight value function if action information does not leak into it.Blindly using feedback can leak action information into the hindsight value and yield an advantage estimate of zero, preventing learning.Source: §4.1 · pp. 7–8
  • In contextual bandits, CCA-PG was generally unaffected by increasing exogenous reward noise, whereas vanilla PG performance decreased; weaker independence enforcement dramatically degraded CCA-PG performance.At very low noise, imperfect independence between the hindsight statistic and action biased the update and reduced CCA-PG performance.Source: §4.1 · p. 8, Figure 2 · p. 8

4.2. Key-to-Door environments

  • In Key-To-Door, picking up a key affects a later door reward, while apple rewards can dominate episode return and obscure that credit-assignment signal.The high-variance version varies apple rewards between 1 and 10 while the door reward remains 1, weakening the learning signal for key pickup and door opening.Source: §4.2 · p. 8
  • On High-Variance Key-To-Door, both CCA-PG architectures outperformed baselines in total reward and door-opening probability; on the low-variance task, CCA-PG matched actor-critic’s perfect performance.The comparison includes actor-critic and state- and return-conditional HCA baselines; the low-variance HCA methods struggled with door-opening and apple-gathering.Source: §4.2 · pp. 8–9, Figure 3 · p. 9

4.3. Task Interleaving

  • The interleaving test presents multiple subtasks in random orders within an episode, without clear subtask boundaries, and requires the agent to infer those boundaries.Episodes combine randomly sampled subtasks and color pairs; each subtask’s query and answer rooms may be interleaved differently across episodes.Source: §4.3 · p. 9, Figure 4 · p. 10
  • CCA-PG solved easy and hard tasks in all three interleaving setups in fewer than 5.108 environment steps, while actor-critic failed on hard tasks and earned lower total reward.Actor-critic remained unable to solve hard tasks after 2.109 environment steps; HCA results were not reported because almost no learning occurred on hard tasks.Source: §4.3 · p. 10, Figure 5 · p. 10

5. Related work

Related work spans model-based counterfactual evaluation, hindsight-conditioned estimators, and other approaches to long-term credit assignment. The paper distinguishes CCA by its model-free, trajectory-centric policy-improvement focus and its learned, action-independent future statistics.

5. Related work

  • Buesing et al. (2019) use causal models and real data to generate counterfactual trajectories for off-policy evaluation, whereas this work is model-free and focuses on policy improvement.Heess et al. (2015) also study model-based counterfactual policy-gradient estimates, for continuous actions and differentiable environments.Source: §5 · p. 10
  • Hindsight Credit Assignment uses fixed future states or returns, while CCA learns arbitrary trajectory statistics; the paper says its FC estimators generalize both HCA and CCA estimators.Unlike approaches that treat rewards separately, CCA takes a trajectory-centric view; related HCA variance surrogates are not guaranteed to lower variance.Source: §5 · p. 11
  • Other approaches redistribute delayed rewards or use hindsight for goal-conditioned policies, but the cited work also includes biased estimators and methods that retain time as a proxy for credit.Hung et al. (2019) use attention and episode memory; Arjona-Medina et al. (2019) redistribute delayed rewards earlier in the episode.Source: §5 · p. 11
  • Several hindsight-baseline methods require expert information, whereas CCA’s approach constrains learned hindsight features from revealing the agent’s actions.The paper also points to generative hindsight information and domain-adaptation mechanisms as related directions or analogies.Source: §5 · p. 11

6. Conclusion

The paper concludes that future-conditional value functions can separate action influence from other random effects and reduce policy-gradient variance. Their unbiased use depends on accurate estimation and minimization of mutual information between actions and hindsight information.

6. Conclusion

  • Future-conditional baselines and critics condition on future events to separate action effects on rewards from other random events, reducing policy-gradient estimator variance.The approach builds on causality theory and structural causal models.Source: §6 · p. 11
  • Unbiasedness relies on accurate estimation and minimization of mutual information; inaccurate hindsight classifiers can miscalibrate luck estimates and bias learning.Scaling to more complex environments and evaluating the more general FC-PG and all-actions estimators remain future research directions.Source: §6 · p. 11

A. Algorithmic and implementation details

The appendix describes a constrained objective for learning hindsight information, practical choices for hindsight functions and independence losses, and experimental environment and network details.

A.1. Constrained optimization

  • The hindsight statistic is trained to predict returns while a constraint limits its conditional dependence on the current action given the state.Exact independence corresponds to βIM = 0, which the text says is hard to achieve exactly in practice.Source: §A.1 · p. 14
  • The method replaces the exact hindsight action-probability term with a classifier trained by supervised loss, then relaxes the constraint with a Lagrangian method.The Lagrangian parameter may be fixed or optimized with an algorithm such as GECO.Source: §A.1 · p. 14

A.2. Parameter updates

  • Parameter updates are specified for each trajectory of states, actions, and rewards.The appendix introduces the updates but these supplied passages do not state their individual terms.Source: §A.2 · p. 14

A.3. Design choices

CHOICE OF THE HINDSIGHT FUNCTION ϕ

  • A backward RNN can extract information from arbitrary-length sequences and make hindsight statistics depend on the entire trajectory, with an inductive bias toward near-future observations.The other investigated architecture is a transformer.Source: CHOICE OF THE HINDSIGHT FUNCTION ϕ · p. 14

INDEPENDENCE MAXIMIZATION CONSTRAINT LIM

  • The independence constraint is estimated with conditional mutual information or KL divergence using the hindsight classifier; the paper reports no significant difference, with KL slightly outperforming mutual information.The conditional mutual information compares action entropy given state with action entropy given state and hindsight information.Source: INDEPENDENCE MAXIMIZATION CONSTRAINT LIM · pp. 14–15

B.1.1. ENVIRONMENT

  • In the bandit environment, reward is negative squared context-action distance plus Gaussian noise, while feedback combines context, action, and noise through fixed matrices.The matrices are sampled from a standard Gaussian distribution for each experimental seed.Source: §B.1.1 · p. 15

B.1.2. ARCHITECTURE

  • The bandit hindsight feature uses a backward RNN, while the hindsight classifier is a two-hidden-layer MLP and policy and value functions use concatenated observation and feedback.The appendix lists several RNN cores and says all weights are jointly trained with Adam.Source: §B.1.2 · p. 15

B.1.3. ADDITIONAL RESULTS

In the multi-agent bandit, CCA-PG learns faster and reaches higher performance than vanilla policy gradient, though it does not learn the optimal policy. The supplied passages also describe the bandit and Key-To-Door settings and the agents’ architecture.

B.1.3. ADDITIONAL RESULTS

  • The multi-agent bandit couples agents’ feedback and rewards, with other agents taking the role of exogenous noise.Each agent acts in its own replica, while the terminal joint reward sums their context-action rewards.Source: §B.1.3 · p. 15
  • CCA-PG agents learn faster and reach higher performance than vanilla policy-gradient agents as interacting agents increase, but never learn the optimal policy.The effective variance of vanilla policy-gradient estimates rises with the number of agents, while each agent’s performance falls.Source: §B.1.3 · p. 15, Figure 6 · p. 16

B.2.1. ENVIRONMENT DETAILS

  • In High-Variance-Key-To-Door, the forward baseline is 55 before the apple value is observed, while hindsight can condition it to 10 or 100.The hindsight feature can represent the current episode’s apple value independently of the agent’s actions.Source: Table 1 · p. 16, §B.2.1 · p. 17

B.2.2. ARCHITECTURE

  • The agent uses a forward LSTM state and computes hindsight features with either a backward LSTM or an attention mechanism.The hindsight baseline adds a learned residual to the forward baseline, using the forward state and hindsight feature.Source: §B.2.2 · p. 17

B.2.3. ADDITIONAL RESULTS

The appendix reports that CCA handles exogenous reward variance in Key-To-Door and solves all six interleaved tasks quickly. It also examines attention, training setup, and the effect of hindsight and unroll choices.

B.2.3. ADDITIONAL RESULTS

  • CCA’s hindsight baseline loss decreases during learning as it accounts for trajectory-specific exogenous factors, while vanilla policy-gradient baseline loss initially increases.The vanilla baseline loss rises as variable apple rewards make collecting more apples increase the forward baseline loss.Source: §B.2.3 · pp. 17–18, Figure 8 · p. 18
  • As apple-reward variance increases, CCA-PG is almost unaffected in key-pickup and door-opening probability, while vanilla policy gradient drops dramatically.The variance comes from differences in apple rewards between episodes.Source: §B.2.3 · p. 18, Figure 9 · p. 18
  • CCA-PG attention focuses most on timesteps just after apple pickups and very little on the door-opening timestep.The figure shows a heatmap over the episode and attention over future timesteps at key pickup; the analysis used one attention head.Source: §B.2.3 · p. 18, Figure 10 · p. 18

B.3.1. ENVIRONMENT DETAILS

  • In the interleaving setup, episodes last 140 steps, and reaching a colored square takes at least nine steps; five of ten squares per task yield positive rewards.The six tasks have rewards 80, 4, 100, 6, 2, and 10; tasks 2, 4, 5, and 6 are designated hard.Source: §B.3.1 · p. 19

B.3.2. ARCHITECTURE

  • Task Interleaving uses the same architecture setup as Key-To-Door, with full-episode trajectories and a discount factor of 0.99.The appendix points to a separate table for the optimal hyperparameters for each algorithm.Source: §B.3.2 · p. 19

B.3.3. ADDITIONAL RESULTS

  • CCA solves all six interleaved tasks quickly, while vanilla policy gradient reliably solves only the two easy tasks.The four remaining tasks have smaller associated rewards.Source: §B.3.3 · p. 19

B.3.4. ABLATION STUDY

The appendix studies how future access and backpropagation windows affect CCA-RNN, relates FC estimators to HCA and CCA, and gives proof steps for estimator validity and variance. Together, the results distinguish access to future information from gradient propagation through the full sequence.

B.3.4. ABLATION STUDY

  • With full rollouts, hard-task learning is not much affected by limiting how many steps gradients backpropagate into the hindsight RNN.The hindsight function still sees the whole future despite the limited backward-gradient window.Source: §B.3.4 · p. 19, Figure 12 · p. 20
  • CCA-RNN appears to need access to many future steps for hard tasks, but not backpropagation through the full sequence.The unroll-length analysis reports that less future access makes hard credit-assignment tasks harder to solve.Source: §B.3.4 · pp. 19–20, Figure 13 · p. 20

C. Relation between HCA, CCA, and FC estimators

  • FC estimators generalize HCA and CCA; state and return HCA arise from FC, while CCA follows under conditional independence of Φ_t and A_t.For state HCA, the construction uses Φ_t = X_{t+k} for each reward offset k; HCA and CCA use different unbiasedness approaches.Source: C. Relation between HCA, CCA, and FC… · p. 21

D.1. Policy gradients

  • The policy-gradient proof rewrites the expected return over trajectories and uses score-function terms; subtracting a state-value baseline has zero expected score contribution.The proof establishes E[S_tV(X_t)] = 0 by conditioning on X_t.Source: §D.1 · pp. 21–22

D.2. Proof of FC-PG theorem

  • The FC-PG proof validates its baseline by conditioning successively on the state and Φ_t, with Φ_t not being the action that generated it.The proof passage identifies that action choice as producing a biased estimator.Source: §D.2 · pp. 22–23

D.3. Proof of proposition 1 (existence of independence-maximization loss)

  • The CCA-PG theorem follows from the FC-PG theorem when Φ_t and A_t are conditionally independent given X_t.The appendix also derives conditional independence from a zero expected KL divergence between action distributions.Source: §D.4 · p. 23, §D.3 · p. 23

D.4. Proof of CCA-PG theorem

  • The hindsight advantage estimate has variance no greater than the forward-baseline advantage estimate.The appendix derives this inequality using Jensen’s inequality.Source: §D.4 · p. 23

D.5. Proof of generative-CCA results

The generative-CCA argument establishes valid hindsight baselines from action-uncorrelated information. The variance analysis relates excess policy-gradient variance to the conditional variance of the advantage estimate.

D.5. Proof of generative-CCA results

  • Under the prior, ε and A_t are conditionally independent, giving the factorization used to establish the generative-CCA result.The proof also assumes an exact model and posterior when equating the sampled posterior with the data-distribution posterior.Source: §D.5 · p. 24
  • Averaging valid baselines sampled from the posterior yields a valid baseline, and the posterior-weighted value function is also uncorrelated with the action.The proof uses the strong law of large numbers for the sequence average.Source: §D.5 · p. 24

E.1. Relation between variance of advantage and variance of policy gradient

  • The extra variance from an unbiased advantage estimate Y_t, relative to the exact advantage, is E[S_t^2 V[Y_t|X_t,A_t]|X_t].The variance of the policy-gradient term also depends on score-function variance and its correlation with the advantage.Source: §E.1 · p. 25
  • For discrete actions with a softmax over logits, excess advantage variance upper-bounds excess variance in the policy-gradient estimate up to a constant factor.The analysis bounds score derivatives with respect to logits by 1.Source: §E.1 · p. 25

E.2. Variance analysis in the bandit problem

In the two-action bandit analysis, hindsight information can remove reward noise from an unbiased gradient estimate. The paper connects this mechanism to common randomness and represents RL dynamics with structural causal models.

E.2. Variance analysis in the bandit problem

  • Choosing αᵀW = 1 and αᵀV₀ = αᵀV₁ = 0 makes the hindsight baseline remove reward noise while remaining independent of the action, yielding an unbiased lower-variance estimator.A different choice that also cancels action-specific terms can produce a biased update because its advantage is independent of the action.Source: §E.2 · pp. 25–26

F. RL algorithms, common randomness, structural causal models

  • The causal view of CCA-PG uses structural causal models and is closely related to common random numbers, a standard optimization technique with simulators.The section presents this as an alternative intuition for the algorithm.Source: F. RL algorithms, common randomness,… · p. 26

F.1. Structural causal model of the MDP

  • A structural causal model represents variables as deterministic functions of their parents and exogenous randomness, with exogenous variables assumed independent and the graph acyclic.The figure distinguishes deterministic square nodes from stochastic circular nodes and shades observed nodes gray.Source: §F.1 · p. 26, Figure 15 · p. 26
  • The MDP representation treats transitions and policy actions as functions of state and exogenous random variables, with reward deterministic in state and action.The return is consequently a deterministic function of the current state, action, and subsequent exogenous randomness.Source: §F.1 · pp. 26–27

F.2. Proof of theorem 4

Under the faithfulness assumption, the reduced causal model separates hindsight information from the action conditional on the current state. The following section examines how distinct structural models can yield different counterfactuals despite matching distributions.

F.2. Proof of theorem 4

  • After integrating out the trajectory, faithfulness rules out a causal path from A to Φ, so G depends on X, A, and ε while Φ depends on X and ε.The corresponding reduced-action-selection SCM is shown with and without the trajectory.Source: §F.2 · p. 27, Figure 16 · p. 27
  • The proof uses d-separation to obtain p(ε|x,φ) = p(ε|x,a,φ), which supports the stated counterfactual-return result.The section defines the counterfactual return by conditioning on the observed hindsight information Φ.Source: §F.2 · p. 27

G. Individual Treatment Effects, (Conditional) Average Treatment Effects, Counterfactuals and Counterfactual identifiability

  • Different structural causal models can represent the same distribution yet imply different counterfactuals, motivating analysis of counterfactual identifiability.The paper investigates this issue in simple settings with closed-form computations.Source: G. Individual Treatment Effects,… · p. 27

G.1. Individual and Average Treatment Effects

The treatment-effect discussion distinguishes individual counterfactual effects from population averages and shows how counterfactual estimates relate to CCA-style advantages. Its examples compare counterfactual variance across structural models, while expected policy learning remains aligned across models with the same observed distribution.

G.1. Individual and Average Treatment Effects

  • A counterfactually estimated ITE equals the true ITE when S is identifiable from O and F, and averaging CF-ITE over S and F conditional on H yields the ATE.The true ITE is typically unknowable when exogenous state S is unknown, and residual uncertainty can make CF-ITE differ from ITE.Source: §G.1 · p. 28
  • CCA-like CF-ITE compares an outcome with a counterfactual value, whereas the usual return-based advantage contrasts a sample return with a population value and can have very high variance.The paper relates ATE to a difference of Q functions and ITE to a difference of returns under common randomness.Source: §G.1 · p. 28
  • Different SCMs can imply different individual and counterfactual effects while sharing the same ATE; inaccurate counterfactuals do not alter estimator expectations or the optimal policy given available information.The paper suggests preferring an SCM with lower CF-ITE variance for learning good policies in expectation.Source: §G.1 · p. 29

G.2. Betting against a fair coin

  • Two fair-coin SCMs can have the same zero ATE but different ITEs: model I gives ITE ±1, while model II gives ITE 0.In model I the guess changes the outcome; in model II the guess has no effect.Source: §G.2 · p. 29

G.3. Medical example

  • Model A has CF-ITE variance 1/6, while model B has variance 1, despite both assigning treatment value 2/3 and no treatment value 1/3.The table summarizes counterfactual probabilities, outcomes, effects, and CF-ITE variance for the medical example.Source: §G.3 · p. 29, Table 5 · p. 30
Loading 2011.09464v2…