PaperlayerView exact revision on arXiv

Source-linked AI summary

Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, Nicolas Heess

arXiv:1811.06272v1cs.LGstat.ML

TL;DR

Model-based reinforcement learning needs data but can be biased when synthetic experience mismatches the environment. The paper proposes CF-GPS, which uses structural causal models to evaluate alternative actions in logged episodes, and reports improved policy evaluation and search while connecting the approach to prior RL methods.

  • Problem

    Model-based RL can suffer from biased policy evaluation and search because generating plausible synthetic experience is difficult and learned models may mismatch the environment.

  • Method

    CF-GPS uses structural causal models and counterfactual inference to evaluate arbitrary policies on logged off-policy episodes under actions that were not taken.

  • Results

    The paper reports improved policy evaluation and search results on a non-trivial grid-world task, and shows that Guided Policy Search and Stochastic Value Gradient methods can be interpreted as counterfactual methods.

  • Takeaways & Limitations

    Counterfactual evaluation of concrete past scenarios can mitigate model bias relative to relying only on synthetic scenarios.

  • Takeaways & Limitations

    The approach assumes that off-policy experience is available and that there are no additional hidden confounders in the environment.

Abstract

from arXiv · show

Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.

1 INTRODUCTION

The paper addresses data inefficiency and model bias in reinforcement learning by using counterfactual reasoning over logged experience. It introduces CF-GPS and connects structural causal models with reinforcement learning, causal inference, and prior RL algorithms.

  • Standard model-free RL is data inefficient, while model-based learning can fail when synthetic experience comes from a mismatched model.
  • CF-GPS models alternative outcomes of logged experience under actions that were not actually taken, rather than generating scenarios entirely de novo.
  • Structural causal models provide the framework for formulating model-based reinforcement learning in POMDPs and connecting RL with causal inference.
  • The paper reports that counterfactual reasoning on off-policy data can facilitate solving non-trivial reinforcement-learning tasks.
  • CF-GPS extends the interpretation of Guided Policy Search and Stochastic Value Gradient methods as counterfactual approaches.
  • The paper develops SCM and counterfactual-inference preliminaries before applying them to policy evaluation and policy search.

2 PRELIMINARIES

The preliminaries define POMDP notation, structural causal models, interventions, and counterfactual inference. They represent POMDP dynamics as deterministic mechanisms driven by independent noise variables and use policies as action-generating mechanisms.

  • The setting is an episodic POMDP with states, actions, observations, histories, trajectories, stochastic policies, and undiscounted return.
  • An SCM consists of a directed acyclic graph, independent noise variables, and functions that generate variables from their parents and corresponding noise.
  • The paper calls the noise variables scenarios and the structural functions causal mechanisms.
  • An intervention replaces selected causal mechanisms and produces a modified SCM with its own distribution.
  • Any POMDP under a policy can be represented as an SCM by expressing conditional distributions as deterministic functions of their inputs and independent noise.
  • In this representation, the policy generates actions, while environment mechanisms generate transitions and observations from scenarios and actions.
  • A counterfactual query combines observations, an intervention, and query variables to ask what would have happened under the intervention while keeping everything else the same.

2. Perform intervention I on Mˆxo. This yields Mdo(I)

Counterfactual inference updates latent scenario information using observed outcomes, applies an intervention, and returns the resulting counterfactual distribution. This supports unbiased simulation and reasoning about alternative actions in concrete episodes.

  • After conditioning on observations, counterfactual inference performs the intervention on the updated SCM and returns the marginal distribution of the queried variables.
  • Marginalizing the counterfactual distribution over observations yields an unbiased estimator of the queried variables’ distribution under the intervention.
  • The mixed-sampling corollary permits unbiased samples under an intervention by drawing some noise variables from their posterior and the remainder from their prior.
  • For POMDP models, the method infers scenario variables from data while randomizing action noise, combining observed episode information with prior simulation.
  • In the job-choice example, observed outcomes are used to infer the experienced company scenario before evaluating the outcome under the alternative company.

3 OFF-POLICY EVALUATION: MODEL-FREE, MODEL-BASED AND COUNTERFACTUAL

Off-policy evaluation can use structural causal models to infer scenarios from logged episodes and evaluate alternative policies counterfactually. Under no model mismatch this estimate is unbiased, while experiments show that conditioning on more real data improves accuracy when the prior scenario model is difficult to learn.

  • Importance sampling can have large variance when target and behavior trajectory distributions differ, and it can be useless when the behavior policy lacks target-policy support.
  • Model-based policy evaluation avoids running the target policy but inherits bias from inaccuracies in the environment model.
  • CF-PE infers scenario variables from logged trajectories and applies an intervention changing the behavior policy to the target policy before querying the return.
  • Under no model mismatch, CF-PE produces an unbiased estimate, and mixed posterior-prior sampling can randomize counterfactual action noise.
  • CF-PE can de-bias imperfect models by conditioning on observed episodes, especially when transition and reward mechanisms are accurate but the marginal noise distribution is difficult to model.
  • In PO-SOKOBAN, full-episode inference reliably evaluated policies, whereas the learned initial-state prior generated malformed levels; intermediate data amounts traded off evaluation accuracy and data use.
  • Importance sampling failed in PO-SOKOBAN, with relative error > 0.8 for all three evaluated policies under the uniform behavior policy.

4 OFF-POLICY IMPROVEMENT: COUNTERFACTUALLY-GUIDED POLICY

The section develops CF-GPS by grounding model-based policy-search rollouts in off-policy data rather than sampling scenarios solely from a model prior. In PO-SOKOBAN, this grounding produces better training data and outperforms vanilla model-based and model-free comparisons.

  • 4.2 INCORPORATING OFF-POLICY DATA: COUNTERFACTUALLY-GUIDED POLICY SEARCH: The section generalizes model-based policy search into a counterfactual algorithm intended to increase robustness to model mismatch.The construction starts from return-weighted regression over model rollouts and then anchors those rollouts in observed off-policy episodes.
  • 4.1 STARTING POINT: VANILLA MODEL-BASED RL WITH RETURN WEIGHTED REGRESSION: MB-PS improves the current policy by fitting it to model-generated trajectories weighted toward high-return outcomes.A planner may provide a proposal distribution over trajectories, including through search from sampled full states.
  • 4.2 INCORPORATING OFF-POLICY DATA: COUNTERFACTUALLY-GUIDED POLICY SEARCH: CF-GPS replaces prior scenario sampling with counterfactual trajectories anchored in replay-buffer data and infers scenarios from those observations.Under no model mismatch, the resulting procedure is unbiased; its policy is optimized on rollouts grounded in observed episodes.
  • 4.2 INCORPORATING OFF-POLICY DATA: COUNTERFACTUALLY-GUIDED POLICY SEARCH: When the prior is difficult to model, the counterfactual distribution is expected to concentrate more heavily where the true environment has probability mass.This motivates using inferred scenarios instead of unconditional prior samples.
  • 4.3 EXPERIMENTS: In PO-SOKOBAN, CF-GPS infers levels from complete off-policy histories, whereas MB-PS and GPS-like use unconditional or first-observation-conditioned scenario models.The experiment uses 64 actors collecting real data with a behavior policy that may be slightly outdated.
  • 4.3 EXPERIMENTS: CF-GPS outperforms MB-PS and GPS-like, while its inferred levels yield high-quality training data and better data use than a model-free baseline with the same environment data.The model-free method does not receive the true transition and reward kernels available to CF-GPS.

5 RELATED WORK

The related-work discussion distinguishes CF-GPS from importance-sampling approaches and connects it to Guided Policy Search and Stochastic Value Gradient methods. These connections frame CF-GPS as a counterfactual grounding of model-based search and reparameterized policy optimization.

  • Importance Sampling: Importance-sampling methods reweight off-policy data, whereas CF-GPS infers noise variables from logged data and reuses them to evaluate alternative actions.The related approaches are described as having high-variance estimators, despite improvements to importance sampling.
  • Importance Sampling: CF-GPS is complementary to Hindsight Experience Replay because HER fixes an observed outcome and searches for a policy that would have achieved it, whereas CF-GPS searches alternative outcomes for a policy.The distinction concerns whether the outcome or the policy-side counterfactual is held fixed.
  • Guided Policy Search: In the fully observed MDP setting with a linear SCM and an LQR dynamic-programming planner, CF-GPS is equivalent to Guided Policy Search.The paper also characterizes GPS as the counterfactual variant of the corresponding MB-PS procedure.
  • Guided Policy Search: Counterfactual grounding of model-based rollouts is associated with improved performance over naive prior-sample-based MB-PS.The paper expects CF-GPS to be especially advantageous when transition uncertainty cannot be identified reliably from adjacent observations alone and broader histories help infer it.
  • Stochastic Value Gradient methods: Stochastic Value Gradient methods can be viewed through the same counterfactual lens because reparameterization casts the stochastic model as an SCM.SVG infers transition noise from successive observed states, freezes action noise, and differentiates return with respect to policy parameters.

6 DISCUSSION

The discussion presents counterfactual evaluation of concrete past scenarios as a way to mitigate model bias relative to synthetic scenarios. Its applicability depends on logged experience and structural assumptions about hidden variables and modelability.

  • 6 DISCUSSION: Counterfactual evaluation of alternative actions in past scenarios can mitigate model bias compared with considering only synthetic scenarios.The paper frames this as the main takeaway of the work.
  • 6 DISCUSSION: The methods require available off-policy experience; when collecting it is too costly, they cannot be used and the simulator or model must be relied upon exclusively.This is presented as an assumption limiting applicability.
  • 6 DISCUSSION: The approach assumes no additional hidden confounders and focuses modeling difficulty on the noise-source distribution while treating transition and reward kernels conditional on noise as easy to model.The authors note that this assumption may fit some partially observed grid-worlds but not all environments.
  • 6 DISCUSSION: The paper identifies inference over noise U from observed history as its most restrictive assumption.The supplied discussion states this boundary but does not elaborate further here.

A.1 PROOF OF LEMMA 1

The proof begins by using the fact that root-node noise distributions remain unchanged under interventions, then integrates over noise and observed data to establish the required interventional distribution.

  • A.1 PROOF OF LEMMA 1: Noise-source distributions remain unchanged under interventions because the noise variables are root nodes in the causal graph.This invariance is the proof’s starting point.
  • A.1 PROOF OF LEMMA 1: The proof expresses the interventional density by integrating the intervention-specific conditional density against the posterior noise distribution and the observed-data density.The displayed expression combines pdo(I)(x|u), p(u|x̂o), and p(x̂o) through integration.

A.2 PROOF OF COROLLARY 1

The proof establishes independence between the counterfactual and prior scenario components, then uses averaged inference distributions to recover the prior marginal needed for the corollary.

  • Proof: Independent prior noise variables make uCF and uPrior independent when CF and Prior partition all variables.The factorized prior p(u) = ∏n p(un) implies p(u) = p(uCF)p(uPrior).
  • Proof: Averaging the inference distribution over observations recovers the prior distribution, including the marginal p(uCF).The argument applies to any subset of the variables, not only the full noise vector.
  • Proof: Combining independence with the averaged marginal yields p(u) = E[p(uCF|x̂o)]p(uPrior).This factorization is the displayed conclusion of the proof.
  • Reparametrization: Auto-regressive uniformization represents sequential random variables using functions fn and independent Uniform[0,1] variables.The construction proceeds through conditional inverse-CDF transformations.

C MODEL ARCHITECTURE

The PO-SOKOBAN model assumes known deterministic dynamics and learns only the initial-state distribution with a convolutional DRAW model.

  • Model architecture: Because PO-SOKOBAN transitions are deterministic, the learned component is the initial state distribution p(Us1).The transition and reward kernels are assumed to be known.
  • Model architecture: The initial-state model is learned from data using DRAW, a parametric multilayer latent-variable neural network.The implementation uses the convolutional DRAW architecture.
  • Model architecture: The architecture is selected to model distributions over the initial state in the PO-SOKOBAN environment.The passage identifies convolutional DRAW as the specific architecture used for this purpose.

D MODEL MISMATCH ANALYSIS

The analysis compares learned latent distributions conditioned on increasing observation histories and finds that conditioning improves their match to the data.

  • Experimental setup: The analysis examines separate models for t = 0, 1, and 50 observations, corresponding to unconditional, filtering, and smoothing settings.Each model parameterizes the learned distribution over the initial state using a different amount of conditioning data.
  • Latent-space comparison: The learned prior p(Z|ĥt) is compared with the inferred embedding distribution of true initial states in two latent dimensions.The prior is visualized as a whitened one-standard-deviation contour, while true-level embeddings appear as crosses and aggregated density.
  • Model mismatch: Conditioning on observations reduces the mismatch between learned distributions and the data.The figure summary states that the distributions match the data better as the amount of conditioning data increases.
  • Model mismatch: For t = 0, prior mass occupies regions with little or no true data, and the prior fails to capture structure in the true-data embedding.The passage links this mismatch to unrealistic synthetic data and reduced test performance when policies are trained on it.
Loading 1811.06272v1…