PaperlayerView exact revision on arXiv

Source-linked AI summary

Off-Policy Deep Reinforcement Learning without Exploration

Scott Fujimoto, David Meger, Doina Precup

arXiv:1812.02900v3cs.LGcs.AIstat.ML

TL;DR

Fixed-batch reinforcement learning is needed when new data collection is unavailable or costly, yet standard off-policy deep RL fails when batch data diverges from the current policy because of extrapolation error. The paper introduces batch-constrained reinforcement learning and BCQ, which restrict actions toward those represented in the batch. BCQ is presented as a continuous-control algorithm that learns from arbitrary fixed data without exploration across several tasks.

  • Problem

    Fixed-batch reinforcement learning lacks further environment interaction, while standard off-policy deep RL can fail on data uncorrelated with the current policy because of extrapolation error.

  • Method

    Batch-constrained reinforcement learning restricts selected actions toward batch-supported behavior; BCQ combines a state-conditioned generative model, action perturbation, and Q-network selection.

  • Results

    BCQ is presented as the first continuous-control deep RL algorithm capable of learning from arbitrary fixed batch data without exploration.

  • Takeaways & Limitations

    Batch-constrained learning provides an approach to learning from expert demonstrations and finite suboptimal batches without further exploration.

  • Takeaways & Limitations

    Reweighting cannot ensure accurate estimates when high-likelihood state-action pairs under the current policy are absent from the batch.

Abstract

from arXiv · show

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

1. Introduction

Batch reinforcement learning targets fixed datasets, but standard off-policy deep RL can fail when those datasets do not match the current policy. The paper introduces batch-constrained learning and BCQ to address extrapolation error without further environment interaction.

  • Fixed-dataset reinforcement learning supports applications where data collection is costly, risky, or time-consuming.
  • Standard off-policy deep RL can fail when the dataset is uncorrelated with the current policy, even when behavioral and off-policy agents use the same data.
  • Extrapolation error assigns unrealistic values to unseen state-action pairs because policy-induced and batch data distributions mismatch.
  • Batch-constrained reinforcement learning maximizes reward while minimizing mismatch between policy visitation and state-action pairs in the batch.
  • BCQ uses a state-conditioned generative model to produce previously seen actions and a Q-network to select a high-valued similar action.
  • BCQ learns continuously controlled behavior from fixed batch data without environment interaction, including expert demonstrations and suboptimal data.

2. Background

The background formalizes reinforcement learning as value-based decision-making in an MDP and explains how off-policy Q-learning and actor-critic methods operate with replay data. Continuous control motivates DDPG because directly maximizing over actions is intractable.

  • An MDP specifies states, actions, transition dynamics, rewards, and a discount factor, while the agent seeks to maximize expected discounted return.
  • A policy maps states to actions, and its value function gives the expected return after taking an action and following that policy.
  • The Bellman operator has a unique fixed point for γ in [0, 1), and Q* is the optimal value function obtained through greedy action choices.
  • Q-learning is off-policy because its target does not depend on how the experience was generated, and training typically samples replay tuples from a dataset.
  • In continuous action spaces, analytic maximization is intractable, so actor-critic methods use a policy network and value-based updates for action selection.
  • Combining off-policy deep Q-learning with deterministic policy gradients yields Deep Deterministic Policy Gradients, or DDPG.

3. Extrapolation Error

Extrapolation error arises when off-policy value updates evaluate actions unsupported by the batch, causing inaccurate or unstable estimates. Experiments show DDPG deteriorates on fixed Hopper-v1 datasets, while mismatch, optimistic actions, and limited coverage prevent reliable learning.

  • Extrapolation Error: Extrapolation error occurs when a target policy selects an unfamiliar next-state action that is unlikely or absent in the dataset.
  • Extrapolation Error: Absent state-action data can make Qθ(s′, π(s′)) arbitrarily inaccurate without sufficient nearby data.
  • Extrapolation Error: Sampling transitions from a finite batch can bias estimated transition dynamics in stochastic MDPs.
  • Extrapolation Error: When batch and current-policy distributions differ, the learned value function may poorly estimate actions selected by the current policy.
  • Extrapolation Error: Loss reweighting cannot fix missing high-likelihood state-action pairs, so only a subset of policies may be evaluated accurately.
  • Deep RL Experiments: DDPG performance deteriorates rapidly on uncorrelated data while its deep-Q value estimate diverges, showing that standard off-policy deep RL is ineffective when truly off-policy.
  • Deep RL Experiments: Across Hopper-v1 batch tasks, off-policy agents underperform behavioral agents, including when both use identical data, while value estimates become divergent.
  • Deep RL Experiments: With expert imitation data, the agent quickly selects non-expert actions under optimistic extrapolation, causing value estimates to diverge and learning to fail.

4. Batch-Constrained Reinforcement Learning

Batch-constrained reinforcement learning limits policy actions to batch-supported behavior, reducing extrapolation error while preserving value-based optimization. Its tabular and deep variants provide theoretical guarantees under finite-MDP conditions and implement the constraint through generative action modeling.

  • Batch-Constrained Policies: Batch-constrained policies align state-action visitation with the batch and optimize distance, future-state familiarity, and value.The distance-to-batch objective is primary because future-state estimates and value estimates may otherwise be arbitrarily poor.
  • Finite MDP Analysis: In finite MDPs, extrapolation error arises from transition-distribution mismatch between the batch-induced MDP and the true MDP.The error depends on transition-distribution divergence, value weighting, and errors at succeeding states.
  • Finite MDP Analysis: For deterministic MDPs, batch-constrained policies eliminate extrapolation error exactly when they select only state-action pairs contained in a coherent batch.A coherent batch contains successor states for every nonterminal transition, enabling a batch-constrained policy when the start state is included.
  • Batch-Constrained Q-Learning: BCQL constrains each Q-learning backup to actions observed in the batch and converges to the optimal batch-constrained policy under stated sampling and learning-rate conditions.For deterministic MDPs with coherent batches, this policy is at least as good as any other batch-constrained policy and can outperform behavioral policies.
  • Deep BCQ: BCQ generates batch-similar candidate actions, selects among them with Q-networks, and penalizes rare or unseen states to control overestimation.The approach uses a state-conditioned generative model, a perturbation model, and two Q-networks.

5. Experiments

BCQ is evaluated in MuJoCo batch-learning settings against deep reinforcement learning and imitation-learning baselines. It succeeds across tasks, maintains stable value estimates, and handles noisy demonstrations better than the compared methods.

  • Experimental Setup: BCQ is compared with DDPG, discretized-action DQN, behavioral cloning, and a VAE-based variant in unmodified MuJoCo environments.The evaluation includes final-buffer, concurrent, imitation, and imperfect-demonstration settings.
  • Results: BCQ is the only algorithm that succeeds at all tasks, matching or outperforming the behavioral policy in each instance except imitation learning.Behavioral cloning performs best in the imitation-learning task.
  • Results: BCQ exhibits a highly stable value function with off-policy samples, unlike DDPG and DQN.The reported value estimates are compared with an estimate of BCQ’s true value in the Hopper-v1 analysis.
  • Results: In imperfect demonstrations, BCQ strongly outperforms the noisy demonstrator while deep reinforcement learning and imitation-learning algorithms perform poorly.The task uses expert demonstrations containing two sources of noise.

6. Related Work

Related work covers batch reinforcement learning, imitation learning, and uncertainty estimation. BCQ differs by constraining policies to behaviorally supported regions rather than relying primarily on evaluation, importance sampling, or exploration-oriented uncertainty.

  • Batch Reinforcement Learning: Earlier batch methods offer convergence or function-approximation approaches but generally lack policy-quality guarantees without infinite data or lack convergence guarantees.The paper positions its contribution as constraining the learned policy to adequately evaluable subsets.
  • Batch Reinforcement Learning: Importance-sampling methods may be unsuitable for batch settings because they require behavioral-policy action probabilities and scale poorly with multidimensional actions.BCQ instead constrains the policy to a subset of policies that can be evaluated from the batch.
  • Imitation Learning: BCQ connects to imitation learning and residual policy learning, using a generative model as the initial policy rather than an expert or feedback controller.The generative model supplies the starting policy in this connection.
  • Uncertainty in Reinforcement Learning: Prior uncertainty methods generally estimate value uncertainty to encourage exploration into unseen state-space regions.This contrasts with BCQ’s batch-constrained objective in the paper’s framing.

7. Conclusion

The paper identifies extrapolation error as a critical problem in finite-data off-policy reinforcement learning and presents BCQ as a batch-constrained response. BCQ is described as capable of learning from arbitrary batch data without exploration.

  • Extrapolation error arises when value targets estimate unseen state-action pairs in finite-data off-policy reinforcement learning.
  • Batch-constrained reinforcement learning keeps behavior close to on-policy with respect to the available batch.
  • BCQ is presented as the first continuous-control algorithm capable of learning from arbitrary batch data without exploration.

A.1. Proofs and Details from Section 4.1

The theoretical results characterize batch sampling as learning in a batch-induced MDP and establish convergence properties for batch-constrained policies and BCQ under stated assumptions. The displayed DDPG experiment provides an empirical contrast with these guarantees.

  • A coherent batch contains successor states for every nonterminal transition, enabling a batch-constrained policy when the start state is present.
  • Sampling transitions uniformly from batch B makes Q-learning converge to the optimal value function of the batch-induced MDP M_B.For state-action pairs absent from B, Q-learning never updates the value and retains its initialized value.
  • The batch-induced value error is zero exactly when the batch and true transition distributions agree on state-action pairs visited by the policy.
  • For deterministic MDPs, zero value error is equivalent to the policy being batch-constrained, and a coherent batch supports such a policy when it contains the start state.
  • Under Robbins-Monro learning-rate conditions and standard sampling assumptions, BCQL converges to the optimal value function or the optimal batch-constrained value function, depending on the setting.
  • In the DDPG batch experiments, the final-buffer, concurrent, and imitation settings exhibit poor learning, degraded performance, or divergent value estimates.

B.2. Complete Experimental Results

The complete experiments show that BCQ mitigates extrapolation error across varied fixed-batch settings. The authors also report a hyper-parameter-sensitive instability in one concurrent-learning environment.

  • Only BCQ matches or outperforms the behavioral policy in all evaluated tasks while maintaining a highly stable value function in each task.
  • BCQ successfully mitigates extrapolation error and learns across a variety of fixed-batch settings.
  • With slight hyper-parameter changes, BCQ periodically became unstable in HalfCheetah-v1 concurrent learning after 750,000 or more iterations on some seeds.The authors hypothesize that the instability may reflect the generative model producing out-of-distribution actions.

C. Extrapolation Error in Kernel-Based Reinforcement Learning

The KBRL analysis shows that restricting a batch to a subset of transitions can produce extrapolated values for unseen state-action pairs. In a two-state, two-action example, this causes a degenerate policy despite correct values on observed pairs.

  • In the toy MDP, the optimal batch contains transitions selecting a1 at s0 and a0 at s1, with reward 1 only for selecting a1 at s0.
  • KBRL can converge to the optimal value with the complete deterministic MDP but becomes susceptible to extrapolation when given only a subset.
  • KBRL correctly estimates observed values but extrapolates action values for unseen state-action pairs, producing a policy that continually selects a1.

D.1. Ablation Study of Perturbation Model

The perturbation model’s constraint controls how far BCQ can move sampled actions from the batch. Larger perturbations worsen performance and destabilize value estimates, while uncertainty ensembles stabilize values without sufficiently constraining actions.

  • Perturbation Model: Increasing Φ causes a clear performance drop and greater value-function instability in BCQ.Larger Φ permits actions farther from generated batch actions, where suboptimal actions can receive erroneously high values.
  • Perturbation Model: The perturbation range should remain small enough to stay near generated actions while allowing learning from exploratory actions in the dataset.
  • Uncertainty Estimation: BCQ outperforms uncertainty-based ensemble methods on the Hopper-v1 imitation task because those methods do not sufficiently constrain the action space.
  • Uncertainty Estimation: Ensemble methods stabilize value functions but fail to constrain actions to demonstrated expert actions.The larger ensemble provides a more accurate uncertainty estimate, but neither tested ensemble sufficiently enforces the expert-action constraint.
  • Random Behavioral Policy: On simple tasks with random-policy data, both BCQ and DDPG learn successfully from a 5000-step batch, indicating BCQ does not require expert data.

F. Experimental Details

The experiments use standardized training and evaluation procedures across several batch constructions, while BCQ combines a VAE, perturbation model, and paired value networks. Implementation details specify architectures, targets, regularization, and terminal-state handling.

  • Experimental Protocol: Experiments generally run for 1 million time steps, evaluate every 5000 steps, and report results over 5 random seeds.Evaluations average reward over 10 episodes without exploration noise.
  • Experimental Protocol: Agents train after each episode using one iteration per episode time step and transitions sampled from experiment-specific replay buffers.
  • Batch Construction: The four batch constructions are final-buffer, concurrent-learning, imitation, and imperfect-demonstration datasets.They vary how data is collected and how closely it relates to the learning agent’s policy.
  • BCQ Implementation: BCQ uses a state-conditioned VAE, perturbation model, and two value networks, with corresponding target networks.The perturbation constraint Φ is implemented through a final tanh activation scaled by I · Φ.
  • BCQ Implementation: BCQ samples actions from its generative model, perturbs them, and uses a weighted minimum of paired value estimates as the learning target.Both value networks share the target, with λ = 0.75.
  • VAE Implementation: The VAE reconstructs actions using an encoder-decoder architecture trained with reconstruction and KL-regularization losses.Its latent dimension is twice the action-space dimension, and latent vectors are clipped to [−0.5, 0.5] during inference.
Loading 1812.02900v3…