Source-linked AI summary
Benchmarking Batch Deep Reinforcement Learning Algorithms
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, Joelle Pineau
TL;DR
Batch reinforcement learning must learn from fixed data without further environment interaction, but existing deep methods can suffer from extrapolation error. The paper benchmarks recent off-policy and batch algorithms in a unified Atari setting using one partially trained behavioral policy, and introduces discrete-action BCQ. Under these conditions, many methods underperform online DQN and the behavioral policy, whereas discrete-action BCQ outperforms prior algorithms.
Problem
Existing batch reinforcement learning results are difficult to compare because studies use varied environments and data distributions, while extrapolation error can undermine widely used off-policy methods.
Method
The paper benchmarks recent off-policy and batch reinforcement learning algorithms in Atari under a unified single-behavioral-policy setting and adapts BCQ to discrete actions.
Results
Many algorithms underperform online DQN and the behavioral policy, while discrete-action BCQ outperforms all previous algorithms in this setting.
Takeaways & Limitations
Discrete-action BCQ provides a straightforward strong baseline for batch reinforcement learning in low-diversity Atari data.
Takeaways & Limitations
The experiments treat the behavioral policy as unknown and focus on a single behavioral-policy task with several Atari environments.
Abstract
from arXiv · showhide
Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.
1 Introduction
The paper benchmarks batch and off-policy deep reinforcement learning under a unified Atari setting with data from one partially trained behavioral policy. It finds that standard methods perform poorly, while a discrete-action BCQ variant provides a strong baseline.
- Motivation: Batch reinforcement learning learns from a single fixed data batch without direct environment interaction, which matters when data collection is difficult, costly, or time-consuming.Traditional algorithms lack guarantees on learned-policy quality and scale poorly to high-dimensional tasks.
- Prior findings: Extrapolation error can make widely used off-policy deep reinforcement learning algorithms fail when they evaluate state-action pairs absent from the batch.The resulting erroneous estimates propagate through temporal-difference updates, causing overestimation and poor performance.
- Findings: Under a single behavioral policy, existing batch algorithms fail to outperform the behavioral policy, while standard off-policy deep reinforcement learning algorithms also perform poorly.This contrasts with results obtained from larger, more diverse experience histories.
- Benchmark setting: The benchmark uses 10 million Atari data points generated by a single partially trained behavioral policy to compare several algorithms under one setting.The unified setup addresses difficulties caused by differing environments and data distributions across prior studies.
- Contribution: The discrete-action BCQ variant greatly outperforms prior deep batch reinforcement learning algorithms, including KL-Control.It matches or exceeds the noiseless behavioral policy and online DQN trained with the same amount of data.
- Contribution: The study validates that extrapolation error occurs in challenging discrete-action Atari environments and positions discrete-action BCQ as a strong future baseline.The BCQ variant is described as simple to implement and state of the art in the batch setting.
2 Preliminaries
The preliminaries define batch deep reinforcement learning as learning from fixed data without further environment interaction and explain why neural-network extrapolation creates unstable estimates. They contrast this setting with algorithms that can mitigate such errors through additional interaction.
- Reinforcement Learning: Reinforcement learning models sequential decision making as a Markov decision process with state and action spaces, transitions, rewards, and a discount factor.The agent selects actions through a policy to maximize discounted return.
- Reinforcement Learning: A policy’s value function estimates expected return, and greedy action selection can derive a policy with equal or better performance.The optimal policy is obtained by greedy selection over the optimal value function.
- Deep Reinforcement Learning: DQN approximates the value function with a neural network and minimizes a loss over sampled transition mini-batches using a periodically updated target network.The target network keeps the learning target fixed across multiple updates.
- Batch Deep Reinforcement Learning: In batch reinforcement learning, the dataset is fixed, no further environment interaction occurs, and the behavioral policy is treated as unknown in these experiments.A collection of behavioral policies may be denoted as one behavioral policy for simplicity.
- Batch Deep Reinforcement Learning: Extrapolation error arises when value networks generalize to state-action pairs distant from the batch, producing arbitrarily poor estimates during action selection.Additional environment interaction can mitigate this error when the behavioral policy resembles the target policy.
3 Batch Deep Reinforcement Learning Algorithms
This section surveys batch deep reinforcement learning algorithms, including off-policy and distributional methods, and describes their mechanisms for learning from fixed data while limiting extrapolation error.
- Overview: Batch deep reinforcement learning algorithms are evaluated on fixed data without environment interaction.The section surveys recent batch algorithms, including off-policy methods tested in batch settings.
- QR-DQN: QR-DQN estimates a K-dimensional vector of return quantiles for each action instead of a single value.It computes a pairwise quantile Huber loss and recovers action values through the mean over quantiles.
- REM: REM combines a K-dimensional Q-network output using convex combinations of weights sampled from a (K −1)-simplex.Its policy uses the argmax over the mean output vector.
- BCQ: BCQ constrains Q-learning to actions likely under the behavioral policy, modeled by a state-conditioned generative model.The model represents the batch action distribution and supports selecting likely actions during learning.
- BEAR-QL: BEAR-QL replaces BCQ’s perturbation model with a learned actor while retaining a generative model of the batch distribution.The actor is trained with deterministic policy gradients and variance minimization over an ensemble of Q-networks.
- KL-Control: KL-Control uses a KL-regularized objective with a learned behavioral-policy estimate as its prior.Its Ψ-learning target uses a log-sum-exp over action values and Monte Carlo estimation with dropout masks.
- SPIBB-DQN: SPIBB-DQN matches the behavioral policy when state-action data are insufficient and otherwise uses Q-learning-style policy improvement.The method assumes access to an estimate of the batch state-action distribution.
- SPIBB-DQN: SPIBB-DQN’s deep implementation is limited by the lack of a complete implementation and settings requiring near-exact state-action distribution estimates.Pseudo-count methods are identified as possible ways to estimate the distribution in principle.
4 Discrete Batch-Constrained Deep Q-learning
The paper simplifies BCQ for discrete actions by modeling batch action probabilities and constraining Q-learning to sufficiently likely actions, while preserving a tunable imitation-to-Q-learning range.
- Method: The discrete BCQ variant retains BCQ’s core principles while simplifying the continuous-action algorithm.The simplification is possible because probabilities for every discrete action can be computed directly.
- Constrained policy: The policy selects the highest-valued action among those whose relative generative-model probability exceeds threshold τ.The threshold is scaled by the maximum action probability at the next state.
- Constrained policy: Setting τ = 0 recovers Q-learning, whereas τ = 1 produces an imitator of actions contained in the batch.Thus, the threshold controls the degree of action-space constraint.
- Optimization: The update uses a DQN-style target with constrained action selection and applies Double DQN rather than Clipped Double Q-learning.The algorithm samples minibatches from the batch, updates the Q-network and generative model, and periodically copies target parameters.
- Method: Discrete BCQ trains a generative model with cross-entropy to approximate the behavioral policy’s action probabilities.The learned model is effectively a behavioral cloning network.
5 Experiments
The experiments evaluate off-policy and batch reinforcement learning algorithms on Atari using a fixed dataset collected by a partially trained DQN. Standard off-policy methods generally perform poorly, while discrete BCQ is strongest and stable value estimates align with better performance.
- Experimental Setting: Agents are evaluated on Atari 2600 using a fixed buffer of 10 million transitions collected by a partially trained DQN.The behavioral policy mixes exploratory behavior to generate the offline dataset.
- Results: Standard off-policy algorithms, including DQN, QR-DQN, and REM, perform poorly and generally underperform the noisy behavioral policy.QR-DQN is the strongest of these three but typically remains below the behavioral policy.
- Results: Batch reinforcement learning algorithms perform reasonably well, with BCQ outperforming every other method in all tested games.BCQ often only matches the underlying noise-free online DQN, suggesting robust imitation when exploratory data are limited.
- Results: KL-Control often starts strongly before failing, and its performance drops correspond to negative divergence in value estimates.It performs well in Enduro and Seaquest when the value estimate does not diverge.
- Analysis: The strongest-performing algorithms have stable value estimates, supporting the importance of mitigating extrapolation error in batch reinforcement learning.The experiments connect value-estimate divergence with performance degradation.
6 Conclusion
The paper empirically benchmarks batch reinforcement learning in a single-behavioral-policy Atari setting and introduces a discrete BCQ variant. Existing algorithms underperform online DQN and the behavioral policy, while BCQ outperforms previous algorithms and offers a strong baseline.
- The study analyzes current off-policy and batch reinforcement learning algorithms in a single-behavioral-policy task across several Atari environments.
- Current algorithms underperform online DQN and the behavioral policy in this setting.
- Algorithms that ignore extrapolation error or data distribution perform poorly with low data diversity because of unstable value estimates.
- The introduced discrete BCQ variant outperforms all previous algorithms and is straightforward to implement.The paper proposes BCQ as a strong baseline for future methods.
A.1 Atari Preprocessing.
Atari inputs are consistently preprocessed into stacked image states, with action repetition, sticky actions, clipped rewards, and fixed episode termination rules. These choices standardize the environment across tasks and algorithms.
- Atari frames are grayscaled and resized to 84 by 84 pixels before being supplied to the networks.
- Each network input concatenates four states, producing a (4, 84, 84) tensor representing 16 frames of history.The state is formed from the maximum of the previous two frames.
- The agent selects an action every fourth frame and repeats that action for the next four frames.
- Sticky actions replace the selected action with the previous action with probability p = 0.25.
- Rewards are clipped to [−1, 1], and episodes terminate at game end or after 27k time steps.The time limit corresponds to 108k frames or 30 minutes of real time.
A.2 Architecture and Hyper-parameters.
The experiments use a common convolutional Q-network architecture and largely consistent hyper-parameter settings across algorithms, with separate provisions for generative models and online DQN exploration.
- All networks use the same architecture and hyper-parameters unless stated otherwise.
- Image inputs of size (4, 84, 84) pass through three convolutional layers before a 512-unit hidden fully connected layer outputs one Q-value per action.The convolutional layers use depths 32, 32, and 64, with kernels 8 × 8, 4 × 4, and 2 × 2.
- For methods requiring a generative model, the convolutional network is shared with the Q-network, while a secondary fully connected network outputs action probabilities through softmax.
- Shared hyper-parameters are listed in Table 1 and were chosen to match Rainbow's Dopamine implementation.
- BCQ and KL-Control use a generative-model penalty of 0.01x^2, with dropout additionally applied in KL-Control.
- Online DQN hyper-parameters are listed separately; its exploration uses decayed ϵ-greedy after an initial random-action warmup period.