Source-linked AI summary
Deep Q-learning from Demonstrations
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, Joel Z. Leibo, Audrunas Gruslys
TL;DR
Deep RL often needs extensive data and can perform poorly during learning, limiting real-world deployment without accurate simulators. DQfD combines demonstration pretraining with TD and supervised losses, then continues learning from environment interaction. Across Atari games, it substantially improves early performance and reaches strong results despite small demonstration datasets.
Problem
Deep RL can require huge amounts of data and exhibit extremely poor performance during learning, making real-world tasks difficult without accurate simulators.
Method
DQfD pretrains on demonstration data using TD, supervised, n-step TD, and regularization losses before continuing with reinforcement learning.
Results
DQfD performs better than PDD DQN during the first million steps on 41 of 42 games, while achieving state-of-the-art results on 11 Atari games.
Takeaways & Limitations
Small demonstration datasets can accelerate learning and support strong performance, including on difficult exploration games.
Takeaways & Limitations
Human demonstrations are difficult to learn from because imitation may not perfectly classify demonstrator actions or capture information unavailable in the agent's state representation.
Abstract
from arXiv · showhide
Deep reinforcement learning (RL) has achieved several high profile successes in difficult decision-making problems. However, these algorithms typically require a huge amount of data before they reach reasonable performance. In fact, their performance during learning can be extremely poor. This may be acceptable for a simulator, but it severely limits the applicability of deep RL to many real-world tasks, where the agent must learn in the real environment. In this paper we study a setting where the agent may access data from previous control of the system. We present an algorithm, Deep Q-learning from Demonstrations (DQfD), that leverages small sets of demonstration data to massively accelerate the learning process even from relatively small amounts of demonstration data and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism. DQfD works by combining temporal difference updates with supervised classification of the demonstrator's actions. We show that DQfD has better initial performance than Prioritized Dueling Double Deep Q-Networks (PDD DQN) as it starts with better scores on the first million steps on 41 of 42 games and on average it takes PDD DQN 83 million steps to catch up to DQfD's performance. DQfD learns to out-perform the best demonstration given in 14 of 42 games. In addition, DQfD leverages human demonstrations to achieve state-of-the-art results for 11 games. Finally, we show that DQfD performs better than three related algorithms for incorporating demonstration data into DQN.
Introduction
Deep RL has achieved notable control successes but often requires extensive data and performs poorly during learning, limiting real-world use. DQfD uses small demonstration datasets to pretrain an agent and accelerate subsequent learning.
- Motivation: Real-world RL remains difficult because agents may perform poorly for millions of steps without accurate simulators.Demonstration data from previous controllers can support good online performance from the beginning of learning.
- Contribution: DQfD leverages even very small amounts of demonstration data to massively accelerate learning.It initially pretrains on demonstrations before continuing with self-generated experience.
- Contribution: DQfD combines temporal-difference and supervised losses to imitate demonstrator actions while learning a self-consistent value function.Prioritized replay automatically controls the mixture of demonstration and self-generated data during learning.
- Results: DQfD outperforms PDD DQN during the first million steps on 41 of 42 games.On average, PDD DQN requires 83 million steps to catch up to DQfD's performance.
Background
The paper formulates the task as an MDP and builds on DQN, double Q-learning, and prioritized replay. These components stabilize value learning and focus updates on important transitions.
- MDP formulation: An MDP consists of states, actions, rewards, transitions, and a discount factor, with policies seeking maximal expected discounted reward.The optimal value function satisfies the Bellman equation and induces a greedy policy.
- DQN: DQN approximates action values with a deep neural network, using a target network and replay buffer to stabilize updates.The target network is periodically copied from the regular network, while replay sampling reduces dependence on current experience correlations.
- Double Q-learning: Double Q-learning uses the current network for next-action selection and the target network for evaluation, reducing upward value bias.This separates the value functions used for action selection and evaluation.
- Prioritized replay: Prioritized replay samples transitions according to their priorities, which are based on TD errors plus a positive constant.Importance-sampling weights correct for the changed sampling distribution, with β annealed from β0 to 1.
Related Work
Related approaches use demonstrations for imitation or replay, but DQfD combines demonstration pretraining, supervised losses, and reinforcement learning to improve beyond the demonstrator. Its design addresses sparse demonstration coverage and value consistency.
- Imitation learning: Imitation methods such as DAGGER require expert feedback during training and do not combine imitation with reinforcement learning.Consequently, the described imitation setup cannot improve beyond the expert as DQfD can.
- Replay-based methods: Human Experience Replay mixes agent and demonstration data but does not pretrain the agent or use a supervised loss.The paper reports higher scores for DQfD over a larger variety of games without requiring full environment access.
- Closest related work: ADET also combines TD and classification losses, but uses cross-entropy, does not pretrain initial performance, and relies on trained-DQN demonstrations.DQfD instead uses a large-margin loss and pretraining before environment interaction.
- DQfD design: DQfD pretrains on demonstrations so its value function can imitate the demonstrator while satisfying the Bellman equation for later TD updates.The supervised loss grounds unseen actions, while TD learning constrains values across consecutive states.
- DQfD design: The large-margin loss pushes the demonstrator's action above other action values, whereas n-step returns propagate expert values to earlier states.The full objective combines double-Q, n-step, supervised, and L2 losses.
- DQfD versus PDD DQN: DQfD differs from PDD DQN through permanent demonstrations, demonstration-only pretraining, supervised and regularization losses, n-step returns, and a demonstration priority bonus.These design choices jointly define its demonstration-integrated training procedure.
Experimental Setup
The experiments evaluate DQfD on 42 Atari games using human demonstrations and compare it with pure reinforcement learning and pure imitation. The setup uses standardized visual inputs, shared network architecture, and small game-specific demonstration datasets.
- Environment: DQfD is evaluated in the Arcade Learning Environment using 84x84 grayscale images, four stacked frames, 18 possible actions, and a discount factor of 0.99.Actions are repeated for four Atari frames, and episodes begin with up to 30 no-op actions.
- Compared algorithms: The study compares full DQfD with human demonstrations, PDD DQN without demonstrations, and supervised imitation without environment interaction.All algorithms are averaged across four trials and use parameters shared across the game set after informal tuning on six games.
- Compared algorithms: PDD DQN omits demonstration data, pretraining, supervised losses, and regularization losses, while all algorithms use a dueling state-advantage convolutional network.The baseline includes n-step returns for comparison with DQfD.
- Compared algorithms: The imitation baseline uses cross-entropy classification and L2 regularization but no TD loss or additional environment interaction.It learns only from the pretraining demonstrations.
- Demonstrations: The evaluation uses a randomly selected subset of 42 Atari games with 5,574 to 75,472 human demonstration transitions per game.Each game was played by a human three to twelve times, with episodes lasting until termination or 20 minutes.
- Reward processing: The agent uses unclipped, log-scaled rewards to make rewards more consistent between human demonstrations and agent training.The transformation is r_agent = sign(r) · log(1 + |r|).
Results
DQfD improves early learning, surpasses demonstrators on selected games, and achieves strong results against prior deep-RL and demonstration-learning methods. Its prioritized replay increasingly samples demonstrations on difficult exploration games, while removing supervised or n-step TD losses degrades performance.
- Game performance: DQfD achieved higher scores than any previously published result on Hero and Pitfall, including a best Pitfall score of 394.0 averaged over 3 million steps.No previous approach achieved positive rewards on Pitfall.
- Game performance: On Road Runner, DQfD exceeded PDD DQN for the first 36 million steps and matched its performance afterward despite only 5,574 human demonstration transitions.The human demonstrations had a maximum score of 20,200.
- Early learning: DQfD outperformed PDD DQN during the first million steps on 41 of 42 games, and PDD DQN needed 83 million steps on average to catch up.PDD DQN never surpassed DQfD in mean scores.
- Comparison with prior deep RL: DQfD achieved better scores than the compared deep-RL algorithms on 11 of 42 games.The comparison used the best 3 million step window averaged over four seeds.
- Learning from demonstrations: DQfD learned better than the best demonstration episode in 14 of 42 games, whereas pure imitation learning was worse than the demonstrator in every game.DQfD outperformed the worst demonstration episode in 29 of 42 games.
- Ablations and related methods: Removing either the supervised loss or the n-step TD loss degraded performance, while the three compared demonstration algorithms performed worse than DQfD.The supervised loss was described as critical to good performance, and the n-step TD loss strongly helped learning from limited demonstrations.
Discussion
DQfD combines demonstration data with reinforcement learning to accelerate early performance and support difficult exploration, but learning from human demonstrations remains challenging.
- Method: DQfD pretrains on demonstrations with 1-step TD, n-step TD, supervised, and regularization losses, then samples demonstration and self-generated data during interaction.Prioritized replay automatically controls their mini-batch ratio.
- Results: 41 of 42 Atari games showed better performance than PDD DQN during the first million steps, while DQN required 82 million steps on average to match DQfD.The comparison concerns initial learning performance.
- Results: DQfD achieved state-of-the-art results on 11 Atari games, including difficult exploration games where demonstrations can substitute for smarter exploration.The cited examples include Montezuma’s Revenge, Pitfall, and Private Eye.
- Results: DQfD achieved these results from 5,574 to 75,472 demonstration transitions per game, despite receiving three orders of magnitude more interaction data for reinforcement learning.The demonstrations could be generated in a few minutes of gameplay.
- Discussion: DQfD outperformed the best demonstration in 14 of 42 games, whereas naive combinations of demonstration and reward signals can produce disastrous results.The paper argues that combining all four pretraining losses is critical for maintaining a coherent representation.
- Limitations: Human demonstrations are difficult to learn from because humans may use unavailable information or play differently from the policy an agent would learn.The paper proposes measuring differences between demonstration and agent data as future work.
Supplementary Material
The supplementary material specifies the shared algorithm parameters and describes the evaluation summaries used for Atari performance and demonstration comparisons.
- Parameters: DQfD used 750,000 pre-training mini-batch updates, unit weights for n-step and supervised losses, and L2 regularization weight 10^-5.The expert margin was 0.8 when the action differed from the demonstrator’s action.
- Parameters: The algorithms used ε-greedy exploration with ε = 0.01, prioritized replay exponent α = 0.4, importance-sampling exponent β_0 = 0.6, and 10-step returns.The target network update period was τ = 10,000.
- Evaluation: Figure 3 reports online rewards for three algorithms across 42 Atari games, averaged over four trials with up to 30 random no-op actions at episode start.Scores are Atari-game scores regardless of the agent’s internal reward representation.
- Evaluation: Table 2 compares best and worst human demonstration scores, trial and transition counts, and each algorithm’s average score in its best 3-million-step window.The averages are taken over four seeds.