Source-linked AI summary
Playing Atari with Deep Reinforcement Learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller
TL;DR
Reinforcement learning needed to learn effective policies directly from high-dimensional sensory inputs without hand-crafted representations. The paper uses a convolutional neural network with Q-learning and experience replay on Atari games, outperforming previous methods on six of seven games and surpassing an expert human on three. Its replay memory remains limited by finite size and uniform sampling.
Problem
Reinforcement learning must learn control policies from high-dimensional sensory inputs despite sparse, noisy, delayed rewards and changing data distributions.
Method
A convolutional neural network learns action values from raw video using a Q-learning variant with stochastic updates and experience replay.
Results
Six of seven games achieved state-of-the-art results, while performance surpassed an expert human player on Breakout, Enduro, and Pong.
Takeaways & Limitations
One fixed deep-learning approach can learn difficult Atari control policies from raw pixels without game-specific architecture, hyperparameters, or input features.
Takeaways & Limitations
The finite replay buffer overwrites recent experiences and uniform sampling gives equal importance to all stored transitions.
Abstract
from arXiv · showhide
We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.
1 Introduction
Deep learning offers a way to learn representations directly from raw sensory inputs, but reinforcement learning must cope with sparse, delayed rewards and changing data distributions. This paper addresses these challenges with a convolutional Q-learning agent and evaluates one fixed approach across Atari games.
- Motivation: Raw sensory-input reinforcement learning has traditionally relied on hand-crafted features and linear value functions or policy representations.Performance therefore depends heavily on feature quality.
- Motivation: Deep learning extracts high-level features from raw sensory data, motivating its application to reinforcement learning with sensory inputs.Prior successes include computer vision and speech recognition.
- Challenges: Reinforcement learning must learn from scalar rewards that may be sparse, noisy, and delayed by thousands of timesteps.This differs substantially from supervised learning, where inputs are directly associated with targets.
- Approach: A convolutional neural network trained with a Q-learning variant learns control policies from raw video in complex reinforcement-learning environments.Stochastic gradient descent updates the network, while experience replay smooths training over past behaviors.
- Evaluation: Across seven Atari games, one architecture and training setup outperformed previous reinforcement-learning algorithms on six games and surpassed an expert human on three.The agent used video input, rewards, terminal signals, and legal actions without game-specific information or hand-designed visual features.
2 Background
The Atari agent interacts with an environment through actions, images, and rewards, while partial observability and delayed feedback make current screens insufficient as complete states. Q-learning addresses this by estimating action values with a neural-network function approximator and stochastic updates.
- Environment and State: The agent receives raw screen images and rewards from an Atari emulator after selecting actions from the legal action set.The emulator’s internal state remains unobserved, and rewards represent changes in game score.
- Environment and State: Because current screens can perceptually alias emulator states, the agent represents states as sequences of actions and observations.Treating each sequence as a distinct state yields a finite Markov decision process for applying standard reinforcement-learning methods.
- Value Functions: The objective is to maximize future rewards discounted by γ, represented through the optimal action-value function Q∗(s, a).Q∗ gives the maximum expected return after observing sequence s and taking action a.
- Value Functions: The Bellman equation iteratively updates action values using immediate reward plus the discounted maximum next-state value.This provides the basis for estimating Q∗ rather than storing a separate value for every sequence.
- Q-Learning: A Q-network approximates the action-value function with weights θ, using targets computed from previous network weights.Unlike supervised-learning targets, these targets depend on the network parameters.
- Q-Learning: Stochastic gradient descent replaces full expectations with samples from the behavior distribution and emulator, producing the familiar Q-learning algorithm.The method is model-free and off-policy, learning a greedy strategy while exploring with an ϵ-greedy behavior distribution.
3 Related Work
Earlier reinforcement-learning successes and deep-function-approximation methods exposed limits involving generalization, scalability, and convergence. The paper builds on this history by combining deep networks with stochastic Q-learning and experience replay for Atari control.
- Earlier Reinforcement Learning: TD-Gammon achieved superhuman backgammon play using model-free reinforcement learning, self-play, and a one-hidden-layer multilayer perceptron.It approximated a value function with a neural network.
- Earlier Reinforcement Learning: Less successful applications to chess, Go, and checkers encouraged the belief that TD-Gammon’s success might depend on backgammon’s stochasticity and smooth value function.This interpretation was linked to easier state-space exploration and value estimation.
- Convergence Challenges: Combining Q-learning with nonlinear function approximators or off-policy learning was shown to cause Q-network divergence.Consequently, much reinforcement-learning work focused on linear approximators with stronger convergence guarantees.
- Comparison: TD-Gammon differed from this work by approximating V(s) rather than Q(s, a) and learning on-policy from self-play games.This distinction clarifies the paper’s use of action-value estimation and off-policy learning.
- Recent Methods: Recent work combined deep learning with reinforcement learning to estimate environments, value functions, or policies, while gradient temporal-difference methods addressed some divergence issues.The cited convergence results applied to evaluating a fixed policy with nonlinear function approximation.
- Comparison: Neural fitted Q-learning optimized similar loss functions with RPROP and batch updates, whereas this paper uses low-cost stochastic gradient updates that scale to large datasets.The comparison emphasizes computational scaling rather than only the learning objective.
- Atari Reinforcement Learning: Earlier Atari work used linear function approximation with generic visual features, later improving results through more features, hashing, and evolutionary architectures.These methods provided context for replacing hand-designed visual representations with a deep network.
4 Deep Reinforcement Learning
The paper combines deep neural networks with Q-learning to learn control policies from sensory input, using experience replay and a fixed architecture across Atari games. Its design addresses correlated data, non-stationary training distributions, and the computational cost of evaluating actions.
- Deep Q-learning: Deep Q-learning stores transitions in a replay memory and applies Q-learning minibatch updates to randomly sampled experiences.The agent then selects actions with an ϵ-greedy policy.
- Deep Q-learning: Each experience can contribute to multiple weight updates, improving data efficiency over standard online Q-learning.
- Experience replay: Randomizing replay samples breaks correlations between consecutive experiences and reduces update variance.
- Experience replay: The replay buffer stores only the last N experience tuples and samples them uniformly, so it neither prioritizes important transitions nor preserves older experiences indefinitely.The paper identifies prioritized sampling as a possible refinement.
- Model architecture: The network processes cropped 84 × 84 inputs derived from raw Atari frames and uses two convolutional layers, a 256-unit fully connected layer, and one output per valid action.The same architecture is used across all seven games; valid-action counts range from 4 to 18.
- Model architecture: Separate output units for each action allow all action values to be computed with a single forward pass.This avoids a separate network evaluation for every action.
5 Experiments
Experiments evaluated one shared Atari agent across seven games, tracking training behavior and comparing performance with learning methods, evolutionary policies, and human play. The results show strong generalization across games, including human-level or better performance in several cases.
- Experimental setup: Seven Atari games used the same network architecture, learning algorithm, and hyperparameters, without game-specific information.The games were Beam Rider, Breakout, Enduro, Pong, Q*bert, Seaquest, and Space Invaders.
- Experimental setup: Training used RMSProp minibatches of 32, an annealed ϵ-greedy policy, 10 million frames, and replay memory containing one million recent frames.ϵ was annealed from 1 to 0.1 over the first million frames and then fixed at 0.1.
- Training and stability: Average total reward was noisy during training, so the experiments also tracked average maximum predicted action-value on fixed states as a more stable metric.Figure 2 evaluated reward with ϵ = 0.05 for 10000 steps and measured predicted values on held-out states.
- Visualizing the value function: The learned value function responded to a Seaquest sequence by rising when an enemy appeared, peaking as a torpedo neared impact, and falling after the enemy disappeared.This visualization demonstrates value changes over a reasonably complex sequence of events.
- Main evaluation: DQN outperformed other learning methods by a substantial margin on all seven games, and exceeded expert human performance on Breakout, Enduro, and Pong.It achieved close to human performance on Beam Rider, while Q*bert, Seaquest, and Space Invaders remained far from human performance.
- Main evaluation: Against HNeat, DQN achieved better average and maximum evaluation performance on every game except Space Invaders.HNeat used deterministic policies and specialized representations, whereas DQN was evaluated with ϵ-greedy control sequences.
6 Conclusion
The paper introduced a deep reinforcement-learning model that learned difficult Atari control policies from raw pixels. It combined a neural network with online Q-learning and experience replay, achieving state-of-the-art results in six of seven games without changing the architecture or hyperparameters.
- Conclusion: The model mastered difficult Atari 2600 control policies using only raw pixels as input.It introduced a deep learning model for reinforcement learning.
- Conclusion: The method combined stochastic minibatch updates with experience replay memory in a variant of online Q-learning.This combination was used to ease training deep networks for reinforcement learning.
- Conclusion: Six of seven games produced state-of-the-art results without adjusting the architecture or hyperparameters.The conclusion presents this as evidence of the approach's performance across the tested games.