Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Bradly C. Stadie; Sergey Levine; Pieter Abbeel

Source-linked AI summary

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Bradly C. Stadie, Sergey Levine, Pieter Abbeel

arXiv:1507.00814v3Jul 3, 2015cs.AIcs.LGstat.ML

TL;DR

The paper tackles the difficulty of efficient exploration in complex RL domains, where enumeration-based Bayesian and PAC-MDP methods do not scale. It learns representations and dynamics concurrently with policy learning, using prediction error for exploration bonuses. On challenging Atari games, these bonuses provide the most consistent improvement, while stochastic dynamics remain a limitation of the approach.

Problem
Bayesian and PAC-MDP exploration methods offer formal guarantees but are often impractical in higher-dimensional tasks because they require enumerable state-action representations.
Method
The method concurrently learns a state representation and dynamics model, then uses model prediction error to assign exploration bonuses from observations and actions.
Results
Exploration bonuses most consistently achieve the largest improvement across challenging Atari games, outperforming the other evaluated methods on 7 of 14 games by AUC-100.
Takeaways & Limitations
Learned model-based bonuses provide a scalable exploration approach for RL tasks with complex observations and can accelerate learning on challenging Atari games.
Takeaways & Limitations
The misprediction-error bonus assumes prediction errors reflect model inaccuracies, an assumption violated by stochastic dynamics.

Abstract

from arXiv · show

Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space. Hence, exploration in complex domains is often performed with simple epsilon-greedy methods. In this paper, we consider the challenging Atari games domain, which requires processing raw pixel inputs and delayed rewards. We evaluate several more sophisticated exploration strategies, including Thompson sampling and Boltzman exploration, and propose a new exploration method based on assigning exploration bonuses from a concurrently learned model of the system dynamics. By parameterizing our learned model with a neural network, we are able to develop a scalable and efficient approach to exploration bonuses that can be applied to tasks with complex, high-dimensional state spaces. In the Atari domain, our method provides the most consistent improvement across a range of games that pose a major challenge for prior methods. In addition to raw game-scores, we also develop an AUC-100 metric for the Atari Learning domain to evaluate the impact of exploration on this benchmark.

1 INTRODUCTION

The paper addresses scalable exploration for complex, high-dimensional RL tasks, where enumeration-based bonuses are impractical. It evaluates alternative strategies and proposes learned-dynamics bonuses that consistently improve exploration in Atari games.

Motivation: Bayesian and PAC-MDP exploration bonuses provide formal guarantees but are impractical for large-scale tasks requiring enumerable state-action representations.These methods derive novelty from state-action visitation counts.
Approach: The paper evaluates exploration strategies designed to scale to complex tasks with high-dimensional inputs.
Approach: The proposed method learns state representations and system dynamics concurrently with the policy, using model misprediction error to assign novelty bonuses.Novel states are expected to disagree more strongly with the learned dynamics model.
Findings: Boltzman exploration and Thompson sampling improve on naive epsilon-greedy exploration, while learned model-based bonuses produce the biggest and most consistent improvement.
Evaluation: The study evaluates these methods on Atari games using raw pixel observations and delayed rewards.

2 PRELIMINARIES

The preliminaries frame exploration as optimism under uncertainty: augmenting rewards encourages agents to visit novel state-action pairs. Existing visitation-based bonuses offer guarantees but require concise representations, motivating function approximation and representation learning.

Online reinforcement learning: Online RL updates a policy from sequential state, action, next-state, and reward tuples while selecting subsequent actions to maximize long-term reward.
Exploration: Exploration prevents agents from relying only on previously successful actions, which can quickly lead to a local optimum.
Exploration bonuses: Optimism under uncertainty augments rewards with bonuses for visiting infrequently seen state-action pairs.
Exploration bonuses: The novelty function N(s, a) captures the novelty of a state-action pair, with prior methods often basing it on visitation frequency.
Representation requirements: Visitation-frequency methods require concise, often discrete state-action representations, so the paper turns to function approximation and representation learning.

3 MODEL LEARNING FOR EXPLORATION BONUSES

The paper assigns exploration bonuses from prediction errors in a learned representation of state dynamics, avoiding visitation-count tables and supporting online RL in large problems. Novelty decreases as the dynamics model becomes more accurate, while poorly predicted transitions receive higher bonuses.

Model-based novelty: The method encodes states with σ and trains a dynamics predictor Mφ to predict the encoded next state from the current encoded state and action.Mφ is parameterized by φ and maps σ(S)×A to σ(S).
Bonus construction: The algorithm adds a normalized, decaying novelty bonus to the environmental reward and stores the augmented transition for policy updates.The decay constant satisfies C > 0; the memory bank Ω supports subsequent policy and model updates.
Model-based novelty: Prediction error measures how poorly the model understands a state-action pair and is converted into a novelty signal for exploration.Higher prediction error indicates that more knowledge about that region of the dynamics is needed.
Scope and compatibility: The approach is modular, compatible with Q-learning and actor-critic methods, and scales where Bayesian exploration approaches become impractical.It supports any suitable state representation, dynamics model, and online RL method using observation-action-reward tuples.
Online algorithm: At each step, the agent encodes consecutive observations, computes Euclidean prediction error, updates the policy, and periodically updates the dynamics model.The implementation updates Mφ once per epoch, corresponding to 50,000 observations.

4 DEEP LEARNING ARCHITECTURES

For Atari images, the paper learns a lower-dimensional state representation with an autoencoder and predicts encoded frames with a simpler neural dynamics model. The authors find raw-frame prediction unsuitable, while the learned encoding captures useful state features without overfitting.

Representation choice: Directly predicting Atari pixel intensities produced nearly equal exploration bonuses at most time steps and exhibited extremely poor behavior.This motivates predicting in a learned representation rather than directly in pixel space.
Autoencoders: An eight-hidden-layer autoencoder compresses Atari frames through a 128-unit bottleneck and reconstructs them using expanding decoder layers.Training used 250,000 images and testing used 25,000 images.
Autoencoder training: Static and dynamic autoencoder training differ in whether data are collected by a random agent offline or gathered and periodically retrained during policy learning.Dynamic training begins with epsilon-greedy behavior and retrains after five epochs and thereafter in parallel.
Autoencoder evaluation: Both autoencoder training procedures achieved a small but non-trivial test residual, suggesting learned state features while avoiding overfitting.The result held regardless of which training technique was used.
Autoencoders: The sixth autoencoder layer is used as σ because it produced better model-learning results than the bottleneck layer in the authors’ experiments.The sixth-layer representation was selected after comparisons on Seaquest iterations.
Dynamics model: A two-layer neural network predicts the encoded next frame from the encoded current state and action using Euclidean regression loss.The model initially behaves close to an identity function before its residuals become more differentiated.

5 RELATED WORK

Prior exploration methods provide guarantees or extend to larger spaces, but many rely on discrete transition structures or a suitable state-space metric. The paper contrasts these approaches with Thompson sampling and Boltzman exploration, which control Q values rather than model-based novelty.

Guaranteed exploration: R-Max and E3 achieve exploration that scales polynomially with the number of state-space parameters, but become intractable as state spaces grow.Earlier model-based methods often operate directly on discrete MDP transition matrices.
Metric-based methods: C-PACE and metric-E3 extend more readily to large state spaces but require a good metric whose assumptions may be difficult to satisfy automatically.The paper relates this metric-learning issue to its use of an autoencoder for representation learning.
Q-value exploration: Thompson sampling with network dropout and Boltzman exploration adjust action selection through Q values rather than model-based exploration bonuses.Boltzman exploration assigns positive probability to actions according to expected utility and a temperature parameter.

6 EXPERIMENTAL RESULTS

Across 14 challenging Atari games, the exploration-bonus methods learned faster than epsilon-greedy and most consistently achieved strong results. AUC-100 evaluates learning speed, while examples show bonuses encouraging agents to investigate poorly modeled or repetitive behavior.

Evaluation: The evaluation compares DQN exploration strategies on 14 challenging Arcade Learning Environment games using raw screen images.The study compares epsilon-greedy DQN with Static AE, Dynamic AE, Boltzman exploration, and Thompson sampling.
Evaluation: AUC-100 measures the normalized area under each game-score learning curve over 100 epochs, emphasizing learning rate rather than only final score.The metric uses the trapezoid-rule area and avoids requiring 1000 epochs of play.
Game-specific behavior: In Bowling, bonuses moved the policy beyond a repeated six-pin pattern because the dynamics learner predicted that outcome and encouraged alternative release points.The bonus therefore targeted behavior that had become familiar to the learned dynamics model.
Game-specific behavior: In Frostbite and Seaquest, bonuses encouraged visits to poorly understood dynamics, supporting progress and eventual learning of the correct resurfacing timing.Frostbite required reaching increasingly complex platform configurations; Seaquest involved penalties from resurfacing too early.
Failure case: In Q*bert, bonuses lowered the score because a background-color change caused near-equal bonuses across visited states.The dynamics predictor could not quickly adapt to the dramatic environmental change.
Learning dynamics: Exploration bonuses learned significantly faster than epsilon-greedy and often continued learning after epsilon-greedy had converged.Learning curves were run for 100 epochs, with testing after each policy update and averages over three trials.
Results: Exploration bonuses outperformed the other three methods on 7 of 14 games in AUC-100, providing the fastest and most consistently strong learning.Boltzman exploration and Thompson sampling achieved the best results on three games, while bonuses were strongest overall by AUC-100.

7 CONCLUSION

The paper presents learned-representation dynamics bonuses as a scalable exploration method for high-dimensional reinforcement learning and reports consistent gains across challenging Atari games. It also identifies stochastic dynamics as a limitation because prediction error may conflate randomness with model uncertainty.

Conclusion: The proposed method concurrently learns a dynamics model that predicts a learned state representation and converts prediction error into an exploration bonus.The bonus is added to reward to encourage visits to novel states.
Conclusion: Across challenging Atari games with high-dimensional observations, exploration bonuses most consistently produced the largest improvement, especially where human players outperform prior learning methods.The method is intended to remain scalable and efficient in large reinforcement-learning problems.
Limitations: The misprediction-error metric assumes that every state misprediction reflects model inaccuracy, an assumption violated by stochastic dynamics.Future extensions require distinguishing stochastic dynamics from uncertainty in the learned model.

8 APPENDIX

The appendix examines encoding-layer selection, learned dynamics-model residuals, and AUC-100 as an alternative way to assess learning progress.

8.1 ON AUTO ENCODER LAYER SELECTION: The sixth autoencoder layer was used as the encoded state space instead of the bottleneck layer after better Seaquest results over 20 iterations.The predictive model was trained on next encoded frames rather than raw pixel intensities.
8.2 ON THE QUALITY OF THE LEARNED MODEL DYNAMICS: Mean normalized residuals continually decreased over time across the games and trials used in the experiments.After 100 epochs, the mean normalized residual was approximately half of the maximum reported value.
8.2 ON THE QUALITY OF THE LEARNED MODEL DYNAMICS: Figure 4 reports normalized dynamics-model prediction residuals across all game trials, with each model retrained from scratch for every trial.The plotted quantity is the normalized residual across trials and games.
8.3 RAW AUC-100 SCORES: AUC-100 compares the area under a 100-epoch game-score learning curve with a rectangle based on 100 and the maximum DQN score.The integral uses the trapezoid rule, and the metric is proposed as an alternative to raw game-score because it captures learning rate without 1000 epochs of play.