PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploration by Random Network Distillation

Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov

arXiv:1810.12894v1cs.LGcs.AIstat.ML

TL;DR

Sparse rewards make directed exploration necessary, while existing exploration methods can be difficult to scale. The paper introduces random network distillation and flexible intrinsic–extrinsic reward combination, achieving strong performance on hard Atari exploration games, including state-of-the-art results on Montezuma’s Revenge.

  • Problem

    Reinforcement learning struggles with sparse rewards, and many existing directed-exploration methods are difficult to scale across parallel environments.

  • Method

    RND uses prediction error for a fixed randomly initialized network’s output, combined with separate value heads for intrinsic and extrinsic rewards.

  • Results

    The method achieves state-of-the-art performance on Montezuma’s Revenge, Venture, and Gravitar, with the best agent often finding 22 of 24 Montezuma rooms and occasionally passing the first level.

  • Takeaways & Limitations

    The experiments suggest that simple generic exploration methods, applied at scale and treating intrinsic rewards separately, can advance hard-exploration Atari performance.

  • Takeaways & Limitations

    RND handles local exploration but does not reach global exploration requiring coordinated decisions over long time horizons.

Abstract

from arXiv · show

We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level.

1 INTRODUCTION

Deep reinforcement learning struggles when rewards are sparse and difficult to discover, motivating scalable directed exploration. This paper proposes prediction-error-based exploration and reports strong results on hard Atari games, especially Montezuma’s Revenge.

  • Sparse, hard-to-find rewards make random exploration ineffective, while engineering dense rewards for every task is impractical.
  • Many count-, information-gain-, and prediction-gain-based exploration methods are difficult to scale across large numbers of parallel environments.
  • The method is simple, supports high-dimensional observations and any policy-optimization algorithm, and requires one neural-network forward pass per experience batch.
  • The proposed bonus uses prediction error for a fixed randomly initialized network’s output on the current observation, whose target is deterministic.
  • With separate value heads for intrinsic and extrinsic rewards, the best agent often finds 22 of 24 first-level rooms in Montezuma’s Revenge and occasionally passes the level.
  • The same method achieves state-of-the-art performance on Venture and Gravitar, while performance varies across other hard-exploration Atari games.

2 METHOD

The method uses prediction error from a randomly generated, deterministic target to provide an exploration bonus, while separating intrinsic and extrinsic return streams for flexible optimization.

  • Exploration bonuses: Exploration bonuses add an intrinsic reward to sparse environmental rewards to encourage visits to novel states.Count-based methods implement this principle through visitation-dependent bonuses, while prediction-based methods use transition-related prediction error.
  • Random network distillation: RND uses a fixed randomly initialized target network and a predictor network trained on agent-collected observations.The target maps observations to embeddings, and the predictor is trained by gradient descent to match those embeddings.
  • Random network distillation: The predictor’s MSE is expected to be higher on novel states because it has seen fewer similar training examples.An MNIST experiment tests this behavior by varying target-class examples while holding total training data fixed.
  • Prediction-error sources: RND avoids stochasticity-driven prediction error because its target function is deterministic and within the predictor’s model class.This addresses noisy-TV behavior associated with forward-dynamics prediction, while model-improvement approaches can be computationally expensive to scale.
  • Combining return streams: Non-episodic intrinsic returns treat novel states across game-over boundaries as part of one exploration horizon.The paper argues this can reduce excessive risk aversion during exploration, but non-episodic extrinsic returns can enable repeated early-game reward cycles.
  • Combining return streams: Separate value functions can combine reward streams with different temporal properties and provide an additional supervisory signal.This is especially relevant because extrinsic rewards are stationary whereas intrinsic rewards are non-stationary.

3 EXPERIMENTS

Experiments evaluate RND through pure exploration, reward-stream and discount-factor ablations, scaling with parallel environments, recurrence, and comparisons with baseline exploration methods. Non-episodic intrinsic rewards, suitable discounting, and increased training scale support exploration and performance, while dynamics-based exploration can suffer from stochasticity.

  • 3.1 PURE EXPLORATION: Non-episodic RND exploration performed best on both mean episodic return and rooms explored, with one run exploring 21 rooms.Using γI = 0.999, four of five runs achieved a best return of 6,700.
  • 3.2 COMBINING EPISODIC AND NON-EPISODIC RETURNS: Non-episodic intrinsic rewards increased rooms explored for both CNN and RNN policies, although the difference was less dramatic when extrinsic rewards were present.The benefit was less pronounced for CNN experiments, while RNN results were less stable and performed worse for γE = 0.99.
  • 3.2 COMBINING EPISODIC AND NON-EPISODIC RETURNS: Two value heads provided no benefit over one in the episodic setting but were necessary for combining reward streams with different characteristics.All subsequent experiments therefore used two value heads.
  • 3.3 DISCOUNT FACTORS: Increasing γE to 0.999 while holding γI at 0.99 greatly improved performance, whereas increasing γI to 0.999 hurt performance.This contrasts with the pure-exploration results, where increasing γI did not significantly affect performance.
  • 3.4 SCALING UP TRAINING: Larger batches collected from more parallel environments produced higher mean returns after similar numbers of updates and better final performance.The effect saturated earlier for CNN policies than for RNN policies; one 32-environment RNN run visited all 24 rooms and passed the first level once.
  • 3.6 COMPARISON TO BASELINES: Across six hard-exploration Atari games, RND exceeded PPO on Montezuma’s Revenge and Venture, while performance varied across the remaining games.Dynamics-based exploration performed significantly worse than RND on Montezuma’s Revenge, PrivateEye, and Solaris, partly because stochastic transitions sustained prediction error.

4 RELATED WORK

Prior exploration methods include count-based bonuses, prediction errors, uncertainty measures, and diverse-policy approaches. Montezuma’s Revenge remained difficult without demonstrations or privileged state access, while prior pseudo-count methods reached 15 rooms and a best return of 6,600.

  • Count-based exploration: Count-based methods generalize visitation bonuses to large state spaces using pseudo-counts derived from changes in state-density estimates.Pseudo-counts can be positive for unvisited states similar to previously visited states.
  • Prediction-based exploration: Prediction-error exploration methods can suffer from the noisy-TV problem in stochastic or partially observable environments.This limitation motivated exploration based on uncertainty quantification or prediction improvement.
  • Other exploration methods: Alternative exploration approaches include adversarial self-play, empowerment maximization, parameter noise, diverse policies, and value-function ensembles.
  • Montezuma’s Revenge: Without special exploration methods, agents can reliably achieve a mean return of about 2,500 on Montezuma’s Revenge, while random agents escape the first room only rarely.
  • Montezuma’s Revenge: 15 rooms and a best return of 6,600 were achieved on Montezuma’s Revenge by combining DQN with a pseudo-count exploration bonus.Subsequent methods achieved similar performance without exceeding this result.
  • Montezuma’s Revenge: Prior work improved Montezuma’s Revenge using demonstrations or underlying RAM-state access, with demonstrations producing performance comparable to or better than human experts.Methods using RAM-state access remained inferior to average human performance.
  • Random features: Randomly initialized neural-network features had been studied in supervised learning and more recently applied to exploration.
  • Value functions: Vectorized or multiple value heads had been used to represent reward factors or discretized returns, although one cited Bellman backup was not itself vectorized.

5 DISCUSSION

The discussion presents RND as a simple method for directed exploration in sparse-reward Atari games, while identifying limits on long-horizon exploration and reward-balancing strategies. The authors propose treating intrinsic and extrinsic reward streams separately as a useful source of flexibility.

  • Contribution: RND supports directed exploration on several Atari games with very sparse rewards using a relatively simple generic method applied at scale.
  • Reward streams: Separate treatment of intrinsic and extrinsic rewards, such as separate value heads, may benefit exploration methods.
  • Limitations: RND handles local exploration of short-term decisions but does not reach global exploration requiring coordinated decisions over long time horizons.
  • Montezuma’s Revenge: In Montezuma’s Revenge, the agent must preserve keys instead of taking easier immediate rewards to open the final two doors.
  • Montezuma’s Revenge: RND provides too little incentive for this key-saving strategy, so the agent discovers it only rarely.
  • Future work: Solving this and similar high-level exploration problems is identified as an important direction for future work.

A.1 REINFORCEMENT LEARNING ALGORITHM

The reinforcement learning algorithm adds intrinsic and extrinsic rewards to train PPO. PPO is used as the baseline policy-gradient method because it requires little tuning for good performance.

  • Algorithm: The combined reward is defined as r_t = i_t + e_t, allowing an exploration bonus to be used with any reinforcement learning algorithm.
  • Algorithm: The proposed exploration bonus is combined with Proximal Policy Optimization, a policy-gradient method selected for requiring little tuning.

A.2 RND PSEUDO-CODE

The RND pseudo-code collects rollout experience, computes intrinsic prediction-error rewards, and trains PPO together with the predictor network. Intrinsic and extrinsic returns and advantages are calculated separately before policy optimization.

  • Initialization and rollout: The algorithm initializes observation normalization through M random-action steps before collecting optimization rollouts.
  • Initialization and rollout: Each rollout samples actions from the policy, transitions through the environment, and stores states, actions, extrinsic rewards, and intrinsic rewards in an optimization batch.
  • Intrinsic reward: The intrinsic reward is the squared prediction error between the predictor output and the fixed target-network feature, i_t = ∥f̂(s_{t+1}) − f(s_{t+1})∥².
  • Optimization: The algorithm normalizes intrinsic rewards and separately computes intrinsic and extrinsic returns and advantages before combining the advantages.
  • Optimization: PPO parameters are optimized on the rollout batch, while the predictor network is optimized using the distillation loss.

A.3 PREPROCESSING DETAILS

The experiments use standardized environment preprocessing, including sticky actions, with separate preprocessing details documented for policy/value and target/predictor networks.

  • Environment preprocessing: Sticky actions make the environments non-deterministic, preventing memorization of action sequences.The preprocessing follows recommendations from Machado et al. (2017).
  • Network preprocessing: Tables 3 and 4 separately document preprocessing for policy/value networks and target/predictor networks.
  • Environment preprocessing: Table 2 documents environment preprocessing used across the experiments.

A.4 PPO AND RND HYPERPARAMETERS

The paper documents default PPO and RND hyperparameters, with additional implementation details and a scaling adjustment for experiments using more parallel environments.

  • Network settings: Table 4 provides preprocessing details for the target and predictor networks used in the experiments.
  • Hyperparameters: Table 5 lists PPO and RND hyperparameters, while the accompanying code provides complete usage details.
  • Scaling experiments: When scaling from 32 to 128 environments, the predictor’s effective batch size was kept constant by randomly dThe passage is truncated after describing the scaling procedure.

A.5 ARCHITECTURES

The paper evaluates RNN and CNN policy architectures with matched parameter counts, using convolutional encoders modeled on the standard Atari architecture.

  • Policy architectures: The experiments use both RNN and CNN policy architectures with closely matched numbers of parameters.
  • Policy architectures: Both policies use convolutional encoders identical to those in Mnih et al. (2015), while the RNN additionally uses GRU cells for longer contexts.
  • Target and predictor networks: The target and predictor networks also use convolutional encoders identical to the Mnih et al. (2015) architecture.

A.6 ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments compare RND with autoencoder, dynamics-prediction, and PPO baselines across hard-exploration Atari games, with results averaged across seeds.

  • Autoencoder exploration: The autoencoder-based exploration agent performs worse than RND but better than baseline PPO under otherwise matched choices.The comparison uses CNN policies with γI = 0.99 and γE = 0.999.
  • Autoencoder exploration: The autoencoding task can nevertheless be used successfully as an exploration method.
  • Baseline comparisons: RND significantly outperforms PPO on Montezuma’s Revenge, Private Eye, and Venture among six hard-exploration Atari games.
  • Result aggregation: Each figure averages results across experiment seeds, with standard error shown as the shaded region around each mean curve.
Loading 1810.12894v1…