Source-linked AI summary
Large-Scale Study of Curiosity-Driven Learning
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A. Efros
TL;DR
Hand-designed dense extrinsic rewards are difficult to scale, motivating intrinsic reward functions such as curiosity. The paper studies prediction-error curiosity across 54 environments, compares feature spaces, and finds useful behavior and reward alignment in many settings, while identifying stochastic dynamics as a limitation.
Problem
Designing dense, environment-specific extrinsic rewards is difficult to scale, motivating investigation of intrinsic reward functions for reinforcement learning.
Method
The paper performs a large-scale study of purely curiosity-driven agents, compares feature representations for prediction-error rewards, and evaluates behavior across diverse environments.
Results
Across 54 environments, purely curiosity-trained agents learn useful behaviors, and curiosity often aligns with extrinsic rewards in human-designed games.
Takeaways & Limitations
Curiosity can support learning useful behaviors without extrinsic rewards, making unlabeled environments relevant for studying exploration and possible transfer.
Takeaways & Limitations
Prediction-error curiosity can seek stochastic or unpredictable transitions rather than progress, and feature choices involve trade-offs that affect generalization.
Abstract
from arXiv · showhide
Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the prediction-based rewards in stochastic setups. Game-play videos and code are at https://pathak22.github.io/large-scale-curiosity/
1 Introduction
The paper studies whether agents can learn through curiosity alone, without extrinsic rewards, and examines how feature representations affect this approach across diverse environments.
- Motivation: Curiosity provides an intrinsic reward through prediction error, offering an alternative to difficult-to-engineer dense extrinsic rewards.The approach is intended to guide exploration toward informative or novel transitions.
- Study scope: The study evaluates purely curiosity-driven agents across 54 environments, including video games, physics simulations, and virtual 3D navigation tasks.The dynamics-based curiosity model predicts the consequence of an action from the current state.
- Feature representations: The authors systematically compare feature spaces for curiosity and investigate generalization to unseen environments.The comparison includes random features, pixels, inverse-dynamics features, and variational auto-encoders.
- Limitations: The paper also identifies a limitation of prediction-error rewards: agent-generated stochasticity can produce reward without actual progress.This limitation is demonstrated in a 3D navigation task where the agent controls parts of the environment.
2 Dynamics-based Curiosity-driven Learning
Dynamics-based curiosity rewards agents for prediction error in learned representations of future states. The paper analyzes representation properties and compares pixels, random, variational, and inverse-dynamics features across benchmark environments.
- Curiosity formulation: The exploration reward is based on the error between predicted and actual next-state representations after an action.The method embeds observations with φ and uses a forward dynamics model to predict φ(x_t+1).
- Curiosity formulation: High prediction error favors transitions in less-visited areas or regions with complex dynamics.The paper uses mean-squared error under a fixed-variance Gaussian density as surprisal.
- Feature-space criteria: A useful feature space should be compact, sufficient, and stable for learning forward dynamics.These properties balance tractability, preservation of relevant information, and reduced reward non-stationarity.
- Feature-space comparisons: The study compares pixels, random features, variational auto-encoders, and inverse-dynamics features as representations for curiosity.Each representation trades off dimensionality, information preservation, stability, or sensitivity to irrelevant details.
- Feature-space limitations: Learned feature distributions change during training, while collecting representative pretraining data without internal game state may impair generalization.The paper therefore treats feature-space effectiveness as an empirical question across environments.
- Empirical findings: Random features are a simple and surprisingly strong baseline, while pixel-based curiosity performs poorly across environments.The figure comparison reports that inverse-dynamics features outperform random features in 55% of Atari games.
3 Experiments
The study evaluates purely curiosity-driven learning across diverse environments and finds useful behavior without extrinsic rewards. It also examines feature representations, transfer to novel Mario levels, and combining curiosity with sparse external rewards.
- 54 diverse environments, including 48 Atari games, Super Mario Bros., Roboschool, Two-player Pong, and Unity mazes, were trained without extrinsic rewards.
- Most Atari evaluation curves increased, showing that curiosity-only agents can obtain external game rewards without using them during training.
- Curiosity aligned with many game rewards, but performance could be worse than random when exploration poorly correlated with the extrinsic objective.
- Feature learning methods: Random features performed well across tasks, while raw-pixel dynamics performed poorly and learned features offered a potentially more stable generalization advantage.Random features sometimes outperformed learned features, partly because their frozen representations provide stationary prediction targets.
- Super Mario Bros.: Curiosity-only Mario agents passed several levels, substantially improving over prior exploration results after longer training with PPO.
- Scaling curiosity-driven learning: 2048 parallel environment threads enabled broader Mario exploration, including 11 levels, secret rooms, and bosses, compared with 128 threads.The comparison concerns performance per gradient step rather than sample efficiency.
- Generalization across novel levels: IDF features transferred across both tested Mario level pairs, whereas random features transferred only between the two day-themed levels.Transfer from Level 1-1 to Level 1-3 was weaker because of the day-to-night color-scheme shift.
4 Related Work
Related exploration methods use intrinsic signals based on prediction error, uncertainty, improvement, visitation counts, novelty, or value estimates, while this paper focuses on dynamics-based curiosity.
- Intrinsic-motivation methods reward prediction error, prediction uncertainty, or improvement in a learned forward-dynamics model.
- Count-based exploration smooths state-visitation counts and has achieved strong results with extrinsic rewards, including on Montezuma’s Revenge.
- Novelty search in evolutionary computing rewards events based on their distance from previously observed events rather than solely optimizing fitness.
- Other exploration methods use uncertainty in value estimates or policy perturbations while maximizing a reward function.
- Random features can work well for simpler problems, while learned features appear to generalize better to novel Mario levels.
5 Discussion
Curiosity-driven agents learn useful behaviors across diverse environments and often align with human-designed rewards, but prediction-error rewards can be misled by stochasticity.
- Pure curiosity produces useful behaviors across Atari, Mario, Ant, Robo-school, and two-player Pong without extrinsic rewards.
- In many human-designed games, novelty-seeking appears aligned with extrinsic rewards because designers create task-guiding curricula.
- Stochastic transitions can make prediction-error curiosity seek high-entropy regions even with a perfect dynamics model.
- Adding a noisy TV drastically slows learning, although agents sometimes eventually converge to obtaining the extrinsic reward consistently.
- Random features perform well, whereas learned features appear to generalize better as environments become complex.
- The broader goal is to use unlabeled environments to improve performance on tasks of interest, including possible transfer to labeled environments.
A Implementation Details
The experiments used standardized pixel preprocessing, stacked observations, convolutional architectures, and parallelized rollouts across environments. Training code and environments were released online for reproducibility.
- Reproducibility: Training code and environments were released on the project website.The authors also direct readers to online code and video results.
- Pre-processing: All experiments converted pixel observations to grayscale 84x84 images and used four-frame action repetition for Super Mario Bros. and Atari.The policy and forward dynamics function operated on four historical observations to capture partial observability.
- Pre-processing: The policy and forward-dynamics models received observation stacks [x_t−3, x_t−2, x_t−1, x_t] rather than only the current observation.This design was intended to capture partial observability in the games.
- Architectures: The embedding and policy networks used identical Atari-style convolutional architectures, with 512-dimensional embedding features and no nonlinearity at the selected feature layer.Batch normalization was applied to the embedding network in Unity and Mario generalization experiments to maintain prediction-error scale relative to extrinsic reward.
- Hyper-parameters: Networks used a 0.0001 learning rate, while parallel-environment counts varied by experiment, reaching 2048 in the large-scale Mario study.Rollout length was generally 128, increased to 512 for Unity experiments.
B.1 Atari
Pure curiosity-driven agents were evaluated on Atari using extrinsic returns and episode length, while feature choices and reward combinations were also examined. The results show that exploration can diverge from maximizing game score and that intrinsic–extrinsic combinations were not clearly superior without tuning.
- Atari exploration: Pong’s increasing episode length alongside a returns plateau indicates the agent maximizes ball bounces rather than reward.The episode-length and return measures distinguish exploration behavior from game-score optimization.
- Feature comparisons: Figure 8 compares curiosity-driven Atari agents using Inverse Dynamics and Random features across 48 games.Extrinsic returns are used to evaluate agents trained without extrinsic rewards.
- Reward combinations: Combining intrinsic and extrinsic rewards produced no large difference between settings, likely because the combination required tuning.One intrinsic+extrinsic run on Montezuma’s Revenge explored 10 rooms.
B.2 Mario
Mario scaling experiments examine how batch size affects curiosity-driven exploration and extrinsic returns. Larger batches support more effective exploration and improve both update efficiency and ultimate scores.
- Mario scaling: Figure 9 reports best extrinsic returns for the Mario scaling experiments.The plot is the analogue of the max-extrinsic-return plot shown earlier.
- Mario scaling: Larger batches allow the agent to explore more effectively.The scaling experiment links batch size with exploration effectiveness.
- Mario scaling: Larger batches reach the same performance in fewer parameter updates and achieve better ultimate scores.Both update efficiency and final performance improve in the reported scaling experiments.