Source-linked AI summary
Curiosity-driven Exploration by Self-supervised Prediction
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell
TL;DR
Sparse or absent extrinsic rewards make random exploration ineffective for reaching goals, creating a need for intrinsic exploration signals. The paper uses inverse-dynamics features and forward prediction error to generate curiosity, and evaluates it across sparse-reward, reward-free, and generalization settings. The approach supports exploration and transfers knowledge to new scenarios, while remaining bounded by task-specific policy-learning challenges.
Problem
Sparse extrinsic rewards and difficult-to-construct shaped rewards can make random exploration unlikely to reach pre-specified goals.
Method
The method learns an action-relevant visual feature space with self-supervised inverse dynamics and uses forward prediction error in that space as intrinsic reward.
Results
Across VizDoom and Super Mario Bros, curiosity supports sparse-reward learning, reward-free exploration, and faster exploration of subsequent scenarios using knowledge from earlier experience.
Takeaways & Limitations
Curiosity can drive exploration and support future skill use while ignoring environmental variation that does not affect the agent.
Takeaways & Limitations
In Mario, exploration stops beyond a pit at 38% of the game because crossing requires a specific 15–20-key-press sequence that the policy does not learn.
Abstract
from arXiv · showhide
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward, where curiosity allows for far fewer interactions with the environment to reach the goal; 2) exploration with no extrinsic reward, where curiosity pushes the agent to explore more efficiently; and 3) generalization to unseen scenarios (e.g. new levels of the same game) where the knowledge gained from earlier experience helps the agent explore new places much faster than starting from scratch. Demo video and code available at https://pathak22.github.io/noreward-rl/
1. Introduction
Sparse or absent extrinsic rewards make goal-directed reinforcement learning difficult, motivating curiosity as an intrinsic signal for exploration and future skill acquisition. The paper develops a self-supervised action-relevant feature space for predicting action consequences and evaluates curiosity across sparse-reward, reward-free, and generalization settings.
- Motivation: Sparse rewards can make random exploration unlikely to reach pre-specified goals in environments where shaped rewards cannot be constructed.The agent receives reinforcement only upon reaching the goal state, leaving few useful policy updates.
- Motivation: Curiosity provides intrinsic motivation to explore novel states and learn skills that may support future reward pursuit when extrinsic rewards are sparse.The paper evaluates curiosity in sparse-reward tasks, reward-free exploration, and transfer to new scenarios.
- Challenges: Existing novelty and prediction-error approaches are difficult to construct in high-dimensional visual spaces and must also handle stochastic agent-environment dynamics.Modeling raw images or environmental-state distributions is challenging, especially when observations contain uncontrollable variation.
- Approach: The proposed method predicts action-relevant environmental changes in a self-supervised feature space learned from inverse dynamics, rather than predicting raw pixels.A forward model predicts the next feature representation from the current representation and action, with prediction error serving as intrinsic reward.
- Approach: The learned feature space ignores environmental variation that cannot affect the agent, reducing undesirable curiosity about irrelevant phenomena.The inverse-dynamics objective focuses representation on information useful for predicting the agent’s actions.
- Evaluation: Curiosity is evaluated in VizDoom and Super Mario Bros across sparse extrinsic reward, reward-free exploration, and generalization to unseen scenarios.In sparse-reward VizDoom tasks, curiosity is reported as crucial; without extrinsic rewards, agents explore corridors and make progress through Mario’s Level-1.
2. Curiosity-Driven Exploration
The Intrinsic Curiosity Module combines extrinsic and prediction-error rewards, learning an action-relevant feature space with inverse and forward dynamics models. Its feature-space curiosity avoids pixel prediction and reduces attraction to uncontrollable environmental variation.
- Curiosity-Driven Exploration: The policy maximizes the sum of environment-provided extrinsic reward and ICM-generated intrinsic curiosity reward.The policy samples actions from π, while the reward combines r^e_t and r^i_t.
- Prediction error as curiosity reward: Pixel-space prediction can trap exploration on inherently unpredictable distractors, such as moving leaves, even when those distractors are inconsequential.The paper identifies persistent pixel prediction error as an artificial curiosity trap.
- Prediction error as curiosity reward: The learned feature space represents factors that affect or are affected by the agent while excluding variation unrelated to its actions.This design targets controllable and agent-relevant variation rather than all changes in the observation.
- Self-supervised prediction for exploration: ICM learns features φ(s_t) and φ(s_t+1) by training an inverse model to predict the action taken between consecutive states.The inverse model is trained on tuples collected while the agent interacts with the environment.
- Self-supervised prediction for exploration: A forward model predicts φ(s_t+1) from φ(s_t) and a_t, and its feature-space prediction error supplies the intrinsic reward.The forward model operates on learned features rather than raw pixels.
- Curiosity-Driven Exploration: The formulation uses forward-model prediction error as the policy’s curiosity reward, unlike prior inverse-forward work that used forward error only as a feature-learning regularizer.The policy-learning experiments use A3C, while the curiosity formulation is the paper’s main contribution.
3. Experimental Setup
The experiments evaluate curiosity-driven exploration in VizDoom and Super Mario Bros using visual inputs, multiple environment setups, and comparisons against A3C baselines.
- Environments: The evaluation uses VizDoom 3-D navigation, where the agent chooses among four discrete actions and receives a sparse terminal reward for finding the vest.Episodes end when the vest is found or after 2100 time steps.
- Environment layouts: VizDoom generalization pre-trains on a different map with different textures, while testing uses the DoomMyWayHome-v0 map and lasts up to 2100 steps.An optimal policy takes approximately 350 steps to reach the vest from the farthest room in the sparse-reward map.
- Environments: Super Mario Bros experiments use 14 joystick actions and curiosity-only training across four levels, with later levels reserved for generalization.Long jumps require repeating the same action up to 12 times, creating long-range dependencies.
- Training details: The visual input is grayscale, resized to 42 × 42, and formed by concatenating the current frame with the three previous frames.Training uses action repetition in VizDoom and Mario.
- A3C architecture: The A3C backbone uses four convolutional layers followed by a 256-unit LSTM, with separate value-function and action prediction heads.Each convolution uses 32 filters, 3x3 kernels, stride 2, and ELU nonlinearities.
- Models and baselines: ICM combines inverse and forward models, while ICM-pixels predicts next observations directly in pixel space and serves as a baseline alongside vanilla A3C.The ICM feature vector φ(s_t) has dimensionality 288; Figure 5 compares all three agents across dense, sparse, and very-sparse reward settings.
4. Experiments
Across sparse-reward, no-reward, and novel-scenario settings, curiosity-driven agents solve tasks, explore broadly, and transfer useful exploratory behavior. The experiments also show robustness to uncontrollable visual noise and a limitation when fine-tuning encounters a curiosity blockade.
- Sparse Extrinsic Reward Setting: ICM remains effective under a distractor replacing 40% of visual observations with white noise, whereas ICM-pixels suffers significantly.The authors interpret this as insensitivity to nuisance changes that do not affect the agent.
- No Reward Setting: Curiosity-only agents explore substantially more of VizDoom without extrinsic rewards, traversing corridors, rooms, and sometimes the entire map within 2100 steps.Random exploration often remains stuck in local minima such as against a wall.
- No Reward Setting: Using only curiosity, the Mario agent crosses over 30% of Level-1 and discovers enemy-avoidance behaviors despite receiving no game rewards.The authors suggest that curiosity indirectly supervises behaviors that expose more of the game space.
- No Reward Setting: The reported evaluation presents curiosity-driven learning directly from complex visual pixels without extrinsic rewards as a first demonstration of this kind.The claim covers 3-D navigation and game playing across the evaluated environments.
- Generalization to Novel Scenarios: Policies pretrained with curiosity generalize to new scenarios: Level-1 training transfers to Level-3, fine-tuning helps on Level-2, and curiosity pretraining accelerates VizDoom learning with external rewards.Level-2 fine-tuning outperforms training from scratch, while Level-3 fine-tuning can deteriorate performance after a curiosity blockade.
- Generalization to Novel Scenarios: Fine-tuning a Level-1 exploration policy on Level-3 can degrade performance when progress stops and intrinsic rewards approach zero.The resulting policy slowly degenerates after the agent reaches a curiosity blockade.
5. Related Work
Prior work explores curiosity through surprise, compression progress, information gain, and uncertainty about environment dynamics. Related approaches also use bootstrapping, Thompson sampling, and self-supervised prediction for exploration or data efficiency.
- Curiosity-driven exploration has used surprise and compression progress as intrinsic rewards.
- Exploration methods also include bootstrapping and Thompson sampling through multiple value functions.
- Other approaches measure information gain about the environment or the agent’s belief over its dynamics.
- Concurrent work applies asymmetric self-play and self-supervised prediction tasks to improve reinforcement-learning data efficiency.
6. Discussion
The proposed curiosity mechanism scales to visual inputs and is evaluated across reward-free exploration, generalization, and potential reuse of learned exploratory behavior. Its scope is limited when useful interactions are themselves rare, and policy learning can fail at demanding exploration barriers.
- The method generates curiosity rewards for high-dimensional visual inputs while bypassing pixel prediction and nuisance factors.
- In VizDoom, the agent learns to move through corridors and rooms without environmental rewards, while in Mario it crosses more than 30% of Level-1 reward-free.
- Limitations: A pit requiring 15–20 specific key presses prevents further Mario exploration when the agent fails to execute the sequence.
- Generalization is evaluated by applying a learned policy to a new scenario without further learning or by fine-tuning it there.
- Future research: Learned exploration behavior could serve as a low-level motor primitive for navigation systems.
- Limitations: The approach does not directly extend to settings where opportunities for interaction are rare.