Episodic Curiosity through Reachability

Nikolay Savinov; Anton Raichuk; Raphaël Marinier; Damien Vincent; Marc Pollefeys; Timothy Lillicrap; Sylvain Gelly

Source-linked AI summary

Episodic Curiosity through Reachability

Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly

arXiv:1810.02274v5Oct 4, 2018cs.LGcs.AIcs.CVcs.ROstat.ML

TL;DR

Sparse rewards and surprise-based exploration can make reinforcement learning difficult, including by attracting agents to unpredictable but unproductive behavior. The paper proposes episodic curiosity, which compares current observations with episodic memory using learned reachability and adds novelty bonuses to task rewards. Across visually rich 3D environments, it outperforms ICM on reported navigation tasks and enables first-person-view curiosity locomotion in MuJoCo.

Problem
Sparse rewards hinder standard reinforcement learning, while surprise-based curiosity can produce unproductive “couch-potato” behavior; how to form a useful novelty bonus remains open.
Method
Episodic curiosity compares the current observation with observations in episodic memory using a learned siamese reachability network, then adds a novelty bonus to the task reward.
Results
The method outperforms ICM in visually rich 3D environments from VizDoom and DMLab, while a MuJoCo ant learns locomotion from first-person-view curiosity alone.
Takeaways & Limitations
Reachability-based episodic curiosity avoids the reported couch-potato issue and supports exploration and locomotion across the evaluated 3D tasks.
Takeaways & Limitations
The method adds computational overhead: PPO + EC is 1.84x slower than PPO and its reachability network has 13M trainable variables.

Abstract

from arXiv · show

Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only.

1 INTRODUCTION

The paper introduces episodic curiosity, which rewards observations that are difficult to reach from an episode's explored memory. It uses learned reachability comparisons to form dense rewards while avoiding surprise-driven “couch-potato” exploration.

Motivation: Sparse-reward tasks challenge standard reinforcement learning, motivating reward bonuses for novel observations that are summed with task rewards.The paper frames bonus construction as an open question and highlights surprise-based exploration's failure mode in which agents pursue unpredictable but unproductive actions.
Reachability-based novelty: Episodic curiosity defines novelty by the number of environment steps required to reach a current observation from observations stored in episodic memory.Observations reachable within a threshold are treated as non-novel, while observations requiring more steps receive curiosity reward.
Bonus computation: At each step, the agent compares its current embedding with episodic-memory embeddings, rewards novel observations, stores them, and clears memory at episode end.The episodic curiosity module produces a bonus that is combined with the task reward to create a dense augmented reward.
Evaluation: The method is evaluated in visually rich 3D environments from VizDoom, DMLab, and MuJoCo against PPO and curiosity baselines including ICM.The experiments cover maze goal reaching, key-door navigation, and first-person-view ant locomotion.
Reachability network: A reachability network combines an embedding network and comparator in a siamese architecture trained to predict within-k-step reachability between observations.The method learns from temporally close positive pairs and temporally distant negative pairs in observation sequences.

3 EXPERIMENTAL SETUP

The experiments compare episodic curiosity with PPO, ICM, and a privileged Grid Oracle across visually rich 3D environments. Evaluation uses VizDoom, DMLab, and MuJoCo, with procedural generalization and validation-based tuning where available.

Environments: Experiments span visually rich 3D environments from VizDoom, DMLab, and MuJoCo.VizDoom uses 84 × 84 grayscale images, while DMLab uses 84 × 84 RGB images and discrete navigational actions.
Baselines: PPO is applied to the task reward plus each curiosity method’s bonus, providing the common reinforcement-learning backbone.The compared methods include PPO alone, PPO + ICM, PPO + Grid Oracle, and PPO + EC.
Baselines: ICM is the principal curiosity baseline, while Grid Oracle rewards exploration using privileged agent coordinates unavailable to the other methods.Grid Oracle discretizes the world into 2D cells and rewards visiting more cells during each episode.
Tasks: The maze suite includes static VizDoom layouts and randomized DMLab layouts, including versions with doors.Figure 5 illustrates the three maze types used in the experiments.
Tuning: DMLab hyperparameters are tuned on a validation set and reused across environments, whereas VizDoom tuning uses the same environment because proper splits are unavailable.The DMLab validation set is disjoint from training and testing; VizDoom lacks procedurally generated splits.

4 EXPERIMENTS

The experiments evaluate goal reaching, reward-free exploration, and dense-reward robustness across static and procedurally generated mazes. Episodic curiosity matches ICM on final VizDoom performance while converging at least 2 times faster, outperforms ICM across DMLab maze settings, and avoids significant deterioration on dense-reward tasks.

4.1 STATIC MAZE GOAL REACHING: The VizDoom MyWayHome task provides only +1 reward at goal completion, with zero reward otherwise, making Sparse and Very Sparse starts difficult for standard RL.The agent has 525 4-repeated steps to reach the goal; starting distance varies across subtasks.
4.1 STATIC MAZE GOAL REACHING: At least 2 times faster convergence was achieved by the proposed method than ICM in VizDoom, while both reached 100% success in all three subtasks.The comparison uses shifted curves to account for environment interactions used to train the R-network.
4.2 PROCEDURALLY GENERATED RANDOM MAZE GOAL REACHING: The proposed method outperformed ICM in all three DMLab environments under the same 20M 4-repeated-step interaction budget.Its advantage over PPO and ICM was clearer in Very Sparse and Sparse + Doors than in Sparse.
4.2 PROCEDURALLY GENERATED RANDOM MAZE GOAL REACHING: ICM learned firing-based or wall-adjacent behavior instead of robust maze exploration, whereas reachability-based novelty treats firing as only one step away and therefore not novel.The same failure pattern is associated with prediction-error curiosity being driven by easily produced visual changes.
4.3 NO REWARD/AREA COVERAGE: In reward-free exploration, ICM quality first peaks and then degrades, while the proposed method and Grid Oracle both work.The authors report that ICM’s prediction-error bonus disagrees sharply with area coverage and raises concerns for extremely sparse tasks.
4.4 DENSE REWARD TASKS: The proposed method did not significantly deteriorate plain PPO performance on the two dense-reward DMLab tasks.The same bonus weight used in earlier DMLab tasks was retained for this evaluation.

5 DISCUSSION

The method connects curiosity with episodic memory and temporal-distance prediction, distinguishing reachability-based novelty from prior exploration approaches. Its closest prior methods differ in whether they count visits, generate goals, repeat successful strategies, or predict temporal distance for other tasks.

Curiosity methods commonly use prediction error, state counts, or goal generation to drive exploration.
Count-based curiosity: Count-based methods favor rarely visited observations, but extending them from discrete to continuous visual spaces is non-trivial.Density models and hashing provide notable continuous-space extensions, with experiments historically focused on Atari or limited 3D evaluation.
Goal generation: Goal-generation methods seek goals that are reachable but not too easy, using reachability boundaries or expected learning progress.
Episodic memory: Episodic-memory approaches learn memory functioning for successful-task strategies, unlike this work's focus on exploration without task rewards.
Temporal distance prediction: Temporal-distance prediction has been used for embeddings, threshold classifiers, density estimation, and navigation to provided goals rather than this work's exploration task.

6 CONCLUSION

The paper proposes curiosity based on episodic memory and reachability. It reports overcoming couch-potato issues, outperforming ICM in visually rich 3D environments, and enabling first-person-view curiosity locomotion in MuJoCo.

The proposed curiosity model combines episodic memory with reachability to define novelty.
The method outperforms the previous curiosity state-of-the-art ICM in visually rich 3D VizDoom and DMLab environments.
A MuJoCo ant learns locomotion purely from first-person-view curiosity.
Future work would use reachability-retrieved memory content to guide test-time exploration and investigate few-shot exploration learning.

SUPPLEMENTARY MATERIAL

The supplementary material documents MuJoCo experiments, reachability-network training, hyperparameters, and transfer experiments testing R-network generalization across environments.

The supplement first describes MuJoCo locomotion experiments and then provides training details for the R-network.
It lists hyperparameters and hyperparameter-search details for all methods.
Its experimental results examine whether R-networks generalize and transfer across DMLab levels and individual environments.

S1 MuJoCo ANT LOCOMOTION OUT OF FIRST-PERSON-VIEW CURIOSITY

The MuJoCo supplement evaluates first-person-view curiosity for ant locomotion on textured randomized floors, comparing the proposed method with PPO-based baselines. Without task reward, the method moves around the environment; under Escape Circle, it exceeds the best baseline by a factor of 10.

Environment: The ant operates on a 400 × 400 tiled floor with 4 × 4 tiles, 190 possible textures, random initialization, and 1000-step episodes.
Observation space: Curiosity rewards use only the ant's first-person camera, while the policy receives standard Ant-v2 body features.
Baselines and solver: The experiment uses continuous Ant-v2 actions and PPO, with baselines for task reward alone and task reward plus constant reward 1.
No reward: After 10M training steps with no task reward, the proposed method outperforms the baselines and moves around the environment, unlike random PPO and PPO+1.Random PPO dies quickly, while PPO+1 survives longer but does not move much.
Escape Circle: 10×: in Escape Circle, the proposed method significantly outperforms the best baseline after 10M training steps.The task gives zero reward inside radius 10 and one-time rewards when the ant crosses successive concentric circles.
Comparison: The setup differs from concurrent pixel-curiosity work reporting slow ant motion and from approaches using state features rather than pixels.
Evaluation: Table S1 averages results over 30 random seeds for No reward and 10 random seeds for Escape Circle, without seed tuning.

S2 REACHABILITY NETWORK TRAINING DETAILS

The reachability network is trained on observation pairs using minibatch optimization, with separate training schedules and shared architectural choices across environments.

Training uses minibatches of 64 observation pairs matched within episodes.The R-network is trained for 50K minibatch iterations in VizDoom and 200K in DMLab.
The R-network uses Adam with learning rate 10^-4 and a siamese architecture built from two ResNet-18 branches with 512 outputs.
Online training collects experience and retrains the R-network every 720K four-repeated environment steps.Each collection is used for 10 training epochs, with data shuffled before every epoch.

S3 HYPERPARAMETERS

The supplementary experiments specify environment-specific hyperparameters and examine whether reachability models transfer across DMLab tasks. Universal models slightly reduce performance relative to specialized models but remain better than plain PPO, while transfer works across similar environments.

Hyperparameters are reported separately for VizDoom, DMLab, and the two MuJoCo Ant environments.DMLab hyperparameters are tuned on the Sparse environment for all methods.
The experiments evaluate generalization across available levels and transfer between different tasks.
A universal R-network slightly hurts performance relative to a task-specialized network but still improves reward over plain PPO.The universal R-network is trained on 10M environment interactions split equally across 30 DMLab-30 tasks.
R-network transfer provides reasonable performance between similar enough environments, with unsatisfactory performance in one transfer case from Dense 2.

S5 STABILITY/ABLATION STUDY

The stability and ablation studies examine threshold, memory, training-data, and architecture choices. Performance is relatively robust to several settings, while the comparator is essential and unsharing siamese branches enables a simpler comparator without quality loss.

EC performance is reasonably robust when the positive-example threshold varies from 2 to 10 actions.Each action is repeated four times in these experiments.
Memory size has little impact on EC performance in the No Reward and Very Sparse tasks.
The R-network training and policy-training stages jointly determine EC sample complexity, especially when reachability does not generalize across environments.In the worst case, the R-network must be trained separately for each environment.
A random embedding remains better than plain PPO, but removing the comparator drops quality below plain PPO.The ablation isolates the embedding and comparator components of the R-network.
The comparator is essential for the current architecture, whereas training the embedding is desirable but not necessary.
Unsharing the siamese branches restores simple-comparator validation accuracy from 78% to 93%.The complex-comparator validation accuracy is also around 93%.

S6 RANDOMIZED ENVIRONMENTS

The randomized-environment experiments test curiosity under stochastic visual distractions that make future prediction difficult. EC remains reasonably robust to these stochasticity sources and outperforms the baselines in all tested settings.

The experiments add randomized TVs to DMLab Sparse and Very Sparse environments to introduce strong environment stochasticity.Random images occupy the lower-right quadrant of the agent’s first-person view.
The randomized settings include Image Action, Noise, and Noise Action variants.Image Action changes among k animal images, Noise changes every step independently of actions, and Noise Action changes only after a special action.
At 20M four-repeated environment steps, stochasticity deteriorates performance for almost all methods.
EC remains reasonably robust to all tested stochasticity sources and outperforms the baselines in every setting.Videos indicate that EC continues to explore the maze reasonably well.

S7 COMPUTATIONAL CONSIDERATIONS

The method’s reachability computations are parallelized through mini-batching, while its reported comparisons use repeated runs and adjusted training curves. The implementation uses modest memory for stored embeddings but adds substantial parameters and runtime overhead.

Reachability queries are the algorithm’s most computationally intensive component and are computed in parallel via mini-batching.
The method stores 200 embeddings using 400 KB of memory rather than storing observations.
PPO + EC is 1.84x slower than PPO, compared with 1.09x for PPO + ICM.
The R-network adds 13M trainable variables, versus 1.7M for PPO alone and 2M for PPO + ICM.
Figure S2 shifts the method’s curves by R-network training steps to make method comparisons fair, showing five randomly selected runs from 30 repetitions.