Source-linked AI summary
The NetHack Learning Environment
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, Tim Rocktäschel
TL;DR
RL environments often lack both high complexity and fast simulation, limiting evaluation of exploration and generalization. The paper introduces NLE, a procedurally generated NetHack environment with symbolic observations, task interfaces, and IMPALA/RND baselines. The baselines learn diverse early-game policies, while the environment supports reproducible evaluation on unseen seeds.
Problem
Existing RL environments are rarely both sufficiently complex and fast, leaving a need for testbeds that challenge exploration, planning, skill acquisition, memory, and transfer.
Method
The paper builds NLE as a Gym interface around NetHack, exposing its procedurally generated stochastic dynamics, symbolic observations, configurable tasks, and unseen-seed evaluation protocol.
Results
NLE baselines using IMPALA and Random Network Distillation learn diverse early-game policies, including staircase success rates of 77.26% for monk and 80.42% for wizard.
Takeaways & Limitations
NLE provides a fast, complex environment for advancing RL research and analyzing learned agent behavior.
Abstract
from arXiv · showhide
Progress in Reinforcement Learning (RL) algorithms goes hand-in-hand with the development of challenging environments that test the limits of current methods. While existing RL environments are either sufficiently complex or based on fast simulation, they are rarely both. Here, we present the NetHack Learning Environment (NLE), a scalable, procedurally generated, stochastic, rich, and challenging environment for RL research based on the popular single-player terminal-based roguelike game, NetHack. We argue that NetHack is sufficiently complex to drive long-term research on problems such as exploration, planning, skill acquisition, and language-conditioned RL, while dramatically reducing the computational resources required to gather a large amount of experience. We compare NLE and its task suite to existing alternatives, and discuss why it is an ideal medium for testing the robustness and systematic generalization of RL agents. We demonstrate empirical success for early stages of the game using a distributed Deep RL baseline and Random Network Distillation exploration, alongside qualitative analysis of various agents trained in the environment. NLE is open source at https://github.com/facebookresearch/nle.
1 Introduction
Existing RL environments often trade off complexity against simulation speed, limiting tests of exploration, planning, skill acquisition, memory, and transfer. NLE addresses this gap with a fast, procedurally generated NetHack environment, tasks, and baseline agents that learn diverse early-game policies.
- Motivation: Current RL environments are rarely both sufficiently complex and fast enough to support millions or billions of interactions.Procedurally generated environments improve variation, but existing alternatives are described as either costly to run or limited in complexity.
- Contribution: NLE is a procedurally generated Gym environment built around NetHack that balances computational speed with rich, stochastic game dynamics.NetHack includes hundreds of enemy and object types and a defined dungeon goal.
- Contribution: The paper releases an initial task suite and shows that additional tasks can be added easily within the environment.The task suite supports tractable subgoals while exposing diverse behaviors.
- Contribution: IMPALA and Random Network Distillation baselines produce agents that learn diverse policies during the early stages of NetHack.The contribution summary identifies these models and their early-game learning behavior as core results.
2 NetHack: a Frontier for Reinforcement Learning Research
NetHack combines procedural generation, stochastic dynamics, partial observability, symbolic observations, long horizons, and many entities into a demanding RL research setting. NLE exposes these properties through a lightweight Gym interface, configurable tasks, and an evaluation protocol based on repeated ascensions on unseen seeds.
- NetHack: a Frontier for Reinforcement Learning Research: NetHack is a procedurally generated roguelike centered on exploration, resource management, and continuous discovery, with episodes that are extremely difficult to complete consistently.The game requires descending through over 50 levels, retrieving the Amulet of Yendor, and escaping through final levels.
- NetHack: a Frontier for Reinforcement Learning Research: Procedural generation and stochastic dynamics make exact state repetition unlikely, creating a fundamental challenge for exploration and systematic generalization.The location and structure of important areas vary across episodes.
- NetHack: a Frontier for Reinforcement Learning Research: NetHack’s symbolic observations form an enormous combinatorial space across hundreds of symbols, roles, races, and randomized inventories.The paper argues that state similarity and exploration may require novel methods or entirely new frameworks.
- The NetHack Learning Environment: NLE provides a turn-based Gym interface with multimodal symbolic observations, 93 actions, configurable tasks, and tools for replay analysis.Observations include dungeon tensors, bottom-line statistics, messages, and inventory representations.
- Tasks: NLE’s task suite covers navigation, pet accompaniment, eating, gold collection, score maximization, scouting, and locating the oracle.Tasks are designed to be defined with only a few lines of code and produce diverse trained behaviors.
- Evaluation Protocol: The evaluation protocol defines solving as ten consecutive ascensions to demigodhood on unseen seeds with randomized role, race, alignment, and gender.Held-out seeds are intended to test systematic generalization rather than memorization.
3 Experiments and Results
Experiments evaluate distributed Deep RL and RND across NLE tasks, showing useful early-game learning alongside clear exploration and generalization challenges. Performance varies substantially by task, character, and training setup.
- Staircase: Agents achieved staircase success rates of 77.26% for monk, 50.42% for tourist, 74.62% for valkyrie, and 80.42% for wizard.RND improved monk, tourist, and valkyrie success by 13.58, 6.52, and 16.34 percentage points, respectively, but reduced wizard success by 12.96 points.
- Pet: Pet success rates were lower than staircase rates, reaching 62.02% for monk, 25.66% for tourist, 63.30% for valkyrie, and 66.80% for wizard.The task is harder because the pet can be killed or fall through a trap door; RND produced consistent gains.
- Eat: RND exploration consistently worsened the Eat task, while the valkyrie survived longest at an average of 1713 time steps.Valkyrie and wizard agents gradually learned to hunt for food, unlike monk and tourist agents that began with edible items.
- Gold: Monk agents reached dungeon level 4.2 with the CNN baseline and 5.0 using RND while collecting gold under sparse rewards.Agents learned to descend deeper in search of additional gold.
- Score: 748, 11, 573, and 314 were the average scores for monk, tourist, valkyrie, and wizard, respectively, with RND increasing monk’s average to 780.Monk agents reached dungeon level 5.4 on average, with one run descending to level 11 and scoring 4260.
- Oracle: No agent found the Oracle except in one lucky valkyrie episode, making it a challenging exploration benchmark.The task requires searching many potentially branching dungeon levels; harder goals can be added with few lines of code.
- Generalization Analysis: Test performance increased monotonically with training-seed count, while at least 1000 seeds narrowed the training–test gap and made memorization infeasible.Increasing model capacity improved score performance but not staircase performance.
- Qualitative Analysis: Over training, starvation and traps became less prominent causes of death as agents descended and fought more, while food poisoning later regained prominence in score and scout tasks.Gnome lords, gnome kings, chameleons, and mind flayers became more prominent causes of death over time.
4 Related Work
Related RL environments trade off complexity, procedural variation, and simulation speed. NLE is positioned as a testbed combining these properties with difficult exploration and diverse dynamics.
- Existing RL environments have exposed failures in exploration and generalization while driving progress through increasingly challenging simulations.
- Compared with StarCraft and Minecraft, NLE is intended to reduce computational and entry costs while preserving substantial environmental complexity.
- NLE combines a fast simulator with procedural generation, hard exploration, complex dynamics, and many static and interactive entities.
- Roguelike environments such as Rogue have demonstrated rudimentary navigation, whereas NetHack offers a substantially richer basis for long-term RL research.
- Unlike gym_nethack, NLE leaves NetHack’s game dynamics unchanged rather than removing mechanics such as boulders, traps, locked doors, and item identification.
5 Conclusion and Future Work
The paper presents NLE as a fast, complex, procedurally generated RL environment and reports model-free RL as a sensible baseline. Future work targets richer variability and user-created tasks.
- NLE is presented as a fast, complex, procedurally generated environment for advancing reinforcement learning research.
- Current state-of-the-art model-free RL serves as a sensible baseline in NLE, with analysis of the behaviors learned by agents.
- NetHack’s large state space and varied dynamics create challenges for exploration methods based on novelty or surprise.
- Future NLE versions plan support for NetHack 3.7, Themed Rooms, and Lua-based custom sandbox tasks.
6 Broader Impact
NLE is intended to help RL research address the gap between constrained game environments and the unpredictable real world. Its low computational cost also broadens access to experimentation.
- The work seeks an extremely fast, rich, procedurally generated environment with sparse rewards that tests agent generalization and exploration.
- NLE is presented as a testbed for developing and evaluating RL algorithms that avoid environment-specific heuristics.
- The authors state that solving these challenges is not achieved here; NLE instead provides the challenge and evaluation setting.
- NLE is computationally cheap for running environments and collecting data, supporting researchers in resource-constrained laboratories.
A Further Details on NetHack
NetHack combines stochastic, partially observable gameplay with a large action and observation space. NLE exposes these mechanics through a Gym-compatible interface while retaining implementation-specific constraints.
- NetHack properties: NetHack offers thirteen roles, five races, three alignments, and two genders, with character choices affecting features and interactions.
- NetHack properties: Stochastic entity interactions and severe penalties for careless exploration make NetHack relevant to exploration and safe RL.
- NetHack properties: Fog of War creates partial observability, requiring agents to discover level topology and respond to changes outside visible regions.
- NLE interface: NLE’s Gym wrapper exposes numerical observations including glyphs, characters, colors, statistics, messages, and inventory arrays.
- Observation space: Glyph observations use integer ids in a (21, 79) array, with MAX_GLYPH = 5991 in the reported build.
- Observation space: The observation interface includes a length-25 bottom-line statistics vector and a length-256 padded message vector.
- Observation space: Inventory observations encode up to 55 items through glyph, string, letter, and object-class arrays.
- Implementation boundaries: The exact monster set depends on compile-time options and the target operating system, limiting cross-build equivalence.
D Environment Speed Comparison
This section compares NLE with popular Gym environments using environment steps per second, based on controlled random-policy runs. It also identifies the compass-direction action table used to describe NLE’s movement interface.
- Evaluation setup: The comparison used uniformly random policies with reset on terminal states, on a MacBook Pro running the listed software configuration.The authors caution that the measurements do not necessarily reflect the performance of these environments in general.
- Action interface: Table 3 lists the compass-direction actions available in the environment.
- Environment comparison: Table 4 compares NLE with popular environments through their Python Gym interfaces using environment steps per second (SPS).All environments except ObstacleTowerEnv were run through Gym with standard settings for 60 seconds.
E Task Details
NLE defines diverse tasks around exploration, survival, navigation, resource collection, and progression through NetHack’s dungeon. Tasks use standardized reward and environment settings while exposing increasingly demanding objectives.
- Common task settings: All tasks penalize actions that fail to advance the in-game timer with a reward of −0.001.Autopick is disabled except for Gold, and bones files are disabled across tasks.
- Navigation and exploration: The Staircase task rewards 100 when the agent reaches the next dungeon level’s downward staircase and ends after 1000 agent steps.Success may require opening or kicking doors, searching for hidden passages, avoiding traps, and moving boulders.
- Navigation and exploration: The Pet task gives 100 only when the agent reaches the staircase with its pet next to it.
- Survival and resources: Eat rewards increases in the in-game Hunger status, requiring agents to locate food or monster corpses while managing starvation risks.
- Survival and resources: Gold rewards collected gold, encouraging full-map exploration, dungeon descent, and advanced strategies involving gems, shopkeepers, vaults, or leprechaun halls.
- Exploration and progression: Scout rewards +1 for uncovering previously unknown dungeon tiles, while Score rewards increases in the in-game score dominated early by monster kills and dungeon descent.
- Exploration and progression: The Oracle task rewards 1000 for reaching an Oracle randomly placed between dungeon levels five and nine.Reliable success requires descending multiple staircases and often exhaustively exploring each level.
- Observation model: The task model uses 32-dimensional glyph embeddings and 128-dimensional hidden and LSTM output representations.Full-map and 9 × 9 glyph crops are encoded with a five-layer ConvNet.
G Random Network Distillation Details
The NLE RND setup adapts the original exploration method to symbolic observations and combines intrinsic and extrinsic rewards. Experiments found that intrinsic reward alone was insufficient for meaningful dungeon progress and Oracle discovery.
- RND configuration: RND uses a two-headed value function, separate discount factors, non-episodic intrinsic reward, episodic extrinsic reward, and intrinsic-reward normalization.
- NLE adaptations: The implementation replaces pixel-based feature extraction with the baseline model’s symbolic-observation architecture and omits observation normalization.
- Reward scaling: Intrinsic reward is divided by ten before normalization so it carries less weight than extrinsic reward, while intrinsic rewards are clipped with rc = tanh(r/100).
- Reward scaling: The forward modeling loss is downscaled by 0.01 to slow familiarization because intrinsic reward often collapsed quickly despite normalization.
- Ablations: Using feature subsets or the original RND architecture with symbolic embeddings did not improve results.
- Limitations: Intrinsic reward alone produced no significant dungeon progress even on a fixed seed, and RND was insufficient for learning to find the Oracle.Combined intrinsic and extrinsic reward helped some tasks for some characters.
- Experiment tooling: A NodeJS web dashboard visualizes experiment runs and statistics and supports replaying episodes recorded as tty files.
I NetHack Bots
Earlier NetHack bots primarily used search, heuristics, and planning rather than statistical learning, while some relied on exploits or random-number-generator manipulation. The paper argues that reliably solving NetHack remains exceptionally difficult.
- Prior bots: TAEB, BotHack, and Saiph largely relied on search heuristics and planning, whereas SWAGGINZZZ used lookups, exhaustive simulation, and RNG manipulation.
- Exploits and robustness: BotHack’s pudding-farming exploit strengthened its character and simplified late-game fights, but the strategy was later disabled by a NetHack patch.
- Difficulty of solving NetHack: The authors consider a hand-crafted bot solving NetHack under their definition unlikely because many corner cases can derail play.
- Evaluation measures: Figure 8 reports mean score, dungeon level, experience level, and steps from the last 100 episodes, averaged over five runs.
J Viewing Agent Videos
The paper provides ways to inspect trained NetHack agents through recordings, dashboards, annotated game stages, architecture diagrams, and training analyses.
- Agent recordings can be watched on the Asciinema portal or played in a terminal, with playback speed adjustable through a command-line flag or web controls.The recordings are available at the authors’ Asciinema page.
- Task metrics are averaged over the last 1000 episodes, while the score task also lists its top five episodes from that window.
- Training analysis reports different causes of death averaged over the last 1000 episodes and five runs.
- The web dashboard is included as an interface for viewing the NetHack Learning Environment.
- Annotated examples show an agent at two stages of NetHack, including a procedurally generated first level and Gnomish Mines.
- The released baseline models are summarized in an overview of their core architecture.