PaperlayerView exact revision on arXiv

Source-linked AI summary

MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research

Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, Tim Rocktäschel

arXiv:2109.13202v2cs.LGstat.ML

TL;DR

RL benchmarks often lack controlled tests of specific capabilities, while extending simplified environments toward richer settings can require substantial engineering. MiniHack uses NetHack’s environment assets and dynamics to create and extend custom RL tasks; across navigation, skill-acquisition, and ported environments, added complexity often leaves baseline agents unable to make progress.

  • Problem

    Widely used whole-game benchmarks do not provide controlled tests of specific RL capabilities, while systematically extending simplified environments can require substantial engineering.

    Source: §1 · p. 1

  • Method

    MiniHack uses NetHack’s des-file format and level compiler to create custom environments and extend existing ones with rich entities and dynamics.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    Across navigation, skill-acquisition, and ported tasks, added complexity often leaves baseline agents unable to make progress.

    Source: §4 · pp. 7–8

  • Takeaways & Limitations

    The released task suite provides a unified setting for testing a wide variety of RL capabilities and the limits of RL methods.

    Source: §1 · p. 3, §6 · p. 10

    Some ported environments differ from their originals; for MiniGrid, action-space and dynamics changes include probabilistic door opening in MiniHack.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

Progress in deep reinforcement learning (RL) is heavily driven by the availability of challenging benchmarks used for training agents. However, benchmarks that are widely adopted by the community are not explicitly designed for evaluating specific capabilities of RL methods. While there exist environments for assessing particular open problems in RL (such as exploration, transfer learning, unsupervised environment design, or even language-assisted RL), it is generally difficult to extend these to richer, more complex environments once research goes beyond proof-of-concept results. We present MiniHack, a powerful sandbox framework for easily designing novel RL environments. MiniHack is a one-stop shop for RL experiments with environments ranging from small rooms to complex, procedurally generated worlds. By leveraging the full set of entities and environment dynamics from NetHack, one of the richest grid-based video games, MiniHack allows designing custom RL testbeds that are fast and convenient to use. With this sandbox framework, novel environments can be designed easily, either using a human-readable description language or a simple Python interface. In addition to a variety of RL tasks and baselines, MiniHack can wrap existing RL benchmarks and provide ways to seamlessly add additional complexity.

1 Introduction

MiniHack addresses the gap between whole-game benchmarks and tightly controlled tasks by making it easier to build rich, progressively more difficult RL environments. It draws on NetHack’s entities and dynamics, supports importing and extending existing benchmarks, and provides tasks, baselines, and demonstrations.

1 Introduction

  • Researchers need benchmarks that test specific RL capabilities, but whole-game environments lack controlled proof-of-concept tests and small-scale environments are difficult to extend systematically.Building complex interactions from scratch also requires substantial engineering, leaving a gap for easily specified suites of rich, gradually more difficult tasks.Source: §1 · pp. 1–2
  • MiniHack uses NetHack’s human-readable des-file language and Python interface to build custom environments from pre-implemented entities and complex interactions, then wraps them as Gym environments.The framework also supports enriching existing environments with NetHack elements such as monsters, objects, dungeon features, and stochastic dynamics.Source: §1 · p. 2
  • MiniHack offers increasingly difficult tasks and symbolic, pixel-based, and textual observations, and can port MiniGrid or Boxoban environments for extension with added complexity.The paper identifies unsupervised skill discovery, unsupervised environment design, transfer learning, and language-assisted RL as potential application areas.Source: §1 · p. 2
  • The paper provides baseline agents, benchmarking guidance, a PAIRED demonstration for unsupervised environment design, learning curves, and framework documentation.The baselines use frameworks such as TorchBeast and RLlib.Source: §1 · pp. 2–3

2 Background: NetHack and the NetHack Learning Environment

NetHack is a difficult, stochastic game in randomly generated dungeons, and its full-game scope makes isolated research questions difficult to study. MiniHack provides an extendable sandbox for custom tasks that use NetHack assets and dynamics.

2 Background: NetHack and the NetHack Learning Environment

  • NetHack requires players to descend through more than 50 procedurally generated levels, retrieve a special object, and escape, while managing resources and learning entity mechanics amid stochastic events.The passage notes that winning can take a human years, even with external knowledge, and that the full game is beyond modern RL approaches.Source: §2 · p. 3
  • NLE focuses on the full game and its existing level-generation mechanisms, making it difficult to isolate specific research questions.MiniHack instead provides custom tasks while retaining NetHack’s assets and complex dynamics.Source: §2 · p. 3
  • MiniHack is presented as an extendable sandbox that uses NetHack assets and dynamics to define varied custom tasks.This contrasts with NLE’s focus on the full game.Source: §2 · p. 3

3 MiniHack

MiniHack is a sandbox for designing RL environments whose difficulty can scale by relaxing simplifying assumptions, using NetHack’s des-file format and level compiler. Its sections describe the design language, environment components, Python interface, task suite, and evaluation practices.

3.1 des-file format: A Domain Specific Language for Designing Environments

  • NetHack des-files let users specify human-readable level distributions, with variables, loops, conditionals, probability distributions, underspecified placements, and procedural maze or corridor generation.The examples include random rivers, monsters, terrain replacement, and stairs.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 MiniHack Environments

  • MiniHack draws on NetHack’s diverse monsters and items, and supports multiple observation forms, configurable action spaces, and event-based rewards and termination.The action space can be restricted for targeted skill discovery; skill acquisition tasks use 75 actions.Source: §3.2 · pp. 4–5

3.3 Interface

  • The Gym interface supports either passing a des-file directly or building levels in Python with LevelGenerator, while RewardManager specifies task goals.The example task requires finding and using a wand against a Minotaur before reaching an apple.Source: §3.3 · p. 5

3.4 Tasks: A World of Possibilities

  • The 44 navigation environments target capabilities including exploration, planning, memory, and generalisation, with challenges such as monsters, rivers, and procedurally generated mazes.Tasks can be extended by adding hazards or removing assumptions such as fixed maps or full observability.Source: §3.4 · p. 5
  • Skill acquisition tasks use a 75-action space with context-dependent object interactions and sometimes autoregressive action sequences, making sparse-reward exploration of joint state-action space difficult.Putting on a ring requires selecting PUTON, the ring, and a hand.Source: §3.4 · p. 6
  • MiniHack ports MiniGrid and Boxoban tasks, preserving the capabilities they assess while allowing added complexity; shared observation and action spaces also support transfer-learning comparisons.Examples add monsters, locked doors, or lava to MiniGrid levels and monsters to Boxoban puzzles.Source: §3.4 · p. 6
  • Ported tasks are not always identical to their originals: MiniGrid’s colored keys and doors lack color counterparts in NetHack, and its Dynamic-Obstacles moving objects are absent.The MiniGrid port also changes movement actions and makes door opening probabilistic, while the authors state it assesses the same capabilities.Source: §3.4 · pp. 6–7

3.5 Evaluation Methodology

  • MiniHack evaluation recommendations include matching difficulty-affecting environment parameters, reporting median reward over at least five seeded runs, and testing generalisation on held-out seeds or the full level distribution.The authors also recommend sharing learning curves, wall-clock time, and behavior examples.Source: §3.5 · p. 7

4 Experiments

Experiments use IMPALA baselines, with RND and RIDE also tested on navigation tasks, to assess agent capabilities across MiniHack environments. Increasing task complexity often challenges baseline learning.

Experiments

  • Experiments assess RL capabilities and how incremental environment changes using MiniHack entities challenge existing methods; IMPALA is the main baseline, with RND and RIDE also used for navigation.Agents train and test on the full level distribution of each environment.Source: §4 · p. 7

Navigation Tasks

  • Simpler navigation variants are often solved, but added complexity defeats baselines; none solves the largest KeyRoom variant or the River task with moving monsters and deadly lava.The KeyRoom-Random-S5 task is solved by all baselines, whereas KeyRoom-Random-S15 is not solved by any method.Source: §4 · p. 7, Figure 7 · p. 8

Skill Acquisition Tasks

  • Baselines solve simple single-interaction skill tasks, but make no progress on WoD-Medium or LavaCross when randomized sequences of entity-action relations require broader exploration.The failures persist despite the agents mastering simplified versions of those tasks.Source: §4 · p. 8, Figure 8 · p. 8

Ported Environments

  • In ported MultiRoom tasks, added complexities make learning harder; four-room baselines progress only on the simplest and monster variants, while Extreme two-room runs show no learning progress.All independent runs on the more complex four-room variants fail to obtain positive reward.Source: §4 · pp. 8–9, Figure 9 · p. 8

Unsupervised Environment Design

  • PAIRED’s induced curriculum improves protagonist zero-shot performance on MultiRoom-N2-Extreme, despite training only on a much smaller environment.The adversary places walls, lava, monsters, or locked doors in a small 5x5 room and maximizes regret between antagonist and protagonist returns.Source: §4 · p. 9, Figure 10 · p. 9

Agent Architecture Comparison

  • Medium and large IMPALA architectures perform similarly across five tasks, while the small model performs poorly on Room-Ultimate-15 and CorridorBattle but outperforms larger models on Corridor-3.The comparison varies convolutional layers, hidden MLP sizes, and entity-embedding dimensions.Source: §4 · p. 9, Figure 11 · p. 9

RLlib Example

  • The RLlib example reports DQN, A2C, and PPO results on Room-15x15, illustrating integration of multiple algorithms with MiniHack.RLlib provides access to a wide range of RL algorithms within the framework.Source: §4 · p. 9

5 Related Work

Existing RL benchmarks range from whole games to small controlled tasks, while environment-design frameworks offer different trade-offs in richness, speed, and extensibility. MiniHack draws on NetHack’s entities and mechanics to support custom environments.

Video-game benchmarks

  • Whole-game benchmarks are often computationally expensive, fixed in their levels, or difficult to modify, while ProcGen adds procedural levels but has limited game richness and modification options.MiniHack is reported as 16 times faster than ALE and approximately 10% faster than ProcGen.Source: §5 · pp. 9–10

Small-scale environments

  • MiniGrid offers procedural grid-world tasks but has limited entity and action-space complexity; MiniHack provides hundreds of entities and a high-level DSL for extension.MiniHack extension requires no understanding of underlying implementation details.Source: §5 · p. 10
  • Unlike bsuite’s static task set, MiniHack lets researchers extend existing tasks without needing to understand framework implementation details.Source: §5 · p. 10

Roguelike and sandbox benchmarks

  • NetHack interfaces typically use full-game level generation or fixed subtasks, whereas MiniHack permits custom dungeon layouts and placement of features, monsters, and objects.MiniHack also offers multimodal observations and a Python interface; Griddly is faster on simple environments, while its speed with MiniHack-equivalent complex dynamics is unclear.Source: §5 · p. 10

6 Conclusion

The conclusion presents MiniHack as a sandbox and task suite for targeted RL experimentation, with procedural generation, algorithms, evaluation guidance, and baselines supporting further research. It also describes an agent navigating to the staircase after killing the Minotaur when RewardManager is not used.

6 Conclusion

  • MiniHack combines a framework for creating rich, varied RL environments with a suite of tasks that tests methods across capabilities in a unified setting.Built on NLE and the des-file format, it uses NetHack entities and dynamics for targeted experimentation and allows existing environments to be scaled up in difficulty.Source: §6 · p. 10
  • Procedurally generated environments enable evaluation of agents’ systematic generalization across varied instances.Source: §6 · p. 10
  • MiniHack releases agent-training code for numerous RL algorithms, evaluation best practices, and baseline results across its task suite.Source: §6 · p. 10

B.5 Examples

  • After killing the Minotaur, the agent must navigate to the staircase, the default goal when RewardManager is not used.Source: §B.5 · p. 22

C MiniHack tasks

This section describes MiniHack’s existing tasks and registered configurations, grouping similar tasks together while varying attributes to create harder versions.

C MiniHack tasks

C.1 Navigation Tasks

MiniHack’s navigation tasks span randomized rooms, corridors, mazes, and interaction challenges, with variants adding uncertainty, hazards, memory demands, and deeper exploration.

Room

  • Room tasks vary room size, randomized start and goal positions, monsters, teleportation traps, and darkness, with an ultimate variant combining the added challenges.Source: §C.1 · p. 22

Corridor

  • Corridor tasks randomize rooms and connecting corridors, with R2, R3, and R5 variants increasing the room count from two to five.Source: §C.1 · p. 22

KeyRoom

  • KeyRoom requires finding a key, unlocking a door, and reaching the staircase; variants randomize object locations, enlarge the room, or hide the key in darkness.Source: §C.1 · pp. 22–23

MazeWalk

  • MazeWalk procedurally generates mazes on 9x9, 15x15, and 45x19 grids, while mapped variants reveal the maze and goal locations.Source: §C.1 · p. 23

River

  • River tasks require pushing boulders into water to cross, with harder variants adding multi-boulder planning, monsters, or lava hazards.Source: §C.1 · p. 23

HideNSeek and CorridorBattle

  • HideNSeek combines line-of-sight obstacles and a stronger monster; CorridorBattle rewards luring rats into a narrow corridor, while dark mode adds memory demands.Source: §C.1 · pp. 23–24

Memento and MazeExplore

  • Memento tests choosing a path from an opening cue, while MazeExplore requires collecting apples for +0.5 reward before returning to the staircase for +1.Source: §C.1 · p. 24

C.2 Skill Acquisition Tasks

Skill-acquisition tasks test both multi-step NetHack interactions and abilities such as memory, object handling, directional actions, and navigation.

C.2.1 Skills

  • The skill-task table lists MiniHack navigation tasks and the capabilities they assess.Source: Table 1 · p. 25
  • Confirmation requires responding to an in-game prompt after an interaction action; choosing Y completes the action, while declining leaves the agent in the same state.Source: §C.2.1 · p. 26
  • Other named skills cover picking up objects, selecting inventory items, choosing action directions, and solving mazes or labyrinths.Source: §C.2.1 · p. 26

C.2.2 Tasks

MiniHack’s skill-acquisition tasks range from simple object interactions to complex sequences combining navigation, item use, and combat. Their variants progressively change starting conditions, distractions, and the skills required.

C.2.2 Tasks

  • Skill-acquisition tasks support curriculum and transfer-learning studies, including transfer between MiniHack tasks or to NetHack.The task suite spans a range of complexity.Source: §C.2.2 · p. 26
  • Simple tasks test object interactions; fixed variants fix object and agent locations, while distractor variants add a random object and monster.Eat, Pray, Wear, and LockedDoors tasks can serve as building blocks for more advanced tasks.Source: §C.2.2 · p. 26, Figure 26 · p. 26
  • LavaCross variants progress from starting with a required item to finding randomly placed levitation or freezing objects before crossing lava.Options include levitation potions, rings, boots, a frost horn, and a wand of cold.Source: §C.2.2 · pp. 26–27, Figure 27 · p. 26
  • Wand of Death tasks increase difficulty from zapping a sleeping monster to finding the wand and confronting an awake, hostile monster or a Minotaur in a maze.WoD-Medium also requires picking up the wand, approaching and killing the sleeping monster, then reaching the staircase; WoD-Hard requires reaching the staircase after the kill.Source: §C.2.2 · p. 27, Figure 28 · p. 27
  • Quest tasks combine maze navigation, lava crossing, and combat, with Quest_Hard adding procedurally generated mazes, item-finding, and a powerful monster.Quest_Medium instead requires luring a horde into narrow corridors to survive.Source: §C.2.2 · p. 27, Figure 29 · p. 27

C.3 Ported Tasks

The ported-task suite uses selected tasks from MiniGrid and Boxoban to demonstrate MiniHack’s ability to port existing gridworld environments and enrich them with additional challenges.

C.3 Ported Tasks

  • The experiments use selected MiniGrid and Boxoban tasks to demonstrate porting and enrichment, rather than attempting to port every possible MiniGrid task.The full list used in the experiments is provided in Table 3.Source: §C.3 · p. 27

D Experiment Details

The appendix describes experiment replication and implementation details spanning the agent architecture, TorchBeast and RLlib baselines, architecture comparisons, and PAIRED-based environment design.

D Experiment Details

  • Experiment replication instructions are available in MiniHack’s repository.The section then details individual experiment components.Source: D Experiment Details · p. 27

D.1 Agent and Environment Details

  • The agent encodes entity glyphs into learned embeddings and combines full-screen, agent-centred, and statistics representations; skill-acquisition observations also include the in-game message.A small penalty is applied when an action does not advance NetHack’s timer.Source: §D.1 · p. 29

D.2 TorchBeast Details

  • TorchBeast experiments use specified CNN, LSTM, optimizer, and intrinsic-reward settings, with architecture and throughput details reported for the baselines.IMPALA and RND each train for two million Room-5x5 timesteps using the same stated hardware resources.Source: §D.2 · p. 29

D.3 Agent Architecture Comparison Details

  • The architecture comparison varies model size across CNN depth, hidden dimension, and entity-embedding size while holding other hyperparameters identical.The medium model uses the architecture specified in Appendix D.2.Source: §D.3 · p. 30

D.4 RLlib Details

  • RLlib examples provide DQN, PPO, and A2C training on simple MiniHack environments, with a limited hyperparameter sweep intended as a starting point rather than state-of-the-art tuning.The DQN combines Double Q-Learning, a duelling architecture, and prioritized experience replay; experiment results are shown in Figure 12.Source: §D.4 · p. 30

D.5 Unsupervised Environment Design

  • The PAIRED unsupervised-environment-design adversary builds levels on an initially empty 5 × 5 grid by placing objects, then selecting goal and agent positions.The adversary policy encodes glyph observations, time step, and random noise to produce choices over 169 possible cells; protagonist and antagonist policies use the agent-centred crop and MiniHack action space.Source: §D.5 · p. 31

E Full Results

The full-results figures present baseline mean episode returns across navigation, skill-acquisition, and ported MiniHack tasks.

E Full Results

  • Mean episode returns for all navigation tasks are plotted across five independent runs, with the median run bolded.Figure 30 presents these results.Source: Figure 30 · p. 32
  • Mean episode returns for all skill-acquisition tasks are plotted across five independent runs, with the median run bolded.Figure 31 presents these results.Source: Figure 31 · p. 33
  • Mean episode returns for tasks ported to MiniHack are plotted across five runs, with the median run bolded.Figure 32 presents these results.Source: Figure 32 · p. 33

Related papers

Loading 2109.13202v2…