Source-linked AI summary
MineRL: A Large-Scale Dataset of Minecraft Demonstrations
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, Ruslan Salakhutdinov
TL;DR
Standard deep reinforcement learning requires too many samples for many complex problems, while existing reinforcement-learning datasets lack sufficiently large, structured human demonstrations. MineRL addresses this gap with a simulator-paired Minecraft dataset and extensible packet-level collection platform, showing that human data improves performance and sample efficiency even though the tasks remain difficult for standard agents.
Problem
Existing reinforcement-learning datasets lack the scale, structure, and quality needed to develop and evaluate methods using human demonstrations.
Method
MineRL collects packet-level human gameplay on a public Minecraft server, reconstructs demonstrations, and automatically annotates them as simulator-paired state-action data across related tasks.
Results
Human-data methods perform better and achieve high performance with fewer samples across all evaluated tasks, while learned agents remain significantly worse than humans.
Takeaways & Limitations
MineRL provides a community-accessible resource for studying inverse reinforcement learning, hierarchical learning, lifelong learning, and related sequential-decision methods.
Abstract
from arXiv · showhide
The sample inefficiency of standard deep reinforcement learning methods precludes their application to many real-world problems. Methods which leverage human demonstrations require fewer samples but have been researched less. As demonstrated in the computer vision and natural language processing communities, large-scale datasets have the capacity to facilitate research by serving as an experimental and benchmarking platform for new methods. However, existing datasets compatible with reinforcement learning simulators do not have sufficient scale, structure, and quality to enable the further development and evaluation of methods focused on using human examples. Therefore, we introduce a comprehensive, large-scale, simulator-paired dataset of human demonstrations: MineRL. The dataset consists of over 60 million automatically annotated state-action pairs across a variety of related tasks in Minecraft, a dynamic, 3D, open-world environment. We present a novel data collection scheme which allows for the ongoing introduction of new tasks and the gathering of complete state information suitable for a variety of methods. We demonstrate the hierarchality, diversity, and scale of the MineRL dataset. Further, we show the difficulty of the Minecraft domain along with the potential of MineRL in developing techniques to solve key research challenges within it.
1 Introduction
Standard deep reinforcement learning is highly sample-inefficient, while existing reinforcement-learning datasets lack the scale, structure, and quality needed for human-demonstration research. MineRL addresses this gap with a large-scale, simulator-paired dataset and an extensible collection platform for Minecraft demonstrations.
- Motivation: 44 to over 200 million frames are required for human-level performance in Atari 2600 games, illustrating standard deep reinforcement learning’s sample inefficiency.The cited examples correspond to 200 to over 900 hours of gameplay.
- Motivation: Existing reinforcement-learning simulators lack large-scale labeled human-demonstration datasets for domains with broad structural constraints and tasks.
- Domain challenge: Minecraft’s item hierarchy contains 371 unique items, blocks, and non-player characters, making complete coverage by one player require several hundred hours.
- MineRL: Over 60 million automatically annotated state-action pairs span six related Minecraft tasks involving planning, vision, control, navigation, multi-agent interaction, and task hierarchies.
- MineRL: MineRL records packet-level gameplay information, enabling reconstruction of each player’s view and actions and supporting new tasks and automatic annotation.
2 Environment: Minecraft
Minecraft combines an open-ended, mutable 3D world with long-horizon, situation-dependent hierarchies of resource collection, construction, exploration, and combat. These properties make the domain expressive for reinforcement and imitation learning but difficult for existing methods to model fully.
- 2.1 Description: Minecraft is a 3D, first-person, open-world environment where players gather resources and create structures and items across sessions lasting tens of hours.
- 2.1 Description: Players develop their own subgoals because Minecraft has no single definable objective, producing multiple natural task hierarchies.
- 2.1 Description: Item-collection objectives form a large task hierarchy because tools, materials, and items require strict prerequisite sets.
- 2.1 Description: Construction, exploration, and combat introduce long time horizons and flexible hierarchies whose requirements depend on the situation.
- 2.2 Existing Interest: Prior Minecraft research often restricts tasks to 2D movement, discrete positions, or confined maps, unlike the complexity of fully embodied human gameplay.
3 Methods: MineRL Data Collection Plaform
MineRL provides an end-to-end platform for collecting, reconstructing, annotating, and extending Minecraft player trajectories. Its public server, client plugin, and processing pipeline support scalable datasets paired with sequential decision-making environments.
- Platform architecture: The platform combines a public game server and website, a custom client plugin, and a data-processing pipeline for collecting and annotating player trajectories.
- Data acquisition: Players consent to anonymous recording, install a client plugin, and stream client-server game packets to the MineRL data repository while completing stand-alone tasks.
- Data pipeline: The pipeline resimulates, modifies, and augments recorded trajectories into algorithmically consumable formats while automatically annotating game state.
- Extensibility: A modular server and in-game economy support continued collection for expanding sets of stand-alone tasks and future multi-task datasets.
4 Results: MineRL-v0
MineRL-v0 is a large, simulator-paired dataset of human Minecraft demonstrations designed to support research on difficult, hierarchical tasks. Its scale, detailed state-action records, expert-heavy trajectories, broad coverage, diverse game conditions, and observed subpolicy structure make it useful for imitation and hierarchical reinforcement learning.
- Dataset Details: 500+ hours across six tasks yield over 60 million state-action pairs per dataset version, with low- and medium-resolution sizes of 130 GB and 734 GB.The released versions vary in resolution and texture while retaining the same per-version state-action-pair scale.
- Dataset Details: Each trajectory records tick-level RGB observations, comprehensive game-state features, and player actions including keyboard, view, GUI, chat, and crafting events.Trajectories are sampled every Minecraft game tick at 20 ticks per second.
- Tasks: The six stand-alone tasks target hierarchality, long-term planning, complex orienteering, navigation, resource acquisition, and varied Minecraft gameplay challenges.The task set includes Navigate, Treechop, and four Obtain<Item> tasks, with common human-player action and observation spaces.
- Human Performance: A majority of demonstrations are expert-level, while beginner and intermediate trajectories support methods that leverage both optimal and imperfect demonstrations.Expert-level play is identified using completion times from players with at least five years of Minecraft experience.
- Coverage: MineRL-v0 covers most item-acquisition subtasks, including 371 Survival subtasks demonstrated hundreds to tens of thousands of times, enabling option extraction and skill acquisition.Its label hierarchy also supports metrics for the interpretability and quality of extracted options.
- Coverage and Hierarchality: Data from 1,002 player sessions spans 24,393,057 square meters and randomly initialized worlds, while precedence graphs reveal overlapping and adaptive human subpolicies across tasks.Players move similarly on tasks with overlapping hierarchies and differently on tasks with less overlap; longer paths occur when earlier items are unavailable.
5 Experiments
The experiments evaluate reinforcement and imitation-learning methods on three Minecraft tasks, showing that learned agents remain far below human performance while human data improves performance and sample efficiency.
- Experiment Configuration: The study evaluates DQN, pretrained DQN, A2C, and behavioral cloning on Treechop, Navigate (Sparse), and Navigate (Dense).The evaluation includes the easiest designed tasks and a simplified dense-reward navigation task.
- Experiment Configuration: Observations are converted to grayscale and resized to 64x64, while reinforcement-learning methods use a simplified action space of 10 discrete actions.Behavioral cloning performs similarly without the action-space simplification.
- Experiment Configuration: Agents are compared using the highest average reward over a 100-episode window, alongside random-policy and 50th-percentile human-performance baselines.The results are summarized in Table 1.
- Evaluation and Discussion: In all tasks, learned agents perform significantly worse than humans; on Treechop, humans score 64 while reinforcement agents score less than 4.The authors attribute much of the difficulty to long-horizon credit assignment, such as delayed drowning during water navigation.
- Evaluation and Discussion: In all tasks, methods leveraging human data perform better and achieve high performance using fewer samples.Expert demonstrations are especially helpful in Navigate (Sparse), where random exploration is unlikely to yield reward.
6 Related Work
Related work includes simpler imitation-learning domains, real-world datasets without simulators, and StarCraft II, whereas MineRL targets a challenging open-world simulator-paired setting.
- Prior Demonstration Datasets: Atari and Super Tux Kart have used human-demonstration datasets for imitation learning, but their shallow hierarchies and small action- and state-spaces make them simpler than Minecraft.These domains are not open-world.
- Real-World Datasets: KITTI provides 3 hours of 3D traffic data and Dex-Net provides five million robotic grasps, but both concern real-world tasks where a lack of simulators limits development speed.MineRL is larger relative to the difficulty of its domain.
- Simulator-Paired Domains: MineRL is directly compatible with the Malmo simulator, enabling training in the data-collection domain and comparison with methods not based on imitation learning.This distinguishes it from the cited real-world datasets.
- Simulator-Paired Domains: StarCraft II is the only other complex unsolved domain identified with both a simulator and a large-scale dataset, but it is not open-world and its StarData trajectories are unlabeled standard gameplay.MineRL instead contains labeled related tasks representing different components of Minecraft.
7 Conclusion and Future Work
MineRL-v0 provides a large, expanding collection of procedurally annotated human demonstrations in an open-world, simulator-paired environment. The platform is intended to support diverse sequential-decision-making methods and real-world-environment research.
- Conclusion and Future Work: MineRL-v0 contains 60 million state-action pairs of procedurally annotated human demonstrations across six tasks in an open-world, simulator-paired environment.None of the six tasks can be fully solved with standard deep reinforcement learning methods.
- Conclusion and Future Work: The platform supports ongoing collection of demonstrations for existing and new tasks.The authors host MineRL-v0 through a community-accessible website and plan to gather feedback on new annotations and tasks.
- Conclusion and Future Work: The authors expect MineRL to support methods including inverse reinforcement learning, hierarchical learning, and lifelong learning as the dataset expands.They hope it becomes a central resource for sequential-decision-making research.