Source-linked AI summary
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, Jifeng Dai
TL;DR
Existing Minecraft-agent research focuses narrowly on ObtainDiamond and faces inefficient, poorly scalable RL-based learning for broader open-world tasks. GITM uses LLMs for hierarchical decomposition, structured actions, feedback, text-based knowledge, and memory. It surpasses previous methods on ObtainDiamond and reaches all items in the Minecraft Overworld technology tree.
Problem
Existing Minecraft research predominantly focuses on ObtainDiamond, while RL agents require many training steps and scale poorly to new tasks.
Method
GITM uses LLM-based hierarchical decomposition, structured actions, environmental feedback, external text knowledge, and text-based memory for Minecraft planning and control.
Results
+47.5% success rate on ObtainDiamond is reported for GITM over previous methods, and the agent collects all 262 Minecraft Overworld items.
Takeaways & Limitations
GITM demonstrates the feasibility of using LLMs to develop Generally Capable Agents that handle broad Minecraft objectives and long-horizon open-world tasks.
Abstract
from arXiv · showhide
The captivating realm of Minecraft has attracted substantial research interest in recent years, serving as a rich platform for developing intelligent agents capable of functioning in open-world environments. However, the current research landscape predominantly focuses on specific objectives, such as the popular "ObtainDiamond" task, and has not yet shown effective generalization to a broader spectrum of tasks. Furthermore, the current leading success rate for the "ObtainDiamond" task stands at around 20%, highlighting the limitations of Reinforcement Learning (RL) based controllers used in existing methods. To tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel framework integrates Large Language Models (LLMs) with text-based knowledge and memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These agents, equipped with the logic and common sense capabilities of LLMs, can skillfully navigate complex, sparse-reward environments with text-based interactions. We develop a set of structured actions and leverage LLMs to generate action plans for the agents to execute. The resulting LLM-based agent markedly surpasses previous methods, achieving a remarkable improvement of +47.5% in success rate on the "ObtainDiamond" task, demonstrating superior robustness compared to traditional RL-based controllers. Notably, our agent is the first to procure all items in the Minecraft Overworld technology tree, demonstrating its extensive capabilities. GITM does not need any GPU for training, but a single CPU node with 32 CPU cores is enough. This research shows the potential of LLMs in developing capable agents for handling long-horizon, complex tasks and adapting to uncertainties in open-world environments. See the project website at https://github.com/OpenGVLab/GITM.
1 Introduction
Minecraft provides an open-world testbed for generally capable agents, but existing methods focus narrowly on ObtainDiamond and struggle with long-horizon, adaptable behavior. GITM addresses this with LLM-based hierarchical planning, structured actions, text-based knowledge, and memory.
- Motivation: Minecraft’s unrestricted, long-horizon environment provides a platform for researching autonomous Generally Capable Agents.The paper frames mastering Minecraft’s technical challenges as analogous to mastering multidisciplinary real-world technologies.
- Research gap: Existing research predominantly targets ObtainDiamond, which involves fewer than 5% of Minecraft’s item types.The paper contrasts this specialized objective with obtaining all Minecraft items, which requires broader knowledge and capabilities.
- Research gap: RL agents require vast training budgets and scale poorly to new tasks, limiting their practicality for broad Minecraft objectives.DreamerV3 reportedly needs nearly 30 million steps for diamonds, while VPT uses separate agents for exploration and diamond mining.
- Approach: GITM decomposes complex goals into sub-goals, structured actions, and finally keyboard/mouse operations instead of mapping directly to low-level controls.Its agents also use text-based knowledge and memory to support planning and interaction.
- Reported outcome: +47.5% success rate on ObtainDiamond is reported for GITM compared with previous methods.The introduction presents this result alongside the broader aim of developing generally capable Minecraft agents.
2 Related Work
Prior Minecraft agents largely rely on reinforcement learning, imitation learning, or LLM-assisted high-level planning for narrower tasks. GITM differs by making the LLM-native agent responsible for hierarchical planning while using explicit action abstraction and text-based knowledge and memory.
- Minecraft agents: Earlier Minecraft agents used hierarchical RL, imitation learning, video-trained foundation models, and human-feedback fine-tuning.These approaches established RL as the dominant paradigm before recent LLM-based planning methods.
- LLM planning: Recent LLM-based methods enhance Minecraft agents’ high-level planning through environment feedback, descriptions, explanations, selectors, or skill graphs.The related work names Inner Monologue, DEPS, and Plan4MC as examples.
- GITM’s distinction: GITM uses an LLM-native approach rather than RL-based or RL-with-LLM methods.The paper characterizes its distinction as combining high-level action abstraction with text-based knowledge and memory.
- Task setting: Unlike tool-augmented LLM systems for robotics, GITM targets more complex goals in a highly uncertain open world.The comparison is presented as a difference in task setting and environmental uncertainty.
3 Method
GITM combines hierarchical goal decomposition, structured action planning, feedback, external text knowledge, and textual memory to control Minecraft agents. Its components translate goals into executable actions while revising plans from environmental feedback and prior experience.
- LLM Decomposer: GITM recursively decomposes a goal into a sub-goal tree and schedules prerequisite sub-goals through post-order traversal.The goal format represents an object, count, material, tool, and related information; decomposition continues until prerequisites are exhausted.
- LLM Decomposer: External Minecraft Wiki knowledge and crafting or smelting recipes supply prerequisites, tools, quantities, and resource-distribution information for decomposition.For example, recipes specify materials and tools for wooden pickaxes, while the knowledge base describes where diamonds are found.
- LLM Planner: The LLM Planner generates goal-specific structured action sequences, receives feedback, and revises plans through a closed loop.Planning uses instructions, user queries, previous feedback, and reference plans retrieved from memory.
- Structured actions: Structured actions encode a name, arguments, and description, giving the LLM clear semantics for selecting actions at the cognitive level.The interface abstracts low-level control while preserving execution information for the agent.
- Text-based memory: Text-based memory stores and retrieves successful experience, including summarized reference plans for recurring objectives.Explicit textual knowledge is intended to support direct reuse across similar tasks and improve learning efficiency and generalization.
4 Experiments
GITM evaluates whether LLM-based agents can generalize beyond diamond acquisition by collecting Overworld items, comparing success rates, learning efficiency, and component contributions. It collects all 262 evaluated items, improves diamond performance over prior agents, and benefits from hierarchical decomposition, feedback, external knowledge, and text-based memory.
- Main Result: 262 Overworld item-collection tasks define the evaluation, using technology-tree coverage and success rate as metrics.The evaluation excludes items obtainable only through trading, treasure chests, special structures, or Silk Touch.
- Main Result: GITM collects all 262 Overworld items, compared with 13/262 for DreamerV3, 15/262 for VPT, and 69/262 for DEPS.Prior methods are limited by hard-coded goals, low training efficiency, or reliance on RL controllers for subtasks.
- Main Result: GITM achieves non-zero success rates for every Overworld item and 100% success on simple wooden-tool tasks.Success rates change smoothly across items, indicating robustness in the uncertain open-world setting.
- Learning Efficiency: GITM raises its success rate from 35% to 47.5% within the first 5,000 steps and requires several orders fewer episodes than other methods before doubling its initial success rate.RL agents typically require tens of millions of parameter-updating steps before reaching meaningful non-zero success rates.
- Ablation Study: Goal decomposition raises stone-axe success from 5% to 67.5%, while feedback enables diamond collection when combined with decomposition.External knowledge increases iron-pickaxe and diamond success, and text-based memory yields 95.0% iron-pickaxe and 67.5% diamond success, exceeding no-memory performance by 37.5% and 32.5%.
5 Conclusion
GITM uses LLMs for hierarchical goal decomposition and advances Minecraft agents beyond RL-centered approaches. It improves ObtainDiamond success by +47.5% and collects all items in the Minecraft Overworld.
- 5 Conclusion: GITM uses LLM-based hierarchical decomposition to break goals into sub-goals, structured actions, and keyboard/mouse operations.The framework introduces an LLM Decomposer, LLM Planner, and LLM Interface.
- 5 Conclusion: GITM outperforms all previous methods on ObtainDiamond by +47.5% success rate.The conclusion presents this result as progress toward generally capable Minecraft agents.
- 5 Conclusion: GITM obtains all items in the Minecraft Overworld technology tree.This result marks progress toward Generally Capable Agents in Minecraft.
A.1 LLM Decomposer
The LLM Decomposer converts Minecraft goals into recursively structured sub-goals, while the LLM Planner and Interface translate them into executable actions and environment interactions.
- Goal decomposition: Goals are recursively decomposed into prerequisite materials and tools until no further prerequisites remain.The resulting sub-goal tree starts from the final object and expands prerequisite items as child goals.
- Goal decomposition: The decomposer uses text-based crafting recipes and Internet wiki knowledge to identify materials, tools, and related information.The generated goal representation includes the target object, quantity, prerequisites, tool, and compact knowledge summary.
- Structured actions: Structured actions are represented as (verb, object, tools, materials) tuples, with None used when requirements are absent.Actions are extracted from tree-structured plans generated for MineDojo tasks and balanced at decomposition depth 2.
- Structured actions: The action vocabulary includes equip, explore, approach, mine/attack, dig_down, go_up, build, craft/smelt, and apply.More detailed decomposition is avoided for attack and build to focus on longer-term task completion.
- Action interface: The interface executes abstract actions using LiDAR-based object localization and voxel-based navigation and status observations.RGB is omitted, and exploration uses BFS above ground or DFS underground, while approach uses A∗ path finding.
A.4.2 Implementation of Memory Summarization
Memory summarization condenses successful, state-conditioned action sequences into reusable references while filtering irrelevant steps and lucky-case contingencies.
- Inputs and conditioning: The summarizer receives a goal, agent state, and multiple successful action sequences, then produces one universal reference sequence.State includes inventory, biome, and whether the agent is above or below ground.
- Inputs and conditioning: Summarized sequences remain conditioned on the given state, including inventory and environment information.The summarized plan is intended to achieve the goal from that state rather than from an unspecified starting condition.
- Summarization rules: The summarizer includes necessary actions, removes redundant or irrelevant actions, and excludes lucky successes and contingency handling.This produces a general-case sequence without requiring specific modification for each successful example.
- Output format: The output begins with natural-language summarization thoughts and then provides the summarized action sequence in JSON format.Each action records its name, arguments, and expected result.
B Results of All Items
The evaluation covers all 262 items in the Minecraft Overworld Technology Tree using incremental testing designed to control cost while accommodating rare and biome-dependent items.
- Evaluation scope: The evaluation reports success rates for all 262 items in the entire Minecraft Overworld Technology Tree.The testing strategy is incremental rather than uniform across items.
- Evaluation protocol: Each item starts with 20 games and can scale to 50, 100, or 200 games when successful counts remain low.Testing stops when the success count exceeds 1 or after 200 games.
- Environmental factors: Biome distribution limits success rates for items such as cactus, pumpkin, and melon that occur only in specific biomes.The authors identify biome exploration as a key factor influencing performance.
C Supplementary Ablations
Ablations show that structured actions and hierarchical goal decomposition improve performance, while direct low-level mapping and some RL baselines struggle with long-horizon Minecraft goals.
- Direct mapping: Directly mapping long-horizon goals to keyboard and mouse operations is difficult for both LLM and goal-conditioned RL approaches.The evaluated DEPS model can ignore prerequisites, while the LLM struggles with environment observations and low-level operations.
- Structured action: Adding structured actions significantly improves LLM performance by abstracting environment observations and keyboard/mouse operations.The abstraction allows the LLM to make higher-level decisions while receiving more precise interaction support.
- Goal decomposition: Goal decomposition improves both the goal-conditioned RL model and the proposed method by simplifying long-term tasks into sub-goals.The ablations report its effectiveness in comparisons between Tab. 5(b)(d) and Tab. 5(f)(g).
- RL comparison: Replacing the goal-conditioned RL model with an LLM using structured actions produces a significant performance boost.This comparison changes the model while retaining the structured-action pipeline.
D ObtainDiamond
GITM demonstrates the ObtainDiamond challenge by collecting required materials, crafting necessary tools, and obtaining a diamond.
- The agent collects logs, stone, and iron ore during the ObtainDiamond challenge.
- It crafts a wooden pickaxe, stone pickaxe, furnace, and iron pickaxe.
- The diamond is ultimately obtained.
E Applications
GITM enables survival and Nether exploration by constructing four Minecraft items that support protection, food provision, automation, and access to the Nether world.
- GITM constructs a Shelter with Farmland, an Iron Golem, a Redstone Circuit, and a Nether Portal.
- The Shelter with Farmland protects against nighttime monsters and provides food.
- The Iron Golem automatically attacks monsters to protect the agent and shelter.
- The Redstone Circuit supports automation equipment, while the Nether Portal provides access to the Nether world.