Voyager: An Open-Ended Embodied Agent with Large Language Models

1 Introduction

• Existing LLM-based agents lack lifelong learning that progressively acquires, updates, accumulates, and transfers knowledge over extended periods [31].The paper argues that Minecraft lifelong learners should select tasks appropriate to their skills and world state, refine and retain skills through feedback, and continually explore autonomously.Source: §1 · p. 2
• Voyager combines an automatic curriculum, a skill library, and iterative code prompting to learn continually in Minecraft without human intervention.It uses blackbox GPT-4 through prompting and in-context learning [37] [38], without model-parameter access, gradient-based training, or fine-tuning. Code represents temporally extended, compositional actions.Source: §1 · p. 2
• The automatic curriculum proposes progressively harder tasks based on exploration progress and the agent’s state.GPT-4 generates the curriculum around the overarching goal of discovering as many diverse things as possible.Source: §1 · p. 2
• Voyager stores successful action programs, retrieves them using description embeddings, and composes simpler programs into complex skills.Composition rapidly compounds capabilities and alleviates catastrophic forgetting associated with other continual learning methods [31].Source: §1 · pp. 2–3
• Iterative prompting refines executable programs using environmental observations and interpreter errors until self-verification confirms task completion.After verification, Voyager commits the program to its skill library and asks the automatic curriculum for the next milestone.Source: §1 · p. 3
• 3.3× more unique items, key tech tree milestones unlocked up to 15.3× faster, and 2.3× longer distances are Voyager’s gains over prior SOTA.Evaluation compares Voyager against ReAct, Reflexion, and AutoGPT in MineDojo. Its learned skill library also supports solving novel tasks from scratch in a new Minecraft world, while other methods struggle to generalize.Source: §1 · p. 3

2.1 Automatic Curriculum

• GPT-4 proposes increasingly challenging exploration tasks, adapting to the agent’s current state, completed and failed tasks, and capabilities.Prompt directives encourage diverse, manageable goals using inventory, equipment, surroundings, biome, time, health, hunger, and position. GPT-3.5 supplies additional context by asking and answering questions about the agent’s state and exploration progress, reducing costs.Source: §2.1 · pp. 3–4

2.2 Skill Library

• A vector database stores verified programs under description embeddings and retrieves the top-5 relevant skills using task plans and environment feedback.GPT-3.5 generates program descriptions for indexing and general task-solving suggestions for retrieval queries.Source: Figure 4 · p. 4, §2.2 · p. 5
• Skills are reusable executable programs for temporally extended actions, generated with control APIs, retrieved skills, and chain-of-thought prompting.GPT-4’s code-generation prompt also includes the previous program, environment feedback, execution errors, critique, and the agent’s current state.Source: §2.2 · pp. 4–5

2.3 Iterative Prompting Mechanism

• Iterative prompting improves programs using environment feedback, execution errors, and self-verification, incorporating execution outcomes into GPT-4’s next refinement round.Environment feedback exposes intermediate execution progress, while interpreter errors identify invalid operations and syntax problems.Source: §2.3 · p. 5
• A separate GPT-4 agent acts as a critic [49], checking task success from the current state and task description and suggesting corrections after failure.Verified completion adds the skill to the library and triggers a new curriculum objective. If the agent remains stuck after 4 code-generation rounds, it requests another task.Source: §2.3 · pp. 5–6

3.2 Baselines

• ReAct, Reflexion, and AutoGPT were adapted from NLP settings; pixel-input, low-level-control Minecraft agents [53] were excluded.The authors avoid direct comparison with those Minecraft agents because Voyager uses high-level Mineflayer controls rather than addressing visual perception or sensorimotor control.Source: §3.2 · p. 6

3.3 Evaluation Results

• 15.3× faster wooden-level unlocking, 8.5× faster stone-level unlocking, and 6.4× faster iron-level unlocking put Voyager ahead of baselines.These speedups measure prompting iterations; Voyager alone unlocks the diamond level. Voyager also traverses 2.3× longer distances than baselines across diverse terrains.Source: Table 1 · p. 7, §3.3 · p. 7, Figure 7 · p. 7

3.1 Experimental Setup

• Experiments evaluate exploration, tech-tree mastery, map coverage, and zero-shot generalization using a MineDojo environment with Mineflayer controls.Text completion uses gpt-4-0314 and gpt-3.5-turbo-0301, with text-embedding-ada-002 embeddings. Temperatures are 0 except for the automatic curriculum, which uses 0.1 to encourage task diversity.Source: §3.1 · p. 6, §3.3 · p. 7

3.3 Evaluation Results

• 63 unique items within 160 prompting iterations give Voyager 3.3× as many novel items as the evaluated baselines.AutoGPT discovers substantially fewer items, while ReAct and Reflexion struggle with the abstract open-ended exploration goal without an appropriate curriculum.Source: §3.3 · p. 7
• 50 prompting iterations leave baselines unable to solve any unseen task, whereas Voyager consistently solves all tested tasks.Testing clears inventory and starts a new world; both Voyager and AutoGPT use GPT-4 for subgoal decomposition. Voyager’s learned skill library also improves AutoGPT, demonstrating transferability across methods.Source: Table 2 · p. 8, §3.3 · p. 8, Figure 8 · p. 8

3.4 Ablation Studies

• 93% fewer discovered items result when a random curriculum replaces Voyager’s automatic curriculum.A manually designed curriculum also underperforms and requires Minecraft expertise without adapting to the agent’s live situation. Removing the skill library causes later-stage plateauing, limiting complex actions built from earlier skills.Source: §3.4 · p. 8
• 73% fewer discovered items follow removal of self-verification, the most important feedback type for deciding whether to retry or advance.GPT-4 code generation produces 5.7× more unique items than GPT-3.5. Ablations also examine environment feedback and execution errors, with full Voyager outperforming the alternatives.Source: §3.4 · pp. 8–9, Figure 9 · p. 9

3.5 Multimodal Feedback from Humans

• Human feedback enables Voyager to build complex 3D structures despite its lack of visual perception with the text-only GPT-4 API.Human critics guide code revisions to correct spatial errors, while human curricula break construction into incremental steps. Demonstrated structures include a Nether Portal and a house.Source: §3.5 · p. 9, Figure 10 · p. 9

Cost

• 15× as expensive as GPT-3.5, GPT-4 provides code-generation quality that Voyager requires but GPT-3.5 and open-source LLMs cannot provide.This dependence makes GPT-4 API costs a significant limitation.Source: §4 · p. 10

Inaccuracies

• Voyager can remain stuck generating incorrect skills despite iterative prompting, while self-verification can fail to recognize successful task completion.The automatic curriculum can reattempt failed tasks later.Source: §4 · p. 10

Hallucinations

• Hallucinations produce unachievable curriculum tasks and incorrect code, including nonexistent items, invalid fuel choices, and calls to unavailable control APIs.Examples include requests to craft copper swords or chestplates and using cobblestone as fuel; unavailable API calls cause execution errors.Source: §4 · p. 10

Future Work

• The authors expect improvements in GPT API models and new open-source LLM finetuning techniques to overcome these limitations.Source: §4 · p. 10

Decision-making Agents in Minecraft

• Voyager’s bottom-up, curiosity-driven automatic curriculum enables open-ended exploration, unlike Minecraft planners requiring human interaction or recipe-based subgoals [53] [71].Voyager combines GPT-4 high-level planning with Mineflayer low-level control, following Volum et al.. Earlier approaches include demonstration-based learning, video pre-training in MineDojo and VPT, DreamerV3’s world model, and a success-rate curriculum restricted to curated items.Source: §5 · p. 10

Large Language Models for Agent Planning

• Voyager’s skill library supports increasingly complex behaviors for lifelong learning, whereas the surveyed LLM-based robot planners and text agents lack such a library.Robot-planning precedents include environment feedback in Inner Monologue and executable policies in Code as Policies and ProgPrompt. Text-agent precedents include reasoning and action in ReAct, self-reflection in Reflexion, and memory retrieval in Generative Agents, whose actions are not executable.Source: §5 · p. 10

Code Generation with Execution

• Voyager distinguishes its execution-based code generation through iterative prompting that combines environment feedback, execution errors, and task-success self-verification for embodied control.Earlier approaches use intermediate execution outcomes for program search [88], execution-based majority voting, or LEVER’s learned verifier. CLAIRIFY iteratively provides rule-based verification errors when generating code for chemistry experiments.Source: §5 · p. 11

• VOYAGER exhibits superior performance in item discovery, Minecraft tech-tree progression, terrain traversal, and applying its learned skill library to unseen tasks in a new world.Source: §6 · p. 11
• VOYAGER is the first LLM-powered embodied lifelong learning agent, leveraging GPT-4 for continuous exploration, increasingly sophisticated skill development, and discoveries without human intervention.Source: §6 · p. 11
• VOYAGER offers a starting point for developing powerful generalist agents without tuning model parameters.Source: §6 · p. 11

• Deploying Voyager on physical robots would require additional human attention and safety constraints to ensure responsible and secure deployment.Source: §7 · p. 11
• Voyager is designed to be generally applicable beyond Minecraft, including to robotics.Source: §7 · p. 11
• The research is conducted within Minecraft, a safe and harmless 3D video game environment.Source: §7 · p. 11

A.1 VOYAGER Algorithm

• VOYAGER coordinates curriculum, code generation, execution, self-verification, and skill storage, attempting each task up to four times.The next task depends on the current state and completed/failed tasks; successful code enters the skill library.Source: §A.1 · p. 19

A.2 Prompting

• Each assistant response uses concatenated system and user prompts rather than a multi-round conversation to save tokens.System messages provide overarching behavioral instructions, while user messages specify the immediate response.Source: §A.2 · p. 20

A.3.1 Components in the Prompt

• The curriculum prompt combines diversity constraints, agent state, task history, additional context, and chain-of-thought prompting to reason about progress before proposing tasks.State includes inventory, equipment, nearby/recently seen blocks, entities, chests, biome, time, health, hunger, and position.Source: §A.3.1 · p. 20

A.3.2 Additional Context

• GPT-3.5 supplies additional context through self-asked questions and self-generated answers, optionally grounded in concept-retrieved documents from the wiki knowledge base.The wiki is optional for Minecraft but becomes advantageous when GPT-3.5 lacks pretraining in the target domain.Source: §A.3.2 · p. 20

A.3.3 Warm-up Schedule

• A warm-up schedule progressively introduces agent state and additional context as completed tasks accumulate, moving from basic skills toward more intricate, diverse skills.The same warm-up setting is used across all experiments.Source: §A.3.3 · pp. 20–21

A.3.4 Full Prompt

• Curriculum instructions request one achievable, novel, status-verifiable task, permit repetition only when necessary, and exclude placing, building, planting, and trading.Question prompts request 5–10 self-contained, concept-specific questions. Answer prompts use optional context and Minecraft knowledge, returning Unknown when the answer is not known.Source: §A.3.4 · pp. 22–24

A.4.1 Components in the Prompt

• Code-generation prompts combine Mineflayer control APIs, retrieved skills, previous code, environment feedback, and self-verification critique.The response explains why the previous code failed, plans the solution step by step, and generates code.Source: §A.4.1 · p. 25

A.4.2 Full Prompt

• The code-generation system prompt supplies Mineflayer JavaScript examples and an exploration helper with a maximum duration and callback-based stopping.The helper checks its stopping condition each second and stops when the callback returns a non-null value.Source: §A.4.2 · p. 26
• Each conversation round supplies execution feedback and agent state, requesting an explanation of failures, a step-by-step plan, and executable code.The prompt emphasizes inventory because task-completion checks depend on the final inventory.Source: §A.4.2 · pp. 29–30
• The code-generation prompt requires reusable functions that check inventory, collect missing prerequisites, and compose existing programs.Previously displayed code is not automatically saved or executed. Exploration should use randomized directions, and block searches must stay within 32 blocks.Source: §A.4.2 · p. 30
• A separate prompt generates concise descriptions when adding skills to the library, illustrated through a mining procedure’s prerequisites and actions.Descriptions summarize only the main function in a single line of at most six sentences, excluding function names, chat messages, and helper-function details.Source: §A.4.2 · pp. 31–32

A.4.3 Examples

• Skill-library examples cover crafting wooden planks, mining deepslate below Y=0, smelting iron, and filling a bucket with water.The crafting and smelting examples compose existing skills through calls to mineWoodLog and craftFurnace.Source: §A.4.3 · pp. 32–33
• The fishing example crafts a missing fishing rod, attempts five catches, and retries cancelled fishing attempts without advancing the catch counter.Errors other than fishing cancellation are rethrown.Source: §A.4.3 · p. 34

A.5.1 Components in the Prompt

• The listed prompt components include the agent’s state, an automatic-curriculum task, GPT-3.5 task guidance, chain-of-thought prompting, and few-shot examples for in-context learning [37] [38].The automatic curriculum handles task context through its systematic question-answering mechanism.Source: §A.5.1 · p. 35

A.5.2 Full Prompt

• The verifier treats meeting or exceeding task requirements as success and returns JSON containing reasoning, a success boolean, and critique for failures.Checks use inventory for mining and smelting, nearby blocks for placing or planting, equipment for crafted items, and chest contents for deposits.Source: §A.5.2 · p. 35

A.6 System-level Comparison between VOYAGER and Prior Works

• Voyager alone among the compared methods combines automatic curriculum, iterative planning, and a skill library, while learning Minecraft without gradient updates.Table A.2 presents the system-level comparison with prior works.Source: §A.6 · p. 37, Table A.2 · p. 38

B.1 Experimental Setup

• The simulation combines MineDojo with Mineflayer JavaScript controls, chat-based environment feedback, condition checks, and exception handling for continuous execution.The bot recycles its crafting table and furnace after program execution.Source: §B.1 · p. 38
• When the bot dies, it respawns near the closest ground with its inventory preserved, enabling uninterrupted exploration.Source: §B.1 · p. 38

B.2 Baselines

• ReAct generates reasoning traces and action plans using chain-of-thought prompting, with environment feedback and agent states as observations.It repeatedly performs one round of code generation from scratch followed by three refinement rounds until the prompting limit. All baselines must explore the world and acquire as many items as possible.Source: §B.2 · p. 38
• Reflexion extends ReAct with self-reflection and receives environment feedback, agent states, execution errors, and Voyager’s self-verification module.Like ReAct, it repeatedly performs one round of code generation from scratch followed by three refinement rounds until the prompting limit.Source: §B.2 · p. 38
• AutoGPT lacks Voyager’s skill library, self-verification, and automatic curriculum for accumulating knowledge, assessing task success, and supporting open-ended exploration.Source: §B.2 · p. 38
• Three consecutive subgoals without a new item trigger replanning in the AutoGPT reimplementation.GPT-4 decomposes the goal into subgoals, executed using agent states, environment feedback, and execution errors. Error-free execution counts as completion; otherwise, code receives up to three refinement rounds before advancing to the next subgoal.Source: §B.2 · p. 38

B.3 Ablations

• Curriculum ablations replace automatic task selection with a manually designed diamond-mining sequence or random targets sampled from 101 items obtained by VOYAGER.These belong to six design-choice ablations studying exploration performance. The manual curriculum requires human design effort and is not scalable for open-ended exploration.Source: §B.3 · p. 39
• The skill-library ablation removes the library, eliminating skill retrieval during code generation.Source: §B.3 · p. 39
• The environment-feedback ablation excludes the environment chat log from the code-generation prompt.Source: §B.3 · p. 39
• The execution-error ablation excludes execution errors from the code-generation prompt.Source: §B.3 · p. 39
• The self-verification ablation generates code without self-verification, then refines each task’s program for three rounds, totaling four code-generation rounds.Source: §B.3 · p. 39
• The GPT-3.5 ablation replaces GPT-4 only for code generation, retaining GPT-4 for the automatic curriculum and self-verification.Source: §B.3 · p. 39

B.4.1 Significantly Better Exploration

• 2.3× longer distances are traversed by Voyager compared with baselines while crossing diverse terrains.Figure A.2 presents two bird’s-eye views of the agents’ map coverage.Source: Figure A.2 · p. 41

B.4.2 Extensive Map Traversal

• Map trajectories use positions where agents interact with GPT-4, and Figure 7 summarizes each trajectory using its smallest enclosing circle.Figure A.2 displays the underlying trajectories.Source: §B.4.2 · p. 41, Figure A.2 · p. 41
• Voyager traverses diverse environments across three trials, including deserts, bamboo jungles, dripstone caves, frozen peaks, and oceans.Trial 1 includes desert and bamboo_jungle; Trial 2 includes snowy_plains and frozen_river. Trial 3 includes frozen_peaks, ocean, and stony_shore.Source: §B.4.2 · p. 41
• ReAct traverses plains, desert, and jungle in Trial 1, with different terrain inventories in subsequent trials.Trial 2 covers snowy_plains, snowy_taiga, and snowy_slopes. Trial 3 covers dark_forest, dripstone_caves, grove, and jagged_peaks.Source: §B.4.2 · p. 41
• Reflexion traverses only snowy_taiga in Trial 2, while its other trials cover different terrain combinations.Trial 1 covers plains and flower_forest. Trial 3 covers old_growth_birch_forest, river, ocean, beach, and plains.Source: §B.4.2 · p. 41
• AutoGPT traverses only snowy_taiga in Trial 2, with broader terrain inventories in Trials 1 and 3.Trial 1 covers plains, dripstone_caves, savanna, and meadow. Trial 3 covers plains, stony_shore, forest, and ocean.Source: §B.4.2 · p. 41

B.4.3 Efficient Zero-Shot Generalization to Unseen Tasks

• 50 prompting iterations leave baselines unable to solve either additional unseen task, while VOYAGER solves both.Figure A.3 visualizes intermediate progress on these two tasks.Source: §B.4.3 · pp. 41–42, Figure A.3 · p. 42
• ReAct and Reflexion make no meaningful progress on the two additional unseen tasks and are therefore omitted from Figure A.3.Source: Figure A.3 · p. 42
• The skill library constructed through lifelong learning improves VOYAGER’s performance and also boosts AutoGPT.Source: §B.4.3 · p. 42

B.4.4 Accurate Skill Retrieval

• The top-5 relevant skills are included in the prompt when synthesizing a new skill.Source: §B.4.4 · p. 42
• 96.5% top-5 skill-retrieval accuracy is achieved across 309 evaluated samples.The authors interpret this accuracy as evidence that retrieval is reliable.Source: Table A.4 · p. 42, §B.4.4 · p. 42

B.4.5 Robust to Model Variations

• VOYAGER performs roughly the same with gpt-4-0613 as with gpt-4-0314, supporting robustness to model variations.All main-paper experiments use gpt-4-0314; additional experiments evaluate gpt-4-0613.Source: §B.4.5 · p. 42, Figure A.4 · p. 42