PaperlayerView exact revision on arXiv

Source-linked AI summary

Voyager: An Open-Ended Embodied Agent with Large Language Models

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar

arXiv:2305.16291v2cs.AIcs.LG

TL;DR

Embodied agents need to explore open-ended worlds while acquiring and transferring skills over long time spans, a capability existing LLM-based agents do not provide. Voyager addresses this with an automatic curriculum, reusable executable programs, and iterative GPT-4 code refinement, achieving stronger Minecraft exploration and generalization than prior methods. It also transfers its learned skill library to novel tasks in a new world.

  • Problem

    Existing LLM-based embodied agents generate plans or executable policies but lack demonstrated lifelong acquisition, accumulation, updating, and transfer of knowledge over extended periods.

  • Method

    Voyager uses an automatic curriculum, an executable skill library, and iterative prompting that incorporates environment feedback, execution errors, and self-verification while querying GPT-4 without parameter fine-tuning.

  • Results

    Voyager outperforms prior state-of-the-art methods in Minecraft, obtaining 3.3× more unique items, traveling 2.3× longer distances, and unlocking key tech-tree milestones up to 15.3× faster.

  • Takeaways & Limitations

    Voyager’s learned skill library transfers to a newly instantiated Minecraft world, where it solves unseen tasks from scratch while baselines solve none within 50 prompting iterations.

  • Takeaways & Limitations

    The agent can become stuck generating skills, misidentify success, propose impossible tasks, hallucinate invalid game operations, and call unavailable control functions.

Abstract

from arXiv · show

We introduce Voyager, the first LLM-powered embodied lifelong learning agent in Minecraft that continuously explores the world, acquires diverse skills, and makes novel discoveries without human intervention. Voyager consists of three key components: 1) an automatic curriculum that maximizes exploration, 2) an ever-growing skill library of executable code for storing and retrieving complex behaviors, and 3) a new iterative prompting mechanism that incorporates environment feedback, execution errors, and self-verification for program improvement. Voyager interacts with GPT-4 via blackbox queries, which bypasses the need for model parameter fine-tuning. The skills developed by Voyager are temporally extended, interpretable, and compositional, which compounds the agent's abilities rapidly and alleviates catastrophic forgetting. Empirically, Voyager shows strong in-context lifelong learning capability and exhibits exceptional proficiency in playing Minecraft. It obtains 3.3x more unique items, travels 2.3x longer distances, and unlocks key tech tree milestones up to 15.3x faster than prior SOTA. Voyager is able to utilize the learned skill library in a new Minecraft world to solve novel tasks from scratch, while other techniques struggle to generalize. We open-source our full codebase and prompts at https://voyager.minedojo.org/.

1 Introduction

Voyager addresses the challenge of building embodied lifelong learners by combining exploration, reusable executable skills, and iterative code refinement in Minecraft. It achieves stronger exploration, tech-tree progression, traversal, and generalization than prior LLM-based agents.

  • LLM-based embodied agents generate plans or policies but generally do not progressively acquire, accumulate, update, and transfer knowledge over extended periods.
  • Voyager combines an automatic curriculum, an executable skill library, and iterative prompting to support continuous exploration and lifelong learning.It uses code as the action space for temporally extended and compositional behaviors while querying GPT-4 through prompting rather than parameter fine-tuning.
  • Voyager proposes progressively harder tasks based on exploration progress and agent state, with the overarching goal of discovering diverse things.
  • Voyager refines generated programs using environment observations, execution errors, and self-verification before storing successful programs as reusable skills.
  • Voyager obtains 3.3× more unique items, travels 2.3× longer distances, and unlocks key Minecraft tech-tree milestones up to 15.3× faster than prior state-of-the-art methods.It also uses its learned skill library in a new world to solve novel tasks from scratch, while other methods struggle to generalize.

2 Method

Voyager organizes open-ended Minecraft learning around an automatic curriculum, a reusable executable skill library, and iterative prompting with feedback and verification. These components continually propose tasks, retrieve and compose prior behaviors, and refine programs until tasks are validated or abandoned after repeated failure.

  • Automatic Curriculum: Voyager uses an automatic curriculum to propose objectives for open-ended exploration and increasingly complex tasks.GPT-4 receives directives, the agent’s state, prior successes and failures, and exploration context when generating tasks.
  • Skill Library: Skill retrieval queries the library using self-generated task plans and environment feedback, returning relevant prior skills for code generation.The figure describes retrieving the top-5 relevant skills from a vector database.
  • Automatic Curriculum: The curriculum encourages diverse behaviors while considering the agent’s resources, learned skills, current state, and exploration progress.
  • Skill Library: Voyager represents skills as executable programs for temporally extended actions and stores them in an indexed library for later reuse.Programs are indexed by embeddings of their descriptions, allowing complex skills to be synthesized by composing simpler programs.
  • Iterative Prompting Mechanism: Voyager iteratively refines generated code using environment feedback, interpreter errors, and self-verification of task completion.The prompt also includes control APIs, retrieved skills, the previous program, critique, and the agent’s current state.
  • Iterative Prompting Mechanism: A separate GPT-4 verifier checks whether the task is complete and critiques failures, after which successful programs enter the skill library and a new objective is requested.If the agent remains stuck after 4 code-generation rounds, the curriculum supplies another task.

3 Experiments

VOYAGER is evaluated against adapted LLM-agent baselines in Minecraft across exploration, tech-tree mastery, map coverage, and zero-shot generalization, with ablations of its design choices. It consistently outperforms alternatives, while the ablations identify the automatic curriculum, skill library, feedback mechanisms, and GPT-4 as important contributors.

  • VOYAGER discovers 63 unique items within 160 prompting iterations, achieving 3.3× more novel items than its counterparts.
  • VOYAGER unlocks the wooden, stone, and iron tech-tree levels 15.3×, 8.5×, and 6.4× faster than baselines, respectively, and uniquely reaches the diamond level.
  • VOYAGER traverses distances 2.3× longer than baselines across diverse terrains, whereas baseline agents often remain confined to local areas.
  • In a newly instantiated world with cleared inventory, VOYAGER consistently solves unseen tasks, while baselines solve none within 50 prompting iterations.
  • Ablation Studies: Replacing the automatic curriculum with a random one reduces discovered item count by 93%, while removing the skill library causes later-stage performance to plateau.
  • Ablation Studies: Removing self-verification reduces discovered item count by 73%, and GPT-4 obtains 5.7× more unique items than GPT-3.5 in code generation.

4 Limitations and Future Work

VOYAGER’s main limitations are substantial API cost, occasional failures in skill generation and verification, and hallucinated tasks or code. The authors identify improved models and open-source fine-tuning as possible future remedies.

  • GPT-4 costs 15× more than GPT-3.5, although VOYAGER relies on GPT-4’s higher code-generation quality.The paper reports that GPT-3.5 and open-source LLMs cannot provide the required quality.
  • Despite iterative prompting, VOYAGER sometimes gets stuck, fails to generate the correct skill, or must reattempt the task later.The automatic curriculum can defer and reattempt unsuccessful tasks.
  • The curriculum and code generator sometimes hallucinate impossible tasks, invalid game mechanics, or unavailable control functions.Examples include requesting nonexistent copper equipment, using cobblestone as fuel, and calling absent APIs.
  • The authors expect improved GPT APIs and fine-tuning techniques for open-source LLMs to address these limitations.

5 Related work

Related work spans low-level Minecraft controllers, high-level LLM planners, and execution-guided code generation. VOYAGER combines high-level GPT-4 planning with executable programs and environment-informed refinement in Minecraft.

  • Decision-making Agents in Minecraft: Minecraft decision-making agents include low-level controllers trained with reinforcement learning or demonstrations and high-level planners that generate executable policies.Prior approaches include hierarchical reinforcement learning, video pretraining, world models, and few-shot Codex prompting.
  • Decision-making Agents in Minecraft: Recent Minecraft LLM planners decompose tasks into recipe-based subgoals, while VOYAGER pursues broader exploration through GPT-4 planning.
  • Large Language Models for Agent Planning: LLM planning research covers robot learning, executable policies, multimodal fine-tuning, and text agents using reasoning, reflection, or automated subgoals.
  • Code Generation with Execution: Execution-guided code-generation methods use intermediate outcomes, candidate voting, or verifiers to identify and improve programs.The cited approaches include execution-guided search, majority voting, LEVER, and CLAIRIFY.

6 Conclusion

VOYAGER is presented as an LLM-powered embodied lifelong learning agent that continuously explores Minecraft, develops increasingly sophisticated skills, and makes discoveries without human intervention. The paper positions it as a starting point for generalist agents without model-parameter tuning.

  • VOYAGER uses GPT-4 to explore Minecraft continuously, develop increasingly sophisticated skills, and make new discoveries without human intervention.
  • The paper reports superior performance in discovering items, unlocking the tech tree, traversing diverse terrain, and applying learned skills to unseen tasks in a new world.
  • VOYAGER is proposed as a starting point for developing powerful generalist agents without tuning model parameters.

7 Broader Impacts

The study is conducted in Minecraft, a safe and harmless 3D video-game environment. Applying VOYAGER to physical robots would require human-implemented safety constraints.

  • The research evaluates VOYAGER within Minecraft, a safe and harmless 3D video-game environment.
  • Although VOYAGER is designed to be generally applicable to robotics, physical deployment would require additional attention and human-implemented safety constraints.

A Method

VOYAGER repeatedly proposes tasks from exploration progress, generates executable code, and refines it using environment feedback, errors, and self-verification. Its prompts incorporate agent state and progressively increasing context to guide diverse, achievable tasks.

  • Algorithm: VOYAGER’s algorithm resets the environment, tracks exploration progress, proposes a task, generates code, executes it, and checks success.The loop retries code generation and execution for at most four rounds before moving to another task.
  • Automatic curriculum: The automatic curriculum uses completed tasks, failed tasks, and exploration progress to propose progressively harder objectives.GPT-4 generates tasks around the overarching goal of discovering diverse things while keeping tasks achievable.
  • Prompt construction: GPT-4 receives directives, agent state, and additional context before suggesting the next Minecraft task.State includes inventory, equipment, nearby blocks, recently seen blocks, and nearby entities; prompts can incorporate wiki-derived question-and-answer context.
  • Prompt construction: A warm-up schedule gradually adds agent state and additional context as tasks are completed, moving from basic skills toward intricate and diverse ones.The schedule is used across the experiments to expose GPT-4 to increasing information during exploration.
  • Prompt construction: The curriculum prompt asks for specific, self-contained questions, while the task prompt requires one concise, novel, manageable action.Examples include mining, crafting, smelting, cooking, killing mobs, and equipping an item.

B.1 Experimental Setup

Experiments use a MineDojo-based simulation with Mineflayer controls and compare VOYAGER against ReAct, Reflexion, and AutoGPT on exploration tasks. The setup evaluates item collection and map coverage using repeated trials.

  • Environment: VOYAGER’s simulation environment is built on MineDojo and uses Mineflayer JavaScript APIs for motor controls and environment feedback.The environment also preserves inventory after death and recycles crafting tables and furnaces after execution.
  • Baselines: The comparison includes ReAct, Reflexion, and AutoGPT, with baselines receiving environment feedback, agent states, and, where specified, execution errors.ReAct and Reflexion use code generation followed by refinement rounds; AutoGPT performs task decomposition and subgoal execution.
  • Evaluation: All baselines perform the task “explore the world and get as many items as possible.”The setup therefore compares open-ended exploration under a shared objective.

B.3 Ablations

The ablation study removes or replaces six VOYAGER design choices to measure their effects on exploration performance. Variants include manual and random curricula, removal of the skill library, and omission of feedback signals.

  • Ablation design: The study ablates automatic curriculum, skill library, environment feedback, execution errors, self-verification, and GPT-4 code generation.These six choices are evaluated for their impact on exploration performance.
  • Curriculum variants: The manual-curriculum variant follows a fixed sequence for mining a diamond, from collecting wood through crafting tools and mining diamond.The sequence requires human effort to design and is described as not scalable for open-ended exploration.
  • Ablation variants: The random-curriculum variant randomly selects the next task from 101 items obtained by VOYAGER.Other variants remove skill retrieval, environment feedback, or execution errors from code generation.

B.4.1 Significantly Better Exploration

VOYAGER explores more broadly than the compared agents in the supplied map-coverage and item-collection evidence. Its trajectories span diverse terrains, while the baselines’ listed trials generally contain fewer or different terrain types.

  • Item collection: The item-collection section reports separate trial-level inventories for VOYAGER, ReAct, Reflexion, and AutoGPT.The supplied passages enumerate the items collected by each method across three trials.
  • Map coverage: 2.3× longer distances: VOYAGER traverses farther than baselines while crossing diverse terrains.Figure A.2 plots trajectories based on positions where agents interact with GPT-4.
  • Map coverage: VOYAGER’s three trials span 8, 5, and 12 listed terrain types, respectively.The terrains include forests, rivers, snowy regions, caves, beaches, oceans, and plains across the trials.
  • Map coverage: ReAct’s three trials list 3, 3, and 4 terrain types, while Reflexion’s list 2, 1, and 5.These trial-level terrain lists provide the supplied baseline comparison for map coverage.
  • Map coverage: AutoGPT’s three trials list 4, 1, and 4 terrain types, compared with VOYAGER’s broader listed terrain coverage.The comparison is based on the terrain categories reported for each trial.

B.4.3 Efficient Zero-Shot Generalization to Unseen Tasks

Voyager generalizes its lifelong-learned skills to unseen Minecraft tasks, consistently solving them while baselines make no meaningful progress. Its skill retrieval remains reliable, and performance is roughly unchanged across GPT-4 model variants.

  • Voyager consistently solves unseen tasks, while baselines are unable to solve any task within 50 prompting iterations.
  • Voyager’s lifelong-learning skill library also boosts AutoGPT’s performance on unseen tasks.
  • 96.5% top-5 skill-retrieval accuracy indicates that Voyager reliably retrieves relevant skills for synthesizing new ones.The evaluation contains 309 samples, and the top five retrieved skills are included in the synthesis prompt.
  • Voyager’s performance is roughly the same with GPT-4-0314 and GPT-4-0613, indicating robustness to these model variations.
Loading 2305.16291v2…