Source-linked AI summary
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, Yitao Liang
TL;DR
Open-world agents need to plan and control across diverse, long-horizon tasks while adapting to changing situations and potentially unlimited task experiences. JARVIS-1 combines a pretrained multimodal language model, goal-conditioned control, and multimodal memory in Minecraft, achieving broad task performance and up to a fivefold improvement over prior records. Its memory-based planning and self-improvement support progressively stronger performance during gameplay, although diamond-related tasks remain constrained by controller execution.
Problem
Open-world agents struggle with changing situations, precise long-horizon tasks, and learning potentially infinite new tasks as gameplay progresses.
Method
JARVIS-1 combines pretrained multimodal language models, goal-conditioned control, multimodal memory, and self-instruct exploration for Minecraft planning and action.
Results
Across more than 200 Minecraft tasks, JARVIS-1 achieves up to a 5× increase over previous records and up to 12.5% success on ObtainDiamondPickaxe.
Takeaways & Limitations
JARVIS-1 demonstrates broad multi-task open-world operation and performance that can increase with game time without additional training.
Takeaways & Limitations
Diamond-related failures often arise from the controller’s inability to perfectly execute short-horizon text instructions generated by the language model.
Abstract
from arXiv · showhide
Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., "chopping trees" to long-horizon tasks, e.g., "obtaining a diamond pickaxe". JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task of $\texttt{ObtainDiamondPickaxe}$, JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. The project page is available at https://craftjarvis.org/JARVIS-1
1. Introduction
JARVIS-1 is introduced as a multimodal Minecraft agent that combines language-based planning, embodied control, and memory to address open-world task diversity and long horizons. It achieves broad task coverage and substantially improves long-horizon performance while supporting self-improvement through gameplay.
- Contribution: JARVIS-1 combines a multimodal language model for planning with a low-level controller to translate user and environmental inputs into Minecraft actions.Its multimodal inputs include visual observations and human instructions.
- Contribution: A multimodal memory stores successful scenarios and plans, allowing JARVIS-1 to retrieve relevant experiences and strengthen planning from its own interactions.The approach uses in-context retrieval rather than additional model updates.
- Evaluation: More than 200 Minecraft tasks evaluate JARVIS-1 without demonstrations, spanning early-game tasks through challenging long-horizon objectives.The benchmark includes tasks such as ObtainCraftingTable and ObtainDiamondPickaxe.
- Results: Up to 5× increase to previous records is reported, and JARVIS-1 reaches up to 12.5% success on obtaining a diamond pickaxe.The paper describes this as the first robust completion of the diamond-pickaxe task at that success rate.
- Results: JARVIS-1’s performance on long-horizon tasks continuously increases as game time progresses without additional training.The paper also explores self-instruct exploration and experience accumulation for life-long learning.
2. Challenges for Open-world Agents
Open-world agents must adapt plans to changing situations, handle precise and lengthy task dependencies, and learn across potentially unlimited tasks. JARVIS-1 addresses these challenges through situation-aware and interactive planning, multimodal memory, and self-improvement mechanisms.
- Challenge I: Situation-Aware Planning: Situation-aware planning is needed because viable plans depend on location, inventory, time, weather, and tool condition.Plans must be updated as the agent’s situation changes during an episode.
- Challenge I: Situation-Aware Planning: A GPT planner that plans only at the beginning failed ObtainDiamondPickaxe, whereas human players and JARVIS-1 used situation-aware planning.Observed failures included entering a new biome and tools breaking.
- Challenge II: Task Complexity: Interactive planning gives JARVIS-1 more significant advantages over GPT as task complexity rises from STONE to IRON to DIAMOND.The framework uses interaction to mitigate long-horizon reasoning and execution difficulties.
- Challenge II: Task Complexity: Open-world tasks require long, precise plans because sub-goals impose exact object names, quantities, and preconditions.ObtainEnchantingTable includes more than 20 sub-goals, including mining three obsidian with a diamond pickaxe.
- Challenge III: Life-long Learning: Potentially infinite open-world tasks require life-long learning, while JARVIS-1 retrieves relevant past experiences from multimodal memory as in-context references.Related tasks can help one another when they require nearly identical materials.
- Challenge III: Life-long Learning: JARVIS-1 combines a memory-augmented multimodal language model, a low-level controller, and self-instruct exploration that expands its experience memory.The architecture uses self-proposed tasks to strengthen planning on previously or partially visited tasks.
3. Multi-task Agent with Memory-Augmented MLM
JARVIS-1 combines multimodal planning, interactive self-correction, and experience retrieval to solve long-horizon Minecraft tasks. Its memory stores situated successful plans and supports progressively improved planning throughout gameplay.
- 3.1. Overview: JARVIS-1 combines an interactive planner, a goal-conditioned controller, and multimodal memory to map observations and instructions into human-aligned actions.The agent first generates a multimodal query, retrieves relevant experiences, and uses them with the planning instruction to prompt its planner.
- 3.2. Interactive Planning: Self-check simulates plan execution and verifies intermediate states against goal preconditions before acting, helping identify flaws proactively.This reduces reliance on encountering errors before recovery or replanning becomes necessary.
- 3.2. Interactive Planning: Environment feedback enables closed-loop recovery: JARVIS-1 explains execution errors, locates bugs in the original plan, and produces an improved plan.The process interleaves self-check before execution with self-explain after failures.
- 3.3. Planning with Multimodal Memory in the Loop: Multimodal memory stores task-and-observation keys with successfully executed plans, allowing multiple situated plans for the same task.Current task and situation determine which prior experiences are relevant for retrieval.
- 3.3. Planning with Multimodal Memory in the Loop: Query generation reasons backward over intermediate sub-goals, then retrieves relevant planning experiences using textual matching and perceptual distance.Reasoning is bounded by a limited depth, and only the top entry for each sub-goal is retrieved.
- 3.4. Self-improving Agents: Memory-augmented planning supports in-context lifelong learning, as accumulated gameplay experiences provide better references for challenging tasks without gradient updates.The paper reports improved success rates on tasks such as ObtainDiamondPickaxe as gameplay continues.
4. Experiments
JARVIS-1 is evaluated on over 200 Minecraft tasks using human-like interfaces, repeated-seed success rates, and comparisons with multi-task LLM agents. It performs best across task groups, improves long-horizon reliability through memory and experience, and benefits from multimodal retrieval and longer gameplay, while controller execution remains a bottleneck.
- Experimental Setups: Over 200 Minecraft Universe Benchmark tasks span early-game through challenging long-horizon objectives, grouped into 11 categories.Tasks use survival mode and cover overworld-obtainable items with varying acquisition requirements.
- Experimental Setups: At least 30 tests per task with different seeds are averaged into success rates, with group means and variances also reported.Agents start in survival mode with empty inventories, and success requires obtaining the target within a specified time.
- Main Results: JARVIS-1 achieves the best performance across all task groups, although success rates decline as technology-tree difficulty and planning horizon increase.Reasoning and feedback-based replanning improve performance, but long-horizon Iron and Diamond tasks remain challenging.
- Main Results: 8.99% vs 2.42%: JARVIS-1 nearly triples DEPS success on diamond-related tasks and typically needs 2–3 replanning rounds instead of more than 6.The comparison attributes the advantage to JARVIS-1’s extensive experience and reports savings in tokens and thinking time.
- Limitations: Diamond-task performance is often limited by the controller’s imperfect execution of short-horizon text instructions generated by the LLM.The paper suggests generating more executable plans or improving controller instruction following.
- Language-Model Ablation: Minecraft-specific knowledge matters: pretrained LLaMA2-70B performs poorly, whereas fine-tuned LLaMA2-13B performs similarly to ChatGPT.The language-model comparison is conducted on Minecraft tasks.
- Ablation on Memory: Memory improves retrieval and planning: reasoning before retrieval helps, and multimodal state retrieval outperforms text-only embedding retrieval.The multimodal state includes visual observations and symbolic information such as inventory and location.
- Long-Horizon Challenges: 6.2% to 12.5%: JARVIS-1’s diamond-pickaxe success improves with longer gameplay, while GPT-generated curricula achieve the best 60-minute performance.VPT changes only from 2.5% to 3% when time increases from 20 to 60 minutes; JARVIS-1 replans when its pickaxe breaks.
5. Related Works
Related work uses language models for high-level action planning and develops Minecraft agents from video-derived representations and policies. These approaches support instruction following or short-horizon tasks, but prior agents have limitations on complicated process-oriented tasks.
- LLM-Based Planning: Prior embodied-agent methods use LLMs to decompose commands, jointly decode feasible robot plans, or generate executable action sequences.Examples include text completion, semantic translation, and language-model-weighted planning.
- Minecraft Agents: Minecraft research includes MineCLIP, VPT, and Steve-1, which derive visual-semantic alignment, imitation policies, or instruction-following behavior from online videos.Steve-1 combines MineCLIP and VPT with hindsight relabeling and unCLIP techniques.
- Minecraft Agents: Steve-1 addresses open-world short-horizon tasks but cannot solve complicated process-oriented tasks because of limited goal-space expressivity.This distinguishes its capability boundary from the broader planning requirements of Minecraft.
6. Conclusion
JARVIS-1 combines multimodal language-model planning, embodied action translation, and multimodal memory for human-like planning in Minecraft. It performs across more than 200 tasks and reaches up to five times VPT’s completion rate on the long-horizon diamond-pickaxe task.
- Conclusion: JARVIS-1 combines a pretrained multimodal language model with a goal-conditioned controller to interpret multimodal inputs and translate plans into actions.Its multimodal memory draws on pretrained knowledge and real-time game experiences.
- Conclusion: Across a wide array of Minecraft tasks, JARVIS-1 demonstrates broad task performance, including a long-horizon diamond-pickaxe completion rate up to five times VPT’s.The conclusion presents this comparison as evidence of progress toward versatile agents in complex virtual environments.
A.1. Controller
JARVIS-1 uses a multimodal language model to plan, self-check, and self-explain, while controllers execute structured mine, craft, and smelt goals. Its memory stores successful task trajectories and supports planning from prior experiences.
- Controller: The controller represents Minecraft tasks as structured mine, craft, and smelt actions with objects, quantities, materials, tools, and execution ranks.Examples include mining logs, crafting planks, and parsing action names, conditions, tools, and ranks from plans.
- Interactive Planner: JARVIS-1’s multimodal language model accepts visual, language, and symbolic inputs for planning, self-checking, and self-explaining.Symbolic inputs include inventory, location, home, and current life statistics; all modalities are converted into language for GPT processing.
- Interactive Planner: JARVIS-1 uses separate prompts for goal parsing, self-explaining failed actions, and checking whether plans can be completed.Failure explanations connect missing tools or materials to prerequisite mining and crafting steps.
- Controller: The planning examples decompose wooden-tool tasks into ordered mining and crafting steps, updating inventory quantities after each operation.The demonstrated sequence mines logs, crafts planks and sticks, and then crafts a wooden tool.
- Memory: JARVIS-1’s memory stores each successful trajectory as a dictionary containing the task, completed state, and executed plan.The state includes visual observation and symbolic information returned by the environment.
B.1. Observation Space
JARVIS-1 observes Minecraft through raw game pixels and auxiliary environmental information. It excludes high-level voxel and lidar observations, relying on information available through human-like interaction.
- Observation Space: The observation space combines raw Minecraft pixels with auxiliary information about the agent’s environment.Pixels include the hotbar, health indicators, and hand animations; auxiliary information includes location and weather.
- Observation Space: Table 3 specifies the observation space used in Minecraft.
- Observation Space: JARVIS-1 cannot access high-level observations such as voxels or lidar during inference.The controller perceives raw pixels and interacts with the environment, while environmental information is used to generate text conditions.
B.2. Action Space
JARVIS-1 uses a hybrid action space combining human-like keyboard and mouse controls with additional environment actions. Mouse behavior changes depending on whether in-game interfaces are open.
- Action Space: The hybrid action space includes human-like keypresses, mouse movements, and clicks.Binary functional actions include forward, jump, use, and attack.
- Action Space: Mouse X and Y control yaw and pitch when Minecraft graphical interfaces are closed.When a GUI is open, camera actions instead move the cursor within the interface.
- Action Space: Table 4 presents the action space used in Minecraft.
B.3. Rules
Experiments run Minecraft 1.16.5 in survival mode with controlled environment rules. These rules fix peaceful daytime gameplay, preserve inventory after death, and add night vision for visualization.
- Rules: The evaluation uses survival mode in Minecraft 1.16.5.
- Rules: The environment is set to peaceful difficulty and permanent daytime.The daylight cycle is disabled through a game rule.
- Rules: Agents retain their inventory after death and respawn at the spawn point within each task’s time limit.
- Rules: Night vision is enabled to facilitate displaying agent behavior.
C. Results and Details of 200+ tasks in Minecraft Universe Benchmark
The benchmark evaluates JARVIS-1 across more than 200 Minecraft tasks grouped by item and activity categories, using standardized Survival Mode conditions. Evaluation records task specifications, evaluation times across seeds, and successful-episode rates.
- The evaluation task set documents each task’s name, maximum steps, initial inventory, biome, and language instruction.
- All tasks are evaluated in Minecraft 1.16.5 Survival Mode.
- Evaluation reports include evaluation times across different seeds and successful-episode rates.
- The benchmark organizes task results into Wood, Stone, Iron, Gold, Diamond, Redstone, Blocks, Armor, Decoration, and Food groups.
- Separate result tables report JARVIS-1’s performance for the Blocks, Armor, Decoration, and Food groups.