Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

• Figure 1 illustrates Optimus-1 performing long-horizon tasks in Minecraft, while its caption introduces a task for the Knowledge-Guided Planner without establishing that the task was completed.Source: Figure 1 · p. 1
• The Knowledge-Guided Planner uses knowledge from the Hierarchical Directed Knowledge Graph, and the Action Controller executes planning sequences step by step.Source: Optimus-1 · p. 1, Figure 1 · p. 1
• During execution, the Experience-Driven Reflector periodically retrieves experience from the Abstracted Multimodal Experience Pool to reflect.Source: Optimus-1 · p. 1

1 Introduction

• Existing agents still fall short on long-horizon tasks; the introduction identifies insufficient structured Minecraft knowledge and lack of multimodal experience as challenges.It relates long-term human memory to knowledge and experience that guide behavior and adaptation.Source: §1 · p. 2
• Hybrid Multimodal Memory combines HDKG, which represents object relationships as a directed graph, with AMEP, which stores multimodal histories and includes both successful and failed cases.The introduction contrasts AMEP with approaches that store only successful cases.Source: §1 · pp. 2–3
• Optimus-1 combines a Knowledge-Guided Planner, Experience-Driven Reflector, and Action Controller to form sub-goals, execute low-level actions, and retrieve relevant experience for plan refinement.Source: §1 · p. 3
• Up to 30% improvement over existing agents on long-horizon tasks; with Hybrid Multimodal Memory, Optimus-1 shows 2 to 6 times performance improvement and outperforms GPT-4V on many tasks.Source: §1 · p. 3
• Optimus-1 outperforms all baseline agents on long-horizon task benchmarks and exhibits capabilities close to human players.Source: §1 · p. 3

2.1 Hybrid Multimodal Memory

• The Hybrid Multimodal Memory pairs a Hierarchical Directed Knowledge Graph with an Abstracted Multimodal Experience Pool to store knowledge and experience.The graph supports explicit knowledge retrieval, while the pool stores multimodal task experience.Source: §2.1 · p. 3

2.1.1 Abstracted Multimodal Experience Pool

• AMEP filters video at 1 frame per second, dynamically selects frames in a 16-frame image buffer, and uses MineCLIP [7] to align images with textual sub-goals.It stores aligned experience with environment information, the agent’s initial state, and the planner’s plan.Source: §2.1.1 · p. 4

2.1.2 Hierarchical Directed Knowledge Graph

• HDKG represents objects as graph nodes and directed edges as craftable materials, enabling explicit knowledge retrieval and topological ordering of task materials.The planner uses the resulting material relationships to form a sequence of sub-goals without fine-tuning [32] [58].Source: §2.1.2 · pp. 4–5

2.2 Optimus-1: Framework

• The Knowledge-Guided Planner uses visual observations and HDKG knowledge to generate sub-goals; the Action Controller turns each sub-goal and observation into low-level actions.The Experience-Driven Reflector periodically retrieves AMEP experience and returns COMPLETE, CONTINUE, or REPLAN; REPLAN sends the task back for revision.Source: §2.2 · pp. 5–6
• AMEP retains successful and failed cases as reflection references, and experiments report that the Experience-Driven Reflector significantly enhances long-horizon task success.The paper also reports that incorporating failure cases into in-context learning is effective.Source: §2.2 · p. 6

2.3 Non-parametric Learning of Hybrid Multimodal Memory

• The “free exploration-teacher guidance” method expands shared HDKG and AMEP memory through exploration and guided tasks, then alternates learning with unseen-task inference without parameter updates.The memory grows across iterations, supporting mastery of tasks from easy to hard.Source: §2.3 · p. 6

3.1 Experiments Setting

• The evaluation uses MineRL with Minecraft 1.16.5, a 67-task benchmark across seven groups, and averages success over at least 30 runs per task.The agent starts in survival mode with an empty inventory; average steps and completion time are also measured.Source: §3.1 · pp. 7–8

3.2 Experimental Results

• Optimus-1 achieves near-100% success on the Wood Group, improves over Jarvis-1 by 29.28% on Diamond and 53.40% on Redstone, and averages 5.37% improvement toward human levels.It also achieves the best performance and shortest elapsed time among all task groups.Source: §3.2 · p. 8

3.3 Ablation Study

• Removing the Knowledge-Guided Planner or Experience-Driven Reflector sharply reduces performance; removing HDKG lowers average performance 20%, and removing AMEP lowers it 12%.Without AMEP case retrieval, success falls by an average of 10% across groups; the reflection mechanism considers both success and failure cases.Source: §3.3 · pp. 8–9

3.4 Generalization Ability

• With Hybrid Multimodal Memory, Deepseek-VL and InternLM-XComposer2-VL-based Optimus-1 improve performance by 2 to 6 times across task groups and outperform GPT-4V.The original MLLMs perform poorly on long-horizon tasks, which the paper attributes to insufficient Minecraft knowledge and experience.Source: §3.4 · p. 9, Figure 5 · p. 9

3.5 Self-Evolution via Hybrid Multimodal Memory

• Across four learning epochs, Optimus-1’s success rate on unseen tasks rises as its memory expands through free exploration and teacher guidance.The setup uses 160 learning tasks: 150 for free exploration and 10 for teacher guidance in each period.Source: §3.5 · p. 9, Figure 5 · p. 9

4.1 Agents in Minecraft

• Earlier Minecraft agents struggle with complex tasks due to limitations in instruction comprehension and planning or insufficiently detailed visual information.Optimus-1 uses observations directly as situation-aware planning conditions and HDKG to produce a complete plan in one step.Source: §4.1 · pp. 9–10

4.2 Memory in Agents

• LLM-agent memory has textual and parametric forms, while recent MLLM work explores long-term visual storage and summarization; Optimus-1’s plug-and-play memory provides world knowledge and multimodal experience.Source: §4.2 · p. 10

• Hybrid Multimodal Memory pairs HDKG world knowledge for planning with AMEP historical experience for reflection.Source: §5 · p. 10
• Optimus-1, built on this memory, outperforms existing agents on long-horizon Minecraft tasks.Source: §5 · p. 10
• General-purpose MLLMs using Hybrid Multimodal Memory can exceed GPT-4V without additional parameter updates.The paper describes this as a step toward a general agent with human-like performance.Source: §5 · p. 10

6 Limitation and Future Work

• STEVE-1’s instruction-following and complex-action limits constrain Optimus-1 on tasks such as defeating the Ender Dragon and building a house.The paper identifies improving the action controller as a future research direction.Source: §6 · p. 10

A Broader Impact

• The authors identify MLLM text-generation stochasticity as a deployment risk and call for comprehensive risk assessment before use with Hybrid Multimodal Memory.They anticipate possible applications in robotics, video games, and the web.Source: A Broader Impact · p. 15

B Minecraft

• Minecraft is an open, dynamic environment with 79 biomes, where tasks can require many dependent sub-goals, making it a challenging long-horizon benchmark.The paper contrasts its diversity and complexity with web/app navigation and embodied manipulation.Source: B Minecraft · p. 15

B.1 Basic Rules

• Minecraft gameplay spans broad activities, while time adds survival pressure: a game day lasts 20 real-world minutes and night brings dangerous mobs.The paper also describes progression through crafting, mining, farming, and other activities.Source: §B.1 · pp. 15–16

More Challenge than Diamond

• Progression beyond the Overworld proceeds through the Nether to the End, where defeating the Ender Dragon unlocks the outer End Islands.The outer islands offer elytra and shulker boxes.Source: More Challenge than Diamond · p. 16

B.2 Observation and Action Spaces

• The agent receives 640 × 360 RGB observations with night vision; its controls use mouse and keyboard, with crafting and smelting abstracted as actions.Precise mouse movement matters for tasks involving inventories or crafting tables.Source: §B.2 · pp. 16–17, Table 4 · p. 17

B.3 Long-horizon Tasks

• Long-horizon tasks require world knowledge and multiple indispensable subtasks, with tool tiers determining which blocks can be mined.Wooden tools cannot mine iron-level blocks, while diamond tools can mine any level.Source: §B.3 · p. 17
• Crafting an iron sword requires a sequence of tools, a crafting table, a furnace, raw materials, and two iron ingots.Figure 7 depicts the task, which can take thousands of steps and requires error-free mouse movements for precise crafting and smelting.Source: §B.3 · p. 17, Figure 7 · p. 18

C Theory

• The paper distinguishes semantic memory for conceptual knowledge from episodic memory for personal experiences tied to particular times and places.It uses this distinction to motivate an agent memory divided into knowledge and experience, implemented as HDKG and AMEP.Source: C Theory · p. 17

D.1 Benchmark

• The benchmark contains 67 tasks in seven groups, starts agents with empty inventories, and sets task-specific maximum steps based on human completion times.Diamond generation is increased from its usual probability to 20% to reduce its effect on task completion; the setting also applies to human players.Source: §D.1 · pp. 18–19, Table 5 · p. 19

D.2 Baselines

• The study compares against GPT-3.5, GPT-4V, Minecraft agents, and a human baseline averaged across 10 volunteers.Volunteers had at least 20 hours of Minecraft experience and each performed every benchmark task 10 times.Source: §D.2 · p. 19, Table 7 · p. 20

D.3 Minecraft Agents

• Earlier agents include Transformer-based policy networks and MLLM-based planners, while agent environments differ in whether tasks use APIs or low-level actions.The paper notes that low-level action execution in MineRL is more challenging for long-horizon tasks than API-based sub-goal completion.Source: §D.3 · p. 20
• Optimus-1 significantly outperforms existing agents on the long-horizon benchmark and approaches human-level performance on many tasks.The paper presents this comparison as evidence of the agent’s performance across challenging Minecraft tasks.Source: §D.3 · p. 20

E.1.1 Abstracted Multimodal Experience Pool

• AMEP summarizes task histories into multimodal experiences, addressing the storage and retrieval burden of retaining all visual history while preserving long sequential information.Visual frames are filtered at one frame per second, reduced to a 16-image buffer, and linked to sub-goals using MineCLIP; both successful and failed cases can be retrieved for reflection.Source: §E.1.1 · pp. 20–21

E.1.2 Hierarchical Directed Knowledge Graph

• HDKG represents object and crafting relationships as a directed graph, then retrieves an object’s subgraph and topologically sorts it into required materials and antecedents for planning.The graph can be expanded through local updates rather than rebuilding the entire structure.Source: §E.1.2 · p. 21

E.2 Hybrid Multimodal Memory Driven Optimus-1

• The non-parametric “free exploration-teacher guidance” method grows memory through atomic task attempts and guided completion of longer tasks.Successful free exploration can add crafting relations to HDKG, while attempts can record experience whether they succeed or fail; teacher guidance adds advanced knowledge and multimodal experience.Source: §E.2 · pp. 21–22

E.3 Backbone of Optimus-1

• Optimus-1 uses GPT-4V for planning and reflection with STEVE-1 for action control, and also evaluates Deepseek-VL and InternLM-XComposer2-VL as MLLM backbones.The authors report comparable performance with the open-source backbones as a lower-cost alternative; inference took about 20 hours on four A100 80G GPUs.Source: §E.3 · p. 22

E.4 Prompt for Optimus-1

• The prompts ask the planner to combine task, visual-state information, and a craft graph, while the reflector classifies situations as complete, continue, or replan.The reflector prompt also asks it to identify the environment and, for replanning, distinguish predicaments such as being in water or trapped.Source: §E.4 · p. 23

F.1 Full Results on Our Benchmark

• The full benchmark reports task-level SR, AS, and AT for Wood and Stone tasks, evaluated with empty inventories and random starts in Minecraft 1.16.5 Survival Mode.Tables 8 and 9 cover the Wood and Stone groups.Source: §F.1 · p. 23, Table 8 · p. 24, Table 9 · p. 24
• Optimus-1’s average task completion step (AS) is significantly lower than other baselines.MineRL steps count agent-environment interactions at 20 per second.Source: §F.1 · p. 24

F.2 Results on Other Benchmark

• Comparisons with other agents involve different environments and task controls, including low-level actions for Optimus-1 in MineRL.Voyager uses encapsulated code in Mineflayer, while MP5 needs only a specific crafting control signal in MineDOJO.Source: §F.2 · p. 24
• Optimus-1 outperforms a variety of powerful baseline agents on the reported external benchmarks.Table 15 compares process-dependent tasks with MP5, and Table 16 reports success rates across eight META TASK groups.Source: §F.2 · p. 24, Table 15 · p. 27, Table 16 · p. 28

G Case Study

• Reflection replans after craft failure, ends completed tasks, and continues tasks that remain incomplete but are progressing.The examples include replanning after a failed wooden-pickaxe plan and continuing to chop toward the log goal.Source: G Case Study · p. 29, Figure 9 · p. 29, Figure 11 · p. 30
• HDKG supports one-step planning, while visual signals can change the immediate subgoal, such as leaving a cave before resuming a diamond-mining plan.Figures 12 and 13 illustrate knowledge-based planning adjusted to current visual information.Source: G Case Study · p. 29
• With a zombie approaching, Optimus-1 plans in one step, whereas MP5 and Voyager iteratively generate plans without acting during planning.The example describes the approaching zombie as a danger to the agent.Source: G Case Study · p. 29, Figure 13 · p. 31