PaperlayerView exact revision on arXiv

Source-linked AI summary

MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception

Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, Jing Shao

arXiv:2312.07472v4cs.CV

TL;DR

Long-horizon open-world tasks require both ordered sub-objectives and context-aware execution, challenges that existing approaches struggle to address together. MP5 coordinates five modules and goal-conditioned active perception for embodied planning and action in Minecraft. It reports 22% success on diamond-level tasks and 91% on tasks requiring complex scene understanding.

  • Problem

    Long-horizon open-world tasks require both process-dependent decomposition and context-dependent execution, while recent LLM approaches may assume agents have complete knowledge of their state and environment.

    Source: §1 · p. 2

  • Method

    MP5 coordinates five modules, including a Minecraft MLLM Percipient and multi-round active perception, to support task decomposition, situation-aware planning, and embodied execution.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    MP5 reports 22% success on diamond-level tasks and 91% on tasks requiring complex scene understanding.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that MP5 handles context- and process-dependent tasks in an open-ended environment and can address novel tasks.

    Source: §1 · p. 2, §5 · p. 8

    MP5 relies on GPT-3.5-turbo or GPT-4 APIs, and its evaluation has not been extended beyond Minecraft.

    Source: §5 · p. 8

Abstract

from arXiv · show

It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways. However, existing approaches usually struggle with compound difficulties caused by the logic-aware decomposition and context-aware execution of these tasks. To this end, we introduce MP5, an open-ended multimodal embodied system built upon the challenging Minecraft simulator, which can decompose feasible sub-objectives, design sophisticated situation-aware plans, and perform embodied action control, with frequent communication with a goal-conditioned active perception scheme. Specifically, MP5 is developed on top of recent advances in Multimodal Large Language Models (MLLMs), and the system is modulated into functional modules that can be scheduled and collaborated to ultimately solve pre-defined context- and process-dependent tasks. Extensive experiments prove that MP5 can achieve a 22% success rate on difficult process-dependent tasks and a 91% success rate on tasks that heavily depend on the context. Moreover, MP5 exhibits a remarkable ability to address many open-ended tasks that are entirely novel.

1. Introduction

MP5 targets long-horizon Minecraft tasks that require both ordered sub-objectives and context-aware perception. Its five-module system uses active perception to support planning and execution, achieving reported success on difficult process-dependent and context-dependent tasks.

1. Introduction

  • Existing LLM agents can decompose long-horizon tasks but often assume agents have complete knowledge of their state and environment.MP5’s motivation is to add open-ended, selective perception tailored to planning or execution and compatible with other modules.Source: §1 · p. 2
  • MP5 combines Parser, Percipient, Planner, Performer, and Patroller to decompose tasks, answer image questions, schedule and execute actions, and verify plans or actions.Parser and Planner use external Memory; Percipient is a LoRA-enabled MLLM.Source: §1 · p. 2
  • Multi-round interaction between Percipient and Patroller gathers visual context in response to Planner and Performer queries.Patroller relays feedback to the Planner and Performer, supporting situation-aware planning and more correct embodied execution.Source: §1 · p. 2
  • 22% success on diamond-level tasks and 91% on tasks requiring complex scene understanding demonstrate performance on difficult process-dependent and context-dependent tasks.The latter tasks require perceiving around 4–6 key items in observed images.Source: §1 · p. 2

2. Related Work

The related work spans multimodal language models and Minecraft agents, including methods for open-ended tasks and long-horizon planning. MP5 is positioned as an embodied agent that uses task-conditioned perception alongside hierarchical reasoning.

2.1. Multi-modal Large Language Models

  • MineLLM is designed for Minecraft and supplies Percipient with perception, interaction, and analysis capabilities for objective-conditioned active perception.The paper describes MLLMs including LLaVA, InstructBLIP, and LAMM as examples in the broader model landscape.Source: §2.1 · p. 2

2.2. Agents in Minecraft

  • Minecraft agents have used hierarchical or goal-based reinforcement learning, reward shaping, learned world models, video-trained foundation models, and LLM-based zero-shot planning.Examples include MineCLIP [9], DreamerV3 [12], VPT [1], Steve-1 [18], Voyager [34], DEPS [36], and GITM [43].Source: §2.2 · p. 2
  • MP5 addresses the stated challenge that direct simulator scene data is unrealistic for embodied agents in real scenes, while fixed-output perception cannot adapt to open-ended task needs.The authors motivate an agent able to correct decisions despite inaccurate or erroneous perception information.Source: §2.2 · p. 3

3. Method

MP5 coordinates five modules to solve process- and context-dependent tasks in an embodied environment. Active perception gathers objective-relevant visual information, while planning, execution, and memory support situation-aware action and replanning.

3.1. Overview

  • MP5’s five modules comprise a Minecraft-specific MLLM Percipient, LLM-based Parser, Planner and Patroller, and a Performer interface that turns action sequences into game commands.Retrieval-augmented generation enhances Parser and Planner responses.Source: §3.1 · p. 3

3.3. Perception-aware Planning and Execution

  • The active perception scheme lets Planner schedule or update actions using observed images and inventory status, while Patroller can give feedback on better choices based on visual evidence.The overview also describes updating following sub-objectives when necessary steps are missing.Source: §3.1 · p. 3, §3.3 · pp. 4–5

3.2. Active Perception

  • Active perception uses successive Percipient questions about the sub-objective and current action, with Patroller accumulating answers until it has enough environmental information or reports what is missing.For execution, Performer continues the current action when Patroller cannot gather enough relevant factors.Source: §3.2 · p. 4

3.3. Perception-aware Planning and Execution

  • Performer interacts with active perception during each action round and executes a new action when Patroller identifies environmental information matching the sub-objective and action goal.Successful action sequences and the situation in which they were planned are stored in Performer Memory.Source: §3.3 · p. 5

3.4. Implementation Details

  • Percipient uses a frozen MineCLIP vision encoder, an Alignment Net, and a LoRA-fine-tuned Vicuna-13B-v1.5 LLM, with only the Alignment Net and LoRA parameters optimized.Parser, Planner, and Patroller use GPT-4; Performer maps simple actions such as equip, move, and craft to simulator operations.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate MP5 on context-dependent and process-dependent Minecraft tasks, then examine open-ended tasks and module choices. Results highlight the roles of environment interaction, situation-aware planning, and task-conditioned perception.

4. Experiments

  • The experiments cover quantitative and qualitative performance, open-ended tasks described as never seen before, and analyses of how module choices affect performance.The Minecraft simulation uses MineDojo, ego-view images, and an action space built from MineDojo actions.Source: §4 · p. 5, §4.1 · p. 5

4.1. Experimental Setup

  • The 16 context-dependent tasks span four difficulty levels based on the minimum number of information types required; the 25 process-dependent tasks span five levels based on reasoning steps.Context tasks use predetermined environment details, while process-task environmental factors are randomized.Source: §4.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6

4.2.1 Results of Context-Dependent Tasks

  • Active perception obtains more accurate objective-related information than fine-grained global perception, which can miss or misidentify objective-relevant details.The comparison uses MineLLM, which the authors report performs slightly better than GPT-4V and substantially better than LLaVA-1.5.Source: §4.2.1 · p. 6, Table 3 · p. 7

Embodied action execution is critical for open-ended

  • When an agent cannot interact with the environment or access low-level contextual information during action execution, it is effectively blind to action termination, and Process-Dependent Task success is 0.00%.Embodied execution supports dynamic adjustment of perception and action; perceived environmental information and erroneous actions can be fed back to the planner for replanning.Source: Embodied action execution is critical… · p. 7

4.2.3 Open-Ended Tasks

  • Open-Ended Tasks combine process- and context-dependent elements, requiring perception of different environmental information at multiple stages of sub-objective completion.The authors report validations showing MP5 can complete long sequential tasks in challenging environments.Source: §4.2.3 · p. 7

4.3. Ablation Study

  • LLaVA-1.5 achieves 50.00% success at Mid level and 11.00% at Hard level; with a frozen visual encoder, MineLLM with CLIP performs worse across all levels than MineLLM with MineCLIP.The authors attribute LLaVA-1.5's weakness primarily to training on real-world data rather than Minecraft's pixel-style images.Source: Figure 5 · p. 7, §4.3 · p. 8
  • GPT-4 as zero-shot Planner achieves 22.00% success at Diamond level, while Vicuna-13B-v1.5 and GPT-3.5-turbo each score 0.00%.Vicuna-13Bv1.5 has 1.33% accuracy at Basic level; the authors report stronger inference capabilities yield more concise and accurate execution plans.Source: §4.3 · p. 8, Table 6 · p. 8
  • Without Performer Memory, Diamond-level success drops from 22.00% to 16.67%, while the authors characterize memory mainly as a reference for the Planner.Under Random Drop, Wooden-level success falls from 76.67% to 7.33% without Patroller robustness, and Iron and Diamond success rates drop to 0.00%.Source: §4.3 · p. 8, Table 7 · p. 8

5. Conclusion

MP5 integrates perception, planning, and execution for context- and process-dependent tasks in an open-ended environment. The authors identify reliance on GPT APIs and evaluation limited to Minecraft as limitations.

5. Conclusion

  • MP5 integrates perception, planning, and execution to handle context- and process-dependent tasks in an open-ended environment.The system acquires essential visual information from the scene to support task planning and execution.Source: §5 · p. 8
  • The system relies on GPT-3.5-turbo or GPT-4 APIs, which the authors say limits its usability for people without API access.Its evaluation is limited to Minecraft; extension to other simulation platforms remains future work.Source: §5 · p. 8

Supplementary Material

The supplementary document covers MP5’s modules and implementation, environment and task settings, evaluation details, active-perception strategies, demonstrations, and interactions.

Supplementary Material

  • The supplement begins with discussion of MP5, then describes Percipient, memory, observation and action spaces, and the environment setting.It also includes task details, success rates, ablations, active-perception strategies, application demonstrations, and MP5 interactions.Source: Supplementary Material · p. 11

A. Discussion of MP5

MP5 proposes situation-aware planning and action execution, implemented through five interactive modules and MLLM-based active perception. Its setup uses ego-centric RGB images and a restrained set of human-defined primitives while addressing challenging context- and process-dependent tasks.

A. Discussion of MP5

  • MP5 proposes a paradigm in which embodiment, planning, and execution are enhanced with awareness of task-relevant contextual and procedural information.The authors contrast this with prior methods focused on planning, embodiment, or both, and describe the paradigm as more human-like with potential for harder open-ended tasks.Source: A. Discussion of MP5 · p. 11
  • MP5 implements this paradigm with five interactive modules and an MLLM-based active-perception scheme for situation awareness.The authors report that MP5 can solve the context-dependent tasks in Table 8 through situation-aware planning and execution.Source: A. Discussion of MP5 · p. 11
  • MP5 uses ego-centric RGB images and a restrained set of human-defined primitives while solving the most challenging context- and process-dependent tasks.The passage refers to Table 8 for comparisons of setups and their consequences.Source: A. Discussion of MP5 · p. 11

B.1.1 Data Collection

These appendix sections describe MP5’s data collection and MineLLM training, its retrieval memories, restricted observations, and embodied action execution with visual checks.

B.1.1 Data Collection

  • 50K Minecraft images support 500K image-text instruction-following examples, with collection spanning 60 biomes and nine common mob types.The dataset combines biome and mob batches; snapshots are captured from 12 perspectives, with ground truth screened to the image’s field of view.Source: §B.1.1 · pp. 11–12

B.1.2 MineLLM training details

  • MineLLM aligns frozen MineCLIP image features with Vicuna text embeddings through a two-layer Alignment Net, training that net and LoRA while freezing both backbones.Loss is computed only on system responses.Source: §B.1.2 · p. 12

B.2. Memory

  • MP5 uses Retrieval-Augmented Generation in two modules: Knowledge Memory supports Parser decomposition, while Performer Memory supplies similar action sequences to the Planner.The Planner may consult Performer Memory when planning actions for a sub-objective.Source: §B.2 · p. 13

B.2.1 Knowledge Memory

  • Knowledge Memory retrieves relevant text embeddings from a vector database populated with wiki knowledge, MineDojo crafting recipes, and Reddit tips.When all similarity scores exceed 0.05, the authors manually add the missing knowledge to the database.Source: §B.2.1 · p. 13

B.2.2 Performer Memory

  • Performer Memory stores successful sub-objective descriptions and action sequences, retrieving the two closest matches by task-description embedding.The vector database stores task descriptions and sequence positions; a JSON file maps positions to the successful action sequences.Source: §B.2.2 · p. 13

B.3. Observation Space

  • The observation space combines ego-view RGB images and 3×3×3 voxels with auxiliary status text, while hiding biome, weather, and sky-visibility information.The authors intend this restriction to encourage perception from the current RGB image rather than direct access to those environmental details.Source: §B.3 · p. 13

B.4. Action Space

  • The Performer executes short MineDojo API action combinations with frequent environmental interactions; during “Find,” it checks images at a fixed frequency for the target.“Find” moves forward, jumps over obstacles, and turns left or right if an obstacle cannot be overcome.Source: §B.4 · p. 13

C. Environment Setting

The experiments use MineDojo and define context-dependent tasks around perceiving environmental information. The setup varies initial conditions and task difficulty while keeping the task environment specified by task type.

C. Environment Setting

  • MineDojo supplies the unified observation and action space, with the action-space definition listed in Table 10.The platform is used as the experimental environment.Source: C. Environment Setting · p. 13, Table 10 · p. 14
  • The MineDojo environment randomizes starting position and world seed; agents begin in survival mode with empty inventories as time and weather change.Hostile mobs may spawn, and death clears the inventory; accidental death and failure to finish within the time limit are part of the setting’s challenges.Source: C. Environment Setting · pp. 14–15

D.1.1 Context-Dependent Tasks

  • 16 context-dependent tasks span six information aspects and four difficulty levels, from one required element to four-to-six.The aspects are Object, Mob, Ecology, Time, Weather, and Brightness.Source: §D.1.1 · p. 15
  • Each context task is tested in 50 scenarios across ten predetermined initial biomes, with random seeds and unchanged terrain.The authors say predetermined biomes reduce exploration time and that terrain was not modified to align with other methods’ settings.Source: §D.1.1 · p. 15

D.1.2 Process-Dependent Tasks

The paper defines process-dependent tasks by increasing sub-objective demands and evaluates performance across both context- and process-dependent tasks. It also describes a diamond-pickaxe challenge and an MLLM ablation for context-dependent tasks.

D.1.2 Process-Dependent Tasks

  • 25 process-dependent tasks are grouped into five difficulty levels by increasing reasoning steps, with higher levels requiring longer reasoning and planning.Random environment initialization can cause execution errors and require replanning, adding sub-objectives beyond the basic scenario counts.Source: Table 13 · p. 16, §D.1.2 · p. 16
  • Process-dependent evaluation averages success across 30 games and treats accidental death or exceeding the time limit as failure.The stated limits include a 10-minute game limit and API request timeout.Source: §D.1.2 · p. 16

D.2.1 Context-Dependent Tasks

  • Table 14 reports success rates across context-dependent tasks for methods using active perception or fine-grained global perception.Its caption identifies gray regions as average success rates for each difficulty level.Source: §D.2.1 · p. 16, Table 14 · p. 17

D.2.2 Process-Dependent Tasks

  • Table 16 compares process-task success rates with and without environment interaction during planning or execution; Figure 7 depicts milestone-object acquisition in the diamond-pickaxe challenge.The challenge is described as having 13 reasoning steps, and its figure uses background colors to denote process-task levels.Source: §D.2.2 · pp. 16, 18, Figure 7 · p. 17

D.3.1 Context-Dependent Tasks

  • The context-dependent ablation compares different MLLMs and pretrained visual encoders in the Percipient.The supplied passage identifies the ablation’s scope but does not report its outcomes.Source: §D.3.1 · p. 18

D.3.2 Process-Dependent Tasks

The process-task ablations examine module removal, planner replacements, memory components, and Patroller checking under execution errors. A separate comparison reports higher context-task success with multi-round active-perception query generation.

D.3.2 Process-Dependent Tasks

  • Removing any module significantly reduces process-task success; Diamond-level success falls to 0.00% for every removal except Patroller removal.The Percipient supplies visual input, Memory supplies relevant knowledge, Parser simplifies task decomposition, and Patroller checks action execution.Source: §D.3.2 · p. 18
  • Replacing the Planner with LLaMA2-70B-Chat or Vicuna-13B-v1.5-16k yields 0.00% success at Wooden level, which the authors attribute to difficulty processing long, diverse text.The passage reports that stronger reasoning capability is associated with better understanding of varied text inputs and more effective planning.Source: §D.3.2 · p. 18
  • Without Knowledge Memory, the agent cannot recognize likely iron-ore locations and reaches 0.00% success at both Iron and Diamond levels.The authors report that Knowledge Memory supports item-acquisition knowledge, while Performer Memory supplies similar scenarios as references.Source: §D.3.2 · pp. 18–19
  • With Random Drop, Patroller checking detects execution-error causes and feeds them back for replanning; disabling the check leaves stone-level completion at 0.00%.Random Drop forces an item loss after tasks with more than four reasoning steps, potentially leaving insufficient logs, planks, or sticks.Source: §D.3.2 · p. 19

E. Different Strategy of Active Perception

F.1. Obtain Diamond Pickaxe

  • The paper presents the craft-diamond-pickaxe challenge as a process-dependent task demonstration.The supplied passage identifies the challenge but gives no further outcome.Source: §F.1 · p. 20

F.2. Discovery

  • The paper presents finding a pig near specified conditions as a complex context-dependent task demonstration.The task specifies a sunny day and sufficient brightness.Source: §F.2 · p. 20

F.3. Open-Ended Tasks

The paper illustrates an open-ended task and module interactions through examples of active perception and replanning. In the replanning example, the Patroller identifies a plank shortfall and prompts the Planner to revise the plan.

F.3. Open-Ended Tasks

  • The open-ended-task demonstration asks the agent to dig a sand block on a sunny day.The passage identifies the example without reporting a performance outcome.Source: §F.3 · p. 20

G. Interactions in MP5

  • The interaction examples present communication among MP5’s modules during active perception and replanning.The paper frames these dialogues as illustrations of internal module interactions.Source: G. Interactions in MP5 · p. 20

G.2. Re-planning

  • When crafting a wooden pickaxe with one plank short, the Patroller identifies the execution error and instructs the Planner to replan.The interaction is illustrated in the dialogue and screenshots associated with the task.Source: §G.2 · p. 20, Figure 9 · p. 26, Figure 11 · p. 27

Related papers

Loading 2312.07472v4…