PaperlayerView exact revision on arXiv

Source-linked AI summary

Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, Yitao Liang

arXiv:2302.01560v3cs.AI

TL;DR

Open-world multi-task agents need reliable long-horizon reasoning and plans that account for which parallel sub-goals are currently feasible. DEPS interactively describes failures, uses LLM explanation and replanning, and selects goals by predicted completion horizons. It robustly accomplishes 70+ Minecraft tasks, nearly doubles overall performance, and improves results in ALFWorld and tabletop environments.

  • Problem

    Open-world planning requires precise multi-step reasoning, while conventional planners ignore the agent’s proximity to parallel sub-goals when ordering them.

  • Method

    DEPS combines execution description, LLM self-explanation and replanning, and a trainable selector that ranks candidate goals by estimated completion horizon.

  • Results

    DEPS robustly accomplishes 70+ Minecraft tasks and nearly doubles overall performance, with between on-par and more than 50% relative improvement in ALFWorld and tabletop environments.

  • Takeaways & Limitations

    DEPS provides an interactive planning approach that addresses both long-horizon plan correction and state-aware sub-goal selection across open-ended and non-open-ended domains.

  • Takeaways & Limitations

    The framework relies on privately held LLMs and explicit step-by-step planning, which limits accessibility and may create a planning bottleneck for scaling.

Abstract

from arXiv · show

We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose "$\underline{D}$escribe, $\underline{E}$xplain, $\underline{P}$lan and $\underline{S}$elect" ($\textbf{DEPS}$), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated $\textit{plan}$ by integrating $\textit{description}$ of the plan execution process and providing self-$\textit{explanation}$ of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal $\textit{selector}$, which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the $\texttt{ObtainDiamond}$ grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner.

1 Introduction

Open-world multi-task planning is difficult because tasks require long, dependency-constrained sub-goal sequences and state-aware route selection. DEPS addresses these challenges through interactive plan correction and goal selection, improving performance across Minecraft and other domains.

  • Motivation: Transfer from established robotics domains to open-ended worlds with unlimited exploration and internet-scale knowledge remains unresolved.The authors evaluate this gap in Minecraft alongside ALFWorld and tabletop environments.
  • Challenges: Open-world tasks involve abundant object types, complex dependencies, and long sub-goal sequences that must execute in strict order.Obtaining a diamond in Minecraft requires at least 13 properly ordered sub-goals, whereas tabletop tasks typically require only a few.
  • Challenges: Open-world planners must choose among alternative sub-goal sequences according to the agent’s current accessibility and state.For crafting a bed, reachable spider-derived materials may be preferable to wool from sheep or beds in villages that cannot be reached quickly.
  • Approach: DEPS combines a descriptor, LLM-based explainer and planner, and learned goal selector to correct failed plans and choose accessible sub-tasks.The descriptor summarizes execution feedback, the explainer identifies errors, and the planner refines the plan; the selector ranks sub-tasks by proximity.
  • Results: DEPS nearly doubles overall success relative to language-planner baselines and achieves between on-par and more than 50% relative improvement in ALFWorld and tabletop environments.The comparison uses the same initial state and goal-conditioned controller, and the Minecraft evaluation covers 71 tasks without demonstrations.

2 Background

Hierarchical embodied agents decompose long-horizon language goals into controller-executed sub-goals, while LLM planners can generate such sequences zero-shot. However, one-shot plans become unreliable as task length and state-dependent efficiency demands increase.

  • Hierarchical Planning: A planner decomposes a long-horizon task into K short-horizon sub-goals, which a goal-conditioned policy executes from image observations and language goals.At each time step, the controller conditions its action on the current state and the specified sub-goal.
  • LLM Planning: LLMs can decode task descriptions into ordered sub-goal sequences that low-level controllers execute in embodied environments.Prior work uses models such as Instruct-GPT and Codex as zero-shot planners.
  • Limitations of Prior Planning: The probability of generating a flawless plan decreases as the required number of sub-goals increases, and even correct plans may be inefficient for the current state.These limitations correspond to long-horizon reasoning and state-dependent feasibility.
  • Terminology: Here, “zero-shot” means no gradient updates, although related demonstrations may be provided as inference-time prompts.The distinction concerns training updates rather than whether prompts contain demonstrations.

3 Towards Reliable Planning in Embodied Open-World Environments

DEPS combines interactive LLM-based plan refinement with a horizon-predictive selector to improve execution reliability and efficiency for long-horizon tasks in open-world environments.

  • 3.1 DEPS Overview: The planner decomposes a high-level task into sequential natural-language sub-goals executed by a goal-conditioned controller.The initial plan P0 contains goals such as mining wood, finding cups, or placing one block on another.
  • 3.2 Describe, Explain and Plan with LLM Generates Executable Plans: DEPS iteratively updates an initial LLM-generated plan when controller failures occur, using textual descriptions and LLM self-explanations to locate errors before replanning.The descriptor summarizes the current state and execution outcome, while the LLM explains the failure and generates a revised plan.
  • 3.3 Horizon-Predictive Selector Yields Efficient Plans: The selector ranks parallel goals by predicted remaining completion time and chooses the current goal, modifying the original plan when alternative routes exist.A learned horizon predictor estimates each goal’s remaining steps from the current state, allowing the selector to favor more efficient routes.
  • 3 Towards Reliable Planning in Embodied Open-World Environments: Interactive replanning addresses long-horizon execution because open-world tasks can require many dependent sub-goals, including at least 13 ordered sub-goals for Minecraft’s ObtainDiamond task.One-shot planners often fail because generating a flawless plan with complex preconditions is difficult.
  • 3.3 Horizon-Predictive Selector Yields Efficient Plans: Selector-based ordering matters because dynamic environments can make distant goals inefficient and allow nearby targets to disappear before they are pursued.The paper gives Minecraft as an example: delaying a nearby sheep while collecting wood may make the sheep difficult to find again.

4 Experiments

Experiments evaluate DEPS across 71 Minecraft tasks, multiple environments, selectors, and replanning rounds. DEPS achieves the strongest reported performance, while selector and interactive replanning components improve efficiency and success on difficult tasks.

  • Experimental Setup: Experiments evaluate DEPS on 71 Minecraft tasks grouped into eight meta-task categories across Minedojo, MineRL, and MC-TextWorld.The evaluation uses different Minecraft versions and environments, with a common focus on interactive planning.
  • Main Results: DEPS achieves the best performance across all eight Minedojo meta-task groups, while existing LLM planners nearly fail from MT6 onward.DEP without the Selector already outperforms existing LLM-based planners, supporting the contribution of description, explanation, and replanning.
  • Main Results: The Selector further improves success on hard, efficiency-sensitive tasks, reaching up to +2.7 times the success rate on MT7.Hard tasks offer many candidate sub-goals, making ordering efficiency especially important under limited episode lengths.
  • Ablation on Selector: At 1000 steps, goal models provide greater advantages, and horizon prediction outperforms CLIP and MineCLIP while Selector performance scales with more goals.Longer maximum episode lengths also increase success rates, so the ablation tests efficiency under constrained horizons.
  • Ablation on Re-Planning Rounds: More DEPS description, self-explanation, and replanning rounds produce better results, especially for hard tasks.Round 0 corresponds to the vanilla planner without replanning; each additional round updates the plan through interaction.
  • ObtainDiamond Challenge: DEPS achieves a 0.59% success rate on ObtainDiamond within 10 minutes without task-specific fine-tuning.The experiment evaluates a longstanding Minecraft challenge involving complex item acquisition, crafting, tool use, smelting, and mining dependencies.

5 Related Works

Prior work uses LLMs to generate and execute plans in embodied environments, including approaches that incorporate affordances, scene descriptions, success detection, and interactive feedback. DEPS extends interactive planning with self-explanation and goal selection to address accumulated errors and efficiency.

  • LLM planners generate executable action sequences for high-level embodied tasks, sometimes using skill affordances or text-completion decomposition.
  • Scene descriptions and success detectors help planners produce environment-suitable plans and verify step execution.
  • DEPS addresses accumulated planning errors with explanation and improves plan efficiency through a goal Selector.
  • Minecraft agents have also used hierarchical architectures, learned reward functions, human demonstrations, and learned world models.

6 Limitations

The authors identify limitations involving access to privately held LLMs, explicit step-by-step planning, and underrepresented planning challenges in the adopted environments.

  • DEPS relies on privately held LLMs such as GPT-3 and ChatGPT, limiting accessibility for users without affordable access.
  • Explicit step-by-step planning can create a bottleneck that prevents further scaling, motivating amortization within trainable goal-conditioned policies.
  • Some fundamental planning challenges, including dead ends, may be underrepresented in the adopted environments.

7 Conclusion

The paper concludes that DEPS addresses open-world planning challenges through LLM-based interactive planning and demonstrates broad performance across Minecraft and additional embodied environments. Its experiments emphasize robust Minecraft task completion and generalization to ALFWorld and tabletop manipulation.

  • DEPS targets long-term reasoning and inefficient ordering of parallel sub-goals in open-world planning.
  • The method robustly accomplishes 70+ Minecraft tasks and nearly doubles overall performance relative to counterparts.
  • Additional experiments evaluate DEPS in ALFWorld and Tabletop Manipulation to demonstrate generalization beyond Minecraft.
  • ALFWorld: ALFWorld aligns text-based policies with visually rich embodied execution and includes six task types with varying difficulty.
  • ALFWorld: The ALFWorld experiments compare GPT, Inner Monologue, and DEPS planning approaches.

A.1.2 Results

In ALFWorld, replanning substantially improves long-horizon planning, while DEPS further improves feasibility; in tabletop manipulation, all LLM-based planners perform well and show smaller performance differences. The experiments also identify a task-specific knowledge limitation affecting Place Two & Place tasks.

  • ALFWorld: Replanning enables the short-horizon Inner Monologue approach to outperform the long-horizon GPT approach by a large margin in ALFWorld.
  • ALFWorld: GPT+RP achieves performance ranging from 10% to 52%, while DEPS further enhances planning feasibility with descriptions and self-explanation.
  • ALFWorld: All planning methods fail on Place Two & Place tasks because the LLM lacks the requisite task knowledge.
  • Tabletop Manipulation: Tabletop Manipulation experiments use a UR5e robot with a suction gripper in simulated environments.
  • Tabletop Manipulation: All LLM-based planning methods perform well on tabletop rearrangement tasks, with less pronounced performance gaps than in open-ended Minecraft.

B Minecraft Task Details

The Minecraft evaluation uses 71 overworld tasks grouped by task ingredients and functions, with episode limits scaled to human completion times. Tasks rely on a 262-goal skill space and behavior-cloned goal-conditioned controllers.

  • Task Set: Episode limits are set separately for each task by doubling the average human completion time, reflecting differences in task difficulty.Longer maximum episodes increase success rates, so the study imposes task-specific limits to evaluate efficiency.
  • DEPS Variants: Three DEPS implementations are evaluated across experimental settings, differing in their internal components while all using description and self-explanation.The comparison is summarized in the paper’s implementation table.
  • Task Structure: Minecraft tasks require sequential mine-and-craft goals because raw materials, tools, crafting stations, and item quantities impose dependencies.Mining and crafting objectives can require dozens of step-by-step goals, such as obtaining diamonds with sufficiently advanced tools.
  • Controller and Skills: The agent’s skill space contains 262 goal-conditioned policies, each associated with an objective item used to assess goal achievement.Each skill maps a natural-language goal and current state to an action through a goal-conditioned policy.
  • Controller and Skills: Controller training uses MineDojo observations, 42 discretized actions, imitation learning, and a modified goal-sensitive Impala CNN.Observations include RGB vision, orientation, GPS position, and nearby 3×3 block types.

C.2 LLM as Planner

DEPS augments LLM planning with environment descriptions, self-explanation, replanning, and goal selection. Across LLMs, these additions make final success rates nearly identical despite large differences in initial plan quality, while token limits constrain replanning rounds.

  • Planner Design: DEPS uses Codex for Minecraft, GPT3 for ALFWorld, and ChatGPT for tabletop manipulation, demonstrating compatibility with multiple LLM planners.The planner choice is adapted to each environment, while DEPS retains the same interactive planning principle.
  • Planner Design: DEPS converts execution failures into textual descriptions, asks an explainer to identify plan errors, and replans using the resulting explanation.This process is designed to produce a feasible final plan even when the initial LLM plan is inaccurate.
  • LLM Comparison: DEPS-augmented LLMs achieve almost identical success rates across models, although GPT-4’s Vanilla Planner initial plan accuracy is twice that of other baselines.The result indicates that DEPS can improve plan feasibility even when initial planning ability is weaker.
  • Limitations: DEPS is constrained by each model’s maximum token limit, which determines how many replanning rounds can be supported.Longer replanning sequences are especially beneficial for long-horizon tasks requiring more skills.
  • Execution Interface: The goal parser maps free-form plans to predefined controller skills using LLM parsing and Sentence-BERT semantic similarity for unmatched expressions.Executable goals are represented through structured action, object, tool, and rank fields.
  • Execution Interface: Minecraft success detection monitors inventory changes, while other environments may use an LLM to assess whether the current goal was completed.The detector supplies the execution-status signal needed by the planner-controller loop.
  • Goal Selection: The selector chooses among currently executable goals using the state and plan, improving executability beyond planners that ignore feedback or merely replan.Figure 7 contrasts DEPS with Inner Monologue, Zero-Shot Planner, and Zero-Shot Planner with replanning.

E Discussion on ObtainDiamond Task

ObtainDiamond is used as a demanding open-world test of long-horizon planning. DEPS reaches 60% success with an Oracle Controller but only 0.6% with a behavior-cloned controller, with later milestones declining as complexity and controller constraints accumulate.

  • ObtainDiamond Results: 0.6% success is achieved by DEPS with a behavior-cloned controller in randomly generated Minecraft worlds.The reported bottleneck is the goal-conditioned controller rather than the plans generated by DEPS.
  • Task Difficulty: Diamonds appear only 0.0846% of the time and are predominantly found between depths 2–16, making tool durability and material quantities central planning constraints.An iron pickaxe is typically insufficient for the task, motivating state-dependent replanning.
  • Task Difficulty: DEPS can generate a feasible plan beginning with an iron pickaxe based on the agent’s current state.This illustrates how replanning responds to resource constraints during ObtainDiamond.
  • Milestone Analysis: Success rates decrease across later ObtainDiamond milestones, which the authors attribute to task complexity and controller constraints.The milestone progression is reported in Figure 9.

F Success Rates of ALL Tasks in Minecraft

The Minecraft evaluation compares DEPS with several LLM-based and programmatic planning baselines across all tasks. DEPS is state-of-the-art on almost all tasks, particularly those requiring more skills, under repeated random-world evaluation.

  • Overall Results: DEPS is state-of-the-art on almost all of the evaluated Minecraft tasks, especially difficult tasks requiring more skills.The comparison includes Zero-shot Planner, ProgPrompt, Chain-of-Thought, Inner Monologue, Code as Policies, DEP without Selector, and DEPS.
  • Evaluation Protocol: All tasks are evaluated at least 30 times across different world seeds with the same initial conditions.Average success rates are reported in the complete task comparison table.

G Prompt for Different Tasks and Different Methods

The prompts illustrate how DEPS-style interaction and baseline methods represent Minecraft tasks, execute plans, and respond to failures. They contrast executable planning with descriptive reasoning and feedback-driven replanning.

  • G.1.1 DEPS: DEPS prompts combine planning, execution descriptions, explanations of failures, and replanning for Minecraft tasks.The interaction exposes inventory states and action requirements before producing a corrected plan.
  • G.1.1 DEPS: For simpler tasks, DEPS generates direct action sequences, such as mining a log, crafting planks, and crafting a wooden_button.The prompt records success after all three steps.
  • G.1.1 DEPS: A failed stone_sword plan is repaired by inserting crafting_table construction before wooden_pickaxe production.The revised sequence then mines cobblestone and crafts the sword.
  • G.1.1 DEPS: The DEPS example reports successful completion after four corrected steps, although its final statement names a stone_pickaxe.The dialogue records success through steps 1–4 and then reports obtaining a stone_pickaxe.
  • G.1.2 Baselines: Baseline prompts represent Minecraft plans as either code, explicit step lists with inventory reasoning, or scene-action-success dialogues.The examples include ProgPrompt, Chain-of-Thought, Zero-shot Planner, and related prompt formats.
  • G.1.2 Baselines: The baseline stone_sword plan lists seven ordered actions from mining logs through crafting the final sword.The sequence includes crafting a table and wooden pickaxe before mining cobblestone.
  • G.1.2 Baselines: Chain-of-Thought prompts additionally track inventory changes after each action while preserving the same seven-step stone_sword sequence.The inventory trace records resource consumption and production across the plan.
  • G.1.2 Baselines: Scene-based prompts pair each planned action with a short requirement check and a successful-action outcome for wooden_button and stone_sword tasks.For example, they verify that mining needs no tool and crafting requires the relevant inventory item.

H Full Dialogue

The full dialogue shows iterative replanning for obtaining a diamond in Minecraft. Each failure reveals a missing resource, tool, or workstation, and subsequent plans extend the sequence until later stages succeed.

  • H Full Dialogue: The initial diamond plan mines iron ore, smelts an iron ingot, crafts an iron pickaxe, and mines a diamond.It assumes the required furnace and sufficient iron ingots are already available.
  • H Full Dialogue: The first repair mines one cobblestone before crafting a stone_pickaxe, but diamond mining fails because an iron_pickaxe is required.The agent succeeds on the first two revised steps before encountering the tool mismatch.
  • H Full Dialogue: A subsequent repair adds iron-ore mining, smelting, and iron-pickaxe construction, but smelting fails because no furnace exists.The dialogue explicitly attributes the failure to the missing furnace.
  • H Full Dialogue: Another repair adds furnace construction, yet that step fails because crafting the furnace requires a crafting_table.The plan otherwise proceeds toward smelting and iron-pickaxe construction.
  • H Full Dialogue: The dialogue then mines eight cobblestone before attempting furnace construction, which succeeds but exposes an additional workstation dependency.The later failure is again tied to the crafting context required for furnace construction.
  • H Full Dialogue: The corrected plan uses the crafting_table for furnace construction and successfully completes smelting, while iron-pickaxe crafting still lacks enough iron ingots.The agent has one iron ingot but needs three for the pickaxe.
Loading 2302.01560v3…