Source-linked AI summary
AdaPlanner: Adaptive Planning from Feedback with Language Models
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, Chao Zhang
TL;DR
Existing LLM agents either act greedily without planning or rely on plans that adapt inadequately to environmental feedback, limiting sequential decision-making. AdaPlanner uses closed-loop plan refinement with in-plan and out-of-plan strategies, code-style prompting, and skill discovery. It outperforms baselines on ALFWorld and MiniWoB++ while using substantially fewer samples.
Problem
Existing LLM agents either lack planning or use plans that do not adequately adapt to environmental feedback, creating a gap in sequential decision-making methods.
Method
AdaPlanner combines an LLM planner and refiner with in-plan and out-of-plan feedback refinement, code-style prompting, and skill discovery from successful trajectories.
Results
AdaPlanner achieves 91.79% overall success in ALFWorld and 91.11% in MiniWoB++ tasks with feedback, outperforming existing baselines.
Takeaways & Limitations
AdaPlanner provides an adaptive closed-loop framework that improves planning performance and sample efficiency across ALFWorld and MiniWoB++.
Takeaways & Limitations
AdaPlanner still requires few-shot expert demonstrations for complex tasks.
Abstract
from arXiv · showhide
Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. However, most existing methods either take actions greedily without planning or rely on static plans that are not adaptable to environmental feedback. Consequently, the sequential decision-making performance of LLM agents degenerates with problem complexity and plan horizons increase. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. In AdaPlanner, the LLM agent adaptively refines its plan from feedback with both in-plan and out-of-plan refinement strategies. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities. Furthermore, we propose a skill discovery mechanism that leverages successful plans as few-shot exemplars, enabling the agent to plan and refine with fewer task demonstrations. Our experiments in the ALFWorld and MiniWoB++ environments demonstrate that AdaPlanner outperforms state-of-the-art baselines by 3.73% and 4.11% while utilizing 2x and 600x fewer samples, respectively.
1 Introduction
Existing LLM agents struggle to balance planning with adaptation: open-loop methods ignore feedback, while most closed-loop methods update actions without revising the full plan. AdaPlanner addresses this gap with adaptive plan refinement, code-style prompting, and skill discovery, achieving strong performance with few samples.
- Existing limitations: Open-loop agents follow fixed plans without feedback, whereas most closed-loop agents adapt actions but do not revise the entire plan.These approaches can produce locally responsive but globally suboptimal decisions.
- AdaPlanner: AdaPlanner uses an LLM as both planner and refiner to adaptively respond to in-plan and out-of-plan environmental feedback.Out-of-plan feedback triggers full-plan revision, while in-plan feedback supports information extraction for subsequent actions.
- Prompting and efficiency: Code-style prompting decomposes tasks into subgoals and mitigates hallucination during planning and refinement.The approach operates solely through prompting, without a dedicated training phase.
- Prompting and efficiency: Skill discovery archives successful trajectories as guidance for planning on similar tasks, improving long-term planning and sample efficiency.The mechanism addresses the cost of acquiring expert demonstrations as the number of tasks grows.
- Results: AdaPlanner’s adaptive closed-loop framework enhances planning performance on ALFWorld and MiniWoB++ with limited samples.The introduction reports superior performance on both environments while emphasizing limited sample use.
2 Preliminaries
LLM planning can be formulated as generating an initial action plan from a task and observation, then refining future decisions using interaction feedback. Closed-loop systems revise plans or actions, while implicit action-only refinement can preserve errors from the initial plan.
- Problem Formulation: The agent receives allowed actions, a task definition, and an initial observation, then generates a length-T planning policy over action sequences.During interaction, subsequent observations and executed actions form a trajectory-like context for feedback-conditioned decisions.
- Open-Loop System: Open-loop control fixes the entire plan from the initial task and observation without feedback-based refinement.Its simplicity comes at the cost of vulnerability to environmental changes.
- Closed-Loop Systems: Closed-loop control incorporates environmental feedback to adjust future decisions through initial planning and feedback-based refinement.The formulation distinguishes high-level plan refinement from action generation conditioned on the current context and plan.
- Implicit Closed-Loop Systems: Implicit closed-loop systems modify only the current action while retaining future actions from the initial plan.Although this can select locally optimal actions, initial planning inaccuracies can still cause task failure or non-completion.
- Explicit Closed-Loop Systems: Explicit closed-loop systems revise the entire remaining plan at each step, allowing constant refinement and improvement of future actions.AdaPlanner is classified as an explicit closed-loop system.
3 AdaPlanner
AdaPlanner combines an LLM planner and refiner with skill memory to generate, check, and adapt code-based plans through environmental feedback. Its closed-loop execution supports in-plan reasoning, out-of-plan revision with checkpoint resumption, and filtering of discovered skills for similar tasks.
- Model Architecture: AdaPlanner uses an LLM-based agent as both planner and plan refiner, alongside skill memory for sample-efficient skill discovery.The planner generates a comprehensive plan, while the refiner performs feedback-based refinement; skill memory archives successful plans and interactions.
- Environment Interaction: AdaPlanner checks subgoal success at N crucial timestamps and routes resulting observations back for plan revision, reducing API calls.It proactively refines only when actual outcomes differ from anticipated observations.
- Plan Generation via Code-Based LLM Prompting: Pythonic code prompts decompose complex tasks into subgoals and admissible-action sub-plans, with assertions testing subgoal fulfillment.The code is executed through an environment interface that grounds actions and routes observations back for adaptation.
- Adaptive Closed-Loop Plan Refinement: In-plan refinement uses ask_LLM() to extract useful information from aligned observations and incorporate it into upcoming actions.The action supports self-querying and reasoning over information parsed from natural-language environmental observations.
- Adaptive Closed-Loop Plan Refinement: Out-of-plan refinement revises the entire plan after an assertion fails, then resumes execution from a saved intermediate checkpoint.The refine-then-resume mechanism avoids restarting the episode, significantly speeding task completion and reducing LLM calls.
- Skill Discovery: Skill discovery alternates successful-task acquisition with filtering that archives a discovered solution only when it improves success rate.Successful trajectories can be collected with few or no demonstrations, while episode-specific solutions that fail to generalize are discarded.
4 Evaluation
AdaPlanner is evaluated on ALFWorld and MiniWoB++ using benchmark-specific settings, baselines, and ablations. It achieves strong performance while using relatively few demonstrations, and its feedback, code-interface, and skill-discovery components improve results.
- Evaluation Setup: AdaPlanner is evaluated on 134 ALFWorld tasks and selected MiniWoB++ tasks with and without environmental feedback.The ALFWorld benchmark spans six task types; MiniWoB++ includes nine feedback tasks and 53 tasks evaluated in RCI.
- Main Results: 91.79% overall success rate in ALFWorld and 91.11% on MiniWoB++ tasks with feedback establish AdaPlanner’s state-of-the-art benchmark performance.On MiniWoB++, it also achieves 93.22% on tasks without feedback.
- Sample Efficiency: 600 times fewer samples than CC-Net are required for comparable MiniWoB++ performance, while AdaPlanner achieves the highest ALFWorld performance with the fewest samples.The sample-efficiency comparison is summarized through success rate versus number of demonstrations.
- Ablations: Success rates increase with each subsequent closed-loop refinement across settings with different numbers of demonstration samples.Figure 4a measures total samples across all six ALFWorld tasks.
- Ablations: Code prompts make AdaPlanner more resilient to GPT-3.5 hallucination than ReAct and Reflexion in decision-making tasks.The code interface provides a more formal and constrained generation space.
- Ablations: Skill discovery nearly doubles ALFWorld success and increases MiniWoB++ success by approximately 15%.The comparison uses AdaPlanner with and without skill discovery under GPT-3.5 adaptive closed-loop operation.
5 Related Work
Prior LLM-agent methods range from open-loop generation of complete action sequences to closed-loop interaction and plan refinement. AdaPlanner is positioned among methods that recursively adapt plans, while addressing the data requirements associated with plan selection.
- Open-Loop Systems: Open-loop methods generate complete action sequences or tool-use plans without adapting them to subsequent environmental feedback.Examples include Chain-of-Thoughts, Zero-Shot Planner, HuggingGPT, and Chameleon.
- Closed-Loop Systems: Closed-loop methods use environmental feedback for single-step action selection or revision across trials.ReAct and Inner Monologue act from feedback, while Reflexion extends ReAct with revision from past trials.
- Plan Refinement: Strict closed-loop architectures recursively refine generated plans, but DEPS requires a trained selector and therefore faces generalization challenges.DEPS first proposes an entire plan and then refines it using real-world feedback.
6 Conclusion and Limitations
The paper concludes that AdaPlanner combines adaptive closed-loop refinement with code-style prompting and skill discovery to improve planning and sample efficiency. It also identifies a remaining need to solve complex tasks without demonstrations.
- Conclusion: AdaPlanner adaptively refines generated plans using both in-plan and out-of-plan refinement strategies.These strategies are presented as mechanisms for fully leveraging environmental information.
- Conclusion: Code-style prompting addresses hallucination, while skill discovery supports learning from successful past experience.The two mechanisms complement the closed-loop planning approach.
- Conclusion: AdaPlanner outperforms state-of-the-art baselines and demonstrates better sample efficiency, with ablations supporting the effectiveness of its components.The conclusion reports these findings from comprehensive experiments.
- Limitations: AdaPlanner still requires few-shot expert demonstrations for solving complex tasks.The authors identify solving complex tasks with no demonstrations as future work.
7 Broader Impacts
The paper notes that autonomous LLM-agent research carries risks including security threats, misuse, and unintended consequences such as job displacement. It calls for coordinated regulation and safety-oriented research before deployment.
- Risks: Autonomous LLM agents carry risks including security threats, potential misuse, and job displacement from automation.These risks are identified as inherent to the research line.
- Risk Mitigation: Researchers and policymakers should collaborate on regulations and safety principles that guide deployment toward positive outcomes.The passage emphasizes human values and ethical and moral alignment before industrial deployment.
8.1 Experimental Setup
The experiments evaluate AdaPlanner in challenging ALFWorld and MiniWoB++ environments using success rate as the primary metric. ALFWorld emphasizes long action trajectories, while MiniWoB++ tests dynamic web interactions and varied task difficulties.
- ALFWorld: ALFWorld contains six text-based household task types requiring navigation and object interaction.The tasks are Pick, Clean, Heat, Cool, Examine, and Pick two.
- ALFWorld: More than 50 possible object locations can require trajectories exceeding 50 actions in ALFWorld.This creates a long-horizon exploration challenge for the agent.
- MiniWoB++: MiniWoB++ covers computer tasks ranging from button clicking to long-horizon, reasoning, pop-up, and layout-variation challenges.Examples include click-checkboxes-large, click-checkboxes-soft, login-user-popup, multi-orderings, and multi-layouts.
- MiniWoB++: MiniWoB++ represents each task as a 160px × 210px web environment whose state is HTML code.The action space includes clicking and typing operations.
- MiniWoB++: The evaluation selects nine MiniWoB++ tasks whose HTML observations change after certain actions.These tasks are used to evaluate the proposed closed-loop framework.
- Metrics: Success rate (%) is the number of successful episodes divided by the total number of episodes.The metric is used consistently with previous work.
- Metrics: In ALFWorld, an episode fails when 50 actions are reached while the task remains unsolved.
- Metrics: In MiniWoB++, an episode can fail because of invalid actions or because the task remains unfinished.
8.2 Baseline Details
The experiments compare AdaPlanner with prompting-based, imitation-learning, training, and finetuning baselines across ALFWorld and MiniWoB++. Sample allocations vary to test performance under different demonstration budgets.
- ALFWorld: On ALFWorld, AdaPlanner is compared with BUTLER, ReAct, and Reflexion across 134 environments.
- ALFWorld: BUTLER is an imitation-learning baseline trained with 100k samples per task.
- ALFWorld: ReAct and Reflexion are prompting-based methods using six and eight samples, respectively, across all six ALFWorld tasks.
- ALFWorld: AdaPlanner, ReAct, and Reflexion are evaluated with GPT-3 and GPT-3.5 models, while BUTLER results come from prior work.
- MiniWoB++: The baseline comparison includes RCI, AdaPlanner, and three training or finetuning-based MiniWoB++ methods.
- Sample allocation: Table 4 allocates 2, 4, or 6 prompting samples across the six ALFWorld task types.
- Sample allocation: With two total samples, one Clean trajectory prompts Pick, Clean, Heat, and Cool, while one Examine sample prompts Examine and Pick two.
- MiniWoB++: The MiniWoB++ evaluation uses 59 examples for 53 tasks, including human-written demonstrations and 21 examples obtained through skill discovery.
8.3 Prompts
The prompts encode agent capabilities, tasks, examples, and action procedures in code-style formats. ALFWorld prompts support decomposition, ranked exploration, assertions, and task-specific expert samples.
- 8.3.1 ALFWorld: The basic_info prompt defines the agent and admissible actions and is reused in initial-planning and refinement prompts.
- 8.3.1 ALFWorld: The agent state includes its location, held object, receptacle list, and reporting function for assertions.
- 8.3.1 ALFWorld: The ALFWorld prompt represents household actions as Python-like functions such as take, put, open_receptacle, clean, heat, cool, and turn_on.
- 8.3.1 ALFWorld: Initial planning combines basic information, an expert trajectory sample, a receptacle list, and a natural-language task description.
- 8.3.1 ALFWorld: Each planned ALFWorld step includes a numbered marker, a reason, and an assertion checking whether the action succeeds.
- 8.3.1 ALFWorld: Six expert samples provide one exemplar for each ALFWorld task type.
- 8.3.1 ALFWorld: The Pick example asks the model to rank likely receptacles, inspect them until finding the soapbar, identify and take it, then place it on a countertop.
- 8.3.1 ALFWorld: The Clean example demonstrates cleaning lettuce and placing it on a diningtable, while the prompt exposes admissible actions and environment interaction code.
8.4 Hallucination with GPT-3.5 Model (gpt-3.5-turbo)
With gpt-3.5-turbo, ReAct and Reflexion repeatedly generated inadmissible actions, whereas AdaPlanner completed both illustrated tasks, revising its plan when feedback contradicted an assumption.
- Case 1: examine the book with the desklamp: ReAct hallucinated actions outside ALFWorld’s admissible action space despite two expert demonstrations.Examples include searching for a specific book title and asking household members for information.
- Case 1: examine the book with the desklamp: Reflexion remained unable to formulate a feasible plan after two iterative trials and errors.Its reflections proposed asking household members for help or obtaining a ladder or step stool outside the room.
- Case 1: examine the book with the desklamp: AdaPlanner completed the book-and-desklamp task without revisions, indicating that its code interface mitigated hallucination for gpt-3.5-turbo.The reported trajectory presents only the planner’s major subgoals.
- Case 2: put some watch on safe: ReAct hallucinated infeasible actions in the watch-on-safe task, ultimately leading to task failure.The examples include looking for a manual or instructions and asking someone in the household.
- Case 2: put some watch on safe: Reflexion also hallucinated actions such as seeking additional resources or asking for help, contributing to eventual task failure.The passage attributes the issue to hallucination during this task.
- Case 2: put some watch on safe: AdaPlanner initially assumed the watch was in the safe, but revised its plan after an assertion error and then completed the task.The refinement adapted the plan to the actual environment after the incorrect assumption.
8.5 Detailed Evaluation Results on MiniWoB++
Table 5 presents per-task success rates for AdaPlanner and four comparison methods on MiniWoB++, with unavailable results marked separately and feedback tasks visually distinguished.
- Table 5: Table 5 compares AdaPlanner, CC-Net, WGE, WebN-T5-3B, and RCI using per-task success rate (%).The caption identifies the methods and metric displayed in the table.
- Table 5: “n/a” marks success rates that were not reported in the original method papers.This indicates unavailable reporting rather than a measured zero success rate.
- Table 5: The nine tasks with feedback are marked in gray.The caption specifies this visual grouping but does not state a comparative outcome for those tasks.