Source-linked AI summary
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, Yu Su
TL;DR
Existing embodied agents need costly labeled trajectories and often cannot adapt plans to partially observable environments. LLM-Planner uses few-shot LLM-generated high-level plans with grounded replanning from observations; on ALFRED, it achieves competitive performance using less than 0.5% of paired training data.
Problem
Existing embodied agents require many labeled instruction–trajectory pairs, while static LLM plans and admissible-action assumptions limit adaptation in diverse, partially observable environments.
Method
LLM-Planner prompts an LLM to generate high-level subgoal plans and dynamically replans from perceived objects and environmental observations during execution.
Results
Less than 0.5% of paired training data produces competitive performance with full-data baselines on ALFRED, while existing methods barely complete tasks under the same few-shot setting.
Takeaways & Limitations
LLM-Planner demonstrates a route toward versatile and extremely sample-efficient embodied agents using LLM planning and physical grounding.
Takeaways & Limitations
Prior LLM-based approaches can require enumerating admissible actions, an assumption that is difficult in partially observable environments and scales poorly with object count.
Abstract
from arXiv · showhide
This study focuses on using large language models (LLMs) as a planner for embodied agents that can follow natural language instructions to complete complex tasks in a visually-perceived environment. The high data cost and poor sample efficiency of existing methods hinders the development of versatile agents that are capable of many tasks and can learn new tasks quickly. In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models to do few-shot planning for embodied agents. We further propose a simple but effective way to enhance LLMs with physical grounding to generate and update plans that are grounded in the current environment. Experiments on the ALFRED dataset show that our method can achieve very competitive few-shot performance: Despite using less than 0.5% of paired training data, LLM-Planner achieves competitive performance with recent baselines that are trained using the full training data. Existing methods can barely complete any task successfully under the same few-shot setting. Our work opens the door for developing versatile and sample-efficient embodied agents that can quickly learn many tasks. Website: https://dki-lab.github.io/LLM-Planner
1. Introduction
LLM-Planner uses LLMs for few-shot high-level planning and dynamically grounds plans through environmental perception. On ALFRED, it achieves competitive performance with less than 0.5% of paired training data.
- Existing embodied agents require many labeled instruction–trajectory pairs, limiting versatile agents that must learn diverse tasks quickly.
- Unlike static planning approaches, LLM-Planner dynamically adjusts plans using environmental observations because the suitable plan depends on the current environment.
- LLM-Planner prompts an LLM to generate high-level plans consisting of ordered subgoals, while a low-level planner converts each subgoal into primitive actions.
- Grounded re-planning injects perceived objects into renewed prompts when execution stalls or repeatedly fails, producing a new continuation of the partial plan.
- Less than 0.5% of paired training data yields competitive performance with full-data baselines, while existing methods barely complete tasks under the same few-shot setting.
- The approach is presented as a path toward versatile and extremely sample-efficient embodied agents through LLM planning and physical grounding.
2. Related Work
Related work includes end-to-end and hierarchical VLN systems, auxiliary uses of LLMs, and static LLM planners. LLM-Planner extends these directions with generative planning, environmental grounding, and dynamic replanning without requiring admissible-action lists.
- End-to-end Transformer and BERT-based models perform well on navigation-only VLN, whereas complex ALFRED tasks favor hierarchical separation of high- and low-level planning.
- Hierarchical planning has shown advantages in low-data settings, but prior work had not combined few-shot planning with LLMs.
- Existing LLM applications use models as auxiliary helpers or planners, often assuming that admissible environment actions can be enumerated.
- LLM-Planner generates high-level plans without prior knowledge of current-environment specifics and refines them from new observations, avoiding the admissible-action-list assumption.
- The method is evaluated across diverse ALFRED tasks and uses a pretrained vision model to ground plans in the current environment.
3. Preliminaries
The paper formulates embodied instruction following as hierarchical planning in visually perceived environments. In-context learning supplies few-shot adaptation, while high-level plans condition low-level action planning independently of the original instruction.
- Embodied instruction following, also called VLN here, requires an agent to execute primitive actions in an environment to satisfy a language instruction.
- ALFRED represents tasks with an instruction, environment, and goal test, and supports high-level goals with optional step-by-step instructions.
- Hierarchical planning separates a high-level planner that proposes subgoals from a low-level planner that executes them through primitive actions.
- Once the high-level plan is specified, low-level planning is independent of the language instruction, formalized as P(L_l|I, L_h, E) = P(L_l|L_h, E).
- In-context learning adapts LLMs to downstream tasks from a few demonstrations without updating model parameters.
- The study adopts a true few-shot setting to avoid bias from large validation sets used for prompt design and model selection.
4. LLM-Planner
LLM-Planner uses LLMs for few-shot high-level planning and grounds plans in the current environment through observed objects and dynamic re-planning. Its hierarchical design combines generated subgoal sequences with low-level execution and updates plans when execution fails or progresses for a fixed number of steps.
- LLM-Planner adapts LLMs such as GPT-3 for few-shot grounded high-level planning in embodied agents.
- The static planner uses carefully designed prompts and dynamically retrieved similar instruction–plan examples to generate reasonable high-level plans.A frozen BERT-base model retrieves the K most similar training examples using Euclidean distance between instruction embeddings.
- Observed objects are injected into the prompt and receive logit biases so generated plans can prioritize relevant objects in the environment.This grounding addresses static plans that may contain incorrect objects or unattainable actions.
- Grounded re-planning regenerates a continuation of the completed partial plan when an action fails or after a fixed number of time steps.The algorithm supplies the instruction, observed-object set, and completed subgoals to produce a new high-level plan, then resumes low-level execution.
- The planner provides a generic integration interface requiring an object list and a low-level planner, without assumptions about the agent’s internal workings.
5. Experiments
Experiments evaluate LLM-Planner on ALFRED using few-shot training, high-level planning metrics, baselines, ablations, and sensitivity analyses. The results show competitive performance with limited paired data, benefits from dynamic retrieval, object-focused logit biases, grounded re-planning, and additional examples.
- Dataset: ALFRED spans 7 task types, 207 environments, 115 object types, and 4,703 tasks, providing a diverse benchmark for long-horizon embodied planning.Tasks range from moving a single object to placing a heated object into a receptacle.
- Metrics: The evaluation reports success rate, goal-condition success rate, and high-level planning accuracy, with dynamic HLP accuracy given as a lower–upper bound range.The bounds reflect whether the generated plan is correct overall or was successfully executed by the low-level controller.
- Main Results: Using less than 0.5% of paired training data, LLM-Planner matches the full-data HLSM baseline competitively and outperforms several recent baselines, while few-shot retrained HLSM and FILM barely complete tasks.LLM-Planner averages 7 GPT-3 calls per task versus 22 for SayCan, which still underperforms despite oracle environment information.
- Ablation Studies: Dynamic retrieval materially improves performance over random example selection, while object-focused logit biases provide a decent high-level planning accuracy boost.The biases help resolve ambiguous object references, such as mapping “RecycleBin” to “GarbageCan.”
- Fine-grained Analyses: High-level planning accuracy generally increases with more training examples but shows diminishing returns around 250, while 9 in-context examples is generally effective.Additional in-context examples may yield only slight gains, and are more useful when fewer training examples are available.
- Case Studies: Grounded re-planning helps localize and disambiguate objects, including inferring a cup’s likely cabinet location and grounding “lamp” to a desklamp.These cases illustrate how environmental observations can correct or refine the high-level plan.
6. Conclusion
LLM-Planner is a high-level LLM planner for embodied agents operating in diverse, partially observable, complex environments. It dynamically replans from environmental perception and reduces the human annotations needed for instruction following.
- LLM-Planner provides high-level planning for embodied agents in diverse, partially observable, and complex environments.
- The planner dynamically replans from environmental perception to produce more grounded plans.
- LLM-Planner substantially reduces the human annotations required for learning instruction following.
- The work supports developing versatile and sample-efficient embodied agents through LLM-based planning with physical grounding.
Appendices
The supplementary material provides implementation details, additional comparisons, prompt-design analysis, and fine-grained analyses supporting the main paper.
- Appendix A contains additional model implementation details.
- Appendix B compares LLM-Planner with (SL)3 on ALFRED.
- Appendix C examines prompt-design choices and prompt selection under the true few-shot setting.
- Appendix D presents additional fine-grained analyses corresponding to the main paper’s analysis section.
A. Additional Model Implementation Details
The implementation details describe hierarchical baselines that combine semantic mapping, language-conditioned high-level planning, and low-level action execution. Few-shot adaptations retrain only components requiring paired instruction–trajectory data.
- HLSM: HLSM builds a 3D semantic voxel map, predicts high-level subgoals, and uses a low-level planner for execution.
- HLSM: HLSM’s high-level controller is retrained with the same 100 training examples used for LLM-Planner in the few-shot setting.
- FILM: FILM uses a semantic map, semantic search policy, template-based high-level planner, and low-level planner.
- FILM: FILM’s five BERT-based language classifiers are retrained with the same 100 examples to adapt the model to few-shot learning.
A.3. SayCan
The SayCan implementation adapts its skill-ranking framework to ALFRED by combining LLM relevance with executability estimates. Because the value function could not be trained, the comparison uses oracle environment information, while LLM-Planner requires fewer calls.
- SayCan components: SayCan combines an LLM ranker, a skill set, and a value function to select executable skills for task completion.
- ALFRED adaptation: Each ALFRED skill is treated as analogous to an LLM-Planner high-level action–object pair, enabling use of the same low-level controller.
- Evaluation setting: SayCan receives oracle object and affordance information because a trained value-function policy could not be produced under resource constraints.
- LLM-call efficiency: 22 versus 7 average GPT-3 calls per task: SayCan requires more calls than LLM-Planner despite oracle knowledge shrinking its skill list.
B. Comparison with (SL)3 on ALFRED
LLM-Planner achieves comparable validation performance to (SL)3 while using less than one-twentieth of ALFRED’s training data. The comparison is not performed under the same few-shot setting because (SL)3’s code was unavailable.
- 0.5% vs. 10% of ALFRED’s training data yields comparable validation performance for LLM-Planner and (SL)3.(SL)3 uses 10% of ALFRED’s training data, whereas LLM-Planner uses less than 0.5%.
- The authors could not compare (SL)3 under the same few-shot setting because its code was not publicly available.
- Table 5 reports task success rate for the scaling experiment on ALFRED’s valid unseen split.
D.1. HLP Accuracy by Task Type
LLM-Planner’s high-level planning accuracy is relatively stable across task types rather than closely tracking task difficulty. This pattern suggests robustness across varied tasks from few in-context examples.
- Stack & Place, often considered the most difficult task by success rate, has HLP accuracy similar to easier tasks such as Place two.
- HLP accuracy does not depend strongly on task difficulty across the evaluated task types.
- The authors report that LLM-Planner is not overly sensitive to task complexity.
- The observed task-type pattern suggests that LLM-Planner could generalize to different tasks with only a few in-context examples.
D.2. End-to-End Performance by Task Type
Dynamic LLM-Planner combined with HLSM achieves end-to-end performance comparable to full-data HLSM across a similar task-type distribution. The results also expose substantial dependence on low-level perception and planning quality.
- LLM-Planner + HLSM achieves comparable end-to-end performance to HLSM trained with ALFRED’s full training set.The comparison is not apples-to-apples because LLM-Planner uses few-shot training.
- The similar performance distributions likely reflect the shared low-level planner and object detector, which introduce similar error profiles.
- 36% HLP accuracy for Heat & Place corresponds to only 1.8% success rate because HLSM’s object detector often fails to detect the microwave.
- The authors identify substantial room for improvement through better low-level planners and object detectors.