Source-linked AI summary
Embodied Task Planning with Large Language Models
Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, Haibin Yan
TL;DR
LLMs can bring commonsense to embodied planning, but they cannot perceive the scene and may plan interactions with objects that are absent. TaPA tunes a planner on generated scene–instruction–plan triplets and grounds inference in objects detected from multi-view images. It reports higher success than GPT-3.5, with the largest stated comparison being a 6.38% average gain.
Problem
LLMs have commonsense knowledge for planning, but cannot perceive surrounding scenes and may generate actions involving objects that are not present.
Source: §1 · p. 1Method
TaPA fine-tunes a pre-trained planner on GPT-3.5-generated scene–instruction–plan triplets and grounds inference by conditioning on objects detected across multi-view images.
Source: §1 · p. 2, §3.2 · p. 6Results
TaPA's average success rate was 6.38% higher than GPT-3.5 (61.11% vs. 54.73%) after instruction fine-tuning, and it achieved optimal performance among the compared large models across four scenes.
Source: §4.2 · p. 7Takeaways & Limitations
TaPA provides a benchmark and planning framework for complex embodied tasks in realistic indoor deployment scenarios.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Equipping embodied agents with commonsense is important for robots to successfully complete complex human instructions in general environments. Recent large language models (LLM) can embed rich semantic knowledge for agents in plan generation of complex tasks, while they lack the information about the realistic world and usually yield infeasible action sequences. In this paper, we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models. Specifically, we first construct a multimodal dataset containing triplets of indoor scenes, instructions and action plans, where we provide the designed prompts and the list of existing objects in the scene for GPT-3.5 to generate a large number of instructions and corresponding planned actions. The generated data is leveraged for grounded plan tuning of pre-trained LLMs. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin, which indicates the practicality of embodied task planning in general and complex environments.
1 Introduction
LLMs bring commonsense to embodied planning but cannot perceive surrounding scenes, so they may produce actions involving absent objects. TaPA grounds plans in the objects present in a scene and targets complex tasks across indoor environments.
1 Introduction
- TaPA combines human instructions with detected scene objects to generate executable plans for navigation or manipulation.Its stated aim is grounded planning without restricting task types or target objects.Source: Figure 1 · p. 2, §1 · p. 2
- Prior systems are described as limited to kitchen scenarios or simple tasks in ALFRED, leaving complex tasks and diverse deployment scenarios unmet.The introduction names making sandwiches and setting tables as examples of complex household tasks.Source: §1 · p. 2
- TaPA is presented as the first benchmark for complex embodied task planning in realistic indoor deployment scenarios.The paper also reports a multimodal dataset with 80 indoor scenes and 15K instructions and corresponding action plans.Source: §1 · p. 2
- The paper evaluates LLMs and LMMs on its benchmark and uses ablations to select a visual-scene representation for action generation.The framework collects multiple RGB views, detects scene objects, then conditions planning on the object list and human instruction.Source: §1 · p. 2, Figure 1 · p. 2
- TaPA's instructions are described as more complex than ALFRED's, with longer action sequences for completion.Figure 2 compares frequent verbs and associated nouns, and the percentage of instructions grouped by number of implementation actions.Source: Figure 2 · p. 3
2 Related Work
Related work draws on large pre-trained language and vision models, as well as methods for grounding language-model plans in embodied environments. Existing approaches are described as relying on interaction or prompts, while work using visual scene information has focused on simpler simulated tasks.
2 Related Work
- Large pre-trained language and vision models provide generalization and open-environment object detection, while CLIP aligns image and text representations and LLaVA instruction-tunes multimodal models.The paper builds on LLMs for executable planning with visual information from open-vocabulary detection models.Source: §2 · p. 3
- Embodied planning research grounds LLMs through environment feedback or prompt engineering, including filtering generated plans for executability.Examples include interactive decision-making and prompts pairing simple instructions with actions.Source: §2 · p. 3
- Prior scene-aware methods are described as limited to simple placing and putting tasks in VirtualHome or ALFRED simulators, rather than diverse complex deployment tasks.Some approaches extract visual information as latent features or object names for planning.Source: §2 · p. 3
3 Approach
TaPA uses a multimodal instruction dataset to tune a planner, then grounds inference in detected objects from multi-view images. Its data-generation and image-collection strategies are designed to support complex plans constrained by the physical scene.
3 Approach
- The approach comprises multimodal instruction-dataset construction followed by grounding plans to visual scenes through image collection and open-vocabulary detection.The dataset is used to tune the TaPA task planner.Source: §3 · p. 4
3.1 Data Generation of Embodied Task Planning
- GPT-3.5 generates diverse instructions and executable action plans from a scene's object-name list using designed prompts; targets are constrained to listed objects.Samples pair visual scenes, instructions, and plans, and only instructions with explicitly executable actions are retained.Source: §3.1 · p. 4
- Training expands 80 AI2-THOR scenes to 6,400 by plausibly substituting objects, yielding 15K training samples and 60 evaluation triplets.The original split contains 80 training scenes and 20 evaluation scenes; the 15K samples are used for planner fine-tuning.Source: §3.1 · p. 5
3.2 Grounding Task Plans to Surrounding Scenes
- Scene-image collection samples achievable locations using traversal, random, whole-scene-center, or block-wise-center strategies, with camera views rotated at each location.Block-wise centers use K-means subregions, with the cluster count selected by within-cluster sum of squared errors; they traverse fewer locations than traversal sampling.Source: Figure 3 · p. 4, §3.2 · p. 6
- At inference, open-vocabulary detection across multi-view RGB images yields a deduplicated object list, which TaPA combines with the instruction to generate scene-constrained plans.The agent gathers images from different locations to discover objects that may be novel in deployment scenarios.Source: §3.2 · pp. 5–6
4 Experiment
Experiments used AI2-THOR scenes and a fine-tuned LLaMA-7B planner. Human raters assessed plan executability, and TaPA achieved higher success rates than the compared large models.
4.1 Evaluation Metrics
- Three volunteers judged each plan against the instruction and ground-truth object list; two of three votes marked it successful, with failures annotated as counterfactuals or hallucinations.Counterfactuals violate physical rules; hallucinations require interacting with absent objects.Source: §4.1 · pp. 6–7
4.2 Experimental Results
- 6.38% (61.11% vs. 54.73%) higher average success rate than GPT-3.5; TaPA led all tested large models across four room types.The comparison covered 60 validation samples.Source: §4.2 · p. 7, Table 2 · p. 7
- Kitchen plans had the lowest success because they require more steps; longer plans raise hallucination risk, while LLaVA's single-image view misses overall scene information.Complex kitchen tasks are also more prone to noise from redundant object lists.Source: §4.2 · pp. 7–9
- More image collection did not reliably improve success: smaller traversal grids increased image and computation costs, while extra camera rotations could reduce success through redundancy.The reported traversal success rates were 47.83 vs. 44.78; the rotation comparisons were 47.95 vs. 47.23 and 46.93 vs. 47.23.Source: Table 3 · p. 7, §4.2 · pp. 8–9
- Block-wise center points achieved the highest success rate, combining high scene coverage with few images and avoiding redundant object lists.Traversal positions had the lowest success, with excess images increasing false positives in object detection.Source: Table 3 · p. 7, §4.2 · p. 9
- In a qualitative example, LLaMA's plan was irrelevant and LLaVA's was non-executable, whereas TaPA's steps were more complete and consistent with human values.GPT-3.5 also produced plausible plans.Source: Table 4 · p. 8, §4.2 · p. 9
5 Conclusion
TaPA uses a GPT-3.5-generated multimodal dataset to tune an instruction model for executable task planning, then grounds inference with objects detected across multi-view scene images. The authors report more complex tasks than conventional instruction-following benchmarks and stronger plan plausibility than state-of-the-art LLMs and LMMs.
5 Conclusion
- TaPA outperforms state-of-the-art LLMs and LMMs on the plausibility of generated action plans.The authors also report that its tasks have longer implementation steps and are more complex than conventional instruction-following benchmarks.Source: §5 · p. 9
- A GPT-3.5-generated multimodal dataset pairs visual scenes and instructions with corresponding plans, and is used to tune the instruction model to generate executable actions.GPT-3.5 receives the scene's object list and designed text prompts to generate the dataset.Source: §5 · p. 9
- At inference, TaPA uses multi-view RGB images from achievable locations and open-vocabulary object detection to discover scene objects for the finetuned instruction model.Source: §5 · p. 9
Supplementary Material
The prompts specify a work scene and rules for generating instructions and corresponding actions. Generated actions include object queries to check for hallucinations, with alternatives allowed for absent objects.
Supplementary Material
- Prompts specify a work scene and rules intended to improve the executability and confidence of generated instructions and actions.They ask GPT-3.5 to generate instructions and corresponding actions.Source: Supplementary Material · p. 14
- Each generated action includes a target-object query used to check for hallucinations against the scene's object list.If the object is absent, the procedure checks for an alternative item; an interacting object may also be part of an existing object or its synonym.Source: Supplementary Material · pp. 14–15
- Examples are provided to standardize the form of the generated data's input and output.Source: Supplementary Material · p. 15