Source-linked AI summary
Interactive Task Planning with Language Models
Boyi Li, Philipp Wu, Pieter Abbeel, Jitendra Malik
TL;DR
Robots must decompose tasks such as drink preparation into many steps while accommodating user feedback during execution. ITP combines language-model planning, robot-skill function calls, and pretrained visual grounding; the paper reports successful drink-task performance and replanning from revised requests. Its evaluated scope is a real-world drink-making robot.
Problem
Robot task execution requires multiple intermediate steps, while users may modify or give feedback on an initial request during execution.
Source: §1 · p. 1Method
ITP combines high-level language-model planning with low-level robot-skill function calling and pretrained vision models to ground the scene in language.
Source: §1 · pp. 1–2Results
ITP generates pertinent step-by-step plans and achieves drink-making outcomes, and the authors report that it responds to feedback during execution.
Source: §4.2 · p. 7, §5 · p. 10Takeaways & Limitations
The drink-making robot demonstrates interactive planning and replanning in a real-world robotics setting.
Source: §5 · p. 10Recognition or robot-execution errors can lead to task failure because ITP relies solely on LLMs for replanning.
Source: §4.6 · p. 9
Abstract
from arXiv · showhide
An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals and distinct tasks, even during execution. However, most traditional methods require predefined module design, making it hard to generalize to different goals. Recent large language model based approaches can allow for more open-ended planning but often require heavy prompt engineering or domain specific pretrained models. To tackle this, we propose a simple framework that achieves interactive task planning with language models by incorporating both high-level planning and low-level skill execution through function calling, leveraging pretrained vision models to ground the scene in language. We verify the robustness of our system on the real world task of making milk tea drinks. Our system is able to generate novel high-level instructions for unseen objectives and successfully accomplishes user tasks. Furthermore, when the user sends a new request, our system is able to replan accordingly with precision based on the new request, task guidelines and previously executed steps. Our approach is easy to adapt to different tasks by simply substituting the task guidelines, without the need for additional complex prompt engineering. Please check more details on our https://wuphilipp.github.io/itp_site and https://youtu.be/TrKLuyv26_g.
1 INTRODUCTION
ITP addresses the challenge of planning multi-step robot tasks while remaining responsive to user feedback during execution. It combines language-model planning and skill execution, and can revise a plan using the new request, completed steps, and task guidelines.
1 INTRODUCTION
- A high-level planner turns requests and task guidelines into step-by-step plans, while a low-level executor maps steps to robot skills through a functional API.The API lets GPT-4 call robot skills without exposing code-level details.Source: §1 · pp. 1–2
- ITP is a training-free robot framework that combines language-model planning, vision, and skill execution in real-world drink making.The paper presents the system as a simple framework for interactive task planning.Source: §1 · pp. 1–2
- ITP uses a simple prompt format and natural-language task input rather than code-level prompt engineering.The authors report that the format is effective in simulated and real settings.Source: §1 · p. 2
- When a user changes a request during execution, ITP replans using the updated goal, completed steps, and task guidelines.The illustrated interaction revises a drink order after the robot has begun the initial task.Source: §1 · p. 2, Figure 1 · p. 2
2 RELATED WORK
The related work spans symbolic task planning and language-model-driven robot planning. ITP uses language and contextual information for planning, then executes plans through a robot API grounded in visual input.
2.1 Task planning
- Traditional symbolic task planning formalizes actions, preconditions, and effects, while ITP replaces that pipeline with language models and natural-language problem scope.The paper notes that symbolic definitions can require meticulous parameter specification.Source: §2.1 · p. 3
2.2 Language Models as Planners in Robotics
- Prior language-model robot planners include zero-shot action prediction from language-converted scenes (Huang et al., 2022) and systems coordinating pretrained models, including SayCan (Zeng et al., 2022).Unlike SayCan, ITP relies on task guidelines and robot skills rather than a pretrained value function to ground actions.Source: §2.2 · p. 3
- Code as Policies (Liang et al., 2022) and ProgPrompt (Singh et al., 2023) generate executable code and often require example-code prompt engineering; ITP instead uses a functional robot API.Language-Guided Robot Skill Learning (Ha et al., 2023) also uses hierarchical planning but assumes simulator-provided ground-truth state.Source: §2.2 · p. 3
3 METHOD
ITP separates high-level planning from low-level skill execution and supports replanning when a user changes a request. The method combines natural-language task guidance, visual scene grounding, and callable robot skills.
3 METHOD
- ITP uses a two-level architecture: one language model produces high-level steps, and another realizes each step through the robot’s functional API with visual grounding.A pretrained vision-language model translates the visual scene into language-compatible descriptions.Source: §3 · pp. 3–4
3.1 High-level Planning
- The high-level planner uses GPT-4, a user request, and natural-language task guidelines with few-shot examples and available-material descriptions to generate a step-by-step plan.The guidelines help assess whether a requested drink can be made and support plans for drinks beyond the examples.Source: §3.1 · p. 4
3.2 Low-level Execution
- The low-level executor uses pretrained vision modules to describe object locations and exposes Python robot skills as callable functions derived from function definitions and docstrings.Grounded-DINO supplies object boxes, with projective mapping used to approximate robot-frame locations; the paper notes these assignments can be imprecise.Source: §3.2 · pp. 4–5
3.3 Replanning
- On a changed request, ITP replans using completed steps, task guidelines, the new request, and chat history, then executes the new plan’s remaining steps.The example revises the drink order after the cup has been grasped.Source: Figure 2 · p. 5, §3.3 · p. 6
4 EXPERIMENTS
The experiments evaluate ITP on real-robot drink making, human-in-the-loop replanning, Virtual Home planning, and dishwashing. The reported results show performance on varied requests and tasks, alongside limitations involving task ambiguity, perception, execution, and latency.
4.1 Robot Experiments
- The real-robot setup combines supplied drink ingredients, an overhead camera, Grounded-DINO scene understanding, and predefined manipulation skills.Planning accuracy is measured as successful steps / total steps and averaged across user requests; robot success is assessed by task completion.Source: §4.1 · p. 6, Table 1 · p. 7
4.2 Comparison on Task Planning
- ITP handles unseen drink requests and unavailable ingredients on real robots, while Code as Policies can misorder complex plans and cannot justify missing materials.The comparison gives both systems user requests and task guidelines, and gives Code as Policies additional code examples.Source: §4.2 · p. 7
4.3 Replan with Human-in-the-loop Feedback
- After interruptions before different steps, ITP replans, but matcha requests can add a stirring step, producing 5 and 7 steps instead of 4 and 6.The authors attribute the extra step to ambiguity about mixing matcha and suggest adding restrictions to the task guidelines.Source: Table 2 · p. 8, §4.3 · p. 8
4.4 Comparison on Simulation Tasks
- ITP matches or exceeds ProgPrompt on executable functions across Virtual Home tasks, using ProgPrompt’s low-level execution.Executability counts actions executable in the environment, even when they are irrelevant to the task.Source: Table 3 · p. 8, §4.4 · p. 8
4.5 Adaptation to Other Tasks
- ITP adapts to other tasks by modifying task guidelines and low-level skill documentation; prompt tuning is optional.The dishwashing example replaces drink-making guidelines with dishwashing guidelines.Source: §4.5 · p. 9
4.6 Analysis and Discussion
- ITP’s replanning relies on LLMs, so recognition or execution errors can cause task failure; GPT-4 latency can also lengthen tasks, and VLM-based replanning is proposed.The paper gives misidentifying milk and a line entangling with the robot arm as examples of risks.Source: §4.6 · pp. 9–10
4.5 Adaptation to Other Tasks
- Dishwashing achieved a 100% high-level planning rate after changing the text guidelines, with plans specifying the correct order and utensil locations.The caption attributes the rate to the absence of misleading material or content in the dishwasher task.Source: §4.5 · p. 9, Table 4 · p. 10
4.6 Analysis and Discussion
- The authors position ITP as combining user-friendly task guidelines with replanning from new user requests, which they say prior approaches did not both provide.The discussion contrasts this focus with approaches centered on planning and execution or on the task rather than user experience.Source: §4.6 · p. 10
5 Conclusion
ITP interprets user requests, generates step-by-step plans, and achieves desired outcomes in a drink-making robot that responds to feedback during execution. The authors frame it as a proof of concept whose capabilities could be strengthened for more intricate tasks and richer visual understanding.
5 Conclusion
- ITP precisely interprets user requests, generates pertinent step-by-step plans, and achieves desired outcomes while responding to feedback during drink making.The system is demonstrated on a robot designed to make drinks according to user preferences.Source: §5 · p. 10
- As a proof of concept, ITP needs more powerful robot skills to tackle more intricate tasks.Source: §5 · p. 10
- More precise visual information leveraging 3D information could elevate the robot’s understanding, planning, and interaction with its surroundings.Source: §5 · p. 10
- The authors hope their open-source system inspires research using established and emerging models to enhance real-world robotics.Source: §5 · p. 10