Source-linked AI summary
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg
TL;DR
LLM-based robot planners struggle with situated state and executable action constraints. PROGPROMPT addresses this with program-like prompts describing actions and objects, plus executable examples, and reports strong VirtualHome performance and physical-robot deployment. Its scope is bounded by environment-specific failure modes, incomplete generations, and strict or ambiguous task evaluation.
Problem
LLM planners either enumerate possible next actions for scoring or generate free-form plans that may contain actions or objects unavailable in the robot’s current context.
Method
PROGPROMPT prompts LLMs with available actions, environment objects, executable example programs, comments, and assertions with recovery actions.
Results
PROGPROMPT outperforms prior work across success rate, goal conditions recall, and executability, generalizes to new VirtualHome scenes, and is deployed on a physical robot arm.
Takeaways & Limitations
Program-language structure and situated information enable direct generation of executable plans across environments, agents, and tasks within the demonstrated settings.
Takeaways & Limitations
Generated programs can fail because environment peculiarities are not explicitly communicated, plans can be truncated by API caps, and strict goal checking may reject plausible task completions.
Abstract
from arXiv · showhide
Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at progprompt.github.io
I. INTRODUCTION
Robot task planning needs both commonsense knowledge and feedback about the current environment. PROGPROMPT addresses this by combining program-like prompts with situated state information and executable plans.
- Motivation: Household tasks require commonsense knowledge about object affordances, action order, and task-relevant objects.State feedback is also needed to identify which objects are actually available in the current environment.
- Motivation: Existing LLM planners either score enumerated candidate actions or generate free-form plans that must be mapped to executable robot actions.Free-form outputs may mention objects or actions unavailable to the robot.
- PROGPROMPT: PROGPROMPT supplies imports for available actions, an object list, example tasks, and program functions whose action sequences operate on environment objects.The prompt structure uses programming-language representations alongside natural-language task instructions.
- PROGPROMPT: Assertions encode plan preconditions and trigger recovery actions when state checks fail during execution.This allows generated programs to respond to situated feedback rather than executing entirely open-loop.
II. BACKGROUND AND RELATED WORK
Prior work frames planning through search, learned methods, or language-model prompting, but faces scalability and executability challenges. PROGPROMPT instead generates executable programs using programming-language structures and environment information.
- Task Planning: Robot task planning commonly uses search over predefined domains, while large action and object spaces can make unconditional search difficult to scale.Heuristics and learning-based approaches have been proposed to guide or replace aspects of search.
- Problem Formulation: The task-planning formulation includes objects, properties, actions, transitions, initial and goal states, and a natural-language task description.The agent receives the high-level task description but does not directly access the goal state.
- Planning with LLMs: LLM prompting transforms state observations into textual prompts, after which the model either generates text or scores predefined options.Prompt design is challenging because executable plans are not paired routinely with natural-language instructions.
- Related Work: Open-domain generation and action matching do not ensure that proposed actions are admissible in the current situation, while enumerating admissible actions creates combinatorial challenges.Prior systems therefore differ in how they incorporate environment feedback and action feasibility.
- Related Work: PROGPROMPT extends code-completion planning with imports for robot capabilities, comments for commonsense reasoning, assertions for execution state, and direct generation of an executable plan program.Its answer search generates the entire program rather than scoring an enumerated action set.
III. OUR METHOD: PROGPROMPT
PROGPROMPT represents robot plans as Pythonic programs that combine action calls, object arguments, comments, and assertions. These structures support decomposition, situated execution, and recovery from unmet preconditions.
- Program Representation: PROGPROMPT prompts the LLM with Pythonic code and asks it to complete a situated robot task plan conditioned on a natural-language instruction.The plan is represented as a program rather than free-form action text.
- Program Representation: Plan functions call action primitives with objects as arguments.For example, the “put salmon in the microwave” task includes an action call such as find(salmon).
- Comments: Comments summarize upcoming action sequences and divide high-level tasks into logical sub-tasks.They provide immediate goals that can reduce incoherent, divergent, or repetitive outputs.
- Feedback: Assertions check environmental preconditions before actions and invoke recovery behavior when those conditions do not hold.For example, the plan checks proximity to salmon before grabbing it and calls find(salmon) if necessary.
B. Constructing Programming Language Prompts
PROGPROMPT constructs Pythonic prompts from environment observations, robot primitives, and executable examples. Imports constrain generated actions, while examples demonstrate valid task, object, and action relationships.
- Prompt Construction: The prompt function combines observations, action primitives, and sample tasks and plans into Pythonic code for the LLM to complete.The model predicts the next task as an executable function.
- Action Constraints: Import statements expose the robot’s available action primitives and encourage the LLM to generate only functions supported in the current context.Changing agents requires replacing the imported-function list.
- Examples: Executable example tasks show how task names, actions, and environment objects are related and which combinations are permitted.These examples also demonstrate restrictions on the actions and objects used in plans.
C. Task Plan Generation and Execution
PROGPROMPT generates executable task plans from program-like prompts and executes them with environment feedback. The method is evaluated in both virtual and physical robot settings.
- Table I compares PROGPROMPT with prior task-planning approaches and ablates its prompt features.The comparison includes different language-model backbones and natural-language prompting.
- PROGPROMPT plans are executed by an interpreter on either a virtual agent or physical robot system.Assertion checking occurs closed-loop during execution to provide current environment state feedback.
- The system evaluates generated programs in a virtual household environment and on a physical robot manipulator.
A. Simulation Experiments
The experiments use Virtual Home household scenes and a physical tabletop robot. The setup represents available objects and actions explicitly, with environment feedback used during virtual execution.
- Simulation: Virtual Home is a deterministic simulation platform for household activities with object properties and 11 executable action primitives.The action set includes grab, putin, putback, walk, find, open, close, switchon, switchoff, sit, and standup.
- Simulation: The study uses 3 Virtual Home environments containing 115 unique object instances each, including class-level duplicates.Objects have properties corresponding to action affordances, and some have semantic states such as heated, washed, or used.
- Simulation: The dataset contains 70 household tasks posed as high-level instructions with ground-truth action sequences and symbolic final-state goal conditions.
- Simulation: Virtual Home state feedback is checked against assertions by prompting the LLM with relevant state-graph information.
- Real robot: The physical setup uses a Franka-Emika Panda robot with a parallel-jaw gripper and a pick-and-place policy.Motion generation, collision avoidance, and grasp generation are implemented with MPPI, SceneCollisionNet, and Contact-GraspNet.
- Real robot: For tabletop plans, ViLD detects and segments objects so generated object names can be mapped to pointclouds for grab and putin actions.Unlike the virtual setup, assert-based closed-loop options are not implemented because of real-world uncertainty.
C. Evaluation Metrics
Performance is measured with success rate, goal conditions recall, and executability. These metrics distinguish complete task success, achieved task conditions, and action feasibility.
- Success rate (SR) is the fraction of executions achieving all task-relevant goal conditions.SR equals 1 only when goal conditions recall also equals 1.
- Goal conditions recall (GCR) compares achieved final-state conditions with ground-truth task-specific conditions.It is computed using the set difference between ground-truth final state conditions g and achieved conditions g′, divided by the number of task-specific goal conditions.
- Executability (Exec) is the fraction of planned actions executable in the environment, regardless of task relevance.
A. Virtual Experiment Results
Virtual Home experiments compare PROGPROMPT across models, prompting features, and environments. PROGPROMPT outperforms the cited baseline, benefits from feedback and comments, and adapts to new scenes and object meanings.
- Main comparison: PROGPROMPT substantially outperforms prior work using the same language-model backbone on success rate, goal conditions recall, and executability.CODEX exceeds GPT3 on every metric, while DAVINCI performs worse than base GPT3 under a two-example prompt-length constraint.
- Ablations: Feedback mechanisms improve performance across metrics, except executability increases slightly without feedback when examples contain no comments.
- Ablations: Removing comments from prompt code substantially reduces performance across all metrics.This result highlights the contribution of natural-language guidance within the programming-language structure.
- Baselines: LANGPROMPT generates natural-language action sequences that require a separately fine-tuned GPT2 policy to map text steps to executable simulation actions.
- Metric interpretation: Some tasks show low goal conditions recall despite high executability because evaluation compares against one true goal although multiple final states may be appropriate.For salmon, placing the prepared food on either a table or countertop illustrates this mismatch.
- Additional environments: In new Virtual Home scenes, PROGPROMPT infers that “cutlery” refers to spoons and forks even though examples use knives.
B. Qualitative Analysis and Limitations
The analysis identifies environment-specific limitations, inaccessible objects, missing action feedback, and incomplete or ambiguous plan generation as recurring failure sources. A sorting rollout nevertheless shows the planner using relevant objects while ignoring a distractor.
- Environment artifacts prevent the VirtualHome agent from interacting with nearby objects when sitting and omit some common-sense actions.
- In a sorting rollout, the LLM placed fruits on a plate, put the bottle in a box, and ignored the distractor drill.
- When objects are inaccessible, generated assertions may not suffice to produce prerequisite actions such as opening a cabinet.
- Missing action-success feedback can cause subsequent actions to fail, while assertion recovery does not cover every possibility.
- Some plans are truncated by LLM API caps, and strict final-state checking struggles with multiple valid solutions or extra actions.
C. Physical Robot Results
The physical-robot evaluation tested four increasingly difficult tabletop tasks in scenes with and without distractors. PROGPROMPT almost always succeeded, and every generated plan was executable, but the physical results are intended primarily as a qualitative demonstration.
- Four physical-robot tasks were tested in scenes containing only necessary objects and scenes with one to three distractors.
- The physical setup used PROGPROMPT with comments but without feedback because system-state tracking and assertion checking were unreliable.
- Plan SR measures whether a generated plan would likely succeed assuming successful pick-and-place execution without gripper failures.
- The system almost always succeeded across tasks and distractor conditions, failing only on sorting because of a gripper failure or a soup-can classification error.
- Generated plans had executability Exec=1 in every physical-robot result.
VI. CONCLUSIONS AND FUTURE WORK
The paper concludes that PROGPROMPT combines commonsense reasoning with code understanding to generate executable robot plans grounded in situated environments. It reports improved task performance and broad generalization, while proposing richer programming-language features for future work.
- PROGPROMPT combines LLM commonsense reasoning and code understanding for robot task planning.
- Prompts include situated world and robot-capability information, enabling direct generation of executable plans as programs.
- Experiments report improved task performance across multiple metrics and generalization to new scenes, agents, and tasks, including real-robot deployment.
- Future work will explore real-valued measurements, nested dictionaries for scene graphs, and more complex control flow.