PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application

Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi

arXiv:2304.03893v6cs.RO

TL;DR

Natural-language robot task planning has often been limited in scope, hardware-dependent, or tied to specific datasets. This paper uses customizable prompts to structure ChatGPT’s actions and environment representations; experiments report multi-step planning across environments, with success improving after user feedback.

  • Problem

    Many existing natural-language robot planners are limited in operation scope, hardware-dependent, lack human-in-the-loop functionality, or depend on datasets that require recollection and retraining for new settings.

    Source: §I · p. 2

  • Method

    The authors customize prompts that define application-specific robot actions and represent environments through assets, objects, and spatial-relation states.

    Source: B. The definition of robot actions · p. 3, C. Representation of the environments · p. 3

  • Results

    ChatGPT’s VirtualHome action-sequence success rate was 36% without feedback and approached 100% after several rounds of feedback.

    Source: §IV · p. 10

  • Takeaways & Limitations

    The prompts worked for multi-step instructions in various environments, and users could adjust outputs interactively, making them practical resources for robotics research.

    Source: §IV · p. 10, §VI · p. 16

    The prompts assume a robot with an arm, sufficient degrees of freedom and reachability, and instructions at grasp-manipulation-release granularity.

    Source: §IV · p. 10

Abstract

from arXiv · show

This paper demonstrates how OpenAI's ChatGPT can be used in a few-shot setting to convert natural language instructions into a sequence of executable robot actions. The paper proposes easy-to-customize input prompts for ChatGPT that meet common requirements in practical applications, such as easy integration with robot execution systems and applicability to various environments while minimizing the impact of ChatGPT's token limit. The prompts encourage ChatGPT to output a sequence of predefined robot actions, represent the operating environment in a formalized style, and infer the updated state of the operating environment. Experiments confirmed that the proposed prompts enable ChatGPT to act according to requirements in various environments, and users can adjust ChatGPT's output with natural language feedback for safe and robust operation. The proposed prompts and source code are open-source and publicly available at https://github.com/microsoft/ChatGPT-Robot-Manipulation-Prompts

I. INTRODUCTION

The paper presents customizable ChatGPT prompts for multi-step robot task planning, targeting portability across environments, integration with execution systems, and reduced impact from token limits. The prompts structure actions, environmental information, and updated states; experiments report planning across environments and adjustment through natural-language feedback.

I. INTRODUCTION

  • Existing natural-language robot planners often have limited operation scope, hardware dependence, lack human-in-the-loop functionality, or require dataset recollection and retraining for new settings.The paper positions its work as an extensible ChatGPT task-planning use case.Source: §I · p. 2
  • The study designs customizable prompts to address practical needs: system integration, applicability across home environments, and multi-step instructions despite ChatGPT’s token limit.The prompts are intended as a specific but extensible use case of ChatGPT for task planning.Source: §I · p. 2
  • The prompts specify user-defined robot actions in parseable JSON, represent spatial relations, and estimate post-operation environments to support subsequent planning.The paper reports that users can adjust outputs with natural-language feedback, and VirtualHome tests found executable and correct plans after several feedback rounds.Source: §I · p. 2
  • The paper explores whether ChatGPT can plan effectively across diverse environments and scenarios, extending earlier work focused on specific settings.The prompts are presented as customizable templates and made publicly available.Source: §I · p. 2

II. CHATGPT PROMPTS

  • Prompts one through five are pre-prepared, while the sixth is generated from user instructions and environmental information; preliminary tests favored supplying them as six conversation turns.The prompt design assumes an arm with sufficient degrees of freedom and reachability, and instructions at grasp-manipulation-release granularity.Source: §II · p. 2

A. The role of ChatGPT

The prompt design defines ChatGPT’s role, application-specific robot actions, a structured environment representation, and an output format intended to integrate with other pipelines.

A. The role of ChatGPT

  • The first prompt establishes ChatGPT’s role and instructs it to wait for subsequent prompts until the full set has been provided.The paper describes this as part of a multi-prompt interaction.Source: A. The role of ChatGPT · p. 2, Figure 2 · p. 3

B. The definition of robot actions

  • Robot actions are application-specific; the example defines them as functions changing motion constraints, enabling a theoretically necessary and sufficient action set for manipulation.The experiments use these actions except for the VirtualHome experiment, which uses a separately prepared set.Source: B. The definition of robot actions · p. 3, Figure 3 · p. 3

C. Representation of the environments

  • The environment prompt distinguishes non-manipulable assets from manipulable objects and represents spatial relations as states selected from a customizable STATE LIST.The VirtualHome experiment uses different states to fit its specifications.Source: C. Representation of the environments · p. 3, Figure 4 · p. 3

D. The format of the output produced by ChatGPT

E. Examples of input and output

The prompts pair example inputs and outputs with a user-specific instruction and environment, allowing later planning to reuse ChatGPT’s updated environment. Experiments used a fixed GPT model, and outputs were assumed to be reviewed by a user before saving.

E. Examples of input and output

  • Figure 6 provides examples of the desired prompt inputs and outputs.The full information is available at the URL provided in the text.Source: Figure 6 · p. 4

F. Specific instruction from the user

  • The user supplies the instruction and environment in a dynamic prompt; later planning can reuse the updated environment from ChatGPT’s previous output.The initial environment is supplied separately, for example through manual preparation.Source: F. Specific instruction from the user · pp. 4–5

III. EXPERIMENTS

F. Specific instruction from the user

  • The conversation structure presents task planning as a sequence of prompts, with the updated environment available for subsequent planning.The figure is described as the entire structure of the conversation with ChatGPT in task planning.Source: Figure 8 · p. 5, F. Specific instruction from the user · p. 5

A. Multi-step manipulation of the environment

The authors qualitatively tested multi-step household tasks across object relocation, opening doors, and cleaning, and found the generated plans adequate for executable actions. ChatGPT also adjusted action sequences in response to natural-language feedback.

A. Multi-step manipulation of the environment

  • The results suggest that ChatGPT translated multi-step human instructions into adequate sequences of executable robot actions.Authors manually inspected whether action sequences were executable and whether environment estimates were reasonable.Source: A. Multi-step manipulation of the… · pp. 5–6

1) Relocation of objects on a table:

  • The relocation task moved juice between a shelf and table, discarded spam, then discarded the juice; ChatGPT’s output demonstrated successful planning.The task comprised four successive operations involving two objects and a trash bin.Source: 1) Relocation of objects on a table · p. 6, Figure 9 · p. 6

2) Open a fridge/drawer door:

  • ChatGPT successfully planned opening a refrigerator, widening its door, removing juice, placing it on the floor, and closing the door; drawer-opening results were similar.Source: 2) Open a fridge/drawer door · p. 6

3) Wipe a window with a sponge, and throw it away:

B. Adjustment of the output produced by ChatGPT through user feedback

  • After users requested an action be added or removed, ChatGPT changed the output sequence according to the feedback’s meaning.The authors note that users should review and correct incomplete sequences for safe and robust operation.Source: B. Adjustment of the output produced… · p. 6

C. Quantitative evaluation of task decomposition performance of ChatGPT

The quantitative evaluation tested whether ChatGPT could produce household task plans executable in VirtualHome and correct on visual inspection. The authors prepared 14 multi-action scenarios involving object manipulation and everyday activities.

C. Quantitative evaluation of task decomposition performance of ChatGPT

1) Experimental setup:

  • VirtualHome represents household objects with unique IDs and their relationships as a graph, and provides predefined atomic actions through a Python-based API.Source: 1) Experimental setup · p. 7
  • Fourteen kitchen scenarios were selected to involve sequential actions, at least one object grasp or release, and everyday activities.The authors manually identified action sequences and object IDs for the scenarios.Source: 1) Experimental setup · p. 7, Table I · p. 9

2) Results:

Without feedback, ChatGPT produced successful sequences for about 36% of scenarios; with automatically generated error feedback, it reached successful sequences in all 14. Environment estimates were accurate across scenarios and trials, even when action sequences failed.

2) Results:

  • Across 14 scenarios, generated action sequences achieved an approximately 36% success rate (5 of 14) without feedback, with minimal variation between trials; temperature was set to its maximum to ensure trial-to-trial output variation.Source: 2) Results · p. 9
  • The identified failure patterns were incorrect verb selection and omission of necessary steps, including actions such as opening a container before placing an object inside.The authors investigated whether these errors could be corrected through feedback.Source: 2) Results · p. 9
  • ChatGPT produced successful action sequences in all 14 scenarios after several rounds of automatically generated feedback.The feedback system detected errors and returned generated error messages; the number of rounds varied by scenario.Source: 2) Results · p. 9, Table III · p. 10, Figure 13 · p. 10
  • ChatGPT reflected the semantic content of auto-generated feedback and made necessary output adjustments.Figure 13 illustrates an example of this process.Source: 2) Results · pp. 9–10
  • Environment estimates were accurate in all 14 scenarios across five trials, regardless of whether the action sequence succeeded.Source: 2) Results · p. 10

IV. DISCUSSION: TOWARDS MORE GENERAL ROBOTIC APPLICATIONS

The prompts support multi-step robot planning across environments and interactive adjustment, while the authors identify assumptions and limits on interpreting ChatGPT’s planning process. VirtualHome results show a 36% success rate without feedback, rising to nearly 100% after several rounds.

IV. DISCUSSION: TOWARDS MORE GENERAL ROBOTIC APPLICATIONS

  • The prompts target multi-step instructions across environments and allow users to adjust outputs interactively.The authors present the prompts as practical resources for the robotics research community.Source: §IV · p. 10
  • ChatGPT plans without object-manipulation experience, but its internal computation, including post-operation environment estimation, remains unclear.The authors suggest learned object-manipulation knowledge may contribute, but cannot access the model’s inner computations.Source: §IV · p. 10
  • 36% without feedback, rising to nearly 100% after several rounds, was ChatGPT’s VirtualHome action-sequence success rate.The authors say this aligns with prior LLM task-planning results, despite using a different task set.Source: §IV · p. 10
  • The task planner assumes a capable, reachable robot and instructions at grasp-manipulation-release granularity, assumptions the authors say may restrict some scenarios.Source: §IV · p. 10

A. Handling of conditional branching

The discussion describes prompt extensions for conditional plans and multiple arms or robots, alongside environmental-state handling. These extensions are presented as feasible, while static-world assumptions constrain the current approach.

A. Handling of conditional branching

B. Collaboration of multiple arms and robots

  • Small prompt modifications enabled action sequences involving multiple arms; a separate process composed plans to coordinate arms or robots.The authors describe these results as feasibility evidence for extending the planner.Source: B. Collaboration of multiple arms and… · p. 11

C. Managing environmental changes

  • Environmental information is input to and output from ChatGPT, but it must be prepared for the initial planning instance.The authors propose exploring a separate ChatGPT process to prepare it from symbolic scene understanding supplied by a vision encoder or human explanation.Source: C. Managing environmental changes · pp. 11–12
  • The current approach assumes a static environment whose changes come only from robot actions and whose state stays consistent between planning and execution.Movement, introduction, or disappearance of objects, including people, are identified as dynamic changes for future work.Source: C. Managing environmental changes · p. 12

D. Connection with vision systems and robot controllers

The discussion situates task planning alongside vision and robot control, then considers token constraints and prompting strategies. It identifies action-name confusion and ambiguous action granularity as error sources that prompt changes may address.

D. Connection with vision systems and robot controllers

  • The planner addresses what to do, while vision or robot systems provide how-to-do information such as grasping, focus, or posture.The authors say how-to-do information is often more effectively demonstrated visually, and treat its integration as outside this paper’s scope.Source: D. Connection with vision systems and… · p. 12

A. Token limit

  • Estimating the post-operation environment can reduce the burden of long conversation histories, but does not eliminate token-limit constraints.Long action explanations, extensive environment descriptions, and feedback can consume tokens; lengthy instructions may need to be split into smaller segments.Source: A. Token limit · pp. 12–13

B. Optimal prompting strategies

  • VirtualHome tests revealed incorrect verb selection and omitted necessary steps as two failure patterns.Similar action names may confuse ChatGPT; renaming “Put” and “PutIn” reduced this error type, though the data were not shown.Source: B. Optimal prompting strategies · pp. 13–14
  • Adding an example at the intended action granularity reduced omissions of microwave-opening and -closing steps in a follow-up experiment.The authors connect omissions to ambiguity about whether an instruction denotes one action or a more detailed sequence; the follow-up data were not shown.Source: B. Optimal prompting strategies · p. 14

C. Variations in the expression of instructions

Tests found consistent performance across outcome-focused and differently worded instructions, but the authors limit that conclusion to a reasonable range of variation. They also describe Azure OpenAI safeguards and a vision-to-symbolic-scene workflow.

C. Variations in the expression of instructions

D. Data privacy and security

  • The system uses Azure OpenAI, which the authors say supports compliance with data-security and privacy regulations and standards.They also cite content filtering and abuse monitoring as risk-mitigation features.Source: D. Data privacy and security · pp. 14–15
  • A described workflow uses an image-description model to obtain symbolic scene information, then ChatGPT to format it.Source: Figure 18 · p. 15
  • The authors call for continued assessment and improvement of protective measures as understanding of potential risks evolves.Source: D. Data privacy and security · p. 15

E. Future directions

The paper presents ChatGPT task planning as adaptable across settings through few-shot learning and user feedback, while identifying user studies and cross-model testing as future work.

E. Future directions

  • Few-shot learning and user feedback support adaptation across operational settings without extensive data collection or model retraining.These capabilities are described as facilitating safe and robust task planning.Source: E. Future directions · p. 15
  • The paper links few-shot learning to larger models and extended training, and user-feedback accommodation partly to reinforcement learning from human feedback.It notes that GPT-4 and Llama2-chat have been reported to possess similar features.Source: E. Future directions · p. 15
  • A teaching-system example pairs task planning with stepwise visual demonstrations that yield execution parameters for two robot hardware types.The figure describes an integrated teaching system using the planner.Source: Figure 19 · p. 16
  • Future research will test whether other models can achieve results comparable to ChatGPT for task planning.Source: E. Future directions · p. 16
  • Experiments suggest that reasonable amounts of feedback can adjust ChatGPT’s output, but the study did not test the effect on user-friendliness.The authors identify usability studies and comparisons with direct output editing as future work.Source: E. Future directions · p. 16

VI. CONCLUSION

  • The prompts translated multi-step instructions into executable robot actions across tested environments, and users could adjust outputs with natural-language feedback.The prompts and source code are open source and publicly available.Source: §VI · pp. 16–17

APPENDIX A DETAILED PROMPT FOR DEFINING ROBOT ACTIONS

The appendices describe the prompt’s action definitions, supplementary VirtualHome materials, and an expert-guided robot teaching system that adds visual demonstration and execution stages to task planning.

APPENDIX A DETAILED PROMPT FOR DEFINING ROBOT ACTIONS

APPENDIX B SUPPLEMENTARY INFORMATION FOR VIRTUALHOME EXPERIMENT

  • The VirtualHome supplement lists predefined atomic actions and fourteen scenarios, including textual instructions, manually identified action sequences, and higher-level instructions.It also points to results for the higher-level-instruction experiment.Source: Appendix B · p. 19, Table IV · p. 19

APPENDIX C AN EXAMPLE OF A ROBOT SYSTEM EXPANDING THE PROPOSED TASK PLANNER

  • The teaching system is intended for experts familiar with robot operations and action definitions, not as an automatic solution for non-experts.Its objective is to simplify robot teaching without complex coding, using multimodal demonstrations and language feedback.Source: Appendix C · p. 19
  • The system assumes that the initial environment representation and instruction sequence are prepared manually.Source: Appendix C · p. 19
  • The workflow plans tasks with feedback, gathers execution parameters through stepwise visual demonstration, then simulates before real execution.If execution fails or produces an unexpected result, the planning and demonstration steps are revisited; real testing follows safe simulation.Source: Appendix C · p. 19, Figure 21 · p. 20
  • Visual demonstrations provide robot parameters through RGB-D analysis, including motion features and discrete arm-posture directions represented by 26 unit vectors.The table summarizes actions and parameters obtained from demonstrations and on-site execution.Source: Appendix C · p. 19, Table VII · p. 23
  • Execution assumes the demonstration and execution environments match closely enough not to affect the action sequence or discrete posture representation.The vision system re-detects objects to correct slight misalignments, and some tasks use force feedback.Source: Appendix C · p. 19
  • Robot movements use inverse kinematics with postural constraints or pretrained reinforcement-learning policies, while ChatGPT’s predicted post-operation environment serves only as a planning hint.That predicted state is not used during robot execution in this example system.Source: Appendix C · pp. 19–20

Related papers

Loading 2304.03893v6…