PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration

Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi

arXiv:2311.12015v4cs.ROcs.CLcs.CV

TL;DR

Pipelines combining off-the-shelf VLMs for multimodal task planning remain limited, and grounding task plans in environmental information is a robotics challenge. A two-pipeline system uses GPT-4V and GPT-4 to turn videos and/or text into symbolic robot actions, then analyzes videos for task timing and location and extracts execution affordances. 85–95% end-to-end performance was reported in real-robot experiments, comparable to model-based approaches such as [2].

  • Problem

    Pipelines combining off-the-shelf VLMs for multimodal task planning remain limited, and grounding task plans in environmental information is a robotics challenge.

    Source: §I · p. 1, B. Grounding visual information for… · p. 2

  • Method

    A two-pipeline system uses GPT-4V and GPT-4 to turn videos and/or text into symbolic robot actions, then analyzes videos for task timing and location and extracts execution affordances.

    Source: §I · p. 1, §III · p. 2

  • Results

    85–95% end-to-end performance was reported in real-robot experiments, comparable to model-based approaches such as [2].

    Source: §V · p. 7

  • Takeaways & Limitations

    The off-the-shelf-model pipeline is presented as flexible across robotic hardware and scenarios.

    Source: §VI · p. 7

    Video understanding was limited in cluttered kitchens and under self-motion blur, and selecting five regularly spaced frames can affect inference of critical actions.

    Source: §V · p. 7

Abstract

from arXiv · show

We introduce a pipeline that enhances a general-purpose Vision Language Model, GPT-4V(ision), to facilitate one-shot visual teaching for robotic manipulation. This system analyzes videos of humans performing tasks and outputs executable robot programs that incorporate insights into affordances. The process begins with GPT-4V analyzing the videos to obtain textual explanations of environmental and action details. A GPT-4-based task planner then encodes these details into a symbolic task plan. Subsequently, vision systems spatially and temporally ground the task plan in the videos. Objects are identified using an open-vocabulary object detector, and hand-object interactions are analyzed to pinpoint moments of grasping and releasing. This spatiotemporal grounding allows for the gathering of affordance information (e.g., grasp types, waypoints, and body postures) critical for robot execution. Experiments across various scenarios demonstrate the method's efficacy in enabling real robots to operate from one-shot human demonstrations. Meanwhile, quantitative tests have revealed instances of hallucination in GPT-4V, highlighting the importance of incorporating human supervision within the pipeline. The prompts of GPT-4V/GPT-4 are available at this project page: https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/

I. INTRODUCTION

The introduction identifies limited development of multimodal task-planning pipelines that combine off-the-shelf VLMs, then proposes a GPT-4V/GPT-4 system to turn demonstrations into robot task plans with affordances. It reports real-robot evaluation and highlights GPT-4V hallucinations as a limitation requiring human supervision.

I. INTRODUCTION

  • The paper identifies limited development of pipelines combining off-the-shelf VLMs for multimodal task planning.It positions the system amid off-the-shelf LLM/VLM approaches, which can be adapted by modifying prompts rather than collecting extensive data and retraining for each hardware configuration or scenario.Source: §I · p. 1
  • The pipeline accepts video, text, or both, aligns task steps with videos, extracts execution affordances, and compiles them with the plan into a hardware-independent JSON file.Examples of extracted information include grasp approach direction, grasp type, collision-avoiding waypoints, and upper-limb posture.Source: §I · p. 1
  • The authors report qualitative tests on several real robots and quantitative tests on a manually task-labeled public cooking-video dataset.They also observed GPT-4V hallucinations and identify human supervision as important within the pipeline.Source: §I · p. 1
  • The stated contributions include a ready-to-use planner, an affordance-grounding methodology, and publicly accessible code.The planner uses off-the-shelf VLM and LLM models.Source: §I · p. 1

A. LLM/VLM-based task planning

The related-work discussion positions this system within task-and-motion planning and motivates multimodal grounding and affordance extraction. The approach combines open-vocabulary object localization and hand-object interaction analysis to extract execution-relevant information from demonstrations.

A. LLM/VLM-based task planning

  • Off-the-shelf LLM approaches decompose instructions into high-level subgoals executed by pre-trained skills, and this study extends the TAMP approach to multimodal input with GPT-4V and GPT-4.The paper contrasts these approaches with trained models that may require substantial robot data and recollection or retraining when transferred to other settings.Source: A. LLM/VLM-based task planning · p. 2

B. Grounding visual information for robotics

  • Grounding plans in environmental information is a robotics challenge; this system localizes GPT-4V-named objects and detects grasp and release timing and locations from hand-object interactions.The resulting information supports estimates of grasp types, waypoints, and body postures for robot execution.Source: B. Grounding visual information for… · p. 2

C. Learning affordance

  • The paper treats affordance information as including waypoints, grasp types, and upper-limb postures, which may not be taught explicitly and therefore must be extracted from demonstrations.The proposed pipeline supplies this information to the task plan.Source: C. Learning affordance · p. 2

III. MULTIMODAL TASK PLANNER

The proposed planner connects a symbolic task-planning pipeline to an affordance analyzer, using video, text, or both. Its stages generate and check textual descriptions and plans, then analyze videos for task timing and execution affordances.

III. MULTIMODAL TASK PLANNER

  • The system assumes demonstrations occur in the robot’s operating environment, allows slight correctable object shifts, and uses videos covering grasp, manipulation, and release.The grasp-manipulation-release sequence runs from preparatory grasping actions through object release.Source: §III · p. 2

A. Symbolic task planner

  • The symbolic task planner uses GPT-4V to turn regularly sampled video frames into human-readable action instructions, which a user checks and edits.Five frames were used as a practical number given token limits and latency.Source: A. Symbolic task planner · pp. 2–3
  • The scene analyzer combines the instructions with the video’s first frame to describe recognized objects, graspable properties, and spatial relations relevant to the action.The prompt and input-output examples are shown in Fig. 4.Source: A. Symbolic task planner · p. 3, Figure 4 · p. 3
  • The GPT-4 task planner decomposes instructions into symbolic robot-task sequences and provides explanations, estimated post-operation environments, and summaries for user review and linguistic feedback.The task definitions are embedded in a pre-prompt, so extending the action set requires modifying that pre-prompt rather than preparing a training dataset.Source: A. Symbolic task planner · p. 3

B. Affordance analyzer

The affordance analyzer reanalyzes demonstrations using the symbolic plan to locate task events and derive information for robot execution.

B. Affordance analyzer

1) Attention to human hands to detect grasping and releasing:

The analyzer classifies regularly sampled video clips by whether an object is held at the first and last frames, then uses hand–object proximity to locate grasping and releasing.

1) Attention to human hands to detect grasping and releasing:

  • Clips are classified as grasp clips when an initially empty hand holds something in the final frame, or release clips when the reverse occurs.All other clips fall outside these two definitions; a YOLO-based hand detector and recognizer supports the classification.Source: 1) Attention to human hands to detect… · p. 4

2) Attention to hand-object interaction to detect the spatiotemporal location of grasping and releasing:

  • The hand and object are compared across grasp clips to identify the grasp moment, and the same calculation locates release timing.When several detected objects are candidates, the one closest to the hand is selected as the grasped object.Source: 2) Attention to hand-object… · p. 4

3) Extracting affordance from aligned videos:

After aligning task steps with video, the analyzer extracts affordances including hand directions, waypoints, grasp type, rotation parameters, and arm postures.

3) Extracting affordance from aligned videos:

IV. EXPERIMENTS

The pipeline was tested on two robots using one-shot demonstrations, with successful execution reported across environmental texture changes and a multiple-identical-can scenario.

A. Qualitative results

  • The implementation uses demonstration videos with depth channels to extract spatial affordances and was tested for single-attempt operation.The tested platforms were Nextage and Fetch robots, with a Shadow Dexterous Hand Lite attached.Source: A. Qualitative results · p. 5
  • Execution locates objects with a head-mounted RGB-D camera and Detic, then guides the hand along a demonstration-based reference path.Predefined paths accommodate robot-hand morphology, while grasp-type-specific reinforcement-learning policies adjust hand posture for uncertainty.Source: A. Qualitative results · p. 5
  • The robot operated in a different-texture environment and successfully reused the juice-can shelf-moving scenario with multiple identical cans.The authors interpret the texture result as suggesting robustness to environmental shifts.Source: A. Qualitative results · p. 5

B. Quantitative evaluation

The quantitative evaluation uses manually annotated cooking videos to compare the symbolic planner’s task sequences with human annotations, without comparison to existing applications.

B. Quantitative evaluation

  • The authors manually annotated a small subset because existing datasets use coarse action categories or motion labels that may combine adjacent manipulations.They report no existing dataset labeling tasks at the granularity of robot actions.Source: B. Quantitative evaluation · p. 5
  • The evaluation compares symbolic task-planner outputs with annotated grasp–manipulation–release sequences from 58 videos, focusing only on right-hand manipulation.The videos come from a first-person cooking dataset, and the annotations target task granularity based on changes in object constraints.Source: B. Quantitative evaluation · p. 5
  • The study reports no comparison with existing applications because, to the authors’ knowledge, no methods output robot-task-level action sequences without model training.Source: B. Quantitative evaluation · p. 5

1) Performance of the video analyzer:

The video analyzer correctly transcribed the manipulated-object name and action in only a limited portion of videos, with failures grouped into three patterns.

  • 20.7% of videos were correctly transcribed for manipulated-object name and action.Source: Figure 10 · p. 6
  • Failure cases fell into illusory-object, illusory-motion, and visually difficult patterns.Illusory-object cases selected an incorrect object; illusory-motion cases named the correct object but described incorrect tasks.Source: 1) Performance of the video analyzer · p. 6
  • Visually difficult errors were attributed presumably to limited spatial or temporal resolution or perspective differences, motivating human supervision.Source: 1) Performance of the video analyzer · p. 6

2) Performance of the symbolic task planner:

The symbolic planner was evaluated against task labels, while real-robot experiments reached 85–95% end-to-end performance. The discussion identifies limits in video grounding, task conditions, and prompt design.

2) Performance of the symbolic task planner:

3) End-to-end robot operation:

  • Robot tests moved a juice can between shelves and opened a drawer, using 20 demonstrations per operation; instruction misinterpretations caused 2 out of 20 failures.Execution was tested only on valid task plans, and detection failures were linked to inaccurate hand detection and image classification affecting grasp/release clip identification.Source: 3) End-to-end robot operation · p. 6
  • End-to-end robot performance reached 85-95%, comparable to model-based approaches such as [2].Source: Table III · p. 6, §V · p. 7

V. LIMITATION AND DISCUSSION

  • Five regularly spaced frames are used instead of every frame, and clutter, self-motion, and image shifts or blurring challenge GPT-4V recognition.The authors identify human supervision and corrections or prompt engineering as responses to limited video-understanding performance.Source: §V · p. 7
  • Video grounding at grasping and releasing moments alone limits the affordance information extracted from demonstrations.Further extraction requires precisely localizing each task’s action in the video.Source: §V · p. 7
  • Pre- and post-task conditions primarily rely on object relationships, although completion criteria can extend beyond those relationships, such as dirt removal in cleaning.Source: §V · p. 7
  • Task-planning success is heavily influenced by prompt design, making practical prompt-engineering expertise important.Source: §V · p. 7

VI. CONCLUSION

The paper presents a multimodal pipeline using GPT-4V to convert human actions in videos into robot-executable programs. Its reliance on off-the-shelf models supports application across varied robotic hardware and scenarios.

  • The GPT-4V-based multimodal pipeline converts human actions in videos into robot-executable programs.Source: §VI · p. 7
  • Using off-the-shelf models offers flexibility across a wide range of robotic hardware and scenarios.Source: §VI · p. 7
  • The authors hope the study serves as a practical resource for robotics researchers and inspires further developments.Source: §VI · p. 7
Loading 2311.12015v4…