PaperlayerView exact revision on arXiv

Source-linked AI summary

Task and Motion Planning with Large Language Models for Object Rearrangement

Yan Ding, Xiaohan Zhang, Chris Paxton, Shiqi Zhang

arXiv:2303.06247v3cs.RO

TL;DR

Underspecified rearrangement requests require commonsense knowledge about semantically valid object placements, which is difficult to transfer to robots. LLM-GROP prompts an LLM for object relationships and grounds them with task-motion planning; it receives the highest user ratings and shortest execution time among the compared approaches, with a real-robot demonstration.

  • Problem

    Underspecified rearrangement requests require commonsense object placements, while semantic-learning methods require collected training data that limits their applicability to complex service tasks.

    Source: §I · p. 1

  • Method

    LLM-GROP prompts an LLM for symbolic spatial relationships, grounds them in geometric possibilities, and optimizes task-motion plans for feasibility and efficiency.

    Source: §I · p. 1, A. Generating Symbolic Spatial… · p. 3

  • Results

    LLM-GROP achieves the highest user rating and shortest execution time compared with the three baseline approaches, and is demonstrated on a real robot.

    Source: §IV · pp. 5–6

  • Takeaways & Limitations

    LLM-derived semantic information can support semantically valid object placements within multi-step task and motion planning for natural-language commands in complex environments.

    Source: §V · p. 6

Abstract

from arXiv · show

Multi-object rearrangement is a crucial skill for service robots, and commonsense reasoning is frequently needed in this process. However, achieving commonsense arrangements requires knowledge about objects, which is hard to transfer to robots. Large language models (LLMs) are one potential source of this knowledge, but they do not naively capture information about plausible physical arrangements of the world. We propose LLM-GROP, which uses prompting to extract commonsense knowledge about semantically valid object configurations from an LLM and instantiates them with a task and motion planner in order to generalize to varying scene geometry. LLM-GROP allows us to go from natural-language commands to human-aligned object rearrangement in varied environments. Based on human evaluations, our approach achieves the highest rating while outperforming competitive baselines in terms of success rate while maintaining comparable cumulative action costs. Finally, we demonstrate a practical implementation of LLM-GROP on a mobile manipulator in real-world scenarios. Supplementary materials are available at: https://sites.google.com/view/llm-grop

I. INTRODUCTION

Service-robot rearrangement combines navigation and manipulation with commonsense decisions about where objects belong. LLM-GROP extracts symbolic spatial relationships from an LLM, grounds them in scene geometry, and optimizes task-motion plans; evaluations report higher user satisfaction with similar or lower action costs.

I. INTRODUCTION

  • Underspecified requests require commonsense placements, such as putting a fork left of a knife, alongside indoor navigation around obstacles whose locations are unknown in advance.The task combines manipulation and navigation capabilities.Source: §I · p. 1
  • The illustrated task requires arranging tableware from other tables while accounting for randomly generated obstacles absent from the pre-built map.The robot must plan using both navigation and manipulation behaviors.Source: Figure 1 · p. 1
  • LLM-GROP extracts symbolic object relationships from an LLM, grounds them in geometric possibilities, and optimizes task-motion plans for feasibility and long-term utility.The optimization seeks a trade-off between motion feasibility and task completion.Source: §I · p. 1
  • In dining-room evaluations, users rated LLM-GROP more highly than existing rearrangement methods while cumulative action costs were similar or lower; the approach was also demonstrated on a real robot.The robot computes a commonsense tabletop configuration and a task-motion plan from user instructions.Source: §I · p. 2

II. RELATED WORK

The related-work section surveys object rearrangement, tabletop arrangement methods, and LLM-based planning.

II. RELATED WORK

  • The section first introduces the object rearrangement domain.Source: §II · p. 2
  • It next discusses tabletop object arrangement methods that mostly rely on supervised learning.Source: §II · p. 2
  • It concludes by summarizing research on using large language models for planning.Source: §II · p. 2

A. Object Rearrangement

Prior rearrangement work commonly assumes the desired object arrangement is specified, while LLM-GROP accepts underspecified human instructions and makes motion-level placement decisions.

A. Object Rearrangement

  • Many object-rearrangement methods assume the exact desired object positions are provided as the goal arrangement.Source: A. Object Rearrangement · p. 2
  • ALFRED-style language-based rearrangement operates at a coarse, discrete level rather than making granular motion-level placement decisions.Source: A. Object Rearrangement · p. 2
  • LLM-GROP accepts underspecified requests, such as setting a table with provided tableware, and handles both high-level and motion-level placement decisions.Source: A. Object Rearrangement · p. 2

B. Predicting Complex Object Arrangements

LLM-GROP uses an LLM to acquire commonsense for object arrangements without demonstration data, while optimizing placement-plan feasibility and efficiency.

B. Predicting Complex Object Arrangements

  • Prior work predicts semantically meaningful tabletop structures from vague natural-language instructions, including StructFormer, a transformer-based arrangement model.Source: B. Predicting Complex Object… · p. 2
  • LLM-GROP uses an LLM for commonsense acquisition to avoid demonstration data when computing object positions, and optimizes plan feasibility and efficiency.Source: B. Predicting Complex Object… · p. 2
  • Like DALL-E-Bot, LLM-GROP achieves zero-shot performance with pretrained models, but is not restricted to a single top-down table view and accounts for navigation and manipulation uncertainty in planning.Source: B. Predicting Complex Object… · p. 2

C. Robot Planning with Large Language Models

LLM-GROP is positioned among LLM-based robot-planning methods, with its distinction framed as jointly optimizing feasibility and efficiency while computing semantically valid geometric configurations.

C. Robot Planning with Large Language Models

  • LLM-GROP optimizes both feasibility and efficiency while computing semantically valid geometric configurations, unlike the compared robot-planning methods.The cited passage also describes LLM task-planning approaches, including SayCan, which uses affordance functions to account for action feasibility.Source: C. Robot Planning with Large Language… · p. 2
  • Huang et al. use iteratively augmented prompts for household-domain task planning.SayCan enables planning from natural-language service requests and uses affordance functions to account for action feasibility.Source: C. Robot Planning with Large Language… · p. 2
  • The passage names BERT, GPT-3, ChatGPT, CodeX, and OPT as LLMs that can encode common-sense knowledge.It also notes LLM applications to robot task planning.Source: C. Robot Planning with Large Language… · p. 2

III. THE LLM-GROP APPROACH

LLM-GROP combines LLM-generated symbolic and geometric spatial relationships with task and motion planning. It targets semantically valid table arrangements amid dynamic obstacles and uncertain navigation and manipulation.

III. THE LLM-GROP APPROACH

  • The task begins with scattered tableware and seeks a semantically valid tabletop configuration aligned with common sense.The robot knows table shapes and locations and has loading and unloading skills; chairs are examples of dynamic obstacles sensed during planning.Source: §III · p. 2
  • Navigation and manipulation are uncertain: navigation can fail when goals are too close to tables or chairs, and manipulation when the robot is not close enough to a target.Source: §III · p. 2
  • LLM-GROP uses an LLM to generate symbolic and geometric object relationships, then TAMP to compute an optimal task-motion plan.The approach is presented for mobile-manipulator object rearrangement.Source: §III · p. 2

A. Generating Symbolic Spatial Relationships

The approach prompts an LLM to propose symbolic object relationships, then checks generated relationships for logical consistency. Human-authored rules identify conflicts that could make an arrangement infeasible.

A. Generating Symbolic Spatial Relationships

  • The LLM proposes symbolic spatial relationships among table objects using a template-based prompt that asks for a typical positioning.The prompt can specify relationships, examples, objects, and output constraints.Source: A. Generating Symbolic Spatial… · pp. 2–3
  • Few-shot prompts include examples and can help enforce a predefined response format, while requiring more prompt-engineering effort.Without examples, the prompt is simplified to zero-shot prompting.Source: A. Generating Symbolic Spatial… · p. 3
  • Answer set programming evaluates whether generated candidates are consistent with explicit symbolic constraints, addressing contradictory LLM outputs.ASP expresses problems as logical rules and constraints and supports recursive reasoning.Source: A. Generating Symbolic Spatial… · p. 3
  • Human experts define spatial relationships, their transitions, and conflict-detection rules to ensure generated context is logical and feasible.For example, an object cannot be both below and to the right of another object.Source: A. Generating Symbolic Spatial… · p. 3
  • A sample LLM arrangement contains conflicting relations about the fruit bowl and dinner fork, leaving no feasible solutions.The stated inconsistency is between the implications of Steps 2 and 3 and the fruit bowl's placement in Step 5.Source: A. Generating Symbolic Spatial… · p. 3

B. Generating Geometric Spatial Relationships

The method converts symbolic relations into candidate geometric positions using LLM-recommended distances, then samples and filters configurations against object and table constraints.

B. Generating Geometric Spatial Relationships

  • A second LLM template asks how many centimeters apart objects should be for a specified spatial relationship.The example response recommends placing a dinner knife about 5–7 centimeters to the right of a dinner plate.Source: B. Generating Geometric Spatial… · p. 3
  • Object coordinates are computed relative to a centrally located coordinate origin by adding or subtracting recommended distances along horizontal and vertical directions.The LLM-guided position for object i is denoted (xi, yi).Source: B. Generating Geometric Spatial… · p. 3
  • Because LLM responses omit object attributes such as shape, size, and table constraints, adaptive sampling incorporates those attributes after recommended positions are obtained.The method sequences 2D Gaussian samples with (xi, yi) as the mean vector.Source: B. Generating Geometric Spatial… · p. 3
  • The illustrated four-object plan uses GPT-3 commonsense to produce navigation trajectories intended to minimize overall navigation cost and select an available unloading side.The example task includes a bread plate, fork, knife, and bread in a complex environment.Source: Figure 3 · p. 4
  • Samples are rejected if they violate symbolic relationships, overlap objects, or place an object's bounding box outside the table boundary.Multiple rounds of sampling produce M object-configuration sequences for selection.Source: B. Generating Geometric Spatial… · p. 4

C. Computing Task-Motion Plans

The planner sequences object placements and chooses how the robot approaches the table, then converts that task plan into navigation goals and motion trajectories.

C. Computing Task-Motion Plans

  • The planner sequences object placements and selects how to approach the table, then computes navigation goals that connect task-level and motion-level planning.Placement order can depend on object relations, such as placing a plate before bread on top of it.Source: C. Computing Task-Motion Plans · p. 4
  • After selecting a task plan, the robot plans motion trajectories for navigation and manipulation behaviors.Fig. 3 illustrates a generated plan for a four-object rearrangement task.Source: C. Computing Task-Motion Plans · p. 4
  • For each object configuration, GROP evaluates navigation goals by plan feasibility and efficiency, and the system selects the best plan across configuration groups.The choice accounts for dynamic obstacles and may favor standing close to an object rather than extending the robot’s reach.Source: C. Computing Task-Motion Plans · p. 4

IV. EXPERIMENTS

In tableware rearrangement experiments, LLM-GROP receives the highest user ratings and shortest execution time among the compared methods, and it is demonstrated on a real mobile robot.

Experimental Setup

  • The simulated evaluation covers eight tableware tasks with varying object counts, randomly placed chairs, and 20 executions per task using LLM-GROP.The study uses GPT-3 engines and evaluates arrangements using images captured after task completion.Source: §IV · pp. 4–5, Table I · p. 5

Rating Criteria

  • Five graduate-student volunteers rated 640 images from four methods across eight tasks, producing 3,200 image ratings using a five-point rubric.The rubric assigns 1 point to the poorest arrangements and 5 points to the best.Source: §IV · p. 5, Table III · p. 5

LLM-GROP vs. Baselines

  • The paper presents example LLM-GROP arrangements for eight tableware tasks as semantically valid configurations.The objects used in those tasks are listed in Table I.Source: Figure 5 · p. 5, Table I · p. 5
  • LLM-GROP achieves the highest user rating and shortest execution time among the three baselines across tableware arrangement tasks.GROP is described as similarly efficient but with a significantly lower rating; TPRA and LATP have lower ratings and poor efficiency.Source: §IV · p. 5, Figure 4 · p. 5
  • LLM-GROP outperforms the baselines on user ratings for each of the eight tasks, while tasks involving more objects pose additional challenges.Tasks 1–5 involve three objects, tasks 6–7 four, and task 8 five; tasks 1–5 receive slightly higher scores than tasks 6 and 8.Source: Table I · p. 5, §IV · p. 6, Figure 6 · p. 6

Real Robot Demonstration

  • On a real mobile manipulator, LLM-GROP rearranges a plate, fork, knife, cup, and strawberry into semantically valid placements.The fork ends up left of the plate and the strawberry on the plate.Source: §IV · p. 6

V. CONCLUSION AND FUTURE WORK

The conclusion presents LLM-GROP as a way to use LLM-derived semantic information for commonsense object placement within task-and-motion planning. It identifies unknown objects and scenes, and a wider range of placement problems, as future directions.

V. CONCLUSION AND FUTURE WORK

  • LLM-GROP uses semantic information extracted from LLMs to make commonsense object-placement decisions within task-and-motion planning for natural-language commands in complex environments.Future work may extend grasping and manipulation to fully unknown objects in unknown scenes and broaden the placement problems addressed.Source: §V · p. 6

Related papers

Loading 2303.06247v3…