PaperlayerView exact revision on arXiv

Source-linked AI summary

Task and Motion Planning with Large Language Models for Object Rearrangement

Yan Ding, Xiaohan Zhang, Chris Paxton, Shiqi Zhang

arXiv:2303.06247v4cs.RO

TL;DR

Service robots need commonsense knowledge to arrange multiple objects in semantically valid configurations, but transferring that knowledge to robots is difficult. LLM-GROP prompts an LLM for symbolic spatial relationships, grounds them in geometry, and integrates them with task and motion planning. It achieves the highest user rating and shortest execution time among the compared approaches, while maintaining similar or lower cumulative action costs, and is demonstrated on a real mobile manipulator.

  • Problem

    Service-robot rearrangement requires commonsense object knowledge for semantically valid configurations, but such knowledge is difficult to transfer to robots.

  • Method

    LLM-GROP prompts an LLM for symbolic spatial relationships, grounds them in geometric relationships, checks logical consistency, and optimizes task-motion plans for feasibility and efficiency.

  • Results

    LLM-GROP achieves the highest user rating and shortest execution time among compared approaches, while maintaining similar or lower cumulative action costs and succeeding in a real-robot demonstration.

  • Takeaways & Limitations

    LLM-GROP connects natural-language commands to semantically valid object rearrangements while operating across navigation and manipulation in complex environments.

Abstract

from arXiv · show

Multi-object rearrangement is a crucial skill for service robots, and commonsense reasoning is frequently needed in this process. However, achieving commonsense arrangements requires knowledge about objects, which is hard to transfer to robots. Large language models (LLMs) are one potential source of this knowledge, but they do not naively capture information about plausible physical arrangements of the world. We propose LLM-GROP, which uses prompting to extract commonsense knowledge about semantically valid object configurations from an LLM and instantiates them with a task and motion planner in order to generalize to varying scene geometry. LLM-GROP allows us to go from natural-language commands to human-aligned object rearrangement in varied environments. Based on human evaluations, our approach achieves the highest rating while outperforming competitive baselines in terms of success rate while maintaining comparable cumulative action costs. Finally, we demonstrate a practical implementation of LLM-GROP on a mobile manipulator in real-world scenarios. Supplementary materials are available at: https://sites.google.com/view/llm-grop

I. INTRODUCTION

Service-robot rearrangement tasks require both navigation and manipulation, plus commonsense knowledge for semantically valid object placements. LLM-GROP extracts such relationships from an LLM, grounds them geometrically, and plans efficient rearrangements in environments with obstacles.

  • Motivation: Multi-object rearrangement supports everyday service-robot tasks but requires coordinated manipulation, navigation, semantic placement, and obstacle avoidance.Table setting may require retrieving objects from different locations, arranging them meaningfully, and navigating around unknown chairs or humans.
  • Problem: Existing systems often require explicit goal arrangements, whereas real-world user requests are underspecified and require commonsense decisions about object placement.Examples include determining that a fork belongs to the left of a plate and a knife to its right.
  • Problem: Training-data-based methods for equipping mobile manipulators with semantic information limit applicability to complex service tasks.
  • Approach: LLM-GROP prompts an LLM for symbolic object relationships, grounds them in geometric relationships, and optimizes task-motion plans for feasibility and efficiency.The planner seeks a trade-off between motion feasibility and task-completion efficiency.
  • Evaluation: In a dining-room evaluation, the robot computed commonsense tabletop configurations and task-motion plans, improving user satisfaction while maintaining similar or lower cumulative action costs.The approach was also demonstrated on a real robot.

II. RELATED WORK

Prior object-rearrangement research commonly assumes that exact goal arrangements are provided, while language-based systems operate too coarsely for granular motion-level placement decisions.

  • Object Rearrangement: Object-rearrangement research includes benchmarks and systems for moving objects and placing them in new positions.

A. Object Rearrangement

Object rearrangement requires achieving functional, semantically valid tabletop configurations, and LLM-GROP addresses underspecified human instructions while making both high-level and motion-level decisions.

  • Object Rearrangement: Prior rearrangement methods commonly assume that exact desired object positions are given as input.ALFRED extends rearrangement to language-based multi-step tasks but remains coarse and discrete for placement decisions.
  • Object Rearrangement: LLM-GROP accepts underspecified instructions and extracts commonsense knowledge to make high-level and motion-level placement decisions.

B. Predicting Complex Object Arrangements

Complex object arrangement targets functional, semantically meaningful tabletop configurations from vague instructions. LLM-GROP uses LLM-based commonsense acquisition without demonstration data and optimizes placement-plan feasibility and efficiency.

  • Predicting Complex Object Arrangements: Object arrangement places tabletop items into functional, semantically valid goal configurations, including commonsense relations such as forks left of knives.
  • Predicting Complex Object Arrangements: LLM-GROP uses an LLM for commonsense acquisition to avoid demonstration data when computing object positions.It also optimizes the feasibility and efficiency of tableware-placement plans.
  • Predicting Complex Object Arrangements: Unlike a single top-down-view approach, LLM-GROP achieves zero-shot performance with pretrained models and considers uncertainty in manipulation and navigation.

C. Robot Planning with Large Language Models

Large language models encode common sense and have been applied to robot task planning, including household domains and affordance-aware planning.

  • LLMs such as GPT-3 and ChatGPT can encode large amounts of common sense.
  • Prior work applied LLMs to household task planning by iteratively augmenting prompts.
  • SayCan incorporated affordance functions to account for action feasibility in robot planning.

III. THE LLM-GROP APPROACH

LLM-GROP addresses tabletop rearrangement by combining LLM-generated spatial relationships with task and motion planning under the task’s environmental constraints.

  • III. THE LLM-GROP APPROACH: The task rearranges scattered tableware into a semantically valid tabletop configuration aligned with common sense.The robot has prior knowledge of table shapes and locations and skills for loading and unloading tableware.
  • III. THE LLM-GROP APPROACH: Dynamic obstacles such as chairs are sensed at planning time, while navigation uncertainty is considered.
  • III. THE LLM-GROP APPROACH: LLM-GROP uses an LLM for symbolic and geometric spatial relationships and TAMP for computing an optimal task-motion plan.

A. Generating Symbolic Spatial Relationships

LLM-GROP prompts an LLM to generate symbolic object relationships, then uses logical constraints to identify and correct contradictory outputs.

  • A. Generating Symbolic Spatial Relationships: LLMs extract common-sense symbolic spatial relationships among objects placed on a table.A template-based prompt asks for typical object positioning and can use zero-shot or few-shot examples.
  • A. Generating Symbolic Spatial Relationships: Prompt notes constrain outputs, including requiring each action to start with “Place” and excluding other objects.
  • A. Generating Symbolic Spatial Relationships: Answer set programming evaluates generated candidates against explicit symbolic constraints to prevent logical errors.If an inconsistency is detected, the template is repeatedly fed to the LLM to elicit a logically consistent output.
  • A. Generating Symbolic Spatial Relationships: Human-defined rules detect incompatible relationships, such as an object being both below and to the right of another.
  • A. Generating Symbolic Spatial Relationships: An example identifies contradictions when generated placement steps imply conflicting positions for the fruit bowl.

B. Generating Geometric Spatial Relationships

LLM-GROP converts symbolic relationships into geometric positions using prompted distances, adaptive sampling, and feasibility checks for execution.

  • B. Generating Geometric Spatial Relationships: A second LLM template asks how many centimeters one object should be placed from another.An example produces a recommended distance of about 5–7 centimeters between dinnerware objects.
  • B. Generating Geometric Spatial Relationships: Coordinates are computed from a central reference object by adding or subtracting recommended horizontal and vertical distances.The position of object i is represented as (x_i, y_i).
  • B. Generating Geometric Spatial Relationships: Adaptive 2D Gaussian sampling incorporates object attributes such as shape, size, and table constraints after obtaining recommended positions.
  • B. Generating Geometric Spatial Relationships: Navigation trajectories illustrate an efficient and feasible four-object pick-and-place plan in a complex environment.Efficiency minimizes overall navigation cost, while feasibility includes selecting an available table side for unloading.
  • B. Generating Geometric Spatial Relationships: Samples are rejected when they violate symbolic relationships, cause overlap, or place objects outside table boundaries.The accepted candidate is selected for execution as part of the generated task-motion plan.

C. Computing Task-Motion Plans

LLM-GROP selects task-motion plans by evaluating feasible object configurations and navigation goals, then executes the highest-utility option. The planner accounts for placement order and approach direction in complex environments.

  • Task-level planning: The robot determines object placement order and how to approach the table for each configuration sequence.Objects supporting other objects must be placed first; approach direction is also part of the task-level decision.
  • Navigation-goal selection: Navigation goals are evaluated by feasibility and efficiency when dynamic obstacles make some positions preferable to others.The planner may favor standing close to an object for placement rather than extending reach from farther away.
  • Plan selection: GROP computes maximal task-motion-plan utility for different groups of object configurations, and LLM-GROP selects the best group for execution.This combines configuration selection with task-motion planning before execution.
  • Plan generation: The resulting system generates complete task-motion plans for multi-object rearrangement tasks, including four-object scenarios.Figure 3 is cited as an example of a plan generated by LLM-GROP.

IV. EXPERIMENTS

The experiments evaluate LLM-GROP for tableware rearrangement in simulation and on a real mobile robot, comparing it with three baselines using human ratings, execution time, and qualitative outcomes. LLM-GROP receives the highest ratings and shortest execution time in the reported comparison, while producing semantically valid placements in both simulated and real settings.

  • Experimental setup: The simulated evaluation requires semantically valid arrangements, efficient rearrangement plans, and navigation and manipulation behaviors.A mobile manipulator retrieves objects from varied locations and places them on a central table around which a chair is randomly positioned.
  • Baselines: The study compares LLM-GROP with TPRA, LATP, and GROP, which differ in arrangement generation, navigation, and feasibility capabilities.TPRA and GROP randomly place objects in collision-free positions, while LATP predicts arrangements with an LLM and samples standing positions.
  • Experimental setup: Eight simulated tasks cover three, four, or five objects, and each task is executed 20 times using identical prompt templates.The evaluation uses OpenAI GPT-3 engines, captures post-task images, and later presents those images for human assessment.
  • Human evaluation: Five graduate-student volunteers rate 640 images from four methods across eight tasks using a five-point tableware-arrangement scale.The rating scale ranges from 1 point for the poorest arrangement to 5 points for the best arrangement.
  • Overall comparison: LLM-GROP achieves the highest user rating and shortest execution time, while GROP is similarly efficient but receives a significantly lower rating.TPRA and LATP receive lower ratings and show poor efficiency because they lack navigation capabilities for complex environments.
  • Real-robot demonstration: On real robot hardware, LLM-GROP successfully rearranges a plate, fork, knife, cup, and strawberry into semantically valid placements while avoiding chairs and a human.The demonstration uses a Segway-based mobile platform and a UR5e arm with hard-coded grasping procedures.

V. CONCLUSION AND FUTURE WORK

The paper concludes that LLM-GROP can extract semantic information from LLMs and use it within task and motion planning for commonsense object placement. It demonstrates multi-step rearrangement from natural-language commands in complex environments and identifies broader manipulation as future work.

  • Conclusion: LLM-GROP extracts semantic information from LLMs to make commonsense, semantically valid object-placement decisions within a task and motion planner.The approach is intended for multi-step tasks in complex environments specified through natural-language commands.
  • Future work: Future work includes handling fully unknown objects and scenes and expanding the approach to a wider range of placement problems.The paper specifically points to incorporating information from methods such as M0M for grasping and manipulation.
Loading 2303.06247v4…