PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning

Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Zhu, Hangxin Liu

arXiv:2403.11552v3cs.ROcs.AI

TL;DR

Conventional TAMP depends on manually designed, domain-specific interfaces that limit generalization to emerging tasks. LLM3 replaces this interface with a pre-trained LLM that proposes actions and parameters, reasons over categorized motion failures, and iteratively refines plans. Experiments report effective box-packing solutions, improved efficiency over random parameter sampling, and successful real-robot manipulation, while multiple feedback iterations remain necessary.

  • Problem

    Conventional TAMP relies on manually designed, domain-specific modules to interface symbolic task planning with continuous motion planning, limiting generalizability to novel environments.

    Source: §I · p. 1

  • Method

    LLM3 uses a pre-trained LLM to propose symbolic actions, generate continuous action parameters, and refine plans using motion-planning feedback categorized by failure type.

    Source: §I · pp. 1–2, C. Synthesizing Motion Planning Feedback · p. 5

  • Results

    LLM3 effectively solves box-packing TAMP problems, improves planning efficiency over random parameter sampling, and succeeds in a real-robot manipulation experiment.

    Source: C. Action Parameter Selection · p. 6, §V · p. 6

  • Takeaways & Limitations

    A pre-trained LLM can serve as a domain-independent task planner, informed parameter sampler, and motion-failure reasoner for TAMP.

    Source: §I · p. 2

Abstract

from arXiv · show

Conventional Task and Motion Planning (TAMP) approaches rely on manually crafted interfaces connecting symbolic task planning with continuous motion generation. These domain-specific and labor-intensive modules are limited in addressing emerging tasks in real-world settings. Here, we present LLM^3, a novel Large Language Model (LLM)-based TAMP framework featuring a domain-independent interface. Specifically, we leverage the powerful reasoning and planning capabilities of pre-trained LLMs to propose symbolic action sequences and select continuous action parameters for motion planning. Crucially, LLM^3 incorporates motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failure. Consequently, LLM^3 interfaces between task planning and motion planning, alleviating the intricate design process of handling domain-specific messages between them. Through a series of simulations in a box-packing domain, we quantitatively demonstrate the effectiveness of LLM^3 in solving TAMP problems and the efficiency in selecting action parameters. Ablation studies underscore the significant contribution of motion failure reasoning to the success of LLM^3. Furthermore, we conduct qualitative experiments on a physical manipulator, demonstrating the practical applicability of our approach in real-world settings.

I. INTRODUCTION

LLM3 addresses the domain-specific, labor-intensive interface between symbolic task planning and continuous motion planning by using a pre-trained LLM to propose and refine both plans and parameters through motion-failure reasoning.

Motivation

  • Traditional TAMP interfaces require manually designed, domain-specific modules to connect symbolic actions with continuous motion constraints, limiting generalizability to novel environments.These modules sample continuous action parameters and incorporate motion failures into task planning, requiring substantial expert effort.Source: §I · p. 1

LLM3 Framework

  • LLM3 uses a pre-trained LLM as a domain-independent task planner, informed action-parameter sampler, and motion-failure reasoner.The framework proposes symbolic action sequences, generates continuous parameters, and iteratively refines them using motion-planning feedback.Source: §I · pp. 1–2

Motion Failure Reasoning

  • Motion-failure reasoning categorizes feedback into collision and unreachability, enabling targeted refinement of actions and parameters.The authors report that this reduces the number of planning iterations and motion-planning queries needed to find feasible solutions.Source: §I · p. 2

Evaluation

  • In box-packing simulations, motion-feedback reasoning improves success rates and planning efficiency, while the LLM parameter sampler is more sample-efficient than random sampling.Real-robot experiments further demonstrate applicability to physical manipulation problems.Source: §I · p. 2

A. Related Work

Prior TAMP research addresses the difficult interface between symbolic task planning and continuous motion generation through engineered domains, heuristics, and communication modules. LLM3 instead uses a pre-trained LLM as both task planner and interface, while related LLM-based robotics systems apply language models to planning, feedback, or selected TAMP components.

A. Related Work

  • Manually defining planning domains and interface modules that capture real-world complexity is impractical, while domain-specific heuristics lack generalizability across domains.LLM3 uses a pre-trained LLM as both task planner and interface to provide domain-independent heuristics.Source: A. Related Work · p. 2
  • Earlier LLM robotics systems use language models for task planning, textual feedback, embodied replanning, or translating specifications, whereas LLM3 makes the LLM the core TAMP component.LLM-GROP supplies goals and placements to a classical planner, and AutoTAMP translates natural-language tasks into a formal processable representation.Source: A. Related Work · p. 2

A. Task and Motion Planning

  • TAMP represents actions with symbolic object variables and continuous parameters, whose feasible instantiations are validated by collision-free motion trajectories and state transitions.The objective is a feasible action sequence that reaches the task goal under the transition function.Source: A. Task and Motion Planning · p. 2
  • Conventional TAMP alternates symbolic action-sequence search with continuous parameter sampling, but feasibility checks require costly motion-planning calls.Researchers use symbolic domains and learned heuristic samplers to prune infeasible sequences or improve parameter selection.Source: A. Task and Motion Planning · pp. 2–3
  • LLM3 proposes symbolic actions and continuous parameters from the initial state, goal, and motion-feedback trace, then iterates validation until a feasible plan is found.This sequence-prediction formulation combines LLM generation with a motion planner, transition function, and goal test.Source: Figure 2 · p. 3, A. Task and Motion Planning · p. 3

B. Planning as Sequence Prediction

The paper formulates planning as sequence prediction, using a pre-trained language model to generate symbolic actions and continuous parameters from textualized state, goals, and context. Proposed sequences are validated by a motion planner, with feedback incorporated iteratively until a feasible plan is found.

III. METHOD

LLM3 is a TAMP framework that uses a pre-trained LLM to reason about motion failures and iteratively refine symbolic actions and continuous action parameters.

  • LLM3 alternates LLM-based motion-failure reasoning with generation of refined symbolic actions and continuous action parameters.Source: §III · p. 3

A. The LLM3 Framework

LLM3 repeatedly generates action sequences from the current state and accumulated motion-feedback trace, verifies them with a motion planner, and updates the trace after failures. This search-then-sample process is intended to improve plan quality and parameter-selection efficiency over unguided random sampling.

A. The LLM3 Framework

  • LLM3 iterates between LLM generation informed by prior motion failures and motion-planner verification of each action sequence.Source: A. The LLM3 Framework · p. 3
  • The planner maintains a bounded feedback trace, appending aggregated action or task feedback after failed rollouts for use in the next planning iteration.Source: A. The LLM3 Framework · pp. 3–4
  • The algorithm initializes the state and trace, repeatedly calls the LLM up to a maximum number of attempts, rolls out its plan, and returns feasible actions when the goal is reached.Source: A. The LLM3 Framework · p. 4
  • LLM3 is a search-then-sample TAMP planner that incrementally improves action-sequence quality using LLM heuristics and previous motion failures.Source: A. The LLM3 Framework · p. 4

B. Reasoning and Planning with pre-trained LLM

The framework uses zero-shot Chain-of-Thought prompting to generate motion-failure reasoning and action sequences from task context, state, and feedback. It supports backtrack and from-scratch replanning strategies, with distinct prompt contents for each variant.

B. Reasoning and Planning with pre-trained LLM

  • Zero-shot Chain-of-Thought prompts provide task context, goals, primitive actions, initial state, motion-feedback trace, and an output format for autoregressive planning.Source: B. Reasoning and Planning with… · p. 4
  • Only the motion-feedback trace changes across planning iterations for the same TAMP problem, while the remaining prompt content stays mostly unchanged.Source: B. Reasoning and Planning with… · p. 4
  • The backtrack strategy resumes from a previously feasible action, whereas the from-scratch strategy generates a new sequence intended to avoid the prior motion failure.Source: B. Reasoning and Planning with… · p. 4
  • Figure 3 distinguishes variant-specific prompt contents using orange for backtrack and blue for from-scratch.Source: Figure 3 · p. 4

C. Synthesizing Motion Planning Feedback

LLM3 converts binary motion-planning outcomes into categorized feedback so the LLM can reason about failures and improve subsequent plans.

IV. SIMULATION AND EXPERIMENT

The evaluation combines simulation-based ablations and parameter-sampling comparisons with experiments on a perception-integrated physical manipulator.

  • The study evaluates %SR, #LM, and #MP in simulation, compares random and LLM-based parameter sampling, and tests LLM3 on a physical manipulator.Source: §IV · p. 5

A. Simulation Setup

The experiments use two simulated box-packing settings and ablations to test motion-failure reasoning, planning strategies, and action-parameter sampling efficiency.

A. Simulation Setup

B. Ablation Study

  • The ablation compares LLM3 Backtrack, Backtrack, LLM3 Scratch, and Scratch using success rate, LLM calls, and motion-planner calls.Source: B. Ablation Study · p. 5
  • Integrating motion-planning feedback decreases #LM and #MP while increasing %SR for both backtracking and scratch strategies.Source: B. Ablation Study · p. 5, Table I · p. 6
  • Backtracking is not clearly superior to replanning from scratch; backtracking adjusts failed actions, whereas scratch resamples all actions and can use slightly fewer #LM.Source: B. Ablation Study · pp. 5–6

C. Action Parameter Selection

The action-parameter study compares random sampling with LLM sampling and LLM sampling augmented by motion-planning feedback.

D. Experiment Setup

LLM3 was evaluated on a real-world box-packing task with a Franka Research 3 manipulator under perception and execution uncertainties. The robot successfully identified and manipulated objects, while simulations and experiments supported effective plan generation and refinement.

D. Experiment Setup

  • The robot successfully identified and manipulated objects despite uncertainties and clutter while placing all blocks on a plate.The setup used a Franka Research 3, a third-person-view RGB-D camera, and object point clouds produced through segmentation and projection.Source: D. Experiment Setup · p. 6

V. CONCLUSIONS

  • LLM3 uses a pre-trained LLM to propose action sequences, generate continuous motion parameters, and refine plans after motion-planning failures.The framework does not require a prior planning domain for proposing action sequences.Source: §V · p. 6
  • Across simulations and experiments, LLM3 effectively produced and refined task and motion plans for box-packing problems.The authors describe this as promising for previously unspecified tasks.Source: §V · p. 6
  • Although LLM3 generated parameters more efficiently than random samplers, it still required multiple feedback iterations and motion-planner calls.The authors identify in-context learning or fine-tuning as possible ways to improve efficiency.Source: §V · p. 6

D. Experiment Setup

  • Actions 1 to 8 were successfully proposed by LLM3 and executed by the physical manipulator in the box-packing experiment.The experiment qualitatively evaluated placing all blocks on a plate.Source: D. Experiment Setup · p. 6, Figure 6 · p. 7
Loading 2403.11552v3…