Source-linked AI summary
AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers
Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, Chuchu Fan
TL;DR
Complex TAMP requires jointly reasoning about task actions, motion feasibility, and environmental or temporal constraints, while direct natural-language interfaces remain difficult to connect to formal planners. AutoTAMP translates language into STL with few-shot LLMs and autoregressively checks and corrects syntactic and semantic errors. Across challenging constrained domains, it outperforms direct LLM-planning approaches, though planning and inference can be costly and the demonstrated STL planner is not immediately suited to manipulation.
Problem
Natural-language task descriptions are user-friendly, but direct LLM planning and subgoal factorization struggle with executable long-horizon tasks involving complex geometric and temporal constraints.
Method
AutoTAMP uses few-shot LLM translation from language to STL, then applies formal planning and autoregressive re-prompting to correct syntactic and semantic errors.
Results
AutoTAMP outperforms direct LLM-planning approaches on tasks with complex geometric and temporal constraints, while syntactic and semantic correction significantly improves success across scenarios.
Takeaways & Limitations
In-context learning with pretrained LLMs is well suited to translating language into formal task specifications for solving constrained TAMP problems.
Takeaways & Limitations
Planning time is high, especially with repeated re-prompting, and the STL planner used is not immediately applicable to manipulation tasks.
Abstract
from arXiv · showhide
For effective human-robot interaction, robots need to understand, plan, and execute complex, long-horizon tasks described by natural language. Recent advances in large language models (LLMs) have shown promise for translating natural language into robot action sequences for complex tasks. However, existing approaches either translate the natural language directly into robot trajectories or factor the inference process by decomposing language into task sub-goals and relying on a motion planner to execute each sub-goal. When complex environmental and temporal constraints are involved, inference over planning tasks must be performed jointly with motion plans using traditional task-and-motion planning (TAMP) algorithms, making factorization into subgoals untenable. Rather than using LLMs to directly plan task sub-goals, we instead perform few-shot translation from natural language task descriptions to an intermediate task representation that can then be consumed by a TAMP algorithm to jointly solve the task and motion plan. To improve translation, we automatically detect and correct both syntactic and semantic errors via autoregressive re-prompting, resulting in significant improvements in task completion. We show that our approach outperforms several methods using LLMs as planners in complex task domains. See our project website https://yongchao98.github.io/MIT-REALM-AutoTAMP/ for prompts, videos, and code.
I. INTRODUCTION
AutoTAMP addresses the gap between user-friendly natural-language task descriptions and formal TAMP representations. It translates language into STL and uses autoregressive checking to correct syntactic and semantic errors before planning.
- Classic TAMP uses expressive formal representations, but directly specifying them requires training and experience that nonexperts may lack.
- LLM task-planning approaches often cannot verify executable subtask sequences or handle temporally dependent actions, optimization, and execution constraints jointly.
- AutoTAMP translates natural-language task descriptions into Signal Temporal Logic for consumption by a formal TAMP planner rather than directly planning subgoals.
- An LLM evaluates semantic alignment between the instruction, generated STL, and planner output, while syntax verification supports correction of malformed specifications.
- Experiments in challenging 2D and multi-agent domains report better performance than direct LLM planning under hard geometric and temporal constraints.
II. PROBLEM DESCRIPTION
The problem is to convert language and environment information into a constraint-satisfying robot trajectory represented by timed waypoints.
- The target motion plan encodes a natural-language instruction, including spatial and temporal constraints, as timed waypoints such as (x_i, y_i, t_i).
- The environment state consists of named polygonal obstacles supplied as additional context for trajectory generation.
- The trajectory must satisfy the instruction, respect the robot’s maximum velocity, and remain within the task time limit.
- The full trajectory is assumed to be a linear interpolation between timed waypoints, with dense waypoint sequences representing complex trajectories.
III. METHODS
The compared approaches use language, environment state, and few-shot examples to generate robot trajectories, with AutoTAMP using an LLM as translator and checker.
- The approaches ultimately produce robot trajectories from the instruction and environment state.
- Each approach receives a text-based global environment state, few-shot in-context examples, and a natural-language instruction.
- The LLM-As-Translator & Checker approach is the paper’s contribution and uses these inputs within its prompting and re-prompting process.
A. LLM End-to-end Motion Planning
LLM End-to-end Motion Planning directly generates a trajectory from a language instruction and re-prompts after constraint violations. The approach can repeatedly violate constraints because of weak spatial and numerical reasoning.
- LLM End-to-end Motion Planning directly generates a robot trajectory for a given language instruction.
- When a trajectory violates constraints, the model receives the violation and may be re-prompted up to five times to generate another trajectory.
- The method violates constraints multiple times even with direct correction re-prompts, which the paper attributes to poor spatial and numerical reasoning.
B. LLM Task Planning
LLM task-planning methods generate navigation sub-task sequences from language and rely on a separate motion planner to produce trajectories. The evaluated variants differ in whether they check executability during sequence generation.
- LLM Task Planning: LLM Task Planning generates a sequence of navigation sub-tasks from a language instruction, then converts each sub-task to STL for an independent motion planner.This shared setup permits comparison with the proposed approach using the same STL planner.
- Naive Task Planning: Naive Task Planning generates the entire sub-task sequence without checking executability.
- SayCan: SayCan iteratively selects each next sub-task from five candidates using language-model likelihood combined with feasibility likelihood.The next action is conditioned on previously generated sub-tasks.
- LLM Task Planning + Feedback: LLM Task Planning + Feedback generates full sequences, checks feasibility, and re-prompts with infeasibility feedback when sub-tasks cannot be executed.This combines sequence-level generation with feasibility checking before execution.
C. Autoregressive LLM Specification Translation&Checking + Formal Planner
AutoTAMP translates natural-language tasks into Signal Temporal Logic and uses a formal planner to jointly produce trajectories under geometric, temporal, and dynamical constraints. Autoregressive re-prompting corrects both syntactic and semantic translation errors.
- Autoregressive LLM Specification Translation&Checking + Formal Planner: Unlike LLM Task Planning, AutoTAMP translates natural language into STL and passes the specification to an STL planner for trajectory generation.
- Signal Temporal Logic: STL provides a formal task specification with continuous real-time constraints for time-critical missions.
- Signal Temporal Logic: STL formulas combine atomic predicates, logical operators, and temporal operators including eventually, always, and until over real-time intervals.The action primitives are enter(room name) and not enter(room name).
- STL Trajectory Planner: The STL planner encodes constraints through piece-wise linear paths with timed waypoints, optimizing formula validity and time efficiency under maximum-velocity dynamics.
- Syntactic Checking & Semantic Checking: Syntax checking uses a rules-based verifier and up to five corrective re-prompts, while semantic checking uses the planner’s generated state sequence to assess alignment with the instruction.Semantic re-prompting addresses misalignment between the natural-language instruction and generated STL.
IV. EXPERIMENTAL DESIGN
The evaluation covers six 2D navigation scenarios spanning single-agent and multi-agent settings with geometric and temporal constraints. Experiments vary language models, prompt examples, environments, and randomized task instances.
- Experimental Setup: Each scenario uses 2D navigation with one or more robots, shaped regions, named properties, varying starts, and five in-context examples per method.
- Experimental Setup: The study evaluates six scenarios—three single-agent and three multi-agent—with different combinations of geometric and temporal constraints using GPT-3 and GPT-4.SayCan and LLM Task Planning + Feedback are not tested in multi-agent scenarios.
- HouseWorld1: HouseWorld1 contains 100 paraphrased instructions and 200 test cases from randomized start-end pairs, without a hard trajectory time constraint.
- HouseWorld2: HouseWorld2 matches HouseWorld1 but adds a hard time limit based on completing the correct trajectory at 0.8 maximum velocity.
- Remaining Scenarios: The remaining scenarios use 20 human-checked paraphrases, three environment instances, and five randomized start/end pairs, totaling 300 test cases.
- Chip’s Challenge: Chip’s Challenge imposes strict geometric and logical constraints requiring the robot to reach all goals while acquiring unique keys for corresponding doors.
- Overcooked: Overcooked requires cooperative ingredient gathering and timed return to CookingRoom, while limited space challenges multi-agent maneuvering.
- Rover: Rover requires multiple agents to visit observation regions before transmitting from a red region under time and energy constraints.
V. RESULTS
Across constrained task scenarios, AutoTAMP generally maintains stronger performance than direct or naive LLM planning, while translation errors remain its main failure source. Error correction improves STL translation, and demonstrations extend to simulated and physical environments.
- Without a hard time constraint, all LLM task-planning methods outperform AutoTAMP in HouseWorld1 because direct trajectories lack geometric challenges.
- Strict time constraints substantially reduce competing task-planning performance, while AutoTAMP’s success rate persists in HouseWorld2.
- In geometric-constraint tasks, LLM End-to-end Motion Planning and Naive Task Planning perform quite poorly.
- Most LLM task-planning failures arise from execution-time violations and long-horizon action sequencing, while naive planning also fails to avoid multi-agent collisions.
- Syntactic and semantic error correction significantly improve STL-translation performance across all evaluated scenarios.
- Re-prompting-based error correction is competitive with fine-tuning without requiring additional data or training, while NL2TL performs similarly to pretrained-LLM translation.
- AutoTAMP is demonstrated in 3D simulated environments and on physical differential-drive robots across multiple navigation and manipulation scenarios.
VI. RELATED WORK
Related work includes both direct LLM planning and translation into formal task representations. AutoTAMP follows the translation route while adding semantic-error correction through autoregressive re-prompting.
- Task-and-motion planning jointly addresses high-level task sequencing and low-level continuous motion planning.
- Direct LLM planners generate action sequences or subtasks, but prior work improves executability using few-shot learning, admissible-action constraints, feedback, and iterative generation.
- Language-to-representation methods map natural language into forms including lambda calculus, motion constraints, linear temporal logic, and signal temporal logic.
- AutoTAMP translates language into task specifications that represent complex temporal constraints and automatically detect and correct semantic errors.
- Re-prompting has been used with contextual feedback for actions, preconditions, code, and syntax; AutoTAMP extends it to semantic-error correction.
VII. CONCLUSION
AutoTAMP uses pretrained LLMs as translators to formal task specifications and checkers of syntactic and semantic errors, enabling formal planning for complex geometric and temporal constraints. The authors report that this outperforms direct LLM planning, while noting limitations in prompting, runtime, and planner applicability.
- AutoTAMP translates language task descriptions into formal specifications and uses corrective re-prompting to check syntactic and semantic errors.The framework uses few-shot in-context learning and introduces autoregressive re-prompting for semantic errors.
- LLM-generated task specifications solved by a formal planner outperform direct LLM planners on tasks with complex geometric and temporal constraints.
- The reported results rely on selecting the best prompt from several candidates, and better prompts may improve performance.
- Planning time is high, particularly when re-prompting requires multiple iterations, motivating faster formal planners and LLM inference.
- The STL planner is not immediately applicable to manipulation tasks, although the approach is intended to support integration with more suitable STL planners.