PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Motion: From Natural Language Instructions to Feasible Plans

Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, Jeannette Bohg

arXiv:2303.12153v5cs.ROcs.AIcs.LG

TL;DR

Long-horizon manipulation requires language-based planners to verify both symbolic goals and geometric feasibility before execution. Text2Motion combines LLMs, learned skills, and geometric feasibility planning in a hybrid shooting/search framework, achieving 82% success versus 13% for prior language-based methods. Its scope remains bounded by learned-skill, dynamics-model, and likelihood-calibration limitations.

  • Problem

    Prior language-based planners often execute myopically or open-loop without verifying symbolic correctness and geometric feasibility, which is problematic for long-horizon tasks.

  • Method

    Text2Motion combines an LLM, learned skills, geometric feasibility planning, and hybrid shooting/search strategies to construct and verify multi-step plans before execution.

  • Results

    82% success rate was achieved on challenging tabletop manipulation tasks, compared with 13% for prior language-based planning methods.

  • Takeaways & Limitations

    Geometric feasibility planning, search-based reasoning, hybrid planning, and inferred-goal termination support sequential manipulation with geometric dependencies.

  • Takeaways & Limitations

    Performance depends on the fidelity of learned skills, value functions, and predicted future states, while recency-biased likelihoods can induce cyclic skill repetition.

Abstract

from arXiv · show

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills.

1 Introduction

Long-horizon robot planning must combine symbolic interpretation with geometric feasibility, but prior LLM-based approaches often execute myopically or without verification. Text2Motion addresses this by constructing and verifying feasible plans, achieving higher reported success than prior language-based methods.

  • Traditional long-horizon planning combines symbolic and geometric reasoning, while formal logic can make user intent and preferences cumbersome to specify.
  • Prior LLM-based planners avoid manually specified symbolic domains but commonly use myopic or open-loop execution without symbolic or geometric verification.
  • Long-horizon settings challenge LLM task planning, making pre-execution verification of plan correctness and feasibility a central question.
  • 82% success rate was achieved by Text2Motion on challenging tabletop manipulation tasks, compared with 13% for prior language-based planning methods.

2 Related Work

Prior language-based planners and classical task-and-motion methods leave gaps in long-horizon geometric coordination, plan verification, and flexible language interpretation. Text2Motion combines LLM planning with learned skills and geometric feasibility planning while retaining assumptions about perceived objects and poses.

  • Language-based planning: Language-conditioned policies can address long-horizon tasks but require expensive data collection and training to generalize across diverse instructions.
  • Language-based planning: Recent LLM planners support task planning through admissible actions, PDDL, replanning, code, and fallback programs, but Text2Motion focuses on geometric dependencies across skill sequences.
  • Text2Motion: Text2Motion combines LLMs, learned skills, and geometric feasibility planning to construct task- and motion-level plans and verify inferred symbolic goals before execution.
  • Language-based planning: SayCan and Inner Monologue score and execute one skill at a time, which can miss dependencies spanning a skill sequence and prevents prior verification of desired outcomes.
  • Task and motion planning: Task-and-motion planning typically separates symbolic plan-skeleton generation from motion-planning verification under robot and environmental constraints.
  • Task and motion planning: Text2Motion assumes task-relevant objects and poses are known, while extending language-based planning to unrestricted object classes and properties remains future work in open-world settings.

3 Problem Setup

Text2Motion plans skill sequences that satisfy natural-language goals and remain executable from the initial state. Its formulation combines learned skill models with geometric feasibility planning that optimizes parameters across the sequence.

  • The problem assumes a closed-world initial state containing task-relevant objects and poses, with instruction fulfillment represented by predicates over desired object configurations.
  • Each learned skill pairs a policy with a parameterized manipulation primitive, whose parameters can encode grasps, placements, pulling distances, or pushing distances.
  • The framework uses skill policies, Q-functions, dynamics models, and natural-language state descriptions to support planning, while remaining agnostic to how these models are obtained.
  • The objective is to find a skill sequence that is both likely to satisfy the instruction symbolically and successfully executable from the initial state.
  • A single failed skill causes the entire plan to fail, so symbolic correctness must be paired with sequence-level execution success.
  • Geometric feasibility planning computes primitive parameters that maximize the success probability of the skill sequence and resolve dependencies across skills.
  • Future states are predicted with learned dynamics, and sequence success is approximated by multiplying Q-functions along a sampled trajectory.

4 Methods

Text2Motion converts language into inferred symbolic goals, then searches for skill sequences whose predicted executions are geometrically feasible and goal-reaching. Its hybrid planner uses shooting first and greedy-search as a fallback, evaluating skills in the context of preceding actions.

  • 4.1 Goal prediction: Text2Motion predicts symbolic goal propositions from an instruction and uses a satisfaction function to verify whether predicted geometric states meet any goal.The inferred goals are grounded over scene objects through predicate classifiers.
  • 4.2 Shooting-based planning: Shooting generates K complete skill sequences, evaluates their predicted success probabilities and trajectories, filters invalid candidates, and returns the highest-probability goal-reaching sequence.It queries the LLM once and raises planning failure when no valid candidate remains.
  • 4.3 Search-based planning: Greedy-search ranks candidate skills using language-based usefulness and geometric feasibility, then appends the highest-scoring skill and repeats until the predicted state satisfies the goal or search depth is exhausted.Its skill feasibility is evaluated through geometric planning in the context of the running sequence.
  • 4.3 Search-based planning: Unlike baselines that assess skills only in the current state, greedy-search evaluates each skill within the planned prefix using geometric feasibility planning.This addresses dependencies spanning multiple skills.
  • 4.4 Text2Motion: Text2Motion invokes shooting to plan candidate sequences and falls back to one greedy-search step when shooting fails, before execution begins.The hybrid design uses shooting for efficient full-plan prediction and greedy-search for cases where shooting cannot find a plan.

5 Experiments

The experiments evaluate Text2Motion and related planners on tabletop tasks spanning long horizons, lifted goals, and partial affordance perception. Success requires predicted goal satisfaction before execution, while failures distinguish planning from execution outcomes.

  • Experimental design: The evaluation tests four hypotheses concerning geometric feasibility planning, greedy-search under partial affordance perception, hybrid planning, and a priori goal prediction.These hypotheses target long-horizon manipulation with geometric dependencies and uncertain affordances.
  • Baselines and implementation: Text2Motion, shooting, and greedy-search use the same skill library, independently trained policies and Q-functions, out-of-distribution rejection, and applicable dynamics and geometric planners.This provides a shared planning and skill infrastructure across the evaluated methods.
  • Baselines and implementation: SayCan-GS and Inner Monologue-GS rank or score admissible skills myopically, with generator-scorer limiting the candidate pool to LLM-generated skills.Inner Monologue-GS additionally conditions likelihoods on visited state history and uses symbolic task-progress context.
  • Evaluation tasks: The task suite samples tabletop manipulation instances from initial-state distributions and varies task difficulty across long-horizon, lifted-goal, and partial-affordance properties.Geometric parameters are randomized during evaluation, and long-horizon tasks require skill sequences of length six or greater.
  • Evaluation and metrics: A run is successful when execution reaches a final state satisfying the task instruction, and plans execute only after geometric feasibility planning predicts satisfaction of inferred goals.Planning failure means no qualifying sequence is produced within maximum length; execution failure means the executed plan misses the ground-truth goal.
  • Evaluation and metrics: Low-level execution is closed-loop through geometric replanning of remaining skills after environment feedback, but the LLM is not queried for task-level replanning.This evaluation design compensates for potentially erroneous future-state predictions while retaining the original task-level sequence.

6 Results

Results show that geometric feasibility planning is crucial for long-horizon tasks with geometric dependencies, while Text2Motion combines shooting and search to handle partial affordance perception. Its hybrid strategy matches shooting on non-PAP tasks and improves over greedy-search on PAP tasks.

  • Feasibility planning: Geometric feasibility planning enables substantially higher task success than myopic baselines that only assess individual skills.The myopic baselines remained below 20% success despite completing 50%-80% of subgoals, because downstream feasibility depends on earlier geometric coordination.
  • Feasibility planning: Explicit language feedback alone does not improve success when it is isolated from plan feasibility.Saycan-gs closely matched innermono-gs despite Inner Monologue receiving descriptions of all states encountered during execution.
  • Search-based reasoning: Shooting slightly outperforms greedy-search on the first two LH tasks, while both achieve 100% success on the third LH + LG task.Shooting benefits from testing multiple generated skill sequences and selecting the one with the highest success probability.
  • Search-based reasoning: On PAP tasks, shooting incurs an 80% planning failure rate, whereas greedy-search finds plans over 90% of the time and solves 40%-60% of PAP tasks.Shooting achieves 10% success on Task 4 and solves neither of the latter two PAP tasks, while greedy-search uses feasibility heuristics during search.
  • Hybrid planning: Text2Motion matches shooting on non-PAP tasks but succeeds more often than greedy-search on PAP tasks by interleaving both planning strategies.Text2Motion also incurs fewer planning and execution failures than greedy-search, and over 90% of PAP solutions use both strategies.
  • Plan termination: Predicting goals a priori for plan termination yields a 10% success-rate boost over scoring a dedicated stop skill.The comparison averages results across six tasks and ten seeds for each variation, totaling 120 experiments.

7 Limitations and Future Work

Text2Motion’s limitations concern likelihood reliability, dependence on learned skill components, planning runtime, and closed-world perception assumptions. Future work targets efficiency, visual grounding, and open-world capability.

  • Limitations: Unreliable LLM likelihoods can induce cyclic repetition of feasible skills during greedy-search planning and baseline execution.Combining greedy-search with shooting mitigates these failures; likelihood calibration may improve search-based planning over long horizons.
  • Limitations: Text2Motion depends on the fidelity of learned skills, value functions, and dynamics models for future-state prediction.Extending skills to high-dimensional observations may require stable long-term prediction techniques.
  • Limitations: Each LLM API query requires 2-10 seconds, while repeated geometric feasibility planning increases overall planning time.Runtime complexity was not a core focus, and future advances in LLM inference and distillation may reduce the bottleneck.
  • Limitations: The framework assumes task-relevant objects and their poses are known from an external perception system when instructions arrive.Open-world extensions may require exploration or interaction to discover initially unobservable objects and skills supporting diverse real-world objects.
  • Future Work: Future work includes warm-starting geometric feasibility planning and using visually grounded multimodal models.These directions aim to improve plan-time efficiency and extend Text2Motion toward higher-dimensional visual settings.

8 Conclusion

Text2Motion combines LLMs, learned skills, and geometric feasibility planning to construct and verify long-horizon manipulation plans before execution. The authors report that geometric reasoning, hybrid search and shooting, and inferred-goal termination support challenging sequential tasks.

  • Conclusion: Text2Motion combines LLMs, learned skills, and geometric feasibility planning to solve long-horizon manipulation tasks with geometric dependencies.It constructs task- and motion-level plans and tests planned states against goals inferred from natural language instructions before execution.
  • Conclusion: The planner optimistically queries an LLM for long-horizon plans and falls back to reliable search when optimistic planning fails.This hybrid formalism combines shooting-based and search-based planning strengths.
  • Conclusion: Geometric feasibility planning is important for using LLMs and learned skills on sequential manipulation tasks from natural language.Search-based reasoning addresses tasks with many possible plans but few feasible ones, while inferred symbolic goals provide more reliable termination than prior scoring techniques.
  • Acknowledgments: The work was supported by Toyota Research Institute, Toshiba, and NASA’s Innovative Advanced Concepts program.

Overview

The appendix provides implementation, experimental, algorithmic, and real-world demonstration details for Text2Motion and its language-planning baselines.

  • Appendix Overview: Appendix A covers implementation details for Text2Motion and language-planning baselines.
  • Appendix Overview: Appendix B describes the experiments, while Appendix C provides derivations supporting the algorithms.
  • Appendix Overview: Appendix D presents real-world planning demonstrations, with qualitative results available online.

Appendix A Implementation Details

The implementation combines learned manipulation skills, dynamics and Q-function models, OOD detection, an LLM, and geometric feasibility planning. Training and calibration support feasible skill selection and model-based sequence planning.

  • Planner Components: Text2Motion integrates shooting and greedy-search with learned skills, OOD detection, an LLM, and geometric feasibility planning.
  • Skill and Dynamics Models: Q-functions estimate skill feasibility, while dynamics models predict future states needed for geometric feasibility planning.The optimized skill-sequence success probability is approximated from Q-functions along dynamics-model-predicted trajectories.
  • Skill Library: The skill library contains Pick, Place, Pull, and Push policies that must generalize across objects with differing geometries.Each skill uses geometric state features and parameterized manipulation primitives.
  • Model Evaluation: Over 100 evaluation episodes, policy success rates were 99% for Pick, 90% for Place, 86% for Pull, and 97% for Push.Q-functions achieved precision and recall rates above 95%.
  • Skill and Dynamics Models: The forward prediction loss trains a deterministic dynamics model for each skill.The dynamics models converge to within millimeter accuracy on the validation split.
  • OOD Detection: OOD detection addresses symbolically invalid LLM-proposed skills that can otherwise receive erroneously high Q-values.Deep ensembles classify state-action pairs as OOD when predicted Q-value variance exceeds a calibrated threshold.
  • OOD Detection: OOD thresholds were calibrated using 100K invalid states and actions per skill.The reported thresholds are ϵPick = 0.10, ϵPlace = 0.12, ϵPull = 0.10, and ϵPush = 0.06.

A.3 Task planning with LLMs

The planners use LLM-generated skills and geometric feasibility to extend and rank candidate manipulation plans. Text2Motion combines usefulness and feasibility information while optimizing skill parameters for sequential success.

  • Candidate generation: Text2Motion, greedy-search, and myopic baselines generate five candidate skills at each timestep, while shooting generates five complete skill sequences.All LLM queries use temperature 0; shooting directly outputs full sequences, whereas the other planners generate skills incrementally.
  • Skill scoring: Skills are selected using a combined usefulness and geometric feasibility score.Text2Motion, greedy-search, and shooting compute geometric feasibility with planning, while saycan-gs and innermono-gs use a value-function estimate.
  • Skill scoring: A damped SoftMax with β = 0.3 balances LLM usefulness against feasibility and avoids suppressing highly feasible skills with slightly lower log-probability.Undamped SoftMax scoring produced a winner-takes-all effect in which feasible alternatives could be omitted.
  • Geometric feasibility planning: Geometric feasibility planning optimizes parameters for an entire skill sequence to maximize the success probability of its underlying primitives.For Pick(hook) followed by Pull(box, hook), it computes a 3D hook grasp position that enables the subsequent pull.
  • Geometric feasibility planning: Text2Motion can use STAP’s PolicyCEM variant for geometric feasibility planning, with warm-started optimization followed by ten Cross-Entropy Method iterations.Each iteration samples 10K trajectories and selects 10 elites.

Appendix B Experiment Details

The experiments evaluate symbolic planning for tabletop manipulation using hand-crafted scene relations and varied instruction properties. The setup includes lifted, partial-affordance, and multi-step tasks, while assuming externally supplied object knowledge and leaving perception alternatives for future work.

  • Symbolic representation: Scene context is converted into symbolic states using predicates such as on(a, b), under(a, b), and inhand(a).Hand-crafted predicate classifiers detect the modeled spatial relations from geometric state information.
  • Goal representation: The goal-prediction method outputs grounded predicate sets, including alternative goal sets for instructions such as moving two boxes under a rack.The predicted propositions are grounded over specific objects in the scene.
  • Perception assumptions: The evaluation assumes known task-relevant objects and poses, using hand-crafted spatial-relation heuristics rather than learning perception from high-dimensional observations.The authors leave alternative scene-description techniques for future work.
  • Example tasks: The examples cover varied sequential objectives, including pushing selected or all boxes under a rack and relocating the hook or boxes.Other examples include picking any box, stacking objects, and satisfying multiple relations simultaneously.

Appendix C Derivations

The derivations justify approximations for usefulness and feasibility components of the skill score. They rely on future trajectories generated after geometric planning and on conditional-independence and Markov assumptions, with real-robot experiments using RGB-D perception and simulation-trained policies.

  • Skill-score derivation: The skill score is approximated as a product of usefulness and geometric feasibility terms.The usefulness approximation is defined through the LLM-related term, while the feasibility approximation is derived separately.
  • Usefulness score: The usefulness derivation uses a single-sample Monte Carlo estimate over future state trajectories.The estimate follows an expectation of the probability of the next skill conditioned on instruction, states, prior skills, and rewards.
  • Usefulness score: Future states are sampled only after STAP geometrically plans the running sequence, so they correspond to successful execution and positive prior rewards.This supports the independence assumption on previous rewards used in the derivation.
  • Feasibility score: The feasibility derivation reduces the reward probability through conditional-independence, previous-reward independence, and Markov assumptions.The resulting expression uses parameters computed by STAP and a sampled future trajectory.
  • Real-robot setup: Real-robot experiments use Kinect V2 RGB-D capture, manually adjusted color thresholds, and a Franka Panda with simulation-trained policies and models.Object poses are estimated from segmentation masks and depth images.
  • Real-robot setup: The paper provides a project page with demonstrations of Text2Motion operating on a real robot.
Loading 2303.12153v5…