PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding Long-Horizon Task and Motion Planning with Vision Language Models

Zhutian Yang, Caelan Garrett, Dieter Fox, Tomás Lozano-Pérez, Leslie Pack Kaelbling

arXiv:2410.02193v1cs.RO

TL;DR

VLMs can generate plausible common-sense plans but may omit prerequisite steps and lack geometric reasoning, while TAMP handles feasibility but scales poorly and lacks open-world semantics. VLM-TAMP uses VLM-proposed subgoals with TAMP refinement and replanning; on long-horizon kitchen tasks, it outperforms action-sequence baselines in success and task completion.

  • Problem

    VLMs can omit prerequisite actions and cannot ensure geometric feasibility for a particular robot, while TAMP has difficulty scaling and lacks open-world semantics.

    Source: §I · pp. 1–2

  • Method

    VLM-TAMP uses a VLM to propose intermediate subgoals that TAMP solves in sequence, with replanning when a subgoal or action cannot be refined.

    Source: §I · p. 2, §III · p. 3

  • Results

    VLM-TAMP succeeds 50 to 100% of the time versus 0% for action-prediction baselines, with task completion of 72 to 100% versus 15 to 45%.

    Source: §I · p. 2, D. Results · p. 6

  • Takeaways & Limitations

    The combined system uses VLM decomposition and TAMP refinement to address VLM reasoning shortcomings and TAMP’s computational complexity.

    Source: §VI · p. 7

Abstract

from arXiv · show

Vision-Language Models (VLM) can generate plausible high-level plans when prompted with a goal, the context, an image of the scene, and any planning constraints. However, there is no guarantee that the predicted actions are geometrically and kinematically feasible for a particular robot embodiment. As a result, many prerequisite steps such as opening drawers to access objects are often omitted in their plans. Robot task and motion planners can generate motion trajectories that respect the geometric feasibility of actions and insert physically necessary actions, but do not scale to everyday problems that require common-sense knowledge and involve large state spaces comprised of many variables. We propose VLM-TAMP, a hierarchical planning algorithm that leverages a VLM to generate goth semantically-meaningful and horizon-reducing intermediate subgoals that guide a task and motion planner. When a subgoal or action cannot be refined, the VLM is queried again for replanning. We evaluate VLM- TAMP on kitchen tasks where a robot must accomplish cooking goals that require performing 30-50 actions in sequence and interacting with up to 21 objects. VLM-TAMP substantially outperforms baselines that rigidly and independently execute VLM-generated action sequences, both in terms of success rates (50 to 100% versus 0%) and average task completion percentage (72 to 100% versus 15 to 45%). See project site https://zt-yang.github.io/vlm-tamp-robot/ for more information.

I. INTRODUCTION

VLMs provide common-sense plans but can omit prerequisite steps and lack geometric reasoning; TAMP handles feasibility but scales poorly and lacks open-world semantics. VLM-TAMP combines them by using VLM subgoals for TAMP refinement and replanning.

I. INTRODUCTION

  • VLM plans can omit obvious prerequisite steps and lack geometric reasoning, while TAMP lacks commonsense semantics and scales poorly with problem complexity.TAMP’s guarantees depend on an accurate domain model and common assumptions.Source: §I · pp. 1–2
  • VLM-TAMP uses VLM-generated subgoals for a TAMP planner to refine, add needed actions, and re-prompt after a subgoal cannot be grounded or is infeasible.Re-prompting uses the current world state and objects that collided during the failed planning process.Source: §I · p. 2
  • The evaluation uses two robot embodiments in procedurally generated kitchen scenes and cooking problems with plans of 30 to 50 actions.The tasks involve making chicken soup under different scene initial conditions.Source: §I · p. 2
  • On challenging very long-horizon problems with plans ranging from 30 to 50 actions, VLM-TAMP succeeds 50 to 100% of the time, while naive VLM action prediction plus motion planning never succeeds.The baseline completes on average only 15 to 45% of subproblems before failing to refine a suggested action, even with chances to reprompt.Source: §I · p. 2

II. RELATED WORK

Prior language-model planning approaches address task specification and replanning, but geometric infeasibility remains difficult to handle at scale. VLM-TAMP uses intermediate goals rather than actions and lets TAMP refine them; the method decomposes long-horizon problems into sequential subproblems.

II. RELATED WORK

  • Prior language-model methods translate task descriptions, propose actions, or guide motion, but do not explicitly handle geometrically infeasible action sequences.Text-template replanning has been shown on small tabletop tasks, while describing geometric obstacles this way does not scale to larger environments.Source: §II · p. 2
  • Unlike prior approaches, VLM-TAMP has an LLM generate intermediate goals instead of actions, then uses TAMP to add steps and solve continuous parameters for geometric and kinematic feasibility.A prior shooting-based approach depends on robot- and scene-specific Q-functions and does not use local geometric feasibility to guide the next plans.Source: §II · p. 3

III. METHOD

  • VLM-TAMP breaks a long-horizon manipulation problem into smaller problems that TAMP solves in sequence, with replanning for translation errors, infeasible plans, and planning failures.The overview describes the decomposition, sequential refinement, and replanning mechanism.Source: §III · p. 3

A. Problem Formulation

The problem representation combines a geometric world model and natural-language goal with a textual scene description for the VLM. The system produces robot commands and uses each achieved subgoal’s resulting state to begin the next one.

A. Problem Formulation

  • The problem assumes a correct 3D model of the world and robot plus a natural-language goal; the VLM receives object and relation descriptions because scene images alone do not fully communicate the model.The example scene image uses rendered objects with semantic labels and bounding boxes.Source: A. Problem Formulation · p. 3
  • The system outputs robot joint-angle trajectories for each subproblem, and the state after achieving one subgoal initializes planning and execution for the next.The passage describes commands as joint-angle trajectories.Source: A. Problem Formulation · p. 3
  • The VLM query template requests a short sequence of outputs to accomplish a goal and supplies the scene image, object relations, and planning objects.The conversation templates also show examples for VLM-TAMP and the VLM-plus-motion-planning baseline.Source: A. Problem Formulation · p. 3, Figure 4 · p. 3

B. Approach

VLM-TAMP can request either subgoals or actions, then checks their semantics and constructs a TAMP problem. Its VLM query process translates English proposals into PDDL, with predefined predicates or actions and action preconditions guiding the outputs.

B. Approach

  • The approach has two modes: the VLM predicts subgoals or high-level actions, which undergo semantic checking before a TAMP problem is constructed.For subgoals, TAMP constructs detailed motion plans that may move objects or open cabinets; for actions, it grounds continuous parameters such as grasps and paths.Source: B. Approach · p. 3

C. Using VLM for Subgoal or Action Sequencing

  • Each VLM query first uses the goal, scene objects, and spatial relations to propose English intermediate subgoals, then translates them into PDDL for TAMP.The VLM can instead translate an English action plan into a PDDL action sequence.Source: C. Using VLM for Subgoal or Action… · p. 4
  • Subgoal translation uses a predefined predicate list, while action translation uses legal actions and their preconditions, such as requiring an empty hand before picking an object.The current formulation uses one goal tuple per subgoal.Source: C. Using VLM for Subgoal or Action… · p. 4
  • A symbolic planner verifies each proposed entry for semantic validity; when an entry is impossible, its error message prompts the VLM to try again until the query limit is reached.Errors include nonexistent or wrongly typed objects and incorrect argument counts.Source: C. Using VLM for Subgoal or Action… · p. 4

D. Using TAMP to Refine Subgoals or Action Sequences

The system refines subgoals and actions through TAMP, grounding plans in continuous parameters and motion trajectories. For subgoals, it starts with goal-relevant objects and adds collision-relevant objects when needed.

D. Using TAMP to Refine Subgoals or Action Sequences

  • Given subgoals or actions, TAMP refines them in sequence into grounded action plans and corresponding motion trajectories.A solution is a finite sequence of grounded actions that reaches the goal from the initial state.Source: D. Using TAMP to Refine Subgoals or… · p. 4
  • A TAMP problem comprises objects, initial literals, goal literals, and a domain of predicates and actions that supports continuous values.The domain extends PDDL, and grounded actions deterministically transition the world state.Source: D. Using TAMP to Refine Subgoals or… · p. 4
  • For each subgoal, TAMP first plans over goal-relevant objects, then adds objects implicated in collisions if planning fails.After N_TAMP unsuccessful calls, the subgoal is declared unreachable; successful plans are executed and the current state is updated.Source: D. Using TAMP to Refine Subgoals or… · pp. 4–5
  • Subgoal predicates specify spatial or motion-level entities, since omitting objects can prevent planning when a predicate leaves out directly relevant entities.For example, Heat⟨object⟩ omits the heating surface and appliance handle, while PlaceOn⟨object, surface⟩ names relevant entities.Source: D. Using TAMP to Refine Subgoals or… · p. 5
  • To refine an action, TAMP uses its symbolic effects as a subgoal and is constrained to a partial plan skeleton containing that action.The planner chooses continuous parameters and may add actions such as base motion; this refinement also uses reduced object sets and multiple trials.Source: D. Using TAMP to Refine Subgoals or… · p. 5

E. VLM Replanning after TAMP Failure

After repeated TAMP failure, the VLM receives updated scene information, executed actions, and detected collision objects to replan. This approach assumes subgoals lack long-term low-level dependencies that require one detailed plan for the whole problem.

E. VLM Replanning after TAMP Failure

  • After N_TAMP failed trials, the VLM is reprompted with the updated scene, executed actions, and collision objects, up to N_reprompt times before failure is declared.The approach assumes there are no long-term low-level dependencies among subgoals that require planning the entire problem at once.Source: E. VLM Replanning after TAMP Failure · p. 5

IV. EXPERIMENTS

The experiments examine whether sequencing subgoals or actions yields better task completion, and how reprompting and additional compute budgets relate to the number of solvable problems.

IV. EXPERIMENTS

  • The experiments ask whether increasing compute budgets increases the number of problems VLM-TAMP can solve.Source: §IV · p. 5
  • The experiments compare task completion performance when the VLM sequences subgoals rather than actions.Source: §IV · p. 5
  • The experiments ask how much reprompting improves performance.Source: §IV · p. 5

A. Baselines and Ablations

The study compares VLM subgoal sequencing with TAMP refinement against action sequencing with limited TAMP refinement across kitchen task and robot variations. It evaluates task success and task completion percentage over randomized trials.

A. Baselines and Ablations

  • The comparison includes VLM subgoal sequencing with TAMP refinement and VLM action sequencing with limited TAMP refinement, each with three reprompting variants.The variants allow N_reprompt ∈ {0, 1, 2}; the action baseline cannot add actions other than moving the robot’s base.Source: A. Baselines and Ablations · p. 5

B. Task Suite and Robot Embodiment

  • The chicken-soup task varies initial conditions: all doors are open and the pot lid is on the counter in Easy, versus closed doors and a lid covering the pot in More Obstacles.The scene includes five movable objects, eight surfaces, two enclosed spaces, and six articulated objects; subsequent object poses and joint positions are randomly sampled.Source: B. Task Suite and Robot Embodiment · p. 5
  • The robot uses either its left arm only or both arms, and interacts through pick-and-place and articulated-joint manipulation.The environment includes challenging constraints such as a small sink and limited collision-free positioning near the fridge door and faucet.Source: B. Task Suite and Robot Embodiment · p. 5
  • Six methods are tested on four task variations over 30 random trials per method, using five VLM plans per task and measuring task success and task completion percentage.Task success requires refining all generated actions or subgoals into collision-free trajectories; task completion percentage measures solved subproblems among those generated.Source: B. Task Suite and Robot Embodiment · p. 5

C. Implementation Details

Implementation uses PDDLStream for TAMP and allows up to three planning runs per subproblem, with more world objects considered in successive runs.

C. Implementation Details

  • PDDLStream’s diverse planning mode considers at most 12 plan skeletons, while action refinement restricts the skeleton to the predicted action and base movement.The task planner omits the task-plan feasibility predictor used in the referenced setup.Source: C. Implementation Details · p. 5
  • Each subproblem receives three planning runs, with an increasing number of world objects included in planning.A subproblem counts as successful if it fails on the first two runs but succeeds on the third.Source: C. Implementation Details · pp. 5–6
  • The six methods are evaluated in 30 random trials across four difficulties with increasing numbers of controllable arms and manipulable obstacles.Source: Figure 6 · p. 6

D. Results

VLM-TAMP outperforms action-prediction baselines on task success and completion, while reprompting improves subgoal performance on harder problems. The failure analysis identifies VLM prediction and translation errors, as well as infeasible subproblems.

D. Results

  • VLM-TAMP succeeds 50 to 100% of the time versus 0% for VLM action prediction, and task completion is 72 to 100% versus 15 to 45%.TAMP resolves geometric infeasibility by moving articulated or movable obstacles, using 1 to 2 actions on average.Source: D. Results · p. 6
  • On harder problems, reprompting increases VLM-TAMP task success by 47 to 55%, but does not help the action-prediction variant.More reprompting gives TAMP additional attempts at difficult subproblems and may add intermediate subgoals; it also reduces variance in completion percentage.Source: D. Results · p. 6

V. FAILURE ANALYSIS

  • The system’s VLMs routinely make inaccurate predictions, although the authors expect them to improve over time.Source: §V · p. 6

A. VLM Failures

  • English-to-PDDL translation can omit subgoals or actions, produce tuples with the wrong number of arguments, or misspell object names.Source: A. VLM Failures · p. 6
  • A subproblem can be infeasible when movable or articulated objects block access or when an action’s preconditions are unmet.Examples include a pot blocking the path to a stove knob and the robot’s only arm already holding an object when asked to close a door.Source: A. VLM Failures · p. 6

B. TAMP Failures

TAMP failures include missing a feasible plan skeleton and failing to refine one within the compute budget. The discussion frames VLM-TAMP as combining VLM common-sense reasoning with TAMP’s geometric planning, and points to dexterous dual-arm policies as future work.

B. TAMP Failures

  • The planner may miss a feasible plan skeleton because it tries only 12 skeletons in increasing task-length order.For a subproblem with six planning objects and two obstacles to clear, the correct skeleton may fall outside those 12.Source: B. TAMP Failures · p. 6
  • Even a feasible skeleton may not be refined within the compute budget because low-level samplers have limited attempts and must all succeed in sequence.Source: B. TAMP Failures · pp. 6–7
  • A VLM can still translate an instruction to close a cabinet using a hand holding an object into an infeasible close-door goal, despite a prompt rule requiring an empty hand.Source: B. TAMP Failures · p. 6

VI. DISCUSSION

  • VLM-TAMP combines VLM language and common-sense reasoning with TAMP’s feasible task skeletons and collision-free trajectories, using subgoals and reprompting to address long-horizon complexity.Future work includes training dexterous dual-arm manipulation policies to achieve subgoals from visual inputs.Source: §VI · p. 7

Related papers

Loading 2410.02193v1…