PaperlayerView exact revision on arXiv

Source-linked AI summary

Plan Along the Way: Event-Triggered Foundation-Model Planning for TAMP Execution in Partially Observable Manipulation

Puru Ojha, Narendhiran Vijayakumar, Nav Singhal, Girish Varma, Antony Thomas

arXiv:2608.28075v1cs.RO

TL;DR

Partially observable manipulation can reveal hidden objects after a plan has executed successfully but remains incomplete. ROBUST TAMP plans only from visible relational state, validates and executes task actions through adapters, and replans on discovery or unrecovered failure. Across six RLBench/CoppeliaSim scenes, results show strong performance for 8B and 32B models, substantial gains from in-context learning for implicit non-target handling, and scale-dependent failure modes.

  • Problem

    Foundation-model-guided TAMP often assumes fully specified scenes, leaving hidden task-relevant and non-target objects outside planning under partial observability.

  • Method

    ROBUST TAMP restricts planning to visible relational state, validates actions before execution, and triggers replanning when objects are discovered or failures remain unrecovered.

  • Results

    8B and 32B models achieve mean TSR above 86%, while in-context learning raises 4B LLM grill TSR from 0% to 100% and improves non-target handling.

  • Takeaways & Limitations

    Discovery-triggered replanning integrates planning with execution, while model size alone does not guarantee better performance.

Abstract

from arXiv · show

Manipulation in partially observable environments requires planning under incomplete scene information. In such settings, an initially valid plan may execute successfully yet remain insufficient for task completion. Existing foundation-model-guided task and motion planning (TAMP) systems can generate useful long-horizon task decompositions, subgoals, or constraints, but they often assume having access to a fully specified scene state or invoke model-level replanning after a subgoal, refinement, or execution attempt fails. We present ROBUST TAMP, a modular LLM/VLM-guided planning framework for reactive TAMP where unseen task-relevant and non-target objects may become visible during execution. The framework restricts the foundation-model planner to the currently visible relational scene state, validates generated task-level actions against a strict executable interface, and routes the accepted actions to scene-specific execution adapters. Object discovery is treated as a distinct replanning event and, after a stable execution horizon, the system reconstructs the visible scene state and replans using completed-action history and structured replanning event context. Evaluations are performed on six RLBench/CoppeliaSim kitchen and grill variants involving hidden objects, non-target object discovery, articulated-container interaction, and temporal manipulation procedures. We compare text-only LLM and VLM planners of different sizes under the same validation, execution, monitoring, and replanning pipeline, reporting task success, partial goal completion, discovery- and failure-triggered replanning behavior, implicit non-target-object handling, and planner inference cost.

I. INTRODUCTION

ROBUST TAMP addresses long-horizon manipulation when hidden objects make an initially valid plan insufficient. It combines visible-state planning, validated execution, event-triggered replanning, and layered recovery for partial observability.

  • Motivation: Partial observability can reveal hidden task-relevant objects after an initially valid plan executes successfully.This creates a failure mode distinct from ordinary plan-execution failure.
  • Motivation: Existing foundation-model planning approaches commonly assume fully observable scenes or replan only after plan or execution failure.Prior systems generate task skeletons, constraints, subgoals, or repairs but do not explicitly treat discovery as its own event.
  • Framework: Object discovery triggers scene reconstruction and replanning with completed-action history and structured event context, while recoverable execution failures remain local to the backend.This separates semantic replanning from executor-local recovery and reduces unnecessary foundation-model calls.
  • Framework: ROBUST TAMP plans from the currently visible relational state, validates actions against an executable interface, and delegates them to scene-specific execution adapters.The framework integrates perception, task planning, validation, symbolic execution, motion planning, and monitoring.
  • Evaluation: The evaluation studies six kitchen and grill scenes across hidden objects, non-target discovery, articulated containers, and procedural manipulation.Models vary in modality and scale under a common validation, execution, monitoring, and replanning pipeline.

II. RELATED WORK

Related work uses foundation models to translate goals, generate plans, and guide TAMP, but generally assumes more complete scene knowledge. ROBUST TAMP instead plans only over currently visible objects and treats discovery as a separate replanning event.

  • Foundation-model planning: LLM-based systems translate natural-language goals into formal representations, symbolic plans, constraints, subgoals, or continuous parameters checked by downstream modules.These approaches reduce reliance on manually specified domain abstractions but still commonly encode procedural states and object roles explicitly.
  • Replanning: Most existing systems initiate replanning after a plan, subgoal, refinement, or execution attempt fails.ROBUST TAMP distinguishes discovery-triggered replanning from failure-triggered replanning because hidden objects can alter task requirements.
  • Visible-state planning: ROBUST TAMP receives only the visible relational state and natural-language goal, excluding unobserved objects until interaction reveals them.Procedural requirements are verified deterministically while the planner infers required sequences from observed evidence and goal text.
  • Example: The kitchen example hides a task-relevant object initially, illustrating why interaction must update the planning evidence.The task requires placing mugs in a container and groceries in a cupboard.
  • Visible-state planning: The planner-visible context includes visible objects, region assignments, placement regions, robot and articulation states, completed actions, event context, and the task goal.This context supports later plans without assuming relations for objects that remain unobserved.

B. Planning and Execution Interface

At each planning event, the foundation model maps visible context and history to an ordered task-level action sequence. Validation precedes execution, while task completion combines final relations with procedural predicates.

  • Planning interface: The planner maps context c_k to an ordered sequence α_k = π(c_k) of domain-specific executable actions.Kitchen actions include pick(o), place(o,r), and open(c), while continuous grasps and trajectories are supplied later by execution.
  • Execution interface: A validator checks whether the generated sequence is realizable before the execution backend grounds actions through scene-specific adapters.Adapters report structured success or failure outcomes for subsequent system handling.
  • Planning objective: The planner objective is to synthesize and execute action sequences across planning events until task completion or termination.Planning therefore operates as an event-indexed loop rather than a single fixed sequence.
  • Task objective: Task completion is evaluated using required final object-region relations R_τ and procedural predicates P_τ over action history.Grill tasks use predicates for temporal procedures such as cooking before placing meat on a plate.

IV. METHODOLOGY

The methodology repeatedly reconstructs a visible scene state, queries and validates a foundation-model plan, executes it through adapters, and responds to discoveries or unrecovered failures with structured events.

  • Execution loop: The loop queries an LLM or VLM, validates the action sequence, sends it to the execution layer, and monitors structured outcomes.Algorithm 1 represents this as repeated planning iterations subject to a replanning budget.
  • State construction: Each planning event constructs s^vis_k from currently available perception evidence and excludes objects that remain unobserved.In the kitchen domain, a hidden mug enters the visible state only after the container opens.
  • State construction: The state constructor converts visibility evidence into symbolic object-region relationships while retaining raw geometry for execution grounding.The planner receives lightweight relational information rather than raw coordinates or region bounds.
  • Evaluation setup: The RLBench/CoppeliaSim evaluation uses segmentation masks for visibility and privileged simulator geometry for semantic resolution and execution grounding.These signals isolate planning, monitoring, and replanning behavior from perception errors.
  • Event handling: Execution reports action history, newly visible objects, and unrecovered failures so the system can create discovery or failure events.A newly discovered object causes the loop to continue with updated planning context.

B. FM Planning and Layered Plan Validation

The planner converts visible context and the natural-language goal into a structured task sequence, while layered validation rejects actions that do not satisfy the executable interface before execution.

  • The Query subroutine combines the visible context and natural-language goal to produce a structured prompt and task sequence.
  • During replanning, completed-action history and structured triggering-event context are added to the planner prompt.
  • A constrained parser rejects sequences containing unsupported actions, objects, or regions or violating structural action-sequence requirements.
  • Structural validation is separated from scene-dependent checks, which assess visibility, accessibility, and action preconditions immediately before execution.

C. Heterogeneous Grounding and Execution

ROBUST TAMP separates model-generated task order from physical realization, allowing heterogeneous execution backends and bundling related actions when appropriate.

  • The validated task sequence constrains intended action order, while Xd determines how each action is physically realized.
  • Action bundling groups realizable sequences such as pick(o)-place(o, r) and preserves physical context across related actions.
  • L1 denotes state-independent validation before execution, whereas L2 denotes failures detected through scene checks or low-level execution feedback.
  • Non-bundled actions are executed individually.
  • Scene-specific adapters may use PDDLStream, geometric helpers, or fixed motion primitives and report success or structured execution failure.

D. Closed-Loop Monitoring and Layered Failure Escalation

The monitoring layer checks actions against the current scene, applies local recovery where possible, and escalates only unresolved failures or newly discovered objects to structured replanning.

  • Before, during, and after execution, monitoring checks scene relevance, physical realization, and the intended resulting state.
  • Local fallbacks address recoverable failures through alternative grasps, trajectories, or scene-specific recovery routines.
  • Unresolved failures become structured events containing their source, execution stage, associated action, and supporting evidence for planner revision.
  • An obstructed mug causes an attempted container-opening action to be rejected before execution.
  • Previously unobserved objects trigger replanning even after the preceding execution routine completes successfully.
  • After discovery, execution pauses at a stable horizon, the visible scene is reconstructed, and the planner generates a revised sequence incorporating the new object.

V. EXPERIMENTS AND EVALUATIONS

The evaluation spans six RLBench/CoppeliaSim kitchen and grill variants designed to test hidden-object discovery, progress preservation, and implicit handling of non-target objects.

  • The study evaluates event-based replanning, modality and scale trade-offs, in-context learning, and dominant planning, parsing, or execution failures.
  • Six curated scenes use common model-facing interfaces and execution adapters while varying object arrangements, hidden objects, and task procedures.
  • Kitchen variants require grocery and mug placement, with hidden grocery or mug discoveries and increasing plan horizon across K1–K3.
  • Kitchen variants test whether newly revealed objects can be incorporated while preserving progress made before discovery.
  • Table II reports phone-excluded task-success rates by variant and aggregates mean TSR separately from INH for G1 and G3.
  • Grill variants require cooking raw meat and serving it, while G1 and G3 include a phone that should be relocated from the grill to the table when present.

B. Compared Methods and Model Backends

The evaluation compares six interchangeable LLM and VLM task-planning front-ends across three model sizes, with zero-shot and in-context prompting under an otherwise shared ROBUST TAMP pipeline.

  • Model Backends: Six planners span 4B, 8B, and 32B sizes in both text-only LLM and vision-language VLM variants.The compared models are Qwen3 and Qwen3-VL-Thinking backends at small, medium, and large scales.
  • Controlled Comparison: All model conditions operate as interchangeable task-planning front-ends within the complete ROBUST TAMP pipeline.The same downstream execution stack enables comparisons focused on the planning front-end rather than pipeline changes.
  • Inputs: Each planner receives the same planning inputs, while VLMs additionally receive a fresh stitched RGB composite at every planning and replanning event.The visual composite is captured from simulator cameras for both initial planning and replanning.
  • Prompting Conditions: Both zero-shot and ICL prompting conditions are evaluated using the same constrained parser, scene-dependent checks, monitoring, and deterministic task validation.This keeps downstream execution and validation consistent across model modality, scale, and prompting conditions.

C. Evaluation Protocol and Metrics

The study evaluates repeated model conditions on kitchen and grill tasks using deterministic success and partial-completion measures, alongside discovery, recovery, non-target handling, and failure analyses. Results show that event-triggered replanning supports partial-observability performance, while model scale and ICL affect recovery and implicit object handling in different ways.

  • Evaluation Protocol: Each model–variant–prompting condition is evaluated over 10 independent trials with reset simulator scenes and a replanning limit of 10.Planning uses temperature 0 and a maximum generation length of 4096 tokens.
  • Metrics: Task Success Rate verifies final object–region relationships and, for grill variants, temporally ordered cooking procedures.Successful trials are reported as both counts and percentages.
  • Metrics: Partial Goal Completion measures the proportion of deterministic final relations and procedural predicates satisfied at termination, reaching 1 exactly at complete task success.The metric does not require matching a canonical action sequence.
  • Discovery-Triggered Replanning: Mean TSR exceeds 86% for 8B and 32B models, while the lowest 4B LLM zero-shot TSR is 26.7% versus 42.5% mean PGC.The difference indicates substantial progress can occur before trials terminate unsuccessfully.
  • Model Modality and Scale: Larger models require fewer failure-triggered replans but have higher planner-call latency; 8B models achieve TSR comparable to 32B models at lower computational cost.The results identify medium-scale planners as the most favorable tradeoff among success, recovery quality, and planning cost.
  • Dominant Failures: Parser and executable-interface failures dominate smaller models, whereas larger-model failures primarily involve execution grounding, motion, placement, and post-execution validation.ICL significantly reduces parser/interface failures for all LLMs.

VI. CONCLUSION

ROBUST TAMP integrates discovery-triggered replanning with hierarchical failure detection and recovery for reactive manipulation under partial observability. The evaluation examines trade-offs across model modality, size, and prompting, while identifying unresolved challenges in implicit non-target-object reasoning.

  • Framework and implications: ROBUST TAMP uses discovery-triggered replanning to integrate planning with execution when unseen task-relevant and non-target objects become visible.The framework is described as a closed-loop mechanism for reactive TAMP under partial observability.
  • Framework and implications: The framework combines closed-loop execution with efficient, hierarchical failure detection and recovery.The conclusion links this mechanism to leveraging foundation-model reasoning capabilities in reactive TAMP.
  • Open challenges: The study highlights the need for deeper analysis of how foundation models reason about implicit non-target objects.This remains an identified challenge within the evaluated manipulation setting.
  • Evaluation trade-offs: Fig. 3 compares mean planner time per trial, TSR, and total failure-triggered replans across 4B, 8B, and 32B model scales.Circular and triangular markers distinguish LLM and VLM planners, respectively.
  • Evaluation trade-offs: Fig. 4 presents different failure modes for the evaluated models.The figure is used alongside the study’s analysis of modality, size, and prompting trade-offs.
Loading 2608.28075v1…