PaperlayerView exact revision on arXiv

Source-linked AI summary

TANDEM: Task and Motion Planning with As-Needed Demonstrations for Efficient Vision-Language-Action Model Fine-tuning

Samrat Sahoo, Liang Ji, Tom Silver, Yixuan Huang

arXiv:2609.28314v1cs.RO

TL;DR

Human teleoperators spend time demonstrating behaviors robots can already perform, and fixed TAMP domains may not cover every stage of long-horizon tasks. TANDEM combines TAMP with selective human teleoperation by representing unsupported stages as human-executed planning operators. It collects more demonstrations at matched human intervention time and supports higher VLA task success after fine-tuning.

  • Problem

    Human effort spent demonstrating behaviors robots can already perform limits scalable data collection, while fixed planning domains may not support every stage of long-horizon tasks.

    Source: §I · p. 1

  • Method

    TANDEM extends TAMP with human-executed magic operators for unsupported stages, combining selective teleoperation with autonomous execution.

    Source: §I · pp. 1–2

  • Results

    TANDEM collects approximately 2.9× more demonstrations than full-task teleoperation at matched human intervention time; fine-tuned π0.5-DROID average success rises from 0% to 60.0% across five tasks.

    Source: B. Experimental Results · p. 6

  • Takeaways & Limitations

    TANDEM reduces human effort to collect useful demonstrations for tasks beyond the planner’s capabilities.

    Source: §I · p. 2

    TANDEM inherits limitations of the underlying TAMP system, the dominant source of demonstration failures in the experiments; human low-level execution is delegated entirely to teleoperators.

    Source: §VI · p. 7

Abstract

from arXiv · show

Human teleoperators spend substantial time demonstrating behaviors that robots can already perform autonomously, limiting the scalability of data collection for robot foundation models. Task and motion planning (TAMP) can automate many of these behaviors, but a fixed planning domain may not support every stage of a long-horizon manipulation task. We present TANDEM (Tamp with As-Needed Demonstrations for Efficient Model fine-tuning), a system that combines TAMP with selective human teleoperation to collect demonstrations for tasks beyond the planner's capabilities. Our key idea is to represent human assistance as an on-demand planning capability. Given a language instruction and visual observation, TANDEM uses pretrained vision-language models to extend the planning domain with missing predicates and human-executed magic operators. This allows the planner to interleave autonomous and human-executed stages without task-specific intervention points. After each human stage, TANDEM re-perceives the scene and checks whether the intended effects hold before resuming autonomous planning. To support fine-tuning vision-language-action (VLA) models, TANDEM also uses example pretraining trajectories to align planner-generated motions with the target model's pretraining distribution. We evaluate TANDEM on five long-horizon manipulation tasks beyond the TAMP domain's capabilities. On a representative long-horizon task, TANDEM collects 2.9x as many demonstrations as full-task teleoperation at the same human intervention time. Fine-tuning a pretrained \pi_{0.5}-DROID model on 20 TANDEM demonstrations per task increases average task success from 0% to 60% across the five tasks.

I. INTRODUCTION

Human teleoperators spend time demonstrating behaviors robots can already perform, while fixed TAMP domains may not cover every stage of long-horizon tasks. TANDEM combines autonomous planning with on-demand human assistance, and the paper reports gains in both demonstration efficiency and downstream VLA success.

I. INTRODUCTION

  • TANDEM uses VLMs to extend TAMP with predicates and human-executed magic operators, allowing unsupported stages to be delegated while TAMP executes the rest.Example pretraining trajectories are used to align planner-generated motions with the target VLA’s pretraining distribution.Source: §I · pp. 1–2

C. Open-Vocabulary Planning and Model Learning

  • TANDEM uses VLMs to invent and ground symbolic predicates and operators, composing human-executed stages with existing robot skills.The compared methods learn models for composing supplied low-level skills; TANDEM specifies and verifies stages without autonomous controllers.Source: §I · p. 2

I. INTRODUCTION

  • On Cover Bread Rolls, TANDEM collects approximately 2.9× as many demonstrations as full-task teleoperation at matched human intervention time; fine-tuning raises average success from 0% to 60%.The success result is reported for a pretrained π0.5-DROID model evaluated across five long-horizon tasks.Source: §I · p. 2

A. Planning-Based Demonstration Generation

  • Planning-based data generation includes adapting human demonstrations, generating TAMP demonstrations, and aligning planner-generated motions with VLA pretraining data [6].TANDEM adopts the alignment approach for autonomous portions and extends planning-based collection to stages beyond planner capabilities.Source: A. Planning-Based Demonstration… · p. 2

B. Human-Assisted Demonstration Collection

  • Unlike HITL-TAMP, TANDEM does not require task-specific teleoperated action schemas and uses complete demonstrations to fine-tune a VLA rather than local policies.HITL-TAMP retains TAMP at deployment after training local policies for human-executed segments.Source: B. Human-Assisted Demonstration… · p. 2

III. PROBLEM SETUP

The problem setup treats demonstration collection as adapting a pretrained VLA to long-horizon tasks using successful observation–action episodes. It frames collection under a human-effort budget and evaluates the resulting policy by average success across target tasks.

A. Demonstration Generation for VLA Fine-tuning

  • The policy maps observation–action history and language instructions to distributions over robot actions.Observations contain camera RGB images and robot proprioception data.Source: A. Demonstration Generation for VLA… · p. 2
  • TANDEM’s demonstration pipeline interleaves autonomous TAMP phases with human-assisted phases, then re-perceives the scene and verifies human-stage effects.The overview describes planning from a language instruction and initial visual observation.Source: Figure 2 · p. 3
  • A dataset D contains successful task episodes, and fine-tuning the pretrained VLA on D yields a task-adapted policy.Each recorded episode pairs its observation–action sequence with the task’s language instruction.Source: A. Demonstration Generation for VLA… · pp. 2–3

B. Human-Effort-Constrained Data Generation

  • Human effort is the sum of teleoperation time for successful demonstrations, and admissible datasets must fit a fixed effort budget.The setup measures data usefulness by average success probability of the fine-tuned policy over the target-task set.Source: B. Human-Effort-Constrained Data… · p. 3

IV. PROPOSED APPROACH: TANDEM

TANDEM combines a pretrained VLM with TAMP to generate complete demonstrations, adding missing predicates and human-executed operators when the base domain cannot express or achieve a task stage. The preliminaries define symbolic states and operators used to represent those plans.

IV. PROPOSED APPROACH: TANDEM

  • TANDEM’s VLM generates missing predicates and human-executed magic operators, plans their combination with autonomous actions, and evaluates human-stage effects.The resulting task plan combines autonomous TAMP execution with selective teleoperation.Source: §IV · p. 3

A. Preliminaries

  • The base TAMP domain represents scene properties as predicates and abstracts observation history into a symbolic state containing facts judged to hold.For the base domain, the state facts are defined using its predicate set.Source: A. Preliminaries · p. 3
  • A TAMP operator specifies continuous parameters, symbolic preconditions and effects, and a low-level controller that executes the transition.Preconditions describe facts required before execution, while effects describe facts expected afterward.Source: A. Preliminaries · pp. 3–4
  • The base domain can lack predicates for a desired state or operators to achieve it; for example, it may neither describe an open box nor open one.The example contrasts missing symbolic representation with missing manipulation capability.Source: A. Preliminaries · p. 4

B. Method Overview

TANDEM extends a fixed TAMP domain with missing predicates and human-executed operators, then plans task phases with assigned subgoals and executors. Fresh observations and predicate checks help verify human stages before execution continues.

B. Method Overview

  • When base controllers cannot achieve a desired state, a VLM generates magic operators and any new predicates needed to express their preconditions and effects.Source: B. Method Overview · p. 4
  • The planner orders task phases, each pairing a symbolic subgoal with either robot or human execution.Source: B. Method Overview · p. 4

C. Predicate and Magic Operator Generation

D. Task Plan Generation and Execution

TANDEM executes planned phases in sequence, using TAMP for robot subgoals and teleoperation for human subgoals. It re-perceives and verifies after phases, then records successful trials and aligns planner-generated motions with the target VLA’s pretraining distribution.

D. Task Plan Generation and Execution

  • For each robot phase, TAMP plans from the subgoal and current scene, selects base-domain operators, and generates a trajectory for autonomous execution.Source: D. Task Plan Generation and Execution · p. 4
  • For a human phase, the operator receives a natural-language subgoal and controls the robot, while the planner reasons over the magic operator’s symbolic transition.Source: D. Task Plan Generation and Execution · p. 4
  • After each phase, TANDEM updates its scene observation; after human phases it verifies intended effects, terminating and excluding trials when verification fails.Verification rarely rejected a trial in the experiments.Source: D. Task Plan Generation and Execution · p. 4

E. Demonstration Generation

  • Successful trials combine autonomous and teleoperated phase trajectories in execution order, while example pretraining trajectories guide alignment of TAMP-generated motions.Human-executed portions need no separate alignment step.Source: E. Demonstration Generation · pp. 4–5

V. EXPERIMENTS

TANDEM is evaluated on five long-horizon tasks requiring both autonomous pick-and-place and human-assisted manipulation. Results show improved VLA performance, more demonstrations per unit of human effort, and automatic extension of an incomplete TAMP domain.

A. Experimental Setup

  • The five tasks combine autonomous pick-and-place with unsupported manipulation, and the baselines include Raw TAMP, oracle teleoperation, pretrained π0.5-DROID, and HITL-TAMP.TANDEM and HITL-TAMP use the same underlying TAMP system and teleoperation interface.Source: A. Experimental Setup · p. 5

B. Experimental Results

  • TANDEM collects 100 successful demonstrations from 130 attempts (76.9%) and automatically interleaves human-assisted stages with autonomous TAMP execution.Raw TAMP cannot complete any of the five tasks because each includes a stage outside its pick-and-place capabilities.Source: B. Experimental Results · pp. 5–6, Table I · p. 6
  • Most failed episodes stem from real-robot TAMP execution (93.3%); predicate/operator invention accounts for 6.7%, all on Store Bread in Closed Box.None of the failed episodes is attributed to TAMP planning.Source: Figure 4 · p. 5, B. Experimental Results · p. 6
  • At matched human-time budgets on Cover Bread Rolls, TANDEM collects approximately 2.9× more demonstrations than full-task teleoperation.At 524 seconds, TANDEM collects 40 demonstrations versus 14 for oracle teleoperation.Source: Figure 5 · p. 6, B. Experimental Results · p. 6
  • Across five long-horizon tasks, TANDEM demonstration fine-tuning achieved 60.0% average success versus HITL-TAMP’s 17.0%, with 77.6% average task progress over 20 evaluation trials per task.The pretrained π0.5-DROID achieved 0% success across all five tasks.Source: B. Experimental Results · p. 6, Figure 6 · p. 7
  • On Cover Bread Rolls, success rises from 45% with 20 demonstrations to 80% with 60, then is 75% with 80 demonstrations.The study reports that performance begins to saturate at approximately 60 demonstrations in this experiment.Source: B. Experimental Results · p. 7

VI. CONCLUSION

TANDEM pairs autonomous TAMP with selective human intervention, and its demonstrations improve fine-tuned VLA task success across five tasks. The authors identify inherited TAMP limitations and visual verification and generalization boundaries.

VI. CONCLUSION

  • Fine-tuning π0.5-DROID on 20 TANDEM demonstrations per task raises average task success from 0% to 60% across five tasks; the resulting policy outperforms HITL-TAMP.The resulting policy also supports direct VLA inference at 15 Hz.Source: §VI · p. 7
  • On Cover Bread Rolls, TANDEM collects approximately 2.9× more successful demonstrations than full-task teleoperation at matched human intervention time.Source: §VI · p. 7
  • TANDEM delegates unsupported stages to humans through magic operators while TAMP executes the remaining stages autonomously.The system was evaluated on five long-horizon manipulation tasks.Source: §VI · p. 7
  • TANDEM inherits underlying TAMP limitations, which remain the dominant source of demonstration failures; low-level human execution is delegated entirely to the teleoperator.Source: §VI · p. 7
  • Effects difficult to observe from one image challenge visual verification; evaluation covers one embodiment and one robot foundation model [1].Generalization to more complex embodiments and diverse robot foundation models remains open.Source: §VI · p. 8

APPENDIX

The appendix supplies task-evaluation criteria and additional materials on prompts and failure analyses. Its evaluation rubric scores rollouts by satisfied subgoals and requires all subgoals for success.

APPENDIX

  • The appendix adds task-evaluation criteria, VLM prompts, demonstration-collection failure details, and policy-execution failure details.Source: APPENDIX · p. 10

A. Task Evaluation Criteria

  • A rollout succeeds only when all task subgoals are satisfied; task progress is the fraction satisfied at termination.Mean progress and success rate are reported over rollouts.Source: A. Task Evaluation Criteria · p. 11
  • Table II presents task-specific evaluation subgoals and success criteria alongside each task’s language instruction to the policy.Source: Table II · p. 11

B. VLM Prompt Details

The VLM prompts translate language and workspace observations into an ordered plan of robot and human phases. They constrain robot actions, specify human operators and verifiable effects, and include checks and limited repair.

B. VLM Prompt Details

  • Detection translates the instruction into symbolic goal predicates and supplies detected object names to downstream stages, fixing the planner’s available object vocabulary.This stage uses the default TiPToP prompt [3].Source: B. VLM Prompt Details · p. 11
  • Phase segmentation turns the instruction into an ordered sequence of robot and human phases, specifying the associated TAMP or magic operators.Source: B. VLM Prompt Details · p. 11
  • Robot phases use On, Holding, and HandEmpty; On covers loose placement but not precise fitting, insertion, alignment, or soft-object manipulation.Source: B. VLM Prompt Details · pp. 11–12
  • Each human phase specifies instructions, an operator, preconditions, and add and delete effects; camera checks verify preconditions before execution and effects afterward.Effects must include the phase’s intended atoms, and delete effects remove states that no longer hold.Source: B. VLM Prompt Details · p. 12
  • The prompt assigns pick-and-place work to the robot and reserves human phases for actions it cannot perform, ordering phases so later steps remain physically possible.Intermediate states and fresh planning of each robot phase are allowed.Source: B. VLM Prompt Details · p. 12
  • The full instruction is planned clause by clause, with coverage assigned to phases or marked unrepresented when a clause cannot be expressed or its object is unavailable.Invented predicates need descriptions of what must be visible for a VLM to evaluate them.Source: B. VLM Prompt Details · p. 13
  • Rules require reachable preconditions, consistent effects, valid object names, and robot phases limited to supported predicates and complete pick-and-place actions.The prompt also prohibits consecutive robot phases from redundantly moving the same object.Source: B. VLM Prompt Details · p. 13
  • When a proposal fails, repair is appended for at most three attempts; predicate classification checks human-operator preconditions before and add and delete effects after execution.Source: B. VLM Prompt Details · p. 13

C. Demonstration Collection Failure Analysis

Across 130 TANDEM executions, 100 demonstrations were collected successfully and 30 failed. Most failures were attributed to TAMP execution rather than predicate or operator invention.

C. Demonstration Collection Failure Analysis

D. Downstream Policy Execution Failure Analysis

Failure-mode distributions differ across π0.5-DROID, HITL-TAMP, and TANDEM, with semantic errors leading for π0.5-DROID and TANDEM, and manipulation/placement leading for HITL-TAMP.

D. Downstream Policy Execution Failure Analysis

Loading 2609.28314v1…