PaperlayerView exact revision on arXiv

Source-linked AI summary

DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Models

Samrat Sahoo, Yixuan Huang, Tom Silver

arXiv:2609.12316v1cs.RO

TL;DR

Raw TAMP demonstrations solve tasks but provide little benefit when fine-tuning a pretrained VLA, which the authors hypothesize reflects a mismatch with pretraining behavior. DATAFARM incorporates that distribution into TAMP trajectory generation. It raises average success to 56.7% versus 8.3% for raw TAMP, approaching human teleoperation, while retaining performance on cloth folding beyond TAMP’s scope.

  • Problem

    Successful TAMP trajectories provide little benefit for fine-tuning a pretrained VLA, and the authors hypothesize that planner behavior differs from pretraining data.

    Source: §I · p. 1

  • Method

    DATAFARM incorporates pretraining-distribution alignment in joint configurations, motion style, and execution timing into TAMP trajectory generation.

    Source: §I · p. 2

  • Results

    56.7% average success with DATAFARM compares with 8.3% for raw TAMP and 61.7% for human teleoperation; cloth-folding success is 85% for the fine-tuned model versus 90% for the pretrained model.

    Source: §I · p. 2

  • Takeaways & Limitations

    Distribution-aligned planner demonstrations support VLA fine-tuning on TAMP-solvable tasks while largely preserving performance on a task beyond the planner’s scope.

    Source: §I · p. 2

    The evaluation covers one pretrained VLA checkpoint, one robot embodiment, three target tasks, and one retained capability; broader coverage is needed to establish generality.

    Source: §VI · p. 7

Abstract

from arXiv · show

Collecting high-quality robot data remains a fundamental challenge for training robot foundation models. Task and motion planning (TAMP) offers a scalable way to generate demonstrations, but our experiments show that raw TAMP trajectories provide surprisingly little benefit when used to fine-tune pretrained vision-language-action (VLA) models, despite successfully solving the target tasks. We hypothesize that this failure arises from a behavioral distribution mismatch between planner-generated trajectories and the data used to pretrain the VLA. To address this mismatch, we introduce DATAFARM: Distribution-Aligned Task And motion planning for Fine-tuning A Robot foundation Model, an approach that incorporates the pretraining distribution directly into TAMP trajectory generation. DATAFARM aligns generated trajectories with the pretraining data in robot joint configurations, motion style, and temporal execution profiles. We evaluate DATAFARM on three tabletop manipulation tasks that TAMP can perform and a cloth-folding task beyond the capability of TAMP. DATAFARM achieves an average success rate of 56.7%, substantially outperforming raw TAMP (8.3%) while approaching human teleoperation (61.7%). On Deformable Object Manipulation, which is outside the fine-tuning distribution, the fine-tuned model retains 85% success, compared with 90% for the pretrained model. These results show that aligning planner-generated demonstrations with the pretraining distribution can make TAMP an effective source of data for VLA fine-tuning. Website and code: https://prpl-group.com/datafarm/

I. INTRODUCTION

The paper argues that successful TAMP trajectories transfer poorly to pretrained VLAs when their behavior differs from pretraining data, and introduces DATAFARM to align planner-generated demonstrations with that data. Its evaluations compare fine-tuning outcomes on planner-solvable tasks and retention of a capability beyond TAMP’s scope.

I. INTRODUCTION

  • Raw TiPToP demonstrations raise average success only from 1.7% to 8.3%, despite solving the tasks; the authors hypothesize a mismatch in joint configurations, motion style, or execution speed.The paper frames task-validity as insufficient to ensure effective fine-tuning data for a pretrained VLA.Source: §I · p. 1
  • DATAFARM incorporates pretraining distributions into TAMP objectives for joint configurations, motion style, and timing while retaining task and motion constraints.A configuration model guides endpoint selection, while a frozen trajectory encoder and fitted distribution guide motion style and timing.Source: §I · p. 2
  • 56.7% average success with DATAFARM approaches human teleoperation’s 61.7%, versus 8.3% for raw TAMP; the fine-tuned model retains 85% success on cloth folding.The pretrained model achieves 90% on cloth folding, which is beyond TAMP’s scope and was absent from fine-tuning data.Source: §I · p. 2

A. Planning-Based Demonstration Generation

  • Prior work used TAMP to train visuomotor policies, whereas DATAFARM uses planner-generated fine-tuning data to teach new tasks to an already pretrained VLA while retaining existing capabilities beyond the planner’s scope.HITL-TAMP and SkillMimicGen both retain planning as part of the deployed controller.Source: A. Planning-Based Demonstration… · p. 2

B. Demonstration Collection and Augmentation

  • Demonstration augmentation can adapt examples to new object configurations or geometry and pose variation, whereas DATAFARM uses pretraining demonstrations as a behavior prior without target-task examples.The planner supplies solutions to the new tasks while reference demonstrations supply the behavior prior.Source: B. Demonstration Collection and… · p. 2

C. Synthetic Data for VLA Fine-Tuning

III. PROBLEM SETUP

  • The problem setup seeks autonomously generated and executed demonstrations for target tasks under a data budget, then evaluates both target-task success and retained capabilities outside those tasks.Human help is permitted for environment setup and resets; reference demonstrations need not cover the target tasks.Source: §III · pp. 2–3

IV. DISTRIBUTION-ALIGNED TAMP

DATAFARM adds reference-derived behavior costs to TiPToP’s planning pipeline. The underlying planner enumerates task plans, chooses feasible grasps, placements, and robot configurations, then optimizes connecting motions while respecting constraints.

IV. DISTRIBUTION-ALIGNED TAMP

  • DATAFARM augments TAMP with objectives derived from reference demonstrations to generate fine-tuning data.The paper first reviews TiPToP’s pipeline, then describes how the objectives are constructed and incorporated.Source: §IV · p. 3

A. Preliminaries

  • TiPToP enumerates symbolic plan skeletons, samples grasps and placements, and uses inverse kinematics to obtain candidate robot configurations.The planner refines placements and configurations subject to collision avoidance, placement stability, and kinematic constraints.Source: A. Preliminaries · p. 3
  • For fixed endpoints and sample times, trajectory optimization updates intermediate waypoints to minimize a planning objective; candidate motions are checked for feasibility.The geometric path and its time parameterization are distinct: retiming changes traversal speed while keeping path geometry fixed.Source: A. Preliminaries · pp. 3–4

B. Method Overview

DATAFARM learns fixed reference models before planning: a configuration density and a latent density over encoded trajectories. Their costs guide endpoint selection, connecting-path optimization, and retiming.

B. Method Overview

  • A configuration cost from a Gaussian mixture scores individual robot configurations, while a trajectory cost uses the squared Mahalanobis distance of an encoded segment from the reference latent distribution.The trajectory cost covers a segment’s temporal properties as well as its motion.Source: B. Method Overview · p. 4

C. Learning Reference Models

  • The configuration density is fit to moving-arm frames, so long pauses do not concentrate it on idle configurations; its cost favors configurations common during reference motions.It does not identify whether a configuration accomplishes a task; IK and feasibility checks provide that constraint.Source: C. Learning Reference Models · p. 4
  • The trajectory encoder receives resampled joint positions, velocities, accelerations, and jerks, with temporal crops exposing it to partial motions of different lengths.The physical sampling interval is fixed so the representation changes when the same path is executed at a different speed.Source: C. Learning Reference Models · p. 4
  • An auxiliary prediction objective trains the encoder to capture motion descriptors, and a variational penalty regularizes its latent distribution; the auxiliary head and covariance output are discarded before planning.The latent reference density is fit to embeddings of reference trajectories and their crops, and the encoder and reference densities remain fixed during planning.Source: C. Learning Reference Models · p. 4

D. Distribution-Aligned Planning

DATAFARM applies its learned reference costs at endpoint selection, path optimization, and retiming. The planner preserves its feasibility checks while aligning candidate configurations and motions with reference behavior.

D. Distribution-Aligned Planning

  • Endpoint selection ranks inverse-kinematics solutions for each end-effector pose by configuration cost, favoring arm postures with higher reference density.Selected configurations become endpoints for path optimization, subject to grasp, placement, and feasibility checks.Source: D. Distribution-Aligned Planning · p. 4
  • Path optimization adds the trajectory cost to the planner objective, allowing it to influence motion between fixed endpoints, including waypoint-induced velocities and higher derivatives.The planner retains its constraint penalties and checks resulting motions for feasibility.Source: D. Distribution-Aligned Planning · p. 5
  • Retiming optimizes traversal speed while keeping path geometry fixed, using interval duration fractions and an overall duration to parameterize the time scaling.The same trajectory cost guides retiming, and candidate motions are checked against robot motion limits before acceptance.Source: D. Distribution-Aligned Planning · p. 5

V. EXPERIMENTS

The experiments assess how demonstration source, DATAFARM components, task-specific fine-tuning, and demonstration budgets relate to target-task performance and capability retention. The setup uses separate task-specific checkpoints and evaluates both task completion and partial progress.

V. EXPERIMENTS

  • The experiments ask how demonstration source, alignment components, fine-tuning effects on Deformable Object Manipulation, and tested demonstration budgets relate to performance.These are the four evaluation questions posed for the experiments.Source: §V · p. 5
  • Path optimization and retiming are performed in separate stages because the authors found this worked better in practice.Source: §V · p. 5

A. Experimental Setup

  • Three tabletop tasks each require three object moves, while cloth folding tests retention on a planner-out-of-scope task absent from planner-generated data.The tabletop tasks test geometric constraints, multi-step reasoning, and semantic reasoning; Figure 3 shows task-specific evaluation alongside the OOD evaluation.Source: A. Experimental Setup · p. 5, Figure 3 · p. 5
  • Raw TAMP fine-tunes the pretrained VLA on successful unmodified TiPToP demonstrations; DATAFARM uses aligned demonstrations, with ablations removing one alignment component each.Human teleoperation fine-tunes on human-controlled target-task demonstrations; the table caption defines the oracle by access to these demonstrations, not as an upper bound.Source: A. Experimental Setup · p. 5, Table I · p. 6
  • For each target task and approach, a separate checkpoint is trained from 20 successful demonstrations and evaluated over 20 fresh-scene trials.Demonstrations are not pooled across target tasks, and the pretrained checkpoint receives no target-task demonstrations.Source: A. Experimental Setup · p. 6
  • Success requires completing every evaluation subgoal, while task progress scores the fraction completed at termination, normalized to 100% across 20 trials.Each target task has seven subgoals and Deformable Object Manipulation has two; subgoals score rollouts but are not training labels or policy inputs.Source: A. Experimental Setup · p. 6

B. Experimental Results

Across three tabletop tasks, DATAFARM approaches human teleoperation and substantially outperforms raw TAMP; all three alignment components contribute, while the model largely retains performance on an out-of-scope deformable task.

B. Experimental Results

  • Across the three target tasks, DATAFARM succeeds on 65%, 35%, and 70% of trials, versus 5%, 5%, and 15% for raw TAMP.The raw TAMP demonstrations were successful planner executions, despite their limited downstream benefit.Source: B. Experimental Results · p. 6
  • DATAFARM samples fall inside the displayed DROID contour at 74% in joint space and 96% in trajectory latent space.The corresponding raw TAMP overlaps are 66% and 0%; these projections diagnose overlap in displayed representations, not equality with the full pretraining distribution.Source: B. Experimental Results · p. 6, Figure 4 · p. 7
  • Removing joint-space alignment lowers average success to 23.3%, removing style alignment to 15.0%, and removing timing alignment yields 0% across the tasks.The style ablation retains 89% projected latent overlap with DROID, so high overlap in that projection alone does not ensure task success.Source: B. Experimental Results · p. 6
  • On Geometric Constraints, success reaches 65% with 20 DATAFARM demonstrations and 85% with 80, after plateauing at 75% for 40 and 60.The pretrained checkpoint starts at 0% success and 18% progress; with 80 demonstrations, progress reaches 97%.Source: B. Experimental Results · p. 7, Table II · p. 7
  • 56.7% average success with DATAFARM approaches human demonstrations’ 61.7% and exceeds raw TAMP’s 8.3%.Average task progress is 84.7%, versus 43.3% for raw TAMP and 27.0% for π0.5-DROID.Source: B. Experimental Results · p. 6, §VI · p. 7
  • Fine-tuned policies retain 85% average success on Deformable Object Manipulation, compared with 90% for the pretrained policy, despite no fine-tuning demonstrations of that task.That task is beyond TiPToP’s capability.Source: §VI · p. 7, B. Experimental Results · p. 7

VI. CONCLUSION

  • Evaluation covers one pretrained VLA checkpoint, one robot embodiment, three target tasks, and one retained capability; alignment also requires representative pretraining-distribution samples.The implementation focuses on quasi-static pick-and-place manipulation.Source: §VI · p. 7

APPENDIX

The appendix documents the trajectory encoder inputs, motion descriptors, and architecture used for trajectory-style alignment. Its encoder combines spectral features with a branch that captures motion ordering.

APPENDIX

  • The appendix also documents the trajectory latent space, endpoint selection, path optimization, retiming, hyperparameters, task subgoals, and failure breakdown.It points to separate sections for the latent space and the planner’s alignment objectives.Source: APPENDIX · p. 10

1) Trajectory inputs:

  • At 15 Hz, each trajectory is represented by joint positions, velocities, accelerations, and jerks, then standardized using fixed per-channel training statistics.The input has 28 channels across seven joints.Source: 1) Trajectory inputs · p. 11

2) Motion descriptors:

  • The 113 motion descriptors comprise 14 per-joint descriptors for each of seven joints and 15 whole-trajectory descriptors.The whole-trajectory descriptors include spectral measures of the speed signal, with low, mid, and high bands spanning 0–7.5 Hz.Source: 2) Motion descriptors · p. 11, Table III · p. 11

3) Encoder architecture:

  • The spectral branch captures behavior at multiple temporal scales and pools activations into a 384-dimensional summary, while the temporal branch captures motion ordering.The example ordering is acceleration during free-space transit followed by deceleration on approach to an object.Source: 3) Encoder architecture · p. 11

4) Training details:

The trajectory encoder is trained with descriptor prediction and Gaussian KL regularization, then used to fit a latent reference density. The section also specifies the configuration density used to score joint configurations.

4) Training details:

  • The encoder combines descriptor prediction with Gaussian KL regularization; planning uses its mean embedding, and training includes random trajectory crops.The crops expose the encoder to partial motions of different lengths, matching the segments queried during planning; the frozen encoder’s embeddings and crops are used to fit the latent reference mean and covariance.Source: 4) Training details · pp. 11–12

1) Fitting the configuration density:

  • The configuration density is a 16-component full-covariance Gaussian mixture over seven joint angles, fitted to 987,631 frames from 4,000 reference episodes.Its cost is the negative log density of a configuration.Source: 1) Fitting the configuration density · p. 13

2) IK candidate generation and grasp symmetry:

The section describes selecting among inverse-kinematics solutions, including equivalent grasp orientations, and details the evaluation and gradient-computation pipeline. Including rotated grasps reduces the share of selected endpoint configurations in the extreme tail of reference configuration cost in both scenes.

2) IK candidate generation and grasp symmetry:

  • For candidate end-effector poses, the planner searches from multiple initial guesses for IK solutions and considers both grasp orientations when the grasp model admits the π-rotated alternative.Candidates failing a forward-kinematics check are discarded; if none remain, the planner falls back to its baseline IK procedure.Source: 2) IK candidate generation and grasp… · p. 13

3) Effect of including the rotated grasp:

1) Evaluation pipeline:

  • Trajectory cost evaluation resamples an optimized segment at 15 Hz, standardizes its encoder input, computes its mean embedding, and evaluates squared Mahalanobis distance.The planner weights this cost by λpath.Source: 1) Evaluation pipeline · p. 14

2) Gradient computation and caching:

  • During path optimization, gradients pass through the frozen encoder and preprocessing, while a constant preprocessing Jacobian is precomputed and reused across solver iterations.The sample times or waypoint count must change for the preprocessing operator to require updating; only interior waypoint configurations are optimized.Source: 2) Gradient computation and caching · p. 14

D. Retiming

Retiming changes traversal speed while preserving the fixed path and endpoint configurations. The section bounds interval durations, penalizes segment-boundary speeds relative to reference gripper-event speeds, and selects feasible low-cost candidates.

1) Time-scaling implementation:

  • Tanh-before-softmax time fractions cap the ratio between interval durations at e^2, while retiming preserves the path and endpoint configurations, including gripper-event endpoints.The implementation uses 64 equal arc-length intervals in the time parameterization.Source: 1) Time-scaling implementation · p. 14

2) Boundary-speed penalty:

  • Without the boundary term, retiming produces boundary speeds of 0.30–0.37 rad/s, versus a reference target of 0.07 rad/s around gripper events.The median speed at the first and last frames of reference windows is 0.296 rad/s, compared with 0.335 rad/s in their interiors.Source: 2) Boundary-speed penalty · p. 15

3) Optimization and candidate selection:

  • The optimizer evaluates 16 initial duration candidates in parallel, rejects candidates exceeding velocity or acceleration limits, and returns the survivor with the lowest latent trajectory cost.The boundary penalty contributes during optimization, but final ranking uses latent trajectory cost.Source: 3) Optimization and candidate selection · p. 15

E. Planner and Fine-Tuning Hyperparameters

The section records planner and fine-tuning settings, defines rollout success and progress using task subgoals, and categorizes failures among unsuccessful rollouts.

E. Planner and Fine-Tuning Hyperparameters

  • The planner uses 12 IK solutions per candidate pose, while VLA fine-tuning uses AdamW with a 2.5 × 10−5 peak learning rate and a 1,000-step warmup followed by cosine decay to 2.5 × 10−6.Other planner settings use the base TAMP system defaults; fine-tuning hyperparameters are shared across data sources.Source: E. Planner and Fine-Tuning… · p. 15, Table VIII · p. 15, Table VII · p. 15

F. Task Evaluation Subgoals

  • A rollout succeeds only when it meets every task subgoal; task progress is the fraction of subgoals met at termination, with mean progress and success rate reported across rollouts.The task-specific subgoals and success criteria are listed in Table IX.Source: F. Task Evaluation Subgoals · p. 15, Table IX · p. 16

G. Failure Breakdown

  • Failures are grouped as grasping, manipulation/placement, joint-space misalignment, semantic, stuck, or did-not-finish outcomes.Table X gives failure counts by approach and task, marking the most common mode and runner-up in each row.Source: G. Failure Breakdown · pp. 15–16, Table X · p. 17
Loading 2609.12316v1…