Source-linked AI summary
Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs
Junhao Shi, Siyin Wang, Xiaopeng Yu, Li Ji, Jingjing Gong, Xipeng Qiu
TL;DR
VLA models are bottlenecked by the scarcity of costly expert demonstrations, while physical competence and semantic alignment require different supervision. TAP learns motor priors from unlabeled off-task and exploratory interactions using inverse dynamics, then grounds them in language with minimal expert data. It matches models trained on over 1M expert trajectories, gains 10% absolute over standard behavior cloning, and retains 15–25% success under camera perturbations where internet-scale baselines achieve 0%.
Problem
VLA learning is limited by the scarcity and cost of aligned observation-language-action expert demonstrations.
Method
TAP uses inverse dynamics to pretrain physical priors on task-agnostic interactions, then finetunes them with minimal language-annotated expert demonstrations.
Results
TAP matches models trained on 1M+ expert trajectories, gains 10% absolute over standard Behavior Cloning, and retains 15–25% success under camera perturbations versus 0% for internet-scale baselines.
Takeaways & Limitations
Task-agnostic interaction data provides a complementary, cost-effective foundation for transferable physical competence in robot learning.
Takeaways & Limitations
Approximately 75% of reported failures are semantic and reasoning failures, including distractor selection and freezing in longer-horizon sequences.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models are fundamentally bottlenecked by the scarcity of expert demonstrations -- triplets of observations, instructions, and actions that are costly to collect at scale. We argue that this bottleneck stems from conflating two distinct learning objectives: acquiring physical competence (how to move) and acquiring semantic alignment (what to do). Crucially, only the latter requires language supervision. Building on this Decomposition Hypothesis, we propose Task-Agnostic Pretraining (TAP), a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data -- including discarded off-task trajectories and autonomous robot play -- via a self-supervised Inverse Dynamics objective. A lightweight second stage then grounds these priors in language using minimal expert data. On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data, yielding a 10% absolute gain over standard behavior cloning. On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%, demonstrating that task-agnostic pretraining produces robust, transferable physical representations and offers a scalable path forward for Embodied AI.
1 Introduction
VLA training is constrained by the cost and limited scalability of language-annotated expert demonstrations. TAP separates learning physical competence from semantic alignment by using unlabeled interaction data for the former and minimal expert data for the latter.
- Expert teleoperation data is expensive, labor-intensive, and bottlenecked by human operators, limiting the scale and diversity of VLA training.
- Task-agnostic exploration provides abundant, inexpensive interaction trajectories without human supervision or task-specific annotations.
- TAP uses discarded off-task demonstrations and autonomous random play to learn physical knowledge with a self-supervised Inverse Dynamics objective.
- The method predicts actions connecting consecutive observations, encouraging representations focused on end-effector motion and object displacement rather than static background noise.
- On Simpler and real-world WidowX experiments, TAP improves sample efficiency and generalization while reducing dependence on expensive expert teleoperation.
2 Related Works
Prior VLA work improved general-purpose robotic control through larger, more diverse models and datasets, but remained constrained by the cost of scaling expert demonstrations. TAP instead treats inverse dynamics as a standalone pretraining phase for task-agnostic physical priors.
- Earlier visuomotor policies often struggled to generalize beyond constrained environments, novel objects, and unstructured language instructions.
- Generalist VLA systems unified perception, language, and action while scaling data diversity and model backbones to improve dexterity and generalization.
- Current state-of-the-art VLAs remain bounded by the prohibitive cost of scaling expert human demonstrations.
- Dynamics-learning methods commonly use forward or inverse objectives for visual pretraining, auxiliary learning, or pseudo-labeling.
- TAP differs by using inverse dynamics as a standalone pretraining phase to learn physical priors before task semantics, improving downstream learning efficiency and performance.
3 Task-Agnostic Data for Physical Grounding
TAP operationalizes the Decomposition Hypothesis in two stages: it learns physical priors from unlabeled trajectories, then aligns them with language using a small expert dataset. The framework combines diverse task-agnostic data sources with inverse-dynamics pretraining and behavior-cloning finetuning.
- Task-Agnostic Data for Physical Grounding: The Decomposition Hypothesis factorizes action generation into physical-affordance perception (“how to move”) and semantic-intent grounding (“what to do”).
- Task-Agnostic Data for Physical Grounding: Task-agnostic trajectories lack language instructions but contain valid physical interactions and object responses.
- Task-Agnostic Data for Physical Grounding: TAP uses off-task trajectories, autonomous random play, and existing heterogeneous datasets while discarding original task labels.
- Task-Agnostic Data for Physical Grounding: Autonomous collection constructs safe waypoint trajectories from filtered teleoperation poses, stochastic sampling, contact heuristics, interpolation, and boundary-aware noise.
- Stage 1: Task-Agnostic Pretraining: Stage 1 predicts the action causing a transition between observations, using inverse dynamics to emphasize motion and object displacement over static appearance.
- Stage 1: Task-Agnostic Pretraining: The visual encoder maps observations to token sequences, and training minimizes mean squared error between predicted and ground-truth actions.
- Stage 1: Task-Agnostic Pretraining: After pretraining, the model acquires spatial reasoning, affordance detection, and motor coordination without language supervision.
- Stage 2: Task-Specific Alignment: Stage 2 reuses the pretrained backbone and action head, maps language instructions into semantic conditioning, and finetunes with behavior cloning on expert demonstrations.
4 Experiments
The experiments test whether task-agnostic pretraining improves effectiveness, physical competence, robustness, learning dynamics, scaling, and cross-domain transfer. Across simulation and real-world evaluations, TAP improves performance with limited expert data and supports physical representations that generalize beyond training conditions.
- Effectiveness and Efficiency: TAP-20k achieves 33.32% Avg-All success, exceeding Standard BC by 10% absolute and outperforming OpenVLA at 7.75% and RT-1-X at 3.03%.Performance rises monotonically from 24.47% at 8k episodes to 30.2% at 14k and 33.32% at 20k.
- Mechanism via Partial Success Analysis: 45.82% Avg-Partial success matches Octo’s 42.30% and approaches π0’s 53.10%, indicating strong low-level grasping and contact competence without language supervision during Stage 1.Partial success measures physical sub-goals, whereas Entire success additionally requires full task completion and semantic execution.
- Robustness from Self-Exploration: 65% success under unseen-fruit clutter exceeds Standard BC’s 5% and NORA’s 60%, showing TAP’s stronger adaptability to visual distractors.The authors attribute this robustness to attention on causal gripper–object dynamics rather than static background correlations.
- How Task-Agnostic Data Shapes Learning: Pretrained models continue beyond the Baseline’s roughly 23% plateau to exceed 30%, despite comparable initial learning rates during Stage 2 finetuning.The reported pattern suggests pretraining raises the achievable performance ceiling rather than accelerating semantic acquisition.
- How Task-Agnostic Data Shapes Learning: Scaling Stage 1 from 20k to 100k steps changes performance from stagnating near 18% to exceeding 30%, while longer Stage 2 finetuning alone yields diminishing returns.The heatmap indicates that pretraining scale bounds downstream performance and that task-agnostic data regularizes limited expert finetuning.
- How Task-Agnostic Data Shapes Learning: Without language input, Stage 1 attention concentrates on grippers and nearby objects across simulation and real-world settings, while Stage 2 refines attention toward the gripper.The authors connect this pattern to inverse dynamics training on observation pairs and interpret the cross-domain consistency as evidence of transferable physical representations.
- Failure Analysis: Approximately 75% of failures are semantic or reasoning failures, while approximately 25% involve execution and dynamics.Execution failures include slipping, millimetric pre-grasp misalignment, and depth ambiguity; semantic failures include selecting distractors or losing the instruction over longer horizons.
5 Conclusion
TAP decouples physical affordance learning from semantic task understanding, using cheap unlabeled interactions for “how to move” and expert demonstrations for “what to do.” It improves sample efficiency, transfers motor competence across tasks, and increases robustness under visual perturbations.
- TAP decouples physical affordance learning from semantic task understanding in VLA models.
- 10% absolute gain over standard Behavior Cloning, matching models trained on 1M+ expert trajectories with orders of magnitude less labeled data.
- Partial success increased from 31.8% to 45.8%, supporting transfer of generalizable motor competencies rather than task-specific behaviors.
- 15–25% success under camera perturbations contrasted with 0% in internet-scale baselines, indicating greater robustness from diverse physical experience.
- Task-agnostic interaction provides a complementary, cost-effective foundation for robot learning beyond scaling expert data alone.
A Details of Autonomous Random Play Data Collection
The autonomous random-play pipeline combines a verified safe workspace with constrained, contact-rich trajectory generation. Temporal downsampling makes action-caused visual changes perceptible for inverse-dynamics training.
- A two-phase constrained procedural framework generates safe, contact-rich autonomous interactions instead of redundant free-space motions.
- Safe workspace initialization uses operator teleoperation, unsafe-pose filtering, and voxel-grid downsampling to build a uniform pose library.
- Constrained waypoint sampling and contact-forcing adjustments generate pushing and sliding interactions while preventing end-effector hovering.
- Cosine interpolation and boundary-aware Gaussian noise diversify trajectories, while human intervention is limited to periodic object resets.
- 5Hz downsampling makes visual displacement perceptible and causally attributable to executed actions for inverse dynamics.
B Training and Evaluation Details
The implementation uses a Qwen2.5-VL backbone with a lightweight action head and trains in two stages. Relative delta-pose actions encode local end-effector motion for transferable interaction dynamics.
- Qwen2.5-VL with 3B parameters serves as the VLM backbone, paired with a ViT-based SigLIP visual encoder and a 2-layer MLP action head.
- Stage 1 freezes the visual encoder while training the VLM backbone and action head; Stage 2 jointly finetunes all parameters.
- The 7-dimensional delta-pose action space encodes relative position, axis-angle orientation, and a scalar gripper command.
- Relative motion rather than absolute pose supports learning local interaction dynamics invariant to global workspace coordinates.
B.3 Training Details.
Training runs for 100,000 steps on eight H100 GPUs with distributed execution, AdamW optimization, cosine learning-rate decay, warmup, gradient clipping, and bfloat16 precision.
- 100,000 training steps run on a single node equipped with 8 NVIDIA H100 GPUs.
- Global batch size is 128, with 16 samples per GPU, using AdamW with weight decay 0.05 and β1 = 0.9, β2 = 0.999.
- The learning rate starts at 5 × 10^-5 and follows cosine decay with a 0.05 warmup ratio over the first 5,000 steps.
- Global gradient clipping uses a maximum norm of 1.0, and training runs use bfloat16 precision for efficiency and numerical stability.
B.4 Evaluation Details
The evaluation combines standardized SIMPLER testing with real-world manipulation experiments designed to probe generalization across task conditions. It also compares data requirements and computational resources against large-scale VLA baselines.
- SIMPLER Evaluation: SIMPLER success rates are averaged over 50 episodes per checkpoint using matched Bridge data and training settings.
- Real-World Evaluation: Real-world evaluation covers “Put Carrot on Plate” and “Push Pumpkin to Left” across five scenarios testing generalization.The scenarios are in-domain setups, initial-state perturbations, visual distractors, background texture shifts, and viewpoint variations.
- Real-World Evaluation: Each real-world task uses 20 trials per scenario, with overall success averaged arithmetically across all five scenarios.
- Generalization Conditions: The scenarios test recovery from unseen starting poses, robustness to clutter, invariance to tablecloth textures, and tolerance to camera-extrinsic shifts.Distractors do not obstruct the manipulation trajectory, while viewpoint changes preserve the main visual semantics.
- Resource Comparison: Table 4 compares TAP with leading VLA baselines on expert data, cheap autonomous data, and computational resources.The comparison targets dependence on massive curated datasets and industrial-scale training infrastructure.
C.1 Data Comparison: Expert vs. Task-Agnostic
Large-scale VLA baselines rely on millions of curated expert trajectories, whereas TAP reduces expert-data dependence by using task-agnostic interaction data and lightweight finetuning. The paper also presents TAP as substantially more accessible in computational requirements.
- Expert Data: OpenVLA, Octo, and NORA rely on OXE, which aggregates over 2 million expert trajectories across varying embodiments.Curating and standardizing these datasets requires immense human effort.
- Task-Agnostic Data: TAP minimizes reliance on expert data by separating task-agnostic pretraining from expert-data finetuning.
- Task-Agnostic Data: Stage 1 uses purely autonomous, task-agnostic interaction data such as random exploration or play without human labeling costs.
- Expert Data: 200 trajectories support competitive real-world performance during Stage 2 finetuning, compared with the millions of demonstrations used by baselines.
- Results: TAP reduces expert-data demand by several orders of magnitude while maintaining comparable manipulation proficiency.
- Compute Requirements: TAP training can run on a single node with 8×H100 GPUs in approximately 24 GPU hours, rather than weeks on industrial-scale infrastructure.
D.1 Simulation: Unlocking Task-Irrelevant Data in SIMPLER
Qualitative comparisons illustrate the paper’s decomposition claim in simulation and real-world settings. TAP and NORA execute manipulation successfully where Standard BC or visual grounding failures prevent reliable contact.
- Simulation: On SIMPLER’s “Put the carrot on the plate” task, Standard BC fails at the final grasp while NORA and TAP successfully complete the manipulation.
- Simulation: The comparison supports extracting physical manipulation affordances from task-agnostic data as part of the Decomposition Hypothesis.
- Simulation: With scarce expert data, Standard BC reaches the carrot’s vicinity but fails to execute the fine-grained contact dynamics needed for grasping.
- Real World: On the real-world pumpkin-pushing task under an unseen background texture, Standard BC fails to make proper sustained contact and NORA misses the object after misjudging its location.
- Real World: NORA attempts the semantically correct leftward trajectory but suffers a visual grounding failure against the novel table texture.
E Full Experimental Results
The paper provides a comprehensive scaling-results breakdown across combinations of Stage 1 pretraining and Stage 2 finetuning steps. Reported task columns include multiple manipulation tasks and aggregate metrics.
- Results Presentation: The detailed results are presented as a continuation table covering individual evaluation checkpoints and subtasks.
- Scaling Results: Table 5 reports success rates across all combinations of Stage 1 and Stage 2 training steps.Training steps are reported in thousands.
- Task Breakdown: The reported task columns include Spoon on cloth, Carrot on plate, Stack Blocks, and Eggplant in Basket.
- Metrics: Table 5 includes Avg-P, Avg-E, Avg-All, and partial- versus entire-success metrics for the listed tasks.