PaperlayerView exact revision on arXiv

Source-linked AI summary

LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer

Lihan Zha, Asher J. Hancock, Mingtong Zhang, Tenny Yin, Yixuan Huang, Dhruv Shah, Allen Z. Ren, Anirudha Majumdar

arXiv:2602.10556v2cs.ROcs.AI

TL;DR

Robot foundation policies seek zero-shot deployment across unseen embodiments, but existing VLAs remain tied to training robots and often need costly adaptation. LAP represents low-level actions in natural language and combines this supervision with an efficient action expert. LAP-3B achieves substantial zero-shot transfer on unseen robots, including over 50% average success and roughly 2× improvement over the strongest prior VLAs.

  • Problem

    Existing VLAs rarely operate zero-shot on new robot embodiments and typically require costly per-embodiment adaptation despite multi-embodiment training.

  • Method

    LAP trains a VLM to predict low-level robot actions described in natural language and combines the backbone with a lightweight diffusion-based action expert for efficient control.

  • Results

    LAP-3B achieves over 50% average success across three unseen embodiments, approximately doubling the strongest baseline, with roughly 30% absolute improvement over prior action representations.

  • Takeaways & Limitations

    Language-action representations support substantial zero-shot embodiment transfer while also enabling favorable scaling and efficient fine-tuning.

  • Takeaways & Limitations

    The evaluation focuses on zero-shot transfer across single-arm manipulators and has not yet covered regimes requiring substantially higher control.

Abstract

from arXiv · show

A long-standing goal in robotics is a generalist policy that can be deployed zero-shot on new robot embodiments without per-embodiment adaptation. Despite large-scale multi-embodiment pre-training, existing Vision-Language-Action models (VLAs) remain tightly coupled to their training embodiments and typically require costly fine-tuning. We introduce Language-Action Pre-training (LAP), a simple recipe that represents low-level robot actions directly in natural language, aligning action supervision with the pre-trained vision-language model's input-output distribution. LAP requires no learned tokenizer, no costly annotation, and no embodiment-specific architectural design. Based on LAP, we present LAP-3B, which to the best of our knowledge is the first VLA to achieve substantial zero-shot transfer to previously unseen robot embodiments without any embodiment-specific fine-tuning. Across multiple novel robots and manipulation tasks, LAP-3B attains over 50% average zero-shot success, delivering roughly a 2x improvement over the strongest prior VLAs. We further show that LAP enables efficient adaptation and favorable scaling, while unifying action prediction and VQA in a shared language-action format that yields additional gains through co-training.

1 Introduction

Existing VLAs struggle to transfer zero-shot across robot embodiments because action learning can mismatch pre-trained VLM representations. LAP addresses this by expressing low-level actions in natural language and pairing the resulting language-supervised backbone with an efficient action expert.

  • Existing VLAs rarely transfer zero-shot to new robots, so deployment commonly requires costly per-embodiment adaptation.
  • Zero-shot transfer is difficult because embodiment diversity is expensive to collect and hardware coverage remains limited.
  • Standard VLA training can create distributional mismatch because VLMs are not pretrained for motor-level, high-frequency control signals.
  • LAP represents end-effector actions as natural-language descriptions that align motor supervision with the VLM’s pretrained modality.
  • LAP-3B combines a LAP-trained VLM backbone with a lightweight diffusion-based action expert for efficient continuous control.
  • LAP-3B achieves roughly a 2× relative improvement, approximately 30% absolute, over prior action representations while supporting efficient adaptation and favorable scaling.

2 Related Work

Prior work addresses heterogeneous robot action spaces through unified representations, embodiment conditioning, or alternative action encodings. LAP instead targets zero-shot transfer through language-action representations and large-scale pre-training without explicit embodiment-specific design.

  • VLAs are generally fine-tuned from internet-pretrained VLMs and trained on large, diverse robotic datasets.
  • Cross-embodiment policies commonly unify varying state and action spaces through padding, shared end-effector spaces, projectors, or separate action heads.
  • Other approaches condition policies on embodiment features, including kinematic structure, structural similarity, soft prompts, or explicit embodiment-specific training.
  • LAP-3B pursues zero-shot transfer without explicit embodiment conditioning, matched hardware interfaces, or embodiment-aligned data-collection assumptions.
  • Unlike coarse language-level sub-tasks, LAP uses fine-grained language-actions parsed from continuous end-effector motion for motor-level control.
  • LAP differs from prior natural-language or raw-string action methods by providing a practical real-time VLA and demonstrating scaling to large robot pre-training corpora.

3 LAP: Language-Action Pre-Training for VLAs

LAP reformulates robot action prediction as natural-language modeling and combines that supervision with a lightweight continuous-control expert in LAP-3B. The resulting design derives language-actions from existing trajectories without manual annotation and supports efficient real-time inference.

  • LAP formulation: LAP trains a VLM to predict structured language-actions that summarize continuous end-effector action chunks from observations and task instructions.The language-action sequence is derived deterministically and learned with standard autoregressive cross-entropy language modeling.
  • Dataset curation: LAP obtains language-action supervision directly from end-effector trajectories rather than manual annotation.This makes the representation available at scale from structured robotic datasets.
  • Language-action representation: Language-actions describe end-effector delta motions using fixed coordinate conventions and structured templates, with arbitrary numerical resolution.The templates cover translational and rotational motions, while dataset trajectories provide the supervision without additional annotation.
  • Temporal abstraction: Each language-action summarizes a chunk’s net displacement, producing lower-frequency supervision that abstracts away high-frequency control.The representation uses cumulative translation and rotation over the action chunk and discretizes magnitudes into integer values.
  • Model architecture: LAP-3B combines a LAP-trained VLM backbone with a lightweight flow-matching action expert to retain language-action supervision while enabling continuous control.The VLM predicts structured language-actions, while the action expert predicts continuous action chunks under a separate flow-matching objective.
  • Model architecture: At inference, only the action expert is rolled out, enabling real-time execution at 25 Hz on an NVIDIA RTX 4090 GPU.The VLM and action expert communicate through cross-attention, with gradient blocking used to preserve pretrained VLM representations and stabilize joint training.

4 Experiments

The experiments test whether LAP enables zero-shot transfer, efficient adaptation, transferable representations, VQA co-training gains, and favorable scaling across embodiments and tasks. LAP-3B shows strong unseen-embodiment performance, faster fine-tuning, lower unseen-embodiment prediction error, additional co-training benefits, and improved scaling behavior.

  • Zero-Shot Cross-Embodiment Generalization: LAP-3B matches π0.5-DROID on the seen embodiment while outperforming replicated baselines by approximately 15 percentage points across tasks without embodiment-specific DROID fine-tuning.
  • Zero-Shot Cross-Embodiment Generalization: Across three unseen embodiments and six real-world tasks, LAP-3B exceeds 50% average zero-shot success and improves approximately 2× over the strongest baseline.
  • Fine-Tuning Efficiency on New Embodiments: LAP-3B reaches 96.8% near-maximum LIBERO performance within six epochs and achieves 78% success after one epoch, converging faster than baselines.
  • Fine-Tuning Efficiency on New Embodiments: On real-robot adaptation, LAP-3B reaches approximately 50% YAM task progress using 20 demonstrations, roughly 2.5× fewer than baselines, and consistently outperforms replicated baselines on Franka.
  • Analyzing LAP’s Cross-Embodiment Generalization: LAP-3B has lower unseen-embodiment action prediction error with smoother training dynamics, while its learned embodiment representations align training and unseen robots more closely.
  • VQA Co-Training: Motion-prediction VQA co-training improves action precision, spatial generalization, adaptation speed, and final convergence performance across embodiments.
  • Scaling with Model Size: LAP maintains lower validation loss across model sizes and improves monotonically with capacity, whereas π0.5-replicated saturates early and degrades at larger scales.

5 Conclusions and Discussions

LAP is a scalable VLA pre-training recipe that predicts actions in natural language, and LAP-3B transfers substantially to unseen embodiments without fine-tuning. The paper also identifies strong training efficiency while limiting evaluation to single-arm robots and moderately dexterous tasks.

  • LAP predicts robot actions in natural language, producing representations that transfer effectively across robot embodiments.
  • LAP-3B achieves roughly a 30% absolute improvement over prior action representations and substantial zero-shot success on novel embodiments without fine-tuning.
  • LAP-3B exhibits stable training, favorable scaling, and strong fine-tuning efficiency.
  • The study focuses on zero-shot transfer across single-arm manipulators, although LAP is described as adaptable to bimanual robots and heterogeneous data sources.
  • The paper has not evaluated substantially higher control frequencies or extreme-precision settings such as fast reactive control and fine-grained deformable-object manipulation.

A.1 Qualitative Analysis for Zero-Shot Cross-Embodiment Experiments

Qualitative rollouts show that baseline VLAs often fail to produce meaningful or safe motions, while LAP-3B completes representative manipulation tasks under comparable conditions. The baseline failures mainly involve imprecise trajectories and gripper–object misalignment.

  • OpenVLA, MolmoAct, and π0.5-Base consistently fail to produce meaningful motions across all robot platforms.
  • X-VLA generates smooth trajectories and sometimes approaches targets, but never achieves the gripper–object alignment required for successful manipulation.
  • π0-replicated frequently overshoots targets and grasps the tissue box rather than the tissue itself in the YAM Tissue task.
  • π0.5-replicated produces smoother, directionally reasonable motions but lacks the spatial precision needed for reliable grasping and task completion.
  • On the YAM Sort task, LAP-3B completes the task while replicated baselines move toward the basket without first grasping objects.

A.2 Full Results for the LIBERO Benchmark

Full LIBERO results confirm consistent LAP-3B gains across task families and all three task suites, with additional improvement from VQA co-training. The evaluation spans multiple robot embodiments with differing kinematics, sensing configurations, and control-action choices.

  • LAP-3B shows consistent performance gains across task families and all three LIBERO task suites.
  • LAP-3B+VQA Co. improves over LAP-3B alone and achieves stronger fine-tuning performance with the same gradient steps.
  • Figure 10 reports that VQA co-training accelerates convergence, allowing LAP-3B+VQA Co. to reach high success in fewer gradient steps.
  • The evaluation covers DROID, Custom Franka, YAM, and Kinova, including seen and unseen embodiments with different cameras, arms, and grippers.
  • Fine-tuning uses joint-space actions on YAM and end-effector pose actions on the other robots.

B.2 Training Hyperparameters

LAP-3B combines flow and language-action losses, with the language-action term weighted differently during pre-training and fine-tuning. The pre-training weighting compensates for the faster convergence of language-action supervision.

  • The overall training objective is L = Lflow + λLCE.
  • λ is set to 0.8 during pre-training and 0.4 during fine-tuning.
  • Using λ < 1 during pre-training downweights the faster-converging language-action loss so both model components learn at comparable rates.

B.3 Baseline Implementations

The replicated baselines are tuned and evaluated under shared architectural, data, sampling, and optimization choices, while LAP uses structured language-action supervision and attention mechanisms. Actions are discretized into ordered natural-language clauses, with an action expert trained through flow matching and integrated at inference.

  • Baseline implementations: All replicated baselines are tuned through learning-rate and batch-size sweeps, with additional FAST-token loss-weight and action-horizon sweeps for π0.5-replicated.The selected defaults are batch size 2048, learning rate 1 × 10−4, action horizon 16, and λ = 0.6.
  • Baseline implementations: Replicated baselines share LAP’s network architectures, data mixtures, sampling strategies, and optimization settings beyond explicitly tuned parameters.π0-replicated follows the original OpenPI implementation, while π0.5-replicated adds FAST-token supervision and Knowledge Insulation.
  • Language-action representation: Language-actions are generated as fixed-order clauses describing translations, tilts, rotations, and gripper actions.Translations are discretized in integer centimeters and rotations in integer degrees; zero-valued movements are omitted.
  • Language-action representation: Motion-prediction VQA pairs are generated automatically by asking semantically varied questions about temporal relationships between observations and answering with ground-truth language-actions.This avoids manual annotation while increasing prompt diversity and robustness.
  • Attention and action generation: LAP-3B uses bidirectional attention over image, prompt, and state prefixes, causal attention among language-action tokens, and an action expert that attends to prefixes but not language-action tokens.The action expert uses full self-attention, whereas language-action tokens attend causally to one another.
  • Attention and action generation: The action expert learns a continuous vector field that transports Gaussian noise to valid actions, then generates actions by integrating the learned dynamics from τ = 0 to τ = 1.Training uses a flow-matching objective, and inference uses a fixed-step explicit ODE solver.

B.9 Training Data

Training uses a mixture of datasets whose batch fractions are summarized in Table 5, with DROID receiving greater weight for task and environmental diversity. The reported checkpoints and baselines share a large-scale compute setup, while real-robot inference runs on a single commodity GPU.

  • Training data: The training mixture’s percentages represent each dataset’s true fraction of samples within a batch, with DROID weighted more heavily for broad task coverage and environmental diversity.The complete mixture is summarized in Table 5.
  • Data filtering: Idle action segments and trajectories without task instructions are removed during training.Idle segments are filtered using a small-motion threshold applied across consecutive steps.
  • Compute: Each reported model checkpoint was trained for approximately 50 hours on a TPU v6e-64 slice under an identical large-scale pre-training setup.This includes LAP-3B, its VQA co-training variant, and replicated baselines.
  • Compute: Across approximately 200 large-scale pre-training runs, the experiments consumed over 4,000 TPU v6e-64 hours, excluding prototyping, debugging, and downstream fine-tuning.The total covers LAP-3B and all replicated baselines.
  • Compute: Real-robot evaluations run in real time on a single NVIDIA RTX 4090 without model parallelism, specialized hardware, or acceleration techniques.The reported experiments required approximately 24 hours of total human-supervised runtime.

C Evaluation Protocol

Evaluation uses repeated randomized real-robot trials with explicit termination criteria and task-specific scoring. Zero-shot experiments report binary success with valid confidence intervals, whereas fine-tuning experiments additionally measure partial task completion.

  • Evaluation protocol: Each task is evaluated over 20 independent trials with randomized object placement within workspace bounds.Trials end on success, timeout, or safety-triggered aborts such as collisions, joint-limit violations, or unsafe motion.
  • Evaluation protocol: Zero-shot experiments report binary task success together with 95% finite-sample-valid confidence intervals for binomial success rates.The intervals control the Type-I error, or miscoverage probability.
  • Evaluation protocol: Fine-tuning experiments use staged success scores to capture partial completion, including reaching, grasping, and completing the Hang Tape on Rack task.The stages are scored from 0.25 through 1.0.
  • Evaluation protocol: Fold Towel and Place in Basket uses staged outcomes in which 0.5 denotes a correctly folded towel, 0.75 placement in the basket while unfolded, and 1.0 full task success.These scores distinguish partial completion from the complete task objective.
  • Evaluation protocol: Zero-shot evaluation visualizations show camera-view observations collected across different robot embodiments and tasks.The examples are presented from left to right across the evaluation views.
Loading 2602.10556v2…