Source-linked AI summary
X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, Ya-Qin Zhang, Jiangmiao Pang, Jingjing Liu, Tai Wang, Xianyuan Zhan
TL;DR
Generalist VLA models must learn across heterogeneous robots, hardware configurations, data-collection strategies, and task scenarios. X-VLA addresses this with embodiment-specific Soft Prompts in a flow-matching Transformer framework, achieving SOTA results across broad simulation and real-world evaluations while supporting parameter-efficient adaptation and scaling.
Problem
Generalist VLA models need large, diverse robotics datasets, but hardware, action spaces, camera settings, visual domains, and task distributions create substantial heterogeneity.
Method
X-VLA uses learnable Soft Prompts for individual hardware setups within a generalist flow-matching VLA framework built from standard Transformer encoders.
Results
X-VLA-0.9B achieves SOTA performance across broad benchmarks, including 6 simulation benchmarks and 3 real-world robot platforms, while 1% parameter tuning reaches 93% success on LIBERO and 54% on Simpler-WidowX.
Takeaways & Limitations
Soft Prompts support rapid cross-embodiment adaptation and scalable training across model size, data diversity, and data volume without saturation at the largest tested configuration.
Takeaways & Limitations
Low-dimensional action labels provide limited supervision, and temporal downsampling only partially addresses the lack of higher-level task information.
Abstract
from arXiv · showhide
Successful generalist Vision-Language-Action (VLA) models rely on effective training across diverse robotic platforms with large-scale, cross-embodiment, heterogeneous datasets. To facilitate and leverage the heterogeneity in rich, diverse robotic data sources, we propose a novel Soft Prompt approach with minimally added parameters, by infusing prompt learning concepts into cross-embodiment robot learning and introducing separate sets of learnable embeddings for each distinct data source. These embeddings serve as embodiment-specific prompts, which in unity empower VLA models with effective exploitation of varying cross-embodiment features. Our new X-VLA, a neat flow-matching-based VLA architecture, relies exclusively on soft-prompted standard Transformer encoders, enjoying both scalability and simplicity. Evaluated across 6 simulations as well as 3 real-world robots, our 0.9B instantiation-X-VLA-0.9B simultaneously achieves SOTA performance over a sweep of benchmarks, demonstrating superior results on a wide axes of capabilities, from flexible dexterity to quick adaptation across embodiments, environments, and tasks. Website: https://thu-air-dream.github.io/X-VLA/
1. Introduction
X-VLA addresses the heterogeneity that complicates cross-embodiment VLA training with learnable soft prompts and a scalable flow-matching Transformer architecture. X-VLA-0.9B achieves strong benchmark performance and efficient adaptation across robotic embodiments, environments, and tasks.
- Heterogeneous robotics data varies in hardware, action spaces, cameras, visual domains, and task distributions, challenging generalist VLA pretraining.
- X-VLA assigns learnable embodiment-specific soft prompts to heterogeneous data sources, guiding the model toward hardware-aware learning.
- X-VLA combines soft-prompted multimodal encoding with standard Transformer encoders and flow-matching action generation for scalable VLA training.
- Soft Prompts outperform other SOTA methods on heterogeneous training, with stable learning and favorable scaling to larger models and mixed-robot datasets.
- The 0.9B model is trained through a two-phase pipeline using a curated heterogeneous mixture spanning 290K episodes, seven platforms, and five robotic-arm types.
- 93% success rate on LIBERO and 54% on Simpler-WidowX result from tuning only 1% of X-VLA-0.9B parameters, or 9M parameters.This performance is comparable to π0 while using 300× fewer parameters than the cited 3B-versus-9M comparison.
2. Preliminary
VLA models combine multimodal observations with action generation, commonly learning demonstrated action chunks through behavior cloning. Flow matching instead learns a velocity field that transports noise toward expert actions, while heterogeneous datasets introduce hardware- and distribution-level variation.
- VLA models unify multimodal understanding and robotic action generation, typically starting from VLMs pretrained on image-text data and finetuned on expert trajectories.
- Behavior cloning trains a policy to predict demonstrated action chunks from multimodal observations such as images, language, and proprioceptive states.
- Flow-matching policies learn a velocity field that iteratively transports Gaussian noise toward target action chunks conditioned on observations.
- The flow-matching objective uses an optimal-transport path aligned with the linear interpolation between noise and expert data.
- Mixed cross-embodiment datasets vary in hardware configurations, low-level action signals, visual understanding, and deployment conditions, creating substantial training heterogeneity.
3. Heterogeneous Soft Prompt Learning
The paper studies four strategies for cross-embodiment heterogeneity and proposes soft prompts as a flexible alternative that encodes domain-specific hardware configurations with learnable parameters. These prompts guide embodiment-aware learning without modifying the backbone through intermediate projections or scripted hardware descriptions.
- Design choices: The empirical study compares domain-specific action projection, HPT-style projection, language prompts, and soft prompts for heterogeneous training.
- Alternative strategies: Domain-specific action projection changes only final action generation, overlooking earlier embodiment-aware reasoning and heterogeneity in cameras and task distributions.
- Alternative strategies: HPT-style projection maps multimodal observations into a shared space but can alter feature distributions and destabilize pretrained VLM representations.
- Alternative strategies: Language prompts explicitly describe hardware configurations textually, but carefully scripted descriptions hinder practical adaptability and scalability.
- Soft prompts: Soft prompts introduce domain-specific learnable parameters that implicitly encode hardware configurations and are injected early in action generation.
- Soft prompts: Soft prompts provide a flexible and scalable solution for encoding domain-specific hardware configurations compared with HPT-style projections and language prompts.
4. X-VLA: Soft-Prompted Transformer Enhanced VLA model
X-VLA combines soft prompts with a streamlined multimodal Transformer pipeline to train on heterogeneous robotic data and adapt efficiently to new embodiments. Its training recipe aligns modalities and actions while scaling data and compute to improve validation error and adaptation performance.
- 4. X-VLA: Soft-Prompted Transformer Enhanced VLA model: X-VLA is designed for stable pretraining on heterogeneous datasets and efficient adaptation to new domains, with an ablation path evaluating its components.The model uses a Soft-Prompted Transformer architecture for cross-embodiment VLA learning.
- 4. X-VLA: Soft-Prompted Transformer Enhanced VLA model: Lower pretraining validation error is associated with progressively higher adaptation success, making validation error a proxy for pretraining performance.The ablation evaluates both pretraining validation error and adaptation success rates on Simpler-WidowX.
- 4.1. Architecture: X-VLA uses dedicated encoding strategies for high-dimensional visual-language inputs and low-dimensional proprioceptive-action states before vanilla Transformer stacks fuse modalities.Multi-view images and language form the observation stream, while proprioception, action tokens, and time embeddings form the low-dimensional stream.
- 4.2.1. Pretraining and Finetuning Pipeline: Pretraining jointly optimizes the backbone and soft prompts, producing an embodiment-agnostic foundation for rapid adaptation across heterogeneous robots.The adaptation pipeline introduces prompts for new hardware configurations and then proceeds to policy specialization.
- 4.2.1. Pretraining and Finetuning Pipeline: New embodiments are adapted in two stages: prompt warm-up with pretrained weights frozen, followed by joint optimization of the backbone and warmed-up prompt.The first stage encodes hardware-specific setups using new learnable prompts; the second specializes the full policy.
- 4.2.1. Pretraining and Finetuning Pipeline: Reduced learning rates for soft prompts and vision-language modules limit drift from pretrained representations during pretraining and adaptation.The adjustment is intended to support smoother optimization and more reliable specialization for novel embodiments.
- 4.2.2. Enhanced Data Processing: Actions are standardized as end-effector position, Rotate6D rotation, and discretized gripper state, with MSE for pose components and BCE for gripper state.This representation is used to provide consistent supervision across embodiments.
- 4.2.2. Enhanced Data Processing: Temporal downsampling converts fine-grained demonstrations into 30 anchor points summarizing the intended trajectory over the next 4 seconds.The procedure targets noisy low-level movements that are unsuitable for high-level intention modeling during pretraining.
5. Experiments
The experiments examine X-VLA's scaling, adaptation, and soft-prompt representations across heterogeneous robotic data and platforms. Results show strong benchmark performance, efficient dexterous manipulation, parameter-efficient adaptation, and structured embodiment-specific prompts.
- X-VLA is evaluated for scaling behavior, adaptation performance, and whether soft prompts capture meaningful heterogeneity across mixed data sources.
- Scaling Experiments: X-VLA-0.9B shows no saturation at 0.9B parameters, 290K episodes, and 7 data sources, indicating further scaling may yield additional gains.
- Adaptation Experiments: Across six simulation benchmarks, X-VLA establishes new SOTA results on five and exceeds 90% success on Simpler-WidowX, Libero, and Calvin's first stage.
- Adaptation Experiments: X-VLA surpasses baselines across all five real-world BridgeData V2 tasks spanning distinct capability axes.
- Adaptation Experiments: With 1,200 Soft-Fold trajectories, X-VLA-0.9B achieves nearly 100% success and 33 completed folds per hour in dexterous cloth folding.
- Adaptation Experiments: Using 9M tunable parameters, X-VLA reaches 93% on Libero and 54% on Simpler-WidowX, comparable to fully finetuned models.
- In-Depth Analysis: T-SNE prompts form hardware-aligned clusters while related Franka views intermingle, and adapted prompts converge faster and reach higher success than random prompts.
6. Conclusion
The paper concludes that X-VLA provides a generalist framework for heterogeneous robotic platforms, combining strong benchmark performance with parameter-efficient adaptation. It also identifies responsible curation of large-scale robotics data as an ongoing concern.
- X-VLA-0.9B achieves SOTA performance across broad benchmarks and remains competitive with fully finetuned models using minimal tunable parameters.
- Soft Prompts support scaling across model size, data diversity, and data volume without saturation at 0.9B parameters, 290K episodes, and 7 data sources.
- The paper uses open-sourced robotics datasets to reduce risks associated with privacy-sensitive information and embedded biases in large-scale pretraining data.
- Future robotics-model training should filter privacy-sensitive content and address potential biases for responsible and fair embodied-AI deployment.
C. Architecture Design
The X-VLA architecture combines a pretrained vision-language encoder with a standard Transformer action-generation backbone, integrating soft prompts and separate input streams. This design is intended to improve training stability and validation performance.
- X-VLA uses Florence-Large for vision-language encoding and a 24-layer Transformer with hidden size 1024 for action generation.
- The encoding pipeline integrates soft prompts while explicitly disentangling high- and low-dimensional input streams.
- The architecture yields improved training stability and consistently stronger validation performance.
D. More Results
Additional experiments compare architectures, test joint adaptation across embodiments, and assess data-efficient PEFT. X-VLA maintains stable, strong performance while supporting multi-domain transfer and limited-demonstration adaptation.
- Alternative Architectural Designs: The standard MM-DiT decoder often destabilizes training and performs worse on heterogeneous datasets and downstream adaptation.
- Alternative Architectural Designs: X-VLA consistently achieves the best validation performance among compared backbones while maintaining stable optimization dynamics.
- Cross-Embodiment Joint Training: Joint finetuning preserves strong performance across evaluated embodiments and sometimes improves success rates over single-domain finetuning.
- Data-Constrained Settings: With only 10 demonstrations, PEFT-finetuned X-VLA-0.9B retains a 91.1% success rate on Libero-Goal.
E. Failure Attempts for Absorbing Heterogeneity
The authors tested domain-specific parameter mechanisms for absorbing cross-embodiment heterogeneity. Heterogeneous adapters and MoE routing introduced optimization instability or limited gains, motivating simpler soft prompts.
- Domain-specific low-rank adapters were intended to capture embodiment variations while preserving shared backbone features.The design introduced LoRA-style modules in parallel with the shared backbone.
- Additional heterogeneous adapters often conflicted with backbone optimization, causing instability and degraded cross-domain generalization.
- The heterogeneity-guided MoE router collapsed toward a few experts, wasting capacity and producing only marginal performance gains.
- Load-balancing regularization caused rapid expert switching that destabilized optimization and degraded overall training dynamics.
F. Soft-FOLD: Superior Dexterous Manipulation Model with a high-quality cloth folding dataset
X-VLA-0.9B was adapted for dexterous cloth folding using the Soft-FOLD dataset and staged, targeted data collection. The resulting model combines diverse folding skills with training and validation procedures designed for heterogeneous data.
- Dataset and model: Soft-FOLD is introduced as a high-quality cloth-folding dataset for qualitative evaluation of a finetuned X-VLA-0.9B model.
- Data collection: Demonstrations decompose folding into smoothing and completion stages to reduce inconsistent behavioral modes during policy learning.The first stage emphasizes consistent smoothing until meaningful cloth keypoints emerge before transitioning to completion.
- Data collection: DAgger-style collection iteratively trains ACT, identifies failures, and gathers targeted demonstrations to address them.ACT is retrained after every 100 collected episodes.
- Dexterous manipulation: The complete folding sequence requires localization, picking, placing, and dynamic swinging skills.
- Training: Pretraining balances heterogeneous sources using weighted sampling and cross-domain, cross-trajectory shuffling to stabilize optimization and prevent domain overfitting.The training mixture and sampling design expose the model to diverse samples at every iteration.
- Finetuning: Adaptation updates soft prompts and action heads first, then restores the learning rate for joint training.The procedure uses 1,000 iterations of prompt-and-head updates followed by a 1,000-iteration warm-up phase.
J. Evaluation Details in Real-World Experiments
Real-world evaluation tests X-VLA-0.9B on three robotic embodiments with different hardware, camera configurations, and task domains. The experiments cover manipulation, dexterous control, low-resource adaptation, and autonomous driving.
- WidowX: WidowX evaluates robust manipulation and language-instruction following through real-world pick-and-place tasks.Each task is evaluated 10 times.
- AgileX: AgileX is used for dexterous, fine-grained bimanual control with wrist-mounted cameras.
- AIRBOT: AIRBOT tests parameter-efficient finetuning on an unseen embodiment using only 200 demonstrations for cloth picking.The setup evaluates adaptation under strict data and resource constraints.
- Hardware setups: Three embodiments—WidowX, AgileX, and AIRBOT—form a heterogeneous real-world validation environment.The platforms use distinct camera configurations and target different evaluation aspects.
- Autonomous driving: NAVSIM uses closed-loop assessment and reports PDM, which aggregates collision, drivable-area, safety, comfort, and progress metrics.Higher PDM indicates better performance, and results are averaged over the public test split.
- Simulation benchmarks: The evaluation reports detailed results across simulation benchmarks including NAVSIM and other listed suites.
N. Limitations and future works
The paper identifies three main limitations: modest scale and limited robotics data, sparse action supervision, and continued reliance on embodiment-specific adaptation. Future work targets broader scaling, richer supervision, and more seamless deployment.
- Scaling: X-VLA-0.9B remains modest in scale because computation and high-quality robotics data are limited.Current robotics corpora are less diverse and smaller than language or vision-language datasets.
- Supervision: Low-dimensional action labels provide limited information about task structure, higher-level reasoning, intent, and multi-step dependencies.The authors propose richer spatial, dynamics, subgoal, and self-supervised signals as future directions.
- Deployment: Deployment still requires embodiment-specific adaptation and a small number of demonstrations rather than operating as a plug-and-play solution.This creates overhead and is problematic when demonstrations are scarce, expensive, or risky to collect.