Source-linked AI summary
Cross-Hand Latent Representation for Vision-Language-Action Models
Guangqi Jiang, Yutong Liang, Jianglong Ye, Jia-Yang Huang, Changwei Jing, Rocky Duan, Pieter Abbeel, Xiaolong Wang, Xueyan Zou
TL;DR
Dexterous VLA models need scalable learning across hands because action spaces vary with embodiment and new-hand data collection is costly. XL-VLA addresses this with a unified latent action space integrated into a VLA pipeline. It consistently outperforms raw-joint-space and retargeting baselines, including a mean success rate increase from 0.55 to 0.90 across tasks and hands.
Problem
Robotic action spaces depend on hand morphology, making scalable learning across diverse dexterous embodiments a central challenge.
Method
XL-VLA uses an embodiment-invariant latent action representation shared across dexterous hands within a VLA framework.
Results
XL-VLA increases mean success rate from 0.55 to 0.90 (+0.35) across all tasks and hands and consistently outperforms VLA models without a unified action representation.
Takeaways & Limitations
The results support latent action spaces as a foundation for generalizable, data-efficient dexterous manipulation across embodiments.
Abstract
from arXiv · showhide
Dexterous manipulation is essential for real-world robot autonomy, mirroring the central role of human hand coordination in daily activity. Humans rely on rich multimodal perception--vision, sound, and language-guided intent--to perform dexterous actions, motivating vision-based, language-conditioned manipulation systems for robots. However, training reliable vision-language-action (VLA) models for dexterous manipulation requires large-scale demonstrations across many robotic hands. In addition, as new dexterous embodiments appear rapidly, collecting data for each becomes costly and impractical, creating a need for scalable cross-embodiment learning. We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands. This embodiment-invariant latent space is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data. Experimental results demonstrate that XL-VLA consistently outperforms baseline VLA models operating in raw joint spaces, establishing it as an effective solution for scalable cross-embodiment dexterous manipulation.
1. Introduction
XL-VLA addresses the morphology-dependent variability of dexterous-hand actions with a shared latent action space and a full cross-embodiment VLA pipeline. The work also contributes a multi-hand teleoperation dataset and reports stronger cross-embodiment performance than standard VLA baselines.
- Robotic action spaces vary with hand morphology, raising the need for unified representations and simpler integration of new embodiments.
- XL-VLA introduces an embodiment-invariant latent action space for joint training across heterogeneous dexterous hands.
- The dataset covers 10 manipulation tasks across four dexterous hands and contains 2M state-action pairs.
- XL-VLA is a full VLA pipeline built on the cross-embodiment latent action space.
- XL-VLA achieves stronger cross-embodiment performance than standard VLA baselines and generalizes zero-shot to untrained cross-embodiment task configurations.
2. Related Work
Related work spans dexterous manipulation, cross-embodiment learning, hand retargeting, and latent action spaces. XL-VLA is situated among approaches that seek policies or mappings transferable across diverse robotic embodiments.
- Dexterous Manipulation: Dexterous-manipulation research covers hardware, sensors, learning and control algorithms, and human–robot interaction.
- Cross Embodiment: Cross-embodiment methods aim to adapt a single policy across diverse humanoids or dexterous hands without per-robot retraining.
- Hand/Dex Retargeting: Hand-retargeting research includes kinematic pipelines, unsupervised mapping, contact-aware formulations, functional retargeting, and policy-centric methods.
- Latent Action Space: Latent action spaces support a single hand-agnostic VLA policy across multiple dexterous hands while preserving VLM pretraining benefits.
3. Method
XL-VLA uses a shared, embodiment-invariant latent action space to let a hand-agnostic VLA model operate across diverse dexterous hands. Hand-specific encoders and decoders connect joint commands to this space, which is pretrained with reconstruction, retargeting, and latent regularization objectives.
- Cross-embodiment VLA: The VLA receives latent action histories with vision and language tokens, predicts the next latent chunk, and decodes it into embodiment-specific joint commands.Latent encoders and decoders remain frozen during VLA finetuning.
- Cross-embodiment VLA: XL-VLA represents actions in a shared latent space so one hand-agnostic VLA policy can operate across heterogeneous dexterous hands.Hand identity selects the appropriate encoder and decoder rather than being provided as an explicit VLA input token.
- Latent action modeling: Each hand uses a hand-specific encoder and decoder within a multi-headed VAE-style autoencoder that maps different joint spaces into a common latent distribution.The encoder produces Gaussian posterior parameters, samples a latent code, and the decoder reconstructs the corresponding joint configuration.
- Latent action modeling: The latent space is trained with reconstruction, retargeting, and regularization constraints to preserve hand kinematics while aligning behavior across embodiments.Retargeting uses differentiable forward kinematics to compare fingertip pinch distances and directions across hands.
- Evaluation: XL-VLA achieves consistently higher cross-embodiment success rates than π0 across the evaluated hands and tasks.The comparison evaluates each task 10 times and computes success rates across hands.
- Latent pretraining: Training samples joint configurations within hardware limits, decodes each latent through all hand decoders, and jointly optimizes self-reconstruction and cross-hand retargeting losses.The latent autoencoder is trained without demonstrations or IK-generated trajectories.
4. Experiments
Experiments evaluate XL-VLA across multi-hand dexterous manipulation, cross-robot training, zero-shot task transfer, latent-space effectiveness, and design choices. XL-VLA consistently improves cross-embodiment performance and supports reliable latent replay and unseen task–hand combinations.
- Experimental Setup: The dataset contains 10 diverse manipulation tasks with 50 demonstrations per task per hand, totaling 2000 demonstrations.Experiments use a bimanual 7-DoF xArm and Unitree G1 humanoid with multiple robot hands.
- Cross-Hand Data Scaling: XL-VLA achieves strong and consistent improvements over the π0 baseline across every evaluated hand and task.The baseline performs inconsistently and generally poorly because hand kinematics and actuation differ substantially.
- Cross-Hand Data Scaling: 0.55 to 0.90 (+0.35) is XL-VLA’s increase in mean success rate averaged across all tasks and hands.The largest gains occur on dexterity-heavy tasks including Sort Cans, Hand over Bottle, and Re-arrange Boxes.
- Zero-Shot Task Generalization: XL-VLA consistently outperforms the retargeting VLA baseline on unseen task–hand combinations and never underperforms it on any evaluated hand or task.The advantage is especially pronounced on fine-grained tasks where geometric retargeting struggles with coordinated finger motion.
- Latent Action Space Effectiveness: 0.82 and 0.81 mean success rates are achieved on two hand pairs in latent replay, versus 0.60 and 0.61 for supervised LAD.XL-VLA requires neither supervision data nor paired labels, relying on unsupervised latent alignment.
- Design Choice Comparison: The H128→64 architecture with latent dimension 32 provides a balance between model capacity and latent compactness.Performance is stable across many architectures and dimensions, while substantially larger latent spaces can degrade embodiment-invariant structure.
5. Conclusion.
XL-VLA uses a unified, embodiment-invariant latent action space for scalable cross-embodiment dexterous manipulation. Real-world experiments show stronger performance, zero-shot generalization, and a flexible interface for new hands.
- XL-VLA provides a unified latent action space that supports scalable cross-embodiment dexterous manipulation.The representation is designed to be embodiment-invariant and shared across diverse robotic hands.
- The framework supports seamless training across diverse hands and zero-shot generalization to new hand–task combinations.
- Real-world experiments show that XL-VLA consistently outperforms standard VLA models and retargeting-based baselines.
- XL-VLA offers a flexible, plug-and-play interface for newly introduced hands and supports data-efficient dexterous manipulation systems.
6. Appendix
The appendix documents the experimental embodiments, sensing and teleoperation setups, object and task diversity, latent visualizations, and additional evaluation details. These materials cover both tabletop xArm experiments and G1 experiments, including success-rate reporting conventions.
- 6.1. Latent Visualizations: Latent visualizations show the same representation decoded across all four hands, including continuous trajectories rendered for each embodiment.The appendix highlights X-Hand in one trajectory visualization and reports additional-hand visualizations beyond Figure 5.
- 6.2. Experimental Setup: The real-world tabletop setup uses a single front-mounted RealSense L515 camera, while G1 uses an egocentric L515 view near the robot’s neck.
- 6.2. Experimental Setup: The experiments use four dexterous hands with different shapes, scales, degrees of freedom, and actuated joints.The listed hands are Ability, Inspire, X-Hand1, and Paxini DexH13.
- 6.3. Data Collection: Data collection combines Apple Vision Pro hand tracking with retargeting and inverse kinematics, while G1 teleoperation uses an upper-body system with MANUS Mocap gloves.
- 6.4. Evaluation: For unseen tasks, partial success is recorded only when one bimanual arm finishes while the overall task fails; otherwise, success requires completing the specified task.
- 6.4. Evaluation: Evaluation spans 10 manipulation tasks and diverse everyday objects varying in scale, shape, texture, and weight.The tasks require varied manipulation skills and have different difficulties.