Source-linked AI summary
AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
Zhiyue Zhao, Jingyi Wu, Hairuo Liu, Mingyu Liu, Liyang Li, Hengdi Zhang, Tong He, Zhengxue Cheng
TL;DR
Dexterous manipulation is difficult to scale because robot demonstrations are costly and embodiments use different action spaces and visual appearances. AdvDex combines human demonstrations, a shared action representation, and domain-adversarial learning, achieving improved generalization and effective human-to-robot transfer in real-world experiments.
Problem
Dexterous manipulation lacks scalable demonstrations and directly comparable action spaces, while heterogeneous visual data can encode embodiment-specific appearance.
Method
AdvDex combines the OmniShare human-demonstration dataset, JAAS with an SE(3) wrist pose and 15 finger joints, and domain-adversarial visual representation learning.
Results
Real-world experiments show improved manipulation performance, generalization to unseen objects and environments, and effective human-to-robot skill transfer.
Takeaways & Limitations
Combining scalable human demonstrations, shared action representations, and domain-adversarial learning provides a promising path toward more general cross-embodiment manipulation.
Takeaways & Limitations
Fine-grained transfer remains limited by hardware differences, and evaluation currently focuses on a single dexterous robot platform without modeling embodiment-specific dynamics or contact constraints.
Abstract
from arXiv · showhide
Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an $\mathrm{SE}(3)$ wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.
1 Introduction
AdvDex addresses the data bottleneck and cross-embodiment challenges in dexterous manipulation by combining human demonstrations, a shared action representation, and embodiment-invariant visual learning. Its OmniShare dataset, Joint-Aligned Action Space, and domain-adversarial framework support cross-embodiment transfer, including zero-shot policy transfer and few-shot adaptation.
- Motivation and challenges: Dexterous manipulation data are fragmented across hardware platforms, while embodiment-specific visual cues limit cross-embodiment generalization and zero-shot transfer.Robot hands differ in kinematic structure, degrees of freedom, joint constraints, and appearance, leaving human and robot manipulation without a shared action representation.
- Motivation and challenges: AdvDex jointly addresses kinematic alignment and embodiment-invariant representation learning because pooling trajectories or aligning actions alone is insufficient.The framework targets both action comparability and prevention of embodiment identity becoming a visual shortcut.
- Contributions: OmniShare provides over 100k trajectories spanning 500 tasks and 700 objects, with high-quality kinematic supervision and tactile measurements from human demonstrations.Its microsecond-synchronized sensor suite reduces reliance on robot teleoperation.
- Contributions: The Joint-Aligned Action Space uses an SE(3) wrist pose and 15 finger joints as a shared action interface for human hands, dexterous robot hands, and parallel grippers.This canonical representation functionally aligns distinct embodiments.
- Contributions: AdvDex integrates a VLM-DiT architecture with a Gradient Reversal Layer, and physical experiments show consistent baseline improvements, zero-shot policy transfer, and few-shot adaptation in unseen environments.These components form a unified framework for cross-embodiment manipulation.
2 Related Work
Related work scales VLA action learning through auxiliary data, simulation, and latent representations, while human demonstrations reduce teleoperation costs but leave morphological alignment challenges. Dexterous manipulation has advanced across analytic, simulated, teleoperated, and human-motion approaches, yet many policies remain hardware-specific or task-narrow.
- VLA learning from auxiliary data: Large-scale VLA pre-training learns diverse tasks, but real-robot action supervision remains expensive, motivating auxiliary data, simulation, and latent representations.Visual learning can exploit abundant image and video data, whereas action supervision remains constrained by the cost of collecting real-robot demonstrations.
- Human demonstrations and embodiment alignment: Human demonstrations provide an abundant alternative to teleoperation, but bridging human–robot morphology commonly requires retargeting, human-in-the-loop corrections, or alignment fine-tuning.Earlier methods extracted visual priors or trajectories from videos, while newer approaches use VR devices for end-to-end motion supervision.
- Dexterous manipulation: Dexterous manipulation spans analytic control, simulation-based reinforcement learning, teleoperation, and human-motion imitation, yet many language-conditioned policies remain hardware-specific or narrowly focused.Examples of narrow tasks include static grasping, despite progress in language conditioning.
3 Method · 3.1 OmniShare Dataset
OmniShare is a large-scale multimodal dataset for dexterous manipulation, spanning diverse tasks, objects, and real-world domains. Synchronized kinematic, tactile, visual, and language signals are processed into a canonical hand representation that supports cross-embodiment action supervision.
- 3.1 OmniShare Dataset: Table 1 compares OmniShare with other 3D bimanual motion datasets.
- 3.1 OmniShare Dataset: The dataset comprises over 10k hours of videos covering 500 tasks and 700 objects.
- 3.1.1 Dataset Overview: OmniShare contains over 100k trajectories across five real-world domains, covering more than 500 manipulation tasks and 700 objects.The dataset includes substantial variation in object geometry and materials to support policy generalization.
- 3.1.2 Data Collection System: A microsecond-synchronized sensor suite records demonstrations using a data glove with 29 magnetic rotary encoders for sub-degree hand kinematics.
- 3.1.2 Data Collection System: A Hall-effect tactile array measures normal contact forces while synchronized recordings capture hand motion, contact, and multi-view scene observations.The shared timeline also preserves instructions, object context, and surrounding geometry.
- 3.1.3 Data Processing Pipeline: ArUco markers and FoundationPose estimate 6D wrist and object poses before physics-aware optimization retargets hand states into a canonical MANO representation.The optimization jointly minimizes kinematic and tactile discrepancies.
- 3.1.3 Data Processing Pipeline: A distance-aware decay function modulates tactile signals to preserve contact timing and grasp-force variation.
- 3.1.3 Data Processing Pipeline: Demonstrations retain visual observations and language instructions while motion is mapped through an intermediate hand space into JAAS.This lets human trajectories supervise actions across heterogeneous embodiments without robot-specific labels for every human joint.
3.2 Unified Cross-Embodiment Framework
AdvDex unifies human and robot manipulation through the Joint-Aligned Action Space, which maps heterogeneous embodiments into shared wrist and finger-joint semantics. A domain-adversarial VLA then promotes embodiment-invariant visual representations while retaining task-relevant information for action prediction.
- Joint-Aligned Action Space: JAAS represents human hands, dexterous robot hands, and parallel grippers with a shared SE(3) wrist pose and 15 finger joints.The representation uses 3D translation, continuous 3D rotation, and three 3-DoF Euler joints per finger.
- Joint-Aligned Action Space: JAAS uses functional correspondence to assign embodiment-specific joints to canonical slots, masking unavailable slots during action-loss computation.MANO hands and 19-DoF dexterous hands map to corresponding slots, while a parallel-gripper jaw maps to two canonical finger slots.
- Domain-Adversarial Learning: The domain-adversarial architecture combines a VLM backbone, Diffusion Transformer action expert, and domain discriminator to predict joint-aligned action chunks.The action expert conditions on cognition and kinematic-state tokens, while the discriminator predicts embodiment-domain labels.
- Domain-Adversarial Learning: Gradient reversal multiplies VLM gradients by −λ, encouraging suppression of hardware-specific appearance cues while retaining transferable task-relevant geometry.The discriminator classifies embodiment domains, whereas the reversed gradient drives the visual encoder toward domain-invariant representations.
- Training Objective: The framework is trained end-to-end with diffusion denoising MSE and domain-classification cross-entropy losses combined through a minimax objective.The adversarial strength is controlled by λ, and state conditioning helps distinguish kinematically explained differences from residual visual appearance cues.
4 Experiments
Experiments evaluate AdvDex on unseen human-manipulation trajectories, real-world dexterous tasks, zero-shot human-to-robot transfer, and few-shot adaptation. Across these settings, OmniShare pre-training and domain-adversarial learning improve generalization, execution, and data efficiency.
- Hand-Motion Prediction: The hand-motion benchmark combines 200 OmniShare-Unseen trajectories across 20 novel object categories with 200 HOI4D trajectories for unseen-environment generalization.Evaluation uses pre-grasp distance and execution-phase MPJPE and MWTE metrics, reported in millimeters.
- Hand-Motion Prediction: Adding OmniShare improves performance over training without it, while adding the domain-adversarial objective further improves all metrics on OmniShare-Unseen and HOI4D.The framework is comparable to VITRA when trained without OmniShare, indicating benefits from OmniShare’s ground-truth trajectories.
- Real-World Dexterous Manipulation: On five real-world manipulation tasks, AdvDex matches or outperforms baselines across tasks, while removing pre-training, OmniShare, or domain-adversarial learning reduces success rates.The protocol evaluates AdvDex, π0.5, and VITRA using identical demonstrations, action representations, training steps, and 20 trials per task.
- Zero-Shot Human-to-Robot Transfer: AdvDex transfers human-demonstrated skills zero-shot to robot-only evaluation tasks, and removing adversarial learning during pre-training or post-training reduces execution performance.Co-training uses 1,000 robot teleoperation trajectories and 1,000 OmniShare human trajectories with mutually exclusive task sets.
- Few-Shot Adaptation: AdvDex achieves non-zero success without target-domain fine-tuning, outperforms failed baseline completion in the zero-shot setting, and improves substantially with five demonstrations.The data-efficiency study evaluates single-object grasping with 0, 5, and 20 target-domain demonstrations.
- Representation Analysis: With a Gradient Reversal Layer, features from different embodiments show greater t-SNE overlap than without it, suggesting reduced embodiment-specific separation.Without the GRL, embodiment features form distinct clusters during both pre-training and post-training.
5 Conclusions
AdvDex is presented as a unified Vision-Language-Action framework for dexterous manipulation across human and robot embodiments. Its contributions combine large-scale human demonstrations, a shared hand-and-gripper action representation, and domain-adversarial visual learning, with real-world experiments demonstrating competitive manipulation performance.
- Framework: AdvDex unifies dexterous manipulation learning across human and robot embodiments within a Vision-Language-Action framework.The framework is designed to support manipulation across different embodiments.
- OmniShare: OmniShare provides large-scale, high-quality human manipulation demonstrations for training.The dataset reduces reliance on robot-specific demonstrations by supplying human data.
- Joint-Aligned Action Space: The Joint-Aligned Action Space maps different hand and gripper kinematics into a shared action representation.This representation enables functional alignment across human hands, robot hands, and grippers.
- Domain-Adversarial Learning: Domain-adversarial learning reduces embodiment-specific information in visual features.The method targets embodiment-related variation in learned visual representations.
- Experiments: Real-world experiments demonstrate competitive manipulation performance.The conclusion also reports improved generalization, but the supplied passage is truncated before specifying its scope or results.
6 Limitations and Future Work
AdvDex’s cross-embodiment action alignment does not eliminate hardware-related barriers to transferring fine-grained manipulation skills. Limitations include insufficient embodiment-specific strategies in human data, evaluation on one robot platform, and action representations that omit embodiment-specific dynamics and contact constraints.
- Hardware differences still limit transfer of fine-grained manipulation skills despite cross-embodiment action alignment.
- Human imitation data alone may not capture embodiment-specific strategies required for high-precision control.
- Physical evaluation currently focuses on one dexterous robot platform, while the shared action representation omits embodiment-specific dynamics and contact constraints.