PaperlayerView exact revision on arXiv

Source-linked AI summary

UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos

Gu Zhang, Qicheng Xu, Haozhe Zhang, Jianhan Ma, Long He, Yiming Bao, Zeyu Ping, Zhecheng Yuan, Chenhao Lu, Chengbo Yuan, Tianhai Liang, Xiaoyu Tian, Maanping Shao, Feihong Zhang, Mingyu Ding, Yang Gao, Hao Zhao, Hang Zhao, Huazhe Xu

arXiv:2603.22264v1cs.RO

TL;DR

Dexterous manipulation lacks scalable data and unified control across costly, heterogeneous, high-dimensional robot hands. UniDex builds a robot-centric dataset from egocentric videos, aligns cross-hand actions through FAAS, trains a 3D VLA policy, and adds a portable capture pipeline for human–robot co-training. On five tool-use tasks across two hands, UniDex-VLA reaches 81% average task progress, with strong spatial, object and zero-shot cross-hand generalization.

  • Problem

    Dexterous manipulation is constrained by costly real-robot demonstrations, heterogeneous hand embodiments and high-dimensional control, while dexterous-hand foundation datasets and policies remain limited.

  • Method

    UniDex constructs a large robot-centric dataset from egocentric human videos, uses FAAS for shared cross-hand action coordinates, trains a 3D VLA policy, and captures additional human data for co-training.

  • Results

    81% average task progress was achieved across five real-world tool-use tasks on two hands, versus 38% for π0, with strong spatial, object and zero-shot cross-hand generalization.

  • Takeaways & Limitations

    UniDex provides a scalable foundation suite that supports dexterous-hand pretraining, cross-hand skill transfer and reduced reliance on costly robot demonstrations.

  • Takeaways & Limitations

    The current work does not yet leverage large action-free or weakly labeled egocentric activity datasets for further dexterous pretraining scale.

Abstract

from arXiv · show

Dexterous manipulation remains challenging due to the cost of collecting real-robot teleoperation data, the heterogeneity of hand embodiments, and the high dimensionality of control. We present UniDex, a robot foundation suite that couples a large-scale robot-centric dataset with a unified vision-language-action (VLA) policy and a practical human-data capture setup for universal dexterous hand control. First, we construct UniDex-Dataset, a robot-centric dataset over 50K trajectories across eight dexterous hands (6--24 DoFs), derived from egocentric human video datasets. To transform human data into robot-executable trajectories, we employ a human-in-the-loop retargeting procedure to align fingertip trajectories while preserving plausible hand-object contacts, and we operate on explicit 3D pointclouds with human hands masked to narrow kinematic and visual gaps. Second, we introduce the Function-Actuator-Aligned Space (FAAS), a unified action space that maps functionally similar actuators to shared coordinates, enabling cross-hand transfer. Leveraging FAAS as the action parameterization, we train UniDex-VLA, a 3D VLA policy pretrained on UniDex-Dataset and finetuned with task demonstrations. In addition, we build UniDex-Cap, a simple portable capture setup that records synchronized RGB-D streams and human hand poses and converts them into robot-executable trajectories to enable human-robot data co-training that reduces reliance on costly robot demonstrations. On challenging tool-use tasks across two different hands, UniDex-VLA achieves 81% average task progress and outperforms prior VLA baselines by a large margin, while exhibiting strong spatial, object, and zero-shot cross-hand generalization. Together, UniDex-Dataset, UniDex-VLA, and UniDex-Cap provide a scalable foundation suite for universal dexterous manipulation.

1. Introduction

UniDex addresses dexterous manipulation’s data, embodiment, and control challenges with a robot-centric dataset, a unified action space and VLA policy, and a portable human-data capture pipeline. Across real-world tool-use tasks, the resulting system shows strong performance and generalization while reducing reliance on robot demonstrations.

  • Dexterous manipulation is limited by costly robot demonstrations, heterogeneous hands, and high-dimensional control, while gripper-focused foundation policies do not cover many tool-use tasks.
  • UniDex-Dataset contains 9M paired image–pointcloud–action frames and over 50K trajectories spanning eight dexterous hands with 6–24 active DoFs.The dataset is derived from egocentric RGB-D manipulation videos and is designed to scale through protocols for adding hands or human datasets.
  • FAAS maps functionally similar actuators to shared coordinates, providing a unified control interface for skill transfer across different hands.UniDex-VLA uses FAAS as its action parameterization and is pretrained on UniDex-Dataset before task-specific finetuning.
  • UniDex-Cap records synchronized RGB-D streams and human hand poses, converting them into robot-centric trajectories for human–robot co-training with less teleoperation data.The setup supports combining transformed human data with smaller amounts of robot data while preserving performance.
  • 81% average task progress was achieved by UniDex-VLA across five real-world tool-use tasks on two hands, compared with 38% for π0.The evaluation also reports strong spatial, object and zero-shot cross-hand generalization.
  • The suite combines a scalable dexterous-hand dataset, a cross-hand VLA policy and a capture pipeline intended to reduce dependence on costly robot demonstrations.These components are presented as a foundation for universal dexterous-hand manipulation.

2. Related Work

Prior dexterous-manipulation research has advanced from analytic control to learning-based methods, but broad generalization remains limited by task, hardware, data and representation constraints. UniDex is positioned against gripper-centric foundation models and hand-specific or narrowly supervised dexterous approaches.

  • Earlier dexterous-manipulation methods enabled skills such as in-hand reorientation, rotation and grasping but often remained tailored to specific tasks or hardware.
  • Most VLA foundation models are pretrained on gripper-centric datasets, while dexterous VLA efforts commonly rely on simulation or limited real-world data and hand-specific representations.
  • Existing unified action-space approaches primarily target gripper-centric actions, leaving embodiment heterogeneity in dexterous hands insufficiently addressed.
  • Human videos can reduce data costs but introduce visual and kinematic domain gaps that must be handled when transferring demonstrations to robots.
  • Prior human-data pipelines include planning, retargeting, human-in-the-loop correction and robot-data co-training, but many target grippers or do not scale robustly.

3. UniDex-Dataset

UniDex-Dataset transforms egocentric human RGB-D manipulation data into robot-executable trajectories spanning diverse dexterous hands, tasks, and perception modalities. Its pipeline combines pointcloud processing with human-in-the-loop fingertip retargeting and iterative kinematic correction to improve hand–object contact plausibility.

  • Human-to-robot transformation: The transformation pipeline masks human hands from scene pointclouds, retargets human motion to robot hands, and attaches the resulting robot meshes to the scene.Human hands are removed to reduce the visual gap, while retargeted robot geometry is rendered into the pointcloud and reprojected to the RGB-D frame.
  • Dataset construction: UniDex-Dataset contains 9M paired image–pointcloud–action frames and over 50K trajectories across eight dexterous hands with 6–24 active DoFs.The dataset is derived from four open-source egocentric RGB-D human-manipulation datasets and supports diverse daily tasks.
  • Dataset construction: The dataset spans diverse daily manipulation tasks and compares favorably with existing datasets in trajectories, hand and scene variety, and perception modalities.Examples include mobile-phone use, milk-carton opening, spatula stir-frying, chair lifting, and Rubik’s-cube solving.
  • Kinematic retargeting: Kinematic retargeting aligns robot fingertip trajectories with human targets while allowing a global hand-base adjustment for more plausible hand–object contact.The process extracts fingertip targets, introduces a 6-DoF dummy-base offset, and minimizes stacked fingertip residuals through inverse kinematics.
  • Kinematic retargeting: The two-stage retargeting procedure combines automatic inverse kinematics with interactive adjustment of the dummy-base offset and other solver settings.Mimic joints receive iterative corrections after the primary IK solve, with fingertip error re-evaluated until convergence.

4. UniDex-VLA

UniDex-VLA uses FAAS to provide a function-centric action representation across heterogeneous dexterous hands, combining 3D observations, language, and proprioception for high-DoF control. Its policy predicts action chunks in FAAS using a pointcloud encoder and conditional flow matching.

  • Unified Action Space: FAAS: FAAS maps actuators with similar functional roles into shared coordinates, enabling skill transfer across different dexterous hands.It replaces embodiment-specific joint semantics with a function-centric control interface.
  • Unified Action Space: FAAS: FAAS represents actions as an 82-dimensional vector with wrist poses, shared base actuator slots, and hand-specific degrees of freedom.The vector contains 18 wrist-pose dimensions and 64 joint-command dimensions, including 21 shared base actuator slots.
  • VLA Policy: By coupling 3D visual inputs with FAAS, UniDex-VLA supports spatial, object, and cross-hand generalization.The unified representation aligns geometric perception and control across embodiments.
  • VLA Policy: At time t, UniDex-VLA consumes a colored pointcloud, language instruction, and proprioception, then predicts an H-step FAAS action chunk.Both proprioceptive states and dexterous-hand actions use FAAS representations, with relative wrist poses for action outputs.
  • VLA Policy: UniDex-VLA uses Uni3D to encode pointclouds, fuses visual, language, and proprioceptive features, and trains with conditional flow matching.The policy generates denoised action chunks at inference time.

5. Experiments

UniDex-VLA is evaluated on challenging real-world dexterous tool-use tasks, including spatial, object, and cross-hand generalization. UniDex-Cap is further assessed for combining transformed human demonstrations with robot data.

  • Experimental Setup: The benchmark evaluates five challenging real-world tool-use tasks across dexterous hands, using average task progress as the primary metric.Experiments use 50 demonstrations per task for fine-tuning.
  • Performance: 84.6% is UniDex-VLA’s largest relative improvement in average task progress over the best competing method, occurring on Use Scissors to Cut Bags.The authors associate pretraining with stronger motion priors for dexterous hand control.
  • Generalization: UniDex-VLA generalizes across spatial configurations and unseen objects, including a smaller purple kettle differing in color, size, handle, and spout.Pointcloud editing and DemoGen are used for out-of-distribution spatial placements.
  • Generalization: 60% success on Oymotion and 40% on Wuji are achieved zero-shot from a policy trained on Inspire Hand, while baselines are near zero.The transfer uses no fine-tuning and combines diverse-hand pretraining with FAAS.
  • UniDex-Cap: Approximately two transformed human demonstrations substitute for one robot demonstration, while human demonstrations are approximately 5.2× faster to collect on Make Coffee.Robot data remains indispensable, because success stays near zero without robot demonstrations.

6. Conclusion and Limitation

UniDex combines a dataset, VLA policy, and capture system into a foundation suite for scalable and transferable dexterous manipulation. The stated limitation is that large action-free or weakly labeled egocentric datasets are not yet used.

  • UniDex comprises UniDex-Dataset, UniDex-VLA, and UniDex-Cap as a robot foundation suite built from egocentric human videos.
  • The suite is presented as a practical foundation platform for accelerating general, scalable, and transferable dexterous manipulation.
  • UniDex does not yet leverage large action-free or weakly labeled egocentric activity datasets, which the authors identify as a direction for scaling pretraining.

A.1. UniDex-VLA Flow-Matching Loss

UniDex-VLA is trained with conditional flow matching, which constructs noisy action states and learns a conditional vector field for action generation.

  • UniDex-VLA minimizes a conditional flow-matching loss over action trajectories conditioned on observations.
  • The training process interpolates action trajectories with Gaussian noise and defines a target vector field as the difference between clean actions and noise.
  • At inference, UniDex-VLA integrates the learned vector field with forward Euler to generate a denoised action chunk.

A.2. UniDex-VLA Pretraining

The reported pretraining configuration uses eight NVIDIA H800 GPUs and a batch size of 128, followed by a three-epoch training schedule for the post-training model.

  • Pretraining uses 8 NVIDIA H800 GPUs with a total batch size of 128.
  • The subsequent post-training model runs for 3 epochs, approximately 30k steps, and takes around 24 hours.
  • Optimization uses AdamW, cosine learning-rate decay from 1e-4, weight decay 1e-10, and gradient clipping at norm 1.0.

A.3. UniDex-VLA Post-training

UniDex-VLA post-training uses separate optimization settings for the proposed model and comparison baselines, while a web GUI reduces human effort in retargeting demonstrations.

  • Training setup: UniDex-VLA post-training uses 2 NVIDIA H800 GPUs per task, batch size 8, AdamW, and a 2.5e-5 initial learning rate.Common data are trained for 50 epochs, while DemoGen-augmented data use 2 epochs.
  • Baseline training: DP, DP3, and π0 baselines are post-trained until convergence on the validation set.
  • Baseline training: DP and DP3 use a 1e-4 learning rate, batch size 32, 400 epochs, state horizon 4, and action horizon 32.
  • Baseline training: π0 uses a 2.5e-5 learning rate, batch size 8, 50 epochs, and 10 diffusion steps.
  • Ablation setup: The no-pretraining UniDex-VLA baseline uses the same training hyperparameters as pretrained UniDex-VLA.
  • Retargeting support: A web-based GUI lets users adjust dummy base links, inverse-kinematics parameters, and other settings to obtain satisfactory robot trajectories.

C. FAAS Details

This section details FAAS joint encoding, cross-hand joint mappings, and the calibrated sensor setup used to align captured human poses with robot-oriented point clouds.

  • FAAS encoding: FAAS encodes dexterous hand joints in 32 dimensions, allocating five five-dimensional finger blocks, two Shadow wrist dimensions, and five reserved dimensions.Dimensions 0–24 represent the thumb, index, middle, ring, and little fingers; dimensions 25–26 are Shadow wrist joints and 27–31 remain unused for new hands.
  • Retargeting interface: The retargeting GUI exposes inverse-kinematics parameters and dummy links so users can adjust robot trajectories interactively.
  • FAAS mappings: Joint mappings in FAAS cover Ability, Allegro, Inspire, Leap, Oymotion, Shadow, Wuji, and Xhand robotic hands.The figure presents two views of the right-hand mappings.
  • Capture calibration: UniDex-Cap combines Vision Pro hand and head poses with RealSense L515 RGB-D, using a rigid mount to stabilize their relative pose.A GUI estimates remaining extrinsics by aligning Vision Pro skeleton poses with RealSense 3D hand point clouds.
Loading 2603.22264v1…