Source-linked AI summary
DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation
Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso, Shuran Song
TL;DR
DexUMI addresses the challenge of transferring dexterous human-hand skills across robot hands with different embodiments. It combines a wearable exoskeleton and robot-hand video inpainting to align action, tactile, and visual data, achieving 86% average success across four tasks.
Problem
Dexterous skill transfer is limited by embodiment differences across human and robot hands and by diverse robot-hand hardware designs.
Method
DexUMI combines a robot-specific wearable exoskeleton for feasible motion and haptic data collection with software that replaces human-hand video content using corresponding robot-hand inpainting.
Results
86% average success rate was achieved across four tasks on two dexterous robot hand platforms.
Takeaways & Limitations
DexUMI supports dexterous policy learning across diverse manipulation scenarios, including long-horizon tasks and coordinated multi-finger contact.
Takeaways & Limitations
The exoskeleton optimization still requires hardware-specific tuning, especially for wearability, and the visual pipeline requires robot hardware to obtain robot-hand images.
Abstract
from arXiv · showhide
We present DexUMI - a data collection and policy learning framework that uses the human hand as the natural interface to transfer dexterous manipulation skills to various robot hands. DexUMI includes hardware and software adaptations to minimize the embodiment gap between the human hand and various robot hands. The hardware adaptation bridges the kinematics gap using a wearable hand exoskeleton. It allows direct haptic feedback in manipulation data collection and adapts human motion to feasible robot hand motion. The software adaptation bridges the visual gap by replacing the human hand in video data with high-fidelity robot hand inpainting. We demonstrate DexUMI's capabilities through comprehensive real-world experiments on two different dexterous robot hand hardware platforms, achieving an average task success rate of 86%.
1 Introduction
DexUMI addresses the embodiment gap between human and robot hands with hardware and software adaptations that support universal human-hand interfaces for diverse robot hands. Real-world experiments across two robot hands and four tasks achieved 86% average success and 3.2× teleoperation data-collection efficiency.
- DexUMI targets embodiment gaps in kinematics, contact surfaces, tactile information, and visual appearance between human and robot hands.
- The framework uses hardware and software adaptations to minimize action and observation gaps across diverse robot hands.
- The wearable exoskeleton enables direct haptic demonstrations, feasible robot-compatible motion, precise encoder-based joint capture, and matching fingertip tactile sensing.
- The software pipeline removes the human hand and exoskeleton from videos, then inpaints corresponding robot-hand and environment backgrounds for consistent visual inputs.
- 86% average success rate was achieved across four tasks on Inspire and XHand platforms, while data collection was 3.2 times more efficient than teleoperation.
2 Related Work
Prior real-world dexterous manipulation methods include teleoperation, human-hand video learning, and wearable devices, but each has constraints in transfer, hardware requirements, or robot-hand generality. DexUMI is positioned as a robot-independent, multi-fingered data-collection interface.
- The related work review focuses on real-world data-collection methods rather than simulated-environment learning.
- Teleoperation uses gloves, virtual-reality devices, or camera tracking, but retargeting struggles with morphological differences, especially thumb flexibility.
- Teleoperation and kinesthetic teaching require robot hardware, whereas DexUMI collects manipulation data without physical robots.
- Human-hand video methods often require additional real-world robot data or simulation with privileged information for real-world deployment.
- Wearable data-collection systems have shown promise but primarily target simple parallel or pinch grippers and are difficult to adapt to multi-fingered systems.
3 Hardware Adaptation to Bridge the Embodiment Gap
DexUMI’s hardware adaptation combines robot-specific wearable exoskeleton optimization with sensors that capture robot-compatible actions, wrist motion, vision, and tactile information. Its visual adaptation transforms human demonstrations into robot-hand videos through segmentation, inpainting, and action replay.
- Hardware adaptation: The hardware adaptation uses a wearable exoskeleton to adapt human motion into feasible robot actions through mechanism optimization and sensor integration.
- Mechanism design optimization: The exoskeleton optimization preserves target robot fingertip and joint-action mappings while imposing wearability constraints.
- Mechanism design optimization: The design prioritizes exact fingertip-link kinematics while allowing more flexibility in links less likely to contact objects.
- Mechanism design optimization: The objective compares sampled exoskeleton and robot fingertip workspaces in SE(3), while constraining the exoskeleton workspace to remain within robot capabilities.
- Sensor integration: Joint encoders, wrist tracking, cameras, and matched tactile sensors are integrated to capture actions and observations with reduced embodiment shift.
- Sensor integration: A 150° wrist-mounted camera is placed identically on the exoskeleton and robot hand to maintain visual consistency between training and deployment.
- Software adaptation: The visual pipeline segments the human hand and exoskeleton, inpaints the background, replays joint actions on the robot hand, and composites visible robot-hand pixels into the demonstration.
5 Evaluation
DexUMI is evaluated across two dexterous robot hands and four real-world tasks using comparisons of action representations, tactile sensing, and visual rendering. Results show robust policy learning, benefits from relative trajectories and task-dependent tactile feedback, and substantially higher collection efficiency than teleoperation.
- Evaluation Setup and Overall Performance: DexUMI achieves high success rates across four tasks on two robot hands, including precise, long-horizon, and coordinated multi-finger manipulation.The evaluation uses the Inspire Hand and XHand, with 20 episodes per task and stage-wise accumulated success rates for long-horizon tasks.
- Action Representation: Relative finger trajectories consistently achieve better success across all tasks and are more robust to noise and hardware imperfections.Relative actions can accumulate corrective deltas until contact events, whereas absolute mappings may stall when errors occur.
- Tactile Sensing: Relative trajectories are the only action representation that benefits from noisy tactile feedback, while absolute trajectories suffer larger performance drops.This pattern holds despite tactile degradation from high-pressure drift on XHand and even noisier manually installed sensors on Inspire.
- Tactile Sensing: Tactile feedback benefits tasks with clear force profiles and limited visual information, but noisy sensors can reduce performance elsewhere.On salt picking with XHand, tactile input changes behavior toward inserting fingers before closing; it does not help tweezer manipulation, and noisy Inspire sensors worsen results.
- Data Collection Efficiency: DexUMI collects 3.2 times more demonstrations than traditional teleoperation within 15-minute sessions, although bare-hand manipulation remains faster.Collection throughput is measured by the number of successful demonstrations on the tea-picking-with-tool task.
- Evaluation Takeaway: The framework combines hardware and software adaptations to support efficient, precise, contact-rich, and long-horizon dexterous policy learning.The evaluation findings support DexUMI’s broader goal of collecting real-world dexterous hand data beyond traditional teleoperation.
7 Limitation and Future Work
DexUMI’s limitations span hardware-specific exoskeleton design, incomplete contact modeling, wearability, robot-hand precision, and size compatibility. Future work targets automated co-design, broader contact modeling, improved materials, and more reliable hardware and visual adaptation.
- Hardware adaptation: Hardware-specific tuning remains necessary for each robot hand, particularly to achieve wearability.The authors propose automating optimization from robot-hand and human-hand descriptions.
- Hardware adaptation: The current exoskeleton optimization matches fingertip workspaces but does not model other contact geometries such as the palm.The authors identify broader contact modeling as a future direction.
- Hardware adaptation: Wearability is constrained by target-hand design and 3D-printing material strength, motivating soft-material integration.TPU is suggested for parts contacting the human hand.
- Software adaptation: Software adaptation still requires real-world robot-hand images and can produce blurred regions or imperfect illumination reproduction.The authors propose image generation conditioned on motor values to remove the hardware-image requirement.
- Existing robot hand hardware: Robot-hand precision is limited by backlash and friction, producing direction-dependent fingertip positions and discrepancies in inpainting and action mapping.The same commanded motor value can yield different fingertip positions depending on motion direction.
- Existing robot hand hardware: Robot-human size differences can prevent the human hand and exoskeleton from reaching configurations required by a much larger robot hand.The authors suggest reverse co-design beginning with a comfortable, operable exoskeleton.
A.1 Inpainting Results
The software adaptation replaces human-hand and exoskeleton pixels with robot-hand imagery while preserving hand-object occlusions. The action-distribution comparison shows relative actions are unimodal, whereas absolute actions are multimodal.
- Inpainting: Software adaptation removes the human hand and exoskeleton, then inpaints corresponding robot-hand and background pixels for policy training.The replacement preserves natural occlusion relationships during hand-object interactions.
- Inpainting: SAM2 usually segments the human hand and exoskeleton effectively, but can miss small exoskeleton regions.Segmentation errors are one source of visual-adaptation deficiencies.
- Inpainting: Flow-based inpainting produces high overall quality, but some image regions remain blurry.Gaussian blur augmentation is used during policy training to reduce sensitivity to this artifact.
- Inpainting: The robot hand’s illumination cannot be fully reproduced in the inpainted output.This remains a reported deficiency despite generally good inpainting quality.
- Action distribution: Relative thumb-swing actions form a simple unimodal distribution, while absolute actions are multimodal in the Kitchen task.This distributional difference accompanies the paper’s comparison of relative and absolute action representations.
B Evaluation Details
Evaluation uses shared, manually selected initial states and task-specific success criteria across dexterous manipulation tasks. The implementation combines motion capture, URDF-based design, sensor-derived actions, and hardware-specific execution handling.
- Evaluation setup: Initial states are manually selected, diversified, shared across methods, and recorded with an additional side camera for consistency.Evaluation overlays recorded and current initial-state images to guide setup adjustment.
- Evaluation setup: Some Inspire-Hand initial states are infeasible for XHand because differing wrist-camera placement can prevent object visibility.A tea-cup position more than 45° left of the tea pot is given as an example.
- Evaluation setup: Kitchen evaluation maximizes seasoning-bowl diversity within the kinematically feasible workspace of a fixed-base single UR5.The large workspace and fixed stove and knob positions constrain coverage.
- Success criteria: Success criteria cover cube placement, egg-carton opening beyond 30°, tool and leaf picking, and sequential kitchen subtasks.Later subtasks automatically fail when an earlier required subtask fails.
- Policy execution: The policy predicts 16 future actions but executes only the first 8 at 10 Hz, with interpolation to hardware frequencies.The UR5 runs at 125 Hz, Inspire Hand at 10 Hz, and XHand at 60 Hz.
- Policy execution: Relative hand actions add the predicted delta to the current motor position, while absolute actions send the predicted motor value directly.The relative end-effector action is similarly converted to an absolute UR5 position.
- Policy execution: A virtual XHand motor position improves grasping precision by accumulating executed relative actions despite hardware drift.The virtual position is initialized from hardware at evaluation start and then updated without repeated direct reads.
- Exoskeleton design: Inspire-Hand exoskeleton design uses motion-captured fingertip poses because its closed-loop four-bar kinematics cannot be directly represented in URDF.Sampled motor commands provide fingertip poses in flange coordinates for subsequent optimization.
E.1 Data Collection
Data collection uses task-specific trajectory counts and records multimodal demonstrations from exoskeleton-equipped operators. Recording rates differ between hands because higher-frequency XHand tactile readings become unstable.
- Trajectory collection: 310 Cube Picking, 175 Egg Carton Opening, and 400 Tea Picking Using Tools trajectories were collected for policy training.The 400 Tea Picking trajectories were collected for both Inspire Hand and XHand.
- Trajectory collection: The kitchen dataset contains 370 trajectories covering all four subtasks plus 100 additional knob-closing trajectories.These counts describe the collected training data for the kitchen task.
- Sensing and recording: Inspire-Hand demonstrations record wrist position, visual observations, encoder joint angles, and tactile feedback at 45 FPS.Each data type includes its receive timestamp in the recording buffer.
- Sensing and recording: XHand demonstrations are recorded at 30 FPS because tactile readings become unstable at higher recording frequencies.The same multimodal data streams are recorded with receive timestamps.
- Data collection setup: Green gloves and green PLA-CF exoskeletons provide consistent color that improves SAM2 segmentation during data collection.The color choice is an operational adaptation for visual preprocessing.
E.2 Training Data Latency Management
DexUMI aligns multimodal training data by correcting sensor and encoder timing differences before interpolation to camera timestamps.
- E.2 Training Data Latency Management: tcapture = treceive − l_sensor computes the actual sensor capture time by subtracting capture-to-receive latency from receive time.The latency term is specific to each sensor.
- E.2 Training Data Latency Management: Camera latency is measured as l_camera = treceive − t_display − l_display using a rolling QR code and monitor refresh timing.The method measures both iPhone and OAK camera latency.
- E.2 Training Data Latency Management: Encoder latency is tuned by overlaying recorded exoskeleton images with robot-hand action replay until the fingers align.Excessive latency makes robot fingers lead; insufficient latency makes them lag.
- E.2 Training Data Latency Management: After timestamp adjustment, joint angles and tactile readings are linearly interpolated to produce samples aligned with camera timestamps.This places visual, tactile, and action data on a common timeline.
E.3 Policy Training
The policy combines pretrained visual features with tactile readings and predicts short-horizon robot actions for end-effector and dexterous-hand control.
- E.3 Policy Training: DINO-V2 CLS visual features are concatenated with tactile readings as input to the diffusion policy.Visual inputs receive random crop, color jitter, grayscale, and Gaussian Blur augmentation before DINO-V2.
- E.3 Policy Training: The diffusion policy predicts 16 steps of robot actions containing 6-DoF relative end-effector actions plus hand actions.Hand actions use 6-DoF for Inspire Hand and 12-DoF for XHand.