Source-linked AI summary
DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection
Amber Xie, Jiawei Fu, Xiaofeng Guo, Carmelo Sferrazza, Guanya Shi, Haozhi Qi
TL;DR
Collecting executable dexterous-hand demonstrations is difficult because teleoperation needs a complete robot setup and human-hand data must be retargeted. DEXPORT uses the deployment hand with a skill-level portable interface and trains policies directly from recorded commands. Final-task success averages 64% versus 11%.
Problem
Teleoperation requires a complete robot setup and coordination of many finger joints, while human-hand demonstrations require motion estimation and retargeting and do not directly provide robot actions.
Source: §I · p. 1Method
DEXPORT collects with the deployment hand through a compact skill interface and trains policies to predict recorded wrist poses and raw hand joint targets without skill labels.
Source: §I · p. 2, C. Policy Learning · p. 4Results
Final-task success averages 64% versus 11%.
Source: §I · p. 2Takeaways & Limitations
The skills assist demonstration collection, while learned policies directly predict raw hand commands without skill or task-phase labels.
Source: C. Policy Learning · p. 4, §V · p. 7
Abstract
from arXiv · showhide
Collecting high-quality demonstrations remains a central bottleneck for learning dexterous manipulation. Teleoperation records executable robot actions, but it requires a complete robot setup and places the burden of coordinating many finger joints on the operator. Human demonstrations are easier to collect, but their motions must be estimated and retargeted to the robot, and they do not directly provide robot actions. In this paper, we introduce DexPort, a portable data-collection interface for high-DoF robot hands. It uses the same hand hardware for collection and deployment, avoiding a hand embodiment gap, and records both commanded joint targets and measured joint states. Our key observation is that a small set of skills can support a wide range of dexterous tasks. During collection, the operator moves the hand-held interface to specify the robot wrist motion and selects or adjusts skills that coordinate the finger motion. This removes the need to control each joint independently while retaining behaviors such as precise grasping and in-hand manipulation. We collect demonstrations across several real-world tasks and in unstructured environments without a robot arm. We then use behavior cloning to train visuomotor policies directly on the recorded wrist motions and hand commands, without skill labels. We evaluate across 4 tasks: pick-and-place with diverse objects, screwing with in-hand tool use, drawer stowage which requires simultaneous grasp and pulling, and a bimanual scanning task. We collect over 2,500 demonstrations with our portable device, in both laboratory and in-the-wild settings. Across our tasks, we see an improvement from 11% to 64% when using recorded robot commands instead of states, and an improvement from 36% to 78% for task success when using in-the-wild data.
I. INTRODUCTION
DEXPORT addresses the difficulty of collecting executable dexterous-hand demonstrations without a full robot setup or joint-by-joint operator control. It combines the deployed hand hardware with a skill-level interface and evaluates policies trained from its recorded demonstrations.
I. INTRODUCTION
- Teleoperation records executable robot actions but requires a complete robot setup and coordination of many finger joints, while human-hand demonstrations require motion estimation and retargeting.Human demonstrations also do not directly provide robot actions.Source: §I · p. 1
- Final-task success averages 64% versus 11%.Co-training with in-the-wild demonstrations further improves pick-and-place across the reported conditions.Source: §I · p. 2
- DEXPORT uses the same hand hardware for collection and deployment and records commanded joint targets alongside measured states.This preserves the hand embodiment and captures commands that can differ from measured states during contact.Source: §I · p. 2
- A compact set of parameterized and learned skills replaces joint-by-joint control while retaining dexterous behaviors.The operator specifies wrist motion and selects or adjusts skills that coordinate finger motion.Source: §I · p. 2
- The evaluation covers four real-world tasks and trains visuomotor policies directly on recorded wrist motions and hand commands without skill labels.Demonstrations include collection in unstructured environments.Source: §I · p. 2
II. RELATED WORK
Prior portable collection systems offer natural data collection but leave open how to control high-DoF hands while preserving deployment-hand actions. DEXPORT draws on hand-held interfaces and skill-level control to address that gap.
II. RELATED WORK
- Hand-held interfaces support portable collection, but parallel-jaw grippers offer only one actuated DoF, while dexterous hands require practical control of a larger action space.UMI combines a hand-held gripper, wide-field camera, visual-inertial tracking, and a policy interface that accounts for deployment latency.Source: §II · p. 2
- Dexterous learning from simulation, human demonstrations, and teleoperation has distinct data or deployment constraints.Simulation transfer remains difficult; human-hand data require estimation and retargeting; teleoperation generally needs a complete robot setup and operator coordination of many finger joints.Source: §II · p. 2
- Prior skill-level controllers reduce operator burden, but do not address portable collection with a high-DoF hand.HATO maps controller inputs to coordinated finger motions, while DexScrew uses a simulation-trained rotation policy as a teleoperation skill.Source: §II · p. 3
- Portable dexterous interfaces based on passive mechanisms record hand configurations rather than both commands and resulting states of the deployed hand.RealDexUMI records robot commands using the deployment hand but demonstrates the approach on a hand with six active DoFs; DEXPORT targets 20 DoFs.Source: §II · p. 3
III. METHOD
The hardware design uses a transferable hand-camera module and a controller-based collection handle. Reusing the module across collection and deployment preserves the relationship between the hand, camera, and robot.
A. Hardware Design
- A transferable module combines a five-fingered, 20-DoF Wuji hand and fisheye camera, mounting to the collection handle during recording and to the robot wrist during deployment.The flange can be adapted to other hand-camera pairs.Source: A. Hardware Design · p. 3
- Reusing the same module preserves hand embodiment and camera-to-hand extrinsics, allowing recorded hand targets to transfer without retargeting.The rigid connections keep the controller-to-hand and camera-to-hand transforms fixed.Source: A. Hardware Design · p. 3
- The Quest 2 controller provides wrist motion targets while the camera records task context and the hand controller records commanded joint targets and resulting states.The recorded hand actions are the commands sent to the hand rather than actions inferred from measured states.Source: A. Hardware Design · p. 3
- The collection handle and strap support the assembly while leaving the thumb and index finger free to operate controls.The strap carries part of the load and reduces sustained grip force; positioning the controller near the center of mass shortens the moment arm and reduces wrist torque.Source: A. Hardware Design · p. 3
B. Dexterous Control via Parameterized Skills
DEXPORT uses parameterized skills to translate operator intent into coordinated finger motion on a high-dimensional hand. The operator controls wrist motion while selecting or adjusting skills for grasping and in-hand manipulation.
B. Dexterous Control via Parameterized Skills
- Parameterized skills separate operator intent from the finger motions needed to realize it, replacing direct joint control with adjustable commands.For example, the joystick incrementally opens or closes the grasp rather than selecting a single fixed grasp.Source: B. Dexterous Control via Parameterized… · p. 4
- The interface keeps all 20 hand joints available while using Quest 2 buttons, triggers, and a joystick to select and modulate skills.Thumb control positions the thumb independently; power grasp, pinch, and two-finger grasp coordinate different finger groups.Source: B. Dexterous Control via Parameterized… · p. 4
- A learned closed-loop skill generates the repeated finger gaiting used to rotate a grasped tool while the operator controls global hand motion.The skill is trained in simulation to relieve the operator of difficult low-level in-hand manipulation, particularly without finger-level haptic feedback.Source: B. Dexterous Control via Parameterized… · p. 4
- The rotation policy learns delta actions over 12 actuated joints with the ring and pinky fingers fixed in a simplified fixed-base simulation.It observes a three-step history of measured joint positions, commanded targets, and previous action; the privileged critic accesses handle rotation angle.Source: B. Dexterous Control via Parameterized… · p. 4
- The simulation training uses domain randomization over object and control properties and runs the actor at 20 Hz on hand joint encoders.Randomized factors include mass, center of mass, friction, restitution, finger PD gains, initial grasp, and disturbance forces.Source: B. Dexterous Control via Parameterized… · p. 4
C. Policy Learning
Skills structure demonstration collection, while policies learn directly from wrist motions and commanded hand targets without skill labels. Deployment calibrates the controller-to-end-effector transform for wrist motion; hand targets need no conversion.
C. Policy Learning
- The policy predicts commanded 20-DoF hand targets and wrist poses, without skill identities, parameters, or task-phase labels.The state-as-action baseline changes only the hand action labels from commanded targets to measured states.Source: C. Policy Learning · p. 4
- A diffusion policy uses two recent observations and predicts 16-step chunks of wrist poses and absolute hand targets.Each action step has 9 wrist dimensions and 20 hand dimensions; bimanual inputs and outputs are concatenated without changing the architecture.Source: C. Policy Learning · p. 4
- Deployment estimates the fixed controller-to-end-effector transform from paired poses and applies it to wrist actions; hand targets require no conversion.The transform is fit by least squares from pairwise constraints across N pose samples.Source: C. Policy Learning · p. 5
IV. EXPERIMENTS
The experiments evaluate DEXPORT on four dexterous tasks using laboratory and in-the-wild demonstrations, then compare commanded-target policies, co-training, and collection throughput. Task phases support partial-success scoring, and each policy is evaluated for 15 episodes per condition.
A. Experimental Setup
- The evaluation covers vegetable pick-and-place, screwdriver rotation, drawer stowing while retaining an object, and bimanual barcode scanning.The tasks test grasp selection, sustained contact and finger gaiting, articulated-object manipulation, and coordinated two-hand positioning.Source: A. Experimental Setup · p. 5
- The dataset contains 1,605 laboratory demonstrations across four tasks and 1,000 additional pick-and-place demonstrations from apartment and office environments.Co-trained pick-and-place policies use both laboratory and in-the-wild data; comparisons include commanded targets versus measured-state labels and lab-only versus co-trained policies.Source: A. Experimental Setup · p. 5
- Fig. 4 annotates successful task rollouts with phases used to score partial success and grasp types observed during execution.The grasp annotations describe observed behavior; policies predict raw joint targets without skill labels.Source: Figure 4 · p. 6
- The figure reports imitation-learning performance across four tasks, with recorded hand-command policies outperforming the state-as-action baseline in success rates and screwdriver rotation speed.Each policy is evaluated for 15 episodes per task; drawer-stowing and scanning phase success requires all preceding phases to succeed.Source: Figure 5 · p. 6
B. Policy Learning Across Dexterous Tasks (Q1)
Policies trained on DEXPORT demonstrations acquire the target behaviors across the four dexterous tasks, while the state-as-action baseline performs worse on every task. Its failures include weak grasps, dropped objects, and no completed barcode scans.
B. Policy Learning Across Dexterous Tasks (Q1)
- Policies trained on DEXPORT demonstrations acquire all target behaviors, including grasp selection, drawer pulling, in-hand screwdriver rotation, and bimanual scanning.The policies select pinch and power grasps without grasp labels.Source: B. Policy Learning Across Dexterous… · p. 6
- The state-as-action baseline is worse on every task, and never completes a bimanual barcode scan.It grasps the first object in 80% of episodes and holds both in 13%.Source: B. Policy Learning Across Dexterous… · pp. 6–7
- On the screwdriver task, the baseline holds the tool for 31.7 s versus 15.6 s for our policy, yet rotates it at a third of the policy’s rate and is much less effective.Source: B. Policy Learning Across Dexterous… · p. 7
C. Co-Training with In-the-Wild Data (Q2)
Co-training with in-the-wild demonstrations improves pick-and-place full success across the evaluated conditions. With 1,000 in-the-wild demonstrations combined with 700 laboratory demonstrations, gains also appear on unseen objects and seen objects.
C. Co-Training with In-the-Wild Data (Q2)
- The in-the-wild dataset varies scenes, backgrounds, and lighting while using the same training objects as the laboratory data.The portable collection required neither a robot arm nor external tracking infrastructure.Source: C. Co-Training with In-the-Wild Data… · p. 7
- With 1,000 in-the-wild demonstrations, full success rises from 40% to 87% under distractors and from 20% to 73% on unseen objects, showing transfer to new objects.Full success on seen objects rises from 47% to 73%.Source: C. Co-Training with In-the-Wild Data… · p. 7
D. Data Collection Throughput (Q3)
DEXPORT collects trajectories faster than the teleoperation interfaces compared in the pick-and-place throughput evaluation. The conclusion also identifies device weight and incomplete skill coverage as constraints.
D. Data Collection Throughput (Q3)
- DEXPORT collects roughly 1.4× as many trajectories as Glove Teleoperation and 1.8× as many as Skill Teleoperation.The comparison counts trajectories during 30 minutes of continuous operation on unimanual pick-and-place; DEXPORT collects 122 trajectories per 30 minutes.Source: D. Data Collection Throughput (Q3) · p. 7, §V · p. 7
- Human hand data collection is about 2.3× faster than DEXPORT, but is treated as a speed upper bound rather than a competing training-data source.It lacks the deployed hand’s embodiment and commanded actions.Source: D. Data Collection Throughput (Q3) · p. 7
V. CONCLUSIONS
- Device weight limits uninterrupted use and rotation-heavy motions, and the current skills cover only part of dexterous behaviors.The paper points to lighter, smaller hands and additional skills as future directions.Source: §V · pp. 7–8
APPENDIX
In-the-wild demonstrations improve pick-and-place success in several settings, while the appendix details the data-collection setup and simulated in-hand rotation skill.
A. Scaling In-the-Wild Data
- Because the in-the-wild demonstrations reuse laboratory objects, they diversify scenes rather than object geometry; unseen-object transfer may require more laboratory demonstrations.The authors hypothesize that transferring robustness to new objects requires a sufficiently large laboratory dataset.Source: A. Scaling In-the-Wild Data · p. 9
B. Hardware and Data Collection
- Collection uses a 30 Hz loop for camera, hand, and controller streams, with latencies assumed to roughly match and no post-hoc temporal alignment.At deployment, asynchronous inference discards actions whose time has passed.Source: B. Hardware and Data Collection · p. 9
A. Scaling In-the-Wild Data
- In-the-wild data improves full success on seen objects and distractor scenes in both data regimes, but unseen-object success only with 700 lab demos.The table caption identifies the evaluations as seen objects, unseen objects, and scenes containing distractors.Source: Table III · p. 9, A. Scaling In-the-Wild Data · p. 9
C. In-Hand Rotation Skill
- The in-hand rotation skill trains in simulation and controls 12 joints, fixing the ring and pinky fingers while updating targets in bounded increments.The simulated screwdriver is a 12-sided prism; the policy observes recent joint positions and targets, among other inputs, and acts at 200 Hz simulation frequency with 10 physics steps per policy step.Source: C. In-Hand Rotation Skill · pp. 9–10, Table IV · p. 9
D. Policy Learning
The policy-learning section represents wrist motion as relative poses and hand actions as absolute targets; deployment converts predicted wrist motions into robot-arm commands.
D. Policy Learning
- Predicted wrist actions are relative poses, while hand actions are absolute 20-D targets; bimanual policies concatenate both hands’ observations and actions.The end-effector pose uses a 3-D translation and 6-D rotation relative to the current end-effector frame.Source: D. Policy Learning · p. 10
E. Deployment
- Deployment calibrates the controller-to-flange transform and converts predicted wrist motions into flange targets using inverse kinematics.Inference runs asynchronously at 30 Hz; each new 16-step chunk keeps its first 8 steps, drops elapsed steps, and averages overlapping commands.Source: E. Deployment · p. 10