PaperlayerView exact revision on arXiv

Source-linked AI summary

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, C. Karen Liu

arXiv:2403.07788v2cs.ROcs.AIcs.CVcs.LG

TL;DR

DexCap and DexIL address the difficulty of collecting portable, precise human hand data and converting it into dexterous robot policies. DexCap combines real-time wrist and finger tracking with 3D observations, while DexIL uses retargeting and point-cloud imitation learning, with optional human correction. Across the reported evaluations, point-cloud and diffusion-policy variants achieved stronger task performance than image-based or MLP-based alternatives, and the system learned from in-the-wild mocap data.

  • Problem

    Existing hand mocap systems lack portability or robust, precise 3D hand and environment data, limiting direct low-level robot learning from human motion.

  • Method

    DEXCAP collects real-time wrist and finger motion with SLAM and EMF sensing plus RGB-D observations, while DEXIL retargets it with inverse kinematics and trains point-cloud-based imitation policies.

  • Results

    Point-cloud methods exceeded 60% task success, while diffusion policy achieved 25% higher averaged task success than MLP-based BC-RNN policies.

  • Takeaways & Limitations

    The system demonstrates dexterous robot learning directly from human mocap data, including in-the-wild collection, with optional human correction for further performance improvement.

  • Takeaways & Limitations

    The paper’s demonstrated scope centers on six challenging dexterous manipulation tasks and reports hardware and code as open-source for future scaling.

Abstract

from arXiv · show

Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, substantial challenges persist, particularly with the portability of existing hand motion capture (mocap) systems and the complexity of translating mocap data into effective robotic policies. To tackle these issues, we introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data. DexCap offers precise, occlusion-resistant tracking of wrist and finger motions based on SLAM and electromagnetic field together with 3D observations of the environment. Utilizing this rich dataset, DexIL employs inverse kinematics and point cloud-based imitation learning to seamlessly replicate human actions with robot hands. Beyond direct learning from human motion, DexCap also offers an optional human-in-the-loop correction mechanism during policy rollouts to refine and further improve task performance. Through extensive evaluation across six challenging dexterous manipulation tasks, our approach not only demonstrates superior performance but also showcases the system's capability to effectively learn from in-the-wild mocap data, paving the way for future data collection methods in the pursuit of human-level robot dexterity. More details can be found at https://dex-cap.github.io

I. INTRODUCTION

DexCap and DexIL address the hardware and algorithmic barriers to learning dexterous robot manipulation directly from human hand motion. DexCap provides portable, precise 3D mocap data, while DexIL retargets that data and trains robot policies with optional human correction.

  • Human-level dexterous manipulation remains difficult because everyday environments suit human hands while robotic systems face hardware and algorithmic challenges.
  • Vision-based hand tracking is difficult to scale because single-view systems may lack accurate 3D information and can suffer from occlusions.
  • Mocap offers accurate, occlusion-robust 3D hand data without requiring robot hardware, but scalable systems also need portability, long capture sessions, precise wrist and finger poses, and environment observations.
  • DEXCAP is a portable system that tracks wrist 6-DoF poses and finger motions in real time at 60Hz while recording RGB-D observations.
  • DEXIL addresses the human–robot embodiment gap by retargeting mocap data with inverse kinematics and training point-cloud-based imitation policies, with optional human-in-the-loop correction.
  • The work contributes DEXCAP, DEXIL, and a correction mechanism intended to enhance robot performance in complex dexterous tasks.

II. RELATED WORKS

Prior work spans dexterous control, hand mocap, and imitation learning, but existing approaches leave gaps in portability, fine-grained 3D tracking, and direct low-level learning from human data. This work targets multi-finger dexterous manipulation with a portable mocap system and compatible learning pipeline.

  • Traditional optimal control often simplifies contacts, while reinforcement learning has demonstrated complex dexterous behaviors primarily in simulation.
  • Hand motion capture: Camera-based hand mocap degrades under occlusion and may require nonportable calibrated cameras, whereas EMF gloves miss grounded 6-DoF palm poses and visual observations.
  • Robot learning with human demonstration: Human-motion imitation data can avoid robot hardware, but 2D human motions cannot directly train 6-DoF policies in 3D environments and often require additional teleoperation data.
  • Existing low-cost hand-held gripper systems target parallel-gripper collection, while this work targets multi-finger hand motion for tasks such as scissor use and bottle-cap unscrewing.
  • The proposed system combines portable human mocap with a bimanual dexterous robot setup for evaluating policies learned from collected data.

A. DexCap

DexCap is designed to collect detailed, calibrated hand motion and 3D environmental observations outside the laboratory. Its data can be retargeted to dexterous robot hands, with optional online correction for policy behavior.

  • DEXCAP targets detailed finger tracking, accurate 6-DoF wrist estimation, aligned 3D observations, and portability for real-world data collection.
  • EMF gloves track fingertip 3D locations relative to a glove-mounted receiver, providing robustness to visual occlusions during hand–object interaction.
  • The system estimates wrist poses with camera-based tracking and records chest-mounted RGB-D LiDAR observations for policy inputs.
  • Tracking cameras are calibrated in rack slots, moved to glove mounts, and transformed into the chest-camera observation frame using a constant initial transformation.
  • DEXCAP runs from a backpack-mounted Intel NUC and 40000mAh power bank for approximately 40 minutes, with a total backpack weight of 3.96 pounds.
  • The robot setup uses two Franka arms with four-fingered, 16-joint LEAP hands and reuses the human data-collection camera through a quick-release mounting system.
  • DexIL constructs point clouds, retargets hand motion with fingertip IK, learns future goal-position actions, and can fine-tune with sampled human corrections.

IV. LEARNING ALGORITHM: DEXIL

DEXIL trains dexterous robot policies from human hand mocap by retargeting motions into the robot embodiment and learning action trajectories from point-cloud observations. The approach addresses high-dimensional bimanual control with a diffusion policy.

  • DEXIL retargets human hand motion into the robot’s action and observation spaces before policy training.The framework uses inverse kinematics to map human motion to the robot embodiment.
  • A. Data re-targeting: Inverse kinematics matches robot fingertips to human fingertip positions while using the captured wrist pose to initialize wrist retargeting.The retargeted robot state contains wrist pose and finger joint positions, with the next future state used as the action label.
  • B. Observation post-processing: DEXIL converts RGB-D observations into consistent-world-frame point clouds to accommodate the moving camera during natural human motion capture.Point clouds are formed using the LiDAR camera parameters and provide the policy with 3D environmental observations.
  • B. Point cloud-based diffusion policy: The policy processes point-cloud observations and robot proprioception to generate a future action trajectory over d steps.Input point clouds are uniformly down-sampled before being combined with RGB information.
  • B. Point cloud-based diffusion policy: A diffusion policy decodes high-dimensional actions for two 7-DoF arms and two 16-DoF hands, outperforming a traditional MLP architecture empirically.The action output spans d steps, creating a high-dimensional regression problem in the bimanual setup.

C. Human-in-the-loop correction

DEXIL supplements policy execution with human-in-the-loop correction through residual actions and direct teleoperation. Corrected rollouts are stored and mixed with the original data to fine-tune the policy.

  • The retargeting method does not fully resolve embodiment differences, because fingertip or joint matching may not reproduce the forces required for tool use.For scissors, stable holding requires inserting fingers deeply into the handle, but differing finger proportions can prevent equivalent force application.
  • Residual correction: Residual correction applies measured human hand-position changes to policy actions during rollout, scaled by separate position and joint factors.This mode is intended for real-time corrective actions while the policy remains active.
  • Residual correction: A small joint-motion scale β < 0.1 provides the best user experience by avoiding excessively fast finger movements.The reported setting concerns the correction mechanism’s joint component.
  • Teleoperation: Teleoperation switches the system from policy rollout to direct human wrist control, with robot fingertips following human fingertips through inverse kinematics.A foot-pedal press activates teleoperation, and users can return to residual correction after fixing an error.
  • Policy fine-tuning: Because corrections usually affect only a small portion of a rollout, the corrected dataset reduces human effort during policy refinement.The corrected actions and observations are stored in D′, which is sampled equally with the original dataset D for fine-tuning.

V. EXPERIMENTS

The experiments evaluate DEXIL and DEXCAP across six dexterous manipulation tasks and address data quality, direct policy learning, architecture choices, in-the-wild data, correction, and bimanual difficulty. The task suite spans simple coordination through long-horizon and tool-use behaviors.

  • The experiments ask whether DEXCAP data is high quality and whether DEXIL can learn policies without on-robot data.These questions define the evaluation of the data collection system and direct imitation approach.
  • The evaluation also examines critical model architecture choices, learning from in-the-wild DEXCAP data, and the value of human-in-the-loop correction.The in-the-wild data question concerns whether performance can be learned from data collected across varied scenes.
  • Tasks: Six tasks range from sponge picking, ball collecting, and plate wiping to packaging, scissor cutting, and tea preparing.The suite includes single-handed coordination, dual-handed coordination, generalization, precise tool use, and long-horizon behavior.
  • Tasks: Packaging evaluates bimanual manipulation and generalization with familiar and new objects.One hand places items in a box while the other pushes or stabilizes them before the lid is closed.
  • Tasks: Scissor cutting tests human-in-the-loop correction for precise tool use, while tea preparing tests a long-horizon sequence of intricate actions.Tea preparing includes grasping and uncapping a bottle, extracting tea with tweezers, and pouring it into a pot.

A. Experiment setups

The experiments evaluate DEXIL across six dexterous manipulation tasks, compare sensing and policy-design choices, and test learning from both standard and in-the-wild DEXCAP data. They also assess human-in-the-loop correction and performance on challenging long-horizon tasks.

  • Tasks: DEXIL is evaluated on six tasks spanning single-hand, dual-hand, bimanual, object-generalization, precise tool-use, and long-horizon manipulation.The tasks include Sponge picking, Ball collecting, Plate wiping, Packaging, Scissor cutting, and Tea preparing.
  • Data: The evaluation uses DEXCAP demonstrations and human-in-the-loop correction data, with in-the-wild data collected across more than 10 scenes.Data were recorded at 60Hz and downsampled to 20Hz for robot control; correction data were collected directly at 20Hz.
  • Baselines: The baselines compare BC-RNN with diffusion policy, image inputs with point clouds, and PointNet with Perceiver encoders.These comparisons target imitation-learning architecture, observation type, and point-cloud encoding.
  • Metrics: Each model variant is tested for 20 randomized trials per task, while Packaging uses 5 trials for each of 6 trained and 9 unseen objects.Packaging therefore includes 30 trained-object trials and 45 unseen-object trials.
  • System evaluation: DEXCAP provides 3D hand-motion capture that remains robust under occlusion, while Fig. 6 visualizes human-to-robot retargeting across multiple views.The retargeted robot hand motions use fingertip inverse kinematics to align with human motions in the scene point cloud.
  • Results: DEXIL’s point-cloud policies exceed 60% task success, and DEXCAP data alone reaches up to 72% average success on selected tasks.The reported 72% average covers Sponge picking, Ball collecting, and Plate wiping.
  • Results: In-the-wild training achieves 70% Subtask and 47% full-task performance, while human correction improves the point-cloud approach by 10%.Correction fine-tuning has only a minor effect on unseen objects with 30 trials of correction data.
  • Results: The framework reaches 45% success for picking up scissors, 20% for cutting tape, and 25% for the full Tea preparing task after correction.For Tea preparing, 30 human corrections improve uncapping success by 35%.

VI. CONCLUSION AND LIMITATIONS

The paper presents DEXCAP and DEXIL for learning dexterous manipulation directly from human mocap data, including optional human-in-the-loop correction. It reports challenging-task demonstrations and identifies hardware, embodiment, and sensing limitations for future work.

  • DEXCAP captures fine-grained 3D hand motion, overcomes occlusions, records RGB-D observations, and supports data collection outside the lab.
  • DEXIL uses DEXCAP data to teach robots complex dexterous manipulation tasks, with optional human-in-the-loop correction to improve performance.
  • The framework demonstrates dexterous performance on tasks including scissor cutting and tea preparation.
  • The authors state that all hardware designs and code will be open-source to support future scaling of dexterous manipulation data collection.
  • DEXCAP’s power consumption currently limits collection sessions to at most 40 minutes.
  • DEXIL can face difficult retargeting when robotic fingers differ substantially in size from human fingers, and DEXCAP currently lacks force sensing.

APPENDIX A IMPLEMENTATION DETAILS

The appendix describes DEXCAP’s camera-mount hardware, preparation workflow, data-collection settings, and retargeting pipeline. Raw observations and hand actions are transformed into robot-space training data using point clouds and inverse kinematics.

  • The chest mount carries one RGB-D LiDAR camera and three T265 SLAM cameras, while two detachable T265 cameras track hand wrist poses from glove mounts.
  • Data collection begins with all cameras on the chest, followed by SLAM map building and relocation of the bottom T265 cameras to the glove mounts.
  • The glove camera is angled 45 degrees upward, and the slide guide indentation maintains a consistent insertion position across experiments.
  • DEXCAP data are collected both in robot space, with the chest setup between robot arms, and in the wild.
  • Retargeting separates observation and action processing before adapting the collected data for robot-policy training.
  • RGB-D inputs become color-preserving point clouds aligned to the world frame and manually adjusted into robot operation space.
  • Hand joints are transformed into the robot operation space, then retargeted through inverse kinematics.
  • Point clouds are uniformly downsampled to 5000 points and stored with robot proprioception states and actions in HDF5 files.

D. Robot controller details

The robot controller uses hierarchical position control, while experiments compare image- and point-cloud-based policy configurations across manipulation tasks. The appendix also specifies task layouts, objectives, and the policy hyperparameter references.

  • Robot controller details: The high-level policy outputs each arm’s 6-DoF end-effector pose and a 16-dimensional finger-joint position, while a low-level OSC interpolates trajectories.
  • Robot controller details: Image-input methods use ResNet-18 encoders, and diffusion-policy models use DDIM denoising iterations.
  • Robot controller details: Point-cloud inputs are uniformly downsampled to 1000 points, with architecture hyperparameters listed in Tables V–VII.
  • Robot controller details: The task suite includes sponge picking, ball collecting, plate wiping, packaging, scissor cutting, and tea preparing.
  • Robot controller details: Packaging tests generalization across objects, including unseen objects absent from the training dataset.
  • Robot controller details: Scissor cutting requires coordinated bimanual grasping, lifting, and cutting of paper tape.
  • Robot controller details: Tea preparing requires bottle grasping, cap unscrewing, tweezer use, pouring, and returning objects to their positions.

G. Human-in-the-loop implementations

DEXCAP supports human-in-the-loop correction during policy rollouts through teleoperation and residual correction. The implementation streams tracking data to the robot and records controller commands for correction data.

  • DEXCAP provides teleoperation and residual correction methods for collecting additional rollout correction data.
  • The mini-PC streams T265 tracking data through a local Redis server to the workstation running the learned policy.
  • The tracking comparison disables SLAM mapping and pose correction for IMU-only tracking, whose endpoint fails to return to the starting point because of pose drift.
  • Correction collection logs action commands sent to the robot controller rather than the robot’s actual positional changes, which is important for physical-contact tasks.

APPENDIX B SUPPLEMENTARY EXPERIMENT RESULTS

The SLAM-IMU approach tracks hand poses more accurately than the IMU-based method, reducing average error from 11.3 cm to 0.8 cm.

  • 0.8 cm average error with SLAM-IMU versus 11.3 cm with the IMU-based method.The IMU-based method suffers pose drifting during tracking.
Loading 2403.07788v2…