Source-linked AI summary
DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation
Hao-Shu Fang, Branden Romero, Yichen Xie, Arthur Hu, Bo-Ruei Huang, Juan Alvarez, Matthew Kim, Gabriel Margolis, Kavya Anbarasu, Masayoshi Tomizuka, Edward Adelson, Pulkit Agrawal
TL;DR
Dexterous robot learning is constrained by the need for large amounts of rich, transferable manipulation data. The paper introduces perioperation and implements it in DEXOP, a passive hand exoskeleton that sensorizes natural human manipulation; across dexterous tasks, DEXOP data improves policy performance per unit collection time compared with teleoperation.
Problem
Dexterous manipulation remains challenging, while machine-learning methods face a major bottleneck in obtaining large amounts of data and recovering fine-grained contact forces.
Method
DEXOP mechanically couples a passive robotic hand with a wearable exoskeleton, combining multisensory recording, whole-hand tactile sensing, force transparency, and pose mirroring.
Results
DEXOP supports diverse dexterous tasks and user studies report superior control and significantly improved data-collection throughput compared with traditional teleoperation.
Takeaways & Limitations
DEXOP provides a tool for scalable, real-world collection of rich tactile and contact-driven data for dexterous robot learning.
Takeaways & Limitations
Manufacturing defects, calibration errors, and mismatches between the exoskeleton and target robot can degrade performance, so DEXOP data alone may be insufficient for deployment.
Abstract
from arXiv · showhide
We introduce perioperation, a paradigm for robotic data collection that sensorizes and records human manipulation while maximizing the transferability of the data to real robots. We implement this paradigm in DEXOP, a passive hand exoskeleton designed to maximize human ability to collect rich sensory (vision + tactile) data for diverse dexterous manipulation tasks in natural environments. DEXOP mechanically connects human fingers to robot fingers, providing users with direct contact feedback (via proprioception) and mirrors the human hand pose to the passive robot hand to maximize the transfer of demonstrated skills to the robot. The force feedback and pose mirroring make task demonstrations more natural for humans compared to teleoperation, increasing both speed and accuracy. We evaluate DEXOP across a range of dexterous, contact-rich tasks, demonstrating its ability to collect high-quality demonstration data at scale. Policies learned with DEXOP data significantly improve task performance per unit time of data collection compared to teleoperation, making DEXOP a powerful tool for advancing robot dexterity. Our project page is at https://dex-op.github.io.
1 Introduction
DEXOP addresses the data bottleneck in dexterous robotics through perioperation, sensorizing natural human manipulation while improving transferability to robots. Its passive exoskeleton combines force-rich sensing, kinematic coupling, and scalable collection to support diverse tasks.
- Perioperation: DEXOP introduces perioperation, which records multisensory human manipulation while maximizing transferability of demonstrated skills to robots.The paradigm captures vision, proprioception, touch, and action through wearable devices rather than remotely controlling a robot.
- DEXOP design: DEXOP uses a passive robotic hand mechanically coupled to a wearable exoskeleton, allowing users to actuate the robot while performing tasks in natural environments.High-resolution tactile sensors are mounted on the passive robot hand rather than directly on a glove.
- Design objectives: Force transparency provides joint-level proprioceptive feedback, while pose mirroring removes the need for unintuitive visual hand-pose correction during demonstrations.These mechanisms are intended to make collection faster, more precise, intuitive, and easier to scale.
- Tactile sensing: Whole-hand tactile sensing preserves action-relevant contact information because forces at multiple contact points are otherwise insufficient to infer joint torques reliably.DEXOP captures detailed force and contact information across the hand for replay on the robot.
- Evaluation: DEXOP variants and user studies span drilling, lamp installation, box packaging, and bottle opening, with reported advantages in control and data-collection throughput over traditional teleoperation.The work presents three variants with different degrees of freedom and reports utility across diverse dexterous tasks.
2 Related Work
DEXOP extends prior robot-learning hardware by combining multi-finger dexterity with whole-hand force sensing and force-transparent, passive control. This supports manipulation capabilities beyond simpler grippers and feedback-limited teleoperation systems.
- Teleoperation for Dexterous Manipulation: DEXOP provides normal and shear haptic feedback through mechanical linkages, extending feedback beyond teleoperation systems that lack haptics or provide only limited fingertip cues.Feedback can optionally extend to each finger segment in DEXOP-7.
- Data Collection Hardware for Robot Learning: The hardware overview includes DEXOP-12 with 4 fingers and 12 DOF, DEXOP-9 with 3 fingers and 9 DOF, and DEXOP-7 with 3 fingers and 7 DOF.DEXOP-7 is co-designed with the EyeSight hand.
- Data Collection Hardware for Robot Learning: Unlike passive gripper approaches, DEXOP supports relative finger-object motion, small-object manipulation, constrained-space tasks, and articulated objects.Additional degrees of freedom enable tasks such as in-hand reorientation and spray-bottle manipulation.
- Data Collection Hardware for Robot Learning: DEXOP emphasizes fine, precise, whole-hand manipulation and records force information, whereas related passive exoskeletons largely support basic open-close motions.The comparison distinguishes DEXOP from systems designed primarily for flexion-based gripper-like use.
3 Hardware Design
DEXOP combines a passive robotic hand, wearable exoskeleton, and linkage system to mirror human hand motion while transmitting interaction forces. Its kinematic and sensing design supports diverse manipulation tasks and records hand, tactile, visual, and global-motion data.
- 3.1 Kinematics of the DEXOP: DEXOP mechanically links a passive robotic hand to a wearable exoskeleton, transmitting human motion to the robot and interaction forces back to the user.The shared linkage provides bidirectional force transmission between the human fingers and passive robotic hand.
- 3 Hardware Design: DEXOP includes DEXOP-12, DEXOP-9, and DEXOP-7, spanning 12, 9, and 7 degrees of freedom for different demonstration and robot-transfer settings.DEXOP-12 and DEXOP-9 support dexterous perioperation, while DEXOP-7 is co-designed with an actual robotic hand for skill transfer.
- 3.1 Kinematics of the DEXOP: 12-DOF DEXOP-12 uses three fingers for in-hand manipulation, a fourth for whole-hand support, and thumb opposition for stable antipodal grasps.Abduction changes inter-finger span, while thumb flexion moves the thumb opposite the index and middle fingers.
- 3.2 Linkage Design: The exoskeleton matches the passive hand’s kinematic chain, enabling motion transmission through 4-bar linkages for the fingers and rotary linkages for the thumb.The design uses standoffs as a virtual ground frame and separate linkage arrangements for finger and thumb degrees of freedom.
- 3.3 Tactile Sensor: Whole-hand force sensing is needed because joint positions alone cannot uniquely recover torques when forces act at multiple contact points.Forces measured at each phalanx can be combined with joint positions to compute joint torques using the Jacobian transpose method.
- 3.4 In-the-wild Data Collection: DEXOP records global hand position, joint angles, tactile images, and camera views to support dexterous manipulation data collection in natural environments.Global position can come from SLAM or an arm exoskeleton, while cameras capture in-hand and scene observations.
4 Experiments
DEXOP hardware closely matches the robotic hand in force transmission, workspace, and relevant finger-speed capabilities. Across four dexterous tasks, DEXOP achieved substantially higher throughput than teleoperation, with task-specific gains over teleoperation but remaining below direct human performance.
- 4.1 Hardware Characteristics: DEXOP transmitted approximately 60 N at the index and middle fingertips and 70 N at the thumb, comparable to the robotic hand.These measurements were used to assess force transmission from the human hand to the passive robotic hand.
- 4.1 Hardware Characteristics: DEXOP closely mirrored the robotic hand’s workspace, with approximately 110–120° MCP rotation, up to 105° PIP rotation, and matching thumb motion.Its MCP speed reached 35 rad/s versus 37 rad/s for the robotic hand, while PIP and IP speeds were 15 and 9 rad/s.
- 4.2 Comparison with Teleoperation: DEXOP achieves much higher task throughput than teleoperation across drilling, bulb installation, box packaging, and bottle opening.The study compares DEXOP, teleoperation, and direct human performance across four tasks using task throughput as the primary metric.
- 4.2 Comparison with Teleoperation: 8 times faster: DEXOP completed bulb installation in 11 seconds on average versus 86 seconds under teleoperation.Participants completed the task in approximately 4 seconds using their own hands.
- 4.2 Comparison with Teleoperation: 7 times faster: DEXOP completed box packaging at 5 times per minute, while teleoperation succeeded in only 3 of 20 trials.Direct human performance reached 16 completions per minute.
- 4.2 Comparison with Teleoperation: 2.4× faster: DEXOP achieved 12 bottle-opening completions per minute versus 5 with teleoperation.Direct human performance reached 22 completions per minute.
5 Preliminary Policy Learning Experiments
DEXOP-collected data was used to train policies for a long-horizon bimanual bulb-installation task on a humanoid robot. The mixed 160 DEXOP + 40 teleoperation policy achieved the strongest performance across stages, while exoskeleton and operator alignment errors constrained direct deployment.
- 5.1 Robot Platform Setup: The evaluation uses two EyeSight Hands on a Unitree H1, with DEXOP data collected through an AirExo-2 system and aligned directly to the robot.The setup streams joint, wrist-image, and tactile data, while the shared kinematics and sensors avoid further embodiment-gap processing.
- 5.4 Task Setup: The bulb-installation task contains six stages requiring grasping, precision insertion, tactile state recognition, and coordinated bimanual cover placement.The stages include grasping the base and bulb, inserting and installing the bulb, grasping the shade, and covering the bulb.
- 5.6 Results of DEXOP and TeleOP: Accumulated arm-exoskeleton joint errors and torso-induced misalignment required compensation and limited direct use of DEXOP data.The errors arose from fabrication variability and operators bending their backs while the robot remained upright.
- 5.6 Results of DEXOP and TeleOP: The 160 DEXOP + 40 teleop policy achieves the highest success rates across all stages, especially bulb insertion and grasp lamp cover.It outperforms 40 teleop and performs considerably better than 100 teleop despite equal collection time for the latter comparison.
- 5.6 Results of DEXOP and TeleOP: The 200 teleop policy is worse than 160 DEXOP + 40 teleop, with the largest gaps occurring in bulb insertion and grasp lamp cover.The authors hypothesize that perioperation introduces more variation in object positioning because operators are more relaxed.
- 5.6 Results of DEXOP and TeleOP: Teleoperation can bias policies toward excessive bulb rotation because limited sensory feedback makes tightening state difficult to judge, causing failures before lamp-shade grasping.The reported dataset bias prioritizes rotation over transition to the next stage.
6 Discussion
The discussion identifies scalability benefits for perioperation while emphasizing calibration, embodiment, hardware, and dexterity constraints. It also outlines future work toward better sensing, models, and force-aware learning.
- Perioperation for large-scale data collection: Perioperation systems are sensitive to manufacturing defects, calibration, and platform mismatches, so DEXOP data alone may not always suffice for deployment.The paper compensates for such errors with teleoperation data and proposes improved calibration and error-robust models.
- Dexterity needs careful design: DEXOP’s dexterity depends on careful hardware design, especially for small objects and articulated-object manipulation.Without these design considerations, perioperation with a multi-finger hand can be inferior to a parallel-jaw gripper for small-object grasping.
- Limitations and future work: Current limitations include torque estimation and calibration, fewer robot-hand degrees of freedom than the human hand, and missing tactile feedback to users.Future work includes combining tactile and vision and enabling precise force control.
- Broader vision: The authors position DEXOP as a scalable layer for collecting tactile, contact-driven data that can support robotic generalization.They connect this direction to the co-evolution of better data, hardware, and dexterous robots.
Supplementary
The supplementary material documents DEXOP variants, hardware electronics, phalange attachment, and co-design with the EyeSight Hand. These design choices align human and robot kinematics while adapting the system for dexterous contact.
- S1. Electronics of DEXOP: DEXOP’s electronics use 12-bit joint encoders, RS-485 communication, a custom aggregation PCB, and fisheye tactile cameras.The encoder resolution is 1.5e-3 radians.
- S2. Design of DEXOP-9 and DEXOP-7: DEXOP-9 removes the ring finger from DEXOP-12, while DEXOP-7 additionally removes MCP abduction.The variants use different kinematic degrees of freedom.
- S2. Design of DEXOP-9 and DEXOP-7: DEXOP-7 can attach both distal and proximal finger phalanges to provide force feedback at each finger segment.This attachment uses Velcro fasteners instead of a fingertip cot.
- S3. Co-design of DEXOP-7 and EyeSight Hand: The DEXOP and EyeSight Hand were co-designed around anatomically similar kinematic chains to support direct motion transfer.The design required modifying the EyeSight thumb and middle fingertip to preserve natural wearability and contact.
- S3. Co-design of DEXOP-7 and EyeSight Hand: The thumb IP joint was reoriented outward and the middle fingertip tilted to restore a broad thumb–middle-finger contact area.These changes were incorporated into the EyeSight Hand for smoother policy transfer.
- S4. Details of hardware enhancements: Hardware enhancements include fingernails, abduction joints, adjustable finger cots, and a palm pad for small-object handling and whole-hand manipulation.These features address grasping, reorientation, fit, and object stabilization.