Source-linked AI summary
EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation
Branden Romero, Hao-Shu Fang, Pulkit Agrawal, Edward Adelson
TL;DR
Dexterous robot hands need human-like morphology, compliant actuation, and dense tactile sensing for robust real-world learning and manipulation. EyeSight Hand combines these properties in a 7-DoF humanoid hand with quasi-direct drive actuation, eight vision-based tactile sensors, and imitation learning with vision dropout. Across three challenging tasks, tactile sensing improved success rates, with vision dropout providing additional gains for plasticine cutting and bottle opening.
Problem
Existing robotic hands provide limited combinations of compliance, full actuation, dense tactile sensing, and human-like morphology, constraining practical dexterous robot learning.
Method
The paper develops a low-cost 7-DoF humanoid hand with quasi-direct drive actuation, eight GelSim(ple) tactile sensors, and imitation-learning policies using vision dropout.
Results
Tactile sensing improved task success from 50% to 100% for plate pick and place and from 50% to 70% for plasticine cutting, while vision dropout reached 90% and 30% for plasticine cutting and bottle opening, respectively.
Takeaways & Limitations
Rich tactile sensing significantly enhances imitation-learning performance on challenging real-world dexterous manipulation tasks.
Abstract
from arXiv · showhide
In this work, we introduce the EyeSight Hand, a novel 7 degrees of freedom (DoF) humanoid hand featuring integrated vision-based tactile sensors tailored for enhanced whole-hand manipulation. Additionally, we introduce an actuation scheme centered around quasi-direct drive actuation to achieve human-like strength and speed while ensuring robustness for large-scale data collection. We evaluate the EyeSight Hand on three challenging tasks: bottle opening, plasticine cutting, and plate pick and place, which require a blend of complex manipulation, tool use, and precise force application. Imitation learning models trained on these tasks, with a novel vision dropout strategy, showcase the benefits of tactile feedback in enhancing task success rates. Our results reveal that the integration of tactile sensing dramatically improves task performance, underscoring the critical role of tactile information in dexterous manipulation.
I. INTRODUCTION
EyeSight Hand addresses the difficulty of combining human-like morphology, compliant actuation, and dense tactile sensing in a practical robotic hand. It introduces a low-cost 7-DoF platform with quasi-direct drive actuation, eight vision-based tactile sensors, and teleoperation and learning evaluations on challenging manipulation tasks.
- Existing robotic hands typically capture only subsets of human functionality because fully actuated hands lack compliance, soft hands lack precise full actuation, and tactile hands often have limited DoF or non-human morphology.The paper identifies compliance, dense tactile sensing, and human-like morphology as important for robot learning and data collection.
- The EyeSight Hand is a low-cost, 7-DoF humanoid hand co-designed around actuation, tactile sensing, and kinematics.Its reported cost is less than $2500.
- Quasi-direct drive actuation provides compliant operation with human-level finger strength and speed while remaining robust for large-scale data collection.
- GelSim(ple) is a simulatable, morphology-adaptable vision-based tactile sensor, and eight sensors are integrated into the hand for sensitive whole-hand manipulation.
- Human-like kinematics and dimensions enable simple teleoperation with minimal retargeting, while bottle opening, plasticine cutting, and plate pick and place test contact-rich manipulation and force application.The hand was teleoperated robustly across hundreds of trials, and imitation learning used vision dropout to better exploit tactile sensing.
II. BACKGROUND AND RELATED WORK
Prior humanoid robotic hands trade off dexterity, robustness, speed, or actuation quality. EyeSight Hand explores quasi-direct drive actuation as a way to combine robust and dynamic manipulation with humanoid-hand design.
- A. Humanoid Robotic Hands: Existing humanoid hands face trade-offs among cost, reliability, reflected inertia, robustness to force, and actuation speed.Tendon-driven hands can be unreliable, highly geared servos resist external forces, and linkage-based designs can be slow.
B. Hands with Vision Based Tactile Sensors
Prior vision-based tactile hands often use limited-DoF morphologies, while dexterous-hand learning remains difficult because of high-dimensional control and precise sensing under occlusion. The paper positions its 7-DoF tactile hand as a real-world imitation-learning platform for contact-rich tasks.
- B. Hands with Vision Based Tactile Sensors: Earlier vision-based tactile sensors were commonly flat, round, or fingertip-mounted, while newer designs extend across fingers but largely target 2- or 3-DoF grippers.EyeSight Hand instead integrates a new sensor design with a self-designed 7-DoF hand.
- C. Learning for Dexterous Hand: Dexterous manipulation learning is difficult because many degrees of freedom and severe occlusion require precise sensing, while real-world imitation learning also faces data and hardware constraints.The paper emphasizes rich tactile perception for efficient imitation learning on real-world contact-rich tasks.
- C. Learning for Dexterous Hand: Figure 3 presents simplified human-hand kinematics as a reference for the hand-design discussion.
- III. MECHANICAL DESIGN: The mechanical design targets a hand robust enough for large-scale data collection and intuitive enough for teleoperation and imitation learning.
A. Kinematics
EyeSight Hand compromises some human kinematic joints to support a compact design, then uses mixed worm-gear and BLDC actuation to provide compliance, robustness, force, and speed. The resulting fingers are designed for high-force dynamic manipulation.
- A. Kinematics: The hand omits selected index, middle, and thumb joints relative to human kinematics, including the index and middle DIP joints and thumb MCP joint.
- B. Actuation: Worm-drive and BLDC actuators are arranged in series to actuate finger and thumb joints directly or through four-bar linkages, improving robustness over tendon-driven designs.The worm drive actuates PIP or IP joints, while BLDC motors actuate MCP or TM-joint axes.
- B. Actuation: The actuator selection reduces resistance to external forces and increases durability through self-locking worm gears, low-reduction BLDC transmission, and planetary load distribution.These choices are also described as supporting high shock-load handling and reduced breakage risk.
- B. Actuation: 19N continuous fingertip force, 57N maximum fingertip force, and 420 RPM maximum speed characterize the actuation scheme's reported strength and speed.
C. Electronics
The EyeSight Hand combines integrated tactile sensing with electronics supporting synchronized, high-rate data capture. GelSim(ple) simplifies illumination and simulation while preserving geometric contact information.
- C. Electronics: The BLDC actuators use magnetic encoders, current sensing for torque estimation, and 1kHz CAN-FD communication, while micro DC motors use dedicated drivers and serial control.These electronics support actuator position and torque-related feedback across the hand’s motor systems.
- IV. TACTILE SENSOR DESIGN: Eight GelSim(ple) sensors provide whole-hand tactile coverage across seven tactile surfaces and stream images from finger segments and the palm.The design targets adaptable sensor morphologies, high data throughput, and simulation for sim2real approaches.
- A. Illumination: GelSim(ple) uses non-directional overhead illumination that removes cast shadows, preserves geometric contact cues, and enables single-channel image transmission.An MLP simulates pixel values from surface normals, viewing direction, and positional encoding of deformation.
B. Implementation Details
The implementation packages GelSim(ple) into compact camera-based tactile modules and evaluates the hand through imitation learning on diverse manipulation tasks. Each module streams synchronized single-channel tactile images at 60Hz, while experiments collect multimodal demonstrations.
- B. Implementation Details: Each GelSim(ple) module combines a fisheye camera, shaped LED diffuser, acrylic backing, molded silicone sensing surface, and remote camera connection.Two camera-array interfaces connect eight synchronized modules to Raspberry Pi computers.
- B. Implementation Details: The tactile system transmits one 640x480 single-channel image per Raspberry Pi at 60Hz, supporting high-throughput sensing during operation.The implementation uses two Raspberry Pi 4 computers and ZeroMQ for host communication.
- V. EXPERIMENT SETUP: The evaluation uses imitation learning across plasticine cutting, stacked-plate pick and place, and bottle opening to test manipulation, tool use, force sensing, and dexterity.The setup collects proprioception, global and wrist-camera images, tactile images, and 100 demonstrations per task.
A. Teleoperation
Teleoperation maps four human-hand sensor poses to robotic joint commands through task-space vector optimization. The implementation solves this retargeting problem with SLSQP and operates at 125Hz.
- A. Teleoperation: Task-space vector optimization estimates robotic joint angles by matching vectors between human and robot frames while accounting for pose changes over time.The optimization uses four human sensor poses and corresponding robot frames; α weights the significance of robotic pose changes.
- A. Teleoperation: The evaluation tasks require coordinated approach, constraint, sliding, lifting, tool removal, and cutting behaviors across bottle, plate, and knife-manipulation scenarios.These task sequences are illustrated for bottle opening, plate pick and place, and plasticine cutting.
- A. Teleoperation: The teleoperation system solves the retargeting objective with SLSQP, JAX automatic differentiation, and rigid-body-dynamics tools before directly commanding robot joint angles.The complete teleoperation loop operates at 125Hz.
B. Task Specification
The task specification tests manipulation under perturbed initial conditions, requiring controlled contact and force. Bottle opening emphasizes stabilization and complete lid motion, while plate pick and place requires careful sliding without moving both plates.
- B. Task Specification: For each task, the study collects 100 demonstrations using proprioceptive data, global-camera images, wrist-mounted fisheye-camera images, and tactile-sensor images at 30Hz.
- 1) Bottle Opening:: Bottle opening requires constraining an unfixed, slightly perturbed ketchup bottle with downward force before using the thumb to fully swing open its lid.A lid that is not fully open counts as a failure.
- 2) Plate Pick and Place:: Plate pick and place requires sliding the top plate over the bottom plate into a graspable position, then lifting it and placing it beside the other plate.Excessive force can make both plates slide, and initial positions are slightly perturbed.
3) Plasticine Cutting:
Plasticine cutting combines unstable tool and workpiece placement with precise force control, and is learned using multimodal imitation policies that integrate vision and tactile inputs.
- 3) Plasticine Cutting:: The task perturbs the knife box, plasticine shape, and plasticine position while requiring a full cut despite unfastened equipment.Excessive force can cause the knife box and cutting board to slide.
- C. Imitation Learning: ACT predicts future joint-space action trajectories from visual features, tactile features, joint positions, and a latent style variable.Eight tactile images are grouped into two 2×2 super-images before shared tactile encoding.
- C. Imitation Learning: The vision-only variant excludes tactile inputs, whereas the vision-tactile variant uses both visual and tactile images during training and testing.These variants provide the comparison basis for evaluating tactile integration.
- C. Imitation Learning: Vision dropout zeros the global and wrist camera images with 30% probability during training, while testing uses unaltered images.The strategy is designed to encourage greater reliance on tactile information.
VI. EXPERIMENTAL RESULTS
Across three real-robot tasks, tactile sensing improved policy success, while vision dropout further benefited the difficult plasticine-cutting and bottle-opening tasks.
- VI. EXPERIMENTAL RESULTS: Plate pick and place improved from 50% with vision-only input to 100% when tactile sensing was incorporated.Without tactile sensing, failures involved insufficient contact or force during sliding and unstable grasps during movement.
- VI. EXPERIMENTAL RESULTS: Plasticine cutting reached 90% with vision dropout, compared with 70% for conventional vision-tactile input and 50% for vision-only input.Tactile sensing reduced partial cuts and excessive force that moved the cutting board.
- VI. EXPERIMENTAL RESULTS: Bottle opening achieved 30% with vision dropout, while both vision-only and vanilla vision-tactile policies failed in all trials.The dropout strategy encouraged greater reliance on tactile sensing when visual cues for lid opening were limited.
VII. CONCLUSION
The EyeSight Hand combines fully actuated 7-DoF morphology, high-resolution tactile sensing, and compliant actuation for robot learning and dexterous manipulation.
- VII. CONCLUSION: The hand’s rich sensing, collision robustness, and human-like morphology supported imitation learning on challenging real-world tasks.The authors identify extending the hand with additional fingers and improving algorithms for using tactile sensing as future work.