Source-linked AI summary
Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network
Shuang Li, Xiaojian Ma, Hongzhuo Liang, Michael Görner, Philipp Ruppel, Bing Fang, Fuchun Sun, Jianwei Zhang
TL;DR
This paper addresses direct, intuitive vision-based teleoperation of dexterous robotic hands from human-hand depth images. TeachNet predicts robot joint angles using paired human-robot data and a consistency loss. Evaluation and robot experiments report higher accuracy and lower error than end-to-end baselines, with faster grasp tasks than the state-of-the-art vision-based method.
Problem
Directly mapping human-hand depth images to robot joint angles is challenging because the human and robot hands occupy different domains.
Source: §I · p. 1Method
TeachNet uses robot-teacher and human-student branches with consistency loss, trained using synchronized human-robot hand data.
Source: §I · p. 1, §III · p. 3Results
TeachNet achieves higher accuracy and lower error than other end-to-end baselines, and robot experiments report significantly faster grasp tasks than the state-of-the-art vision-based method.
Source: §I · p. 1Takeaways & Limitations
The method supports human gesture imitation and grasp tasks teleoperated by novice users.
Source: §I · p. 1The operator's hand must remain within a fixed range, occluded joints have higher error, and the reported grasp tasks do not use tactile feedback.
Source: §VI · p. 6
Abstract
from arXiv · showhide
In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.
I. INTRODUCTION
Teleoperation can handle fast decisions and corner cases, while markerless vision avoids wearable devices and preserves natural motion. The paper proposes direct depth-to-joint control with TeachNet and reports improved evaluation and teleoperation results.
I. INTRODUCTION
- Teleoperation can support fast decisions and corner cases when autonomous manipulation handles complex perception.Source: §I · p. 1
- Existing data-driven teleoperation methods depend on accurate pose estimation or classification and incur post-processing time.Source: §I · p. 1
- TeachNet maps a human-hand depth image directly to robot joint angles, using an end-to-end network and synchronized human-robot data.The paper also proposes an optimized mapping method that matches hand positions and link directions while accounting for possible self-collisions.Source: §I · p. 1
- TeachNet achieves higher accuracy and lower error than other end-to-end baselines, while robot experiments report faster grasp tasks than the state-of-the-art vision-based method.Source: §I · p. 1
II. RELATED WORK
Prior teleoperation approaches use contact devices, analytical vision, or data-driven pose processing. The paper positions its end-to-end CNN as a way to generate continuous robot poses without relying on a separate pose-estimation or classification stage.
II. RELATED WORK
- Contacting devices can hinder natural limb motion, whereas markerless vision-based teleoperation is less invasive and supports natural, comfortable gestures.Source: §II · p. 2
- Model-based vision methods compute continuous poses but can be computationally costly and depend on multicamera systems, while appearance-based methods recognize discrete trained poses.Source: §II · p. 2
- Data-driven approaches may depend on accurate pose estimation or classification and spend time on post-processing; this work targets continuous robot poses with an end-to-end CNN.Source: §II · p. 2
- Prior master-slave pairing by synchronous human imitation is costly and can yield noisy correspondences because human-robot imitation is subjective.Source: §II · p. 2
- Joint, fingertip, and pose mapping each have limits, including fingertip mapping's omission of phalanx position and orientation and master-slave mechanical differences.Source: §II · p. 2
III. TEACHER-STUDENT NETWORK
TeachNet uses robot images as a teacher branch and human images as a student branch to predict robot joint angles. Its consistency and physical losses address cross-domain pose alignment and robot constraints.
III. TEACHER-STUDENT NETWORK
- TeachNet has robot-teacher and human-student branches that predict robot joint angles from their respective depth images.Each branch is supervised with a mean squared error loss against ground-truth joint angles.Source: §III · p. 3
- A consistency loss aligns latent features from paired human and robot images so the human branch is supervised in a pose space shared with the robot branch.Source: §III · p. 3
- The paper explores hard feature alignment with an extra mean squared error and soft alignment using a discriminator's realism score.Source: §III · p. 3
- The dataset-generation figure depicts mapping a 21-joint human hand to a 24-joint Shadow hand with 19 degrees of freedom.Source: Figure 3 · p. 3
- Early teaching aligns features after the encoder and embedding module, while late teaching places alignment at the penultimate layer.Source: §III · p. 3
- A physical loss enforces physical constraints and joint limits, and the complete branch objective combines this loss with joint-angle loss.Source: §III · p. 3
IV. DATASET GENERATION
The training set pairs human-hand depth images with simulated robot-hand images and robot joint angles, using BigHand2.2M data and an optimized mapping pipeline.
IV. DATASET GENERATION
- The generated dataset contains 400K pairs of simulated robot depth images and human-hand depth images, with corresponding robot joint angles and poses.It is built using the BigHand2.2M human-hand dataset and an optimized mapping pipeline.Source: §IV · p. 3
A. Human-Robot Hand Kinematic Configuration
The Shadow hand has five fingers and 19 degrees of freedom, compared with 31 degrees of freedom in the human hand model. The mapping simplifies this difference by fixing the Shadow wrist and considering 15 joint keypoints.
A. Human-Robot Hand Kinematic Configuration
- The Shadow hand has 19 degrees of freedom: 17 in the hand and two in the wrist.Each finger has four joints, with the first joint stiff; the little finger and thumb have an extra joint for holding objects.Source: A. Human-Robot Hand Kinematic… · p. 4
- The human hand model has 21 joints and 31 degrees of freedom, while the Shadow hand has five fingers.The main kinematic differences are the robot's more limited joint-angle ranges and wrist-joint structure.Source: A. Human-Robot Hand Kinematic… · p. 4
- For the simplified mapping, the two Shadow wrist joints are fixed at 0 rad and 15 TIP, PIP, and MCP keypoints are considered.Source: A. Human-Robot Hand Kinematic… · p. 4
B. Optimized Mapping Method
The optimized mapping combines position and orientation constraints while accounting for possible self-collisions. It uses weighted landmark constraints and checks candidate robot motions for collisions.
B. Optimized Mapping Method
- The mapping integrates position and orientation constraints and accounts for possible self-collisions when imitating human hand poses.It weights fingertip positions at 1, PIP positions at 0.2, and direction mappings at 0.2.Source: B. Optimized Mapping Method · p. 4
- The method uses BioIK to determine robot joint angles, then checks the resulting Gazebo motion for self-collision with MoveIt.When a collision is detected, the cost function measures distance between two links; the minimum collision-free radius and link positions define the relevant quantities.Source: B. Optimized Mapping Method · p. 4
- Nine simulated depth cameras capture each robot pose simultaneously to diversify robot viewpoints in line with the human dataset.Fig. 4 shows Shadow depth images from nine viewpoints for one human gesture.Source: B. Optimized Mapping Method · p. 4, Figure 4 · p. 4
A. TeachNet Evaluation
TeachNet evaluation compared network variants, ablations, and HandIK across joint-angle and joint-distance metrics. Teach Hard-Late showed a high-precision gain over Single Human, while thumb joints remained harder to learn.
A. TeachNet Evaluation
- The evaluation compared four TeachNet variants, consistency-loss and single-branch ablations, and HandIK using joint-angle and joint-distance metrics.HandIK maps estimated human joint locations to robot joint angles; network inputs were fixed-size hand-centered depth cubes resized to 100×100, with one of nine Shadow views randomly selected during training.Source: A. TeachNet Evaluation · p. 4
- Single Robot ranked best across evaluation metrics as the training supervisor, while Teach Hard-Late outperformed the other baselines.The evaluation included maximum and average joint-angle and joint-distance errors, plus average angle error across all angles.Source: A. TeachNet Evaluation · p. 4
- 3.63% improvement in accuracy below a maximum joint angle for Teach Hard-Late over Single Human under high-precision conditions.Table I reports the high-precision comparison.Source: A. TeachNet Evaluation · p. 5, Table I · p. 5
- The thumb's second, third, and base joints were harder to learn, with fixed joints, larger ranges, and human-Shadow thumb differences cited as factors.Figure 6 compares average angle error by joint and depicts the Shadow hand's kinematic chain.Source: A. TeachNet Evaluation · p. 5, Figure 6 · p. 5
B. Robotic Experiments
Real-world experiments involved five adult subjects teleoperating the Shadow hand with a close-range depth sensor, constrained hand poses, and bright, uniform lighting.
B. Robotic Experiments
- Five adult subjects tested real-world teleoperation using a Shadow hand whose first joint of each finger was fixed and an Intel RealSense F200 depth sensor.The sensor was described as suitable for close-range tracking.Source: B. Robotic Experiments · p. 5
- Operators used their right hands within a viewpoint range of 70°–120° and a camera distance range of 15–40 mm, with experiments conducted under bright, uniform lighting as much as possible.Source: B. Robotic Experiments · p. 5
- Teach Hard-Late's average computation time was 0.1051 s on an Alienware 15 with an Intel Core i7-4720HQ CPU.Source: B. Robotic Experiments · p. 5
1) Simulation Experiments:
TeachNet supports continuous online imitation of novice gestures without explicit joint-level correspondences. In manipulation experiments, novices took longer with HandIK, with performance affected by thumb accuracy and object geometry.
1) Simulation Experiments:
- TeachNet enabled continuous online imitation of novice gestures without explicit joint-level correspondences, including gestures performed in a disordered sequence.Five novice teleoperators performed American sign language digits 0–9 and common gestures.Source: 1) Simulation Experiments · pp. 5–6
- Because two Shadow-hand wrist joints were fixed, the sensor did not need to capture the teleoperator’s wrist.Source: 1) Simulation Experiments · p. 6
- Visible errors occurred mainly at the fingers’ second and third joints and the thumb base, possibly due to slave-hand kinematics, occlusions, and uncertain lighting.Source: 1) Simulation Experiments · p. 6
2) Manipulation Experiments:
- Novices took longer with HandIK; low accuracy, especially for the thumb, and post-processing contributed, while large-diameter objects particularly challenged HandIK.The comparison used average time to grasp and release objects; the supplied table caption does not provide individual times.Source: Table II · p. 6, 2) Manipulation Experiments · p. 6
VI. CONCLUSIONS AND FUTURE WORK
The paper presents an end-to-end teacher-student network and a method for building paired human-robot hand data to learn kinematic mappings. Its experiments report faster, more accurate novice grasping than HandIK, while identifying operating-range and occlusion constraints and the omission of tactile feedback.
VI. CONCLUSIONS AND FUTURE WORK
- The method combines TeachNet with a dataset of 400K paired human-hand depth images, simulated robot depth images in matching poses, and corresponding robot joint angles.The dataset-generation method maps keypoint positions and link directions, then records robot state in simulation.Source: §VI · p. 6
- Experiments report that novice teleoperators grasped in-hand objects faster and more accurately with the end-to-end method than with HandIK.Source: §VI · p. 6
- The method requires the operator’s hand within a fixed range and has higher error on occluded joints; grasp tasks did not use tactile feedback.The authors propose combining tactile input for more complicated tasks and extending teleoperation to other robot body parts.Source: §VI · p. 6