Source-linked AI summary
DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox
TL;DR
DexPilot targets the high cost and limited control of existing teleoperation systems for highly actuated, multi-fingered robots. It uses vision-based direct imitation of the bare human hand to control a 23-DoA hand-arm system, achieving high task success rates across two pilots while supporting complex manipulation.
Problem
High-DoA, multi-fingered teleoperation is often cost-prohibitive, while low-cost systems typically provide fewer control degrees of freedom.
Method
DexPilot combines markerless, glove-free vision, neural hand-state estimation, nonlinear kinematic retargeting, and motion policies to imitate bare-hand motion on a robotic hand-arm system.
Results
High task success rates were achieved across two trained human demonstrators despite the lack of tactile feedback.
Takeaways & Limitations
The system enabled complex manipulation demonstrations that can provide high-dimensional, multimodal state-action data for future sensorimotor policy learning.
Takeaways & Limitations
The observable pilot work volume limits tasks covering greater distances, and projection schemes can interfere with finger gaiting and timely release of small objects.
Abstract
from arXiv · showhide
Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enables operators to carry out a variety of complex manipulation tasks that go beyond simple pick-and-place operations. This allows for collection of high dimensional, multi-modality, state-action data that can be leveraged in the future to learn sensorimotor policies for challenging manipulation tasks. The system performance was measured through speed and reliability metrics across two human demonstrators on a variety of tasks. The videos of the experiments can be found at https://sites.google.com/view/dex-pilot.
I. INTRODUCTION
DexPilot addresses the cost and control limitations of teleoperating highly actuated, multi-fingered robots with a low-cost, markerless, glove-free vision system. It directly imitates human hand motion to support dexterous manipulation, and evaluations with two pilots achieved high task success rates despite no tactile feedback.
- Teleoperation harnesses human cognition, creativity, and reactivity to perform complex robotic tasks across applications including search and rescue, space, medicine, prosthetics, and machine learning.
- A low-cost setup combining four depth cameras, two GPUs, deep learning, nonlinear optimization, and motion generation captures fine dexterous manipulation without tactile feedback.
- DexPilot provides markerless, glove-free, entirely vision-based control of a highly actuated robotic hand-arm system through direct imitation.
- Novel cost functions and projection schemes retarget human hand joints to Allegro hand joints while preserving dexterity and feasible precision grasps under tracking error.
- DexPilot supports fine-manipulation tasks such as extracting currency from a wallet and grasping two cubes with four fingers.
- High task success rates were achieved across two trained human demonstrators despite the absence of tactile feedback.
III. HARDWARE
The hardware setup places a KUKA arm and Allegro hand beside a four-camera human pilot arena for line-of-sight imitation teleoperation. Tactile sensors instrument the robot for later data use, while three computational threads coordinate learning, tracking, retargeting, and control.
- The robot system combines a KUKA LBR iiwa7 R800 arm with a Wonik Robotics Allegro hand.
- Four fingertip Syntouch Biotac sensors produce 23 tactile signals for later learning from demonstrations, while grip tape and sensor surfaces increase friction.
- The imitation paradigm continuously observes human pose and finger configuration, then relays comparable curling, grasping, reorientation, and translation to the robot.
- Four cameras point toward the table while the pilot moves their hand near the robot system to maintain line of sight for teleoperation.
- Three threads run learning, DART tracking with kinematic retargeting, and Riemannian-motion-policy control across three computers.
- The complete system produces approximately one second of latency.
V. DART
DexPilot combines model-based DART tracking with neural hand-state priors, segmentation, and joint prediction to maintain robust tracking for teleoperation. A glove-assisted first phase supplies annotations, enabling a second phase that tracks the bare hand from fused depth point clouds.
- V. DART: DART continuously tracks human hand pose and joint angles by matching an articulated hand model to an input point cloud.
- V. DART: DART can fail when initialization leaves its nonlinear optimization basin or when spurious local minima attract the hand model during long tracking periods.
- V. DART: Neural hand-pose priors and segmentation are introduced to prevent DART from converging to incorrect local minima.
- VI. ESTIMATING HAND POSE AND JOINT ANGLES WITH NEURAL NETWORKS: A glove-assisted first phase uses colored blobs and four RGB-D cameras to generate hand-pose and joint-angle annotations for raw depth data.
- VI. ESTIMATING HAND POSE AND JOINT ANGLES WITH NEURAL NETWORKS: The second phase removes the glove by estimating pose directly from fused point clouds of the bare hand, after table-plane removal and hand localization.
- VI. ESTIMATING HAND POSE AND JOINT ANGLES WITH NEURAL NETWORKS: JointNet maps 23 predicted keypoint locations to a 20-dimensional vector of finger joint angles for use as DART priors.
- VI. ESTIMATING HAND POSE AND JOINT ANGLES WITH NEURAL NETWORKS: 9.7 mm average keypoint error and 1.33 degrees per joint error were measured on a 7,000-image validation set.
A. Kinematic Retargeting
DexPilot retargets observed human-hand motion to Allegro-hand joint states using learned pose estimation, geometric projections, and real-time optimization. The retargeting design preserves dexterity while reducing collisions and undesirable redundant configurations.
- A. Kinematic Retargeting: Human hand joints are mapped to Allegro joints because the two hands are kinematically disparate.The approach uses task-space relationships between palm and fingertips for the mapping.
- A. Kinematic Retargeting: A two-stage PointNet++ inspired network segments the hand, estimates a rough pose, and refines keypoints from the original point cloud.The second stage addresses sparse finger sampling caused by uniform input subsampling.
- A. Kinematic Retargeting: The retargeting cost function weights task-space vector errors, applies distance-dependent projections, and regularizes Allegro joint angles toward an open-hand configuration.The regularizer reduces redundancy and helps prevent solutions in which fingers embed into the palm.
- A. Kinematic Retargeting: Projection distances use η1 = 1 × 10^-4 m for primary-finger/thumb closure and η2 = 3 × 10^-2 m for minimum separation between primary fingers.These projections keep precision-grasp contacts close while avoiding primary-finger collisions.
- A. Kinematic Retargeting: The optimization is minimized in real time with SLSQP, initializing from zero Allegro angles and then from the preceding solution.
B. Riemannian Motion Policies
Riemannian Motion Policies generate real-time palm motion while combining Cartesian trajectory priorities with collision avoidance. The system maps the retargeted hand states into robot control and registers pilot and robot coordinates for intuitive movement.
- B. Riemannian Motion Policies: RMPs calculate acceleration fields from potential-function gradients and Riemannian metrics for real-time motion generation.
- B. Riemannian Motion Policies: RMPs combine multipriority Cartesian trajectories and collision avoidance to control the Allegro palm while avoiding arm-palm collisions with the table or operator.
- B. Riemannian Motion Policies: Task-space vectors between human and Allegro fingertips and palms provide the geometric relationships used for retargeting optimization.The vectors encode both distance and direction, while local coordinates also capture relative orientation.
- B. Riemannian Motion Policies: Human and robot coordinates are registered from the initial hand and robot poses so the robot moves in the pilot’s hand direction.
VIII. EXPERIMENTS
DexPilot was evaluated on 15 manipulation tasks spanning grasping, dexterous object handling, and long-horizon sequences. Two pilots completed five consecutive trials per task, with completion time and success rate measuring speed and reliability.
- VIII. EXPERIMENTS: The test suite covered precision and power grasps, prehensile and non-prehensile manipulation, and finger gaiting.Objects and representative tasks included wallets, tea drawers, and peanut jars.
- VIII. EXPERIMENTS: Opening a wallet required repositioning it to a particular vantage point before extracting paper currency while maintaining a fingertip pinch.
- VIII. EXPERIMENTS: Opening a tea drawer and extracting a tea bag formed a long-horizon task requiring drawer dexterity and sustained object holding.
- VIII. EXPERIMENTS: Opening a peanut jar required repeatedly rotating the cap while maintaining contact.
- VIII. EXPERIMENTS: 15 tasks ranged from classic pick-and-place to multi-step long-horizon manipulation, with failures defined when objects left the workspace.
- VIII. EXPERIMENTS: Five consecutive trials were conducted for each pilot and task after 3–5 nonconsecutive warm-up attempts.Mean completion time and success rate captured teleoperation speed and reliability.
IX. DISCUSSION
DexPilot provides low-cost, vision-based teleoperation for a highly actuated hand-arm system, translating human hand and finger motion into robot motion. It enables complex manipulation tasks that are difficult to hand-engineer or solve directly with learning methods.
- DexPilot translates observed human hand and finger motion into motor solutions for a highly actuated robotic hand-arm system.
- The system supports complete control of the robot’s arm and fingers during varied manipulation tasks.
- Tasks such as extracting money from a wallet and opening a cardboard box inside a plastic container were especially complex.
- The pilot directs the robot using task knowledge while the robot handles finer details of implementing the solution.
- DexPilot is presented as a viable, low-cost solution for teleoperating a high-DoA robotic system.
XII. APPENDIX
The appendix describes a vision-based glove-tracking pipeline using colored blobs, neural keypoint detection, augmentation, confidence estimation, and outlier rejection. Because fingertip blob predictions were inconsistent under occlusion, hand-pose estimation relied on palm-back blobs.
- GloveNet detects colored hand blobs from an instrumented black-glove and black-cloth scene using HSV thresholding for annotations.
- Data augmentation and depth-based background manipulation are used to improve accurate and stable keypoint detection.
- Test-time augmentation estimates each fingertip location’s confidence from the standard deviation of shifted predictions.
- At test time, low-confidence predictions are discarded, while confident predictions are filtered through distance-based probabilities and a geometric median.
- GloveNet regresses eight keypoint locations—five finger points and three palm points—and scales predictions for 320×240 images.
- Because fingertip blob centers were inconsistent across frames due to occlusions, the system used only palm-back blobs for hand-pose estimation.
B. Architecture for Hand Pose Estimation with PointNet++
The hand-pose estimator uses PointNet++ to process point features at multiple resolutions and predict 23 keypoint locations. Voting, segmentation, and multilayer perceptrons produce keypoint, hand-mask, and pose-related outputs.
- PointNet++ set-abstraction layers subsample points and summarize local context, while feature-propagation layers interpolate features back to higher resolution.
- The PointNet++ backbone produces an N × C embedding that different MLPs map to the desired outputs.
- The voting layer predicts relative offsets to 23 keypoints, which are added to input coordinates to obtain keypoint locations.
- Segmentation layers identify point neighborhoods contributing to each keypoint and separate the hand from the background.
- Euclidean losses train voting and keypoint predictions, while sigmoid cross-entropy trains hand segmentation.
C. Architecture for JointNet
The JointNet architecture converts the 23 three-dimensional hand keypoints into joint angles. Its three-layer design was selected as an accuracy–efficiency trade-off, and evaluation reports completion times across repeated task trials.
- The 23×3 keypoint locations are flattened into a 69-dimensional vector before entering JointNet.
- JointNet maps the flattened keypoint representation to a 20-dimensional vector of joint angles.
- The selected JointNet architecture was reported as an optimal trade-off between accuracy and efficiency among tested hand-designed architectures.
- JointNet comprises three layers, with BatchNorm1d and ReLU in its first two linear layers.
- Evaluation records completion times for five consecutive trials per task, marking failed trials with F.
E. Retargeting With Neural Networks
DexPilot combines model-based and model-free tracking, using learned initialization to support robust hand tracking. For retargeting, neural networks were less accurate than online sequential least-squares optimization, while combined tracking enabled long-duration operation without failures.
- Neural-network retargeting produced less accurate results than online sequential least-squares optimization.
- Combining model-based and model-free tracking significantly improved hand tracking and enabled long-duration tracking without failures.
- Model-based tracking improves accuracy but requires good initialization because its optimization is highly nonlinear.
- The model-free tracker is a neural network trained on data generated by the model-based system to provide initialization.