PaperlayerView exact revision on arXiv

Source-linked AI summary

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Aravind Sivakumar, Kenneth Shaw, Deepak Pathak

arXiv:2202.10448v2cs.ROcs.AIcs.CVcs.LG

TL;DR

The paper addresses dexterous robot teleoperation from a single uncalibrated camera, where human-to-robot translation lacks paired correspondence data and is geometrically under-constrained. It combines human pose estimation with neural motion retargeting trained on diverse internet videos, enabling low-cost real-time control and successful manipulation by trained and previously untrained operators.

  • Problem

    Dexterous teleoperation from one uncalibrated camera is under-constrained, while paired human-robot pose data needed for direct retargeting is unavailable.

  • Method

    The system estimates human hand and body poses from a single RGB stream and retargets them to robot hand-arm commands using neural networks trained with internet human videos and an energy-based objective.

  • Results

    The system enables real-time dexterous teleoperation and outperforms the baseline on 7 out of 10 tasks while performing similarly on the other 3.

  • Takeaways & Limitations

    A low-cost, glove-free, marker-free setup lets experts and novices control a dexterous robot using only a single uncalibrated color camera.

Abstract

from arXiv · show

We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired human-robot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots in learning to act autonomously in the real world. Videos at https://robotic-telekinesis.github.io/

I. INTRODUCTION

The paper targets accessible dexterous teleoperation without expensive or restrictive hardware. It uses diverse internet human videos to learn human-to-robot retargeting and demonstrates a low-cost system across tasks and operators.

  • Motivation: Dexterous teleoperation remains difficult because existing approaches often require expensive hardware, specialized engineering, expert operators, or motion-restricting apparatus.The challenge increases with robot-hand complexity, making high-dimensional dexterous manipulation especially difficult to teleoperate.
  • Objective: The system aims to let any untrained operator control a dexterous robot hand in any environment using one uncalibrated color camera.The operator controls the robot remotely without gloves, markers, bulky motion-capture equipment, or multi-camera rigs.
  • Approach: The key insight is to leverage massive unlabeled internet video collections instead of collecting paired human-robot correspondence data.The videos contain varied people, viewpoints, tasks, and environments, supporting the intended generalization.
  • Approach: The method combines 3D human-pose estimation with motion retargeting to generate physically plausible robot hand-arm actions from passive online data.Training does not require active fine-tuning on the robot in the laboratory setup.
  • Evaluation: The resulting glove-free, marker-free system uses one uncalibrated camera to control a 16-DoF Allegro hand and is evaluated on ten manipulation tasks with ten previously untrained operators.The paper reports usability, versatility, generality, and robustness studies.

III. ROBOTIC TELEKINESIS

Robotic Telekinesis converts a single RGB image stream into simultaneous robot hand and arm commands. Its pipeline estimates human pose, retargets hand motion across different morphologies, and uses learned priors to mitigate single-camera ambiguity.

  • System overview: Each camera image is converted in real time into robot hand and arm poses matching the operator’s hand-arm configuration.The system uses an xArm6, a 16-DoF Allegro hand, and a single uncalibrated RGB camera.
  • Pipeline: The hand branch crops the operator’s hand, estimates its pose, and maps the estimated human hand pose to a robot hand pose.The arm branch estimates body pose and uses cross-body correspondences to determine the robot end-effector pose.
  • Challenges: Single-camera teleoperation is under-constrained because 2D images must specify 3D robot control and because human and robot hands differ in morphology, shape, and functionality.Inter-hand occlusions and ambiguous depth perception are typical failure modes of the color-only setup.
  • Retargeting: Deep neural networks learn priors from passively collected internet-scale human datasets to support human pose estimation and human-to-robot transfer.The approach separates hand retargeting into 3D human-hand pose estimation and mapping that pose to robot joint commands.
  • Generalization: The hand pose estimator is intended to generalize across operators, cameras, and environments without further fine-tuning.The authors attribute this generalization to training on millions of diverse images.

2) 3D Human Hand to Robot Hand Pose

The system retargets estimated 3D human hand poses to Allegro joint angles using a neural network trained without paired human-robot labels. Its differentiable energy compares relative hand keyvectors, enabling fast inference despite differing embodiments.

  • Challenges: Human-to-robot retargeting is underconstrained because human and Allegro hands differ in shape, size, and joint structure, while the system must generalize across operators and tasks.Real-time operation requires a solution above 15 Hz, making direct large-scale paired-data collection impractical.
  • Training Data: The method trains a deep retargeter using human data alone because millions of human poses lack ground-truth robot poses for direct regression.The dataset combines internet-scale hand images and videos, including Epic Kitchens and 100 Days of Hands.
  • Energy Function: Its energy function matches ten relative keyvectors between four human fingertips plus the palm and corresponding Allegro-hand keyvectors after scaling.The formulation emphasizes relative fingertip geometry rather than target fingertip poses relative to a fixed base.
  • Optimization: Because forward kinematics is differentiable, the energy provides a loss for gradient-descent training of the retargeter network.This differs from a conventional IK problem, whose constraints target fingertip poses relative to a fixed base.
  • Retargeter Network: The retargeter maps a 55-dimensional MANO hand shape-and-pose vector to 16 Allegro joint angles with a two-hidden-layer MLP.At inference, the network directly outputs robot joint angles, with a reported forward-pass time of about 3 ms (333 Hz).

3) Collision Avoidance via Adversarial Training

The system addresses self-colliding retargeted hand poses by adding a learned collision penalty during training. A classifier supplies differentiable gradients to the retargeter while expensive computation remains offline.

  • Deployment Advantage: Neural retargeting permits slow-to-compute energy terms to be evaluated during training rather than at deployment.This allows differentiable training objectives to incorporate operations whose computation would otherwise burden real-time control.
  • Collision Problem: Keyvector matching can produce Allegro configurations in which fingers collide with one another or the palm, but self-collisionness is not differentiable in joint angles.This makes direct incorporation of a collision penalty into the energy function difficult.
  • Collision Classifier: A separately trained MLP classifier predicts whether an Allegro joint-angle vector causes self-collision using programmatically sampled configurations and checker-generated binary labels.Training samples are drawn within legal joint limits.
  • Adversarial Training: During retargeter training, the classifier acts as a discriminator whose predicted collision score is added to the loss and backpropagated into the retargeter.The classifier’s gradient flows through the collision network to update retargeter weights.

B. Arm Teleoperation: Human Body to Robot Arm Poses

The arm branch estimates human body motion from a single floating RGB camera and transfers the wrist’s torso-relative transformation to the robot. Filtering and inverse kinematics then produce smooth arm commands, while experiments compare the system with an online-optimization baseline.

  • Arm Retargeting: The arm branch is necessary because a dexterous hand without arm mobility cannot solve many useful tasks.It computes the robot-arm pose from images of the human operator.
  • Pose Estimation: The pipeline detects the operator’s body, crops it, and estimates 3D joints using OpenPose-derived detection, FrankMocap, and the SMPL-X body model.The body pose estimator predicts 3D positions along the human kinematic chain.
  • Single-Camera Constraints: A single floating color camera leaves hand depth ambiguous and provides no known camera-to-robot transformation without depth sensing, intrinsics, or calibration.These constraints are central to operating with arbitrary uncalibrated cameras.
  • Relative Transform: The method transfers the human wrist’s torso-relative position and orientation to the robot by treating the human torso and a selected robot torso point as corresponding anchors.The robot wrist transformation is computed by traversing the human torso-to-wrist kinematic chain.
  • Control: Outlier rejection and low-pass filtering smooth estimated wrist poses before inverse kinematics computes arm joints placing the robot wrist at the desired relative transformation.The control stack also includes sampling and safety clipping before commands reach the robot.
  • Evaluation: The neural-network system outperformed the DexPilot-Monocular* online-optimization baseline on 7 out of 10 tasks and performed similarly on the other 3.The comparison used the same single-camera setup, while the baseline differed primarily in its online hand-pose retargeting procedure.

A. Success Rate: Trained Operator Study

The system supported dexterous manipulation by a trained operator and was evaluated against DexPilot-Monocular∗ and with previously untrained users. Operators generally found it responsive and easy to learn, though difficult hand poses caused periodic retargeting errors.

  • Success Rate: The trained operator completed diverse manipulation tasks successfully, with Robotic Telekinesis outperforming DexPilot-Monocular∗ on 7 of 10 tasks and performing similarly on the other 3.Fine-grained grasps of small, slippery objects such as plastic cups were occasionally difficult.
  • Success Rate: The baseline sometimes became stuck in local minima and produced unnatural hand poses, whereas the proposed method maintained natural poses and fluid, responsive operation.The proposed method was trained on YouTube data and did not use the baseline’s previous-pose seed.
  • Usability: Ten previously untrained operators each completed three tasks seven times, and the total learning-and-testing process took approximately 15 minutes per subject.The tasks were plush dice pickup, drawer opening, and placing a cup onto a plate.
  • Usability: All operators found the interface easy to understand and use despite differences in motion style, stance, and appearance.The study reported no noticeable discrepancies in system behavior or result distributions across operators.
  • Usability: Subjects often struggled during their first trials but quickly learned the system, reporting no noticeable lag or jitter and feeling safe and comfortable during teleoperation.Participants frequently complimented the system’s responsiveness and fluidity.
  • Limitations: Periodic retargeting errors were the main frustration, especially for complicated poses, thumb flexion, and fully open hands viewed with the palm parallel to the camera.The authors attribute these issues to human–Allegro thumb differences and single-camera ambiguity.

V. ANALYSIS

The analysis compares the neural retargeter with DexPilot-Monocular∗ and examines the trade-off between self-collision avoidance and pose similarity. The neural network better matched the oracle, while intermediate collision-loss weighting provided a practical balance between competing objectives.

  • Accuracy of retargeter network: The evaluation compared both retargeters on 500 DexYCB sequences using a pseudoground-truth oracle because ground-truth Allegro joint angles were unavailable.The oracle was DexPilot-Monocular∗ run with an infinite time budget until convergence.
  • Accuracy of retargeter network: 0.17 radians (about 10 degrees) RMSE was achieved by the neural retargeter versus 0.25 radians per joint (about 14 degrees) for DexPilot-Monocular∗ when matching the oracle.The comparison used root mean squared error between each retargeter’s outputs and the oracle’s outputs.
  • Self-Collision Avoidance: Increasing the self-collision-loss weight produced fewer offending joint configurations but degraded minimization of the other energy terms.The trade-off was evaluated using 3000 held-out FreiHand poses and six collision-loss weights from 0 to 1.
  • Self-Collision Avoidance: Higher adversarial self-collision-loss weight reduced self-collisions while increasing energy and reducing pose similarity to the operator.A higher energy indicates that the predicted robot pose is more dissimilar to the operator’s pose.
  • Self-Collision Avoidance: The authors report that a middle ground between self-collision avoidance and pose similarity worked effectively in practice.They did not confidently identify one objective as more valuable than the other.

APPENDIX A HAND/BODY BOUNDING BOX DETECTION

This section describes the perception and hand-retargeting components that convert cropped human imagery and estimated hand pose into Allegro joint configurations. The retargeter matches human and robot keyvectors while tuning scale factors for dexterous teleoperation.

  • Detection and hand pose estimation: The pipeline detects body and right-hand bounding boxes, prioritizing complete coverage over perfectly tight boxes.The hand box is centered at the right wrist and uses a conservative 150-pixel side length for 480x640 images.
  • Detection and hand pose estimation: A CNN processes a 224x224 hand crop to estimate SMPL-X hand shape, pose, and global orientation parameters.The parameters describe hand dimensions, finger arrangement, and root-frame orientation.
  • Human-to-robot hand retargeting: The retargeting module offers online gradient-based and offline neural-network implementations that map human hand poses to Allegro joint angles.The human global orientation is excluded from hand retargeting because the robot arm, rather than the Allegro hand, matches wrist orientation.
  • Human-to-robot hand retargeting: The shared energy function measures functional similarity through ten human and Allegro keyvectors, with robot vectors scaled by tunable constants.Each term is an Euclidean difference between corresponding human and scaled Allegro keyvectors.
  • Scaling choices: Values below 1 for the scaling constants are empirically preferred for dexterous manipulation teleoperation, despite perfect keyvector matching suggesting 1 in theory.A value around 0.625 is suggested for aesthetically appealing generic gestures, while functional similarity theoretically favors 1.

B. Computing the keyvectors on the human hand.

The system computes human and robot keyvectors in compatible coordinate frames, then exploits differentiable forward kinematics to optimize Allegro joint angles. An offline network is trained from both annotated hand configurations and diverse YouTube hand videos.

  • Computing human keyvectors: Human keyvectors are computed by posing a SMPL-X hand mesh, transforming it to a palm-centered canonical frame, and aligning keypoint frames with Kabsch-Umeyama.The canonical axes point outward from the hand, toward the thumb, and toward the middle fingertip.
  • Computing robot keyvectors: Allegro keyvectors are obtained with forward kinematics, which maps the 16-joint angle vector to link transformations corresponding to the hand keypoints.The Allegro URDF supplies the kinematic skeleton used by this mapping.
  • Optimization: Differentiable forward kinematics makes the energy differentiable with respect to Allegro joint angles, enabling gradient descent retargeting.The differentiability follows from the chain of trigonometric functions and matrix multiplications in forward kinematics.
  • Optimization: The online retargeter uses SGD with a zero-vector initial seed on the first iteration and the previous solution thereafter.It applies a fixed learning rate of 0.05 for a fixed number of steps.
  • Optimization: The offline retargeter is an MLP mapping 55-dimensional human hand parameters to 16 Allegro joint angles, with outputs constrained to valid joint ranges.The network has hidden layers of 256, 256, and 128 units with tanh activations.
  • Training data: Training combines over 30,000 annotated FreiHand configurations with hundreds of millions of frames from 100 Days of Hands YouTube videos.The latter supplies diverse in-the-wild hand motions without pose annotations.

A. Rough Body Pose Estimation via a CNN

The body-pose module estimates SMPL-X body parameters from a cropped image and improves the right-hand region using a dedicated hand estimate. The resulting human body pose is mapped through defined torso and wrist correspondences to the xArm6 end effector, then smoothed and converted to joint commands.

  • Rough body pose estimation: A ResNet50-based CNN and MLP estimate body shape, pose, and global orientation from a 224x224 body crop.The pose uses 24 joint rotation matrices within the SMPL-X body representation.
  • Body-hand integration: The hand estimate refines the rough body estimate because body inference can miss fine hand details and misestimate right-wrist rotation.The integration uses a zoomed-in hand crop whose global orientation is often more accurate.
  • Human-to-robot body retargeting: Body retargeting defines corresponding human and robot torso and wrist coordinate frames to determine the xArm6 end-effector pose relative to its base.The robot torso is defined 25cm above the base, with matching axis conventions for torso and wrist frames.
  • Trajectory smoothing: The controller clips large end-effector pose discrepancies and combines the target with an exponential moving average to smooth noisy estimates.The smoothing is intended to maintain stable motion when pose estimation and retargeting are noisy.
  • Trajectory smoothing: The system uses α = 0.25 for the moving average, while lower values can introduce lag.The authors report that high operating frequency makes this lag unproblematic in practice.
  • Trajectory execution: Linear interpolation creates waypoints that an SDLS inverse-kinematics solver converts into six xArm6 joint commands.The resulting commands are sent as a stream to the xArm6 controller.

APPENDIX G SOFTWARE ARCHITECTURE

The software is organized as a dataflow graph in which perception, pose estimation, integration, retargeting, and robot-control nodes exchange outputs through a publisher-subscriber architecture. Separate controller nodes interpolate hand and arm commands, while the camera supplies RGB frames at 30Hz.

  • Architecture: The architecture represents the system as computation nodes connected by dataflow edges and uses parallel computation within a publisher-subscriber design.This organization is intended to support efficient runtime execution.
  • Perception nodes: The CameraNode captures operator RGB images at 30Hz, while bounding-box detector nodes identify the hand and body regions.These detections feed the downstream pose-estimation modules.
  • Pose-estimation nodes: HandPoseEstimationNode and BodyPoseEstimationNode estimate SMPL-X parameters for the operator’s hand and body crops.BodyHandIntegrationNode combines both estimates using the Copy-and-Paste integration method.
  • Retargeting: HandRetargetNode converts hand pose estimates into Allegro joint angles that maximize similarity with the operator’s hand.The resulting target is passed to the Allegro hand controller.
  • Retargeting: BodyRetargetNode converts refined body pose into an xArm6 end-effector target relative to the robot base.It computes the transformation between the operator’s right wrist and torso before producing the robot target.
  • Robot control: AllegroHandControllerNode and xArm6ControllerNode interpolate target differences into fixed-size command streams sent at fixed frequencies.The two controller nodes separately handle hand joint angles and xArm6 joint configurations.

B. Optimizing performance via parallel computation.

The system improved runtime by replacing sequential module chaining with parallel ROS processes, while using a camera-based robot setup tested with previously inexperienced operators. The study included safety procedures and required subjects to adapt to the system during simple manipulation tasks.

  • Parallel computation: Parallel ROS processes increased runtime to approximately 25Hz by making retargeting depend only on the slowest node.The earlier sequential implementation achieved approximately 3Hz.
  • System setup: The robot setup used an xArm6 arm, an Allegro Hand with modified fingertips and grip tape, and an Intel RealSense D415 camera.
  • User study: Ten volunteer colleagues had never used the system before, although a few were familiar with the project.
  • Safety and procedure: Subjects were instructed to stay in camera frame, avoid moving too quickly, and could be stopped using an emergency switch.
  • User study: Many subjects were initially confused but quickly adapted while attempting the tasks without additional information.
  • Evaluation protocol: Failures were recorded when time expired, the object state made completion impossible, or a subject requested a reset.
Loading 2202.10448v2…