PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Human-level Dexterous Teleoperation

Puhao Li, Zeyuan Chen, Yingying Wu, Pengkun Wei, Yuyang Li, Tianyu Wang, Jiaxiao Shi, Mingrui Yu, Baoxiong Jia, Song-chun Zhu, Tengyu Liu, Siyuan Huang

arXiv:2607.11481v1cs.RO

TL;DR

Dexterous teleoperation still falls short of human-level in-hand manipulation, despite its potential for transferring operator skill to robots. TELEDEXTER trains a hand-object co-tracking controller from human references using consecutive subgoals, then transfers it zero-shot to real robots. Across seven tasks and two hands, it averages 75% success where baselines consistently fail.

  • Problem

    Current teleoperation systems remain short of human-level in-hand dexterity, while kinematic retargeting does not account for contact forces or object inertia.

    Source: §1 · p. 2

  • Method

    TELEDEXTER trains a low-level controller to track synchronized fingertip and object targets from human references using consecutive subgoals and a hybrid reward.

    Source: §1 · p. 2

  • Results

    75% average success across seven tasks and two dexterous hands, while baseline models consistently fail.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TELEDEXTER demonstrations also train autonomous policies via behavioral cloning for closed-loop manipulation without a human in the loop.

    Source: §1 · p. 3

    The controller is object-specific, so adapting it to a new object requires new human interaction data and dedicated policy training.

    Source: §6 · p. 9

Abstract

from arXiv · show

Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.

1 Introduction

Human-level in-hand dexterity remains an open challenge for teleoperation, as existing approaches struggle to execute dynamic, contact-rich manipulation. TELEDEXTER pairs hand-and-object targets with a learned controller and reports 75% average success across seven tasks where baselines consistently fail.

1 Introduction

  • Human-level in-hand dexterity remains an open challenge despite teleoperation’s potential to let operators teach robots directly.Such dexterity involves dynamically reorienting, translating, and regrasping objects through coordinated finger-object contacts.Source: §1 · p. 2
  • Kinematic retargeting ignores contact forces and object inertia, making high-acceleration, non-prehensile, and finger-gaiting actions unstable and prone to slips or drops.Learned action priors can instead rely on synthetic grasp transitions that may lack physical feasibility and accumulate drift during long-horizon deployment.Source: §1 · p. 2
  • TELEDEXTER specifies synchronized fingertip and object-pose targets, leaving a learned low-level controller to realize them through multi-contact physics.The controller is trained entirely in simulation with reinforcement learning.Source: §1 · p. 2
  • Consecutive subgoal co-tracking replaces rigid frame-wise imitation with ordered fingertip-and-object targets and combines sparse subgoal objectives with dense tracking rewards.The approach also uses geometry-aware reference construction and random action masking with domain randomization for zero-shot transfer.Source: §1 · pp. 2–3
  • 75% average success across seven tasks and two dexterous hands, while baseline models consistently fail.Collected demonstrations also train autonomous policies via behavioral cloning.Source: §1 · p. 3

2 Related Work

Prior work includes direct kinematic retargeting and learned dexterous action priors, but both leave challenges for contact-rich, long-horizon manipulation. TELEDEXTER instead trains an RL controller from human hand-object references using consecutive subgoals.

2 Related Work

  • Kinematic retargeting offers an intuitive, low-latency interface but lacks a dynamics prior for contact-rich reorientation, finger gaiting, and tool use.It maps human hand configurations to robot joints through vision, gloves, or learned neural mappings.Source: §2 · p. 3
  • DexGen [13] improves contact robustness with a simulated action prior, but synthetic grasp-transition goals may be physically infeasible and generative execution errors compound.These errors degrade real-world performance.Source: §2 · p. 3
  • Consecutive subgoal tracking addresses limitations of dense frame-wise imitation, which can restrict exploration of dynamic contact strategies and leave learned policies short of diverse in-hand skills.TELEDEXTER combines this formulation with action masking and domain randomization for zero-shot real-robot transfer.Source: §2 · p. 3

3 TELEDEXTER

TELEDEXTER treats dexterous teleoperation as hand-object co-tracking: an operator supplies fingertip and object targets, and a learned controller executes the contact dynamics. The pipeline constructs reference motions, trains the controller with consecutive subgoals, and deploys it zero-shot.

3 TELEDEXTER

  • The learned controller receives the current hand-object state and a goal specifying fingertip positions and object pose, while choosing how to realize those targets through contact.During teleoperation, goals come from captured hand-object poses; the arm independently tracks the wrist via inverse kinematics.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1 Learning a Co-tracking Controller

  • Consecutive subgoal co-tracking advances through ordered hand-object targets only after each is reached, allowing the policy to discover its own contact strategy between goals.A hybrid reward makes consecutive goal-reaching the dominant signal and adds dense tracking reward for early exploration.Source: §3.1 · pp. 4–5
  • Training uses a curriculum that increases gravity, tightens tracking tolerances, and lengthens inter-subgoal steps, with random-frame and cross-trajectory resets.Controllers are trained in a single RL stage without task-specific reward engineering.Source: §3.1 · p. 5
  • Domain randomization perturbs object and hand properties, external forces, observations, and latency; random action masking freezes sampled joint commands to reduce simulation overfitting.Masked action dimensions retain their previous command for a randomly sampled duration.Source: §3.1 · p. 5

3.2 Hand-Object Reference Motion Construction

  • Reference motions begin with motion-captured hand-object interactions spanning in-hand translation, rotation, and free-play, including arbitrary grasps, finger gaiting, and tool use.Geometry-aware retargeting first aligns robot hand geometry to human motion, then refines the trajectory using the object mesh.Source: §3.2 · p. 5
  • The retargeting objective attracts near-contact hand points to the object surface, penalizes penetration and self-collision, and smooths joint trajectories before producing fingertip and object targets.The surface-distance term uses a differentiable signed distance; the resulting targets form the co-tracking goals.Source: §3.2 · p. 6

3.3 Real-world Teleoperation Deployment

  • For real-world teleoperation, captured wrist, fingertip, and object poses supply controller goals, while kinematic retargeting initializes pre-grasp positioning before the operator switches to co-tracking.The learned controller deploys zero-shot.Source: §3.3 · p. 6

4 Experiments

TELEDEXTER is evaluated across real-world tasks, two hand embodiments, autonomous policy learning, and ablations of its tracking and action-masking designs. It achieves high success across seven teleoperation tasks, while demonstrations support behavioral cloning and both ablations identify benefits of the proposed designs.

4 Experiments

  • The experiments compare TELEDEXTER with representative baselines on seven tasks across two hand embodiments, then assess learned demonstrations and two design ablations.Each teleoperation task uses 15 trials and reports success rate and task progress.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 7

4.1 Experimental Setup

  • The evaluation spans seven staged tasks: three test in-hand reorientation across geometries, and four test long-horizon tool use requiring functional grasp transitions, finger gaiting, and tool application.Experiments use a Franka FR3 arm with LeapHand and SharpaWave embodiments; operators’ hand and object poses are tracked in real time, and baselines include kinematic retargeting and a learned action prior.Source: §4.2 · pp. 6–7, §4.1 · p. 6, Figure 3 · p. 7

4.2 Dexterous Teleoperation Evaluation

  • 75.2% average SR and 87.1% average TP across seven tasks; all baselines near-uniformly fail.TELEDEXTER reaches 66.7–80.0% SR on reorientation tasks and 66.7–86.7% SR on tool-use tasks; LeapHand achieves 60.0–73.3% SR on reorientation tasks using the same human reference motions.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.3 From Teleoperation to Autonomy

  • With 50 demonstrations per task, autonomous policies achieve 73.3% SR on HammerDriver, 46.7% on BulbInstall, and 40.0% on BrushForward.The primary bottlenecks are nail driving, precision alignment, and grasping, respectively; BrushForward trials that survive grasping nearly all complete rotation and sweep (6/7).Source: §4.3 · pp. 8–9, Table 3 · p. 8

4.4 Ablation Studies

  • Sparse consecutive subgoal tracking yields longer episodes than dense frame-wise tracking, while removing random action masking degrades all three evaluated real-world tasks.The text attributes the tracking advantage to allowing feasible contact strategies between goals, and describes masking as regularizing the action space against simulation-specific dynamics.Source: §4.4 · p. 9, Table 4 · p. 9, Table 5 · p. 9

5 Conclusion

TELEDEXTER combines consecutive subgoal tracking and a hybrid reward with random action masking to learn diverse in-hand skills through single-stage RL and transfer to real robots. Across seven challenging long-horizon tasks, baselines uniformly fail, while its demonstrations train autonomous policies.

  • TELEDEXTER learns diverse in-hand skills with consecutive subgoal tracking and hybrid rewards in single-stage RL, then transfers to real robots.The conclusion also identifies random action masking as part of the approach.Source: §5 · p. 9
  • Across seven challenging long-horizon dexterous tasks, TELEDEXTER achieves a strong success rate where baselines uniformly fail.The conclusion does not state a numerical success rate.Source: §5 · p. 9
  • Collected demonstrations successfully train policies to autonomously execute long-horizon dexterous tasks.Source: §5 · p. 9

6 Limitations

TELEDEXTER currently requires object-specific controllers and motion-capture-based real-time pose estimation for real-world deployment.

  • TELEDEXTER currently learns an object-specific controller, so adapting to a new object requires new human hand-object interaction data and a dedicated policy.A unified object-conditioned controller without per-object collection and training is described as future work.Source: §6 · p. 9
  • Real-world deployment relies on motion capture for real-time hand and object pose estimation.The authors propose markerless vision-based tracking as a way to lower deployment barriers.Source: §6 · p. 9

Supplementary Materials of Towards Human-level Dexterous Teleoperation

The appendix provides extended results and implementation details, covering evaluation, hardware and interface, TELEDEXTER, baselines, and autonomous policy training.

A.1 Any-to-Any Reposition Evaluation

The appendix tests arbitrary in-hand goal transitions on LeapHand and compares two training-recipe choices on Hammer. Results indicate strong repositioning on both tested objects, with differences associated with object geometry, curriculum, and optimizer choice.

A.1 Any-to-Any Reposition Evaluation

  • The any-to-any test streams randomly sampled feasible hand-object targets and counts consecutive reaches until the object slips or a target remains unreached for 20 seconds.A target counts as reached when position error is below 2 cm and rotation error below 20 degrees; 15 trials were run per object.Source: §A.1 · p. 13
  • 41.1 consecutive targets and 97.6% per-target success on Cylinder compare with 12.1 consecutive successes and 92.3% on Cuboid.The authors attribute the gap to the cylinder’s continuous, symmetric surface and cuboid edges and corners that can obstruct finger motion during regrasps.Source: Table 6 · p. 13, §A.1 · p. 13

A.2 Additional Ablation Studies

  • Disabling the Hammer curriculum yields faster initial progress but lower final performance; its advantage continues growing after annealing ends within 2 × 10^9 environment steps.The curriculum starts with small subgoal steps, permissive tolerances, and reduced gravity to establish stable contact primitives.Source: §A.2 · p. 14
  • Replacing SAPG with PPO substantially reduces Hammer reward and consecutive successes, as PPO’s unimodal updates under-cover diverse in-hand contact modes.SAPG aggregates gradients from multiple independently exploring policy blocks.Source: §A.2 · p. 14

A.3 Stage-Wise Teleoperation Success Analysis

Across four long-horizon tool-use tasks, TELEDEXTER retains trials through demanding contact-intensive stages where baselines sharply collapse. The failure analysis identifies impact dynamics, contact-transition jams, and missing tactile observations as distinct limitations.

A.3 Stage-Wise Teleoperation Success Analysis

  • 13–15/15 trials survive TELEDEXTER’s demanding mid-task stages across four tasks, while every baseline sharply collapses at or shortly after the first in-hand reorientation or grasp-transition stage.The stage-wise plot compares trials reaching each stage across methods.Source: §A.3 · p. 14, Figure 6 · p. 15
  • 15/15 HammerUse trials reach stage 3 and 10/15 reach final placement; no teleoperation baseline retains a trial through stage 4.SimToolReal† reaches 14/15 at stage 2 before collapsing at stage 3, while SimToolReal‡ maintains 3/15 through completion.Source: §A.3 · pp. 14–15
  • 11/15 TELEDEXTER trials complete BrushSweep, versus 4/15 for the strongest baseline, SimToolReal†.The cited bottleneck is the stage-4 bristles-right rotation while maintaining grasp.Source: §A.3 · p. 15
  • 11/15 TELEDEXTER trials finish ScrewdriverUse after 13/15 persist through tightening, while DexGen and both SimToolReal variants fall to zero at tightening.DexGen’s action prior cannot sustain continuous axial rotation.Source: §A.3 · p. 15
  • 15/15 TELEDEXTER trials pass both BulbReplace rotational stages and 13/15 finish, while GeoRT and DexGen reach zero at the screw-in stage.SimToolReal is omitted because BulbReplace falls outside its object-centric formulation.Source: §A.3 · p. 15

A.4 TELEDEXTER Failure Analysis

  • Three real-world failure modes are identified: impacts shift grasps beyond the free-space training distribution, compliant fingers can jam during contact transitions, and absent tactile sensing impedes regrasp adaptation.The analysis links these respectively to missing impact dynamics in training, a simulation-to-hardware compliance mismatch, and inability to distinguish pressing from sliding contact.Source: §A.4 · pp. 16–17, Figure 7 · p. 16

B.1 Teleoperation Task Definitions

This section defines the teleoperation and autonomous tasks and describes the sensing, control, training, and reference-motion procedures used for TeleDexter. It covers hand–object co-tracking and geometry-aware retargeting across two robot hands, including progressive training conditions and sim-to-real measures such as domain randomization and random action masking.

B.1 Teleoperation Task Definitions

  • Teleoperation trials use ordered task stages: task progress averages the furthest-stage fraction reached, success rate counts trials completing all stages, and stage 0 denotes failure to pick up the object.The tested objects are wrapped in medical bandage tape to increase surface friction.Source: §B.1 · p. 17
  • Three reorientation tasks share a three-stage sequence—pickup, in-hand reorientation to a target pose, and placement—across cylindrical, corner-rich, and irregular freeform objects.The tool-use tasks extend this staged format to repeated hammering and nail pulling, multi-direction brush sweeping, screwdriver use, and bulb installation followed by removal.Source: §B.1 · p. 17

B.2 Autonomous Policy Task Definitions

  • Autonomous Diffusion Policies are evaluated on simplified subsets of bulb replacement, hammer use, and brush sweeping that retain dexterous stages but remove return-and-place phases.The subsets respectively cover bulb installation, hammer rotation and driving nails into foam, and a single forward brush sweep; trials end on unrecoverable failure.Source: §B.2 · p. 18

B.3.1 Motion Capture System; B.3.2 Teleoperation Interface

  • NOKOV motion capture supplies object pose in both settings, while dense offline glove markers capture hand joints and the compact teleoperation glove provides wrist pose and fingertip positions at 30 Hz.The interface maps wrist pose to a Franka arm target and feeds wrist-frame fingertip and object pose goals to the learned hand policy; kinematic vector retargeting handles reaching and grasp acquisition before policy-controlled in-hand motion.Source: §B.3.1 · p. 18, §B.3.2 · p. 19

C.1 Observation & Action Space

  • The observation omits joint velocities and contact forces, relying on kinematic state, previous action, and a gravity cue; a soft action deadzone lets the policy hold its current joint command.The observation uses gravity direction as its only world-frame orientation cue; action outputs within ±0.1 produce zero command delta.Source: §C.1 · pp. 19–20

C.2 Complete Reward Design

  • The co-tracking reward combines sparse consecutive-subgoal bonuses, dense tracking, and a time cost; subgoal success requires fingertip and object pose errors to remain within tolerance for consecutive frames.The step weight scales the sparse bonus with the temporal gap between subgoals, discouraging exploitation of trivially close goals, while dense object tracking ramps in as training hardens.Source: §C.2 · pp. 20–21

C.3 Curriculum Schedule

  • Training progressively increases subgoal distance and action-mask duration, ramps in dense object tracking, and anneals simulator gravity from zero to −9.8 m/s2.The subgoal distance bound grows from 40 to about 80 reference frames, and the action-mask duration bound grows from 1 to 10 frames.Source: §C.3 · pp. 21–22

C.4 Reset Conditions; C.6 Hand–Object Reference Motion Construction

  • After each hit, the next subgoal can switch to another trajectory for the same object, training transitions across demonstrations needed for operator motions beyond any recorded trajectory.Reference motion consists of 150 unscripted 20-second trajectories per object, spanning in-hand translation, rotation, and free-play with arbitrary grasps, finger gaiting, and tool use.Source: §C.4 · p. 22, §C.6 · p. 23

C.6 Hand–Object Reference Motion Construction

  • Retargeted LeapHand and SharpaWave clips reproduce operator grasp poses and contact transitions for Cylinder and Cuboid despite human–robot kinematic differences.The clips compare source MANO configurations with retargeted robot-hand configurations.Source: §C.6 · p. 23, Figure 9 · p. 24

C.7 Domain Randomization

  • Domain randomization independently samples hand and object dynamics, external perturbations, sensing noise, and observation latency at each episode’s start, holding values fixed throughout.The parameter ranges are listed in Tab. 9.Source: §C.7 · p. 24, Table 9 · p. 25

C.8 Random Action Masking Details

  • Random action masking regularizes the action space against simulation’s synchronized actuation, while its stale joint commands train recovery from real-hardware motor lag, backlash, and missed commands.It was the single most impactful sim-to-real intervention tested; masks freeze selected DoFs by reusing previously commanded actions for a sampled duration.Source: §C.8 · pp. 24–25

C.9 RL Training Hyperparameters and Compute

The section reports SAPG hyperparameters and the compute setup used to train the controllers.

C.9 RL Training Hyperparameters and Compute

  • Training used 4 NVIDIA RTX 5090 GPUs, with 15,600 parallel environments per GPU and 62,400 environments total.Source: §C.9 · p. 25
  • Each controller was trained for approximately 10^10 environment steps, taking about 1 day on this setup.Source: §C.9 · p. 25
  • Tab. 10 lists the hyperparameters used for SAPG.The controllers were trained in a single RL stage.Source: §C.9 · p. 25, Table 10 · p. 26

D Baseline Implementation Details

The section describes how the baselines were implemented for evaluation and outlines the autonomous policy’s architecture and training hyperparameters.

D Baseline Implementation Details

E Autonomous Policy Details

  • The autonomous policy uses RGB input from third-person and wrist cameras and predicts 16-step action chunks from the last 2 frames.Separate frozen DINOv2 ViT-S/14 encoders process the two camera views; Tab. 11 lists hyperparameters.Source: E Autonomous Policy Details · p. 27, Table 11 · p. 27

Related papers

Loading 2607.11481v1…