PaperlayerView exact revision on arXiv

Source-linked AI summary

DITTO-X: Forward and Reverse Teleoperation for Dexterous Manipulation and Human Intervention

Zhanpeng He, Joaquin Palacios, Zhangyu Wang, Chenhao Li, Katelyn Lee, Matei Ciocarlie, C. Karen Liu, Jiajun Wu

arXiv:2610.00781v1cs.RO

TL;DR

Vision-only, parallel-jaw teleoperation limits dexterous manipulation and leaves operators without contact cues, while occlusion and inherited hand-state mismatch complicate mid-task intervention. DITTO-X provides hand-agnostic force and fingertip feedback and uses reverse teleoperation to match the operator to the robot before takeover. The paper reports improved data quality and throughput for regular collection and human-in-the-loop policy deployment.

  • Problem

    Vision-only, parallel-jaw teleoperation limits dexterous expression and leaves operators without contact cues, while occlusion and hand-state mismatch complicate mid-task intervention.

    Source: §I · p. 1

  • Method

    DITTO-X provides joint-level force and fingertip contact feedback across commercial dexterous hands, and back-drives the operator’s fingers to the robot’s configuration before intervention.

    Source: §I · p. 2

  • Results

    DITTO-X improves the throughput and quality of contact-rich policy-training data, with reverse-teleoperation interventions yielding better data for subsequent policy improvement.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results point toward a more efficient way to collect high-quality robot data across regular teleoperation and human-in-the-loop policy deployment.

    Source: §VI · p. 8

Abstract

from arXiv · show

Teleoperated demonstrations are a primary source of data for robot manipulation, and teleoperated interventions are a primary mechanism for correcting policies at deployment. Yet most teleoperation systems close the loop through vision alone and are built around parallel-jaw grippers, limiting both what the robot can execute and what the operator can express through it. This is most damaging in shared autonomy, where the operator sees the scene only through occluded cameras and must take over a dexterous hand mid-task, often with an object already grasped. We present DITTO-X, a hand-agnostic dexterous teleoperation interface that renders joint-level force and fingertip contact events from sensing already on the robot hand, and drives three commercial dexterous hands (Sharpa, Wuji, and Inspire) without per-hand redesign. Because the exoskeleton is actuated, DITTO-X also supports reverse teleoperation, in which the robot back-drives the operator's fingers into its own configuration before control is transferred, so the human enters the loop already matched to the state they inherit. Our results show that DITTO-X improves demonstration quality and throughput over a commercial hand-tracking glove, both in regular data collection and in human intervention during policy deployment for contact-rich manipulation tasks. More information can be found from our website: https://tml.stanford.edu/ditto-x/.

I. INTRODUCTION

Vision-only, gripper-centered teleoperation limits both dexterous robot actions and what operators can demonstrate, with occlusion and mid-task state mismatch making policy interventions especially difficult. DITTO-X combines force and fingertip feedback across commercial hands with reverse teleoperation, and the reported results show improved data quality and throughput.

I. INTRODUCTION

  • Most existing systems use parallel-jaw grippers and vision alone, limiting dexterous expression and leaving operators without contact cues, which can make demonstrations slow and conservative.Operators regulate contact forces, detect incipient slip, and reshape grasps using cues they feel.Source: §I · p. 1
  • During policy intervention, onboard-camera occlusion and inherited hand-state mismatch can make takeover discontinuities drop or break an object, causing unrecoverable failures.These constraints are especially relevant during tool use or in-hand manipulation.Source: §I · p. 1
  • DITTO-X renders sustained joint-level force through exoskeleton motors and discrete fingertip contact events through haptics, using sensing already available on the robot hand.The framework supports Sharpa Wave, Wuji 2, and Inspire RH56DFX without per-hand redesign.Source: §I · p. 2
  • DITTO-X improves the throughput and quality of contact-rich policy-training data, while reverse-teleoperation interventions yield better data for subsequent policy improvement.The paper reports fluent interventions without dropping objects already in hand.Source: §I · p. 2
  • In reverse teleoperation, the actuated exoskeleton back-drives the operator’s fingers to match the robot’s configuration before control transfers.This makes intervention begin from a matched state rather than a discontinuous handoff.Source: §I · p. 2
  • The authors open-source the system and release 1.8k contact-rich episodes spanning three embodiments, including demonstrations, human interventions, and autonomous policy rollouts.Source: §I · p. 2

II. RELATED WORK

Prior dexterous teleoperation work addresses hand kinematics or renders force at limited resolutions, while intervention research largely studies when to request help. DITTO-X extends joint-level feedback across five-finger hands and uses reverse teleoperation to match the operator’s state before takeover.

II. RELATED WORK

  • Dexterous teleoperation has largely targeted closing the operator’s hand-to-robot kinematic gap; vision-based hand tracking, non-wearable leader-follower designs, and learned retargeting priors [11] leave contact inferred from vision alone.Source: §II · p. 2
  • DITTO-X removes DITTO’s one-to-one leader-follower restriction and extends joint-level feedback to five-finger hands.DITTO [8] provided joint-level feedback but was tied to a single two-fingered manipulator.Source: §II · p. 2
  • Where intervention research often studies when to request help, DITTO-X focuses on how to intervene on a policy.Interactive imitation learning aggregates expert corrections during execution, and prior work also studies intervention timing.Source: §II · p. 2
  • Reverse teleoperation conforms the human hand to the robot before authority passes, addressing a mid-task pose mismatch that can release or displace a grasped object.Unlike blended or traded control, this approach makes the operator’s state continuous at transfer.Source: §II · p. 2

III. METHOD

  • Forward teleoperation maps human motion to the robot and returns contact through force and haptic feedback; reverse teleoperation drives the operator’s fingers to the robot’s configuration for takeover.Source: §III · p. 2

A. Forward Teleoperation Across Dexterous Hands

DITTO-X adapts the DITTO exoskeleton for heterogeneous dexterous hands by adding middle-finger actuation and fingertip haptics, while using different mappings for directly corresponding joints, thumbs, and underactuated fingers.

A. Forward Teleoperation Across Dexterous Hands

  • DITTO-X builds on DITTO’s anatomically aligned thumb and index axes, whose one-to-one kinematics support joint-level position control and force feedback but limit use with other hands.Source: A. Forward Teleoperation Across… · p. 3
  • DITTO-X adds a motorized middle-finger module, couples target-hand ring and pinky motion to that finger, simplifies the thumb, and adds fingertip vibrotactile actuators.The design assumes three independent fingers suffice for varied dexterous primitives, with coupled fingers supporting power grasps.Source: A. Forward Teleoperation Across… · p. 3

1) Position Retargeting under Heterogeneous Kinematics:

  • For index and middle fingers, corresponding MCP abduction, MCP flexion, and PIP flexion joints map directly, while unmatched joints stay neutral.Source: 1) Position Retargeting under… · p. 3
  • Thumb retargeting uses palm-frame thumb-tip pose and inverse kinematics, while underactuated fingers map total DITTO-X flexion to one target-hand flexion degree of freedom.Thumb scaling accounts for differences in thumb length.Source: 1) Position Retargeting under… · p. 3
  • Table I compares supported robotic hands and indicates whether each target finger permits direct one-to-one joint mapping with DITTO-X.Source: Table I · p. 3

2) Force and Haptic Feedback under Heterogeneous Sensing:

DITTO-X derives force feedback from each robot hand’s available sensing and maps it to the exoskeleton’s joints. It complements sustained force feedback with fingertip vibration for contact events.

2) Force and Haptic Feedback under Heterogeneous Sensing:

  • Motor-current estimates or fingertip tactile sensing provide the target-hand forces used to compute desired feedback torques.Sharpa’s tactile force vectors can bypass joint-to-task-space conversion; Inspire’s per-finger force magnitudes are treated as fingertip forces.Source: 2) Force and Haptic Feedback under… · p. 3
  • With 1:1 joint correspondence, estimated torque maps directly to the matching exoskeleton joint; otherwise, force is retargeted through fingertip Jacobians and frame transforms.For hands without joint correspondence, motor-current torque estimates are converted to fingertip force; Inspire’s force magnitudes use the same Jacobian-transpose mapping.Source: 2) Force and Haptic Feedback under… · p. 3
  • The DITTO control law converts desired feedback torque into proportional actuator current, using a tunable gain and, for motor-current estimates, a threshold to reject free-space bias.For tactile-derived torques, the threshold is zero because the sensor directly measures external contact force.Source: 2) Force and Haptic Feedback under… · p. 4
  • Fingertip vibration conveys fast contact events, while joint-torque feedback continues to convey sustained grasp force.Contact-event estimation is supported only for hands with fingertip tactile sensing; vibration marks contact onset and decays before it can be retriggered.Source: 2) Force and Haptic Feedback under… · p. 4

B. Reverse Teleoperation for Fluent Human Intervention

Reverse teleoperation keeps the operator’s hand matched to the robot during policy control, addressing configuration mismatch at takeover. A motion threshold gates transfer, and the operator can return control to the policy before the episode ends.

B. Reverse Teleoperation for Fluent Human Intervention

  • During policy control, reverse teleoperation projects the robot hand’s commanded configuration onto the exoskeleton so takeover begins from a matched state.This is intended to avoid the handoff discontinuity that can drop or crush a grasped object and fail the task.Source: B. Reverse Teleoperation for Fluent… · p. 4
  • When joint correspondence is 1:1, reverse projection is exact; otherwise, the system solves for exoskeleton joints matching the target hand’s fingertip pose.For retargeted fingers, it expresses the fingertip pose in the exoskeleton palm frame, rescales it, and solves for joint positions.Source: B. Reverse Teleoperation for Fluent… · p. 4
  • A keyboard takeover request transfers control only after the measured exoskeleton configuration departs from the projected configuration beyond a joint threshold.The inverse-kinematics solve is seeded from the previous projection so the reference does not follow the operator’s displaced hand.Source: B. Reverse Teleoperation for Fluent… · p. 4
  • The operator can return control to the policy through the same trigger and interleave forward and reverse teleoperation within an episode.Source: B. Reverse Teleoperation for Fluent… · p. 4
  • Safety measures include comfortable software joint limits, conservative joint-dependent motor-current limits under 200 mA, and an emergency stop.The stated motor-current limits correspond to ≤0.08 N⋅m.Source: B. Reverse Teleoperation for Fluent… · pp. 4–5

IV. EXPERIMENTS

The experiments assess force information and data-collection performance, reverse teleoperation for policy intervention, and use across diverse dexterous hands.

IV. EXPERIMENTS

  • The experiments ask whether DITTO-X provides useful force information and improves throughput and success rates when collecting dexterous, forceful task data.Source: §IV · p. 5
  • A second question is whether reverse teleoperation improves intervention on a learned policy and yields data for continued policy improvement.Source: §IV · p. 5
  • The evaluation also asks whether DITTO-X can be used with diverse dexterous hands.Source: §IV · p. 5

A. User Study

The user study tests perception and task performance with DITTO-X, while the policy-learning experiments assess data collected through different interfaces on contact-rich tasks. The protocol includes intervention on a running tool-use policy and evaluation across matched rollout conditions.

A. User Study

  • Six participants perform blind object-size and compliance discrimination using feedback alone, including soft, rigid, and no-object conditions.Size trials use cylinders of 30, 50, or 70 mm diameter; each participant completes six trials per condition.Source: A. User Study · p. 5
  • Participants compare DITTO-X with Manus Pro on tong use, with success rate and time per success as measures.Manus tracks hand pose but provides no feedback and cannot drive the operator’s hand, isolating the bilateral loop.Source: A. User Study · p. 5
  • For intervention trials, participants take over after the robot picks up tongs; a dropped tool counts as full-task failure.Each participant performs ten trials per condition.Source: A. User Study · p. 5

B. Contact-rich Data Collection for Policy Training

  • The policy-training tasks include tong use, fragile raspberry grasping, and battery insertion, with episodes designed to preserve an existing grasp during takeover.The raspberry tests grip regulation near the lower limit of a stable grasp; battery insertion requires directional force while maintaining the grasp.Source: B. Contact-rich Data Collection for… · p. 5, Figure 4 · p. 5
  • Each interface supplies 80 tong demonstrations and 60 demonstrations each for raspberry and battery; policies share architecture and training settings, then receive 30 paired rollouts per task.At least ten rollout starting conditions per task are outside the pretraining object-placement distribution, and a stage counts only after all preceding stages succeed.Source: B. Contact-rich Data Collection for… · pp. 5–6

C. DAgger with Human-Intervened Data

The DAgger experiments use human interventions collected during policy rollouts, with operators taking over at their discretion after evaluating the current policy. DITTO-X matches the operator’s hand to the robot’s pose before takeover, while a quantity-matched baseline tests whether iterative corrections add value beyond more demonstrations.

Intervention collection

  • Each DAgger round starts with a full policy evaluation, then collects 20 intervention trajectories as the operator takes over at their discretion.Successful interventions are retained, and the policy is fine-tuned for 10 epochs; each interface supplies data only to its own policy.Source: C. DAgger with Human-Intervened Data · p. 6
  • DITTO-X back-drives the operator’s fingers to the robot’s current configuration before takeover; Manus transfers control directly, leaving the operator to reconcile the pose unaided.All DITTO-X interventions in the reported DAgger runs used reverse teleoperation.Source: C. DAgger with Human-Intervened Data · p. 6

Quantity-matched comparison

  • A quantity-matched baseline uses the same demonstration count as DAgger round two, but collects non-iteratively from the original task distribution to isolate the value of policy-failure states.The matched counts are 120 demonstrations for tong and 100 each for raspberry and battery.Source: C. DAgger with Human-Intervened Data · p. 6

D. DITTO-WUJI and DITTO-INSPIRE

DITTO-X is evaluated on additional tasks using Wuji and Inspire hands, and in user studies comparing data collection and policy intervention with Manus. The reported studies find higher success and throughput with DITTO-X, while the multi-hand tasks exercise varied grasping and hand use.

D. DITTO-WUJI and DITTO-INSPIRE

  • Wuji demonstrations cover toy pick-and-place and drawer opening, while Inspire demonstrations cover bunny lifting; the drawer task switches among hook, pinch, and fist grasps.The drawer task also tests a grasp relying on coupled fingers alone.Source: D. DITTO-WUJI and DITTO-INSPIRE · p. 6

A. User study results

  • With both feedback modes active, subjects identified object size correctly on 86.1% of trials and compliance on 91.7%; disabling either mode impaired both tasks.Without vibrotactile feedback, size accuracy was 61.1% and compliance accuracy 80.6%; without force feedback, they were 61.1% and 47.2%, respectively.Source: A. User study results · p. 6
  • On T3 data collection, DITTO-X succeeded on 70.0% of trials versus 46.7% for Manus, and time per success fell from 85.5 s to 56.1 s.The success-rate difference held for all six subjects.Source: A. User study results · p. 6
  • Even the subject with over 20 hours of prior Manus experience and no prior DITTO-X exposure succeeded more often with DITTO-X.This subject was S3 in the T3 data-collection study.Source: A. User study results · pp. 6–7
  • On T4 intervention, DITTO-X achieved 78.3% success versus 27.7% for Manus, while time per success fell from 97.4 s to 35.7 s.The authors note that policy assistance raised success for DITTO-X, but not Manus, where intervention failures offset the assistance.Source: A. User study results · p. 7
  • In the battery example, the operator recovers a slipping battery during policy execution, places it in the holder, and hands control back for insertion.The figure distinguishes intervention with a blue border and policy execution in green; its upper and lower rows show DITTO-X and Manus examples.Source: Figure 6 · p. 6, A. User study results · p. 7

B. Policy Learning and DAgger Results

Policy-learning experiments compare DITTO-X and Manus demonstrations, iterative DAgger corrections, and a quantity-matched non-iterative baseline. DITTO-X policies perform better across the reported tasks, with corrections addressing failure states and distinct stage-level gains.

Demonstration quality

  • Before DAgger, policies trained on DITTO-X demonstrations outperform those trained through Manus on all three tasks.The largest margin is on tong, where grasp instability can propagate through the task.Source: B. Policy Learning and DAgger Results · p. 7
  • DITTO-X picks up the toy on 66.7% of rollouts versus 23.3% for Manus, despite comparable tong-acquisition rates; the gap emerges at stable tool use.The stage-wise table credits a stage only when all preceding stages succeeded.Source: B. Policy Learning and DAgger Results · p. 7, Table II · p. 7

Iterative correction

  • DITTO-X improves monotonically on every task across DAgger rounds, while Manus improves at round one then declines on two of three tasks.Manus ends below its round-one performance on tong and raspberry.Source: B. Policy Learning and DAgger Results · p. 7
  • On battery, DITTO-X’s second DAgger round raises insertion success from 56.7% to 90.0%, while grasping and alignment are already largely reliable after round one.The remaining failures after round one occur mainly at insertion.Source: B. Policy Learning and DAgger Results · p. 7

Data quantity vs DAgger

  • The quantity-matched non-iterative baseline falls short on all three tasks and does not improve battery performance over pretraining, unlike corrections collected at policy-failure states.The comparison matches the number of demonstrations in DAgger round two.Source: B. Policy Learning and DAgger Results · p. 8
  • With DAgger, the raspberry policy learns to reach out-of-distribution bowl locations, where the pretrained policy drops the raspberry in the wrong locations.This is an example of iterative corrections covering states absent from the original task distribution.Source: B. Policy Learning and DAgger Results · p. 8

Data collection and intervention quality

  • A proficient operator produced usable trajectories in 93.9% of DITTO-X versus 71.7% of Manus pretraining attempts, and 96.5% versus 58.2% during intervention.The operator was distinct from the six novices in the user study and proficient with both interfaces.Source: B. Policy Learning and DAgger Results · p. 8

VI. CONCLUSIONS

DITTO-X combines forceful teleoperation and human intervention across diverse robotic hands through multi-DoF force and fingertip haptic feedback. Its actuated design supports reverse teleoperation, and the paper reports improved success and throughput in regular teleoperation and policy deployment.

VI. CONCLUSIONS

  • DITTO-X supports forceful teleoperation and fluent human intervention across diverse hands, with multi-DoF force and fingertip haptic feedback.Its actuated design lets the operator match the robot’s state before intervening in a learned policy.Source: §VI · p. 8
Loading 2610.00781v1…