XRoboToolKit-T: Teleoperation with High Stability and Precision with Tactile Sensing for Contact-rich Manipulation

I. INTRODUCTION

• Visual information alone is insufficient for contact-rich tasks, while abrupt or unintended operator inputs can cause excessive force and task failure.Precise control of where force is applied also matters.Source: §I · p. 1
• XRoboToolkit-T adds a contact force stabilizer and a Vision-Language-Action-based action refiner to provide tactile-informed assistance.The stabilizer estimates tangential forces from pressure distributions; the refiner uses tactile images and task context to generate refined actions.Source: §I · pp. 1–2
• The stabilizer filters aggressive or unintentional operator motion, while the refiner generates corrective actions from tactile observations and task context.Their outputs are fused to adjust the operator’s manipulation plan.Source: §I · p. 2
• The system is designed for high-frequency tactile-enabled data collection and adapts to gripper- and dexterous-hand teleoperation with deformable objects.Source: §I · p. 2
• The authors evaluate the system on rubber-pipette liquid transfer, bottle and cup grasping, and medical-syringe insertion tasks.Source: §I · p. 2

A. Teleoperation

• Teleoperation demonstrations support robot skill learning, while accessible vision-based systems and data-acquisition platforms reduce reliance on specialized wearable hardware.The section cites AnyTeleop, Open-Teach, and low-cost hardware configurations as examples.Source: A. Teleoperation · p. 2
• Low-cost interfaces have also supported learning fine-grained bimanual manipulation policies from human demonstrations.Source: A. Teleoperation · p. 2

B. XR Human Robot Interaction

• XR teleoperation uses spatial perception and embodied interaction; related systems provide immersive visual feedback and XR interfaces for simulation control.Source: B. XR Human Robot Interaction · p. 2

C. Contact-rich Manipulation

• Visual-only manipulation policies lack tactile feedback for reasoning about contact dynamics and regulating interaction forces.The cited work also describes tactile sensing for assembly, in-hand rotation, and fragile-object grasping.Source: C. Contact-rich Manipulation · p. 2

A. Tactile Perception

• The system supports both end-effectors with integrated tactile sensing and two-finger grippers augmented with external tactile sensors.A generic sensor-reading module converts measurements into a unified format and synchronizes them with robot kinematics.Source: A. Tactile Perception · p. 3, Figure 2 · p. 3
• Tactile visualizations encode contact strength through pressure heatmaps, with gray regions indicating weak or absent contact.Source: A. Tactile Perception · p. 3

B. Dexterous Hand Control

• A morphology-agnostic interface supports sensorless and tactile-enabled dexterous hands, mapping commands to joint actuation while accounting for kinematic structures and joint limits.Source: B. Dexterous Hand Control · p. 3
• Operators can control dexterous hands through tracked hand gestures retargeted to robot joints or controller inputs transmitted at up to 90 Hz.Gesture tracking represents the human hand with 26 joint poses.Source: B. Dexterous Hand Control · p. 3

C. Teleoperation with haptic feedback

• The feedback stream asynchronously publishes joint states, end-effector pose, and tactile or force measurements to client applications.Source: C. Teleoperation with haptic feedback · p. 4
• Tactile feedback represents normal-force pressure grids, tangential shear forces, and binary contact states for detecting contact transitions.Source: C. Teleoperation with haptic feedback · p. 4
• End-effector feedback includes type, active degrees of freedom, pose, velocity, torque, and status, while robot-joint feedback includes joint state and mobile-base information.Source: C. Teleoperation with haptic feedback · pp. 4–5

A. Tactile assistance

• Pseudo-shear direction follows the maximum-pressure cell’s displacement across frames, while its magnitude is proportional to maximum normal pressure.The stabilizer uses this estimate to modulate teleoperation; large shear suppresses commands, while shear below a threshold permits normal control.Source: A. Tactile assistance · p. 5
• Abrupt changes in the maximum-pressure location’s direction indicate unstable contact, prompting the stabilizer to temporarily disable teleoperation.The stabilizer coefficient is set to zero in these cases.Source: A. Tactile assistance · p. 5
• The stabilizer operates at 20 Hz to support stable and consistent tactile feedback during manipulation.Source: A. Tactile assistance · p. 5
• The tactile refiner uses a VLA model to predict corrective end-effector motion from tactile pressure images and task instructions at approximately 7 Hz.Its output is a small Cartesian displacement along x, y, and z.Source: A. Tactile assistance · p. 5

B. Tactile-driven teleoperation

• The system combines stabilizer control and VLA refinement at multiple temporal scales.Source: B. Tactile-driven teleoperation · p. 5
• At approximately 20 Hz, the stabilizer uses tactile pressure-map variation to estimate contact stability and suppresses excessive operator motion when unstable shear or abrupt contact changes are detected.Source: B. Tactile-driven teleoperation · p. 5
• The VLA processes tactile images with task-level context to produce motion refinements that compensate for small pose errors during contact-rich interaction.The refinement is applied only when a new signal is available, as indicated by Is(t).Source: B. Tactile-driven teleoperation · p. 5

V. EXPERIMENTS AND DEMONSTRATIONS

• The two-finger setup mounts Robotiq grippers on dual UR5e arms, with each gripper carrying two 6×5 piezoresistive tactile arrays.Source: §V · p. 5
• The dexterous-hand setup replaces the grippers with two six-motor Realhand L6 hands, each integrating five 12 × 6 piezoresistive tactile arrays.Source: §V · p. 5
• The described hardware configurations support contact-force measurement during manipulation through tactile sensor arrays.Source: §V · p. 5

A. Two-finger grippers and dexterous hands application

• The syringe task proceeds from retrieval and cap removal to needle insertion into an artificial vessel in a venipuncture training pad.The needle is under 1 mm in diameter, and the pad contains three artificial vessels of approximately 5 mm diameter.Source: A. Two-finger grippers and dexterous… · pp. 5–6
• The rubber-pipette task transfers liquid using a deformable bulb and is performed with both a two-finger gripper and a dexterous hand.The bulb measures 3 cm long and 2.1 cm in diameter.Source: A. Two-finger grippers and dexterous… · p. 6
• The table reports successful manipulations completed in 15 minutes for different end-effectors.Source: Table II · p. 6
• The dual-arm grasping application handles water bottles and paper cups from a cluttered workspace, lifting them and placing them into a designated container.Source: A. Two-finger grippers and dexterous… · p. 6, Figure 5 · p. 6

• 138 successful grasps in 15 minutes: XRT-T with a two-finger gripper exceeds Twist2’s 128 successful manipulations in the same duration on paper cups.The dexterous-hand XRT-T configuration completes 132 successful grasps in 15 minutes.Source: B. Results · p. 7

Control frequency

• 96.4 Hz: tactile-enabled XRT-T exceeds hand-tracking XRT-T at 48.6 Hz and original XRT at 20.2 Hz in dexterous-hand control frequency.The passage attributes the higher rate to direct tactile feedback independent of the vision pipeline.Source: B. Results · p. 7, Table III · p. 7

Qualitative results

• XRoboToolkit-T performs micro-adjustments during syringe insertion and extracts liquid, while XRoboToolkit bends the needle and fails to extract liquid.The baseline’s accumulated contact force also triggers a robot safety emergency stop.Source: B. Results · p. 7, Figure 6 · p. 7

C. Ablation study

• 9.6 manipulations per minute: combining the stabilizer and refiner outperforms XRT at 8.5, XRT-T + refiner at 9.0, and XRT-T + stabilizer at 8.8.The combined configuration switches between cautious contact and rapid transfer phases based on tactile feedback, reducing failure-induced delays while maintaining transfer speed.Source: C. Ablation study · pp. 7–8

VI. CONCLUSIONS

• The system was validated on rubber-pipette grasping and liquid transfer, syringe insertion, and water-bottle and paper-cup manipulation.The conclusion reports improved manipulation stability and demonstration-collection efficiency.Source: §VI · p. 8

• Trials used dual-UR arms with dexterous hands or parallel grippers, a Pico4U XR headset, and continuous grasp-and-place manipulation of paper cups and water bottles.Each configuration was evaluated for 15 minutes, with successful manipulations recorded to measure data-collection efficiency.Source: APPENDIX · p. 8