OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation

• Task demonstrations are a fundamental requirement for learning-based robotics across manipulation, locomotion, and aerial robotics.Source: §I · p. 2
• Dexterous teleoperation is challenging because high-dimensional actions often require expensive gloves, extensive calibration, or systems vulnerable to monocular occlusions.Source: §I · p. 2
• OPEN TEACH is an open-source $500 framework using VR hand gestures at 90Hz to support bimanual and multi-finger manipulation.Source: §I · p. 2
• 38 tasks span single-arm, bimanual, multi-fingered, and mobile manipulation in simulation and the real world.Source: §I · p. 2
• Policies trained on OPEN TEACH data achieve an average success rate of 86% across 10 tasks, supporting the utility of the collected demonstrations.Source: §I · p. 2
• OPEN TEACH is presented as a plug-and-play, fully open-source framework for demonstrations across robot morphologies, with a 15-user study indicating efficacy for experienced and new users.Source: §I · p. 2

Unified Teleoperation Frameworks

• Versatile teleoperation systems aim to adapt across robot variants, environments, and morphologies while keeping data collection setup costs low.Source: B. Unified Teleoperation Frameworks · p. 3
• AnyTeleop advances robot-agnostic teleoperation by supporting multiple hands and arms, and OPEN TEACH builds on this direction.Source: B. Unified Teleoperation Frameworks · p. 3

Behavior Cloning

• Behavior cloning learns a policy from observation-action demonstrations without online environment interaction or a reward function.Source: A. Behavior Cloning · p. 3
• Under unimodal isotropic Gaussian assumptions, maximum-likelihood behavior cloning reduces to minimizing mean squared error between actions and policy predictions.Source: A. Behavior Cloning · p. 3

• The teleoperation module transmits VR hand and wrist keypoints to a robot server, which transforms and retargets them before returning real-time scene feedback.Source: Figure 2 · p. 4
• OPEN TEACH uses a VR headset to create a teaching world, retarget hand and wrist poses to robot joints, and control the robot.Source: §IV · p. 4
• The Quest 3 provides 2064 × 2208 resolution, a native 90Hz refresh rate, full-color passthrough, and a $499 base price.Source: A. Placing an Operator in a Virtual… · p. 4
• The integrated hand-pose estimator streams hand poses at 90Hz and avoids specialized calibration routines required by prior multi-camera frameworks.Source: B. Pose Estimation with VR Headsets · p. 4

• VR hand and wrist joint positions are combined into wrappers that map human poses to robot poses for different robot morphologies.Source: C. Human to Robot Pose Retargeting · p. 4

Robot Arm

• For robot arms, wrist position controls end-effector position while changes in a palm-based 3D coordinate system control orientation.Source: C. Human to Robot Pose Retargeting · p. 4

Robot Hand

• For robot hands, corresponding finger joint angles are commanded directly, while thumb motion uses spatial mapping and inverse kinematics.Source: C. Human to Robot Pose Retargeting · p. 4

• High-resolution images stream to the VR application at 90Hz, while camera-view widgets support reactive control and fine-grained movements.Source: Figure 3 · p. 5

Two-fingered gripper

• A pinky-thumb pinch toggles a two-fingered gripper between opening and closing states.Source: C. Human to Robot Pose Retargeting · p. 5

Mobile manipulator

• Mobile-manipulator control maps wrist depth, height, and lateral movements to arm extension, robot height, and wheel-driven sideways motion.Source: C. Human to Robot Pose Retargeting · p. 5

Control implementation

• 300Hz torque control with gravity compensation supports low-error Allegro Hand teleoperation.The Allegro Hand is controlled asynchronously through ROS, while a PD controller outputs desired torques.Source: D. Robot Control · p. 5
• OPEN TEACH evaluates xArm, Franka Emika Panda, Kinova Jaco, Allegro, and Hello Stretch controllers at platform-specific frequencies.The stated frequencies include 90Hz for xArm, 60Hz for Franka and Jaco, 60Hz for Allegro, and 5Hz for Hello Stretch.Source: D. Robot Control · pp. 5–6
• High-frequency teleoperation lets teachers observe robot motion in real time and immediately correct execution errors.The framework uses different control frequencies across robot arms and hands.Source: D. Robot Control · p. 6

Evaluation questions

• The experiments assess versatility, policy success, long-horizon task performance, and intuitiveness for new users.These questions structure the evaluation of OPEN TEACH across robotics setups and users.Source: §V · p. 6

Robot configurations

• Six setups—four real-world and two simulated—test OPEN TEACH across varied robot arms and end effectors.Each setup combines a robot arm variant with either an Allegro Hand or a two-fingered gripper.Source: A. Experimental Setup · p. 6
• The real-world setups are Franka-Allegro, Kinova-Allegro, Bimanual xArm7, and Hello Stretch.They cover single-arm dexterous manipulation, bimanual manipulation, and mobile manipulation.Source: A. Experimental Setup · p. 6, C. How versatile is OPEN TEACH across… · p. 6

Data collection

• Data collection uses one RealSense camera for Franka-Allegro and Kinova-Allegro, five cameras for Bimanual, and an iPhone on Stretch’s wrist.The simulated environments are Allegro Sim and LIBERO Sim.Source: A. Experimental Setup · p. 6

Policy learning

• Collected demonstrations train visual and visuotactile policies using behavior cloning and inverse reinforcement learning.TAVI, FISH, and transformer-based behavior cloning are used for the evaluated setups.Source: A. Experimental Setup · p. 6, B. Imitation Learning with OPEN TEACH… · p. 6

Framework design

• OPEN TEACH is designed as a plug-and-play framework for teleoperating varied robot setups without additional hardware setup cost.The study applies it to four real-world combinations of arms and end effectors.Source: C. How versatile is OPEN TEACH across… · p. 6

Policy performance

• 86% average success rate is achieved across tasks and robot morphologies using policies trained on OPEN TEACH data.This overall result combines imitation-learning policies across real-world and simulated setups.Source: D. How successful are policies trained… · p. 6
• 93% average success rate is achieved by behavior-cloning policies on LIBERO Sim.The result is reported as evidence of high-quality collected action data.Source: D. How successful are policies trained… · p. 6
• 82% average success rate is achieved by TAVI and FISH policies learned within 20 minutes on Franka-Allegro and Allegro Sim.These policies use visuotactile or visual data collected through OPEN TEACH.Source: D. How successful are policies trained… · p. 6

Long-horizon tasks

• OPEN TEACH demonstrates intricate, long-horizon tasks across Bimanual, Franka-Allegro, Kinova-Allegro, and Stretch setups.Examples include USB insertion, cucumber slicing, basket lifting, laptop opening, and sliding a tea sachet.Source: E. Can OPEN TEACH be used for… · pp. 6, 8, Figure 4 · p. 7

User study

• 15 new users receive 10 minutes of practice before completing five trials for each of three tasks.The study uses the Franka-Allegro setup to assess both robot-arm and robot-hand teleoperation.Source: F. How intuitive is the system for new… · p. 8

Baseline comparison

• OPEN TEACH achieves higher success rates and significantly shorter median completion times than Holo-Dex and AnyTeleop on cube flipping and pinch grasping.The comparison is limited to tasks without arm manipulation because baseline arm-retargeting code was unavailable.Source: F. How intuitive is the system for new… · p. 8

Expert comparison

• New users reach 76% of expert success rates on average and take 2.25× longer to complete tasks.Some new users match or exceed experts on individual tasks.Source: F. How intuitive is the system for new… · p. 8

Learning curve

• The authors report that prolonged training substantially improves new-user performance and may yield further gains with continued practice.The study also identifies variation in ability across individuals.Source: F. How intuitive is the system for new… · p. 8

Limitations

• Occluded fingers and misdetected finger positions can reduce hand-tracking quality and hinder precise gestures such as gripper closing.The limitation concerns reliance on the VR headset’s built-in hand-pose detector.Source: §VI · pp. 8–9

System architecture

• The teleoperation pipeline comprises a detector, keypoint transformer, operator, controller, and visualizer communicating through ZeroMQ.The detector receives Quest 3 hand keypoints, while later components transform poses, compute actions, execute them, and provide visualization.Source: 1) Structure of the framework · p. 12

Data collection

• The data recorder logs RGB-D images, tactile readings, timestamps, joint states, and gripper states, then aligns them by timestamp.Source: 1) Structure of the framework · p. 12

Thumb retargeting

• OPEN TEACH retargets most robot-hand joints directly from teacher hand angles but uses a specialized method for the thumb.The thumb method handles workspace bounds and maps both planar position and height.Source: 2) Thumb Retargeting for Robot Hand · p. 12
• Perspective thumb retargeting uses the nearest in-bound point when the human thumb leaves its workspace, avoiding stagnation and enabling smoother motion.Height is also mapped along the palm between maximum and minimum bounds, supporting more complex tasks than Holo-Dex.Source: 2) Thumb Retargeting for Robot Hand · p. 12

Timing

• Average demonstration-collection times are reported for 16 tasks across five setups spanning three real-world and two simulated environments.The real setups are Franka-Allegro, Kinova-Allegro, and Bimanual; the simulated environments are Allegro sim and LIBERO sim.Source: 1) Demo Collections times · p. 12

Task rollouts

• Figures 6–11 provide labeled rollouts for all tasks performed in real-world and simulated environments.Each rollout includes the task name and a task description.Source: 2) Task Descriptions · p. 12

Study results

• Tables VI and VII report success rates and average completion times for 15 users, each performing roughly three tasks with five trials per task.Table VI covers success rates, while Table VII covers completion times for successful trials and marks zero-success cases as NS.Source: C. User Study · p. 12, Table VI · p. 13, Table VII · p. 14
• User success rates and completion times vary widely, demonstrating substantial differences across users.Source: C. User Study · p. 12

Baseline scope

• Holo-Dex and AnyTeleop were not evaluated on tasks involving arm movements because their arm-retargeting code was unavailable.Source: C. User Study · p. 12