Source-linked AI summary
OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation
Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey, Siddhant Haldar, Soumith Chintala, Lerrel Pinto
TL;DR
Robotics learning requires extensive demonstrations, but existing teleoperation platforms can be costly, proprietary, difficult to use, or limited to particular robot morphologies. OPEN TEACH provides an open-source VR framework that retargets natural hand gestures for real-time control across diverse robots. Across 38 tasks, policies trained on collected data achieve an average success rate of 86% across 10 tasks, while the authors identify hand-pose tracking inaccuracies as a limitation.
Problem
Learning-based robotics requires task demonstrations, while existing teleoperation systems can involve expensive hardware, calibration, or robot-specific designs.
Method
OPEN TEACH uses a VR headset to collect demonstrations by retargeting users’ hand and wrist poses to diverse robot morphologies.
Results
OPEN TEACH was evaluated on 38 tasks, and learned policies achieved an average success rate of 86% across 10 tasks.
Takeaways & Limitations
The collected demonstrations support effective policy learning across diverse robot morphologies and task settings.
Takeaways & Limitations
OPEN TEACH depends on headset hand-pose accuracy, which can degrade under finger occlusion or misinterpret precise gripper-closing pinches.
Abstract
from arXiv · showhide
Open-sourced, user-friendly tools form the bedrock of scientific advancement across disciplines. The widespread adoption of data-driven learning has led to remarkable progress in multi-fingered dexterity, bimanual manipulation, and applications ranging from logistics to home robotics. However, existing data collection platforms are often proprietary, costly, or tailored to specific robotic morphologies. We present OPEN TEACH, a new teleoperation system leveraging VR headsets to immerse users in mixed reality for intuitive robot control. Built on the affordable Meta Quest 3, which costs $500, OPEN TEACH enables real-time control of various robots, including multi-fingered hands and bimanual arms, through an easy-to-use app. Using natural hand gestures and movements, users can manipulate robots at up to 90Hz with smooth visual feedback and interface widgets offering closeup environment views. We demonstrate the versatility of OPEN TEACH across 38 tasks on different robots. A comprehensive user study indicates significant improvement in teleoperation capability over the AnyTeleop framework. Further experiments exhibit that the collected data is compatible with policy learning on 10 dexterous and contact-rich manipulation tasks. Currently supporting Franka, xArm, Jaco, and Allegro platforms, OPEN TEACH is fully open-sourced to promote broader adoption. Videos are available at https://open-teach.github.io/.
I. INTRODUCTION
Learning-based robotics depends on task demonstrations, but existing teleoperation systems can be costly, unintuitive, difficult to calibrate, or robot-specific. OPEN TEACH addresses these constraints with an open-source, $500 VR framework evaluated across diverse robots and tasks.
- Task demonstrations are a fundamental requirement for learning-based robotics across manipulation, locomotion, and aerial robotics.
- Dexterous teleoperation is challenging because high-dimensional actions often require expensive gloves, extensive calibration, or systems vulnerable to monocular occlusions.
- OPEN TEACH is an open-source $500 framework using VR hand gestures at 90Hz to support bimanual and multi-finger manipulation.
- 38 tasks span single-arm, bimanual, multi-fingered, and mobile manipulation in simulation and the real world.
- Policies trained on OPEN TEACH data achieve an average success rate of 86% across 10 tasks, supporting the utility of the collected demonstrations.
- OPEN TEACH is presented as a plug-and-play, fully open-source framework for demonstrations across robot morphologies, with a 15-user study indicating efficacy for experienced and new users.
B. Unified Teleoperation Frameworks
Versatile teleoperation seeks to support diverse robots and environments while enabling abundant data collection with minimal setup costs. OPEN TEACH builds on robot-agnostic efforts, while behavior cloning learns policies by maximizing the likelihood of observed demonstrations.
- Unified Teleoperation Frameworks: Versatile teleoperation systems aim to adapt across robot variants, environments, and morphologies while keeping data collection setup costs low.
- Unified Teleoperation Frameworks: AnyTeleop advances robot-agnostic teleoperation by supporting multiple hands and arms, and OPEN TEACH builds on this direction.
- Behavior Cloning: Behavior cloning learns a policy from observation-action demonstrations without online environment interaction or a reward function.
- Behavior Cloning: Under unimodal isotropic Gaussian assumptions, maximum-likelihood behavior cloning reduces to minimizing mean squared error between actions and policy predictions.
B. Inverse Reinforcement Learning
OPEN TEACH combines VR-based demonstration collection with robot-specific pose retargeting and visual feedback. Its headset-based design uses integrated hand tracking and supports multiple robot types without specialized calibration routines.
- The teleoperation module transmits VR hand and wrist keypoints to a robot server, which transforms and retargets them before returning real-time scene feedback.
- OPEN TEACH uses a VR headset to create a teaching world, retarget hand and wrist poses to robot joints, and control the robot.
- The Quest 3 provides 2064 × 2208 resolution, a native 90Hz refresh rate, full-color passthrough, and a $499 base price.
- The integrated hand-pose estimator streams hand poses at 90Hz and avoids specialized calibration routines required by prior multi-camera frameworks.
C. Human to Robot Pose Retargeting
OPEN TEACH maps VR hand and wrist poses to robot actions through morphology-specific wrappers. Its interface supports arm, hand, gripper, and mobile-manipulator control while streaming high-resolution visual feedback at 90Hz.
- VR hand and wrist joint positions are combined into wrappers that map human poses to robot poses for different robot morphologies.
- Robot Arm: For robot arms, wrist position controls end-effector position while changes in a palm-based 3D coordinate system control orientation.
- Robot Hand: For robot hands, corresponding finger joint angles are commanded directly, while thumb motion uses spatial mapping and inverse kinematics.
- High-resolution images stream to the VR application at 90Hz, while camera-view widgets support reactive control and fine-grained movements.
- Two-fingered gripper: A pinky-thumb pinch toggles a two-fingered gripper between opening and closing states.
- Mobile manipulator: Mobile-manipulator control maps wrist depth, height, and lateral movements to arm extension, robot height, and wheel-driven sideways motion.
D. Robot Control
OPEN TEACH uses high-frequency robot control to support real-time correction across multiple robot platforms. The evaluation examines versatility, policy success, long-horizon tasks, and user intuitiveness.
- Control implementation: 300Hz torque control with gravity compensation supports low-error Allegro Hand teleoperation.The Allegro Hand is controlled asynchronously through ROS, while a PD controller outputs desired torques.
- Control implementation: OPEN TEACH evaluates xArm, Franka Emika Panda, Kinova Jaco, Allegro, and Hello Stretch controllers at platform-specific frequencies.The stated frequencies include 90Hz for xArm, 60Hz for Franka and Jaco, 60Hz for Allegro, and 5Hz for Hello Stretch.
- Control implementation: High-frequency teleoperation lets teachers observe robot motion in real time and immediately correct execution errors.The framework uses different control frequencies across robot arms and hands.
- Evaluation questions: The experiments assess versatility, policy success, long-horizon task performance, and intuitiveness for new users.These questions structure the evaluation of OPEN TEACH across robotics setups and users.
A. Experimental Setup
The experimental setup spans six robot configurations across real-world and simulated environments. Collected demonstrations support visual, visuotactile, behavior-cloning, and inverse-reinforcement-learning policies.
- Robot configurations: Six setups—four real-world and two simulated—test OPEN TEACH across varied robot arms and end effectors.Each setup combines a robot arm variant with either an Allegro Hand or a two-fingered gripper.
- Robot configurations: The real-world setups are Franka-Allegro, Kinova-Allegro, Bimanual xArm7, and Hello Stretch.They cover single-arm dexterous manipulation, bimanual manipulation, and mobile manipulation.
- Data collection: Data collection uses one RealSense camera for Franka-Allegro and Kinova-Allegro, five cameras for Bimanual, and an iPhone on Stretch’s wrist.The simulated environments are Allegro Sim and LIBERO Sim.
- Policy learning: Collected demonstrations train visual and visuotactile policies using behavior cloning and inverse reinforcement learning.TAVI, FISH, and transformer-based behavior cloning are used for the evaluated setups.
- Framework design: OPEN TEACH is designed as a plug-and-play framework for teleoperating varied robot setups without additional hardware setup cost.The study applies it to four real-world combinations of arms and end effectors.
D. How successful are policies trained with OPEN TEACH?
Policies trained on OPEN TEACH demonstrations achieve strong performance across real-world and simulated setups. The experiments also show the system supports intricate, extended manipulation tasks across multiple robot configurations.
- Policy performance: 86% average success rate is achieved across tasks and robot morphologies using policies trained on OPEN TEACH data.This overall result combines imitation-learning policies across real-world and simulated setups.
- Policy performance: 93% average success rate is achieved by behavior-cloning policies on LIBERO Sim.The result is reported as evidence of high-quality collected action data.
- Policy performance: 82% average success rate is achieved by TAVI and FISH policies learned within 20 minutes on Franka-Allegro and Allegro Sim.These policies use visuotactile or visual data collected through OPEN TEACH.
- Long-horizon tasks: OPEN TEACH demonstrates intricate, long-horizon tasks across Bimanual, Franka-Allegro, Kinova-Allegro, and Stretch setups.Examples include USB insertion, cucumber slicing, basket lifting, laptop opening, and sliding a tea sachet.
F. How intuitive is the system for new users?
A 15-user study evaluates OPEN TEACH after brief practice and compares it with established teleoperation baselines. New users perform well relative to experts, while the system remains sensitive to hand-tracking limitations.
- User study: 15 new users receive 10 minutes of practice before completing five trials for each of three tasks.The study uses the Franka-Allegro setup to assess both robot-arm and robot-hand teleoperation.
- Baseline comparison: OPEN TEACH achieves higher success rates and significantly shorter median completion times than Holo-Dex and AnyTeleop on cube flipping and pinch grasping.The comparison is limited to tasks without arm manipulation because baseline arm-retargeting code was unavailable.
- Expert comparison: New users reach 76% of expert success rates on average and take 2.25× longer to complete tasks.Some new users match or exceed experts on individual tasks.
- Learning curve: The authors report that prolonged training substantially improves new-user performance and may yield further gains with continued practice.The study also identifies variation in ability across individuals.
- Limitations: Occluded fingers and misdetected finger positions can reduce hand-tracking quality and hinder precise gestures such as gripper closing.The limitation concerns reliance on the VR headset’s built-in hand-pose detector.
APPENDIX
OPEN TEACH separates teleoperation from data collection and maps VR hand poses to robot actions through modular components. Its hand wrapper adds specialized thumb retargeting for smoother, more complex manipulation.
- System architecture: The teleoperation pipeline comprises a detector, keypoint transformer, operator, controller, and visualizer communicating through ZeroMQ.The detector receives Quest 3 hand keypoints, while later components transform poses, compute actions, execute them, and provide visualization.
- Data collection: The data recorder logs RGB-D images, tactile readings, timestamps, joint states, and gripper states, then aligns them by timestamp.
- Thumb retargeting: OPEN TEACH retargets most robot-hand joints directly from teacher hand angles but uses a specialized method for the thumb.The thumb method handles workspace bounds and maps both planar position and height.
- Thumb retargeting: Perspective thumb retargeting uses the nearest in-bound point when the human thumb leaves its workspace, avoiding stagnation and enabling smoother motion.Height is also mapped along the palm between maximum and minimum bounds, supporting more complex tasks than Holo-Dex.
B. Task Details
The appendix reports demonstration-collection timing across multiple real and simulated setups and documents rollouts for the evaluated tasks.
- Timing: Average demonstration-collection times are reported for 16 tasks across five setups spanning three real-world and two simulated environments.The real setups are Franka-Allegro, Kinova-Allegro, and Bimanual; the simulated environments are Allegro sim and LIBERO sim.
- Task rollouts: Figures 6–11 provide labeled rollouts for all tasks performed in real-world and simulated environments.Each rollout includes the task name and a task description.
C. User Study
The user-study appendix reports per-user success rates and completion times across tasks, while noting differences in user performance and baseline coverage. Tables VI and VII summarize these outcomes for 15 users.
- Study results: Tables VI and VII report success rates and average completion times for 15 users, each performing roughly three tasks with five trials per task.Table VI covers success rates, while Table VII covers completion times for successful trials and marks zero-success cases as NS.
- Study results: User success rates and completion times vary widely, demonstrating substantial differences across users.
- Baseline scope: Holo-Dex and AnyTeleop were not evaluated on tasks involving arm movements because their arm-retargeting code was unavailable.