Source-linked AI summary
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, Xiaolong Wang
TL;DR
Teleoperation needs intuitive, precise perception and control for collecting useful robot-learning demonstrations, but existing systems can suffer from occlusion or limited remote and dexterous operation. Open-TeleVision combines active stereoscopic vision with VR-based arm and hand retargeting across humanoid robots, and demonstrates precise long-horizon manipulation with data usable by imitation learning. Its scope remains limited by missing feedback modalities such as haptics and by the absence of expert-data relabeling.
Problem
Existing teleoperation systems can impose visual occlusion or limit remote control and dexterous-hand operation, constraining intuitive data collection.
Method
Open-TeleVision combines actuated-neck active stereoscopic video with VR-based retargeting of human head, arm, wrist, and hand motions to robots.
Results
The system enables precise, long-horizon manipulation, and its demonstrations can be used by imitation-learning algorithms across multiple tasks and robots.
Takeaways & Limitations
Active stereo perception supports operators’ spatial understanding, while the collected data supports real-world imitation learning for fine-grained manipulation.
Takeaways & Limitations
The system lacks other feedback forms such as haptic feedback and does not currently support expert-data relabeling.
Abstract
from arXiv · showhide
Teleoperation serves as a powerful method for collecting on-robot data essential for robot learning from demonstrations. The intuitiveness and ease of use of the teleoperation system are crucial for ensuring high-quality, diverse, and scalable data. To achieve this, we propose an immersive teleoperation system Open-TeleVision that allows operators to actively perceive the robot's surroundings in a stereoscopic manner. Additionally, the system mirrors the operator's arm and hand movements on the robot, creating an immersive experience as if the operator's mind is transmitted to a robot embodiment. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks (Can Sorting, Can Insertion, Folding, and Unloading) for 2 different humanoid robots and deploy them in the real world. The system is open-sourced at: https://robot-tv.github.io/
1 Introduction
Open-TeleVision addresses teleoperation limitations by combining VR-based motion retargeting with active stereoscopic visual feedback. It supports intuitive, remote-capable control and improves perception and imitation learning across precise, long-horizon manipulation tasks.
- Motivation: Existing teleoperation systems provide precise control but can require co-location, hardware coupling, or lack support for dexterous hands.Direct robot viewing also causes visual occlusion during manipulation.
- System: Open-TeleVision retargets operators’ hand, head, and wrist poses to different robots and manipulators using inverse kinematics and motion retargeting.The framework is demonstrated on Unitree H1 with multi-finger hands and Fourier GR-1 with grippers.
- Active visual feedback: An actuated stereo camera follows the operator’s head and streams real-time egocentric 3D observations to the VR device.This active sensing lets operators inspect broader workspaces and focus on regions relevant to detailed interactions.
- Active visual feedback: Stereo perception improves spatial understanding for operators and improves policy performance compared with static captured views.The policy also learns to move the robot head actively with manipulation, reducing pixels processed for control.
- Evaluation: Experiments evaluate four fine-grained tasks across two robots, showing benefits of stereoscopic video for user success rate and completion time.Imitation policies perform faster inference and execute long-horizon tasks such as sequential can insertion.
2 TeleVision System
The system streams VR pose data to a web server for human-to-robot retargeting while returning stereo video from the robot at 60 Hz. Its hardware and control design support humanoids with either dexterous hands or grippers.
- System overview: VR devices stream the operator’s hand, head, and wrist poses in SE(3) to a web server for human-to-robot retargeting.The robot returns stereo video at 480x640 per eye, and the complete loop runs at 60 Hz.
- Hardware: The reference hardware uses Unitree H1 with six-DoF hands and Fourier GR-1 with a one-DoF jaw gripper.H1 uses a custom two-DoF yaw-pitch gimbal, while GR-1 uses its manufacturer-provided three-DoF neck.
- Hardware: The framework is tailored to humanoids but can also apply to setups with two arms and one camera.The stated design scope excludes unused robot degrees of freedom in the evaluated hardware.
- Arm control: Arm control matches human-to-head and robot-end-effector relative positions while aligning robot wrist orientations with the operator’s wrist orientations.This differentiated treatment stabilizes end-effectors as the robot head follows the operator’s head.
- Hand control: Hand retargeting formulates keypoint matching as a real-time optimization with scaling and temporal-consistency penalties.The method uses SLSQP, forward kinematics, and seven vectors for dexterous hands; grippers use one thumb-index vector.
- Data collection: Demonstrations end with a standardized operator gesture that stops recording, and the learned policy retains the gesture to react to task completion.The ending gesture is not cropped from the dataset.
3 Experiments
Experiments evaluate Open-TeleVision across precise, long-horizon manipulation tasks, testing imitation-learning design choices, generalization, and teleoperation usability. Stereo active sensing and the proposed model outperform the tested baselines across most reported settings, while robot morphology and temporal memory constrain some outcomes.
- Experimental Setup: The study evaluates four precision-focused tasks across H1 and GR-1, including randomized manipulation, long horizons, and distinct grasping strategies.Can Insertion uses a two-finger pinch for inserting cans into slots only slightly larger than the can, while Unloading requires extraction, in-hand passing, and placement.
- Imitation Learning Results: The proposed model achieves the highest H1 Can Sorting success rates for both picking and sorting, whereas w/o Stereo Input records 23/50 picking and 26/50 sorting successes.The reported results associate the stereo-input baseline’s failures with missing implicit depth information and frequent experimenter assistance during grasping.
- Imitation Learning Results: On GR-1 Can Sorting, the proposed model consistently outperforms both baselines for picking, but no model reaches satisfactory placing accuracy because the gripper occludes can color.The authors also identify one-second effective memory from chunk size 60 at 60 Hz as a limitation when color must be remembered until dropping.
- Imitation Learning Results: For Folding, both the proposed model and w. ResNet18 achieve 100%, while w/o Stereo Input fails in 2/5 trials during the towel repositioning stage.The authors attribute the single-image failure to insufficient depth information for adjusting hand force relative to the table.
- Imitation Learning Results: The proposed model surpasses both baselines on Can Insertion, reaches 100% in all three Unloading stages, and maintains 100% success across the sampled H1 Can Sorting grid.The baselines fail more often during tube extraction and lack the same performance on precise insertion or spatially randomized grasping.
- Teleoperation and Active Sensing: Active sensing avoids the limited coverage and extra computation of a static wide-angle view, while stereo teleoperation substantially improves success rate and completion time over monocular input.The system is reported as 2x faster for training at the same batch size and able to accommodate 4x data in one batch on an RTX 4090; users also report less discomfort with stereo images.
4 Related Work
The paper situates Open-TeleVision among teleoperation systems for collecting high-fidelity manipulation demonstrations, emphasizing bimanual control, multi-finger dexterity, and remote immersive operation.
- Real-robot demonstrations are effective for manipulation imitation learning because complex contacts create a large Sim2Real gap.
- High-fidelity teleoperation requires intuitive, responsive interfaces, but many existing systems are restricted to grippers or single-hand setups.
- Multi-finger bimanual control supports more robust manipulation across diverse tasks than parallel-jaw grippers alone.The paper gives stably grasping a Pringles Chips tube as an example challenging for a parallel-jaw gripper.
- Unlike prior bimanual setups using direct viewing or fixed RGB cameras, Open-TeleVision combines first-person stereo display with active head rotations.
- The system also supports remote control, including operation across opposite U.S. coasts.
5 Conclusion and Limitations
The paper concludes that stereoscopic video and active perception enable precise, long-horizon manipulation and produce data usable by imitation learning. It also identifies missing haptic feedback and expert-data relabeling as limitations.
- Open-TeleVision combines stereoscopic video streaming with active perception through actuated necks.
- The system enables precise, long-horizon manipulation tasks, and its collected data can be used by imitation learning algorithms.
- User studies show that stereo perception is important for operators’ spatial understanding.
- The system lacks other feedback forms, such as haptic feedback, which is typically dominant in first-person visual occlusion and tactile-intensive tasks.
- Expert-data relabeling is missing and could help increase success rate.
A Discussion on Comparing with Prior Teleoperation Systems
Prior teleoperation research addresses actuation and perception separately, while Open-TeleVision targets the perception trade-off between remote control and stereoscopic depth sensing.
- Actuation: Teleoperation systems use visual tracking, motion capture, or joint copying to translate human commands into robot actions.
- Actuation: Commercial motion-capture gloves can be intuitive but are costly and may not estimate wrist poses.
- Actuation: Joint copying offers high control bandwidth and precision but requires the operator and robot to share a location.
- Perception: Direct viewing preserves human stereoscopic depth sensing but prevents remote operation, whereas RGB streaming enables remote control while losing that depth sensing.
- Perception: Before Open-TeleVision, no described system simultaneously provided remote control and depth sensing; OPEN TEACH still required proximity to the robot.
B Discussion of Visual Occlusion
A controlled GR-1 Can Sorting experiment shows that labeling cans reduces visual occlusion and substantially improves placing success, while spatial stereo information remains important for picking.
- The controlled experiment adds color labels to cans while keeping the other GR-1 Can Sorting settings identical.
- 100% placing accuracy is achieved by the authors’ model with labeled cans, compared with 0.60 using unlabeled cans.
- w. ResNet18 placing success rises from 0.50 to 0.97 with labeled cans.
- w/o Stereo Input placing success rises from 0.63 to 0.93 with labeled cans.
- The results support the claim that successful can-picking requires spatial information from stereo images.
- With labeled cans, the model reaches 100% grasping accuracy in nearly all locations of the 4x4 grid, with each cell measuring 3 cm.
C Dexterous Hand
The H1 setup uses anthropomorphic Inspire Hands with five fingers and 12 DoFs, including six actuated DoFs for dexterous manipulation.
- The setup is evaluated with a can-labeling and generalization figure that measures successful pickings across locations.The heatmap reports five trials at each location.
- Each Inspire Hand has five fingers and 12 DoFs.Six DoFs are actuated: two on the thumb and one on each remaining finger.
- The hand’s non-thumb fingers are actuated at their metacarpophalangeal joints.Their proximal interphalangeal joints are not described as independently actuated in the supplied passage.
D Teleoperation Interface
The teleoperation interface includes the Inspire Hand hardware and a web-based interface that supports access from VR devices, laptops, tablets, and phones.
- The web-based cross-platform interface is accessible from VR devices, laptops, tablets, and phones.The supplied passage identifies this as the interface shown in Figure 12.
E.1 Experimental Details
The experiments collect task demonstrations across multiple devices and use different demonstration counts for repetitive Can Sorting versus the other tasks.
- 20 human demonstrations are used for every task except Can Sorting.Both H1 Can Sorting and GR-1 Can Sorting use 10 demonstrations.
- The web-based system supports access from VR devices, laptops, tablets, and phones.This broadens the available interface devices for the experimental setup.
- Table 6 summarizes the collected demonstration data for each task.
- 10 Can Sorting demonstrations provide 100 individual sorting rollouts, or 60 for GR-1 Can Sorting.Each episode contains 10 individual sortings for H1 and six for GR-1.
E.2 Hyperparameters
The ACT training setup uses mostly shared hyperparameters, with task-specific chunk sizes and temporal weighting to balance memory, responsiveness, and stability.
- Most ACT hyperparameters are shared across baselines and tasks, with exceptions including chunk size and temporal weighting.The complete hyperparameter details are listed in Table 7.
- A chunk size of 60 is used for all tasks except Can Insertion, which uses 100.At a 60 Hz action frequency, chunk size 60 corresponds to approximately one second of memory.
- Temporal weighting uses w_i = exp(−m ∗i), assigning different weights to actions at different timesteps.The parameter m controls the weighting behavior, with w_0 representing the oldest action’s weight.
- A temporal weight m of 0.01 balances responsiveness and stability for most tasks.Lower m values emphasize recent actions more strongly, making the model more reactive but less steady.
- Unloading uses m = 0.05 for greater in-hand-passing stability, while Can Sorting uses m = 0.005 for quicker movements.