Source-linked AI summary
Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning
Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, Xiaolong Wang
TL;DR
Bimanual dexterous teleoperation remains difficult because operators must coordinate complex, high-DoF motions while maintaining responsive and safe control. Bunny-VisionPro uses Vision Pro tracking, modular motion control and retargeting, and low-cost haptic feedback; it improves benchmark performance and imitation-learning generalization while supporting long-horizon tasks. Its hand tracking and light-contact haptics remain limited in specific conditions.
Problem
Bimanual high-DoF teleoperation requires coordinated, responsive manipulation while mitigating collision and singularity risks.
Method
Bunny-VisionPro combines Vision Pro hand tracking with modular hand retargeting, arm control, and human haptic feedback for real-time bimanual teleoperation.
Results
Bunny-VisionPro achieved 11% higher success rates and reduced task completion time by 45% versus prior systems on the Telekinesis benchmark, while improving imitation-learning generalization.
Takeaways & Limitations
The system enables high-quality demonstrations for dexterous, bimanual, and multi-stage long-horizon imitation-learning tasks.
Takeaways & Limitations
Vision Pro hand tracking becomes inaccurate during finger self-occlusion, and subtle haptic changes are difficult to sense during light contact.
Abstract
from arXiv · showhide
Teleoperation is a crucial tool for collecting human demonstrations, but controlling robots with bimanual dexterous hands remains a challenge. Existing teleoperation systems struggle to handle the complexity of coordinating two hands for intricate manipulations. We introduce Bunny-VisionPro, a real-time bimanual dexterous teleoperation system that leverages a VR headset. Unlike previous vision-based teleoperation systems, we design novel low-cost devices to provide haptic feedback to the operator, enhancing immersion. Our system prioritizes safety by incorporating collision and singularity avoidance while maintaining real-time performance through innovative designs. Bunny-VisionPro outperforms prior systems on a standard task suite, achieving higher success rates and reduced task completion times. Moreover, the high-quality teleoperation demonstrations improve downstream imitation learning performance, leading to better generalizability. Notably, Bunny-VisionPro enables imitation learning with challenging multi-stage, long-horizon dexterous manipulation tasks, which have rarely been addressed in previous work. Our system's ability to handle bimanual manipulations while prioritizing safety and real-time performance makes it a powerful tool for advancing dexterous manipulation and imitation learning.
1 Introduction
Bunny-VisionPro addresses the coordination, responsiveness, and safety challenges of bimanual high-DoF teleoperation with a VR-based system that also supports high-quality demonstration collection. It improves benchmark performance and downstream imitation-learning generalization over prior systems.
- Challenges: Bimanual high-DoF teleoperation requires spatiotemporal coordination, responsive control, and collision and singularity mitigation.These challenges arise from coordinating two arms and hands during human-like manipulation.
- System: Bunny-VisionPro uses Vision Pro tracking to translate hand movements into robotic commands while combining arm control, dexterous-hand retargeting, and low-cost haptic feedback.Its arm module handles collision and singularity avoidance, its retargeting module supports loop-joint hands, and ERM actuators cost $1.2 each.
- Demonstrations: The system enables high-quality demonstrations for dexterous, bimanual, and long-horizon imitation-learning tasks.The paper states that its responsive teleoperation supports intricate manipulations and diverse data collection for learning and generalization.
- Evaluation: 11% higher success rates and 45% lower task completion time were achieved on the Telekinesis benchmark versus prior systems.The improvements were especially pronounced for multi-stage, long-horizon tasks.
- Evaluation: Imitation-learning policies trained on Bunny-VisionPro demonstrations showed 20% better generalization to novel poses and unseen objects than policies using previous-work data.This result links the collected demonstrations to improved downstream generalization.
2 Related Work
Prior teleoperation approaches trade off robot specificity, operator-managed safety, dexterous-hand complexity, or computational cost. Vision-based and VR systems broaden access, while imitation learning depends critically on labor-intensive demonstration collection.
- Teleoperation with Gripper: Joint-space leader-follower teleoperation can support bimanual manipulation but requires kinematic equivalence between leader and follower robots.It also leaves collision and singularity management to the human operator.
- Dexterous Teleoperation: Glove-based dexterous teleoperation tracks finger movements but can be costly and require specific hand sizes.The passage uses the MANUS glove as an example.
- System Context: Bunny-VisionPro combines visual and haptic feedback with four real-time capabilities to provide an immersive VR teleoperation experience.The operator controls the robot hand with finger poses and the arm with wrist poses.
- Dexterous Teleoperation: AnyTeleop enables dexterous hand-arm teleoperation but requires sophisticated GPU processing and is primarily designed for a single arm.A concurrent reduced-DoF approach avoids retargeting latency through grip buttons but sacrifices dexterous finger gaiting.
- Imitation Learning from Demonstration: Imitation learning uses expert demonstrations to generate robot control commands, but collecting demonstrations is labor-intensive and critical for effective learning.Prior work increasingly incorporates tactile data and addresses bimanual systems.
3 Teleoperation System
Bunny-VisionPro combines hand retargeting, arm control, and haptic feedback into a modular VR teleoperation system for high-DoF bimanual robots. Its design addresses dexterous mapping, loop joints, safety constraints, real-time execution, and tactile feedback.
- System Overview: The modular system separates hand motion retargeting, arm motion control, and human haptic feedback into decoupled components.The architecture also allows modules to run in separate computation processes to prevent latency accumulation and support real-time control.
- Hand Motion Retargeting: Hand retargeting minimizes fingertip-vector discrepancies while enforcing temporal smoothness, joint limits, hand-size scaling, and SQP-based optimization.The objective matches scaled human fingertip vectors to robot forward-kinematics vectors and penalizes large joint changes between consecutive frames.
- Hand Motion Retargeting: Loop-joint handling reduces optimization from n total joints to k active joints by computing passive joints forward and backpropagating their gradients.This avoids explicit equality constraints for passive joints and makes the optimization more tractable.
- Hand Motion Retargeting: 10.2x speedup is achieved for solving the Ability Robot Hand’s four-bar linkage structure with the reduced-dimension loop-joint method.The method avoids SQP affine approximations of nonlinear loop-joint constraints.
- Robot Arm Motion Control: Arm control optimizes wrist-to-end-effector position and orientation while incorporating singularity and collision avoidance objectives.Singularity penalties activate near low smallest singular values, while collision costs are computed using collections of spheres representing robot links.
- Human Haptic Feedback Device: The haptic-feedback device calibrates and filters fingertip FSR signals before converting them into ERM actuator vibrations for tactile feedback.The system uses zero-drift calibration, low-pass filtering, PWM-based vibration control, and BJT circuitry for stable individual actuator operation.
4 System Evaluation
The evaluation measures Bunny-VisionPro’s real-time performance and teleoperation quality across benchmark and custom manipulation tasks. It compares task success, efficiency, stability, and arm-motion changes against prior systems.
- Profiling: Bunny-VisionPro runs motion control in real time at > 60 Hz while handling singularity and collision avoidance.The profiling evaluates system modules under specified conditions.
- Evaluation Tasks: The evaluation covers ten single-arm benchmark tasks plus six custom tasks, including short- and long-horizon bimanual manipulation.The custom tasks include grasping, cleaning, pouring, wiping, sweeping, and coffee preparation.
- Benchmark Results: Bunny-VisionPro matches or surpasses baselines in 9 out of 10 Telekinesis tasks.The scissor pickup task is the exception because limited hand DoF makes finger insertion into the handles difficult.
- Custom-Task Results: On custom tasks, Bunny-VisionPro achieves 11% higher success rate and 45% lower task completion time than AnyTeleop+.It also reduces episode lengths by 19%, while AnyTeleop+ produces a 43% increase in arm joint position changes.
5 Imitation Learning
The imitation-learning evaluation tests whether demonstrations collected by Bunny-VisionPro support policy performance and generalization. It also examines long-horizon learning and the contribution of tactile data.
- Main Results: Bunny-VisionPro demonstrations produce an average 22% higher success rate than AnyTeleop+ across three tasks and three policies.The evaluated policies are ACT, Diffusion Policy, and DP3.
- Generalization: Bunny-VisionPro exceeds AnyTeleop+ by 14% in spatial generalization and 26% on unseen objects.Spatial generalization uses random object poses, while unseen-object evaluation varies shape, size, and color.
- Long-horizon Tasks: Long-horizon policies achieve 73% sub-task success and 38% entire-task success with only 30 demonstrations.These results concern complicated multi-stage tasks.
- Tactile Data: Tactile data does not necessarily improve outcomes in the evaluated tasks.The discussion attributes this partly to vision sufficing for object identification and localization; impulse data slightly outperforms force data, possibly because of force-sensor zero-point drift.
6 Conclusion and Limitation
Bunny-VisionPro combines VR-based bimanual dexterous control, collision and singularity handling, and haptic feedback for safe, responsive operation. The supplied passages also identify supplementary implementation sections and limitations involving hand-tracking occlusion and weak-contact haptic sensing.
- Limitations: Vision Pro hand tracking becomes inaccurate during finger self-occlusion, producing jerky control commands.The paper suggests fusing wearable-device data with Vision Pro as a possible mitigation.
- Limitations: Subtle haptic changes are difficult to sense when the robot lightly touches an object.The paper proposes larger-contact-area, more-sensitive actuators such as piezoelectric actuators for future work.
- Conclusion: Bunny-VisionPro uses Vision Pro hand poses to generate control commands while addressing retargeting, collision avoidance, and singularity for accurate and responsive operation.The system also includes a haptic feedback device intended to improve control accuracy and immersion.
- Conclusion: The system provides tactile sensations to the operator through a developed haptic feedback device.This feedback is presented as supporting control accuracy and a more immersive experience.
- Supplementary materials: Supplementary materials cover initialization modes, robot-link sphere modeling, task design, haptic feedback, tactile-data processing, and imitation-learning implementation.These topics are listed as Sections A–F of the supplementary materials.
B Sphere Modeling in Collision Checking
The system models robot-arm links with multiple spheres of varying sizes to support fast, differentiable collision checking and avoidance during real-time teleoperation.
- Sphere modeling: Each robot-arm link is represented by multiple spheres of varying sizes for collision checking and avoidance.The sphere model is used to support fast and differentiable computation during teleoperation.
- Collision checking: Collisions between spheres on the same link or between a parent link and its direct child are ignored to accelerate detection.This simplification enables efficient computation of the gradient used for motion-control optimization.
C Task Suits Design
The supplementary task design describes pipelines for teleoperation and imitation learning, alongside haptic user-study tasks. Demonstrations use synchronized visual, point-cloud, and proprioceptive streams with different collection counts for short- and long-horizon tasks.
- Teleoperation and imitation learning: The teleoperation and imitation-learning suite includes three short-horizon tasks and three long-horizon tasks.The short-horizon tasks are grasping toy, uncovering and pouring, and cleaning pan; the long-horizon tasks are wiping glass, sweeping floor, and preparing coffee.
- Haptic user study: The haptic user study uses ball moving and toy handover tasks with precise control and deformable-object manipulation demands.Ball moving involves minimally deformable balls, whereas toy handover involves soft, deformable toys.
- Demonstration collection: Camera images and processed point clouds are collected at approximately 30 Hz, while proprioceptive data is recorded above 100 Hz.The streams are timestamp-aligned by interpolation and downsampled to 10 Hz for imitation training.
- Demonstration collection: The dataset contains 50 demonstrations per short-horizon task and 30 demonstrations per long-horizon task.These counts describe the supplementary demonstration collection procedure.
D Haptic Feedback Implementation
Haptic feedback is implemented through server-side tactile processing and Arduino-side motor control, converting robot-hand tactile signals into PWM commands for wearable vibration motors. The supplementary materials also identify the associated task pipelines and algorithms.
- Implementation: The haptic implementation has server-side signal processing and Arduino-side motor-control components.The server prepares tactile signals, while the Arduino applies received commands to the motors.
- Server-side signal processing: Server-side processing calibrates, filters, and normalizes tactile readings before converting them into PWM values sent to an ELEGOO UNO board.This processing prepares tactile data for motor activation and haptic feedback control.
- Arduino-side motor control: The Arduino parses serially received PWM values and maps them directly to board pins controlling ERM motors.This allows the motor intensity to be adjusted according to server-side input.
- Task pipelines: The task pipelines include short- and long-horizon teleoperation and imitation-learning tasks, plus ball moving and toy handover for the haptic study.The task sets are described in Figures 6 and 7.
- Server-side algorithm: Algorithm 1 defines tactile acquisition, calibration and smoothing, threshold setting, normalization, clipping to [0, 255], and serial transmission.The described loop repeatedly converts tactile sensor values into PWM values.
- Arduino-side algorithm: Algorithm 2 implements Arduino-side motor control, with motor pins configured as outputs and PWM values written after serial parsing.The setup initializes serial communication and the control loop updates each motor pin.
E Tactile Data Processing
Fingertip tactile signals are calibrated and filtered before entering the imitation-learning pipeline, with force impulse optionally derived from temporal force changes.
- Raw FSR pressure data undergo zero-drift calibration and low-pass filtering before use in imitation learning.
- Force impulse can be computed by differentiating the filtered force signal over time.
F Imitation Learning Implementation
The implementation evaluates multiple imitation-learning algorithms and tests whether demonstrations support robustness across spatial changes and unseen objects.
- Learning Algorithms: ACT, Diffusion Policy, and 3D Diffusion Policy are implemented to evaluate demonstration quality.
- Learning Algorithms: Diffusion Policy and 3D Diffusion Policy use multi-view inputs with horizon 8, comprising 2 observation steps and 6 action steps.
- Generalization Evaluation: Generalization experiments assess policy performance at different object locations and with objects absent from the demonstrations.
- Generalization Evaluation: The spatial generalization evaluation covers a large working space.
- Generalization Evaluation: Unseen objects vary in size, color, and shape to provide a comprehensive robustness evaluation.