Source-linked AI summary
TWIST: Teleoperated Whole-Body Imitation System
Yanjie Ze, Zixuan Chen, João Pedro Araújo, Zi-ang Cao, Xue Bin Peng, Jiajun Wu, C. Karen Liu
TL;DR
Humanoid teleoperation needs coordinated whole-body behavior, but existing systems commonly remain limited to isolated skills and lack robust real-time tracking. TWIST retargets human MoCap motions and trains a unified RL-plus-BC controller with future-motion guidance and real-world motion data. On real humanoids, it performs diverse whole-body manipulation, legged manipulation, locomotion, and expressive movement with one neural network controller.
Problem
Existing humanoid teleoperation systems often provide isolated locomotion or manipulation rather than coordinated whole-body behavior, while real-time tracking targets and robust controllers remain challenging.
Method
TWIST retargets human MoCap movements into humanoid targets and trains a unified whole-body controller with reinforcement learning, behavior cloning, future-motion guidance, and online MoCap data.
Results
TWIST enables real-world humanoid robots to perform diverse coordinated skills spanning whole-body manipulation, legged manipulation, locomotion, and expressive movement with a single neural network controller.
Takeaways & Limitations
The system fully leverages humanoid whole-body joints and expands demonstrated teleoperation capabilities across versatile coordinated skills.
Takeaways & Limitations
Without egocentric robotic vision and tactile feedback, visual occlusion and grasp-success uncertainty make teleoperation less effective and less reliable.
Abstract
from arXiv · showhide
Teleoperating humanoid robots in a whole-body manner marks a fundamental step toward developing general-purpose robotic intelligence, with human motion providing an ideal interface for controlling all degrees of freedom. Yet, most current humanoid teleoperation systems fall short of enabling coordinated whole-body behavior, typically limiting themselves to isolated locomotion or manipulation tasks. We present the Teleoperated Whole-Body Imitation System (TWIST), a system for humanoid teleoperation through whole-body motion imitation. We first generate reference motion clips by retargeting human motion capture data to the humanoid robot. We then develop a robust, adaptive, and responsive whole-body controller using a combination of reinforcement learning and behavior cloning (RL+BC). Through systematic analysis, we demonstrate how incorporating privileged future motion frames and real-world motion capture (MoCap) data improves tracking accuracy. TWIST enables real-world humanoid robots to achieve unprecedented, versatile, and coordinated whole-body motor skills--spanning whole-body manipulation, legged manipulation, locomotion, and expressive movement--using a single unified neural network controller. Our project website: https://humanoid-teleop.github.io
1 Introduction
TWIST addresses the challenge of coordinated whole-body humanoid teleoperation by combining real-time human-motion retargeting with a unified learned controller. The system integrates future-motion guidance, online MoCap data, robust retargeting, and force-oriented training to enable diverse real-world skills.
- Motivation: Coordinated whole-body actions are important for humanoid robots intended to live and work alongside humans in household environments.Examples include opening doors while carrying objects, reaching under beds, and balancing while kicking.
- Motivation: Existing teleoperation systems remain limited because modular controllers separate teleoperation and balance, while learning-based systems lack accurate real-time targets and robust tracking.These limitations hinder skills such as crouching to lift a box from the ground.
- System: TWIST retargets arbitrary MoCap movements into humanoid joint positions and root velocities, then trains one balanced tracking policy with RL and human motion data.Its pipeline includes motion-data curation, simulated controller training, and real-robot teleoperation with MoCap devices.
- Controller Training: Future-motion frames guide a teacher policy before distilling behavior to a single-frame student, reducing conservative and hesitant teleoperation behavior.The student preserves low-latency observation while benefiting from the teacher’s smoother behavior.
- Data: A small online MoCap dataset of 150 clips combined with 15K offline clips improves stability on unseen motions and in real-world teleoperation.The online data addresses distribution shift between smooth offline motions and unstable real-time retargeting.
- Results: TWIST enables a 29-DoF Unitree G1 to perform diverse real-world skills using a single neural network controller.Demonstrated behaviors span whole-body manipulation, legged manipulation, locomotion, and expressive motion.
2 Related Works
Prior humanoid-control research has advanced task-specific learning-based skills, while teleoperation research has explored several input modalities. However, existing systems generally provide limited coordinated whole-body behavior for complex household tasks.
- Learning-Based Whole-Body Control for Humanoid Robots: Learning-based humanoid controllers have enabled skills including walking, jumping, parkour, dancing, hopping, and fall recovery.Most prior methods focus on a single specific task, limiting their generality.
- Teleoperation Systems for Humanoid Robots: Humanoid teleoperation systems use sparse VR keypoints, exoskeletons, motion-capture suits, and camera-based pose estimation.Despite these varied modalities, most systems show limited coordinated whole-body skills for general household tasks.
3 Method
TWIST builds a unified whole-body teleoperation controller by retargeting human motion and training with simulation, real-time MoCap data, future-motion supervision, and RL+BC.
- TWIST frames whole-body humanoid teleoperation as real-time motion tracking from retargeted human motions to target robot joint positions.
- Motion Data Curation: The training pipeline combines over 15,000 public motion clips with 150 in-house MoCap clips that reflect noisy real-world teleoperation conditions.The public datasets contain around 42 hours of motion, while the in-house dataset contains around 0.5 hours.
- Motion Data Curation: Offline and online retargeting use different optimization strategies, with joint optimization of 3D positions and orientations mitigating the offline-to-online quality gap.Offline retargeting prioritizes quality through iterative optimization, whereas online retargeting prioritizes fast inference.
- Controller Training: The teacher policy observes 2 seconds of future reference frames, enabling anticipation and smoother locomotion while tracking local-frame joint positions and root velocities.Its reward emphasizes tracking accuracy and penalizes jitter and foot slippage.
- Controller Training: The deployable student policy uses proprioception and immediate targets, and is trained with RL plus KL-based behavior cloning from the teacher.The imitation weight λ is gradually reduced during training.
- Deployment and Evaluation: RL+BC significantly outperforms pure BC and pure RL, producing smoother motions and better generalization while supporting zero-shot real-robot deployment.The real-time pipeline captures motion at 120Hz and retargets it at 50Hz.
4 Experiments
TWIST demonstrates versatile whole-body teleoperation on humanoid robots and analyzes controller, data, retargeting, reachability, latency, and hardware factors affecting performance.
- Main Results: TWIST enables Unitree G1 to perform whole-body manipulation, legged manipulation, locomotion, and expressive movement with one neural network controller.Demonstrated skills include lifting and carrying objects, kicking, walking, crouching, boxing, and dancing.
- Main Results: The controller successfully tracks coordinated whole-body motions on Booster T1, including arm swinging, deep crouching, and walking.
- Ablation Experiments: RL+BC achieves superior tracking accuracy and motion smoothness compared with pure RL and DAgger-based behavior cloning.Pure RL can produce feet-sliding artifacts, while DAgger occasionally fails to robustly track unseen motions.
- Ablation Experiments: Adding a small set of in-house MoCap sequences substantially reduces tracking errors on unseen motions by exposing the controller to real-world capture and retargeting imperfections.The in-house data contains calibration drift, occlusions, and less-smooth online-retargeted references.
- Ablation Experiments: Training with end-effector perturbations significantly improves stability during stationary poses and contact-rich tasks.Without perturbations, policies can drift and become unstable during stationary poses.
- Ablation Experiments: Jointly optimizing 3D joint positions and orientations produces smoother online-retargeted motions and reduces overall tracking errors.The improvement benefits controllers trained both with and without MoCap data.
- System Analyses: Tracking errors are largest at end-effectors and generally higher in lower-body parts than upper-body parts.Feet and knees are harder to track because of more complex contact dynamics.
- System Analyses: The system has approximately 0.9 seconds of teleoperation delay, with 0.7 seconds from tracking-goal generation and 0.2 seconds from policy inference.Reducing latency is identified as a focus for future improvements.
5 Conclusion
TWIST teleoperates real-world humanoid robots using whole-body MoCap data and a unified system for coordinated skills. The system demonstrates expanded reachability and identifies future work on visuomotor learning.
- TWIST teleoperates real-world humanoid robots using whole-body human motion captured from MoCap devices.
- The system fully leverages the robot’s whole-body joints to execute versatile and coordinated skills previously unachieved in the literature.
- TWIST demonstrates exceptional reachability compared with prior work, expanding humanoid robot capabilities.
- Future work will study visuomotor policies combining TWIST-collected teleoperation data with egocentric human data.
6 Limitations
TWIST’s limitations concern operator feedback, robot hardware endurance, and dependence on a non-portable motion-capture system.
- The system provides no egocentric vision or tactile feedback, making occlusion difficult to manage and grasping less natural and reliable.
- Current humanoid hardware cannot sustain long-term continuous operation, constraining system capability and robustness.
- TWIST relies on a motion-capture system that is not portable and difficult to democratize.