TWIST: Teleoperated Whole-Body Imitation System

Motivation

• Coordinated whole-body actions are important for humanoid robots intended to live and work alongside humans in household environments.Examples include opening doors while carrying objects, reaching under beds, and balancing while kicking.Source: §1 · p. 1
• Existing teleoperation systems remain limited because modular controllers separate teleoperation and balance, while learning-based systems lack accurate real-time targets and robust tracking.These limitations hinder skills such as crouching to lift a box from the ground.Source: §1 · p. 3

System

• TWIST retargets arbitrary MoCap movements into humanoid joint positions and root velocities, then trains one balanced tracking policy with RL and human motion data.Its pipeline includes motion-data curation, simulated controller training, and real-robot teleoperation with MoCap devices.Source: §1 · p. 3, Figure 3 · p. 4

Controller Training

• Future-motion frames guide a teacher policy before distilling behavior to a single-frame student, reducing conservative and hesitant teleoperation behavior.The student preserves low-latency observation while benefiting from the teacher’s smoother behavior.Source: §1 · p. 3

Data

• A small online MoCap dataset of 150 clips combined with 15K offline clips improves stability on unseen motions and in real-world teleoperation.The online data addresses distribution shift between smooth offline motions and unstable real-time retargeting.Source: §1 · p. 3

Results

• TWIST enables a 29-DoF Unitree G1 to perform diverse real-world skills using a single neural network controller.Demonstrated behaviors span whole-body manipulation, legged manipulation, locomotion, and expressive motion.Source: Figure 2 · p. 2, §1 · p. 3

Learning-Based Whole-Body Control for Humanoid Robots

• Learning-based humanoid controllers have enabled skills including walking, jumping, parkour, dancing, hopping, and fall recovery.Most prior methods focus on a single specific task, limiting their generality.Source: §2 · p. 4

Teleoperation Systems for Humanoid Robots

• Humanoid teleoperation systems use sparse VR keypoints, exoskeletons, motion-capture suits, and camera-based pose estimation.Despite these varied modalities, most systems show limited coordinated whole-body skills for general household tasks.Source: §2 · p. 4

• TWIST frames whole-body humanoid teleoperation as real-time motion tracking from retargeted human motions to target robot joint positions.Source: §3 · pp. 4–5

Motion Data Curation

• The training pipeline combines over 15,000 public motion clips with 150 in-house MoCap clips that reflect noisy real-world teleoperation conditions.The public datasets contain around 42 hours of motion, while the in-house dataset contains around 0.5 hours.Source: §3.1 · p. 5
• Offline and online retargeting use different optimization strategies, with joint optimization of 3D positions and orientations mitigating the offline-to-online quality gap.Offline retargeting prioritizes quality through iterative optimization, whereas online retargeting prioritizes fast inference.Source: §3.1 · p. 5

Controller Training

• The teacher policy observes 2 seconds of future reference frames, enabling anticipation and smoother locomotion while tracking local-frame joint positions and root velocities.Its reward emphasizes tracking accuracy and penalizes jitter and foot slippage.Source: §3.2 · pp. 5–6
• The deployable student policy uses proprioception and immediate targets, and is trained with RL plus KL-based behavior cloning from the teacher.The imitation weight λ is gradually reduced during training.Source: §3.2 · p. 6

Deployment and Evaluation

• RL+BC significantly outperforms pure BC and pure RL, producing smoother motions and better generalization while supporting zero-shot real-robot deployment.The real-time pipeline captures motion at 120Hz and retargets it at 50Hz.Source: §3.2 · p. 6, §3.3 · p. 6

Main Results

• TWIST enables Unitree G1 to perform whole-body manipulation, legged manipulation, locomotion, and expressive movement with one neural network controller.Demonstrated skills include lifting and carrying objects, kicking, walking, crouching, boxing, and dancing.Source: §4.1 · p. 7
• The controller successfully tracks coordinated whole-body motions on Booster T1, including arm swinging, deep crouching, and walking.Source: §4.1 · p. 7

Ablation Experiments

• RL+BC achieves superior tracking accuracy and motion smoothness compared with pure RL and DAgger-based behavior cloning.Pure RL can produce feet-sliding artifacts, while DAgger occasionally fails to robustly track unseen motions.Source: §4.2 · pp. 7–8
• Adding a small set of in-house MoCap sequences substantially reduces tracking errors on unseen motions by exposing the controller to real-world capture and retargeting imperfections.The in-house data contains calibration drift, occlusions, and less-smooth online-retargeted references.Source: §4.2 · p. 8
• Training with end-effector perturbations significantly improves stability during stationary poses and contact-rich tasks.Without perturbations, policies can drift and become unstable during stationary poses.Source: §4.2 · p. 8
• Jointly optimizing 3D joint positions and orientations produces smoother online-retargeted motions and reduces overall tracking errors.The improvement benefits controllers trained both with and without MoCap data.Source: §4.2 · p. 8

System Analyses

• Tracking errors are largest at end-effectors and generally higher in lower-body parts than upper-body parts.Feet and knees are harder to track because of more complex contact dynamics.Source: §4.3 · p. 8, Figure 8 · p. 8
• The system has approximately 0.9 seconds of teleoperation delay, with 0.7 seconds from tracking-goal generation and 0.2 seconds from policy inference.Reducing latency is identified as a focus for future improvements.Source: §4.3 · p. 8

• TWIST teleoperates real-world humanoid robots using whole-body human motion captured from MoCap devices.Source: §5 · p. 9
• The system fully leverages the robot’s whole-body joints to execute versatile and coordinated skills previously unachieved in the literature.Source: §5 · p. 9
• TWIST demonstrates exceptional reachability compared with prior work, expanding humanoid robot capabilities.Source: §5 · p. 9
• Future work will study visuomotor policies combining TWIST-collected teleoperation data with egocentric human data.Source: §5 · p. 9

• The system provides no egocentric vision or tactile feedback, making occlusion difficult to manage and grasping less natural and reliable.Source: §6 · p. 9
• Current humanoid hardware cannot sustain long-term continuous operation, constraining system capability and robustness.Source: §6 · p. 9
• TWIST relies on a motion-capture system that is not portable and difficult to democratize.Source: §6 · p. 9