Source-linked AI summary
TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection System
Yanjie Ze, Siheng Zhao, Weizhuo Wang, Angjoo Kanazawa, Rocky Duan, Pieter Abbeel, Guanya Shi, Jiajun Wu, C. Karen Liu
TL;DR
Humanoid robotics lacks scalable data collection systems that preserve full-body control without expensive motion capture. TWIST2 combines portable VR teleoperation, an attachable egocentric-vision neck, and hierarchical visuomotor learning, demonstrating long-horizon skills and autonomous whole-body behaviors.
Problem
Humanoid robotics lacks effective scalable data collection, while existing systems either decouple control or depend on expensive motion capture.
Method
TWIST2 combines PICO4U whole-body motion streaming, a low-cost attachable neck for egocentric vision, and hierarchical visuomotor policy learning.
Results
TWIST2 demonstrates long-horizon dexterous and mobile teleoperation, plus autonomous whole-body pick-and-place and kicking behaviors.
Takeaways & Limitations
The open-sourced system and dataset provide a reproducible pipeline for portable humanoid teleoperation, scalable demonstrations, and full-body visuomotor control.
Takeaways & Limitations
The general motion tracker struggles with highly dynamic movements, and PICO pose estimation is less accurate than high-cost motion capture, especially at elbows and knees.
Abstract
from arXiv · showhide
Large-scale data has driven breakthroughs in robotics, from language models to vision-language-action models in bimanual manipulation. However, humanoid robotics lacks equally effective data collection frameworks. Existing humanoid teleoperation systems either use decoupled control or depend on expensive motion capture setups. We introduce TWIST2, a portable, mocap-free humanoid teleoperation and data collection system that preserves full whole-body control while advancing scalability. Our system leverages PICO4U VR for obtaining real-time whole-body human motions, with a custom 2-DoF robot neck (cost around $250) for egocentric vision, enabling holistic human-to-humanoid control. We demonstrate long-horizon dexterous and mobile humanoid skills and we can collect 100 demonstrations in 15 minutes with an almost 100% success rate. Building on this pipeline, we propose a hierarchical visuomotor policy framework that autonomously controls the full humanoid body based on egocentric vision. Our visuomotor policy successfully demonstrates whole-body dexterous manipulation and dynamic kicking tasks. The entire system is fully reproducible and open-sourced at https://yanjieze.com/TWIST2 . Our collected dataset is also open-sourced at https://twist-data.github.io .
I. INTRODUCTION
TWIST2 addresses the lack of scalable humanoid data collection by combining portable, mocap-free full-body teleoperation with egocentric active vision. It also uses the resulting demonstrations to train a hierarchical policy for autonomous whole-body control.
- Humanoid robotics lacks data collection frameworks as effective and scalable as those supporting recent advances in language, vision-language-action, and bimanual manipulation models.
- Existing systems trade off portability and control scope: some decouple body regions, while others provide full-body control but rely on expensive, non-portable motion capture.
- TWIST2 combines PICO4U whole-body motion streaming, a low-cost attachable neck, and full-body control for portable, mocap-free teleoperation and scalable data collection.
- Around 100 successful demonstrations can be collected in 20 minutes without failure, while the system supports long-horizon dexterous and mobile skills.
- A hierarchical framework pairs a motion-tracking low-level controller with a Diffusion Policy that predicts whole-body joint positions from visual observations.
- The autonomous policy demonstrates consecutive whole-body dexterous pick-and-place and continuous kicking of a T-shaped box toward target regions.
II. RELATED WORK
The paper positions full-body teleoperation as central to humanoid loco-manipulation and organizes TWIST2 around a unified control and data-collection pipeline. Its interface uses visual and proprioceptive inputs to produce whole-body motion commands for teleoperation or autonomous policies.
- II. RELATED WORK: Full whole-body control is presented as promising for coordinated humanoid dexterity, whereas prior systems identify unresolved requirements for scalable real-world teleoperation and data collection.
- II. RELATED WORK: TWIST2 integrates active vision, portable VR motion capture, human-to-robot retargeting, and a general motion tracker for scalable collection and autonomous execution.
- II. RELATED WORK: The low-level controller treats motion tracking as task-agnostic and converts reference motions into desired joint targets for physical-robot control.
- II. RELATED WORK: At 50Hz, the controller outputs desired joint positions that a PD controller tracks to generate final torque.
- II. RELATED WORK: The high-level controller maps visual observations and proprioceptive states to task-specific motion commands for teleoperation or autonomous policies.
- II. RELATED WORK: Relative root translations and rotations reduce dependence on accurate global-state estimation, while whole-body joint positions retain precise lower-body control.
B. Humanoid Robot with Active Vision
TWIST2 equips the Unitree G1 with an attachable active-vision neck designed for practical whole-body teleoperation. The system overview connects this hardware to portable data collection and hierarchical visuomotor policy learning.
- The Unitree G1 has 29 DoF and two 7 DoF Dex31 hands, with neck degrees of freedom added for effective long-horizon teleoperation.
- TWIST2 uses a portable yaw-and-pitch neck module that attaches to the G1 without disassembling its original head.
- The system overview combines portable devices and egocentric active vision for scalable imitation data collection, followed by hierarchical policy learning.
- MuJoCo XML files for the neck support simulation research and data standardization.
C. Portable MoCap-Free Whole-Body Human Data Source
TWIST2 uses PICO 4U with two ankle-mounted motion trackers to stream whole-body human poses without heavy calibration or third-person cameras. A two-DoF yaw-and-pitch neck supports major human neck movements while setup takes about one minute.
- Portable motion capture: PICO 4U with two calf-mounted motion trackers provides portable real-time full-body human pose estimation for about $1000.The two-tracker configuration was found to provide more stable pose estimation, with motion streamed at 100Hz.
- Portable motion capture: Around 1 minute is required to set up PICO, without the heavy calibration associated with optical motion capture.The system is described as cheaper and more practical than optical MoCap.
- Portability: PICO whole-body estimation avoids the extra third-person camera setup required by HTC Vive Tracker-based systems.This makes the setup more flexible for portable operation.
- Active neck sensing: A 2-DoF robot neck with yaw and pitch is sufficient to mimic major human neck movements.The neck design supports the paper’s goal of obtaining whole-body human motion for humanoid control.
D. Holistic Human-to-Humanoid Retargeting
TWIST2 retargets VR human motion to humanoid body, hand, and neck commands using pelvis-centered lower-body constraints and local-rotation upper-body control. It simplifies Dex31 hand control into task-specific gripper interpolation while preserving foot placement and teleportation robustness.
- Body retargeting: The retargeting pipeline adapts GMR to PICO motion and separately handles lower-body position-rotation constraints and upper-body motion.The method modifies GMR’s second optimization stage because PICO can produce inaccurate global pose estimates.
- Body retargeting: Pelvis-centered positions reduce sensitivity to noisy global pose estimation and support user teleportation.Selected lower-body points are expressed in the human pelvis frame during optimization.
- Body retargeting: Foot and ankle placement constraints mitigate foot sliding, while upper-body retargeting uses local rotations to avoid teleportation artifacts.Upper-body positional terms are omitted so global-pose jumps do not introduce artifacts.
- Hand retargeting: Dex31 is treated as a gripper: a scalar grasp command interpolates between open and closed configurations.The system avoids direct five-finger hand-pose mapping because Dex31 has three fingers and limited degrees of freedom.
- Hand retargeting: Separate open and closed configurations support both power grasps and fine-grained pinching tasks.The configuration sets are used for examples such as grasping a cup and folding cloths.
- Neck retargeting: A two-DoF yaw-and-pitch neck maps relative head-to-spine rotation into robot neck targets.The neck retargeting formulation uses the relative rotation between human head and spine.
E. Training General Motion Trackers for Low-Level Control
TWIST2 trains a general low-level motion tracker with one-stage PPO over a broad humanoid motion dataset, augmented by a small set of PICO motions to bridge the teleoperation domain gap. History compression improves learning efficiency.
- Controller training: A one-stage whole-body controller πlow is trained for general motion tracking instead of using a complex teacher-student pipeline.The controller maps reference motions to desired PD targets on the physical robot.
- Motion data: Around 20k motion clips combine GMR-retargeted data, TWIST data, AMASS, OMOMO, and in-house MoCap.The mixture is intended to provide broad motion coverage, including omnidirectional walking.
- Motion data: Only 73 PICO-collected motions are added to bridge the domain gap, covering walking, crouching, and manipulation.These motions are described as covering most daily movements needed by the tracker.
- Objective: The reward combines tracking and regularization terms, with regularization including penalties on action changes.The tracking term measures the distance between commanded and achieved robot states.
- Controller training: Compressing history proprioceptions and reference motions into a latent vector boosts learning efficiency.The actor uses a convolutional history encoder and an MLP backbone and is trained with PPO.
F. Scalable Humanoid Data Collection
TWIST2 combines streamed VR motion, whole-body retargeting, low-level tracking, stereoscopic egocentric vision, and single-operator controls for scalable humanoid data collection. Its framework supports coordinated whole-body visuomotor learning and smooth long-horizon operation with pause-state interpolation.
- Teleoperation pipeline: Real-time PICO motion is mapped into robot commands and sent to the low-level tracker during teleoperation.The system also provides stereoscopic robot vision through a custom shader and streams visual data for collection.
- Whole-body learning: The visuomotor framework uses TWIST2 data to control the entire body, including coordinated tasks such as Kick-T.The framework is built around whole-body control rather than separate upper-body manipulation or lower-body locomotion.
- Teleoperator interface: The PICO interface places robot vision at the center of the teleoperator’s view.This provides the operator with an egocentric view during teleoperation.
- Single-operator operation: A single operator can manage teleoperation using handheld controllers as the system’s control center.The design addresses session start, pause, and termination through the controller interface.
- Safe control: Pause recovery interpolates from the last robot pose to the current target pose for smooth state transitions.The origin joystick activates pause, and interpolation is used when pause mode ends.
G. Whole-Body Visuomotor Policy Learning
TWIST2 uses a hierarchical visuomotor policy that converts egocentric visual and proprioceptive observations into full-body humanoid motion through a low-level controller.
- Observation and Action Space: The policy operates on visual observations and proprioceptive information to generate motion commands.Visual input comes from a ZED Mini RGB camera, while historical command sequences are used instead of raw robot states.
- Observation and Action Space: Using historical commands rather than raw states decouples the high-level policy from the low-level controller and mitigates error accumulation.
- Deployment and Inference: The trained Diffusion Policy runs at 20Hz on a single NVIDIA RTX 4090 after ONNX conversion.The system executes 48 of each predicted 64-step action chunk at 30Hz.
IV. EXPERIMENT RESULTS
TWIST2 supports long-horizon whole-body teleoperation, rapid demonstration collection, and remote operation enabled by active egocentric stereo vision.
- Long-Horizon Teleoperation: TWIST2 enables long-horizon dexterous and mobile tasks through egocentric active perception and smooth whole-body tracking.Demonstrated tasks include folding towels and transporting baskets through a door using one teleoperator.
- Efficient Data Collection: Around 100 successful bimanual pick&place demonstrations can be collected within 20 minutes.
- User Study: TWIST2 achieves the shortest completion times and highest success rates across the evaluated user and setup configurations.The study compares an experienced user with a first-time novice and progressively removes system features.
- Design Analysis: Removing stereo vision increases grasp failures, removing the neck blocks perception beyond the fixed field of view, and third-person viewing is unsuitable for remote mobile manipulation.An expert completed 10 episodes in 43 seconds with third-person VR pass-through, but only while standing beside the robot.
C. Whole-Body Policy Learning Results
The hierarchical policy demonstrates autonomous whole-body dexterous manipulation and kicking using demonstrations collected for each task.
- Whole-body dexterous pick & place: The WB-Dex policy is trained with 170 human demonstrations to pick a cup from a shelf and place it into a ground-level box.The policy reliably reaches the cup in most cases, but slight drift can cause grasp failure because the cup is very light.
- Whole-body dexterous pick & place: The WB-Dex evaluation reports success and failure rates for autonomous whole-body dexterous pick-and-place behavior.
- Kick T-shaped box to target: The Kick-T policy is trained with 50 demonstrations to kick a T-shaped box toward a fixed T-shaped target.The learned action pattern uses the left foot for kicking and a right-foot step for balance.
V. CONCLUSIONS AND LIMITATIONS
TWIST2 combines portable mocap-free full-body data collection with autonomous whole-body skills, while its motion tracking remains limited for highly dynamic movement and some joints.
- Conclusions: TWIST2 provides portable, holistic, mocap-free humanoid data collection with full whole-body control and an attachable neck for egocentric vision.
- Conclusions: The framework enables autonomous whole-body skills including dexterous manipulation and Kick-T.
- Limitations: The general motion tracker struggles with highly dynamic movements such as sprinting.The stated challenge is tracking fast, complex motions.
- Limitations: PICO whole-body pose estimation is less accurate than high-cost motion capture, especially at elbows and knees without trackers.This limitation results in reduced motion quality.
- Future Scaling: Scaling data collection remains constrained by cross-embodiment discrepancies across the growing range of humanoid platforms.The authors advocate standardizing humanoid hardware and adopt Unitree G1 in this work.