TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection System

• Humanoid robotics lacks data collection frameworks as effective and scalable as those supporting recent advances in language, vision-language-action, and bimanual manipulation models.Source: §I · p. 1
• Existing systems trade off portability and control scope: some decouple body regions, while others provide full-body control but rely on expensive, non-portable motion capture.Source: §I · pp. 1–2, Table I · p. 2
• TWIST2 combines PICO4U whole-body motion streaming, a low-cost attachable neck, and full-body control for portable, mocap-free teleoperation and scalable data collection.Source: §I · p. 2
• Around 100 successful demonstrations can be collected in 20 minutes without failure, while the system supports long-horizon dexterous and mobile skills.Source: §I · p. 2
• A hierarchical framework pairs a motion-tracking low-level controller with a Diffusion Policy that predicts whole-body joint positions from visual observations.Source: §I · p. 2
• The autonomous policy demonstrates consecutive whole-body dexterous pick-and-place and continuous kicking of a T-shaped box toward target regions.Source: §I · p. 2

II. RELATED WORK

• Full whole-body control is presented as promising for coordinated humanoid dexterity, whereas prior systems identify unresolved requirements for scalable real-world teleoperation and data collection.Source: A. Whole-Body Humanoid Teleoperation · pp. 2–3
• TWIST2 integrates active vision, portable VR motion capture, human-to-robot retargeting, and a general motion tracker for scalable collection and autonomous execution.Source: §III · p. 3
• The low-level controller treats motion tracking as task-agnostic and converts reference motions into desired joint targets for physical-robot control.Source: A. Problem Formulation · p. 3
• At 50Hz, the controller outputs desired joint positions that a PD controller tracks to generate final torque.Source: A. Problem Formulation · p. 3
• The high-level controller maps visual observations and proprioceptive states to task-specific motion commands for teleoperation or autonomous policies.Source: A. Problem Formulation · p. 3
• Relative root translations and rotations reduce dependence on accurate global-state estimation, while whole-body joint positions retain precise lower-body control.Source: A. Problem Formulation · p. 3

• The Unitree G1 has 29 DoF and two 7 DoF Dex31 hands, with neck degrees of freedom added for effective long-horizon teleoperation.Source: B. Humanoid Robot with Active Vision · p. 3
• TWIST2 uses a portable yaw-and-pitch neck module that attaches to the G1 without disassembling its original head.Source: B. Humanoid Robot with Active Vision · p. 3, Figure 3 · p. 4
• The system overview combines portable devices and egocentric active vision for scalable imitation data collection, followed by hierarchical policy learning.Source: Figure 2 · p. 4
• MuJoCo XML files for the neck support simulation research and data standardization.Source: Figure 4 · p. 4

Portable motion capture

• PICO 4U with two calf-mounted motion trackers provides portable real-time full-body human pose estimation for about $1000.The two-tracker configuration was found to provide more stable pose estimation, with motion streamed at 100Hz.Source: C. Portable MoCap-Free Whole-Body… · p. 4
• Around 1 minute is required to set up PICO, without the heavy calibration associated with optical motion capture.The system is described as cheaper and more practical than optical MoCap.Source: C. Portable MoCap-Free Whole-Body… · p. 4

Portability

• PICO whole-body estimation avoids the extra third-person camera setup required by HTC Vive Tracker-based systems.This makes the setup more flexible for portable operation.Source: C. Portable MoCap-Free Whole-Body… · p. 4

Active neck sensing

• A 2-DoF robot neck with yaw and pitch is sufficient to mimic major human neck movements.The neck design supports the paper’s goal of obtaining whole-body human motion for humanoid control.Source: Figure 5 · p. 4

Body retargeting

• The retargeting pipeline adapts GMR to PICO motion and separately handles lower-body position-rotation constraints and upper-body motion.The method modifies GMR’s second optimization stage because PICO can produce inaccurate global pose estimates.Source: D. Holistic Human-to-Humanoid… · p. 4
• Pelvis-centered positions reduce sensitivity to noisy global pose estimation and support user teleportation.Selected lower-body points are expressed in the human pelvis frame during optimization.Source: D. Holistic Human-to-Humanoid… · p. 5
• Foot and ankle placement constraints mitigate foot sliding, while upper-body retargeting uses local rotations to avoid teleportation artifacts.Upper-body positional terms are omitted so global-pose jumps do not introduce artifacts.Source: D. Holistic Human-to-Humanoid… · p. 5

Hand retargeting

• Dex31 is treated as a gripper: a scalar grasp command interpolates between open and closed configurations.The system avoids direct five-finger hand-pose mapping because Dex31 has three fingers and limited degrees of freedom.Source: D. Holistic Human-to-Humanoid… · p. 5
• Separate open and closed configurations support both power grasps and fine-grained pinching tasks.The configuration sets are used for examples such as grasping a cup and folding cloths.Source: D. Holistic Human-to-Humanoid… · p. 5

Neck retargeting

• A two-DoF yaw-and-pitch neck maps relative head-to-spine rotation into robot neck targets.The neck retargeting formulation uses the relative rotation between human head and spine.Source: D. Holistic Human-to-Humanoid… · p. 5

Controller training

• A one-stage whole-body controller πlow is trained for general motion tracking instead of using a complex teacher-student pipeline.The controller maps reference motions to desired PD targets on the physical robot.Source: E. Training General Motion Trackers… · p. 5

Motion data

• Around 20k motion clips combine GMR-retargeted data, TWIST data, AMASS, OMOMO, and in-house MoCap.The mixture is intended to provide broad motion coverage, including omnidirectional walking.Source: E. Training General Motion Trackers… · p. 5
• Only 73 PICO-collected motions are added to bridge the domain gap, covering walking, crouching, and manipulation.These motions are described as covering most daily movements needed by the tracker.Source: E. Training General Motion Trackers… · p. 5

Objective

• The reward combines tracking and regularization terms, with regularization including penalties on action changes.The tracking term measures the distance between commanded and achieved robot states.Source: E. Training General Motion Trackers… · p. 5

Controller training

• Compressing history proprioceptions and reference motions into a latent vector boosts learning efficiency.The actor uses a convolutional history encoder and an MLP backbone and is trained with PPO.Source: E. Training General Motion Trackers… · p. 5

Teleoperation pipeline

• Real-time PICO motion is mapped into robot commands and sent to the low-level tracker during teleoperation.The system also provides stereoscopic robot vision through a custom shader and streams visual data for collection.Source: F. Scalable Humanoid Data Collection · p. 5

Whole-body learning

• The visuomotor framework uses TWIST2 data to control the entire body, including coordinated tasks such as Kick-T.The framework is built around whole-body control rather than separate upper-body manipulation or lower-body locomotion.Source: Figure 7 · p. 6

Teleoperator interface

• The PICO interface places robot vision at the center of the teleoperator’s view.This provides the operator with an egocentric view during teleoperation.Source: Figure 8 · p. 6

Single-operator operation

• A single operator can manage teleoperation using handheld controllers as the system’s control center.The design addresses session start, pause, and termination through the controller interface.Source: F. Scalable Humanoid Data Collection · p. 6

Safe control

• Pause recovery interpolates from the last robot pose to the current target pose for smooth state transitions.The origin joystick activates pause, and interpolation is used when pause mode ends.Source: Figure 9 · p. 6

Observation and Action Space

• The policy operates on visual observations and proprioceptive information to generate motion commands.Visual input comes from a ZED Mini RGB camera, while historical command sequences are used instead of raw robot states.Source: G. Whole-Body Visuomotor Policy Learning · pp. 6–7
• Using historical commands rather than raw states decouples the high-level policy from the low-level controller and mitigates error accumulation.Source: G. Whole-Body Visuomotor Policy Learning · p. 7

Deployment and Inference

• The trained Diffusion Policy runs at 20Hz on a single NVIDIA RTX 4090 after ONNX conversion.The system executes 48 of each predicted 64-step action chunk at 30Hz.Source: G. Whole-Body Visuomotor Policy Learning · p. 7

Long-Horizon Teleoperation

• TWIST2 enables long-horizon dexterous and mobile tasks through egocentric active perception and smooth whole-body tracking.Demonstrated tasks include folding towels and transporting baskets through a door using one teleoperator.Source: A. Long-Horizon Teleoperation · pp. 7–8, §IV · p. 7

Efficient Data Collection

• Around 100 successful bimanual pick&place demonstrations can be collected within 20 minutes.Source: B. Efficient Data Collection · p. 8

User Study

• TWIST2 achieves the shortest completion times and highest success rates across the evaluated user and setup configurations.The study compares an experienced user with a first-time novice and progressively removes system features.Source: B. Efficient Data Collection · pp. 8–9, Table II · p. 8

Design Analysis

• Removing stereo vision increases grasp failures, removing the neck blocks perception beyond the fixed field of view, and third-person viewing is unsuitable for remote mobile manipulation.An expert completed 10 episodes in 43 seconds with third-person VR pass-through, but only while standing beside the robot.Source: B. Efficient Data Collection · p. 9

Whole-body dexterous pick & place

• The WB-Dex policy is trained with 170 human demonstrations to pick a cup from a shelf and place it into a ground-level box.The policy reliably reaches the cup in most cases, but slight drift can cause grasp failure because the cup is very light.Source: C. Whole-Body Policy Learning Results · p. 9
• The WB-Dex evaluation reports success and failure rates for autonomous whole-body dexterous pick-and-place behavior.Source: C. Whole-Body Policy Learning Results · p. 9, Figure 14 · p. 9

Kick T-shaped box to target

• The Kick-T policy is trained with 50 demonstrations to kick a T-shaped box toward a fixed T-shaped target.The learned action pattern uses the left foot for kicking and a right-foot step for balance.Source: C. Whole-Body Policy Learning Results · p. 9

Conclusions

• TWIST2 provides portable, holistic, mocap-free humanoid data collection with full whole-body control and an attachable neck for egocentric vision.Source: §V · p. 9
• The framework enables autonomous whole-body skills including dexterous manipulation and Kick-T.Source: §V · p. 9

Limitations

• The general motion tracker struggles with highly dynamic movements such as sprinting.The stated challenge is tracking fast, complex motions.Source: §V · p. 9
• PICO whole-body pose estimation is less accurate than high-cost motion capture, especially at elbows and knees without trackers.This limitation results in reduced motion quality.Source: §V · p. 9

Future Scaling

• Scaling data collection remains constrained by cross-embodiment discrepancies across the growing range of humanoid platforms.The authors advocate standardizing humanoid hardware and adopt Unitree G1 in this work.Source: §VI · pp. 9–10