Source-linked AI summary
Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations
Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, Zunhao Chen, Yihang Hu, Yingdong Hu, Tong Zhang, Chuan Wen, Yang Gao
TL;DR
Humanoid whole-body manipulation is limited by the labor and hardware demands of teleoperation and visual sim-to-real reinforcement learning. HuMI provides robot-free portable demonstrations and hierarchical learning for transferring whole-body human motions to humanoids. Across five tasks, it reports 3× higher data-collection throughput than teleoperation and 70% success in unseen objects and environments.
Problem
Teleoperation and visual sim-to-real reinforcement learning are labor-intensive, and existing demonstrations cover few autonomous whole-body tasks in fixed laboratory environments.
Method
HuMI uses portable robot-free hardware to capture whole-body human trajectories and trains hierarchical high- and low-level policies to transfer them to humanoids.
Results
Across five diverse whole-body tasks, HuMI achieves 3× higher data-collection throughput than teleoperation and 70% success in unseen objects and environments.
Takeaways & Limitations
HuMI supports robot-free collection and transfer of diverse humanoid manipulation skills across tasks and environments.
Takeaways & Limitations
The paper’s conclusion frames HuMI as addressing the embodiment gap between humans and humanoids, rather than eliminating that gap entirely.
Abstract
from arXiv · showhide
Current approaches for humanoid whole-body manipulation, primarily relying on teleoperation or visual sim-to-real reinforcement learning, are hindered by hardware logistics and complex reward engineering. Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables robot-free data collection by capturing rich whole-body motion using portable hardware. This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks--including kneeling, squatting, tossing, walking, and bimanual manipulation--demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.
I. INTRODUCTION
HuMI addresses the limits of teleoperation and visual sim-to-real reinforcement learning with robot-free, portable whole-body demonstration and hierarchical learning. It transfers human manipulation motions to humanoids across diverse tasks and environments while improving collection efficiency and unseen-environment success.
- Motivation: Current approaches rely mainly on teleoperation or visual sim-to-real reinforcement learning, yet demonstrate few autonomous tasks in fixed laboratory environments.These methods are labor-intensive and typically provide limited whole-body coordination beyond upright walking with simple actions.
- Evaluation: 3× higher data-collection throughput than teleoperation and 70% success in unseen objects and environments demonstrate HuMI’s efficiency and generalization.The evaluation covers five diverse whole-body tasks, including marriage proposal, squatting, tossing, unsheathing a sword, and walking to clean a table.
- HuMI framework: HuMI introduces the first robot-free demonstration system for humanoid whole-body manipulation tasks.The framework transfers manipulation skills from humans to humanoids while addressing the embodiment gap.
- HuMI framework: HuMI collects whole-body trajectories and image observations, using them to train a high-level policy and a low-level controller.The high-level Diffusion Policy maps image observations to target keypoint trajectories, while the low-level controller outputs joint angles for tracking.
- Data collection: Portable handheld grippers and trackers capture pelvis, hand, and foot motion without requiring a robot or base station.The system uses full-body tracking and an online IK preview to adapt motions for robot feasibility while preserving object-related spatial alignment.
B. Manipulation-Centric Whole-Body Controller
HuMI’s whole-body controller combines coordinated motion tracking with manipulation-specific end-effector precision. It uses adaptive rewards and variable execution speeds to preserve stability while learning accurate interactions.
- Controller motivation: 4–6 cm tracking deviations in existing trackers are insufficient for fine manipulation, while simply tightening end-effector tolerance harms whole-body coordination and stability.HuMI therefore combines adaptive tracking rewards with variable-speed augmentation.
- Adaptive end-effector tracking: The final tracking objective combines whole-body and end-effector rewards as r_tracking = w_bodyr_body + w_EEr_EE.The whole-body terms cover position, orientation, and linear/angular velocity metrics.
- Adaptive end-effector tracking: Adaptive end-effector rewards require high accuracy during slow interactions while allowing greater tolerance as reference velocity increases.The reward is gated to deactivate end-effector tracking when base velocity exceeds a threshold, prioritizing stability during rapid movement.
- Training curriculum: A curriculum gradually increases end-effector reward weight and anneals the minimum precision tolerance to avoid prematurely producing uncoordinated postures.This staged training preserves whole-body coordination while increasing manipulation precision.
- Variable-speed augmentation: Variable-speed augmentation samples execution speed factors within [s_min, s_max] during each episode, giving the policy more time to correct small errors.This supports learning highly precise movements when fixed-speed references advance too quickly.
C. Policy Interface for Improved System Integration
HuMI redesigns the policy interface to preserve temporal continuity and reduce drift when translating whole-body demonstrations into humanoid actions. It uses scheduled target references and relative tracking for non-vision-grounded keypoints.
- Interface integration: Naively feeding action targets to the low-level controller can make whole-body execution fragile because errors from both policy levels compound.The interface is redesigned to improve coordination between high- and low-level control.
- Target pose as high-level action reference: Using the previous target pose as the action reference preserves continuity across action chunks despite tracking error.Referencing the lagging executed pose can cause sudden reversals and disrupt momentum.
- Keypoint representation: The interface supports richer whole-body control by incorporating keypoints beyond grippers, including the pelvis and feet.Non-vision-grounded keypoints require relative rather than absolute tracking.
- Relative pose tracking: Relative tracking for non-vision-grounded keypoints avoids relying on absolute transforms corrupted by accumulated drift.Pelvis height drift can exceed 5 cm during stationary grasping.
III. EXPERIMENTS
HuMI is evaluated on whole-body coordination, precise bimanual manipulation, dynamic motion, and loco-manipulation using robot-free demonstrations. The experiments include capability tests and ablations that identify the importance of kinematic adaptation, whole-body supervision, and variable-speed augmentation.
- Whole-body capability: HuMI completes the marriage-proposal whole-body motion in 17/20 = 85% cases.The robot balances through kneeling, grasps the ring-shaped toy, and lifts it smoothly.
- Ablations: Removing human-in-the-loop kinematic adaptation reduces success from 17/20 = 85% to 1/10 = 10%.The variant often produces kinematically inappropriate motions such as excessively splayed kneeling.
- Ablations: Removing whole-body supervision reduces success to 0/10 because sparse end-effector waypoints underspecify coordinated motion.The controller often converges to kinematically or dynamically inappropriate solutions.
- Precise bimanual manipulation: HuMI completes sword unsheathing in 17/20 = 85% trials with an average end-effector tracking error of 15.7 mm.The robot maintains precise grasps and synchronized bimanual motion without slippage or rack collisions.
- Ablations: Removing variable-speed augmentation lowers success from 85% to 5/10 = 50% and increases average tracking error to 21.2 mm.Failures include unreliable grasping and insufficient bimanual synchronization.
C. Temporally Coherent Dynamic Control
HuMI combines temporally coherent control with whole-body and loco-manipulation capabilities, while robot-free demonstrations support evaluation beyond the training environment. Its experiments show successful dynamic tossing, table cleaning, and generalization to unseen scenes and objects.
- Temporally coherent dynamic control: HuMI completes dynamic tossing in 15/20 = 75% trials with smooth, stable trajectories and reliable object placement.The robot releases near the peak of the forward swing, producing accurate velocity and direction.
- Temporally coherent dynamic control: Using the previous target pose instead of the actual executed pose prevents trajectory reversals that disrupt the forward swing.The ablation drops success from 15/20 = 75% to 4/10 = 40%.
- Loco-manipulation: HuMI completes walking-to-cleaning in 15/20 trials by transitioning from navigation to precise wiping.Successful trials position the robot within arm reach and clear the tabletop with overlapping strokes.
- Loco-manipulation: Adding pelvis and feet targets improves whole-body intent expression and supports coordination across walking and wiping.EE-only succeeds in 6/10 = 60%, compared with 15/20 = 75% for EE+pelvis and 7/10 = 70% for EE+pelvis+feet.
- Generalization ability: HuMI achieves 14/20 = 70% success across unseen environments and unseen objects.The evaluation uses four new scenes and six novel items, including a vase unlike the training bottles.
VI. DATA COLLECTION EFFICIENCY
HuMI is evaluated against teleoperation for efficient, reliable, and broad whole-body demonstration collection. It achieves higher throughput and acceptance while capturing motions that teleoperation cannot reliably produce.
- Evaluation criteria: HuMI and TWIST2 were compared using throughput, acceptance rate, and whole-body motion coverage.The evaluation used unsheathing for throughput and marriage-proposal motion for challenging coverage.
- Acceptance rate: 96.7% versus 64.3% acceptance rates were achieved by HuMI and TWIST2, respectively.An acceptable episode had to complete the task and be replayable end-to-end by a policy trained on the collected trajectories.
- Efficiency: HuMI reduced average time per acceptable episode to 30.0% of TWIST2’s time.This compares the time required to obtain episodes usable for downstream humanoid policy learning.
- Whole-body motion coverage: HuMI collected 50 marriage-proposal demonstrations in 15 minutes with 100% acceptance, whereas TWIST2 produced none.The target required deep kneeling, which the teleoperated humanoid could not reliably realize while maintaining stability.
- Scope: HuMI is positioned as the first robot-free data-collection system specifically for humanoid whole-body manipulation.Prior robot-free systems typically rely only on end-effectors and lack complex whole-body coordination.
APPENDIX
HuMI combines portable sensing with real-time inverse-kinematics preview to collect whole-body demonstrations without a robot. Its hardware tracks hand, waist, and foot motion while the interface enforces feasible, collision-free configurations.
- Hardware: HuMI uses portable, base-station-free hardware intended for demonstration collection across diverse environments.The setup is contrasted with outside-in motion-capture systems that lack portability.
- Hardware: The data-collection system uses two sensorized grippers and five wearable 6DoF trackers.Trackers are attached to the grippers, waist, and feet, while gripper cameras record wrist-view RGB observations.
- IK interface: A real-time IK preview helps demonstrators adapt motions to be kinematically feasible and task-compliant.Recorded world-frame SE(3) transforms are used as IK targets, with scaling applied only to pelvis height.
- IK interface: The IK problem tracks five keypoints while avoiding self-collisions and maintaining a natural posture.Mink solves the problem in real time, and Viser visualizes the resulting robot configuration.
- Data processing: Recorded demonstrations pair gripper-camera MP4 videos with time-stamped SE(3) trajectories from the five trackers.Processing begins by synchronizing tracker and video data using timestamps and angular-velocity cross-correlation.
4) Data Collection Protocol:
HuMI’s collection protocol maps each new scene, calibrates and synchronizes sensing, and records repeated demonstrations while the user follows real-time IK feedback. The resulting data supports hierarchical control on the Unitree G1.
- Data Collection Protocol: Each new scene begins with a VIVE tracking-map setup that typically takes 1–2 minutes.The demonstrator follows VIVE prompts before recording task demonstrations.
- Data Collection Protocol: Demonstration recording uses optional gripper calibration, GoPro timestamp synchronization, and voice-controlled video capture.Episode boundaries are defined by the start and stop times of recorded videos.
- Data Collection Protocol: Demonstrators adapt their motions using the real-time IK preview during recording.This connects the collection protocol to the kinematic-feasibility interface.
- Experimental Setup: The real-world experiments use a Unitree G1 with an external workstation and HTC Vive Ultimate Trackers for global localization.The setup includes custom end-effectors, gripper-mounted visual observations, and hierarchical control.
- Hierarchical Control: The high-level policy runs at 5 Hz and publishes desired keypoint trajectories and gripper commands through ZeroMQ.It combines external visual streams with robot proprioception on the workstation.
- Hierarchical Control: The low-level whole-body controller runs at 50 Hz and converts target keypoints into joint-position commands.Execution occurs on the robot’s onboard computer through its built-in PD controller, with trackers providing global references.
C. Additional Experiments
Additional experiments test whether adaptive end-effector rewards and dense whole-body supervision are necessary for stable manipulation. Fixed tight tracking and sparse waypoints substantially degrade task success.
- Experiment Design: The squat-and-pick-up task tests whether prioritizing end-effector precision compromises whole-body stability.The benchmark requires simultaneous high-precision manipulation and dynamic balance.
- Fixed End-Effector Reward: The fixed-reward baseline enforces a uniformly tight end-effector tracking constraint throughout the motion.It replaces the adaptive end-effector reward with a fixed formulation.
- Fixed End-Effector Reward: 85% to 50% was the success-rate change after removing the adaptive end-effector reward.Failures commonly occurred during deep squats through falls or marginally stable poses that prevented subsequent manipulation.
D. Generalization Experiments Details
Generalization experiments use seven training environments and objects, then evaluate across four testing environments and six testing objects.
- Seven training environments each contribute 50 demonstrations, totaling 350 training trajectories.
- Four testing environments are evaluated with five experiments per environment.
- Seven training objects and six testing objects are used for policy training and evaluation.
E. Low-Level Controller Training Details
The low-level controller combines privileged-state teacher training with student distillation, whole-body observations, waypoint commands, tracking rewards, and domain randomization.
- A teacher tracker uses privileged states and full-body reference commands before distilling into a student policy with DAgger.
- The controller state includes full-body joint positions and velocities, base angular velocity, and body-frame gravity, while references include joint and link targets.
- Student observations include 25 steps of state and past-action history, with commands aligned to high-level policy outputs.
- Each action chunk contains 10 waypoints sampled over the next 2 s, using localized end-effector poses, deltas, and relative commands for blind points.
- The reward combines tracking and regularization terms, with adaptive end-effector tolerances and scheduled weight and tolerance changes during training.
- Training uses physical, velocity, reset, and speed domain randomization to expose the controller to deployment mismatches and variable motion speeds.
F. High-Level Policy Training Details
The high-level policy uses visual and proprioceptive observations to predict keypoint and gripper actions, with training data spanning capability and multi-environment tasks.
- Diffusion Policy is the high-level backbone, while ACT can be substituted as an alternative architecture.
- Training hyperparameters remain consistent across experimental tasks.
- The policy combines two 224 × 224 gripper-camera RGB images with lower-body joint angles.
- Actions specify keypoint positions and rotations together with gripper widths, using the UMI action parameterization.
- Capability tasks use 100 demonstrations in one environment, while squatting uses 350 demonstrations across seven environments.