PaperlayerView exact revision on arXiv

Source-linked AI summary

Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations

Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, Zunhao Chen, Yihang Hu, Yingdong Hu, Tong Zhang, Chuan Wen, Yang Gao

arXiv:2602.06643v2cs.ROcs.AIcs.LG

TL;DR

Humanoid whole-body manipulation is limited by the labor and hardware demands of teleoperation and visual sim-to-real reinforcement learning. HuMI provides robot-free portable demonstrations and hierarchical learning for transferring whole-body human motions to humanoids. Across five tasks, it reports 3× higher data-collection throughput than teleoperation and 70% success in unseen objects and environments.

  • Problem

    Teleoperation and visual sim-to-real reinforcement learning are labor-intensive, and existing demonstrations cover few autonomous whole-body tasks in fixed laboratory environments.

    Source: §I · p. 1

  • Method

    HuMI uses portable robot-free hardware to capture whole-body human trajectories and trains hierarchical high- and low-level policies to transfer them to humanoids.

    Source: §I · p. 1

  • Results

    Across five diverse whole-body tasks, HuMI achieves 3× higher data-collection throughput than teleoperation and 70% success in unseen objects and environments.

    Source: §I · p. 2

  • Takeaways & Limitations

    HuMI supports robot-free collection and transfer of diverse humanoid manipulation skills across tasks and environments.

    Source: §I · p. 2

    The paper’s conclusion frames HuMI as addressing the embodiment gap between humans and humanoids, rather than eliminating that gap entirely.

    Source: §VIII · pp. 8–9

Abstract

from arXiv · show

Current approaches for humanoid whole-body manipulation, primarily relying on teleoperation or visual sim-to-real reinforcement learning, are hindered by hardware logistics and complex reward engineering. Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables robot-free data collection by capturing rich whole-body motion using portable hardware. This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks--including kneeling, squatting, tossing, walking, and bimanual manipulation--demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.

I. INTRODUCTION

HuMI addresses the limits of teleoperation and visual sim-to-real reinforcement learning with robot-free, portable whole-body demonstration and hierarchical learning. It transfers human manipulation motions to humanoids across diverse tasks and environments while improving collection efficiency and unseen-environment success.

Motivation

  • Current approaches rely mainly on teleoperation or visual sim-to-real reinforcement learning, yet demonstrate few autonomous tasks in fixed laboratory environments.These methods are labor-intensive and typically provide limited whole-body coordination beyond upright walking with simple actions.Source: §I · p. 1

Evaluation

  • 3× higher data-collection throughput than teleoperation and 70% success in unseen objects and environments demonstrate HuMI’s efficiency and generalization.The evaluation covers five diverse whole-body tasks, including marriage proposal, squatting, tossing, unsheathing a sword, and walking to clean a table.Source: §I · p. 2

HuMI framework

  • HuMI introduces the first robot-free demonstration system for humanoid whole-body manipulation tasks.The framework transfers manipulation skills from humans to humanoids while addressing the embodiment gap.Source: §I · p. 2
  • HuMI collects whole-body trajectories and image observations, using them to train a high-level policy and a low-level controller.The high-level Diffusion Policy maps image observations to target keypoint trajectories, while the low-level controller outputs joint angles for tracking.Source: §II · p. 2, A. Robot-Free Demonstration System · p. 2

Data collection

  • Portable handheld grippers and trackers capture pelvis, hand, and foot motion without requiring a robot or base station.The system uses full-body tracking and an online IK preview to adapt motions for robot feasibility while preserving object-related spatial alignment.Source: A. Robot-Free Demonstration System · pp. 2–3

B. Manipulation-Centric Whole-Body Controller

HuMI’s whole-body controller combines coordinated motion tracking with manipulation-specific end-effector precision. It uses adaptive rewards and variable execution speeds to preserve stability while learning accurate interactions.

Controller motivation

  • 4–6 cm tracking deviations in existing trackers are insufficient for fine manipulation, while simply tightening end-effector tolerance harms whole-body coordination and stability.HuMI therefore combines adaptive tracking rewards with variable-speed augmentation.Source: B. Manipulation-Centric Whole-Body… · pp. 3–4

Adaptive end-effector tracking

  • The final tracking objective combines whole-body and end-effector rewards as r_tracking = w_bodyr_body + w_EEr_EE.The whole-body terms cover position, orientation, and linear/angular velocity metrics.Source: B. Manipulation-Centric Whole-Body… · p. 4
  • Adaptive end-effector rewards require high accuracy during slow interactions while allowing greater tolerance as reference velocity increases.The reward is gated to deactivate end-effector tracking when base velocity exceeds a threshold, prioritizing stability during rapid movement.Source: B. Manipulation-Centric Whole-Body… · p. 4

Training curriculum

  • A curriculum gradually increases end-effector reward weight and anneals the minimum precision tolerance to avoid prematurely producing uncoordinated postures.This staged training preserves whole-body coordination while increasing manipulation precision.Source: B. Manipulation-Centric Whole-Body… · p. 4

Variable-speed augmentation

  • Variable-speed augmentation samples execution speed factors within [s_min, s_max] during each episode, giving the policy more time to correct small errors.This supports learning highly precise movements when fixed-speed references advance too quickly.Source: B. Manipulation-Centric Whole-Body… · p. 4

C. Policy Interface for Improved System Integration

HuMI redesigns the policy interface to preserve temporal continuity and reduce drift when translating whole-body demonstrations into humanoid actions. It uses scheduled target references and relative tracking for non-vision-grounded keypoints.

Interface integration

  • Naively feeding action targets to the low-level controller can make whole-body execution fragile because errors from both policy levels compound.The interface is redesigned to improve coordination between high- and low-level control.Source: C. Policy Interface for Improved… · p. 4

Target pose as high-level action reference

Keypoint representation

  • The interface supports richer whole-body control by incorporating keypoints beyond grippers, including the pelvis and feet.Non-vision-grounded keypoints require relative rather than absolute tracking.Source: C. Policy Interface for Improved… · p. 5

Relative pose tracking

III. EXPERIMENTS

HuMI is evaluated on whole-body coordination, precise bimanual manipulation, dynamic motion, and loco-manipulation using robot-free demonstrations. The experiments include capability tests and ablations that identify the importance of kinematic adaptation, whole-body supervision, and variable-speed augmentation.

Whole-body capability

Ablations

  • Removing human-in-the-loop kinematic adaptation reduces success from 17/20 = 85% to 1/10 = 10%.The variant often produces kinematically inappropriate motions such as excessively splayed kneeling.Source: A. Learning Feasible Whole-Body Skills… · p. 6
  • Removing whole-body supervision reduces success to 0/10 because sparse end-effector waypoints underspecify coordinated motion.The controller often converges to kinematically or dynamically inappropriate solutions.Source: A. Learning Feasible Whole-Body Skills… · p. 6

Precise bimanual manipulation

  • HuMI completes sword unsheathing in 17/20 = 85% trials with an average end-effector tracking error of 15.7 mm.The robot maintains precise grasps and synchronized bimanual motion without slippage or rack collisions.Source: B. Precise Humanoid Bimanual… · pp. 6–7

Ablations

  • Removing variable-speed augmentation lowers success from 85% to 5/10 = 50% and increases average tracking error to 21.2 mm.Failures include unreliable grasping and insufficient bimanual synchronization.Source: B. Precise Humanoid Bimanual… · p. 7

C. Temporally Coherent Dynamic Control

HuMI combines temporally coherent control with whole-body and loco-manipulation capabilities, while robot-free demonstrations support evaluation beyond the training environment. Its experiments show successful dynamic tossing, table cleaning, and generalization to unseen scenes and objects.

Temporally coherent dynamic control

  • HuMI completes dynamic tossing in 15/20 = 75% trials with smooth, stable trajectories and reliable object placement.The robot releases near the peak of the forward swing, producing accurate velocity and direction.Source: C. Temporally Coherent Dynamic Control · p. 7
  • Using the previous target pose instead of the actual executed pose prevents trajectory reversals that disrupt the forward swing.The ablation drops success from 15/20 = 75% to 4/10 = 40%.Source: C. Temporally Coherent Dynamic Control · p. 7

Loco-manipulation

  • HuMI completes walking-to-cleaning in 15/20 trials by transitioning from navigation to precise wiping.Successful trials position the robot within arm reach and clear the tabletop with overlapping strokes.Source: D. Long-Range Loco-Manipulation · p. 7
  • Adding pelvis and feet targets improves whole-body intent expression and supports coordination across walking and wiping.EE-only succeeds in 6/10 = 60%, compared with 15/20 = 75% for EE+pelvis and 7/10 = 70% for EE+pelvis+feet.Source: D. Long-Range Loco-Manipulation · p. 7

Generalization ability

  • HuMI achieves 14/20 = 70% success across unseen environments and unseen objects.The evaluation uses four new scenes and six novel items, including a vase unlike the training bottles.Source: §V · pp. 7–8

VI. DATA COLLECTION EFFICIENCY

HuMI is evaluated against teleoperation for efficient, reliable, and broad whole-body demonstration collection. It achieves higher throughput and acceptance while capturing motions that teleoperation cannot reliably produce.

Evaluation criteria

  • HuMI and TWIST2 were compared using throughput, acceptance rate, and whole-body motion coverage.The evaluation used unsheathing for throughput and marriage-proposal motion for challenging coverage.Source: §VI · p. 8

Acceptance rate

  • 96.7% versus 64.3% acceptance rates were achieved by HuMI and TWIST2, respectively.An acceptable episode had to complete the task and be replayable end-to-end by a policy trained on the collected trajectories.Source: §VI · p. 8

Efficiency

  • HuMI reduced average time per acceptable episode to 30.0% of TWIST2’s time.This compares the time required to obtain episodes usable for downstream humanoid policy learning.Source: §VI · p. 8

Whole-body motion coverage

  • HuMI collected 50 marriage-proposal demonstrations in 15 minutes with 100% acceptance, whereas TWIST2 produced none.The target required deep kneeling, which the teleoperated humanoid could not reliably realize while maintaining stability.Source: §VI · p. 8

Scope

  • HuMI is positioned as the first robot-free data-collection system specifically for humanoid whole-body manipulation.Prior robot-free systems typically rely only on end-effectors and lack complex whole-body coordination.Source: §VII · p. 8, §VIII · p. 8

APPENDIX

HuMI combines portable sensing with real-time inverse-kinematics preview to collect whole-body demonstrations without a robot. Its hardware tracks hand, waist, and foot motion while the interface enforces feasible, collision-free configurations.

Hardware

  • HuMI uses portable, base-station-free hardware intended for demonstration collection across diverse environments.The setup is contrasted with outside-in motion-capture systems that lack portability.Source: 1) Hardware Components · p. 12
  • The data-collection system uses two sensorized grippers and five wearable 6DoF trackers.Trackers are attached to the grippers, waist, and feet, while gripper cameras record wrist-view RGB observations.Source: 1) Hardware Components · p. 12

IK interface

  • A real-time IK preview helps demonstrators adapt motions to be kinematically feasible and task-compliant.Recorded world-frame SE(3) transforms are used as IK targets, with scaling applied only to pelvis height.Source: 2) IK Interface · p. 12
  • The IK problem tracks five keypoints while avoiding self-collisions and maintaining a natural posture.Mink solves the problem in real time, and Viser visualizes the resulting robot configuration.Source: 2) IK Interface · p. 12

Data processing

  • Recorded demonstrations pair gripper-camera MP4 videos with time-stamped SE(3) trajectories from the five trackers.Processing begins by synchronizing tracker and video data using timestamps and angular-velocity cross-correlation.Source: 2) IK Interface · p. 12

4) Data Collection Protocol:

HuMI’s collection protocol maps each new scene, calibrates and synchronizes sensing, and records repeated demonstrations while the user follows real-time IK feedback. The resulting data supports hierarchical control on the Unitree G1.

Data Collection Protocol

  • Each new scene begins with a VIVE tracking-map setup that typically takes 1–2 minutes.The demonstrator follows VIVE prompts before recording task demonstrations.Source: 4) Data Collection Protocol · p. 12
  • Demonstration recording uses optional gripper calibration, GoPro timestamp synchronization, and voice-controlled video capture.Episode boundaries are defined by the start and stop times of recorded videos.Source: 4) Data Collection Protocol · p. 12
  • Demonstrators adapt their motions using the real-time IK preview during recording.This connects the collection protocol to the kinematic-feasibility interface.Source: 4) Data Collection Protocol · p. 13

Experimental Setup

  • The real-world experiments use a Unitree G1 with an external workstation and HTC Vive Ultimate Trackers for global localization.The setup includes custom end-effectors, gripper-mounted visual observations, and hierarchical control.Source: B. Deployment Details · p. 13

Hierarchical Control

  • The high-level policy runs at 5 Hz and publishes desired keypoint trajectories and gripper commands through ZeroMQ.It combines external visual streams with robot proprioception on the workstation.Source: 3) System Architecture · p. 13
  • The low-level whole-body controller runs at 50 Hz and converts target keypoints into joint-position commands.Execution occurs on the robot’s onboard computer through its built-in PD controller, with trackers providing global references.Source: 3) System Architecture · p. 13

C. Additional Experiments

Additional experiments test whether adaptive end-effector rewards and dense whole-body supervision are necessary for stable manipulation. Fixed tight tracking and sparse waypoints substantially degrade task success.

Experiment Design

  • The squat-and-pick-up task tests whether prioritizing end-effector precision compromises whole-body stability.The benchmark requires simultaneous high-precision manipulation and dynamic balance.Source: C. Additional Experiments · p. 13

Fixed End-Effector Reward

  • The fixed-reward baseline enforces a uniformly tight end-effector tracking constraint throughout the motion.It replaces the adaptive end-effector reward with a fixed formulation.Source: C. Additional Experiments · p. 13
  • 85% to 50% was the success-rate change after removing the adaptive end-effector reward.Failures commonly occurred during deep squats through falls or marginally stable poses that prevented subsequent manipulation.Source: C. Additional Experiments · p. 13

D. Generalization Experiments Details

Generalization experiments use seven training environments and objects, then evaluate across four testing environments and six testing objects.

E. Low-Level Controller Training Details

The low-level controller combines privileged-state teacher training with student distillation, whole-body observations, waypoint commands, tracking rewards, and domain randomization.

F. High-Level Policy Training Details

The high-level policy uses visual and proprioceptive observations to predict keypoint and gripper actions, with training data spanning capability and multi-environment tasks.

Related papers

Loading 2602.06643v2…