Source-linked AI summary
HumanPlus: Humanoid Shadowing and Imitation from Humans
Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, Chelsea Finn
TL;DR
Humanoid robots could exploit abundant human motion and skill data, but perception, control, morphology gaps, and limited teleoperation pipelines make this difficult. HumanPlus combines simulation-trained low-level shadowing, single-camera whole-body teleoperation, and egocentric imitation learning, achieving 60-100% success across diverse tasks with up to 40 demonstrations. The authors report that the system enables autonomous humanoid skills including walking, manipulation, typing, and robot greeting.
Problem
Humanoids lack practical, scalable ways to learn motion and autonomous skills from human data because of perception and control complexity, physical gaps, and limited whole-body teleoperation pipelines.
Method
HumanPlus trains a low-level whole-body policy in simulation on human motion data, uses RGB-based shadowing for teleoperation and data collection, then trains egocentric vision skill policies by behavior cloning.
Results
60-100% success rates were achieved on diverse autonomous tasks using up to 40 demonstrations, including wearing a shoe to stand and walk, unloading racks, folding, rearranging, typing, and greeting another robot.
Takeaways & Limitations
The integrated shadowing and imitation-learning system supports whole-body humanoid skill learning from human data across manipulation and locomotion tasks.
Takeaways & Limitations
The system is constrained by fewer robot degrees of freedom than human anatomy, occlusion-sensitive pose estimation, fixed egocentric cameras, and limited treatment of long-horizon navigation.
Abstract
from arXiv · showhide
One of the key arguments for building robots that have similar form factors to human beings is that we can leverage the massive human data for training. Yet, doing so has remained challenging in practice due to the complexities in humanoid perception and control, lingering physical gaps between humanoids and humans in morphologies and actuation, and lack of a data pipeline for humanoids to learn autonomous skills from egocentric vision. In this paper, we introduce a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train a low-level policy in simulation via reinforcement learning using existing 40-hour human motion datasets. This policy transfers to the real world and allows humanoid robots to follow human body and hand motion in real time using only a RGB camera, i.e. shadowing. Through shadowing, human operators can teleoperate humanoids to collect whole-body data for learning different tasks in the real world. Using the data collected, we then perform supervised behavior cloning to train skill policies using egocentric vision, allowing humanoids to complete different tasks autonomously by imitating human skills. We demonstrate the system on our customized 33-DoF 180cm humanoid, autonomously completing tasks such as wearing a shoe to stand up and walk, unloading objects from warehouse racks, folding a sweatshirt, rearranging objects, typing, and greeting another robot with 60-100% success rates using up to 40 demonstrations. Project website: https://humanoid-ai.github.io/
1. Introduction
HumanPlus addresses the difficulty of learning humanoid motion and autonomous skills from human data with a full-stack system combining simulation-trained shadowing, RGB teleoperation, and imitation learning. The system enables a customized humanoid to complete diverse tasks autonomously with 60-100% success using up to 40 demonstrations.
- Human-like morphology offers humanoids access to human motion and skill data for learning tasks designed around human environments, tools, and bodies.
- Humanoid learning from human data remains difficult because of complex perception and control, physical morphology and actuation gaps, and limited whole-body teleoperation pipelines.
- HumanPlus trains a task-agnostic low-level whole-body policy in simulation from 40 hours of AMASS human motion data, then transfers it zero-shot to real hardware.
- A single RGB camera estimates human body and hand motion for real-time shadowing, enabling nearby operators to teleoperate the customized 33-DoF, 180cm humanoid and collect whole-body task data.
- Behavior cloning trains vision-based skill policies from shadowing data, using egocentric RGB vision to predict humanoid body and hand poses.
- The full-stack contribution combines single-camera whole-body shadowing with the Humanoid Shadowing Transformer and the Humanoid Imitation Transformer for binocular perception and high-DoF control.
- 60-100% success rates were achieved across tasks including walking after wearing a shoe, warehouse unloading, folding, rearranging, typing, and robot greeting, using up to 40 demonstrations.
2. Related Work
Related work spans reinforcement learning for humanoid locomotion, teleoperation, and robot learning from human data. HumanPlus differs by integrating single-camera whole-body teleoperation with autonomous imitation learning for humanoids.
- Learning-based reinforcement learning methods have achieved robust humanoid and biped locomotion through randomized simulation training and adaptation.
- Prior teleoperation systems use RGB cameras, VR controllers, exoskeletons, or operation-space interfaces, but differ in body coverage and hardware flexibility.
- HumanPlus provides low-cost whole-body teleoperation using a single RGB camera to control every humanoid joint, integrated with imitation learning.
- Human data supports robot pretraining, visual representation learning, and in-domain imitation learning, including applications to dexterous robotic hands.
3. HumanPlus Hardware
The HumanPlus platform is a Unitree H1-based humanoid with 33 degrees of freedom, dexterous hands, head-mounted RGB cameras, and customized wrist hardware.
- The humanoid has 33 degrees of freedom: two 6-DoF hands, two 1-DoF wrists, and a 19-DoF body.
- Its hardware includes two head-mounted RGB webcams, Inspire-Robots RH56DFX hands, customized wrists, 10N finger forces, 7.5kg arm payloads, and leg torques up to 360Nm.
4. Human Body and Hand Data
HumanPlus combines offline human motion data, real-time body and hand pose estimation, pose retargeting, and simulation randomization to train its shadowing system.
- AMASS provides 40 hours of human motion across more than 11,000 motion sequences for training the Humanoid Shadowing Transformer.
- Human body and hand motions are parameterized with SMPL-X and retargeted to humanoid poses using mappings for finger and wrist motion.
- The model architecture uses decoder-only transformers for both low-level shadowing control and imitation learning.
- Real-time hand pose estimation uses HaMeR with a single RGB camera and runs at 10 fps on an NVIDIA RTX4090 GPU.
5. Shadowing of Human Motion
The Humanoid Shadowing Transformer is a simulation-trained low-level controller that converts human-retargeted target poses into whole-body humanoid motion. Deployed with onboard sensing and a single RGB camera, it supports real-time shadowing and whole-body teleoperation.
- Policy formulation: The low-level policy is formulated as a decoder-only transformer receiving humanoid proprioception and retargeted target poses.Inputs include root state, joint positions and velocities, the last action, target velocities, orientation, and target joint angles.
- Simulation training: PPO trains the policy in simulation using a discounted expected return whose reward matches target poses while saving energy and avoiding foot slipping.The simulated environment randomizes physical parameters for sim-to-real transfer.
- Evaluation: The evaluation compares averaged teleoperation completion time for six participants across two tasks.The supplied table caption identifies the participant and task counts but does not provide the numerical table entries.
- Real-world shadowing: The system estimates human body and hand poses from a single RGB camera, retargets them to humanoid target poses, and deploys the policy zero-shot for real-time shadowing.Humanoid proprioception is measured using onboard sensors including an IMU and joint encoders.
6. Imitation of Human Skills
The paper trains vision-based humanoid skill policies from demonstrations collected through shadowing. Its decoder-only HIT architecture combines egocentric vision, proprioception, and action-chunk prediction for high-DoF imitation.
- Skill learning: Real-world shadowing data are used with supervised behavior cloning to train skill policies from humanoid binocular egocentric RGB vision.The skill policy predicts desired humanoid body and hand poses.
- HIT architecture: Humanoid Imitation Transformer modifies Action Chunking Transformer into a decoder-only architecture for faster inference on limited onboard compute.HIT processes two egocentric RGB image streams, proprioception, and fixed positional embeddings.
- Action prediction: HIT predicts a chunk of 50 target poses from current image features, proprioception, and positional embeddings.Image features are encoded with a pretrained ResNet encoder.
7. Tasks
The task suite tests autonomous imitation and shadowing across dexterous manipulation, whole-body coordination, mobility, balance, and interaction. Autonomous tasks include multi-step object, clothing, typing, greeting, warehouse, and shoe-wearing behaviors.
- Task suite: The paper selects six imitation tasks and five shadowing tasks spanning diverse capabilities and objects relevant to practical applications.The task definitions cover both autonomous imitation and real-time shadowing behaviors.
- Wear a Shoe and Walk: Wear a Shoe and Walk combines bimanual shoe manipulation, lace tying, standing up, and walking.The task contains ten ordered subtasks and each demonstration lasts 1250 steps or 50 seconds.
- Warehouse: The Warehouse task tests whole-body manipulation by retrieving a paint spray, squatting, placing it on a cart, and standing up.Each demonstration lasts 500 steps or 20 seconds, with standing location randomized along a 10cm line.
- Fold Clothes: Fold Clothes requires balanced bimanual manipulation of sweatshirt sleeves and bottom fabric while maintaining an upright pose.The root yaw, garment position, and garment rotation are randomized.
- Rearrange Objects: Rearrange Objects requires selecting an appropriate hand to pick and place varied soft objects into a basket while maintaining balance.Four soft-object types and left-or-right placement relative to the basket are randomized.
- Shadowing tasks: Shadowing tasks include boxing, cabinet opening, tossing, piano playing, table tennis, and typing “Hello World.”These tasks showcase mobility, stability, fast diverse motions, and manipulation of heavy objects.
8. Experiments on Shadowing
Experiments compare the shadowing system with alternative teleoperation methods and evaluate low-level robustness. The system is reported as the only compared method supporting whole-body teleoperation, while its controller recovers faster and enables additional whole-body skills.
- Imitation evaluation: Across six imitation tasks, Table 5 compares HIT with monocular HIT, ACT, and open-loop trajectory replay using success rates.The table caption reports that the full HIT system outperforms the alternatives overall.
- Teleoperation comparisons: The shadowing system is compared with Kinesthetic Teaching, ALOHA, and Meta Quest using teleoperation efficiency and whole-body-control criteria.The baselines do not support whole-body control and require at least two operators for hand pose estimation.
- Teleoperation comparisons: The system uses a single RGB camera, while ALOHA and Meta Quest have hardware or control constraints and Kinesthetic Teaching can destabilize the humanoid.The comparison includes fixed hardware, arm singularities, Cartesian pose mismatch, and external-force issues.
- User study: The system is reported to have the lowest completion time, highest stable-standing success rate, and unique ability to solve the lower-object rearrangement task through whole-body teleoperation.The user study involved six participants performing Rearrange Objects and Rearrange Lower Objects.
- Robustness evaluation: The low-level policy withstands larger disturbance forces than the H1 Default controller and recovers in one or two steps and below 3 seconds.The manufacturer controller takes several steps and up to 20 seconds to stabilize.
- Robustness evaluation: The policy enables whole-body skills unavailable to the default controller, including squatting, high jumping, and standing from a chair.These skills are reported alongside the recovery comparison.
9. Experiments on Imitation
The experiments compare Humanoid Imitation Transformer with three baselines across six tasks, using success rates defined per sub-task. HIT achieves higher success rates across all tasks and uniquely completes Wear a Shoe and Walk.
- Experimental setup: HIT is compared with Monocular, ACT, and open-loop trajectory replay across six manipulation and locomotion tasks.The tasks include Fold Clothes, Rearrange Objects, Type “AI”, Two-Robot Greeting, Warehouse, and Wear a Shoes and Walk.
- Evaluation: Success rate is computed as successful attempts divided by total attempts for each sub-task.For sequential tasks, attempts at a sub-task depend on successful completion of the preceding sub-task.
- Results: HIT achieves higher success rates than the other baselines across all tasks.The comparison includes monocular HIT, ACT, and open-loop trajectory replay.
- Results: 60% success with 40 demonstrations is achieved on Wear a Shoe and Walk, while every other method fails.HIT is the only evaluated method that solves this task.
- Analysis: Binocular perception and reactive visual feedback distinguish HIT from baselines that lack depth information or overfit to proprioception.Monocular fails Wear a Shoe and Walk, ACT fails Wear a Shoe and Walk and Typing “AI”, and open-loop replay fails tasks requiring reactive control.
10. Conclusion, Limitations and Future Directions
HumanPlus is limited by hardware differences from human anatomy, fixed sensing and retargeting, and pose-estimation failures under heavy occlusion. The current scope emphasizes manipulation and short locomotion tasks rather than long-horizon navigation.
- Hardware limitations: The humanoid’s reduced degrees of freedom restrict agile movements, 6DoF arm control, and some reachable regions during shadowing.The platform has 1-DoF ankles and 5-DoF arms including the wrist.
- Perception limitations: Fixed, inactive head-mounted cameras can cause hands and interactions to fall out of view.This creates a persistent sensing constraint during task execution.
- Retargeting limitations: Fixed human-to-humanoid retargeting omits human joints absent from the hardware, potentially restricting learned motions to a smaller subset.The limitation concerns the diversity of human motions available for learning.
- Perception limitations: Pose estimation performs poorly under large occlusions, limiting the operating regions of human operators.The authors identify improving this limitation as future work.
- Scope and future directions: The work focuses on manipulation and short locomotion tasks, while long-horizon navigation requires more demonstrations and accurate real-world velocity tracking.The stated locomotion examples include squatting, standing up, and walking.