Source-linked AI summary
Humanoid Policy ~ Human Policy
Ri-Zhao Qiu, Shiqi Yang, Xuxin Cheng, Chaitanya Chawla, Jialong Li, Tairan He, Ge Yan, David J. Yoon, Ryan Hoque, Lars Paulsen, Ge Yang, Jian Zhang, Sha Yi, Guanya Shi, Xiaolong Wang
TL;DR
Robot-only manipulation training requires costly, difficult-to-scale teleoperation data. The paper introduces PH2D and HAT to co-train humans and humanoids in a unified action space, reporting improved robustness and generalization compared with robot-only training. Human data also offers more efficient collection, while current limitations include simple policy architecture and possible hand-tracking failures under occlusion.
Problem
Robot demonstrations are labor-intensive and expensive to collect, limiting scalable training for robust and generalizable humanoid manipulation policies.
Method
The paper collects task-oriented egocentric human data in PH2D and trains HAT with a unified human-humanoid state-action space and differentiable retargeting to robot actions.
Results
HAT trained with mixed human-humanoid data shows improved generalization and robustness compared with training using only real-robot data.
Takeaways & Limitations
Egocentric human demonstrations can serve as cross-embodiment training data for humanoid manipulation when aligned with robot data.
Takeaways & Limitations
The current policy uses a relatively simple architecture, hand tracking can fail under heavy occlusion, and evaluations are limited to current robot morphologies.
Abstract
from arXiv · showhide
Training manipulation policies for humanoid robots with diverse data enhances their robustness and generalization across tasks and platforms. However, learning solely from robot demonstrations is labor-intensive, requiring expensive tele-operated data collection which is difficult to scale. This paper investigates a more scalable data source, egocentric human demonstrations, to serve as cross-embodiment training data for robot learning. We mitigate the embodiment gap between humanoids and humans from both the data and modeling perspectives. We collect an egocentric task-oriented dataset (PH2D) that is directly aligned with humanoid manipulation demonstrations. We then train a human-humanoid behavior policy, which we term Human Action Transformer (HAT). The state-action space of HAT is unified for both humans and humanoid robots and can be differentiably retargeted to robot actions. Co-trained with smaller-scale robot data, HAT directly models humanoid robots and humans as different embodiments without additional supervision. We show that human data improves both generalization and robustness of HAT with significantly better data collection efficiency. Code and data: https://human-as-robot.github.io/
1 Introduction
The paper addresses the cost and scalability limits of robot demonstrations by using task-oriented egocentric human data for cross-human-humanoid policy training. It introduces PH2D and HAT, whose aligned representations support improved robustness and generalization.
- Motivation: Robot demonstration collection is expensive and time-consuming, motivating egocentric human demonstrations as a scalable training source.Human demonstrations can be collected efficiently, in varied settings, without a robot.
- Method: HAT unifies human and humanoid state-action spaces and converts human actions to robot actions through inverse kinematics and hand retargeting.The policy predicts future hand-finger trajectories in a unified human-centric representation.
- Results: Co-training with diverse human data improves robustness and generalization, including performance in spatial and background conditions unseen in robot data.The paper validates these findings through real-robot evaluations and ablation studies.
- Dataset: PH2D is a large egocentric, task-oriented human-humanoid dataset with accurate hand and wrist poses for modeling human behavior.It covers manipulation tasks, diverse objects and scenes, and includes language annotations.
2 Related Work
Related work uses human videos and cross-embodiment data to improve robot learning, but transferring human actions remains difficult because embodiments, actuation, and control dynamics differ. This paper emphasizes direct human-action modeling and avoids requiring robot hardware for human-data collection.
- Learning from Human Videos: Human-video approaches align either observations or actions, with action alignment addressing differences in embodiment, actuation, and control dynamics.Methods include trajectory, flow, object-centric, masking, and hand-tracking approaches.
- Learning from Human Videos: Robot policies trained from real-robot data still face limited generalization across tasks, environments, and platforms because real-world collection is constrained.This motivates using the large supply of human video data.
- Action Alignment: The paper differs from HumanPlus by collecting human data without robot hardware and directly using human hand poses as policy states and actions.HumanPlus performs retargeting during teleoperation, which still requires robots.
- Cross-Embodiment: Cross-embodiment pre-training can improve adaptability and generalization, while end-to-end approaches aim to reduce compounding perceptual errors.Some end-to-end methods have also produced behaviors such as retrying.
- Concurrent Work: Concurrent egocentric approaches differ in sensing and alignment requirements, including DexCap’s 3D inputs and EgoMimic’s strict visual sensor alignment.These distinctions position the paper’s approach among wearable-device methods for cross-embodiment learning.
3 Method
The method uses scalable, task-oriented egocentric human demonstrations and a unified human-humanoid policy to augment robot data. PH2D addresses data-collection and domain gaps, while HAT retargets human-centric predictions to humanoid actions and improves out-of-distribution execution.
- PH2D: Task-oriented Physical Humanoid-Human Data: PH2D provides task-oriented egocentric human demonstrations aligned with robot execution using consumer-grade VR devices and pose supervision.The dataset diversifies tasks and camera sensors while reducing whole-body movement to reduce visual and behavioral domain gaps.
- PH2D: Task-oriented Physical Humanoid-Human Data: PH2D uses adapted VR-device SDKs and multiple camera configurations to collect world-frame 3D head and hand poses without specialized hardware.Configurations include Apple Vision Pro’s built-in camera and a low-cost ZED Mini Stereo setup costing under $700.
- PH2D: Task-oriented Physical Humanoid-Human Data: Human operators perform robot-overlapping tasks under language instructions while proprioceptive and visual inputs are synchronized by closest timestamps.Example tasks include grasping and pouring.
- HAT: Human Action Transformer: The method mitigates action and visual gaps by constraining human movement, slowing human actions during training, and training across varied sensors and end-effector appearances.The slowdown factor is fixed at αslow = 4 for all tasks; the loss emphasizes wrist end-effector positions with λ = 2.
- HAT: Human Action Transformer: HAT models humans and humanoid robots in a unified state-action space, predicting human-centric wrist and fingertip actions that are retargeted to robot poses.The policy is transformer-based and predicts action chunks for multi-step execution.
- HAT: Human Action Transformer: Co-training with human data improves out-of-distribution performance on Humanoid A, with nearly 100% relative improvement across all tasks.The evaluation also ablates embodiment-specific normalization choices.
4 Experiments
Experiments evaluate HAT on two humanoid platforms, in-distribution and out-of-distribution settings, few-shot transfer, data efficiency, and state-action design. Co-training with human and heterogeneous robot data improves O.O.D. generalization and low-data adaptation, while unified inputs and interpolated outputs prevent characteristic failures.
- Experimental setup: Experiments use Humanoid A, a Unitree H1, primarily for data collection, and Humanoid B, a Unitree H1-2 with different arm configurations, for cross-humanoid transfer.
- Experimental setup: Human demonstrations use more diverse backgrounds, object types, object positions, and human-to-table positions than the robot demonstrations.
- Main evaluation: Co-training drastically improves O.O.D. performance, achieving nearly 100% relative improvement in settings unseen in robot data across background, placement, and appearance changes.
- Few-shot transfer: Using only 20 Humanoid B demonstrations, policies co-trained with Humanoid A data, with or without human data, substantially outperform Humanoid B-only baselines across all task settings.
- Few-shot transfer: Co-training Humanoid B, Humanoid A, and human data consistently outperforms isolated Humanoid B training as the number of Humanoid B demonstrations increases, especially in the few-data regime.
- Ablation study: Without interpolating human actions, predicted speeds fluctuate and destabilize rollouts; without unified state inputs, O.O.D. performance becomes significantly worse despite on-par I.D. performance.
5 Conclusions
The paper presents PH2D and HAT as a pipeline for using aligned human behavior in cross-embodiment robot learning. Mixed human-humanoid training yields better generalization and robustness than robot-only training.
- PH2D and HAT enable direct imitation learning from mixed human-humanoid data without training surrogates when human and robot data are aligned.
- The learned policy improves generalization and robustness compared with a counterpart trained only on real-robot data.
6 Limitations
The paper identifies limitations in policy architecture, hand tracking, and evaluated robot morphology. These constraints bound the current scope of language-conditioned learning, motion capture reliability, and platform coverage.
- The current policy uses a relatively simple architecture and does not yet investigate generalization to novel language using the collected language instructions.
- Off-the-shelf VR hand tracking can fail during motions with heavy occlusion because its SDKs were trained mostly for VR applications.
- Current evaluations use robots equipped with dexterous hands, although the method conceptually extends to more robot morphologies.
A More Ablation Study - Data Collection
The ablation examines whole-body movement, collection speed, and practical benefits of human demonstrations. Human data is presented as scalable, while retargeting remains the main source of collection overhead.
- Autonomous Whole-body Movement: Humanoid platforms cannot yet reproduce difficult upper-body movements, so operators are instructed to minimize unconscious shoulder and waist motion during human data collection.
- Efficiency of Data Collection: Human task completion time remains nearly unchanged when operators wear a VR device compared with standard human manipulation.
- Efficiency of Data Collection: Retargeting causes most collection overhead because of latency and the constrained workspace of 7-DoF robotic arms.
- Efficiency of Data Collection: Human demonstrations are safer, require less supervisory labor, and support more varied environments without robot-specific adaptations.
B Normalization of different embodiments.
The analysis finds minor differences between normalization coefficients for human and humanoid state-action vectors, while one approach achieves the highest overall success rate but remains spatially biased.
- Different normalization coefficients produce only minor performance differences for human and humanoid state-action vectors.
- The same normalization approach achieves the highest overall success rate in vertical grasping.Its success distribution is biased toward the grid’s upper-right region.
- Humans’ larger workspace may cause human data to encompass humanoid proprioception as a subset.This yields a relatively smaller distribution for the robot state-action space.
C In-Depth Analysis of Different Types of Generalization
Co-training with diverse human data improves robustness and generalization across backgrounds, object appearances, and placements that are underrepresented or absent in robot demonstrations.
- Background Generalization: Nearly 50% higher success rate is achieved on background generalization with diverse human data than without human data.Human demonstrations included more than five backgrounds, while teleoperation data used only one paper background.
- Object Appearance Generalization: Human data significantly enhances grasping on challenging objects that differ in color and shape from the teleoperation object.The horizontal grasping evaluation records success over 10 attempts per object.
- Object Appearance Generalization: Co-training improves overall performance on novel objects, including objects absent from human training demonstrations.Box 2 and the can are cited as completely novel objects; box 1 appears in human data.
- Object Placement Generalization: Cross-embodiment learning significantly improves grasping across unseen placement cells, doubling the overall success rate.Robot-only data came from two cells, with 50 attempts on the right and 10 on the left; evaluation used a 3×3 grid of 10cm × 10cm cells and 10 trials per cell.
- Object Placement Generalization: The object-placement evaluation contrasts robot-only and human-plus-robot models under intentionally imbalanced teleoperation data.The real-robot data was collected in two cells inside the dashed lines, while human data covered a more diverse range of settings.
D In-Depth Comparison between Humanoid A and Humanoid B configurations
Humanoid A and Humanoid B are morphologically similar but differ substantially in arm configurations, mechanical limits, and wrist articulation, creating challenges for direct policy transfer.
- The comparison focuses on the arm configurations of Humanoid A and Humanoid B and their implications for manipulation.Other body parts were not exclusively explored in this work.
- The platforms have different motor specifications, mechanical limits, and proximal-joint ranges of motion.The compared proximal joints are shoulder pitch, shoulder roll, shoulder yaw, and elbow.
- Humanoid A has one wrist-roll joint, limiting wrist articulation and in-hand manipulation to one rotational degree of freedom.
- Humanoid B has independently actuated wrist pitch, roll, and yaw joints, providing additional degrees of freedom for wrist orientation.