Source-linked AI summary
Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining
Chongyang Xu, Zhao Wu, Jin Chen, Yiming Jiang, Jinhui Ye, Yuming Jiang, Shifeng Zhang, Ziliang Feng, Mu Xu, Yilun Chen, Li Lu, Steven C. H. Hoi
TL;DR
Humanoid loco-manipulation needs scalable supervision that coordinates whole-body movement with hand–object interaction, a gap left by hand-focused video and costly teleoperation. The paper introduces HUMANVERSE-500 and trains λ0 through three stages; it reports higher success rates than evaluated baselines on SIMPLE and four real-world tasks.
Problem
Robot teleoperation for whole-body demonstrations is costly to scale, while wrist and hand trajectories from egocentric videos do not capture the body–hand coordination required for humanoid loco-manipulation.
Source: §1 · p. 2Method
The paper introduces a 500-hour dataset of synchronized egocentric video, body motion, and hand motion, then trains λ0 through three stages with a shared human–robot representation and domain-specific interfaces.
Source: §1 · p. 2Results
λ0 improves success rates over the strongest evaluated baselines by 3.9 percentage points on SIMPLE and 22.5 percentage points on four real-world tasks.
Source: §1 · p. 2Takeaways & Limitations
Experiments report benefits from scaling whole-body human data, improved loco-manipulation performance, and transfer to robot-unseen objects.
Source: §1 · p. 2, §6 · p. 10Evaluation focuses on G1, leaving transfer to full-sized humanoids untested; human–robot alignment also remains imperfect despite camera alignment and motion retargeting.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Humanoid whole-body manipulation has advanced rapidly, enabling policies to coordinate locomotion, posture, bimanual interaction, and dexterous hand movements. Meanwhile, egocentric human videos provide diverse examples of everyday interactions across objects and scenes, offering scalable supervision without robot operation. However, existing supervision from these videos provides limited coverage of whole-body movement and coordination with hand-object interaction, while obtaining such supervision through humanoid teleoperation is also costly and difficult to scale. We therefore explore how human experience can support scalable learning of humanoid loco-manipulation. To support this study, we introduce HumanVerse-500, a 500-hour dataset of diverse human loco-manipulation behaviors in open-world environments, collected with a lightweight wearable system that synchronizes egocentric video with body and hand motion. Building on this dataset, we develop $\lambda_0$, a whole-body humanoid vision-language-action policy, through three-stage training that first learns interaction from diverse egocentric datasets, then coordinates body and hand motion using HumanVerse-500, and finally adapts the policy to downstream tasks and robot embodiments. Across these stages, $\lambda_0$ learns a shared representation space for human experience transfer, while domain-specific interfaces handle differences between human and robot states and actions. We evaluate $\lambda_0$ on SIMPLE and 4 real-world loco-manipulation tasks, achieving state-of-the-art performance, and further analyze its scaling behavior, generalization, and training-stage contributions to understand how human data support downstream whole-body humanoid control. We will release our code, models, and data to support further research.
1 INTRODUCTION
Scalable whole-body humanoid loco-manipulation needs demonstrations that coordinate body movement with object interaction, but robot teleoperation is costly and hand-only video supervision misses this coordination. The paper introduces HUMANVERSE-500 and λ0, reporting improved benchmark performance and benefits from scaling whole-body human data.
1 INTRODUCTION
- Robot teleoperation requires substantial robot time and human effort, while egocentric wrist and hand trajectories do not capture the whole-body coordination required for loco-manipulation.The paper identifies approach, posture adjustment, and maintaining a grasp while carrying an object as examples of that coordination.Source: §1 · p. 2
- λ0 transfers human experience through three-stage training, using a shared representation space and domain-specific interfaces for human and robot states and actions.The stages learn interaction from egocentric datasets, coordinate body and hand motion using HUMANVERSE-500, and adapt to downstream tasks and embodiments.Source: §1 · p. 2
- λ0 improves success rates over the strongest evaluated baselines by 3.9 percentage points on SIMPLE and 22.5 percentage points on four real-world tasks.The paper also reports scaling trends and benefits for generalization and downstream performance.Source: §1 · p. 2
- HUMANVERSE-500 provides 500 hours of open-world human loco-manipulation with synchronized egocentric video, body motion, and hand motion.The dataset spans diverse activities and is collected with a lightweight wearable system.Source: §1 · p. 2
- Whole-body human pretraining shows a consistent scaling trend, with reported benefits for downstream performance and generalization in humanoid loco-manipulation.The introduction also reports that more whole-body human data reduces validation loss and improves real-world task progress across evaluated data scales.Source: §1 · p. 2
2 RELATED WORK
Related work covers generalist policies and data collection, learning robot actions from human videos, and humanoid whole-body control. Across these areas, scalable supervision that captures whole-body coordination remains a challenge.
2.1 GENERALIST ROBOT POLICIES
- Robot datasets and shared-model policies support generalist learning, while teleoperation and portable interfaces seek to reduce collection effort; robot collection remains tied to hardware and resets.Portable interfaces are limited to their supported action spaces.Source: §2.1 · pp. 2–3
2.2 LEARNING FROM HUMAN VIDEOS
- Human-video methods use temporal and geometric annotations, motion reconstruction, learned visual representations, and imitation or alignment to connect observations with robot actions.Human videos provide diverse activities, objects, and environments, but whole-body coordination remains challenging to capture for humanoid learning.Source: §2.2 · p. 3
2.3 HUMANOID WHOLE-BODY CONTROL
- Humanoid whole-body policies build on control methods that coordinate locomotion, posture, and manipulation, while using human experience through latent actions, geometric pretraining, or joint training.Predictive models and integrated whole-body systems offer complementary ways to connect perception, motion, and control.Source: §2.3 · p. 3
3 HUMANVERSE-500: EGOCENTRIC WHOLE-BODY HUMAN DATA
HUMANVERSE-500 pairs egocentric video with synchronized body and hand motion across 500 hours and 827 task types. The section describes its coverage and reconstruction pipeline, human-to-robot retargeting, and λ0’s training and deployment.
3 HUMANVERSE-500: EGOCENTRIC WHOLE-BODY HUMAN DATA
- HUMANVERSE-500 contains 500 hours, 32,993 episodes, and 87 million pose frames across 827 task types, pairing first-person video with synchronized body and hand motion.The dataset spans diverse objects and scenes.Source: §3 · p. 3
3.1 DATA COVERAGE
- Activities across indoor and outdoor settings combine locomotion, posture changes, and hand–object interaction, including walking while sustaining a grasp during transport.Continuous recordings capture transitions between interactions and body–hand coordination throughout tasks.Source: §3.1 · p. 4
3.2 WHOLE-BODY MOTION RECONSTRUCTION
- Whole-body motion combines wearable-tracked 24-joint body kinematics with video-reconstructed 21-joint motion for each hand, aligned to video frames by timestamps and coordinate transforms.Validity masks preserve usable body supervision when hands are missing or occluded.Source: §3.2 · p. 4
3.3 HUMAN-TO-ROBOT RETARGETING
- For G1 training data, a frozen SONIC encoder and controller produce robot body states from human motion, while reconstructed hand motion is retargeted to bilateral BrainCo Revo 2 commands.Paired records share egocentric video and timestamps while retaining distinct body and hand representations.Source: §3.3 · p. 5
4.1 POLICY ARCHITECTURE AND ACTION INTERFACES
- λ0 shares its vision–language backbone and action expert across stages, while action-space-specific input encoders and output decoders accommodate differing state–action formats.The policy conditions on an egocentric image, language instruction, and current state to predict a 50-step action chunk.Source: §4.1 · p. 5
4.2 THREE-STAGE TRAINING
- Training progresses from egocentric hand-motion pretraining through HUMANVERSE-500 whole-body mid-training to robot-demonstration post-training for downstream executable control.Stage II uses only human data and freezes the Stage-I hand head; robot demonstrations enter in Stage III.Source: §4.2 · pp. 5–6
4.3 TRAINING OBJECTIVE AND DEPLOYMENT
- Training minimizes masked velocity-field error over valid non-prefix entries; at inference, the action chunk is denoised from Gaussian noise in 10 Euler steps, and G1 motion and hand commands go to separate controllers.The predicted 64-dimensional motion latent goes to the SONIC controller, while two 6-dimensional hand commands go to the Revo2 hands.Source: §4.3 · p. 6
5 EXPERIMENTS
The experiments assess humanoid loco-manipulation, human-data scaling, training-stage contributions, and transfer to robot-unseen objects. Across SIMPLE and real-world tasks, λ0 performs strongly, while larger amounts of whole-body human data improve validation loss and task progress.
5.1 EVALUATION SETUP AND METRICS
- Evaluation covers six SIMPLE tasks at Levels 0–2 and four real-world tasks, reporting both task success and milestone-based progress.Standard trials use robot-seen objects; generalization trials use objects appearing only in human demonstrations.Source: §5 · p. 7, §5.1 · p. 7, Figure 7 · p. 7
5.2 Q1: HOW WELL DOES THE MODEL PERFORM ON HUMANOID LOCO-MANIPULATION?
- λ0 reaches 62.5% average real-world success and 80.4% weighted progress, exceeding the strongest external baselines by 22.5 and 25.8 percentage points, respectively.It records the highest success count on all four tasks, while progress exceeds 75% on each.Source: §5.2 · p. 7, Table 1 · p. 8
- λ0 achieves 90.0% overall SIMPLE success, exceeding StarVLA by 3.9 percentage points and Ψ0 by 4.4 percentage points.It leads StarVLA at each evaluation level, though its advantage narrows to 1.7 points at Level 2.Source: Table 2 · p. 8, §5.2 · p. 8
5.3 Q2: HOW DO HUMAN-DATA SCALE AND THE TRAINING RECIPE AFFECT LEARNING?
- At 5% and 10% data, validation loss rises after early minima, whereas at 50% and 100% final losses remain within 1.4% and 0.9% of minima.The 25% run is intermediate, finishing 6.4–6.5% above minima; smaller subsets benefit from earlier checkpoint selection.Source: §5.3 · pp. 8–9
- From 5% to 100% of Stage-II data, minimum validation loss falls by 35.5% on representative holdouts and 40.6% on clean holdouts.The reported single-run loglinear fits describe the measured range, not cross-seed reproducibility or extrapolation.Source: Figure 8 · p. 8, §5.3 · p. 9
- Removing Stage II reduces real-world success by 40.0 percentage points, compared with 12.5 points without Stage I, under fixed robot supervision.The full recipe reaches 62.5% real-world success versus 22.5% without Stage II.Source: §5.3 · p. 9, Table 3 · p. 9
5.4 Q3: DOES HUMAN-DATA SCALING IMPROVE DOWNSTREAM TRANSFER?
- Real-world task progress increases from 59.9% with no Stage-II data to 80.4% with the full corpus, a 20.5-point gain.Model capacity, robot demonstrations, Stage-III update budget, and evaluation configurations are fixed across the comparison.Source: §5.4 · p. 9, Figure 9 · p. 9
- On robot-unseen objects, λ0 reaches 55.8% Human-guided progress versus 46.5% for π0.5, and improves 7.4 points over its Zero-shot variant.Performance remains below the Seen setting.Source: §5.4 · p. 9, Figure 10 · p. 10
6 CONCLUSION
The paper presents λ0 and HUMANVERSE-500, then reports improved loco-manipulation and unseen-object transfer alongside benefits from scaling whole-body human data. Its evaluation is limited to the G1, leaving full-sized humanoid transfer untested.
6 CONCLUSION
- Experiments show improved loco-manipulation, transfer to robot-unseen objects, and consistent benefits from scaling whole-body human data.The findings support scalable human activity as rich supervision for humanoid control.Source: §6 · p. 10
- HUMANVERSE-500 provides 500 hours of open-world human activities with synchronized egocentric video, body motion, and hand motion; λ0 transfers human experience through three-stage training.The policy is a whole-body humanoid vision–language–action model.Source: §6 · p. 10
- Evaluation focuses on the G1, so transfer to full-sized humanoids remains untested.The authors also identify imperfect human–robot alignment across body proportions, viewpoints, and feasible motions.Source: §6 · p. 10