PaperlayerView exact revision on arXiv

Source-linked AI summary

$\lambda_0$: Scaling Open-World Egocentric Human Data for Humanoid Loco-Manipulation

humanoid Team, Chongyang Xu, Jin Chen, Jinhui Ye, Yilun Chen, Yiming Jiang, Zhao Wu, Hu Wei, Lin Qu, Shifeng Zhang, Yisheng He, Yuming Jiang, Zenan Meng, Zheng Wu, Mu Xu, Steven Hoi

arXiv:2610.18950v2cs.ROcs.CV

TL;DR

Humanoid loco-manipulation needs demonstrations that coordinate locomotion, posture, and manipulation, but robot-operated collection is costly and egocentric hand trajectories omit whole-body coordination. The paper introduces HUMANVERSE and an alignment pipeline, then trains λ0 with robot-aligned human supervision. Increasing whole-body human data improves task progress, and λ0 achieves strong reported success on SIMPLE and real-world loco-manipulation tasks.

  • Problem

    Robot-operated demonstrations make broad task and environment coverage costly, while egocentric wrist and hand trajectories omit locomotion and postural changes needed for whole-body interaction.

    Source: §1 · p. 3

  • Method

    The paper builds HUMANVERSE, a 500-hour synchronized egocentric video and body-and-hand-motion dataset, aligns human observations and actions to humanoid control, and uses the data for λ0 training.

    Source: §1 · p. 3

  • Results

    90.0% success on SIMPLE and 62.5% across four real-world loco-manipulation tasks; increasing whole-body human data raises real-world progress from 59.9% without whole-body mid-training to 80.4% with the full corpus.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support egocentric whole-body human experience as a practical data source for humanoid learning that bridges diverse human interaction videos and limited, costly robot demonstrations.

    Source: §6 · p. 14

    HUMANVERSE’s 500-hour duration covers only its synchronized egocentric whole-body corpus, excluding external hand–object datasets and robot demonstrations.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Humanoid loco-manipulation requires coordinated whole-body interaction with the physical world, yet collecting robot demonstrations at scale remains costly. Human demonstrations offer a scalable alternative, but existing egocentric human supervision often omits locomotion and posture, while embodiment differences and controller dynamics hinder accurate end-effector execution. We introduce HumanVerse, a 500-hour dataset of open-world human loco-manipulation pairing egocentric video with synchronized body and hand motion. Our human-to-humanoid pipeline aligns camera viewpoints and arm appearance, and refines motion references and controller inputs offline to reduce end-effector errors while preserving whole-body coordination. We validate the pipeline through zero-shot transfer to Unitree G1 on representative whole-body manipulation and long-range loco-manipulation tasks, without target-task robot demonstrations. Building on this pipeline, we develop $\lambda_0$, a humanoid foundation model for whole-body loco-manipulation, with robot-aligned human demonstrations providing supervision for mid-training. Increasing whole-body human data consistently improves downstream task progress. $\lambda_0$ achieves 90.0% success on SIMPLE and 62.5% across real-world loco-manipulation tasks. We expect this study to guide scalable whole-body human data collection toward humanoid foundation models for richer interaction with the physical world. Homepage: https://lambdahumanoid.github.io/

1 Introduction

The paper addresses the need for scalable humanoid demonstrations that include coordinated locomotion, posture, and manipulation, and asks whether egocentric whole-body human experience can transfer to humanoids. It introduces HUMANVERSE, a human-to-humanoid alignment pipeline, and λ0; increasing whole-body human data improves downstream loco-manipulation.

1 Introduction

  • The paper motivates whole-body supervision because wrist and hand trajectories alone omit the locomotion and postural changes involved in humanoid interaction.It also notes that retargeting and tracking errors can compromise grasping and contact.Source: §1 · p. 3
  • 90.0% success on SIMPLE and 62.5% across four real-world loco-manipulation tasks; λ0 improves over the strongest evaluated baselines by 3.9% and 22.5%, respectively.Increasing whole-body human data also raises real-world task progress monotonically from 59.9% without whole-body mid-training to 80.4% with the full corpus.Source: §1 · p. 3
  • The human-to-humanoid pipeline aligns observations and actions to the downstream humanoid and transfers whole-body skills to Unitree G1 without target-task robot demonstrations.Across seven representative tasks, mean progress is 51.7% for coordinated whole-body manipulation and 76.2% for long-range loco-manipulation.Source: §1 · p. 3
  • HUMANVERSE is a 500-hour dataset pairing egocentric video with synchronized body and hand motion across open-world scenes and tasks.It is built from human demonstrations captured without a robot.Source: §1 · p. 3
  • λ0 progressively adapts egocentric human interaction experience to humanoid control, using HUMANVERSE for mid-training between hand-motion pretraining and robot post-training.Robot post-training adapts learned behaviors to the target humanoid.Source: §1 · p. 3

2 Related Work

Prior work addresses scalable robot data collection, learning from human video, and humanoid whole-body control through distinct approaches. This paper positions synchronized body and hand motion, aligned to G1’s action interface, as supervision for bipedal loco-manipulation.

2.1 Generalist Robot Policies

  • Robot datasets and portable interfaces support generalist policies, but robot collection depends on hardware and repeated resets, while portable interfaces have limited action spaces.UMI on Legs extends robot-free hand-held demonstrations to quadruped manipulation using a task-level policy and simulation-trained whole-body controller.Source: §2.1 · p. 4

2.2 Learning from Human Videos

  • Human-video research connects observations to actions through annotations, motion reconstruction, imitation, and aligned human–robot state–action spaces.The paper distinguishes its approach by using synchronized body and hand motion as action supervision for bipedal humanoid loco-manipulation and measuring its benefit while holding robot demonstrations fixed.Source: §2.2 · p. 4

2.3 Humanoid Whole-Body Control

  • Humanoid whole-body control spans retargeting, human-experience policies, simulation, and integrated systems; this pipeline addresses geometric mismatch and controller execution error in motion references.Figure 2 depicts the wearable capture setup, task and motion distributions, and paired egocentric video and whole-body motions; Supplementary Table 5 compares human-data interfaces, training strategies, and scaling studies.Source: §2.3 · pp. 4–5, Figure 2 · p. 5

3 HumanVerse: Scaling Egocentric Whole-Body Data Collection in the Open World

HUMANVERSE is a 500-hour corpus of open-world egocentric human loco-manipulation paired with synchronized body and hand motion. The paper describes a robot-alignment pipeline and an λ0 training process that uses human-derived supervision alongside robot data.

3 HumanVerse: Scaling Egocentric Whole-Body Data Collection in the Open World

  • HUMANVERSE contains 500 hours, 32,993 episodes, 87 million pose frames, and 827 task types of open-world human loco-manipulation.The 500-hour total describes the recorded corpus before conversion and validity filtering.Source: §3 · p. 5

3.2 Dataset Scope and Coverage

  • The corpus spans 827 task types and varied settings, with activities including cleaning, cooking, personal care, laundry, and large-object transport.The 60-frame gallery illustrates qualitative diversity rather than estimating dataset-wide task frequencies.Source: §3 · p. 5, §3.2 · p. 6

3.1 Data Collection Setup and Protocol

  • Collectors capture egocentric video and body motion with a robot-matched camera setup, reconstruct hand motion from video, and collect without an on-site robot.Activity-level prompts preserve variation in walking paths, posture, objects, grasp strategies, and intermediate corrections.Source: §3.1 · pp. 5–6

3.3 Human Motion Reconstruction and Annotation

  • Video and body streams are synchronized to a common timeline, while wearable tracking supplies body kinematics and video-based reconstruction supplies hand trajectories.Motion records retain coordinate transforms, visibility, estimator confidence, and separate body and hand validity masks.Source: §3.3 · p. 6

3.4 Human-to-Humanoid Alignment Pipeline

  • 17.03 to 3.83 cm position error and 36.06° to 10.85° orientation error follow validation of robot-reference and dynamics-aware alignment on 50 tasks.The three offline stages adjust the human reference, correct the robot reference with constrained whole-body IK, and optimize motion tokens using rollout feedback; end-effector targets stay fixed.Source: Figure 6 · p. 7, §3.4 · p. 8
  • Observation alignment matches camera settings and capture height, then composites rendered G1 arms over inpainted human-arm regions while preserving the scene.Training uses crops, rotations, and photometric jitter to improve robustness to remaining viewpoint differences.Source: §3.4 · pp. 7–8

4.1 Policy Architecture and Action Interfaces

  • λ0 combines a vision–language backbone with an action expert, taking egocentric images, language, and state in Stages II and III and predicting action chunks.For G1, each action step comprises a 64-dimensional SONIC motion latent and two 6-dimensional Revo2 hand commands.Source: §4.1 · p. 8

4.2 Three-Stage Training

  • Training progresses from hand-state prediction on egocentric datasets, to HUMANVERSE whole-body mid-training, then robot-demonstration post-training for executable control.Stage I uses diverse hand-pose supervision; Stage II converts HUMANVERSE demonstrations into the G1 state–action format.Source: §4.2 · p. 9

5 Experiments on Humanoid: Alignment, Scale, and Transfer

The experiments evaluate aligned human supervision, λ0’s benchmark performance, the effect of human-data scale and training stages, and transfer to robot-unseen objects. Across these settings, larger whole-body human-data fractions correspond to higher task progress, and λ0 performs strongly on simulation and real-world tasks.

5.1 Effectiveness and Efficiency of Robot-Aligned Human Data

  • Aligned human demonstrations yield mean progress of 51.7% across four whole-body tasks and 76.2% across three long-range tasks, or 62.2% across all seven.These task-specific human demonstrations are evaluated without target-task robot data; long-range results average seen and unseen target locations.Source: Figure 8 · p. 10, §5.1 · pp. 10–11
  • Human collection uses 33.30 person-seconds per demonstration versus 140.85 for robot collection, while policies trained on 200 human demonstrations score 53.3% versus 60.0% from 50 robot demonstrations.The labor comparison averages Close Laptop and Pull Curtain and includes capture, amortized setup, calibration, troubleshooting, and resets; the measurements use different accounting conventions.Source: Figure 9 · p. 10, §5.1 · p. 11

5.3 Human-Data Scale and Training-Stage Ablations

  • From 5% to 100% data, minimum validation loss falls by 35.5% on representative and 40.6% on clean holdouts.The single-run log-linear fits describe the measured range, without establishing cross-seed reproducibility or extrapolation.Source: Figure 11 · p. 12, §5.3 · p. 13
  • At 5% and 10% data, representative validation loss finishes 24.7% and 19.0% above its minima, whereas at 50% and 100% final losses remain within 1.4% and 0.9% of minima on representative and clean holdouts.The 25% run is intermediate, finishing 6.4–6.5% above its minima; smaller subsets benefit from earlier checkpoint selection.Source: §5.3 · pp. 12–13, Figure 11 · p. 12

5.2 Performance on Simulation and Real-World Tasks

  • λ0 reaches 90.0% overall success on SIMPLE, beating StarVLA by 3.9% and Ψ0 by 4.4%, and 62.5% real-world success, 22.5% above the strongest external baseline.It also achieves 80.4% weighted real-world progress, 25.8% above the strongest external baseline.Source: §5.2 · pp. 11–12, Table 2 · p. 12, Table 1 · p. 12

5.3 Human-Data Scale and Training-Stage Ablations

  • Real-world task progress rises monotonically from 59.9% without Stage-II data to 80.4% with the full corpus, a 20.5% gain.The evaluated fractions are 0%, 5%, 10%, 25%, 50%, and 100%, with Stage-I initialization, robot demonstrations, and other controls fixed.Source: Figure 12 · p. 13, §5.3 · p. 13
  • Removing Stage II lowers real-world success by 40.0%, versus 12.5% when removing Stage I, under fixed robot supervision.Full-recipe real-world success/progress is 62.5%/80.4%, compared with 22.5%/59.9% without Stage II and 50.0%/75.9% without Stage I.Source: Table 3 · p. 13, §5.3 · p. 13

5.4 Transfer to Robot-Unseen Object Instances

  • On robot-unseen object instances, λ0 reaches 55.8% Human-guided progress versus 46.5% for π0.5, and improves 7.4% over its Robot-only variant.The comparison measures combined human-data training; it does not isolate the contribution of human demonstrations of object B from other human-data stages.Source: §5.4 · p. 13, Figure 13 · p. 14

6 Conclusion

The paper presents λ0, a humanoid loco-manipulation foundation model, and HUMANVERSE, an egocentric whole-body human dataset. Its conclusion is that scaling robot-aligned whole-body human supervision improves downstream loco-manipulation.

6 Conclusion

  • HUMANVERSE provides open-world human interaction experience without robot-operated collection, while human-to-humanoid alignment makes that experience usable for coordinated body and hand actions.Whole-body human mid-training bridges diverse human interaction videos and limited, costly robot demonstrations before adaptation to the target humanoid.Source: §6 · p. 14
  • λ0 progressively adapts human interaction experience to humanoid control, extending hand-motion pretraining to whole-body control before target-humanoid adaptation.The conclusion describes robot post-training as the adaptation stage.Source: §6 · p. 14
  • The reported results support egocentric whole-body human experience as a practical data source for humanoid learning, with scaling improving downstream loco-manipulation.Source: §6 · p. 14

7 Contributions

The supplied material lists contributor categories and describes supplementary resources and related-work comparison tables. It also records qualifications on dataset and comparison claims.

7 Contributions

  • The contributions list separates core contributors, contributors, and supervisors, with authors listed alphabetically by first name within each category.Source: §7 · p. 14

Appendix

  • The supplementary material covers data quality and split validation, training formalism and implementation, baseline implementations, and validation-scaling analysis.The project website and overview video accompany the paper.Source: Appendix · p. 20

A Related-Work Comparisons

  • The related-work tables compare human-data resources and documented training interfaces, plus problem-level experimental questions rather than reported scores.The dataset comparison notes that wearable-device body kinematics are not optical motion-capture ground truth and hand geometry is reconstructed offline.Source: Table 5 · p. 20, Table 4 · p. 20, A Related-Work Comparisons · p. 20

B Action Interfaces and Training Details

The section defines hand-state tokens and G1-compatible state–action interfaces for human-derived supervision and robot control.

B.1 State–Action Definitions

  • The hand-state representation includes wrist pose and 20 non-wrist joints in a wrist-local, palm-aligned, scale-normalized frame.The wrist rotation uses a continuous 6D representation.Source: §B.1 · p. 21
  • Stage I discretizes the current and future hand states into 256 bins per normalized scalar and predicts future hand-state tokens without an action expert or continuous action head.The bin indices are serialized as text and passed through the vision–language tokenizer.Source: §B.1 · p. 21
  • Converted human data use a frozen SONIC motion token, G1 body-joint positions and projected gravity, plus six normalized Revo2 commands per hand.The G1-compatible interface combines closed-loop free-base rollout data with hand retargeting.Source: §B.1 · p. 21
  • Human-derived mid-training and native G1 post-training use the same state–action format, while other embodiments retain their native state and actuator blocks.Domain identifiers and separate input/output projections distinguish the human-derived and native G1 data.Source: §B.1 · p. 21

B.2 Model Architecture

The model uses a shared vision–language backbone and action expert, with flow matching for continuous action prediction in Stages II and III.

B.3 Model Architecture

  • The Qwen3.5-2B backbone supplies projected features to each action-expert layer, while embodiment-conditioned soft prompts use 32 prompt tokens and a learned embodiment bias.Domain-specific input projections encode noisy actions and noise embeddings; output projections predict action velocities.Source: §B.2 · p. 21
  • Stages II and III use flow matching with squared velocity error over valid scalar targets, while Stage I predicts discrete hand-state tokens within the backbone.Training excludes missing annotations, padding, and action prefixes from the loss; RTC prefixes are noise-free and excluded from loss.Source: §B.2 · p. 21

B.3 Training Configuration

The 2B model uses an EMA-linked three-stage training schedule, with Stage I sampling seven datasets by usable frame count.

B.3 Training Configuration

  • The Stage-I backbone EMA initializes Stage II, and the Stage-II policy EMA initializes Stage III, with EMA decay 0.999 after each optimizer step.Table 6 summarizes the default 2B schedule, and Stages II and III predict 50-step action chunks.Source: §B.3 · p. 21, Table 6 · p. 21
  • Stage I uses batch size 1024, learning rate 10^-5, a 0.1 backbone learning-rate multiplier, and square-root sampling across seven datasets.VITRA-1M additionally indexes activity from Ego4D, EPIC-KITCHENS, Ego-Exo4D, and Something-Something V2.Source: §B.3 · p. 21

C Data Quality and Scaling Protocol

The protocol checks annotation and split quality, then scales HUMANVERSE data under fixed training and evaluation conditions while adapting four baselines to the G1 interface.

C Data Quality and Scaling Protocol

  • Data-quality checks verify timestamps, stream overlap, coordinate conventions, finite values, hand reprojection, and synchronized visual consistency; invalid targets are masked and failures retained.Splits are by recording session before temporal-window extraction, with task-specific human demonstrations kept separate from general-purpose mid-training data.Source: C Data Quality and Scaling Protocol · p. 22
  • Clean selects held-out sessions meeting upper-quartile validity thresholds for both SONIC whole-body and bilateral-hand annotations; Clean and Representative each contain 44 mutually disjoint sessions.The sets contain 1,008 and 513 episodes, respectively, and remain fixed across data fractions.Source: C Data Quality and Scaling Protocol · p. 22
  • The scaling comparison fixes the Stage-I checkpoint, robot post-training set, Stage-III update budget, validation sets, and evaluation manifest across nested HUMANVERSE fractions.Nonzero fractions are sampled by episode and stratified by activity and recording session; the primary compute-matched comparison uses equal Stage-II updates.Source: C Data Quality and Scaling Protocol · p. 22
  • Each scaling fraction has one run without across-seed error bars, and fitted minima describe only the observed data range rather than establishing reproducibility or extrapolation accuracy.Minimum loss is selected independently for each holdout over 0–100k updates, and closed-loop robot performance is evaluated separately from offline prediction metrics.Source: C Data Quality and Scaling Protocol · p. 22

D.1 Baseline Implementations

  • Four baselines are adapted to the G1 interface using absolute actions at 50 Hz and a 50-frame future buffer, with dataset-specific training-statistic normalization.GR00T N1.6 uses its official Eagle backbone and 128-dimensional action expert; Ψ0 uses Qwen3-VL-2B and a 30-frame SONIC chunk; π0.5 uses its official pretrained parameters; StarVLA uses Qwen3-VL-4B with a 50-step horizon.Source: §D.1 · pp. 22–23

D.2 Simulation Evaluation

The SIMPLE evaluation reports 162 successes in 180 trials across six tasks, but summarizes the best recorded configuration per task rather than a shared configuration.

D.2 Simulation Evaluation

  • Each of the six tasks was evaluated with ten rollouts at each of Levels 0–2, and success rates were averaged equally across tasks and levels.λ0 used single-task training; StarVLA and λ0 results came from the authors’ evaluations, while public baselines retained their source protocols.Source: §D.2 · p. 23
  • 162 successes in 180 trials were recorded across the six SIMPLE tasks, using task-wise best recorded configurations.These results should not be interpreted as a fixed-configuration comparison.Source: §D.2 · p. 23
  • Public baseline counts follow the official leaderboard accessed September 13, 2026, including updated ACT results.Mobile P&P denotes locomotion pick between tables.Source: §D.2 · p. 23

D.3 Real-World Evaluation and Scoring

Real-world evaluations use controlled trial conditions, stage-based progress scores, and a separate three-second terminal-success criterion.

D.3 Real-World Evaluation and Scoring

  • Policies were tested on ten fixed initial configurations per task with shared objects, targets, instructions, timeouts, and success criteria.Configuration order was randomized, and manual adjudicators were blinded to model identity; results report success counts and mean progress averaged equally across tasks, without seed-based confidence intervals.Source: §D.3 · p. 23
  • Progress sums completed-stage points to 100 per task, while terminal success separately requires the task-specific condition to hold for three seconds.Stages are scored independently of preceding stages with identical weights across methods and data scales; the operational definitions are listed in Table 7.Source: §D.3 · p. 23, Table 7 · p. 24
  • Human-guided and Robot-only evaluations share identical test instances, scenes, targets, initial configurations, instructions, and success criteria.Figure 13 reports mean weighted progress over ten trials per task, method, and condition across four tasks, totaling 600 trials.Source: §D.3 · p. 24

Related papers

Loading 2610.18950v2…