PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoHumanoid-V2: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation

Jin Chen, Yiming Jiang, Chongyang Xu, Modi Shi, Shijia Peng, Li Chen, Tianyu Li, Mu Xu, Yilun Chen, Steven Hoi, Hongyang Li

arXiv:2609.37181v1cs.ROcs.AI

TL;DR

Direct transfer of coordinated whole-body loco-manipulation from egocentric human demonstrations remains less explored than transfer under decoupled control. EgoHumanoid-V2 combines coarse-to-fine action alignment with visual adaptation; on four real-world tasks, policies trained on human data achieve zero-shot transfer without target-task robot demonstrations.

  • Problem

    Prior egocentric transfer has emphasized decoupled-control loco-manipulation, leaving direct transfer of coordinated whole-body skills less explored; matching end-effector trajectories alone can also miss the coordination required for whole-body tracking.

    Source: §1 · p. 2

  • Method

    EgoHumanoid-V2 uses offline kinematic reference correction followed by dynamics-aware refinement, alongside robot-arm rendering and training-time image augmentation.

    Source: §1 · p. 2

  • Results

    51.67% mean score across four real-world tasks with 200 human demonstrations per task, without target-task robot demonstrations.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support human motion as direct supervision for coordinated humanoid skills on the tested tasks.

    Source: §5 · p. 9

    Alignment can still produce unstable motions, while IK and repeated rollouts limit throughput.

    Source: §5 · p. 9

Abstract

from arXiv · show

Human demonstrations capture diverse scenes and rich whole-body skills without requiring robot teleoperation. Prior work on egocentric transfer has emphasized scene generalization in loco-manipulation under decoupled control, leaving direct transfer of coordinated whole-body skills less explored. We present EgoHumanoid-V2, the first egocentric human-to-humanoid skill transfer framework for coordinated whole-body loco-manipulation. At its core, coarse-to-fine action alignment combines kinematic reference correction with dynamics-aware refinement. It improves end-effector pose accuracy while preserving whole-body coordination. We also use robot-arm rendering and training-time image augmentation to reduce the visual embodiment gap and improve viewpoint robustness. On four real-world tasks, vision-language-action (VLA) policies trained on aligned human data show zero-shot skill transfer without target-task robot demonstrations. Task scores are comparable to those of policies trained on teleoperation data at a lower collection cost. These results support human data as direct skill supervision.

1 INTRODUCTION

Direct transfer of coordinated whole-body loco-manipulation from human demonstrations remains less explored than decoupled-control transfer. EgoHumanoid-V2 aligns actions and visuals to preserve coordination while improving interaction accuracy, and demonstrates zero-shot transfer on four real-world tasks.

Motivation

  • Prior egocentric transfer methods primarily address bimanual manipulation and navigation, rather than coordinated whole-body loco-manipulation skills.Human demonstrations capture diverse scenes and manipulation experience without robot teleoperation.Source: §1 · p. 2
  • Whole-body tracking jointly controls the body, but reference tracking can leave end-effector error, and online IK corrections may disrupt balance by overriding only part of the coordinated output.The paper reports that its online IK implementations produced this risk in implementation and evaluation.Source: §1 · p. 2

Contributions

  • EgoHumanoid-V2 combines offline kinematic reference correction and dynamics-aware refinement to improve manipulation accuracy while preserving whole-body coordination.Robot-arm rendering and training-time image augmentation address the visual embodiment gap and viewpoint robustness.Source: §1 · p. 2

Results

  • On four real-world tasks, policies trained on 200 human demonstrations per task achieve a 51.67% mean score without target-task robot demonstrations.Human collection requires 76.36% less labor per demonstration than teleoperation on two tasks.Source: §1 · p. 2

Contributions

  • The paper contributes a first egocentric human-to-humanoid transfer framework for coordinated whole-body loco-manipulation, with analyses of accuracy, policy scores, and collection cost.Its evaluation reports zero-shot transfer on four real-world tasks.Source: §1 · p. 2

2 RELATED WORK

Prior work spans decoupled and whole-body humanoid control, as well as multiple ways to learn from human demonstrations. EgoHumanoid-V2 positions its contribution as offline action-supervision refinement under a fixed tracker.

2.1 WHOLE-BODY CONTROL FOR LOCO-MANIPULATION

  • Whole-body tracking policies use motion references and full-body feedback for joint control, while decoupled loco-manipulation combines arm targets or IK with separate lower-body control.Execution errors nevertheless remain; prior work addresses retargeting and residual learning in different ways.Source: §2.1 · p. 2

2.2 LEARNING FROM HUMAN DEMONSTRATIONS

  • Human-to-robot learning uses visual representations, predictive pretraining, latent or world action models, captured trajectories, and wearable hand capture, while recent egocentric work includes humanoid transfer under decoupled control.EgoHumanoid-V2 instead refines action supervision offline under a fixed tracker, without task-specific training or fine-tuning.Source: §2.2 · p. 3

3 METHOD

The method converts synchronized egocentric human demonstrations into robot observations and actions for per-task VLA post-training without target-task robot demonstrations. Its alignment pipeline refines actions in stages and adapts training images to robot appearance and viewpoint variation.

3.1 PROBLEM SETUP

  • For each task, synchronized egocentric images, body motion, hand signals, and an instruction are converted into robot observations and actions for VLA post-training.The policy predicts whole-body and hand actions from images, proprioceptive states, and instructions.Source: §3.1 · p. 3

3.2 DATA COLLECTION AND POLICY LEARNING

  • A wearable system captures human demonstrations without robot hardware, and the adapted images and aligned actions form policy-training pairs.A neck-mounted GoPro and PICO system capture synchronized streams at 50 Hz; deployment uses a Unitree G1 with a head-mounted GoPro and Revo2 hands.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3 ACTION ALIGNMENT

  • Action alignment starts from a SONIC rollout of human motion, then corrects reference geometry and iteratively refines motion tokens using execution feedback.Human palm poses define fixed targets in pelvis-centered frames, with target heights adjusted by the human–robot pelvis-height difference.Source: §3.3 · p. 4

3.3.1 KINEMATIC REFERENCE CORRECTION (K)

  • Constrained offline IK moves the initial robot reference closer to human targets while reducing end-effector pose error and limiting motion changes to preserve temporal continuity.The corrected reference is encoded using SONIC’s robot motion encoder; token replay measures residual tracking errors for the fine stage.Source: §3.3.1 · p. 4

3.3.2 DYNAMICS-AWARE REFINEMENT (D)

  • Dynamics-aware refinement uses rollout residuals and an MLP response model to predict execution changes from kinematic changes and iteratively compensate for current end-effector error.Each updated sequence is evaluated in a fresh rollout, and a validated sequence from the optimization history is selected for policy training.Source: §3.3.2 · p. 4, Figure 3 · p. 5

3.4 VISUAL ALIGNMENT

  • Robot-arm rendering addresses appearance differences, while random resized crops, small rotations, and photometric jitter improve robustness to viewpoint variation.At inference, the policy uses robot images directly.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments test human-to-humanoid transfer, demonstration collection trade-offs, action alignment, and visual alignment on real-world tasks, using simulation to assess alignment quality. Human demonstrations transfer arm and foot manipulation, while the alignment stages improve execution accuracy and policy scores.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers four Unitree G1 tasks, scores policies over 20 real-world trials, and uses simulation to compare endpoint error and motion quality.The tasks include hand and foot interactions with posture adjustments or stepping; task scores average success or failure across three sequential subtasks.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 Q1: SKILL TRANSFER FROM HUMAN DEMONSTRATIONS

  • With 200 human demonstrations per task, the four-task mean score is 51.67%, ranging from 38.33% on Open Fridge to 61.67% on Close Drawer.The policies use no target-task robot demonstrations; Figure 4 shows results for budgets of 25, 50, 100, and 200 human demonstrations.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 Q2: HUMAN VS. ROBOT DEMONSTRATIONS

  • 76.36% less average collection labor per human episode (33.30 versus 140.85 person-seconds) accompanies task-level reductions of 31.41% on Close Laptop and 26.63% on Pull Curtain.The labor measure sums operator time for capture, setup, calibration, troubleshooting, and scene resets.Source: §4.3 · p. 7
  • On Close Laptop, 100 human demonstrations score 48.33% versus 63.33% from 50 robot demonstrations; on Pull Curtain, 200 human demonstrations score 53.33% versus 56.67%.Figure 5 compares scores and collection labor on these two tasks.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4 Q3: ACTION ALIGNMENT FOR IMPROVED ACCURACY

  • SONIC rollout has similar position error to retargeting (3.41 versus 3.42 cm) but higher orientation error (6.71° versus 5.32°); the learned MLP improves errors over unit response, especially Close Drawer (9.18 to 5.80 cm).SONIC also reduces extra yaw from 11.45° to 7.18° and improves the other reported motion-quality metrics.Source: §4.4 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 9
  • Mean position error falls from 11.21 cm at H to 3.41 cm at H + K + Ds, while orientation error falls from 21.22° to 6.71°; mean policy score rises from 21.25% to 51.67%.Kinematic correction removes much of the initial position mismatch, and dynamics-aware refinement further improves execution accuracy; the full pipeline improves scores over H + K on every task.Source: §4.4 · p. 7, Table 1 · p. 8
  • One K round followed by eight D rounds reduces position error to 3.97 cm and orientation error from 16.46° to 8.08°, whereas repeating K eight times changes position error only from 4.86 to 4.77 cm.Eight D rounds alone leave errors of 8.40 cm and 16.40°; Figure 6 compares stage outputs.Source: §4.4 · p. 8, Figure 6 · p. 9

4.5 Q4: VISUAL ALIGNMENT FOR EASIER TRANSFER

  • Across the three arm tasks, rendering, augmentation, and their combination score 38.89%, 42.78%, and 48.33%, respectively, with unchanged action labels.The combination performs best on Open Fridge and Pull Curtain, while augmentation alone is best on Close Laptop; rendering is not evaluated on foot-operated Close Drawer.Source: §4.5 · pp. 8–9, Table 4 · p. 9

5 CONCLUSION

EgoHumanoid-V2 transfers coordinated whole-body loco-manipulation from egocentric demonstrations through coarse-to-fine action alignment and visual adaptation. Across four real-world tasks, policies trained on 200 human demonstrations per task achieve a 51.67% mean score without target-task robot demonstrations, while alignment stability and processing throughput remain limitations.

5 CONCLUSION

  • 51.67% mean score across four real-world tasks, using 200 human demonstrations per task without target-task robot demonstrations.The results demonstrate zero-shot skill transfer on the tested tasks and support human motion as direct supervision for coordinated humanoid skills.Source: §5 · p. 9
  • Coarse-to-fine action alignment and visual adaptation transfer coordinated whole-body loco-manipulation skills from egocentric demonstrations.Source: §5 · p. 9
  • Alignment can still produce unstable motions, while IK and repeated rollouts limit throughput.Stronger stability constraints and faster optimization remain directions for improving reliability and processing efficiency.Source: §5 · p. 9

AI USE STATEMENT

Generative AI tools assisted with literature work, feedback, manuscript preparation, figures, and software development; the authors reviewed the work and take responsibility for the final materials.

AI USE STATEMENT

  • Generative AI tools assisted with literature discovery and synthesis.Source: AI USE STATEMENT · p. 10
  • The tools provided feedback on methods, experiments, manuscript organization, drafting and editing, and reference formatting.Source: AI USE STATEMENT · p. 10
  • The tools also assisted with figure and software development, while the authors reviewed the work and take responsibility for the final text, claims, citations, code, and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility materials describe data synchronization and validation, then detail how human motion targets guide kinematic correction and dynamics-aware refinement before accepted replays supply policy training pairs.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement directs readers to Sections 3–4 and Appendices A–D for the learning interface, controlled comparisons, implementation, evaluation, and supplementary alignment experiments.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DATA PREPARATION AND VALIDATION

  • GoPro RGB, PICO tracking, and hand-pose signals are synchronized into 50 Hz image–trajectory pairs, with validation and heading calibration preceding motion initialization.Validation checks timestamps, video–trajectory frame counts, endpoint tracking error, joint changes, base tilt, and pelvis displacement; without an external pulse, timing checks measure repeatability rather than absolute synchronization error.Source: §A.1 · p. 14

A.2 ACTION ALIGNMENT DETAILS

  • Kinematic alignment adjusts the robot reference toward fixed human hand or foot targets, then dynamics-aware refinement compensates for measured execution residuals.Targets use a pelvis-centered frame that removes world translation and heading while keeping its z axis vertical; foot targets approximate sole centers rather than measured contacts. Kinematic correction is bounded around the initial reference, and joint-preservation weights are soft penalties. Refinement locally linearizes endpoint geometry, uses fresh rollouts for execution context, and applies a schedule of one K round, seven regular D rounds, and one conservative D round.Source: §A.2 · p. 14

A.3 RESPONSE MODEL AND TRAINING

The response model learns how command changes affect executed endpoints from paired rollouts, and candidate motions are validated and selected before their state–action pairs support policy training. The evaluation pipeline uses a shared post-training configuration and image augmentation.

A.3 RESPONSE MODEL AND TRAINING

  • The response model predicts how a kinematic command change affects execution, using paired reference and perturbed-token rollouts from identical simulator states and controller histories.Its input is a Jacobian-based endpoint change and its target is the difference between executed endpoint poses one policy step later; training uses 345,080 frame–perturbation pairs from 250 motions, with 20,420 validation pairs from 13 other motions.Source: §A.3 · p. 15

A.4 MOTION SELECTION AND VALIDATION

  • Candidate motions passing endpoint-error, execution-consistency, and stability checks are ranked by largest and then mean normalized endpoint error, independently replayed, and rejected if none pass.The checks include 5 cm position error, 15° orientation error where applicable, 0.10 m pelvis-height drop, and 25° base tilt; accepted replays supply state–action pairs.Source: §A.4 · p. 15

B POLICY TRAINING AND EVALUATION

  • The policy experiments test whether aligned supervision supports physical task execution, with task-specific preprocessing and a shared low-level controller checkpoint used without target-task training or fine-tuning.Source: B POLICY TRAINING AND EVALUATION · p. 16, §A.4 · p. 16

B.1 POST-TRAINING CONFIGURATION

  • Controlled comparisons post-train π0.5 from pi05_base for 30,000 steps, while training images receive random resized crops, in-plane rotation, and photometric jitter.Deployment applies standard preprocessing to robot images only.Source: §B.1 · p. 16

B.2 TRAINING DATA AND CONTROLLED COMPARISONS

The section specifies the human- and robot-data comparisons, the supervision components varied in ablations, and the task-scoring procedure.

B.2 TRAINING DATA AND CONTROLLED COMPARISONS

  • Human-data policies use H + K + Ds alignment, robot-arm rendering, and image augmentation by default, except that Close Drawer omits arm rendering.The comparisons vary data budget or supervision components while holding the policy backbone, action interface, and evaluation protocol fixed.Source: §B.2 · p. 16
  • Data-budget comparisons use 25, 50, 100, or 200 human demonstrations per task; separate human- and robot-data policies are trained at the specified budgets.Action and visual ablations each use 200 demonstrations per task; action-label conditions compare H, H + K, and validated historical candidates.Source: §B.2 · p. 16

B.3 TASK SCORING CRITERIA

  • Each task has three sequential binary-scored subtasks, with evaluation stopping at the first failure and each remaining subtask scored zero.A trial score can therefore be 0, 1/3, 2/3, or 1; Table 5 lists the ordered task criteria.Source: §B.3 · p. 16

B.4 RESULT AGGREGATION

These sections define how task scores and supplementary alignment experiments are aggregated and compared. They report processing-cost and response-model protocols alongside the expanded motion-initialization analyses.

B.4 RESULT AGGREGATION

  • Scores summarize partial task completion, cross-task means weight tasks equally, and visual-ablation rows with arm rendering exclude Close Drawer; no error bars are reported.The four policies are task-specific, so starting-pose and object variations test within-task execution rather than unseen-task generalization by one policy.Source: §B.4 · p. 16

C SUPPLEMENTARY ACTION-ALIGNMENT EXPERIMENTS

  • Supplementary experiments test alignment stages, motion initialization, refinement rounds, and the response model on four tasks, with a separate 50-task study.The measures include tracking accuracy, motion quality, and processing time.Source: C SUPPLEMENTARY ACTION-ALIGNMENT… · p. 17

C.1 SHARED PROTOCOL AND METRICS

  • The four-task protocol matches targets, registration, controller, reset states, and evaluation windows across a fixed 25-trajectory sample per task.Position errors are in centimeters and orientation errors in degrees; task means are equally weighted, while candidate-selection rules vary across comparisons.Source: §C.1 · p. 17

C.2 STAGE COMPARISON AND HISTORICAL SELECTION

  • Stage comparisons select among initialization, kinematic correction, and refinement candidates; selected H + K + Ds output can retain H or H + K.The recorded measurements are used without validation replay or acceptance filtering.Source: §C.2 · p. 17

C.3 MOTION INITIALIZATION AND WHOLE-BODY MOTION QUALITY

  • Each initializer receives the same SONIC replay, one K round, and eight D rounds; selection uses only D1–D8, without a training-data acceptance filter.Motion quality is assessed with pelvis drop, base tilt, maximum joint-angle step between frames, and extra yaw comparing robot and human heading changes.Source: §C.3 · pp. 17–18

C.4 REFINEMENT DEPTH AND PROCESSING COST

  • K1D8 costs 1.49 s/frame; increasing K depth to eight raises the cost to 3.38 s/frame with similar endpoint error.Times pool runs across hosts and concurrent episodes, so they characterize these processing runs rather than single-machine latency.Source: §C.4 · p. 19, Table 8 · p. 19

C.5 RESPONSE-MODEL ABLATION

  • The response-model ablation compares a learned execution response with a unit response under shared H/K inputs, targets, refinement settings, and eight D rounds.The unit response assumes executed endpoint change equals the change predicted by the kinematic Jacobian.Source: §C.5 · p. 19

C.6 ACTION ALIGNMENT ACROSS 50 TASKS

The expanded study compares action-alignment stages, refinement depth, and learned execution response across 50 tasks. A separate online-IK study tests three command compositions under a whole-body tracking controller.

C.6 ACTION ALIGNMENT ACROSS 50 TASKS

  • 17.03 cm/36.06° position/orientation errors fall to 5.22 cm/21.81° after K, then to 3.83 cm/10.85° after refinement and historical selection.The expanded benchmark uses 112 paired trajectories across 50 tasks; lower endpoint errors are better.Source: §C.6 · p. 19, Table 9 · p. 20
  • One K plus eight D rounds reaches 4.10 cm and 9.21°, versus 5.22 cm and 21.81° after K alone; eight D rounds without K yield 14.16 cm and 21.91°.These depth curves report raw outputs without selecting an earlier candidate; repeated K offers little gain.Source: §C.6 · p. 20, Figure 7 · p. 20
  • After one K and eight D rounds, the MLP lowers position error from 4.25 to 4.10 cm and orientation error from 9.39° to 9.21°.The comparison uses 112 trajectories across 50 tasks and does not use historical selection.Source: §C.6 · pp. 19–20, Table 10 · p. 20

D ONLINE IK UNDER A WHOLE-BODY TRACKING CONTROLLER

  • The online-IK study compares three command compositions on five trajectories per task, totaling 60 runs across four tasks, without accuracy filtering or retries.This cohort is separate from the expanded action-alignment study.Source: D ONLINE IK UNDER A WHOLE-BODY… · p. 20

D.1 THREE COMMAND COMPOSITIONS

  • Variant A uses a broader IK joint set without balance contribution, B adds that contribution, and C restricts IK joints while leaving others under SONIC control.Balance and contact forces dynamically couple the base and limbs, including limbs whose commands remain unchanged.Source: §D.1 · pp. 20–21, Table 11 · p. 21

D.2 ACCURACY AND STABILITY

The tested online-IK variants show instability across the evaluated runs, and the accompanying sequences illustrate failure patterns rather than their frequency.

D.2 ACCURACY AND STABILITY

  • All 60 runs show instability: all 15 drawer runs fall during first-pose preparation, while 45 hand-task runs reach their final frames without stable tracking.The finding is limited to these SONIC implementations.Source: §D.2 · p. 21, Table 12 · p. 21
  • Variant B reaches 1.50–2.43 cm hand-task position errors but remains visibly unstable, so the tested variants are excluded from policy training.The study evaluates endpoint accuracy and stability together because reaching the final replay frame does not establish stable execution.Source: §D.2 · p. 21

D.3 QUALITATIVE FAILURE SEQUENCES

  • Figures 8–10 show one sequence per implementation and task, selected by peak base tilt across five runs; these examples illustrate patterns, not frequency.The sequence set includes initialization, and the fixed crops preserve relative motion and comparable robot scale across tasks.Source: §D.3 · p. 21, Figure 8 · p. 22, Figure 9 · p. 23

Related papers

Loading 2609.37181v1…