PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoAlign: Bridging the Human-Humanoid Gap for Long-Range Loco-Manipulation

Yiming Jiang, Chen Jin, Chongyang Xu, Yilun Chen, Aimin Hao, Yisheng He

arXiv:2609.38046v1cs.RO

TL;DR

Egocentric human demonstrations offer accessible task experience, but body-scale and controller-response differences and missing robot states limit their value as humanoid supervision. EgoAlign adapts motion and reconstructs robot states for a continuous whole-body controller, then evaluates human-only training with zero-shot physical deployment. The policies support object relocation, navigation to held-out goal positions, and foot interaction; refinement also reduces simulated endpoint error.

  • Problem

    Human demonstrations provide navigation and interaction experience, but body-scale and controller-response differences and missing robot state history limit their suitability as humanoid training supervision.

    Source: §I · p. 1

  • Method

    EgoAlign adapts human motion through dynamics-guided collection and scale alignment, then causally replays it to pair adapted actions with robot-state supervision for a continuous whole-body interface.

    Source: §I · pp. 1–2

  • Results

    Policies support long-range object relocation, navigation to held-out goal positions, and independently evaluated foot interaction; macro mean errors are 14.43, 7.14, 4.76, and 3.59 cm for NoAlign and Rounds 0–2, respectively, and 14.09 cm for GMR–SONIC.

    Source: B. Zero-Shot Loco-Manipulation on the G1 · p. 5, 4) Scale Alignment · p. 8

  • Takeaways & Limitations

    Human-only task training followed by zero-shot physical deployment shows the constructed supervision supports long-range loco-manipulation through a continuous whole-body interface.

    Source: §I · p. 2, §VI · p. 8

Abstract

from arXiv · show

Egocentric human demonstrations offer an accessible source of task experience, but differences in body scale and controller response, together with missing robot states, limit their value as humanoid training supervision. We present EgoAlign, a data-construction framework that converts these demonstrations into action and state supervision compatible with a general-purpose, continuous whole-body controller, without collecting physical-robot demonstrations. Using the target-robot model and simulator, EgoAlign guides demonstration collection through execution feedback. It preserves locomotion references for visually guided periodic stepping while adapting upper-body interaction geometry through scale alignment and controller-in-the-loop refinement. A final causal replay reconstructs the corresponding robot states and motion-token labels for training with the human observations. We assess the resulting supervision by fine-tuning a vision--language--action model solely on adapted human demonstrations and deploying it zero-shot on a physical humanoid. The resulting policies perform long-range object relocation, navigation to unseen goal positions, and independently evaluated foot interaction. Refinement improves simulated hand alignment and physical pickup success over kinematic alignment alone, while human collection reduces on-site acquisition time relative to teleoperation. https://lambdahumanoid.github.io/EgoAlign/

I. INTRODUCTION

Egocentric human demonstrations offer accessible experience, but body-scale and controller-response differences and missing robot states limit their value as humanoid supervision. EgoAlign adapts motion and reconstructs robot states, then tests the resulting supervision through human-only training and zero-shot physical deployment.

I. INTRODUCTION

  • Human-derived actions may not suit robot training because body scale and controller response alter realized hand trajectories, while recordings lack corresponding robot state history.These gaps matter for long-range loco-manipulation, which combines distant navigation, sustained interaction during locomotion, and local contact.Source: §I · p. 1
  • Alignment reduces simulated right-palm error from 14.27 to 1.65 cm and increases physical object-pickup success from 0% to 90%.The evaluation also includes human-only training and zero-shot deployment on long-range object relocation, position-generalized navigation, and independently tested foot interaction.Source: §I · pp. 1–2
  • EgoAlign constructs robot-compatible action and state supervision from egocentric human demonstrations for VLA learning through a continuous whole-body control interface.Its motion adaptation and causal state reconstruction use no physical-robot demonstrations.Source: §I · pp. 1–2
  • EgoAlign adapts motion through dynamics-guided collection and upper-body scale alignment, then uses causal replay to pair adapted actions with corresponding robot states.The framework constructs supervision for SONIC’s general-purpose, continuous whole-body interface.Source: §I · pp. 1–2

II. RELATED WORK

The related work spans general VLA models, robot-free human-demonstration systems, and humanoid whole-body control. EgoAlign focuses on constructing controller-compatible action and state supervision from recorded human demonstrations.

A. Vision-Language-Action Models

  • VLA work ranges from generalist robot pretraining and modular model comparisons to open-world mobile manipulation, unified action spaces, and sound-centric manipulation.For humanoids, Ψ0 combines egocentric-video pretraining with a whole-body action expert trained on humanoid trajectories.Source: A. Vision-Language-Action Models · p. 2

B. Robot-Free Human Demonstrations

  • Robot-free human-demonstration work scales egocentric experience through routine annotation, data standardization, predictable scaling, joint human-robot alignment, and interaction-centric transfer.These efforts motivate human demonstrations as scalable supervision while identifying visual and kinematic embodiment gaps.Source: B. Robot-Free Human Demonstrations · p. 2

C. Humanoid Loco-Manipulation and Whole-Body Control

  • EgoAlign differs from prior humanoid systems by constructing controller-compatible action and state supervision from recorded human demonstrations, with navigation and foot interaction through a continuous whole-body interface.The cited landscape includes systems using task-space policies, discrete lower-body primitives, planned motions, or synthesized interactions.Source: C. Humanoid Loco-Manipulation and… · p. 2

A. Problem Formulation and System Overview

  • The system overview links synchronized human observations and motion to execution feedback, scale alignment, state reconstruction, and human-only policy training for zero-shot deployment.The VLA predicts motion tokens and hand commands; SONIC decodes tokens with state history into body commands.Source: A. Problem Formulation and System… · p. 2, Figure 1 · p. 3

B. Causal State Reconstruction

Causal state reconstruction supplies the robot-state history absent from human demonstrations and pairs it with motion-token labels under simulated replay. The procedure captures pre-action feedback before the corresponding action is executed.

B. Causal State Reconstruction

  • Reconstruction addresses the missing robot state history by replaying each finalized motion to obtain proprioceptive states and paired motion tokens for VLA training.The synchronized episode is first staged as a deterministic replay reference with body motion and hand commands.Source: B. Causal State Reconstruction · p. 3
  • During the 50 Hz SONIC–MuJoCo rollout, the controller reads the simulated state and updates its history before evaluating the current action.The pre-action state and controller history are constructed before executing that action.Source: B. Causal State Reconstruction · p. 3
  • Feedback injection pairs each 43-D robot proprioceptive state with the 64-D SONIC token captured at the same control tick and the hand-command vectors.The state comprises 29 body-joint positions and two 7-D hand states.Source: B. Causal State Reconstruction · p. 3

C. Human Demonstration Acquisition

Human demonstrations are acquired with tracked body motion, hand commands, and synchronized egocentric views. A downward camera covers near-field interaction while a level camera supplies distant navigation cues.

C. Human Demonstration Acquisition

  • A PICO 4 Ultra headset and five wrist, ankle, and waist trackers capture motion, which is converted to 24-joint SMPL references while retaining wrist orientations and hand commands.PICO hand recognition supplies discrete grasp/non-grasp commands for collection and physical execution.Source: C. Human Demonstration Acquisition · p. 3
  • Two chest-mounted cameras capture a downward view of hands, objects, and nearby workspace and a horizontal level view of distant targets.Human and robot cameras have matched ground-relative heights, and both image streams are synchronized to the PICO trajectory for training.Source: C. Human Demonstration Acquisition · p. 3, Figure 2 · p. 3

D. Human-to-Humanoid Motion Adaptation

EgoAlign uses execution feedback during collection to expose motions suppressed by scale, controller response, or robot dynamics, then adapts upper-body interaction geometry while preserving root travel and lower-body references. Kinematic alignment is followed by controller-in-the-loop replay refinement, and only trajectories meeting acceptance criteria proceed to causal-data validation.

1) Dynamics-Guided Demonstration Collection:

  • SONIC–MuJoCo replay exposes suppressed motions during collection, letting demonstrators repeat or amplify them until the intended robot movement is visible in simulation.This feedback is collected without operating a physical robot; offline alignment separately corrects recorded motion before state reconstruction.Source: 1) Dynamics-Guided Demonstration… · pp. 3–4

2) Scale Alignment:

  • Scale alignment adapts upper-body interaction geometry while preserving global root travel and lower-body motion in the SMPL reference.It uses a robot-scale proxy fitted to target morphology while keeping human-reference hand targets fixed in a shared interaction frame.Source: 2) Scale Alignment · p. 4
  • The bounded pose residual is optimized on task-selected joints and regularized for magnitude and temporal smoothness, while other joints retain their baseline trajectories.The optimization adjusts collar, shoulder, and elbow joints; spine and wrist-local rotations retain their per-frame baseline trajectories.Source: 2) Scale Alignment · p. 4
  • Controller-in-the-loop refinement replays the kinematically aligned motion, compensates estimated lag for geometric comparison, and updates targets using bounded corrections and response gains.At most two refinement rounds are run; trajectories failing closed-loop endpoint and rollout acceptance criteria after Round 2 are excluded from training.Source: 2) Scale Alignment · p. 4
  • After optimization, motion references are regenerated for causal state reconstruction, and accepted trajectories undergo separate causal-data integrity validation.Gradients pass through the optimization skeleton, not SONIC or MuJoCo.Source: 2) Scale Alignment · pp. 4–5

E. Human-Only VLA Fine-Tuning and Humanoid Execution

The VLA is fine-tuned on human observations and pre-action states paired with robot action targets from causal replay. At deployment, it predicts from live robot observations and state, while SONIC decodes tokens using state history and hand commands execute separately.

E. Human-Only VLA Fine-Tuning and Humanoid Execution

IV. EXPERIMENTS

The experiments assess physical performance from policies trained solely on human demonstrations, the effects of selected data and supervision choices, refinement outcomes, and collection efficiency relative to teleoperation.

IV. EXPERIMENTS

  • The study asks how well policies fine-tuned solely on human task demonstrations perform on a physical humanoid.Source: §IV · p. 5
  • A second evaluation examines how navigation data, level-view perception, state reconstruction, and scale alignment affect performance.Source: §IV · p. 5
  • The refinement evaluation measures simulated endpoint error and physical pickup success.Source: §IV · p. 5
  • The collection-efficiency comparison measures on-site acquisition time for human versus teleoperated collection.Source: §IV · p. 5

A. Experimental Setup

The setup uses a Unitree G1 with SONIC control and MuJoCo replay, and evaluates object relocation, navigation, and foot interaction. Demonstrations cover seen and offset unseen goal positions, while policies share a common training recipe.

A. Experimental Setup

  • Experiments use a Unitree G1 with a 1.25 m reference height, SONIC control, and MuJoCo replay; human-scale SMPL targets use a 1.55 m canonical height.Cameras at approximately 1.3 m cover only the next 2 m downward, motivating the level view.Source: 1) Tasks and Data · p. 5
  • Object relocation includes basket pickup, approximately 10 m of transport, and table placement; navigation and foot interaction combine approximately 5 m of navigation with pedal-bin opening under one policy.Quantitative tests separate the navigation and foot-interaction subtasks, with foot interaction beginning from standing.Source: 1) Tasks and Data · p. 5
  • Object relocation and navigation test direct, five seen, and five offset unseen goal positions, while foot interaction uses multi-position protocols.Three demonstrators collected 200 full-trajectory relocation demonstrations and 200 each for navigation and foot interaction; relocation also used 100 basket-carrying navigation demonstrations, with approximately 90% retained after filtering.Source: 1) Tasks and Data · p. 5
  • Replay models ground contact but not task objects or hand/foot–object contacts; object-relocation replay applies a grasp-triggered downward force equivalent to an empty 200 g basket.Source: 1) Tasks and Data · p. 5
  • Policies use π0.5 initialized from PaliGemma-3B-PT-224 and train for 50,000 steps on eight A800 GPUs; final checkpoints are evaluated without physical-feedback-based selection or tuning.Each 50-step action chunk executes at 50 Hz before the policy reobserves and requests inference.Source: 2) Training and Evaluation · p. 5

2) Training and Evaluation:

Physical evaluation tests long-range relocation and independently scored navigation and foot interaction, using 20 trials per setting and stage-based success criteria. The policies achieve performance across seen and unseen endpoints, while human collection reduces per-demonstration acquisition time.

2) Training and Evaluation:

  • Each setting has 20 trials; trials end at completion, a two-minute timeout, or clear failure, with failures and interventions counted unsuccessful.Multi-position protocols use five positions with four trials each.Source: 2) Training and Evaluation · p. 5

3) Scoring Stages and Success Criteria:

  • Stage score is 100 times the mean fraction of stages completed per trial, while full success requires passing every stage.Stages after the first failure score zero; object relocation has four stages, navigation two, and foot interaction three.Source: 3) Scoring Stages and Success Criteria · p. 5, Figure 4 · p. 6

B. Zero-Shot Loco-Manipulation on the G1

C. Human Data-Collection Efficiency

D. Ablation Study

The ablations test navigation demonstrations, visual context, and state reconstruction. Added navigation data and level-view perception correspond to better multi-position results, while all four comparisons fail at the first Direct stage.

D. Ablation Study

  • All four comparisons fail at S1 in every Direct trial and therefore are not evaluated on multi-position settings.The navigation-data ablation removes additional navigation demonstrations; the other variants share selected source demonstrations.Source: D. Ablation Study · pp. 6–7

1) Navigation Data:

  • With added basket-carrying navigation demonstrations, full success is 40% on both multi-position settings, versus 20% and 30% without them.Both methods achieve 65% on Direct; successful placement requires accurate navigation to the placement location.Source: 1) Navigation Data · p. 7

2) Downward-View and Level-View Perception:

  • Removing level-view perception reduces full success by 25 and 30 percentage points on Multi-seen and Multi-unseen, with marked first-stage declines.The Downward-only and Dual-view variants use identical motion and state data and perform comparably on Direct.Source: 2) Downward-View and Level-View… · p. 7, Figure 6 · p. 7
  • The w/o State Recon. ablation retains images and scale-aligned human references but zeros the SONIC decoder’s history.The passage identifies this as an object-relocation ablation.Source: 3) Causal State Reconstruction · p. 7

3) Causal State Reconstruction:

EgoAlign’s state reconstruction and scale-alignment refinement improve robot supervision: the full method outperforms tested alternatives in physical relocation and simulated hand alignment. The discussion bounds replay’s support to light-load, contact-tolerant tasks, while the conclusion reports human-only training followed by zero-shot deployment through a continuous whole-body interface.

3) Causal State Reconstruction:

  • The full reconstruction passes S1 in 20/20 Direct trials, versus 0/20 for the tested alternative; training uses 43-D robot states and recomputed token labels.Token labels use reference-based orientation anchoring rather than rollout robot-base orientation.Source: 3) Causal State Reconstruction · p. 7

4) Scale Alignment:

  • With lag compensation, simulated macro error on common active right-palm frames falls from 14.27 cm for NoAlign to 6.48 cm after kinematic alignment and 1.65 cm after two refinement rounds.Without lag compensation, macro mean errors are 14.43, 7.14, 4.76, and 3.59 cm for NoAlign and Rounds 0–2, respectively, and 14.09 cm for GMR–SONIC.Source: 4) Scale Alignment · pp. 7–8
  • S1/S2 success is 100%/90% for the full method, versus 50%/0% for Round 0 and 0%/0% for NoAlign in 20 Direct trials per setting.Round 0 trials end after S2.Source: 4) Scale Alignment · p. 7
  • EgoAlign’s 1.65 cm right-palm macro error is lower than GMR–SONIC’s 14.17 cm; all physical GMR–SONIC relocation trials fail at S1.GMR–SONIC uses matched palm targets and G1 geometry, then feeds joint references through SONIC without post-decoder overrides or refinement.Source: 4) Scale Alignment · p. 8

V. DISCUSSION:

  • Simplified replay is supported as a practical supervision approximation for light-load, contact-tolerant tasks, not as exact reconstruction of object-contact dynamics.The evaluated foot operation is pedal pressing, whose comparatively large contact region accommodates greater positioning error; precision-sensitive hand operations need further endpoint refinement.Source: §V · p. 8

VI. CONCLUSION:

  • EgoAlign pairs adapted human actions with corresponding robot states, and human-only training followed by zero-shot deployment supports long-range loco-manipulation through a continuous whole-body interface.The conclusion frames this as connecting accessible egocentric demonstrations with training value for a target humanoid.Source: §VI · p. 8
Loading 2609.38046v1…