PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration

Modi Shi, Shijia Peng, Jin Chen, Haoran Jiang, Tianyu Li, Di Huang, Ping Luo, Hongyang Li, Li Chen

arXiv:2602.10106v2cs.RO

TL;DR

Humanoid loco-manipulation lacks diverse, scalable demonstrations because robot teleoperation is costly and laboratory-bound. EgoHumanoid co-trains a VLA policy on egocentric human and teleoperated robot data after aligning viewpoints and actions. Human data improves performance over robot-only training, especially in unseen environments, while the method remains limited by orientation ambiguity, scaling uncertainty, and restricted whole-body expressiveness.

  • Problem

    Humanoid loco-manipulation remains bottlenecked by scarce, diverse demonstrations, while robot teleoperation is costly and largely laboratory-bound.

  • Method

    EgoHumanoid co-trains a VLA policy on egocentric human and teleoperated robot demonstrations using view and action alignment.

  • Results

    51% performance gain is achieved in scenes unseen by the robot data when human demonstrations are incorporated.

  • Takeaways & Limitations

    Egocentric human data supports humanoid loco-manipulation generalization beyond the environments covered by robot teleoperation.

  • Takeaways & Limitations

    The method remains limited by orientation ambiguity in delta end-effector representations, uncertain human-data scaling laws, and restricted expressive whole-body control.

Abstract

from arXiv · show

Human demonstrations offer rich environmental diversity and scale naturally, making them an appealing alternative to robot teleoperation. While this paradigm has advanced robot-arm manipulation, its potential for the more challenging, data-hungry problem of humanoid loco-manipulation remains largely unexplored. We present EgoHumanoid, the first framework to co-train a vision-language-action policy using abundant egocentric human demonstrations together with a limited amount of robot data, enabling humanoids to perform loco-manipulation across diverse real-world environments. To bridge the embodiment gap between humans and robots, including discrepancies in physical morphology and viewpoint, we introduce a systematic alignment pipeline spanning from hardware design to data processing. A portable system for scalable human data collection is developed, and we establish practical collection protocols to improve transferability. At the core of our human-to-humanoid alignment pipeline lies two key components. The view alignment reduces visual domain discrepancies caused by camera height and perspective variation. The action alignment maps human motions into a unified, kinematically feasible action space for humanoid control. Extensive real-world experiments demonstrate that incorporating robot-free egocentric data significantly outperforms robot-only baselines by 51\%, particularly in unseen environments. Our analysis further reveals which behaviors transfer effectively and the potential for scaling human data.

I. INTRODUCTION

EgoHumanoid addresses scarce and laboratory-bound humanoid demonstrations by co-training on portable egocentric human data and limited robot data. Its alignment pipeline targets visual and action differences, improving transfer to diverse environments.

  • Humanoid loco-manipulation requires coordinated locomotion, posture adjustment, manipulation, and dynamic balance in unstructured environments.
  • Robot teleoperation provides embodiment-consistent supervision but is costly, complex, hardware-dependent, and largely confined to laboratories.
  • Portable wearable sensing enables scalable human demonstrations across diverse environments without robot hardware.
  • Human demonstrations face morphology, viewpoint, kinematic, and dynamics mismatches when transferred to humanoid robots.
  • EgoHumanoid combines robot-free human demonstrations with teleoperated robot data and aligns observations and actions for co-training.
  • 20% average improvement and a 51% gain in unseen scenes are reported when human data is incorporated.

A. Egocentric Human Data for Robots

Egocentric human data offers a scalable alternative to robot-centered supervision, but humanoid loco-manipulation requires cross-embodiment alignment. EgoHumanoid applies co-training to this setting and evaluates generalization beyond robot-data scenes.

  • Egocentric data collection can scale across diverse environments without robot hardware, supporting embodied-AI learning.
  • Co-training uses aligned human and robot actions as supervision, extending prior fixed-base manipulation approaches to humanoid loco-manipulation.
  • Human-to-robot transfer is harder than robot-to-robot transfer because human and robot embodiments differ substantially.
  • Humanoid loco-manipulation couples locomotion, manipulation, and task planning while requiring reliable and scalable supervision.
  • Existing pipelines remain primarily robot-centered, limiting environmental diversity and study of generalization in human-centric settings.
  • EgoHumanoid trains a VLA policy on combined teleoperated robot and egocentric human demonstrations for novel environments.

B. Data Collection System

EgoHumanoid uses a portable VR-based system to collect human and robot demonstrations with shared sensing components. Its alignment pipeline converts human observations and motions into robot-compatible training signals.

  • Robot data requires expensive hardware and laboratory setup, whereas wearable devices support cheaper, flexible collection in diverse settings.
  • The integrated VR setup uses the same camera for egocentric recordings, trackers for coarse human poses, and a controller for robot teleoperation.
  • Human demonstrations are recorded with a portable PICO VR system, motion trackers, and a head-mounted egocentric camera.
  • The collection system records navigation commands, end-effector wrist poses, grasp states, and synchronized egocentric RGB images.
  • View alignment addresses camera-height differences, while action alignment addresses action-representation differences caused by morphology.
  • Upper Body: Upper-body actions use 6-DoF delta end-effector poses to enable cross-embodiment comparison without a globally aligned base frame.
  • Lower body: Lower-body human motion is converted into discrete navigation commands after smoothing and estimating local-frame velocities and yaw rate.
  • Gripper: Gripper behavior is represented as a binary open-or-closed action inferred from tracked human hand motion.

D. Loco-Manipulation Policy Co-Training

EgoHumanoid co-trains a single policy on aligned human and robot demonstrations for whole-body tasks spanning locomotion and manipulation. Evaluation compares robot-only and co-trained policies in laboratory-like and human-centric environments.

  • The policy receives egocentric RGB observations and language instructions and outputs actions in a unified action space.
  • The evaluation asks whether human data improves generalization, which behaviors transfer, and whether performance scales with human demonstrations.
  • The benchmark covers pillow placement, trash disposal, toy transfer, and cart stowing, spanning carrying, grasping, placement, and sustained contact.
  • The evaluation contrasts robot teleoperation in laboratories with human-centric scenes reserved for generalization testing.
  • Co-training improves performance over robot-only baselines across in-domain and generalized environments.
  • The four tasks require 1–5m locomotion and coordinated manipulation, with stopping accuracy affecting downstream manipulation success.
  • 20 trials per setting use position perturbations and normalized scores to evaluate performance.

B. Will human data improve humanoid loco-manipulation?

Co-training human egocentric demonstrations with robot data improves humanoid loco-manipulation in both laboratory and unseen environments. Human data transfers especially well to navigation, while precision-critical manipulation benefits most from combining both sources.

  • Co-training achieves 78% average score versus 59% for robot-only in-domain and 82% versus 31% in generalization.The larger generalization gap shows stronger performance in scenes covered by human data but absent from robot data.
  • Transfer by subskill: Human-only achieves 100% on navigation-dominated subtasks, while robot-only produces near-zero success on these stages.On Toy Transfer s3, Human-only reaches 45%, only 15% below co-training.
  • Transfer by subskill: Human data transfers to manipulation, but effectiveness diminishes as precision requirements increase.Human-only outperforms robot-only on coarse manipulation, but reaches 5% versus 15% on precision-critical Cart Stowing s2.
  • Transfer by subskill: Co-training reaches 60% on precision-critical Cart Stowing s2, compared with 5% for human-only and 15% for robot-only.This pattern indicates that human demonstrations provide useful manipulation priors when combined with robot demonstrations.
  • Failure analysis: Human-only exhibits three times more manipulation failures than locomotion failures, whereas robot-only failures are balanced across both phases.The failure analysis separates failed episodes into locomotion and manipulation causes for robot-only, human-only, and co-training settings.

D. Does Performance Scale with Human Data?

Increasing human demonstrations generally improves generalized loco-manipulation performance, but the best robot-to-human sampling ratio depends on task characteristics. View alignment and scene diversity further support transfer, especially when viewpoints or environments vary.

  • Sampling strategy: Coarse-grasping tasks favor a 1:2 robot-to-human sampling ratio, whereas fine-manipulation tasks favor a 2:1 ratio.Pillow Delivery benefits from more human data, while Toy Transfer and Cart Shopping favor more robot data.
  • Human-data scaling: Performance improves consistently as human demonstrations increase from 0 to 300 across sampling ratios.The scaling study reports improved loco-manipulation performance without overfitting to dominant human data.
  • View alignment: View alignment yields consistent improvements across all tasks, with the largest gains on Toy Transfer and Cart Stowing.These tasks involve objects at varying heights, producing viewpoints that differ between human and robot data.
  • Scene diversity: Zero-shot performance on a novel scene rises monotonically from 57.5% with robot-only data to 82.5% with three human scenes.The data quantity remains fixed while the number of distinct human scenes varies, isolating scene diversity as a driver of generalization.

G. Data Collection Efficiency

Human demonstrations improve collection efficiency and support generalization, but humanoid loco-manipulation remains constrained by embodiment-related limitations and substantial data requirements.

  • Collection Efficiency: 39.7 s average human demonstration collection time is lower than 62.1 s for robot teleoperation across four tasks.Human demonstrations are roughly 2× faster per episode.
  • Limitations: Delta end-effector poses leave human-hand to robot-gripper orientation correspondence ambiguous without proprioceptive input.This limits reliable fine-grained rotational control in manipulation tasks.
  • Data Requirements: 2–3× more demonstrations are required for comparable loco-manipulation success than for fixed-base manipulation tasks of similar difficulty.Navigation creates diverse manipulation viewpoints because the robot does not stop at precisely the same position.
  • Generalization: Human demonstrations enable deployment in diverse environments without requiring in-the-wild robot data.The framework reports substantial generalization improvements from aligned human and robot data.
  • Limitations: EgoHumanoid is presented as an early human-to-humanoid transfer framework with unresolved orientation ambiguity, scaling, and expressive whole-body-control limitations.Future work targets advanced egocentric hardware and larger-scale human data.

APPENDIX

The appendix explains why human data is co-trained after alignment, discusses portability and extensions, and records limitations and future research directions.

  • Motivating Questions: Post-training co-training uses aligned human and robot actions to investigate egocentric data for humanoid loco-manipulation.Human-data pre-training is described as compatible but separate from this initial co-training investigation.
  • Other Embodiments: The alignment pipeline can be instantiated on humanoids with suitable inverse kinematics and locomotion controllers.Deployment on the 1.3 m Unitree G1 demonstrates transfer across a substantial height gap from 1.6–1.8 m human demonstrators.
  • Expressive Whole-Body VLA: Expressive locomotion policies are identified as a future extension for tasks requiring precise leg movements.The existing human demonstration pipeline already captures full-body motion, supporting such extensions.
  • Sensor Design: A minimalist egocentric sensor setup preserves a general alignment baseline, while wrist cameras and hand inpainting introduce embodiment-specific concerns.Wrist cameras may amplify hand-geometry differences, and inpainting can introduce detection and rendering errors.
  • Limitations and Future Endeavors: Key limitations include rotation ambiguity, higher demonstration requirements from navigation-induced viewpoint variability, and discrete locomotion commands.Future work includes advanced egocentric hardware, internet-scale egocentric-video pre-training, and expressive action representations.
  • Data Collection: Occluded or rapidly changing human motion may require tactile and SLAM-based sensing to compensate for estimation errors.These conditions are identified as practical challenges for demonstration collection.
  • Broader Implications: The alignment approach suggests cross-embodiment adaptation may support learning across humanoids, mobile manipulators, and other non-anthropomorphic platforms.The passage frames this as potential for future foundation models with minimal adaptation.

B. Practical Guidelines for Data Collection

The data-collection guidelines constrain demonstrator motions and sensing conditions to reduce alignment noise, while the pipeline transforms observations and actions for policy training.

  • Hand-Wrist Pose Consistency: Demonstrators should maintain consistent hand-wrist orientations because small wrist rotations create conflicting delta end-effector action labels.Such inconsistency can degrade policy learning by pairing similar observations with different targets.
  • Base Coordinate Stability: Demonstrators should keep torso posture relatively stable because locomotion commands use the human waist as the base coordinate frame.Excessive human-like upper-body sway can introduce systematic velocity-estimation mismatch.
  • Hand Visibility Maintenance: Maintaining hand visibility is important for reliable pose estimation when only head-mounted egocentric cameras are used.Occlusion can produce missing or inaccurate action labels.
  • View Alignment: View alignment uses depth-based reprojection, a rigid 0.25 m camera translation, ±0.05 m translation noise, and inpainting to approximate robot viewpoints.MoGe predicts depth before warping and Stable Diffusion 2.0 Inpainting fills resulting occlusions.
  • Action Alignment: Action alignment smooths upper- and lower-body trajectories and quantizes forward, lateral, and yaw velocities into three bins each.The same Savitzky–Golay configuration is used for translation, rotation, and pelvis-trajectory smoothing.
  • Policy Architecture and Training: The policy outputs an 18-dimensional action space covering two-arm end-effector poses, locomotion, grippers, and delta height.The action chunk size is 50 steps.
  • Policy Architecture and Training: Training uses eight NVIDIA A100 GPUs with an effective batch size of 256 and no proprioceptive inputs from either data source.Optimization runs for 20,000 steps with mixed-precision training.
  • Robot Platform: Robot head cameras capture egocentric RGB at 20 Hz, while robot demonstrations are downsampled from 100 Hz to 20 Hz for training.A low-level controller converts high-level delta end-effector commands into joint-level targets.

D. Scoring Criteria for Real-world Experiments

The experiments evaluate four real-world loco-manipulation tasks as ordered stage sequences, requiring every stage to succeed for overall task success. They report both task-level and stage-wise outcomes across laboratory and generalized environments.

  • Each task is decomposed into ordered locomotion- or manipulation-dominant stages, and a trial succeeds only when all stages succeed in sequence.Stage-wise success and failure are also recorded for fine-grained analysis.
  • Average Score measures the mean stage completion rate across a task’s ordered stages.For each stage, sk is 1 when successful and 0 otherwise; scores are averaged over trials.
  • The evaluation covers four tasks: Trash Disposal, Toy Transfer, Cart Stowing, and Pillow Placement.These tasks combine locomotion with carrying, reaching, grasping, placement, disposal, or pushing requirements.
  • In-domain trials use laboratory settings, while generalized trials relocate tasks to outdoor, household, or home-like environments.Examples include a garden, household furniture, and a standard bedroom scene.
  • Success criteria require stable locomotion and manipulation outcomes, including feasible poses, secure grasps, successful object placement or disposal, and controlled cart pushing.Failures include drops, unstable motion, infeasible poses, misses, and cases requiring human intervention.

E. Qualitative Results on Human-to-Humanoid Alignment

The view alignment pipeline converts human egocentric observations into robot-like RGB views through depth estimation, reprojection, and diffusion-based inpainting. Qualitative examples show this process across diverse indoor and outdoor task scenes.

  • View alignment shifts human egocentric imagery downward to approximate the robot’s lower camera height.The pipeline estimates depth, transforms the point cloud into the robot camera frame, and reprojections introduce disoccluded regions.
  • Diffusion-based inpainting fills black or invalid regions while preserving scene structure and object appearance.The resulting RGB images closely approximate the robot’s egocentric observations.
  • The qualitative pipeline is demonstrated across bedrooms, gardens, cluttered tabletops, and retail environments.These examples span the diverse scenes and scenarios used in the experiments.

F. License of Assets

The implementation incorporates multiple open-source assets released under permissive licenses, primarily MIT and Apache 2.0.

  • EGOHUMANOID uses PICO XRoboToolkit, MoGe, and latent diffusion under the MIT License.
  • DINOv2, openpi, and GR00T-WholeBodyControl are incorporated under the Apache 2.0 License.
Loading 2602.10106v2…