PaperlayerView exact revision on arXiv

Source-linked AI summary

HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience

Yang Liu, Jiacheng Liu, Hengtao Li, Yuheng Huang, Zijian Yang, Ji Dai, Maoxin Wang, Yi Gao, Qishen Wang, Baohua Yin, Peiyang Li, Shuanghao Bai, Mingyang Sun, Pengxiang Ding, Jianqin Yin, Lixin Yang, Donglin Wang

arXiv:2610.34186v1cs.RO

TL;DR

Humanoid VLAs must coordinate whole-body motion, while diverse robot demonstrations are costly and egocentric human data differs in appearance and proprioception. HuGe combines existing VLA architectures with a staged human-and-robot-data training pipeline and data adaptations. It reports leading HumanoidArena performance and higher real-world pick-and-drop success when visually adapted egocentric demonstrations supplement robot data.

  • Problem

    Humanoid VLA learning requires coordinated whole-body control and diverse demonstrations, but robot data is costly and egocentric human data differs visually and lacks robot proprioception.

    Source: §1 · p. 1

  • Method

    HuGe trains three existing VLA architectures through egocentric pre-training, humanoid adaptation, and downstream fine-tuning, using simulated pseudo-proprioception and visual adaptation.

    Source: §1 · p. 2, §3.2.1 · p. 4, §3.2.3 · p. 5

  • Results

    HuGe-GR00T and HuGe-Pi exceed π0.5 by 12.86 and 11.19 percentage points, respectively, while visually adapted egocentric demonstrations raise pick-and-drop success to 66.7% from 60.0%.

    Source: §4.1.4 · p. 7, §4.2.2 · p. 8

  • Takeaways & Limitations

    Real-world G1 experiments support supplementing robot demonstrations with visually adapted egocentric demonstrations for the pick-and-drop task.

    Source: §4.2.2 · p. 8, §5 · p. 9

Abstract

from arXiv · show

Extending Vision-Language-Action models (VLAs) to humanoid robots requires coordinated whole-body control and diverse demonstrations that are costly to collect in real-world scenarios. Although egocentric human demonstrations provide a complementary source of task experience, differences in visual embodiment and missing robot proprioception complicate transfer to humanoid policies. To address these challenges, we introduce HuGe, a family of Humanoid Generalist VLAs pre-trained on over 2,000 hours of egocentric human and robot data. Specifically, HuGe employs three existing representative VLA architectures to predict latent motion tokens for a pretrained whole-body controller, together with hand-control signals. Based on this, we introduce a three-stage training pipeline that combines egocentric pre-training, humanoid adaptation, and downstream fine-tuning. To facilitate transfer across embodiments, we construct pseudo-proprioceptive states through simulation playback during pre-training and modify arm appearance in egocentric observations through re-rendering during downstream fine-tuning. With this pipeline, HuGe achieves the state-of-the-art on simulation benchmark HumanoidArena. Furthermore, real-world experiments on the Unitree G1 indicate that augmenting robot demonstrations with visually adapted egocentric demonstrations significantly increases success on a pick-and-drop task.

1 Introduction

Extending VLAs to humanoids requires coordinated whole-body control as well as diverse demonstrations, but robot data is costly and egocentric human data differs in visual embodiment and proprioception. HuGe combines a whole-body controller with a three-stage training pipeline using human and robot data, and reports state-of-the-art HumanoidArena performance.

1 Introduction

  • Humanoid VLA execution must coordinate arms, torso, and lower body while maintaining balance and stability.The action interface must connect task-level predictions to coordinated whole-body execution.Source: §1 · pp. 1–2
  • Collecting diverse whole-body robot demonstrations is costly, while egocentric human demonstrations lack robot proprioception and differ visually.These gaps complicate direct transfer to humanoid policy learning.Source: §1 · p. 2
  • HuGe adapts egocentric data with simulation-playback pseudo-proprioceptive states during pre-training and arm-region re-rendering during downstream fine-tuning.The paper also reports state-of-the-art performance on HumanoidArena and real-world experiments supporting its training recipe.Source: §1 · p. 2
  • HuGe trains humanoid generalist VLAs on over 2,000 hours of egocentric human and robot data through a three-stage pipeline.The stages incorporate egocentric human data, humanoid teleoperation data, and downstream demonstrations.Source: §1 · p. 2

2 Related Work

Related work establishes the VLA and whole-body-control foundations for HuGe, while prior egocentric-learning research motivates transferring human experience through aligned actions and visual adaptation.

2.1 Vision-Language-Action Models

  • Prior VLA and world-action models advanced fixed-base manipulation; HuGe studies three representative architectures for humanoid whole-body control without modifying their designs.The selected architectures are OFT [15], GR00T, and π.Source: §2.1 · p. 2

2.2 Humanoid Whole-Body Control

  • Humanoid-control research progressed from simulated locomotion to motion tracking and hierarchical VLA control, while HuGe focuses on egocentric demonstrations for whole-body learning.Related approaches use reinforcement-learning controllers or generate full-body motion references and latents.Source: §2.2 · pp. 2–3

2.3 Egocentric Learning

  • Egocentric-learning work spans visual representations, human-motion action prediction, and robot-compatible adaptation; HuGe investigates controller-compatible supervision, simulated pseudo-proprioception, and visual adaptation.Prior embodiment adaptation includes human-arm removal, robot rendering, and embodiment masking [17] [14] [55].Source: §2.3 · p. 3

3 Methodology

HuGe uses existing VLA designs with a shared whole-body action interface, trained through egocentric pre-training, humanoid adaptation, and task-specific fine-tuning. The stages progressively incorporate human experience, robot demonstrations, and task data.

3.1 Architecture

  • HuGe predicts a 64-dimensional SONIC latent motion token and two binary hand controls, which the pretrained controller converts into whole-body joint commands.The action prediction is conditioned on visual observations, proprioceptive state, and a language instruction.Source: §3.1 · p. 3
  • OFT, GR00T, and π retain their established architectures but use distinct action modules to predict HuGe’s shared humanoid action interface.OFT uses an MLP, GR00T a diffusion transformer with cross-attention, and π a DiT action expert connected to multiple backbone layers.Source: §3.1 · pp. 3–4

3.2 Training Recipe

  • The three-stage recipe progresses from egocentric pre-training to humanoid adaptation and then task-specific fine-tuning.Table 1 summarizes the data sources and dataset sizes used at each stage.Source: §3.2 · p. 4, Table 1 · p. 4

3.2.1 Stage 1: Egocentric Pre-Training

  • Egocentric pre-training combines human and simulated-robot demonstrations in SONIC motion space and uses simulated G1 playback to create 34-dimensional pseudo-proprioceptive states.The state contains 29 joint angles, two hand-state entries, and body-frame gravity direction; unavailable hand-action labels are masked during training.Source: §3.2.1 · p. 4

3.2.2 Stage 2: Humanoid Adaptation

  • Humanoid adaptation continues training on real G1 demonstrations using a shared action and state interface across heterogeneous datasets.The representation combines SONIC motion tokens, hand-control labels, and proprioceptive inputs derived from recorded body joints, hand events, and gravity directions.Source: §3.2.2 · p. 5

3.2.3 Stage 3: Downstream Fine-Tuning

  • Downstream fine-tuning combines robot demonstrations with egocentric demonstrations paired with SONIC supervision and simulated-G1 pseudo-states.For the cola task, the visual pipeline removes human arms, inpaints the scene, and composites rendered robotic hands while retaining original action targets, states, and instructions.Source: §3.2.3 · p. 5

4 Experimentation

The experiments first refine HuGe’s training strategy and egocentric pre-training on HumanoidArena, then study real-world task adaptation on the Unitree G1.

4 Experimentation

  • HumanoidArena experiments examine training strategies, pseudo-proprioceptive states during pre-training, and egocentric pre-training data scale.These simulations are used to iteratively refine the training strategy and pre-training recipe.Source: §4 · p. 5
  • Real-world experiments on the Unitree G1 focus on robot-versus-egocentric demonstration composition and visual adaptation of egocentric observations.The evaluated task is the cola pick-and-drop task.Source: §4 · p. 5
  • Implementation details for the experiments are provided in Appendix A.Source: §4 · p. 5

4.1 Simulation Evaluation

The simulation evaluation tests training strategies and egocentric pre-training adaptations before comparing HuGe variants on HumanoidArena. Full fine-tuning performs best among the tested strategies, while HuGe-GR00T and HuGe-Pi exceed the strongest listed baseline overall.

4.1 Simulation Evaluation

  • Evaluation covers seven HumanoidArena tasks—three HOI and four HSI—with 100 demonstrations per task and 60 evaluation episodes per task.The study reports task success rates and unweighted averages for HOI, HSI, and all seven tasks.Source: §4.1 · p. 5

4.1.1 Training Strategy

  • 55.00%, 58.33%, and 60.24% average success rates for HuGe-OFT, HuGe-GR00T, and HuGe-Pi make full fine-tuning outperform LoRA and frozen-backbone training.LoRA also outperforms frozen-backbone training for all three variants; full fine-tuning is used in subsequent experiments.Source: §4.1.1 · pp. 5–6, Table 2 · p. 6

4.1.2 Pseudo Proprioceptive States

  • Pseudo-proprioception raises average success by 5.00 and 2.86 points for HuGe-GR00T and HuGe-Pi, but lowers HuGe-OFT from 56.43% to 54.05%.For HuGe-OFT, textual state serialization and input truncation leave incomplete descriptions that introduce ambiguity into action prediction.Source: Table 3 · p. 6, §4.1.2 · p. 6

4.1.3 Scaling Egocentric Pre-Training

  • With the full egocentric corpus, HuGe-GR00T and HuGe-Pi reach average performance of 66.67% and 65.00%, while HuGe-OFT declines at larger scale.The reported decline is consistent with the state-truncation issue described for HuGe-OFT.Source: Figure 2 · p. 6, §4.1.3 · p. 6

4.1.4 Performance Comparison

  • 66.67% and 65.00% average success rates for HuGe-GR00T and HuGe-Pi exceed the strongest baseline, π0.5, by 12.86 and 11.19 percentage points.Both variants outperform listed baselines in HOI and HSI averages; HuGe-OFT leads Boxing and VisNavi at 73.33% each, while all variants score 0%–15.00% on DoubleDesk.Source: Table 4 · p. 7, §4.1.4 · p. 7

4.2 Real-World Evaluation

The G1 pick-and-drop evaluation tests whether egocentric demonstrations and visual adaptation improve downstream fine-tuning. Visual adaptation replaces human hands and arms with rendered robot hands, and adapted demonstrations are associated with higher task success.

4.2 Real-World Evaluation

  • The task combines locomotion, grasping, and object placement, and randomized bottle positions assess generalization across placements.The study uses the π-style HuGe architecture and reports success over 30 trials per configuration.Source: §4.2 · p. 7

4.2.1 Visual Adaptation

  • Visual adaptation removes visible human hands and arms, inpaints the exposed regions, and composites simulation-rendered robot hands while preserving action, state, and language supervision.Feature-distribution Fréchet distance fell from 2.0311 to 1.3499 (33.54%) for Wan2.2, from 7.5697 to 5.5204 (27.07%) for VideoMAE V2 [49], and from 305.5143 to 263.2697 (13.83%) for V-JEPA 2 [1].Source: §4.2.1 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8

4.2.2 Downstream Fine-Tuning

  • 66.7% (20/30) success with visually adapted egocentric demonstrations exceeded 60.0% (18/30) with raw egocentric demonstrations and 43.3% (13/30) with 110 robot demonstrations alone.Each augmented condition used 400 egocentric demonstrations; the task required walking, grasping, turning, and placing a bottle, with randomized bottle positions.Source: §4.2 · p. 7, §4.2.2 · p. 8, Figure 3 · p. 8
  • Bottle pickup accounted for five of the ten failed trials and was the largest source of failure; the authors attributed some grasp failures to balance instability during manipulation.Three execution sequences offer qualitative evidence of adaptation to a bottle knocked over during grasping and generalization across different tabletop positions.Source: §4.2.2 · p. 9

5 Conclusion

HuGe combines egocentric human and robot demonstrations in a three-stage training pipeline, with pseudo-proprioceptive states and visual adaptation supporting transfer to humanoid policies. The paper reports support from HumanoidArena experiments and a real-world G1 pick-and-drop task, alongside statements on AI use, ethics, and reproducibility.

5 Conclusion

  • Experiments on HumanoidArena and a real-world G1 pick-and-drop task support the value of egocentric human experience for whole-body policy learning.HuGe uses a three-stage pipeline and incorporates pseudo-proprioceptive states and visual adaptation to support human-to-humanoid transfer.Source: §5 · p. 9

AI Use Statement

  • The authors report using generative AI for wording and language polishing, specified code implementation, and two Stage 3 Figure 1 illustrations, but not for research questions, methodology, experiments, or result interpretation.The authors take responsibility for the final manuscript and research artifacts.Source: AI Use Statement · p. 9

Ethics Statement

  • Newly collected human demonstrations were provided by the authors, and a suspension safety frame provided fall protection during physical robot experiments.The authors state they will provide processing scripts for existing datasets, not redistribute the processed data.Source: Ethics Statement · p. 9

Reproducibility Statement

  • The paper and appendices document architectures, action representations, data processing, training, evaluation settings, trial counts, and success-rate computation to support reimplementation and verification.Appendix A supplies training and hardware details; Appendix B specifies the proprioceptive state representation and normalization, and Appendix C provides task instructions.Source: Reproducibility Statement · p. 9

A Implementation Details

The implementation uses a shared image pipeline and training setup across HuGe variants, while downstream configurations hold robot demonstrations fixed and balance expected sampling probability across robot and egocentric samples.

A Implementation Details

  • HuGe variants jointly fine-tune the Qwen3-VL-2B-Instruct backbone, projection layers, and action modules while keeping the SONIC controller fixed.Distributed training uses DeepSpeed [40] on one node with eight NVIDIA 4090 GPUs, each with 48 GB memory; Stage 1 and Stage 2 take approximately 38 and 12 hours, respectively.Source: A Implementation Details · pp. 13–14
  • Table 5 is identified as the source of HuGe training hyperparameters, but the supplied passage does not specify the table values.The implementation passage reports the overall training durations separately.Source: Table 5 · p. 14, A Implementation Details · p. 14
  • All stages resize processed RGB images to 256 × 256, with random cropping and color jitter during training and center cropping without jitter at inference.Images are first aspect-ratio-preserving downsampled, square-padded, and resized; training crops 243 × 243 and resizes back to 256 × 256.Source: A Implementation Details · p. 14
  • Across all three real-world configurations, robot demonstrations remain fixed; in augmented configurations, robot and egocentric sample starts receive equal expected sampling probability.Raw and visually adapted egocentric videos use the same action, state, and language supervision, and all conditions share normalization statistics pooled from preceding training data.Source: A Implementation Details · p. 14

B Proprioceptive State Representation

HuGe conditions action-chunk prediction on a single 34-dimensional G1 state at the starting time index. The appendix describes hand-state and gravity components and normalizes each state dimension using pooled percentile bounds.

B Proprioceptive State Representation

  • Each HuGe variant conditions action-chunk prediction on one 34-dimensional G1 state at the starting index t.The state representation includes left and right hand states, encoded as 0 for open and 1 for closed, and a unit gravity direction in the body frame.Source: B Proprioceptive State Representation · p. 14
  • Each state dimension is clipped and scaled using its 1st and 99th percentile bounds, computed per dimension from pooled training frames across cumulative stages.At inference, the Stage 3 normalization statistics saved with the checkpoint are reused.Source: B Proprioceptive State Representation · p. 14

C Task Instructions

The section identifies task instructions for seven HumanoidArena tasks [50] and a real-world cola pick-and-drop task. Table 7 contains the Stage 3 instructions.

C Task Instructions

  • Table 7 lists instructions for seven HumanoidArena tasks [50] and the real-world cola pick-and-drop task.The table is identified as covering Stage 3 instructions.Source: C Task Instructions · p. 14, Table 7 · p. 15
  • The G1 proprioceptive state used by HuGe has 34 dimensions, including joint angles, hand states, and body-frame gravity-direction components.Joint positions are in radians; hand states encode open as 0 and closed as 1.Source: Table 6 · p. 15
  • Table 7 is captioned as the source of task instructions for Stage 3.Source: Table 7 · p. 15

Related papers

Loading 2610.34186v1…