Source-linked AI summary
Humanoid-VLA: Towards Universal Humanoid Control with Visual Integration
Pengxiang Ding, Jianfei Ma, Xinyang Tong, Binghong Zou, Xinxin Luo, Yiguo Fan, Ting Wang, Hongchao Lu, Panzhong Mo, Jinxin Liu, Yuefan Wang, Huaicheng Zhou, Wenshuo Feng, Jiacheng Liu, Siteng Huang, Donglin Wang
TL;DR
Humanoid-VLA addresses reactive humanoid control and limited egocentric training data by combining language-motion alignment, visual conditioning, and self-supervised pseudoannotations. Built with whole-body control, it demonstrates adaptive object interaction and environment exploration, while high-quality humanoid data remains limited.
Problem
Existing humanoid controllers are mainly reactive, while motion datasets lack synchronized egocentric vision and sufficient high-quality data for foundation-model training.
Method
Humanoid-VLA pre-aligns language and motion, adds egocentric visual context through cross-attention, generates motion-derived pseudoannotations, and integrates a whole-body controller.
Results
Humanoid-VLA facilitates adaptive object interaction and environment exploration, with evaluations reporting motion-generation quality and execution success on real humanoid robots.
Takeaways & Limitations
The framework provides a unified approach to language understanding, egocentric scene perception, and humanoid motion control with enhanced contextual understanding.
Takeaways & Limitations
High-quality manually annotated and real-robot execution data remain limited, and some existing datasets are unsuitable for general robotics because of restricted robot configurations.
Abstract
from arXiv · showhide
This paper addresses the limitations of current humanoid robot control frameworks, which primarily rely on reactive mechanisms and lack autonomous interaction capabilities due to data scarcity. We propose Humanoid-VLA, a novel framework that integrates language understanding, egocentric scene perception, and motion control, enabling universal humanoid control. Humanoid-VLA begins with language-motion pre-alignment using non-egocentric human motion datasets paired with textual descriptions, allowing the model to learn universal motion patterns and action semantics. We then incorporate egocentric visual context through a parameter efficient video-conditioned fine-tuning, enabling context-aware motion generation. Furthermore, we introduce a self-supervised data augmentation strategy that automatically generates pseudoannotations directly derived from motion data. This process converts raw motion sequences into informative question-answer pairs, facilitating the effective use of large-scale unlabeled video data. Built upon whole-body control architectures, extensive experiments show that Humanoid-VLA achieves object interaction and environment exploration tasks with enhanced contextual awareness, demonstrating a more human-like capacity for adaptive and intelligent engagement.
1. Introduction
Humanoid-VLA addresses reactive humanoid control and scarce egocentric training data by aligning language with motion, adding visual context, and generating self-supervised pseudoannotations. Integrated with whole-body control, it supports autonomous object interaction and environment exploration.
- Current humanoid controllers achieve high-fidelity motion but remain primarily reactive and cannot autonomously infer interaction targets.
- Egocentric visual training is constrained because motion datasets lack synchronized first-person information and teleoperation is prohibitively expensive.
- Humanoid-VLA pre-aligns language and motion using non-egocentric human datasets, then fuses egocentric visual features through parameter-efficient cross-attention.
- Self-supervised motion analysis masks body joints, reconstructs occluded movements, and converts raw sequences into question-answer pseudoannotations.
- The unified system combines language understanding, scene perception, and motion control to enhance autonomous interaction in real-world environments.
2. Related Works
Prior humanoid-control and dataset efforts provide useful motion capabilities but remain limited by adaptability, egocentric visual integration, annotation scale, or acquisition cost. Humanoid-VLA is positioned against these constraints through language, vision, and self-supervised data augmentation.
- Traditional methods offer accuracy and stability, whereas learning-based methods provide flexibility but depend on limited humanoid motion data.
- Existing human-motion datasets provide scale, but humanoid retargeting datasets often have sparse text annotations and limited scale.
- Humanoid-VLA uses self-supervised augmentation to exploit large-scale unlabeled motion data without manually annotated textual descriptions.
3. Humanoid-VLA
Humanoid-VLA integrates language, motion, and egocentric vision to address the limited generalization and data scarcity of prior humanoid control approaches. It combines compositional motion representation, self-supervised augmentation, and vision-conditioned learning with whole-body control.
- Motivation: Prior humanoid control methods mainly imitate poses without egocentric vision, limiting access to task intent and generalization across environments.Existing datasets also lack synchronized first-person visual information, while teleoperation is too costly for large-scale data collection.
- Humanoid-VLA: Humanoid-VLA aligns non-egocentric motion data with language to learn motion patterns and action semantics before adding visual context.The framework has two main stages: language-motion pre-alignment and vision-conditioned fine-tuning.
- Motion Representation: The method decomposes each body pose into five body-part tokens and independently encodes them into a quantized representation for flexible motion manipulation.The parts are the left leg, right leg, torso, left arm, and right arm; token-level operations can replace, perturb, or rearrange body-part representations.
- Data Acquisition: Self-supervised augmentation converts raw motion into instructional question-answer pairs using tasks such as tracking, timing, occlusion, and state reconstruction.These tasks derive semantic supervision directly from motion, including masking joints and reconstructing the missing movement.
- Data Acquisition: The augmentation strategy produced a motion-language interleaved dataset 25 times larger than previous work.The authors present this expansion as addressing data scarcity for training foundational human motion models.
- Vision and Control: The resulting system combines learned motion knowledge with egocentric visuals for visually grounded humanoid interaction and control.Real-world motion capture paired with egocentric visuals supports transfer to object-aware scenarios, while the model is designed to integrate with whole-body control.
4. Experiments
Humanoid-VLA is evaluated for motion quality, physical plausibility, and vision-integrated real-world control. Results indicate strong kinematic fidelity, physically consistent motion, and successful visual object interaction and obstacle avoidance.
- Evaluation scope: The experiments assess kinematic accuracy, physical plausibility, and vision-integrated humanoid control across increasingly complex conditions.The evaluation includes motion-generation metrics, physics-simulator tracking, and real-world visual tasks.
- Kinematic fidelity: Humanoid-VLA achieves an FID of 0.467 on HumanML3D, improving over MDM by 47.5% and T2M-GPT by 12%.FID measures distribution similarity between generated and real motions, with lower values indicating better matching.
- Kinematic fidelity: On Humanoid-S, Humanoid-VLA reaches a Diversity score of 4.466, outperforming MDM by 6%.The result reflects motion variation under challenging linguistic constraints.
- Physical plausibility: The whole-body RL policy keeps mean position error below 40 mm, with 31.07 mm under medium difficulty with caption and time conditions.The policy is evaluated by tracking generated kinematic trajectories in the IsaacGym physics simulator.
- Physical plausibility: The policy reports PA-MPJPE of 1.18 mm, acceleration error of 27.84 mm, and velocity error of 14.76 mm.These transition-related and pose-accuracy metrics are used to assess physically plausible motion, where lower values are better.
- Data augmentation: Adding extensive video motion data reduces FID from 0.557 to 0.467, a 16% improvement in the data-augmentation ablation.The comparison supports using large-scale video-derived motion data for motion-language alignment.
- Vision integration: In real-world tests, the model uses visual information to approach and kick a ball and navigate around obstacles toward target positions.Each task category includes four representative tasks and ten tests per task, evaluated by success rate.
5. Conclusion
Humanoid-VLA aligns language and motion, adds egocentric visual integration, and addresses data scarcity with self-supervised pseudo-annotations. Built on whole-body control, it supports adaptive object interaction and exploration with enhanced contextual understanding.
- Humanoid-VLA aligns language and motion using human motion datasets, incorporates egocentric visual context through cross-attention, and addresses data scarcity with self-supervised pseudo-annotations.
- Built on whole-body control architectures, the framework facilitates adaptive object interaction and environmental exploration with enhanced contextual understanding.
- The framework’s effectiveness was validated through motion-generation quality and execution-success evaluations on real humanoid robots, demonstrating high executability.
A. Data collection
The motion dataset combines motion capture, online videos, and synthetic data, while standardized joint selection and linguistic augmentation improve flexibility and task-description diversity.
- The motion dataset draws from AMASS motion capture, online videos with extracted human motions, and synthetic movements generated from random text.
- The framework selects 15 universal humanoid joint points from the standard 22 SMPL joints used in the open-source datasets.
- GPT-4 instruction rephrasing expands linguistic diversity, while augmentation types can combine with other conditions to create complex tasks.
B. Data templates
The data templates define varied conditional language descriptions for subtasks, enabling systematic expansion of task descriptions; Table 6 illustrates the template format.
- Table 6 presents examples of conditional language descriptions that can be expanded N times from the first example.
- Each subtask category receives N expression variations, expanding the original 59 subtasks by a factor of N.
C. Simulation Performance
The evaluation covers representative object-interaction tasks in real-world and simulation settings. Humanoid-VLA also demonstrates adaptability across distinct robot configurations in simulation and real-world implementations.
- The study evaluates two representative HITR object-interaction tasks using egocentric frames, natural-language instructions, and robot control signals.
- The framework generalizes across distinct robot architectures in both simulation and real-world implementations.
D. More Details
The system uses a universal 15-joint representation and optimization-based mapping to humanoid joints, while defining motion imitation as a goal-conditioned reinforcement-learning problem.
- Universal joint representation: The model generates motion using 15 joints shared by humans and humanoid robots to support universality across configurations.Training data are extracted using the same consistent joint set.
- Joint mapping: An Adam-optimization step maps the 15 generated joints onto the humanoid robot’s 24 joints while preserving end-effector alignment.The mapping maintains the overall motion pattern represented by the keypoints.
- Reinforcement-learning formulation: The control problem is formulated as learning a goal-conditioned RL policy that maps human motion onto a humanoid robot’s joints.The policy operates over 24-joint robot states.
- Reinforcement-learning formulation: The reward function takes observations and goals as input and outputs target positions for a proportional-derivative controller, optimized with PPO.PPO maximizes the accumulated reward for the control policy.
E. Limitation
The paper identifies limited policy robustness, insufficient high-quality data, and a relatively simple training methodology as limitations requiring future refinement.
- Robustness of the RL Policy: The RL policy lacks sufficient robustness, and the authors plan further refinement to improve task completion.This limitation concerns the policy’s reliability in completing tasks.
- Limited Availability of High-Quality Data: High-quality manually annotated and real-world humanoid execution data remain limited for general robotics tasks.Existing datasets such as Mimicking-Bench are unsuitable because of restricted robot configurations.
- Training Approach: The current training methodology is relatively simple and does not fully exploit available data.The authors identify additional motion-generation training strategies for future work.