Source-linked AI summary
EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos
Ruihan Yang, Qinxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Xuxin Cheng, Ri-Zhao Qiu, Hongxu Yin, Sifei Liu, Song Han, Yao Lu, Xiaolong Wang
TL;DR
Robot-data collection limits the scale of imitation-learning datasets, motivating the use of egocentric human videos for VLA training. EgoVLA predicts human wrist and hand actions, aligns them with robot actions, and fine-tunes on limited robot demonstrations. On the proposed humanoid benchmark, it achieves strong performance across diverse tasks and generalization settings, while still requiring moderate task-specific robot data.
Problem
Robot hardware and expert operators constrain the scale of manipulation data, while VLA training typically requires extensive robot datasets.
Method
EgoVLA pretrains a VLA on egocentric human manipulation videos, uses a unified human-robot action space, and fine-tunes on a small set of robot demonstrations.
Results
EgoVLA outperforms specialist and generalist baselines across short- and long-horizon humanoid manipulation tasks and generalizes across visual observations and spatial locations.
Takeaways & Limitations
Human video pretraining enables a generalist manipulation policy with strong performance across diverse tasks using limited robot data.
Takeaways & Limitations
EgoVLA requires hand and wrist pose annotations and moderate robot-data fine-tuning, limiting data availability and direct zero-shot deployment.
Abstract
from arXiv · showhide
Real robot data collection for imitation learning has led to significant advancements in robotic manipulation. However, the requirement for robot hardware in the process fundamentally constrains the scale of the data. In this paper, we explore training Vision-Language-Action (VLA) models using egocentric human videos. The benefit of using human videos is not only for their scale but more importantly for the richness of scenes and tasks. With a VLA trained on human video that predicts human wrist and hand actions, we can perform Inverse Kinematics and retargeting to convert the human actions to robot actions. We fine-tune the model using a few robot manipulation demonstrations to obtain the robot policy, namely EgoVLA. We propose a simulation benchmark called Ego Humanoid Manipulation Benchmark, where we design diverse bimanual manipulation tasks with demonstrations. We fine-tune and evaluate EgoVLA with Ego Humanoid Manipulation Benchmark and show significant improvements over baselines and ablate the importance of human data. Videos can be found on our website: https://rchalyang.github.io/EgoVLA
1 Introduction
EgoVLA addresses the scale and diversity limits of robot-data collection by learning manipulation from egocentric human videos and transferring predicted human actions to robots. It is evaluated with a new humanoid benchmark and reported to outperform specialist and generalist baselines across short- and long-horizon tasks.
- Robot hardware and expert teleoperation fundamentally constrain the scale of manipulation data collection.
- Egocentric human videos offer greater training-data scale and more diverse tasks, scenes, environments, and interactions than robot data alone.
- EgoVLA predicts future human wrist poses and hand joint actions, then transfers them to robots using inverse kinematics and hand retargeting.
- The model is fine-tuned with a few robot demonstrations to correct transfer errors without requiring large-scale robot training data.
- EgoVLA outperforms specialist and generalist baselines on short- and long-horizon tasks and generalizes across visual observations and spatial locations.
2 Related Work
Prior work established strong multimodal models and learning-based dexterous manipulation, but scalable VLA training and fine-grained human-video supervision remain limited. EgoVLA combines egocentric manipulation data with a VLA framework to address these gaps.
- Dexterous Manipulation: Dexterous manipulation research has progressed from control-based to learning-driven methods, yet large-scale manipulation across diverse scenarios remains an open challenge.
- Vision-Language-Action (VLA): Vision-language-action models integrate perception and action but typically require extensive teleoperation, scripted execution, or crowdsourced robot datasets.
- Egocentric Vision: Existing egocentric vision datasets improve coverage but often focus beyond robotic capabilities or omit pose annotations needed for dexterous manipulation learning.
- Learning from In-the-Wild Video: Prior human-video representation learning generally lacks fine-grained hand and wrist pose information and often uses unsupervised objectives.
- Learning from In-the-Wild Video: EgoVLA instead uses high-quality egocentric manipulation data within a VLA framework to improve dexterous policies.
3 Learning Manipulation Skills from Ego-centric Human Videos
EgoVLA is pretrained on a pose-annotated egocentric human manipulation dataset, predicts human wrist and hand actions, and transfers them to humanoid robots through a unified action space. Robot demonstrations support fine-tuning and deployment through geometric alignment, inverse kinematics, and hand-command mapping.
- 3.1 Ego-Centric Human Manipulation Dataset: The egocentric human manipulation dataset combines RGB observations, wrist poses, hand poses, and camera poses from four sources.
- 3.1 Ego-Centric Human Manipulation Dataset: World-frame camera poses project future wrist positions into the current camera frame, producing consistent supervision across continuously moving egocentric views.
- 3.1 Ego-Centric Human Manipulation Dataset: The dataset contains approximately 500,000 image-action pairs sampled at 3 FPS across diverse manipulation tasks.
- 3.2 EgoVLA Model: EgoVLA uses visual history, language instructions, human proprioception, and action-query tokens to predict future human or robot actions with a VLM backbone and action head.
- 3.2 EgoVLA Model: Each action predicts camera-frame wrist translation and rotation plus hand angles represented by the top 15 PCA components of MANO.
- 3.2 EgoVLA Model: The training objective combines wrist-translation, wrist-rotation, and hand-joint losses weighted by λ_wrist trans, λ_wrist rot, and λ_joint.
- 3.3 Transferring EgoVLA to Humanoid Robot: Robot hand configurations are aligned to MANO by minimizing fingertip-position discrepancies, enabling direct fine-tuning without architectural changes or reinitialization.
- 3.3 Transferring EgoVLA to Humanoid Robot: At deployment, predicted wrist poses are converted to robot end-effectors through transformations and inverse kinematics, while MANO keypoints are mapped to hand commands by a lightweight MLP.
4 Ego Humanoid Manipulation Benchmark
The Ego Humanoid Manipulation Benchmark provides a controlled, reproducible testbed for evaluating humanoid bimanual manipulation across diverse tasks and visual conditions.
- Benchmark Design: The benchmark uses a Unitree H1 humanoid with two Inspire dexterous hands and includes 12 short- and long-horizon manipulation tasks.Tasks range from atomic actions such as pushing and opening drawers to multi-stage skills such as sorting and inserting cans.
- Observation and Action Space: Evaluation observations include robot joint positions, end-effector poses, contact forces, and egocentric RGB-D input, while EgoVLA uses a subset of these modalities.The robot uses end-effector control for arms and PD joint control for hands.
- Visual Diversity: Five room textures and five table textures generate 25 visual background combinations for generalization evaluation.Simulation enables direct control over visual conditions.
- Demonstrations: Each task has 100 successful expert demonstrations collected with a Meta Quest 3, using three training room textures and a fixed table texture.Episode lengths range from 100 to 500 frames depending on task complexity.
- Evaluation: The benchmark evaluates manipulation policies using short- and long-horizon task results across seen and unseen visual configurations.The supplied table caption identifies the short-horizon evaluation split; the benchmark description supplies the broader evaluation setting.
5 Experiments
Experiments evaluate human motion modeling and humanoid manipulation under seen and unseen conditions. EgoVLA benefits from human-video pretraining, but still depends on task-specific robot data and shows weaker performance under domain shifts.
- 5.1 Human Manipulation Modeling: EgoVLA models human wrist motion with approximately 8 cm future translation error and around 0.13 normalized 2D image-plane error.Modified language instructions with unchanged visual inputs shift predicted trajectories toward the instructed behavior.
- Evaluation Metrics: Success Rate measures overall task completion, while Progress Rate measures completed subtasks relative to all subtasks in a long-horizon task.These metrics jointly capture final success and partial completion.
- Experimental Setup: The experiments compare EgoVLA with EgoVLA-NoPretrain and task-specific ACT specialists under Seen and Unseen visual backgrounds.Object positions are randomized during rollouts, with separate rollout counts for seen and unseen backgrounds.
- Robot Data Requirement: Zero-shot humanoid deployment without robot-data fine-tuning achieves 0% success across all tasks because of appearance, perception, and kinematic mismatches.Robot-specific post-training remains necessary even for tasks observed during human-video pretraining.
- In-Domain Performance: EgoVLA consistently outperforms EgoVLA-NoPretrain on seen short- and long-horizon tasks, with approximately 20% higher success rates on long-horizon tasks.Generalist models also substantially outperform specialist ACT baselines across both task horizons.
- Out-of-Domain Generalization: On unseen backgrounds, EgoVLA has only a minor short-horizon success-rate drop, while EgoVLA-NoPretrain declines by 23%; long-horizon success is around 30%.Long-horizon progress remains similar despite lower success, suggesting failures occur mainly near final task stages.
- Data Mixture Ablation: Increasing the scale and diversity of human pretraining data consistently raises average Success Rate and Progress Rate across tasks.Positive transfer persists despite noisy annotations, missing language labels, and limited visual diversity in source datasets.
- Trajectory Visualization: EgoVLA successfully executes diverse long-horizon tasks with only 100 demonstrations per task while showing spatial and visual generalization.Trajectory visualizations include can insertion, unloading, saucer placement, and drawer closing.
6 Conclusion
EgoVLA is a vision-language-action model pretrained on egocentric human manipulation videos and fine-tuned with a small set of robot demonstrations. A unified action space aligns human and robot hand representations for dexterous manipulation across diverse tasks.
- EgoVLA is pretrained on a large-scale egocentric human manipulation dataset and fine-tuned on a small set of robot demonstrations.
- A unified action space aligns human and robot hand representations to enable transfer across embodiments.
- Human video pretraining enables a generalist manipulation policy with strong performance across diverse tasks and limited robot data.
- EgoVLA demonstrates strong generalization ability in dexterous manipulation.
7 Limitation
EgoVLA's pretraining requires human videos with hand and wrist pose annotations, which may restrict data availability. The pretrained model also requires further fine-tuning on a moderate amount of robot data before deployment.
- Human hand and wrist pose annotations are required for pretraining, potentially limiting the available training data.
- EgoVLA cannot be directly deployed for manipulation without further fine-tuning on a moderate amount of robot data.
- Future work may improve zero-shot transferability through more embodiment-agnostic pretraining.
1 Dataset Details
The combined egocentric dataset includes visual observations, wrist and hand poses, camera poses, and language information, with harmonization across constituent datasets. Task distributions cover diverse human manipulation activities, while HOT3D lacks task labels and language annotations.
- The combined dataset contains egocentric RGB observations, wrist poses, hand poses, and camera poses.
- HoloAssist, TACO, and HOI4D provide clip-level language labels, whereas HOT3D lacks language annotations and receives placeholder instructions.
- The average MANO hand shape is used to focus the problem on learning manipulation skills from human demonstrations.
- Hand poses from HoloAssist are retargeted to the MANO representation for unified modeling.
- Task distributions are visualized for HOI4D and HoloAssist, but not HOT3D because its sequences lack task labels.
- All included datasets provide egocentric human videos spanning diverse tasks, with logarithmic frequency scales used for the distributions.
2 Training Details
Training uses a weighted objective for wrist translation, wrist rotation, and hand joint regression, with configurations specified for human-video pretraining and robot fine-tuning. The experiments used substantial GPU resources and separate ACT baseline settings.
- All models were trained using 32 A100 GPUs, with hyperparameters reported for human-video pretraining and robot fine-tuning.
- The full training objective combines weighted wrist translation, wrist rotation, and hand joint regression losses.
- Wrist translation regression uses an L2 loss between predicted and ground-truth wrist translations.
- Wrist rotation prediction uses a rotation loss after converting predicted rot6D representations into rotation matrices.
- Hand joint angle regression uses an L2 loss on predicted and ground-truth MANO parameters.
- The weighting coefficients balance wrist translation, wrist rotation, and joint-loss terms.
- ACT baselines used three RTX A4000 GPUs, modified DinoV2 visual backbones, randomized episode segments, and task-dependent action chunk sizes.
3 Detailed Action Formulation
EgoVLA predicts short-horizon human wrist and hand actions, retargets them to humanoid actuation, and is evaluated across diverse simulated manipulation settings and trajectories.
- Inference: EgoVLA predicts 30 actions per timestep at 30 Hz over a 1-second future window, with action-chunking and smoothing for deployment.The smoothing parameter is 0.8, and the simulation policy also operates at 30 Hz.
- Detailed MANO Hand Parameterization: The MANO hand action space uses the first 15 PCA components to compactly represent human hand pose.MANO otherwise models 45 hand degrees of freedom through 15 ball joints.
- Retargeting during deployment: A four-layer retargeting MLP maps both hands’ wrist-frame 3D fingertip positions to actuation values for all hand degrees of freedom.Its hidden-layer sizes are [64, 128, 64], and its training data comes from converting robot demonstrations into human hand representations.
- Human Prediction Visualization: Human trajectory visualizations show accurate, consistent predictions across unseen scenes, object instances, and tasks, including shared semantics across similar kettles.The predicted trajectories cover actions such as approaching, lifting, cutting, operating doors, and placing objects in drawers.
- Policy Trajectories Visualization: Policy visualizations show fundamental short-horizon skills such as pushing, grasping, and placing, alongside long-horizon compositions of atomic actions.Examples include stacking a can and closing a drawer, and sorting four cans into different containers.