Source-linked AI summary
HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining
Juncheng Ma, Jianxin Bi, Yufan Deng, Xuanran Zhai, Kewei Zhang, Ye Huang, Bo Liang, Shukai Gong, Jiankai Tu, Xiaotian Tang, Jiaxin Li, Kaiqi Chen, Duomin Wang, Yuqi Wang, Bingyi Kang, Eric Huang, Zhiyang Dou, Zhen Dong, Enze Xie, Wojciech Matusik, Tat-Seng Chua, Daquan Zhou
TL;DR
Embodied pretraining is constrained by the cost and limited diversity of teleoperated robot data, motivating a comparison with scalable egocentric video. The paper conducts a controlled matched-scale study with identical post-training and evaluation, finding stronger generalization from egocentric pretraining, while noting current scope limits.
Problem
Teleoperated robot data offers precise action supervision and embodiment alignment but has limited scalability, cost, and behavioral and environmental diversity.
Method
The paper compares matched-scale egocentric and multi-embodiment robot pretraining using the same world-action model, identical real-robot post-training, and held-out Seen and Unseen evaluations.
Results
Egocentric pretraining achieves stronger generalization than real-robot pretraining, with the largest gains on unseen tasks.
Takeaways & Limitations
Egocentric video can provide scalable, diverse pretraining while post-training addresses its embodiment gap with kinematically aligned robot data.
Takeaways & Limitations
Current training is limited to 5,000 hours and evaluations focus mainly on world-action models with WAN2.2 as the video-generation backbone.
Abstract
from arXiv · showhide
Embodied foundation models are expected to benefit from data scaling like large language models, but face a much tighter data bottleneck. Teleoperated real-robot trajectories remain the dominant pretraining source due to their precise action supervision and embodiment alignment, yet their scalability is limited by high collection cost, acquisition difficulty, and low behavioral and environmental diversity. These limitations have sparked interest in egocentric human video as a scalable, substantially lower-cost, and more diverse alternative for embodied model pretraining. However, its effectiveness compared to teleoperated real-robot data remains underexplored. To address this question, we conduct a systematic study comparing egocentric human video and teleoperated real-robot trajectories as pretraining data sources for embodied foundation models, under fixed post-training and validation protocols. Surprisingly, we find that egocentric data, when processed through a carefully designed filtering and labeling pipeline, is not merely a viable substitute for model pretraining but can lead to superior performance. With the same amount of pretraining data, models pretrained on egocentric data achieve a 24% lower validation loss on real-robot action prediction, as well as 52.5% and 90% higher success rates on in-distribution and out-of-distribution real-robot task execution, respectively. This finding verifies a scalable paradigm for embodied foundation models: pretrain on egocentric human video to learn diverse world representations, then adapt with a small amount of labeled real-robot data for action-space alignment. We hope this study encourages broader exploration of egocentric data and offers guidance for data quality assessment before costly robot data collection.
PKU, NUS, MIT, UCSB, NVIDIA
The passage identifies equal contribution and corresponding-author roles.
- The authors mark equal contribution with an asterisk.
- The authors mark corresponding authors with a dagger.
- The contribution and correspondence markers are presented together.
1 Introduction
The paper asks whether scalable egocentric human video can outperform costly, less diverse teleoperated robot data for embodied pretraining. A controlled comparison finds stronger generalization from egocentric pretraining, especially on unseen tasks.
- Teleoperated robot trajectories provide action supervision and embodiment alignment but face limited scale, cost, and diversity.
- Egocentric video exposes models to contact-rich interactions, tool use, object changes, and long-horizon behaviors at greater scale.
- The study compares matched-scale egocentric and real-robot pretraining using the same world-action model and post-training data.
- Egocentric pretraining scales consistently as its data amount increases.
- Egocentric pretraining achieves stronger downstream performance than real-robot pretraining, with the largest gains on unseen tasks.
- Egocentric-pretrained policies remain robust under unseen-object shifts, whereas the no-pretraining baseline collapses on real-robot rollouts.
2 Egocentric vs. Real-Robot Data: Coverage, Cost, and Alignment
Egocentric video offers greater scale, lower collection cost, and broader motion, interaction, and scene diversity than real-robot data, while robot data retains stronger embodiment alignment.
- Embodied pretraining benefits from broad coverage of scenes, objects, interactions, and behaviors, whereas post-training adapts models to a specific embodiment and task distribution.
- Public real-robot data totals only ∼2 × 10^4 hours, while egocentric releases can exceed the aggregated robot supply several-fold.
- Robot teleoperation requires costly hardware, operators, staged scenes, and maintenance, while egocentric collection uses passive off-the-shelf cameras.
- Motion quality: Egocentric trajectories are smoother and contain substantially less action idle time than real-robot trajectories at matched sampled duration.
- Diversity: Egocentric data covers broader workspaces, greater inter-session variation, richer interaction vocabularies, and wider visual scenes than real-robot data.
- Egocentric video favors scale, marginal cost, motion diversity, interaction diversity, and scene diversity, while robot data favors embodiment alignment.
3 Embodied Pretraining with Egocentric Human Video
The study pretrains matched world-action models on egocentric or real-robot data, then uses identical real-robot post-training and held-out evaluations to compare scaling and generalization.
- The world-action model combines video-dynamics prediction and action inference through a Mixture-of-Transformers architecture.
- The comparison holds post-training data, compute budget, and evaluation protocol fixed while varying the pretraining substrate.
- The two pretraining sets are matched at 5,000 hours but differ in collection source and action-label properties.
- Egocentric data uses retargeted hand-pose signals for pseudo-action labels, while robot data provides precise kinematically aligned actions.
- Post-training uses 1,500 real-robot trajectories across 15 manipulation tasks.
- Evaluation measures validation flow-matching action loss on Seen and Unseen held-out splits, with Unseen tasks testing out-of-distribution generalization.
- Figure 3 plots best post-training action loss against pretraining hours for egocentric and matched-scale robot pretraining.
4 Experimental Results and Analysis
The experiments compare egocentric and real-robot pretraining under matched scales and fixed post-training protocols. Egocentric pretraining scales consistently and provides stronger unseen-task generalization, including in real-robot rollouts.
- Egocentric pretraining is evaluated against real-robot pretraining through two questions: whether it scales with data and how it compares at matched scale.
- Egocentric scaling: 35% and 24% lower loss: scaling egocentric pretraining from 100 to 5,000 hours reduces seen-task loss from 0.0080 to 0.0067 and unseen-task loss from 0.0234 to 0.0204.The scaling trend is well captured by a log-linear law, with R2 = 0.86 for seen tasks and R2 = 0.94 for unseen tasks.
- Matched-scale comparison: 0.0067 versus 0.0071: at 5,000 hours, egocentric and real-robot pretraining achieve similarly low loss on seen tasks.Egocentric pretraining reaches 0.0067, compared with 0.0071 for real-robot pretraining.
- Matched-scale comparison: 20% lower loss: egocentric pretraining reaches 0.0204 on unseen tasks at 5,000 hours, versus 0.0254 for real-robot pretraining.Real-robot loss remains near 0.025 across scales, while egocentric loss continues improving.
- Scaling factors: Broader diversity supports scaling because egocentric data covers more objects, skills, environments, and behavioral space than constrained robot collections.This broader coverage supports both unseen-task generalization and a less task-specific initialization for seen-task adaptation.
- Real-world validation: 92.5% and 90.0%: the egocentric-pretrained model succeeds on in-distribution and shifted real-robot trials, while the baseline reaches 40.0% and 0%, respectively.The egocentric model drops only 2.5 points under distribution shift, compared with a 40-point baseline collapse.
5 Related Work
Prior embodied-learning work has scaled real-robot data, explored egocentric video, and developed VLA and world-action architectures. This paper positions itself as the first controlled matched-scale comparison showing egocentric pretraining can surpass real-robot pretraining, especially for OOD generalization.
- Real-robot data: Teleoperated robot data provides kinematic alignment but is costly to collect and limited in background, object, interaction, and OOD diversity.
- Egocentric data: Egocentric corpora have supported recognition and representation learning, while newer work uses them for co-training, motion retargeting, trajectory priors, and manipulation pretraining.
- Research gap: The paper identifies a gap in prior work: no head-to-head comparison of egocentric and real-robot pretraining at matched scale under a controlled post-training benchmark.
- Paper contribution: Egocentric data is reported to substitute for and surpass real-robot pretraining, with the largest gains on OOD generalization.The study establishes scaling curves for an autoregressive world-action model.
- Architectures: VLA models attach action heads to pretrained vision-language backbones, whereas world-action models jointly model future states and actions.The cited world-action examples differ in whether video and actions are generated jointly or sequentially.
6 Conclusion
The paper presents a controlled, matched-scale comparison showing that egocentric pretraining surpasses real-robot pretraining under identical post-training and evaluation protocols, especially for out-of-distribution generalization. The authors characterize the findings as encouraging but preliminary and note current limits on data scale and evaluation scope.
- Egocentric video leads real-robot data on pretraining-relevant scale, cost, and diversity, while post-training is intended to address its embodiment gap.
- Egocentric pretraining surpasses real-robot pretraining under identical post-training and evaluation protocols, with the largest gains on out-of-distribution generalization.
- The conclusions remain preliminary because the current training scheme is limited to 5000 hours and evaluations focus mainly on world-action models with WAN2.2.