Source-linked AI summary
EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
Ruijie Zheng, Dantong Niu, Yuqi Xie, Jing Wang, Mengda Xu, Yunfan Jiang, Fernando Castañeda, Fengyuan Hu, You Liang Tan, Letian Fu, Trevor Darrell, Furong Huang, Yuke Zhu, Danfei Xu, Linxi Fan
TL;DR
EgoScale addresses whether large-scale human data can support fine-grained, high-degree-of-freedom dexterous robot manipulation. It combines large-scale egocentric human pretraining with aligned human–robot mid-training, achieving strong long-horizon manipulation, one-shot adaptation, and transfer across embodiments.
Problem
Whether large-scale human data can serve as a primary training signal for high-degree-of-freedom dexterous robot manipulation remains unclear because robot data collection is costly and current hardware is limited.
Method
EgoScale pretrains a vision–language–action policy on over 20,854 hours of egocentric human manipulation data with explicit wrist and retargeted hand-action supervision, then applies lightweight aligned human–robot mid-training.
Results
EgoScale reveals a log-linear relationship between human data scale and action-prediction validation loss, whose strong correlation with real-robot performance accompanies long-horizon manipulation, one-shot adaptation, and transfer across robot embodiments.
Takeaways & Limitations
Large-scale human demonstrations provide a scalable and predictable supervision source, while explicit motion supervision and precise human–robot alignment amplify the effectiveness of limited robot data.
Takeaways & Limitations
The reported scaling trend is not extrapolated beyond the measured range, and its future gains remain contingent on jointly increasing human data and model capacity.
Abstract
from arXiv · showhide
Human behavior is among the most scalable sources of data for learning physical intelligence, yet how to effectively leverage it for dexterous manipulation remains unclear. While prior work demonstrates human to robot transfer in constrained settings, it is unclear whether large scale human data can support fine grained, high degree of freedom dexterous manipulation. We present EgoScale, a human to dexterous manipulation transfer framework built on large scale egocentric human data. We train a Vision Language Action (VLA) model on over 20,854 hours of action labeled egocentric human video, more than 20 times larger than prior efforts, and uncover a log linear scaling law between human data scale and validation loss. This validation loss strongly correlates with downstream real robot performance, establishing large scale human data as a predictable supervision source. Beyond scale, we introduce a simple two stage transfer recipe: large scale human pretraining followed by lightweight aligned human robot mid training. This enables strong long horizon dexterous manipulation and one shot task adaptation with minimal robot supervision. Our final policy improves average success rate by 54% over a no pretraining baseline using a 22 DoF dexterous robotic hand, and transfers effectively to robots with lower DoF hands, indicating that large scale human motion provides a reusable, embodiment agnostic motor prior.
1. Introduction
EgoScale investigates whether large-scale human data can support fine-grained dexterous robot manipulation, addressing prior work’s limited data scale and low-DoF focus. It combines large-scale human pretraining with aligned human–robot mid-training to improve dexterous transfer and adaptation.
- Human-to-robot dexterous transfer remains unclear because prior studies generally use tens to hundreds of human-video hours and focus on grippers or low-DoF hands.
- 20,854 hours of egocentric human manipulation data reveal a log-linear relationship between data volume and hand-action prediction validation loss.The dataset is over 20× larger than those used in prior human–robot transfer studies.
- Validation loss strongly correlates with real-robot performance on long-horizon, complex manipulation tasks, making large-scale human data a predictable supervision source.
- A two-stage recipe combines explicit wrist and retargeted high-DoF hand supervision with lightweight aligned human–robot mid-training to ground representations in robot sensing and control.Matched tasks, scenes, and viewpoints provide precise cross-domain alignment.
- With one robot demonstration, the policy achieves up to 88% average success on shirt folding, while lower-DoF Unitree G1 transfer improves success rate by over 30% absolute across both evaluated tasks.
- The results indicate that effective transfer requires scale, explicit motion supervision, and precise human–robot alignment, with rich human motion acting as a reusable motor prior.
2. Method
EgoScale converts egocentric human observations and hand motion into shared wrist and retargeted hand actions, then trains a flow-based VLA through large-scale human pretraining and aligned human–robot mid-training. Its datasets separate broad human-data scale from precise embodiment alignment.
- Human Action Representation: Egocentric RGB, estimated camera motion, and human hand pose are converted into a unified action representation for pretraining and robot execution.Human hand pose uses 21 keypoints represented as rigid transforms, with the wrist as keypoint 1.
- Human Action Representation: Relative wrist motion removes dependence on absolute camera pose and provides a shared arm-level abstraction across human demonstrations and robot executions.
- Human Action Representation: An optimization-based retargeting procedure maps 21 human hand keypoints into a constrained 22-DoF robot-hand joint space.Joint limits and kinematic constraints are enforced while preserving fine-grained finger articulation.
- Human Data Sources and Processing: Stage I pretrains on 20,854 hours of egocentric video spanning 9,869 scenes, 6,015 tasks, and 43,237 objects, supplemented by 829 hours of accurately tracked EgoDex data.Most recordings are in-the-wild and use off-the-shelf SLAM and hand-pose estimation, while EgoDex supplies higher-precision kinematic signals.
- Human–Robot Mid-Training: Stage II uses a smaller aligned dataset of 344 tabletop tasks, about 50 human-data hours, and 4 robot-data hours captured with matched viewpoints and calibrated intrinsics.This stage grounds broad pretrained representations in the robot’s sensing and action spaces.
- Model Architecture: The flow-based VLA encodes image-language observations, predicts future action chunks with flow matching, and uses embodiment-conditioned adapters for differing robot states and hand spaces.Human demonstrations replace unavailable proprioception with a learnable placeholder token.
3. Experiment
The experiments evaluate EgoScale across five dexterous manipulation tasks and show that large-scale human pretraining, especially when combined with aligned mid-training, improves robot performance, scales predictably with human data, and supports one-shot and cross-embodiment transfer.
- Experiment Setup: The evaluation covers five dexterous manipulation tasks, including shirt rolling, card sorting, tongs-based fruit transfer, bottle-cap unscrewing, and syringe liquid transfer.Each task tests different demands, from deformable coordination and precise insertion to long-horizon multi-step manipulation.
- Experiment Setup: Each task uses 100 teleoperated robot demonstrations, except Shirt Rolling, which uses 20 because it requires less precise control.
- Main Results: Human pretraining improves average task completion by over 55% across tasks compared with training from scratch, and already outperforms mid-training alone on most tasks.This benefit appears despite human demonstrations being noisy, unconstrained, and not task- or sensor-aligned.
- Main Results: Combining human pretraining with aligned mid-training yields the best overall performance by pairing general manipulation structure with executable robot-control alignment.
- Scaling Behavior: Average task completion increases monotonically from 0.30 with 1k hours to 0.71 with 20k hours of human pretraining, with no saturation in the explored regime.
- Scaling Behavior: Human validation loss follows an almost perfect log-linear relationship with data scale, achieving R2 = 0.9983 and closely tracking downstream task completion.Larger datasets improve stably without overfitting, whereas 1k–2k-hour datasets can plateau or degrade during training.
- Few-Shot Transfer: Aligned mid-training enables one-shot generalization, with the combined model reaching success rates of 0.88 on Fold Shirt and 0.55 on Unscrewing Water Bottles.Shared motion primitives support transfer across differing objects and task instances with as little as one target-robot demonstration.
- Generalization and Action Representation: Human-pretrained policies transfer across embodiments when mid-training includes G1 play data, while wrist-only action supervision performs poorly on tasks requiring precise finger articulation and contact timing.Retargeted joint-space hand actions provide the most consistent performance across tasks.
4. Related Work
Prior work increasingly uses human demonstrations for robot learning, but existing transfer studies generally rely on limited data or simpler embodiments. EgoScale instead targets large-scale transfer of wrist motion and dexterous hand articulation for high-DoF manipulation.
- Human demonstrations have progressed from representation learning and intent inference toward planning, high-level control, and dense action supervision.
- Existing human-to-robot transfer methods commonly use relatively small datasets and focus on grippers or low-DoF hands lacking fine-grained finger articulation.
- EgoScale pretrains directly on diverse egocentric human videos to learn both wrist motion and dexterous hand articulation with minimal robot supervision.
- Compared with prior hand-level approaches, EgoScale uses substantially larger human data and evaluates scaling effects on both human validation metrics and downstream robot performance.
- Dexterous manipulation research has evolved from analytic grasping and contact modeling toward data-driven policies with structured interaction representations.
6. Conclusion
EgoScale combines large-scale human pretraining with aligned human–robot mid-training to support long-horizon dexterous manipulation, one-shot adaptation, and transfer across different robot embodiments. The conclusion identifies further scaling, weaker supervision, and more human-like hardware as open directions.
- EgoScale’s central conclusion is that effective human-to-robot transfer for dexterous manipulation is fundamentally a scaling phenomenon.
- Large-scale human pretraining plus a small amount of aligned human–robot mid-training supports long-horizon manipulation, one-shot adaptation, and cross-embodiment transfer.
- The framework transfers across robots with substantially different kinematics and hand designs, including Galaxea R1 Pro 22-DoF hands and Unitree G1 7-DoF tri-finger hands.
- Future directions include jointly scaling human data and model capacity, using weaker or unlabeled video, and exploiting increasingly human-like robotic hardware.
B. Task Descriptions and Evaluation Rubric
The evaluation assigns each task a natural-language instruction and a completion score from 0 to 1 that captures partial progress as well as binary success. Rubrics are either additive for independent sub-skills or progress-based for tightly coupled tasks.
- Each task is evaluated with a completion score in [0, 1] designed to capture partial progress rather than only end-to-end success.
- Additive rubrics sum achieved sub-scores for tasks that decompose into independent, well-defined sub-skills.
- Progress-based rubrics score the furthest milestone reached when intermediate states are difficult to define independently.
- Unless otherwise specified, each checkpoint is evaluated over 10 trials.
R1 Pro Sharpa
The R1 Pro Sharpa evaluation covers five dexterous manipulation tasks and two one-shot adaptation tasks, using progress-based or additive completion rubrics plus binary end-to-end success. Tasks range from deformable-object handling to precise tool manipulation.
- Task I: Shirt Rolling: Shirt rolling requires coordinated two-hand folding, rolling, and basket placement, with progress milestones from no folding to completed placement.
- Task II: Card Sorting: Card sorting evaluates separating one card from a deck and inserting it into the correct holder while minimizing disturbance.
- Task III: Tong Fruit Transfer: Tong fruit transfer scores grasping the tongs, picking up and placing the fruit, and returning the tongs to the table.
- Task IV: Bottle Cap Unscrewing: Bottle-cap unscrewing tests grasping, at least three continuous rotations, cap removal, and placement across four bottle configurations.
- Task V: Syringe Liquid Transfer: Syringe transfer requires pickup, liquid extraction from tube A, injection into tube B, disposal, and precise long-horizon sequencing.
- One-Shot Adaptation: One-shot adaptation evaluates shirt folding and bottle-cap unscrewing after post-training on a single robot demonstration.
- All tasks additionally report binary success, counting a trial successful only when the instruction is completed end-to-end.
G1 Tasks
The Unitree G1 evaluation uses two manipulation tasks with additive scoring based on discrete sub-skills. The tasks assess canister-marker handling and multi-plate dish-rack placement.
- Two Unitree G1 manipulation tasks are evaluated with additive scores reflecting discrete, well-defined sub-skills.
- Pen in Bin: Pen in Bin requires opening a canister, picking up a marker, and placing the marker into the canister.
- Pen in Bin: The Pen in Bin rubric assigns 0.25 each for canister handling, stable placement, marker pickup, and marker placement.
- Dish Handover in Rack: Dish Handover in Rack requires transferring three plates, potentially between hands, and placing them upright into a dish rack.
- Dish Handover in Rack: Each plate receives 0.11 for pickup, hand transfer, and upright placement, with scores summed to a maximum of 1.0 per trial.
C.1. Dataset Statistics and Analysis
The dataset analysis examines Stage I egocentric human activity data through complementary statistical distributions and qualitative examples summarized in Figure 10.
- Figure 10 analyzes the Stage I egocentric human activity dataset using statistical distributions and qualitative egocentric examples.The perspectives include object vocabulary, category distribution, environment distribution, and task statistics.
C.2. Category Distribution
The dataset spans diverse categories, environments, and objects, with dominant domains accompanied by a long tail of less frequent coverage.
- Categories: Retail and consumer goods comprise 20.1% of video hours, followed by fashion at 11.8%, repair services at 11.5%, and food and beverage at 11.5%.Only categories exceeding 50 hours are included in the plotted distribution.
- Figure 10 shows long-tailed coverage across categories, environments, tasks, and objects.
- Categories: Home environments account for 9.5% of video hours, construction and hardware 7.7%, food processing 6.7%, and printing and design 4.4%.
- Categories: The remaining categories form a long tail spanning hospitality, automotive and transport, sports and recreation, education, healthcare, energy, and industrial domains.
- Environments: Frequent recording environments include homes, flower shops, electronics repair shops, furniture repair shops, woodworking shops, and clothing stores.Dozens of additional settings contribute smaller but non-negligible amounts of data.
C.4. Task Coverage
Task coverage includes frequent household, repair, craft, and food activities, while the transfer pipeline retargets human motion and adapts it across robot embodiments. The G1 is aligned through lightweight mid-training rather than trained from scratch, preserving a reusable human-derived manipulation representation.
- C.4. Task Coverage: High-frequency tasks include folding clothes, cleaning shoes, potting plants, ironing, assembling boxes, arranging flowers, sanding wood, and food preparation.
- C.4. Task Coverage: Medium- and low-frequency tasks cover packing, sorting, repairing, decorating, cutting, bottling, inflating, and disassembling, often involving multi-step procedures and sustained physical interaction.
- Human-to-Robot Transfer: Human hand motion is retargeted into the Sharpa Hand’s 22-DoF joint space using per-frame optimization with URDF-based forward kinematics.
- Human-to-Robot Transfer: The retargeting optimization enforces joint limits and kinematic consistency, then applies exponential smoothing to reduce temporal jitter.
- Cross-Embodiment Adaptation: Relative end-effector poses in SE(3) provide a shared wrist-action representation across human demonstrations and robot executions.The representation remains invariant to absolute workspace, camera placement, and arm kinematics.
- Cross-Embodiment Adaptation: Embodiment-conditioned MLP adapters map between the 22-DoF human-pretrained hand space and robots such as the Unitree G1’s 7-DoF tri-finger hand.
- Cross-Embodiment Adaptation: During G1 mid-training, selected vision, action, and state-action modules are updated while the vision-language backbone remains frozen.This adapts the representation to G1 sensing and actuation interfaces while preserving human-derived manipulation structure.
- Cross-Embodiment Adaptation: The G1 is not trained from scratch; aligned mid-training yields substantially higher performance than using G1 data alone.The authors characterize large-scale human pretraining as a reusable, embodiment-agnostic motor prior for differing kinematics and hand designs.