PaperlayerView exact revision on arXiv

Source-linked AI summary

Ego4WAM: What Matters When Scaling Egocentric Human Data for Robot Learning?

Zhihao Sun, Liu Liu, Xinjiang Wang, Haoyi Jiang, Wei Feng, Huiqiang Zhang, Xiaosong Jia, Zhizhong Su, Zuxuan Wu

arXiv:2609.40341v1cs.ROcs.CV

TL;DR

Egocentric human data is plentiful, but it remains unclear which properties drive robot-learning gains and how the data should be used. Ego4WAM studies alignment, data scale and diversity, supervision, and usage in a fixed-backbone WAM. Its findings show distinct roles for these factors, including benefits from video-only experience as well as aligned demonstrations.

  • Problem

    It remains unclear which egocentric data properties drive downstream robot gains and how to use the data throughout the training pipeline.

    Source: §1 · pp. 1–2

  • Method

    With the WAM backbone fixed, the study varies alignment, coverage, supervision, and usage strategies, evaluating closed-loop policy performance.

    Source: §1 · p. 2

  • Results

    Human data reduces robot-data needs without replacing robot data; duration and task diversity have distinct effects, and video-only experience supports subsequent world-action learning.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Combining broad multi-task human experience, aligned demonstrations, and a small amount of robot data can reduce target-task robot-data requirements.

    Source: §5 · p. 9

    The video-only and video-action training stages are not compute-matched because video-only data is naturally available at larger scale.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Egocentric human data provides a scalable source of experience for robot learning, but varies substantially in human-robot alignment, behavioral coverage, and available supervision. Existing work shows favorable scaling with increasing human data, but it remains unclear which data properties drive downstream robot gains and how to use such data throughout the training pipeline. We present a systematic study of egocentric human data with different alignment and supervision under a unified world-action model framework. With the model backbone fixed, we disentangle the effects of human-robot alignment, data duration and task diversity, action supervision, and data usage strategies. We find that aligned human demonstrations substantially improve out-of-distribution generalization and reduce target-task robot data requirements; data duration and task diversity affect downstream capabilities differently; and video-only supervision remains effective without action labels, providing a strong foundation for subsequent video-action training. We validate these findings through closed-loop policy evaluation on both real robots and RoboDojo. Rather than treating data duration as the sole scaling axis, Ego4WAM shows how alignment, task diversity, available supervision, and usage strategy jointly shape the value of egocentric human data for robot learning.

1 INTRODUCTION

Ego4WAM asks which properties and usage strategies of egocentric human data matter for robot learning, and studies them with a fixed-backbone WAM. Its findings distinguish the value of alignment, data duration and diversity, and video-only supervision using closed-loop policy performance.

1 INTRODUCTION

  • Existing scaling results leave unclear which data properties drive downstream gains, while duration alone entangles alignment, behavioral coverage, training objectives, and scale.Reliable human action trajectories require camera- and hand-pose estimation, adding processing cost and reducing label reliability; video without action labels can still provide interaction dynamics for world modeling.Source: §1 · pp. 1–2
  • The paper studies alignment, duration versus task diversity, action supervision, and data-use strategies under a unified WAM framework with its backbone fixed.It examines how video-only data interacts with subsequent video-action training.Source: §1 · p. 2
  • Aligned demonstrations improve out-of-distribution generalization and reduce target-task robot data needs, while video-only experience supports later video-action training.The study evaluates closed-loop policy performance on real-world and RoboDojo manipulation tasks.Source: §1 · p. 2
  • Human demonstrations covering object and scene conditions absent from robot training improve OOD generalization and reduce the amount of target-task robot data required.The paper evaluates these effects through closed-loop performance rather than human-action prediction MSE alone.Source: §1 · p. 2
  • Data duration and task diversity affect downstream capabilities differently, so overall data scale alone does not characterize egocentric data's value.The study evaluates utility through closed-loop policy performance on real robots and simulation.Source: §1 · p. 2

2 RELATED WORKS

Prior work spans low-cost, broad egocentric experience, action-annotated datasets, and more task-aligned demonstrations, alongside varied approaches to human-to-robot transfer. Aggregate scaling and open-loop metrics do not establish which data properties drive closed-loop robot gains.

2 RELATED WORKS

  • Egocentric datasets range from broad everyday video to manipulation data with hand-pose annotations and more robot-aligned demonstrations, each with different collection requirements.Reliable action trajectories need additional sensing, reconstruction, and quality control; stronger task alignment also imposes stricter collection requirements.Source: §2 · p. 2
  • Prior transfer approaches retarget or transform human experience toward the robot domain, but tracking, reconstruction, and synthesis errors can limit reliability.These approaches include hand and wrist retargeting, action-space or viewpoint alignment, and visual embodiment editing.Source: §2 · p. 3
  • Favorable scaling trends and open-loop human prediction metrics leave unresolved which egocentric data properties drive downstream gains under closed-loop deployment.The paper instead studies data properties and usage strategies under a controlled WAM framework.Source: §2 · p. 3

3 METHOD

The method holds the WAM backbone and unified action space fixed while varying data composition, supervision, and usage. It organizes egocentric samples by reliable supervision, supports video-only and video-action learning, and stages training from world-model pretraining through robot adaptation.

3 METHOD

  • The study controls the WAM backbone and unified action space while varying data composition, supervision, and use across experiments.This setup is intended to study egocentric data's contribution under different alignment and supervision settings.Source: §3 · p. 3

3.1 EGOCENTRIC DATA PYRAMID

  • The egocentric data pyramid retains samples at their reliable supervision level, with aligned, quality-controlled action-annotated, and video-only subsets.Video-only samples include valid visual interactions even when actions are unavailable or rejected by action-quality filtering; their motion and outcomes can support world modeling.Source: §3.1 · p. 4

3.2 WORLD-ACTION MODEL FRAMEWORK

  • The WAM pairs a video world-modeling branch with an action-prediction branch, using shared attention and allowing video-only data to supervise world modeling.Language conditions both branches, while proprioceptive states enter only the action branch; video-action data can supervise both prediction objectives.Source: §3.2 · p. 4

3.3 UNIFIED ACTION SPACE

  • The unified action space is 32D bimanual control, and human action supervision uses only EEF pose and gripper state as shared components.Each arm includes a 6D joint signal, a 1D gripper state, and a 9D EEF representation; missing action dimensions use an analytic forward-noise strategy.Source: §3.3 · pp. 4–5

3.4 TRAINING PROTOCOL

  • Training proceeds through video-only world-model pretraining, joint world-action mid-training on reliable-action samples, and downstream robot adaptation.Post-training uses embodiment-specific robot demonstrations, optionally alongside robot-aligned human data.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate egocentric human data through closed-loop policies on real robots and RoboDojo, progressively varying alignment, data scale, supervision, and usage. Results show distinct benefits and trade-offs across these settings.

4.1 ROBOT-ALIGNED HUMAN DEMONSTRATIONS

  • On Place into the Basket, object-OOD human demonstrations raise object-OOD success from 10% to 60%, while scene-OOD demonstrations raise scene-OOD success from 0% to 20%.The robot-only policy achieves 60% ID success; scene-OOD demonstrations also produce a substantially larger gain in task progress.Source: §4.1 · p. 5
  • With 100 Fold Cloth robot demonstrations and 500 fixed human demonstrations, success reaches 80% ID, 60% object-OOD, and 80% scene-OOD; at 20 robot demonstrations, aligned human data alone is insufficient.After multi-task mid-training, the same 20 robot demonstrations and aligned human data yield 80% ID, 80% object-OOD, and 60% scene-OOD success.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 SCALING BEYOND TASK-SPECIFIC EGOCENTRIC DATA

  • With a fixed task set, increasing duration from 1K to 4K hours improves Generalization from 5.0 to 7.8 and Long-Horizon from 12.0 to 16.2.Expanding task diversity from 500 to 6K tasks improves Generalization from 5.0 to 7.0 and Long-Horizon from 12.0 to 16.2, but Precision decreases as longer-tail tasks are added.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 RELAXING ACTION SUPERVISION WITH LARGE-SCALE HUMAN VIDEO

  • Video-only pre-training raises the average RoboDojo score from 6.39 to 14.13 and success rate from 3.15% to 9.45%, despite using no human action supervision.Video-action mid-training after pre-training further improves performance from 14.13 / 9.45% to 15.04 / 10.78%; without prior video pre-training, mid-training reaches only 10.28 / 6.42%.Source: §4.3 · pp. 7–8
  • Starting from video pre-training, joint shared attention raises the average RoboDojo score from 14.13 to 20.29 and success rate from 9.45% to 14.35%.With approximately 0.1K hours of multi-task robot data and 15K hours of egocentric experience, the model reaches a 20.29 average RoboDojo score and 14.35% success rate.Source: §4.3 · pp. 8–9, Table 1 · p. 8

4.4 MORE RESULTS ON THE REAL ROBOT TASKS

  • The real-robot evaluation spans Fold Towel, Tighten a Bottle Cap, Stack Plates and Bowls, and Heat Food with a Microwave.These tasks cover rigid-object manipulation, deformable-object interaction, and longer-horizon composition.Source: §4.4 · p. 9, Figure 6 · p. 9

5 CONCLUSION

The conclusion argues that egocentric data’s value depends on alignment, coverage, supervision, and when it is used, not duration alone. Its takeaways distinguish complementary human and robot data, non-monotonic scaling, and video-derived dynamics priors.

5 CONCLUSION

  • Broad human experience and aligned demonstrations complement a small amount of robot data, reducing target-task robot-data needs without fully replacing robot data.Broad experience supplies transferable manipulation priors, while alignment covers distribution variations and robot data supports embodiment adaptation.Source: §5 · p. 9
  • More egocentric data does not guarantee better performance: denser recurring-task coverage adds limited gains, while sparse long-tail tasks can hurt precision-sensitive performance.The conclusion distinguishes data duration from task diversity and cautions that total hours alone do not characterize useful scale.Source: §5 · p. 9
  • Video-only pretraining can learn dynamics priors without action labels, and those priors further improve subsequent world-action learning.The conclusion says human-data value depends on its supervision and on when and how it is used.Source: §5 · p. 9
  • Ego4WAM offers an empirical perspective for designing egocentric-data pipelines and plans to release code, training recipes, and stage-specific checkpoints.The release is intended to support further research.Source: §5 · pp. 9–10

STATEMENT

The statement describes generative-AI assistance, participant consent and data-use practices, and planned reproducibility resources. The authors retain responsibility for the research and its conclusions.

STATEMENT

  • Generative AI assisted with language editing, manuscript organization, and presentation clarity; the authors developed and verified the research and take responsibility for the paper.The authors also reviewed the research ideas, experimental design, data processing, analyses, and conclusions.Source: STATEMENT · p. 10
  • Participants in internally collected data consented to research use, while public datasets are used under their licenses and data-use requirements.The authors say they do not intentionally collect or release personally identifiable information and use collected data solely for robot-learning research.Source: STATEMENT · p. 10
  • The authors describe model and evaluation protocols and plan to release code, training recipes, and checkpoints for independent verification and standardized RoboDojo comparison.Planned releases include checkpoints from pre-training and mid-training stages and RoboDojo post-trained checkpoints.Source: STATEMENT · p. 10

A.1 EGOCENTRIC DATA PYRAMID

The data pyramid organizes egocentric experience by the strongest reliable supervision each sample supports, with less restrictive supervision allowing more data to remain usable. EgoAlign supplies a separate robot-aligned dataset with reconstructed and quality-filtered hand-action trajectories.

A.1 EGOCENTRIC DATA PYRAMID

  • The pyramid presents a general supervision hierarchy, not the exact experimental mixture: robot-aligned demonstrations are scarcest, followed by video-action data, while video-only data scales larger.Its tiers reflect that stronger supervision requirements leave fewer samples that reliably meet them.Source: §A.1 · p. 13
  • Action-quality filtering assigns reliable annotated samples to video-action data, while visually valid interactions with unreliable or absent action labels can remain video-only.EgoAlign is processed separately as the robot-aligned subset.Source: §A.1 · p. 13
  • Experiments use 15K hours for video pre-training and 12K hours for video-action mid-training; the pyramid’s roughly 120K video-only hours indicate availability, not experimental use.The 15K-hour mix includes the 12K-hour action-valid set plus about 3K sampled video-only hours.Source: §A.1 · p. 13

A.2 EGOALIGN

  • EgoAlign is an internally collected, monocular head-camera dataset recorded under task and environment settings matching the real-robot experiments.Human hand actions are recovered from the videos using a processing pipeline based on OpenAoE.Source: §A.2 · p. 13
  • EgoAlign trajectories combine estimated camera motion and framewise 3D hand poses in a shared coordinate system, then undergo inspection and quality filtering.Only trajectories passing checks for localization, motion, camera tracking, and temporal or geometric consistency are retained as action-annotated demonstrations.Source: §A.2 · p. 13

B METHOD

The method maps human and robot actions into a shared action space while masking unavailable human-control dimensions. Conditional flow-matching objectives train world modeling and action prediction according to the supervision available at each stage.

B.1 UNIFIED ACTION SPACE

  • The human gripper representation uses the thumb–index midpoint as position and their Euclidean distance as gripper openness.Orientation is represented by a hand-centric frame converted to a continuous 6D rotation representation, yielding a 9D end-effector pose.Source: §B.1 · pp. 13–14
  • Human actions supervise shared end-effector and gripper dimensions, while validity masks and an analytic noise path handle unavailable robot-specific dimensions.Robot demonstrations provide both joint and shared end-effector supervision; inactive dimensions remain consistent with the forward-noise process without receiving supervision.Source: §B.1 · p. 14

B.2 OPTIMIZATION OBJECTIVES

  • Conditional flow matching trains both the world-modeling and action-prediction branches using noisy samples and timestep-dependent loss weighting.The world-modeling target is future video latents; the action-prediction target is an action chunk of horizon H.Source: §B.2 · pp. 14–15
  • World-modeling conditions on the current visual observation and language instruction, and applies to egocentric videos with or without reliable action annotations.Action prediction additionally uses available proprioceptive state; for human demonstrations, its loss covers only shared end-effector and gripper dimensions.Source: §B.2 · p. 15
  • Training uses video-only world modeling for pre-training, joint world and action objectives for action-annotated mid-training, and joint objectives for downstream post-training.Post-training can optionally include aligned human demonstrations alongside downstream robot demonstrations.Source: §B.2 · p. 15

B.3 TRAINING DETAILS

The implementation uses five configurations spanning pre-training, mid-training, and robot post-training. Within each scaling or post-training comparison, optimization settings are held constant unless specified otherwise.

B.3 TRAINING DETAILS

  • The implementation extends the StarVLA codebase to support the WAM architecture, unified human–robot action space, and multi-stage training pipeline.Source: §B.3 · p. 15
  • Five configurations cover video pre-training, full-scale and scaling-study mid-training, real-robot post-training, and RoboDojo post-training.Experiments typically use batch size 16 per GPU, eight H20 GPUs per node, cosine learning-rate decay, eight future video frames, and a 32-step action horizon.Source: §B.3 · p. 15
  • Data-duration and task-diversity variants share a training configuration and optimization-step count, changing only egocentric data scale and composition.Real-robot tasks and robot-data-budget ablations share a post-training recipe, while RoboDojo experiments share a separate post-training configuration.Source: §B.3 · p. 15

C EXPERIMENTS

Experiments evaluate closed-loop manipulation on real robots and RoboDojo. The real-robot setup tests controlled object and scene shifts as well as a broader task set, while RoboDojo provides a standardized multi-category benchmark.

C.1 REAL-ROBOT SETUP

  • Controlled real-robot experiments use a bimanual Piper with parallel grippers and compare teleoperated robot demonstrations with separately recorded head-camera human demonstrations.The human and robot demonstrations share task and environment conditions but retain their native viewpoints and embodiments.Source: §C.1 · p. 16
  • Two controlled tasks—Place into the Basket and Fold Cloth—each have 300 in-domain robot demonstrations and 300 in-domain, 100 object-OOD, and 100 scene-OOD human demonstrations.Object- and scene-OOD conditions are absent from robot training unless otherwise specified.Source: §C.1 · p. 16
  • Object-OOD changes manipulated-object category, instance, appearance, or size, whereas scene-OOD changes the visual environment, including surface appearance or illumination.The two distribution-shift axes are complementary and need not be mutually exclusive.Source: §C.1 · p. 16
  • The final model is also evaluated on four real-world tasks spanning rigid-object, deformable-object, and longer-horizon manipulation.The tasks are Fold Towel, Tighten a Bottle Cap, Stack Plates and Bowls, and Heat Food with a Microwave.Source: §C.1 · p. 16
  • Real-robot policies are assessed through closed-loop success rate and normalized task progress, which scores partial completion from 0 to 100.The same task-specific milestone criteria apply across compared methods and settings.Source: §C.1 · p. 16

C.2 ROBODOJO SETUP

  • RoboDojo reports progress and success across Generalization, Precision, Long-Horizon, Memory, and Open categories, plus the official suite average.The main-paper analysis focuses primarily on Generalization, Precision, and Long-Horizon; Figure 7 provides complete per-task scaling results, including tasks with zero success.Source: §C.2 · pp. 16–17, Figure 7 · p. 17

C.3 DATA DURATION AND TASK-DIVERSITY SCALING

The scaling study separates denser experience on recurring tasks from broader task coverage extending into the long tail. Its variants hold initialization, robot data, optimization schedule, and RoboDojo post-training protocol constant.

C.3 DATA DURATION AND TASK-DIVERSITY SCALING

  • The egocentric task distribution is imbalanced: common behaviors account for much of the data, while many less frequent tasks form its long tail.Source: §C.3 · p. 17
  • Duration scaling fixes the 500 most frequent tasks and increases sampled data from 1K to 2K and 4K hours, increasing experience density without expanding task coverage.Source: §C.3 · p. 17
  • Task-diversity scaling expands from 500 to 6K tasks and 1K to 12K hours while approximately preserving data per task, adding increasingly rare categories.All scaling variants share initialization, robot-data component, optimization schedule, and RoboDojo post-training protocol.Source: §C.3 · pp. 17–18
Loading 2609.40341v1…