PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data

Baoyu Li, Xinchen Yin, Mengying Lin, Yixin Zhang, Danfei Xu

arXiv:2607.08436v1cs.ROcs.AI

TL;DR

The best world representation for transferring supervision from egocentric human data to robots has not been systematically established. EgoWAM compares world-prediction targets in a controlled co-training framework and finds that DINO and 3D flow outperform pixels, with complementary strengths.

  • Problem

    Which world representation enables effective cross-embodiment transfer under WAM co-training has not been systematically studied.

    Source: §1 · p. 2

  • Method

    EgoWAM compares pixel, DINO, and 3D-flow world targets while holding the backbone, action supervision, and data mixture fixed.

    Source: §1 · p. 2

  • Results

    WAM co-training transfers more consistently from in-the-wild human data than behavior cloning, while DINO and 3D flow outperform pixels with complementary strengths.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Appearance abstraction, cross-embodiment consistency, and ego-motion factoring are proposed as design criteria for world targets in human-to-robot transfer.

    Source: §1 · p. 2

    The study uses one policy per task, leaving multi-task co-training on large-scale in-the-wild human data for future work.

    Source: §7 · p. 9

Abstract

from arXiv · show

Egocentric human data offers scalable supervision for robot manipulation. However, behavior cloning entangles transferable content like objects, scenes, and task semantics, with non-transferable factors like human morphology, head motion, and behavioral style. We study whether World Action Models (WAMs) provide a better training signal by requiring policies to predict not only actions, but also how the scene evolves. The central question is what world representation best enables human-to-robot transfer. We hypothesize that an effective world target should abstract appearance, capture agent-invariant physical effects, and separate camera motion from environment change. We introduce EgoWAM, a controlled human-robot co-training framework that fixes the policy backbone, action head, and data mixture while varying only the world prediction target, comparing Pixel, DINO, and 3D motion flow. Across three real-world bimanual tasks, WAM co-training scales more effectively with in-the-wild egocentric human data than behavior cloning. Pixel-based prediction transfers weakly, while DINO and 3D flow yield substantial gains: DINO improves out-of-distribution object and scene generalization by up to 4x, and 3D flow improves in-domain performance by 20-30%. More details: https://gatech-rl2.github.io/egowam.github.io

1 Introduction

EgoWAM asks whether predicting future world states can make in-the-wild egocentric human data more transferable to robot manipulation than action-only co-training, and studies how the world representation affects that transfer.

Introduction

  • Egocentric human data supplies diverse objects, scenes, and behaviors at a scale that is prohibitively expensive to collect on robots [1] [2] [3].The paper frames this data as a potential source for scaling robot manipulation.Source: §1 · p. 1
  • BC co-training [4] [5] depends on human data aligned to the robot, because its shared action decoder entangles transferable task content with embodiment-specific execution.Without alignment in viewpoint, motion speed, and behavioral style, human-like motions can degrade downstream performance.Source: §1 · p. 2
  • WAMs [8] [9] [10] [11] add a future-state prediction channel that can shape shared representations through task dynamics while separating world evolution from embodiment-specific actions.The paper hypothesizes that task-relevant dynamics transfer across embodiments more readily than action labels.Source: §1 · p. 2
  • EgoWAM identifies appearance abstraction, cross-embodiment consistency, and ego-motion factoring as desiderata, contrasting pixel-VAE prediction [8] [9] [10] [11] with DINO features [13] and 3D motion flow [14] [15].DINO abstracts appearance and aligns predictions across embodiments but remains image-grid indexed; 3D flow adds camera-frame geometric grounding.Source: §1 · p. 2
  • EGOWAM uses a shared action head and swappable world-model head on a Heterogeneous Pretrained Transformer backbone [16] [7] to compare world representations under matched action supervision and data mixtures.The framework jointly trains on human and robot data across three real-world bimanual tasks spanning spatial, object, and scene generalization.Source: §1 · p. 2
  • Across in-domain and OOD evaluations, WAM co-training transfers more consistently than BC, with DINO strongest for object and scene generalization and 3D flow strongest for spatial generalization and overall scores.The comparison spans three real-world bimanual tasks.Source: §1 · p. 2

2 Related Work

Prior work uses human video for representation pretraining, joint human-robot training, or motion-based action decoding, while WAM research leaves open which world representation best supports generalization. EgoWAM compares pixel latents, DINO features, and camera-stabilized 3D motion flow within one co-training setup.

Robot Learning from Human Data

  • Human video supports action-aware representation pretraining [17] [19] [20] and joint human-robot imitation or world-model training [4] [5] [21] [22] [23] [24] [25] [26] [27], but embodiment gaps require tight alignment [6] [28] [29] [30] [31].The alignment challenge concerns viewpoint, speed, and kinematics.Source: §2 · pp. 2–3
  • Prior methods decode actions at inference from motion extracted from human video, especially 2D point tracks [32] [33] [34] [35] [36] and 3D flow [38] [39] [40] [41] [42], whereas EGOWAM uses scene motion to supervise training and discards it at inference.This transfers task-relevant dynamics without injecting inexecutable motions or adding test-time cost.Source: §2 · p. 3

World Action Models for Robot Learning

  • WAMs couple a world predictor to a policy [26] [10] [11], and EgoWAM addresses the open question of generalizable world representations by comparing pixel latents [59], DINO features [13] [60], and camera-stabilized 3D flow [14] [15].Existing WAM work commonly uses pretrained video models for action decoding [8] [9] [47] [48] [49], while another line uses world models as interactive simulators [50] [51] [52].Source: §2 · p. 3

3 Human-Robot WAM Co-Training

Section 3 strengthens the behavior-cloning baseline by aligning human and robot actions, then adds future-scene prediction so dynamics from both embodiments can shape a shared representation. It identifies the world target as the key design choice for cross-embodiment transfer.

3.1 Aligned Action Channel as a Strong Baseline

  • The BC baseline aligns human and robot actions in a 14-D end-effector space, adjusts temporal windows for speed, and quantile-normalizes workspace range.Human and robot windows span 1 s and 1.5 s, respectively; the remaining transfer gap motivates the world-model interface.Source: §3.1 · p. 3

3.2 WAM as a World-Level Transfer Interface

  • The world head supports pixel-latent, DINO-feature, or camera-stabilized 3D-flow targets; target choice determines whether embodiments converge or are pulled apart.Source: Figure 2 · p. 4, §3.2 · p. 4
  • WAM jointly trains action and future-state heads from a shared latent, letting dynamics from both embodiments shape the trunk even when human action labels transfer poorly.The objective combines action loss with a weighted world-prediction loss.Source: §3.2 · pp. 3–4

4 EGOWAM: A Controlled Study of World Representations

EgoWAM studies how world-prediction targets shape human-to-robot transfer while holding the policy backbone, action head, and data mixture fixed. Across three bimanual tasks, the reported comparison favors DINO for OOD generalization and 3D Flow for ID spatial gains, while Pixel transfers weakly.

4 EGOWAM: A Controlled Study of World Representations

  • EgoWAM tests three target desiderata—appearance abstraction, cross-embodiment consistency, and ego-motion factoring—using Pixel VAE, DINO features, and 3D flow.Image-coordinate targets can conflate head rotation with scene change, producing different supervision for moving human and static robot cameras.Source: §4 · p. 4

4.1 Single-Backbone Architecture and Controlled Variables

  • The shared HPT trunk [16] [7] processes observation, action, and future-prediction tokens, while the trunk, action head, and data mixture stay fixed across world-model targets.A second bank of learnable future tokens feeds a swappable world-model head.Source: §4.1 · pp. 4–5

4.2 World-Model Target Instantiations

  • Pixel VAE predicts photometrically reconstructed future frames and violates all three desiderata, whereas DINO predicts semantically structured features but only partially mitigates ego-motion.The DINO target remains spatially indexed in image coordinates.Source: §4.2 · p. 5
  • 3D Flow stabilizes tracked point positions using Aria VIO camera poses, yielding near-zero background flow and manipulated-object motion proportional to physical displacement.The target is a dense motion field in the camera-stabilized frame; its decoder regresses velocity at query points sampled from the current ego frame.Source: §4.2 · p. 5

4.3 Joint Training and Action-Only Inference

  • Joint action and world-prediction losses supervise a shared trunk during training, but inference drops the world-model head and runs only the action head at 30 Hz.This retains the jointly learned representation at the latency of a same-size BC policy.Source: §4.3 · pp. 5–6
  • WAM co-training consistently outperforms BC: Pixel transfers weakly, DINO drives the strongest OOD generalization, and 3D Flow yields the largest ID spatial gains.The figure caption reports these trends across three bimanual tasks under ID and OOD evaluation; human data often degrades BC yet yields large WAM gains.Source: Figure 3 · p. 6

5 Experiments

The experiments examine whether WAM co-training scales and transfers better than BC with in-the-wild human data, which world representation best enables transfer, and how each paradigm handles action-misaligned human data.

5.1 Experimental Setup

  • The study evaluates cup-on-saucer, fold-clothes, and bag-grocery, spanning rigid-object manipulation, deformable manipulation, and long-horizon sequencing.Tasks use randomized poses or configurations; bag-grocery requires opening a bag and loading three items.Source: §5.1 · p. 7
  • Human-data regimes are 1:1 in-domain demonstrations with unmatched viewpoints and behavior, and approximately 10:1 EgoVerse data with diverse scenes, objects, and demonstrators.Robot data comprises 300–360 demonstrations per task; EgoVerse data has no deliberate scene or object alignment.Source: §5.1 · p. 7
  • Evaluation uses 20 ID rollouts on seen objects and scenes and 20 OOD rollouts covering unseen objects or novel scenes, reporting normalized sub-task score and success rate.The reported evaluation totals 1,800 real-world rollouts.Source: §5.1 · pp. 7–8

5.2 Core Results and Findings

  • Non-transferable detail leaves hallucination and geometric confusion in qualitative rollouts, including a free-space grasp.The passage also points to Fig. 5 for these examples.Source: Figure 5 · p. 7, §5.2 · p. 8
  • With action-misaligned human demonstrations, BC falls below the robot-only baseline, while 3D-flow WAM remains robust and surpasses it.The human demonstrations involve unusual object-picking behavior that maps to inexecutable robot actions.Source: Figure 8 · p. 8, §5.2 · p. 8

6 Conclusion

EgoWAM studies how world-prediction targets support human-to-robot transfer, finding that WAM co-training scales with in-the-wild human data where action-only behavior cloning stalls. Its conclusion emphasizes representation choice as a design axis for scalable robot manipulation.

6 Conclusion

  • EgoWAM introduces a human-robot WAM co-training framework to study which world representation enables cross-embodiment transfer.The framework scales with in-the-wild human data where action-only behavior cloning stalls.Source: §6 · p. 8
  • Pixels transfer weakly, while DINO drives object and scene generalization and 3D flow grounds in-domain spatial gains.The conclusion identifies world representation as critical to transfer.Source: §6 · p. 9
  • The proposed design recipe uses targets that abstract appearance, keep effects consistent across embodiments, and factor out ego-motion.The authors present this as a point of departure, positioning human data as a flywheel for scalable robot learning toward open-world generalization.Source: §6 · p. 9

7 Limitations

EgoWAM’s gains remain at the context level, while learning novel motion skills from human data is still out of reach. The study also leaves multi-task scaling and the best world representation open for future research.

7 Limitations

  • EgoWAM’s gains remain at the context level; learning novel motion primitives or skills from human data remains out of reach.A more unified action representation is left to future study.Source: §7 · p. 9
  • The study isolates world representation with one policy per task, leaving multi-task co-training on large-scale in-the-wild human data for future exploration.Source: §7 · p. 9
  • Although DINO and 3D flow beat pixels for cross-embodiment transfer, the best world representation for scaling robot learning remains an open question.Source: §7 · p. 9

Supplementary Materials · A Additional Real-World Experiment Results and Analysis

This supplementary section provides additional real-world experimental results and analysis, with further rollout and world-model prediction visualizations available on the project website.

A Additional Real-World Experiment Results and Analysis

  • The section adds real-world experimental results and analysis, and points readers to the project website for further rollout and world-model prediction visualizations.The supplied passage does not report additional quantitative findings.Source: A Additional Real-World Experiment… · p. 15

A.1 Ablation on Aligned Human Data … B Additional Implementation Details

Across the ablations, 3D-flow supervision transfers strongly without manual demonstrator alignment, while combining action labels with world-model supervision wins across task splits. Pixel-PT failure analysis shows that pretrained video predictions can hallucinate scene state, while human co-training can correct it.

A.1 Ablation on Aligned Human Data

  • Aligned human data lifts BC above robot-only, while natural in-domain human data causes negative transfer.The authors conclude BC benefits from human data only when demonstrator actions are hand-curated to match the robot’s.Source: §A.1 · p. 15, Figure 9 · p. 15
  • 3D-flow WAM holds at 85% with either natural or aligned human data, whereas Pixel rises 35%→65% and DINO 50%→70% with alignment.The study uses 1:1 human co-training on cup-on-saucer, the task with the strongest human–robot mismatch; 3D flow factors out ego-motion by construction.Source: §A.1 · pp. 15–16, Figure 9 · p. 15

A.2 Ablation on Human Data Modality

  • Joint Action + 3D Flow training wins on every split; on OOD Scene, 3D-Flow-only retains 10% SR while Action-only scores 0%.Action labels condition the trunk on demonstrator intent and mark task-relevant motion, complementing the context-level transfer from 3D-motion prediction.Source: §A.2 · p. 16
  • Adding action labels to Flow-Only lowers 3D-flow prediction loss on both human and robot streams, supporting complementary action-context and task-relevance effects.The paper attributes the gains to action labels conditioning the trunk on intent and focusing it on agent-caused rather than incidental motion.Source: Table 1 · p. 16, §A.2 · p. 16

A.3 Failure Analysis for Pixel-PT on Bag-Grocery

  • On bag-grocery, Pixel-PT with robot-only data hallucinates an already-open bag, causing the policy to skip opening and advance prematurely.Pixel from scratch is blurrier but tracks bag openness faithfully; Pixel-PT + EgoVerse retains sharp output while predicting the closed-then-opening progression correctly, which the authors say improves the policy.Source: §A.3 · pp. 16–17, Figure 12 · p. 17

B.1 Behavior-Cloning Co-Training Architecture … C.2 Human Data Collection and EgoVerse Dataset

The appendices describe the shared policy architecture and alternative world-prediction heads, then detail training, robot demonstrations, and human-data regimes. These include aligned in-domain data and the larger, diverse EgoVerse-A dataset.

B.1 Behavior-Cloning Co-Training Architecture

  • A shared 256-d transformer processes embodiment-specific stems with 64 action and 16 future tokens, while a flow-matching action head predicts chunks of 100 actions.The trunk uses 16 blocks and one-frame observation history; the action head uses six blocks and samples with 50 v-prediction steps.Source: §B.1 · p. 17

B.2 World-Model Heads

  • Pixel and Pixel-PT predict future-frame latents from a frozen Wan video VAE, using the same target and trunk conditioning but different head capacity and initialization.Pixel is trained from scratch with a lightweight DiT; Pixel-PT uses the pretrained Wan 1.3B transformer.Source: §B.2 · p. 18
  • The DINO head predicts DINOv2-B patch features with an RAE-style denoiser, applying supervision in feature space rather than through its pixel decoder.Its DiTDH combines a 384-d backbone with a 2048-d wide head; the decoder is used only for visualization, and the world-model head is discarded at inference.Source: §B.2 · p. 19
  • Camera-stabilized 3D flow makes static backgrounds near-zero while preserving manipulated-object displacement, separating environmental motion from head motion in the prediction target.Track4World estimates scene flow, depth, intrinsics, and poses from RGB; Aria VIO poses re-express future points in the camera frame at time t. The model predicts 1120 anchors over a 100-step horizon; human and robot movement thresholds are 10 mm and 2 mm, respectively.Source: §B.2 · p. 19

B.3 Training and Inference

  • Training mixes 32 robot and 32 human samples per step with equal action and world-loss weights, and deployment removes the frozen world-model head.The policy runs on one RTX 4090 at 30 Hz; removing the head keeps deployment footprint identical across world-representation variants.Source: §B.3 · p. 19

C.1 Robot Platform and Data Collection

  • The robot platform uses two 6-DoF ARX5 arms, Aria head-mounted RGB, and wrist cameras; each task has 300–360 teleoperated demonstrations with randomized object placements and combinations.Demonstrations are collected through a Quest 3 and Touch Pro controller interface, with commanded poses converted to joint angles using inverse kinematics.Source: §C.1 · p. 20

C.2 Human Data Collection and EgoVerse Dataset

  • In-domain human demonstrations use Aria hand and head tracking in the same scenes and objects as robot data, collected at a 1:1 ratio; auxiliary sets vary execution and viewpoint alignment.The aligned set mimics robot motion and matches its static camera viewpoint; the unaligned bag-grocery set includes demonstrations whose retargeted actions are inexecutable by the robot gripper.Source: §C.2 · pp. 20–21
  • EgoVerse-A provides diverse scenes, objects, and demonstrators without scene or object alignment to robot data, at an overall human-to-robot ratio of about 10:1.The per-task flagship subsets cover cup-on-saucer, fold-clothes, and bag-grocery; demonstrations include 3D hand pose and calibrated head pose used to align actions and stabilize the 3D-flow target.Source: §C.2 · p. 21

C.3 Rollout Evaluation Protocol … D.2 Integration and Baselines

The paper specifies a controlled real-world rollout protocol and a reproducible RoboTwin 2.0 setup to examine cross-embodiment co-training and the contribution of the world-model head. The supplied passages describe evaluation and implementation, not comparative results.

C.3 Rollout Evaluation Protocol

  • Evaluation covers 20 ID and 20 OOD rollouts per method, with randomized conditions held common across methods and normalized sub-task scores plus binary success rates reported.OOD rollouts cover unseen objects in the training scene and seen objects in novel scenes; the protocol totals 1,800 rollouts across methods and tasks.Source: §C.3 · p. 21
  • Cup-on-Saucer and Fold-Clothes each award three sub-task points for staged manipulation, while success requires correct saucer placement or clean completion of all folding stages.Cup subtasks include reorientation, handover, and upright placement; folding subtasks are bottom-sleeves, top-sleeves, and final fold in half.Source: §C.3 · p. 21
  • Bag-Grocery awards up to four sub-task points for opening the bag and placing three items left-to-right; success requires all three items loaded in order.The passage says this task's pick-and-place motions align naturally between human and robot, enabling action-level co-training in the Sec. 5.2(Q3) ablation.Source: §C.3 · p. 21

D Robot-to-Robot Transfer in Simulation: RoboTwin

  • RoboTwin 2.0 [70] is used for a controlled robot-to-robot replication testing whether cross-embodiment co-training beats single-embodiment training and whether the world-model head supports transfer with abstraction.The stated aim is validating the two central claims from the real-world experiments in a public, reproducible setting, not reporting a new benchmark result.Source: D Robot-to-Robot Transfer in… · p. 21

D.1 Simulation Setup

  • The simulation uses aloha-agilex as the primary bimanual embodiment and arx-x5, franka, and ur5 as co-training embodiments across three tasks using the shipped clean-50 demonstration split.The tasks are pick-diverse-bottles, stack-bowls-three, and hanging-mug; franka and ur5 have two 7-DoF arms.Source: §D.1 · pp. 21–22

D.2 Integration and Baselines

  • A shared 14-D end-effector action in the head-camera frame lets one action head serve all robots; single- and cross-embodiment variants hold trunk, head, and optimizer fixed.ACT-EE and DP-EE are adapted to the same action space and IK executor as single-embodiment, world-model-free references; all methods use the same 100 held-out seeds.Source: §D.2 · p. 22

D.3 Results and Analysis

In RoboTwin held-out simulation, cross-embodiment co-training improves performance, and DINO and 3D Flow show strong object generalization. All methods remain at or below 1% on the precision-demanding hanging-mug task.

D.3 Results and Analysis

  • 4→28 (DINO), 7→11 (Pixel), and 2→6 (BC) on pick show cross-embodiment gains; cross-training is also the only setting that succeeds at all on stack.Under cross-embodiment training, world-model variants consistently exceed action-only BC; same-action-space ACT-EE/DP-EE references reach ≤5% on pick and 0% elsewhere.Source: Table 4 · p. 22, §D.3 · p. 22
  • 28% (DINO) on pick and 16% (3D Flow) on stack demonstrate strong object generalization by appearance-abstracting targets.Pick uses a 15-instance bottle pool; stack is an appearance-shift test because the bowl material changed while pose and geometry did not, and only DINO (8%) and 3D Flow (16%) avoid 0% collapse for BC and Pixel.Source: Table 4 · p. 22, §D.3 · p. 22
  • ≤1% on hanging-mug shows that every method, including DINO and 3D Flow, fails the millimeter-precise rack insertion.The passage attributes this bottleneck to manipulation precision rather than object generalization or world target; resolving it is left to future work.Source: §D.3 · pp. 22–23
Loading 2607.08436v1…