PaperlayerView exact revision on arXiv

Source-linked AI summary

EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model -- From Semantic Understanding through Visual Foresight to Action

Hao Wang, Jiajun Wen, Jingzhi Liu, Shuoshuo Xue, Zhiliang Chen, Min Lin, Yicheng Chang, Xiaoyu Guo, Yukang Zhuo, Zheng Chong, Yunshuang Nie, Jian Zhang, Weijia Liufu, Qingman Wu, Heming Xu, Bingchang Song, Dantong Wu, Zhiyuan Wang, Hang Xu, Jianhua Han, Bokui Chen, Shen Zhao, Rui Li, Xiaodan Liang

arXiv:2609.39973v1cs.RO

TL;DR

Existing VLA and WAM policies emphasize different capabilities, and exposing policies to both can still leave action computation centered on one expert. EWAM integrates semantic, visual, and action representations through asymmetric attention, where a depth-wise handoff emerges without layer-wise supervision. It exceeds strong baselines across simulation and real-robot evaluations, with human egocentric data and subtask-phase supervision improving reported outcomes.

  • Problem

    VLAs emphasize semantic representations while WAMs learn predictive dynamics, and combining sources does not necessarily produce structured cooperation inside the action policy.

    Source: §1 · p. 2

  • Method

    EWAM uses asymmetric attention so action queries access semantic, predicted-visual, and action representations while perceptual experts retain distinct roles.

    Source: §1 · p. 2

  • Results

    EWAM exceeds strong VLA, WAM, and hybrid baselines across simulation benchmarks and real-robot embodiments, reaching 72.1% on RoboTwin C2R.

    Source: §1 · p. 4, §9 · p. 23

  • Takeaways & Limitations

    The results suggest unified embodied learning can induce a semantic-to-foresight-to-action progression; human egocentric data improve transfer and robustness, and subtask-phase supervision improves long-horizon completion.

    Source: §1 · p. 4, §9 · p. 23

    The analysis characterizes information routing and selected-source functional dependence, but does not provide a complete causal decomposition of computation.

    Source: §8 · p. 22

Abstract

from arXiv · show

Vision-language-action (VLA) policies emphasize semantic understanding, whereas world-action models (WAMs) learn predictive representations of environment dynamics. Systems that expose a policy to both sources often still concentrate action computation on a single expert. We present EWAM, an action-centric unified embodied model whose asymmetric joint attention lets action tokens read semantic, current-visual, predicted-future, and action information at every layer while the perceptual experts retain their distinct roles. Without layer-wise supervision, EWAM develops an emergent depth-wise specialization: action queries attend mainly to vision-language features in shallow layers, to predicted future frames in intermediate layers, and to action tokens themselves in deep layers. This handoff replicates across tasks and is stable across denoising steps. Checkpoint tracking and causal interventions show that it is learned and that action generation depends on it. EWAM is pretrained in two separate regimes, one on cross-embodiment robot trajectories and one on human egocentric video. In simulation and real-robot experiments, it surpasses existing VLA, WAM, and hybrid baselines. Human egocentric data improve both cross-embodiment transfer and real-robot robustness, and subtask-phase supervision improves long-horizon completion. Together, these results suggest that unified embodied learning can induce an ordered internal progression from semantic understanding, through visual foresight, to action formation.

1 Introduction

Generalist robot policies must connect semantic understanding, prediction of physical dynamics, and precise action, yet combining semantic and predictive experts does not ensure that action computation integrates them. EWAM unifies these sources through the action stream, where an unsupervised depth-wise handoff emerges; the paper reports competitive simulation and real-robot results.

1 Introduction

  • VLA policies offer rich semantics but limited explicit dynamics supervision, whereas WAMs learn motion priors without guaranteeing semantic reasoning or fine-grained control.Earlier combinations can leave action computation reliant on one expert or attach independently operating modules without structured cooperation inside the policy.Source: §1 · p. 2
  • EWAM gives action queries access to semantic, predicted-visual, and action representations at every layer while asymmetric attention preserves the distinct roles of perceptual experts.Video and action outputs are trained jointly with flow-matching objectives.Source: §1 · p. 2
  • Without layer-wise supervision, EWAM shifts action attention from vision-language features in shallow layers to predicted futures in intermediate layers and action self-attention in deep layers.The routing figure places layer index vertically and inference step horizontally; its color represents attention mass on a zero-to-one scale.Source: §1 · p. 2, Figure 1 · p. 3
  • Observational analysis, checkpoint tracking, and interventions test the handoff’s task-level stability, emergence during training, and functional consequence.The evaluation includes counterfactual future-feature injection and tests policy performance in simulation and real-world robot settings.Source: §1 · pp. 2–3
  • EWAM reaches 72.1% on RoboTwin C2R, a 92.9% in-domain average, and 98.8% on LIBERO, with consistent performance across three real-robot embodiments.Human egocentric data improve cross-embodiment transfer and robustness under scene variation, while subtask-phase supervision improves long-horizon completion.Source: §1 · pp. 3–4

2 Related Work

VLA and WAM research emphasizes complementary semantic and predictive capabilities, while hybrid approaches combine them through modular pipelines or interfaces. EWAM instead investigates their coordination within one action stream.

2 Related Work

  • VLAs transfer semantic and web-scale knowledge into action prediction, while WAMs build policies around predictive video representations and jointly model future observations and actions.The related-work review describes VLA examples including RT-1, PaLM-E, RT-2, Octo, OpenVLA, and π0.5, and WAM examples including DreamZero, LingBot-VA, and Fast-WAM.Source: §2 · p. 4
  • Hybrid VLA–WAM systems combine semantics, visual foresight, and control, often through modular pipelines or one-way intermediate interfaces.EWAM instead couples semantic, future-visual, and action representations in one layer-aligned architecture, with the action stream interacting with all three sources throughout depth.Source: §2 · p. 4
  • Human-video learning routes range from visual representation transfer and surrogate future or latent-action signals to hand-centric motion targets.EWAM places heterogeneous human corpora and robot data in a shared camera-relative end-effector action space and uses the same joint video–action objective and layer-aligned experts for human pretraining.Source: §2 · pp. 4–5

3 Preliminaries and Problem Formulation

The policy conditions future-video and action trajectory prediction on the current image and language instruction. It trains both outputs by regressing flow velocities from noisy samples toward clean targets.

3 Preliminaries and Problem Formulation

  • The policy conditions future-video observations and an action chunk on the current image observation and language instruction.The passage denotes these as o_t, ℓ, v_{t+1:t+H}, and a_{t:t+K−1}.Source: §3 · p. 5
  • For a clean target x and Gaussian noise ϵ, the noisy sample is formed by interpolation and its target velocity is ϵ − x.The displayed relation is x_τ = (1 − τ)x + τϵ, u⋆(x_τ, τ) = ϵ − x.Source: §3 · p. 5
  • Training regresses a velocity field from the noisy sample to the target, using independently sampled noise for video and action.The passage states that video and action outputs use flow matching.Source: §3 · p. 5

4 A Unified Embodied Model

EWAM jointly predicts future video and action using layer-aligned experts, with action tokens serving as the cross-modal integration interface. Joint flow matching trains both trajectories, and task-specific post-training adds active-subtask and phase-progress supervision.

4.1 Architecture Overview

  • EWAM processes language and image observations, noisy action, and noisy video latents through layer-aligned VL, video, and action experts in one forward process.Pretraining optimizes future prediction and action generation over cross-embodiment trajectories; task-specific post-training retains these objectives and adds subtask-phase supervision.Source: §4.1 · p. 5

4.2 Action-Centric Asymmetric Attention

  • Asymmetric attention makes action tokens the sole cross-modal readers while video and vision-language streams retain modality-specific computation.Action queries can retrieve semantic and predictive evidence; video and vision-language queries cannot read each other or the action stream.Source: §4.2 · pp. 5–6, Figure 2 · p. 6

4.3 Joint Video–Action Flow Matching

  • Joint flow matching independently noises future-video and action targets, then jointly integrates their predicted velocity fields during inference.The observed condition-frame latent stays fixed, allowing evolving future-video features to participate directly in action computation rather than serving as a separately generated prompt.Source: §4.3 · p. 7

4.4 Subtask-Aware Post-Training

  • Task-specific post-training adds active-subtask text and a normalized phase index derived from the same annotated phase partition to the base video–action objective.The phase index marks the current phase’s position in the partition, not elapsed trajectory time; the subtask decoder attends to action and aligned VL representations rather than an independently executed planner.Source: §4.4 · pp. 7–8

5 Emergent Depth-Wise Specialization

This section measures which sources action queries use at different depths, then tests whether the handoff emerges during training and matters for behavior. It finds a shallow semantic, intermediate predictive, and deep action progression that replicates across tasks and is functionally tested.

5.1 Measuring Modality Allocation

  • The allocation statistic averages attention weights over heads, action queries, samples, and denoising steps, then macro-averages 50 tasks equally.The five sources are VL-text, VL-image, condition-frame, future-frame, and action tokens.Source: §5.1 · p. 8, §5.2 · p. 8

5.2 Emergent Specialization Across Depth

  • Action attention shifts from VL-led shallow layers (75.4%) to mixed intermediate routing, then action-self-attention-dominant deep layers (59.5%).The visual handoff is concentrated in layers 11–15, with a cross-task median raw crossover at layer 13 (IQR 12–13); averaged across layers 16–29, VL-image attention is 1.3% versus 11.8% for predicted future frames.Source: §5.2 · pp. 8–9, Figure 3 · p. 9
  • Across 50 tasks, raw image-to-future handoff appears in all tasks (median crossover layer 13, IQR 12–13), and attention profiles remain stable across denoising steps (mean pairwise correlation 0.998).After task-specific token-count normalization, the handoff remains in 49 of 50 tasks, with median crossover at layer 14.Source: Figure 3 · p. 9, §5.2 · p. 9

5.3 Learning the Visual Handoff during Training

  • At 150 steps, future-frame attention first exceeds VL-image attention at layer 8; by 40,000 steps, it exceeds VL-image attention from layer 13 onward.No layer-wise division is visible at 50 steps, and the mask determines source access but not the depth at which each source is used.Source: §5.3 · p. 9, Figure 4 · p. 10

5.4 Causal Validation through Layer-Targeted Interventions

  • 0.25% success after masking all video keys in layers 10–19, versus 92.80% baseline; masking intermediate future features alone leaves 3.55%.The intermediate future-feature intervention produced the dominant effect; masking condition-frame keys alone retained 81.40% success.Source: Table 1 · p. 10, §5.4 · p. 11
  • Without pretraining, the asymmetric model reaches 83% versus 81% for symmetric full attention; removing shallow VL or intermediate video access reduces success to 78%.Restricting layers 24–29 to action tokens preserves the 83% score.Source: Table 1 · p. 10, §5.4 · p. 10
  • Across five tasks, donor future-feature injection reduces aggregate success from 25/25 to 4/25, with all injected rollouts failing on PLACE EMPTY CUP and STACK BLOCKS TWO.In ADJUST BOTTLE, injected futures initially steer arm choice and spatial targeting; target-conditioned control can re-emerge after injection stops.Source: Table 2 · p. 11, §5.4 · p. 11

6 Human Egocentric Learning: Representation and Data Construction

EWAM uses human egocentric video through camera-relative wrist-motion pretraining and offline robot-compatible demonstrations. Retargeted references pass specified checks but retain measurable tool-position mismatch.

6.1 Human Egocentric Pretraining

  • Human-video pretraining reuses the robot architecture and joint video–action objective, expressing wrist displacements in anchor-camera coordinates and masking unavailable action dimensions.The human-pretraining setup uses consistent motion targets across three egocentric corpora.Source: §6.1 · p. 12
  • Camera-relative wrist actions give human wrists and robot end-effectors a common task-space layout while preserving metric translation, without normalizing embodiment scale or removing kinematic differences.Each hand contributes translation, rotation, and two reserved dimensions; actions in a chunk are referenced to one anchor pose rather than step-to-step increments.Source: §6.1 · p. 12
  • The three corpora require different routes to anchor-camera poses; per-hand validity masks exclude unreliable estimates, while preserved metric scale requires consistent units and calibration.Their camera geometry comes from inverted world-to-camera extrinsics, a direct camera-to-world transform, or SLAM body pose plus stereo calibration.Source: §6.1 · p. 12

6.2 Robot-Compatible Demonstrations from Egocentric Capture

  • The pipeline infers grasp and release from hand–object geometry, then aligns human interaction points to robot tool centers and defines bimanual task-space targets with a grasp schedule.Grasp evidence includes finger-to-mask distances, object occupancy in the pinch region, and changes in hand aperture.Source: §6.2 · p. 14
  • Retargeting optimizes all 14 arm joints over the full sequence while keeping tool targets and grasp schedules fixed, balancing tracking, posture regularization, and temporal smoothness.Grasp timing changes orientation tolerance; joint and derivative bounds constrain the optimization.Source: §6.2 · p. 14
  • Offline human images are paired with robot end-effector trajectories and inferred gripper commands, providing visual-motion correspondence for task-specific co-training.The capture pipeline combines metric wrist tracking with egocentric RGB, depth, and SLAM camera poses; articulated hand estimates add geometry for alignment and grasp inference.Source: §6.2 · pp. 13–15

6.3 Retargeting Quality and Robot Replay

  • All five retargeted references pass specified checks, yet largest per-sequence TCP P95 errors are 77.01 mm for the left tool and 19.69 mm for the right.These measurements compare FK-derived reference positions with human-derived tool targets, not physical tracking error.Source: Table 3 · p. 15, §6.3 · p. 15
  • Figure 7 aligns seven phases of a bimanual pouring demonstration across human recording, simulation replay, and physical Unitree G1-D replay; these are trajectory replays, not learned-policy evaluations.The phases run from grasp through pouring to placement and release.Source: §6.3 · p. 15, Figure 7 · p. 16

7 Experiments

EWAM is evaluated across simulation, cross-embodiment transfer, and three real-robot platforms, with additional studies of robustness, human egocentric data, and long-horizon supervision. Across these settings, it performs strongly against comparison policies, while the targeted studies identify benefits and remaining precision challenges.

7.2 RoboTwin Clean-to-Random Evaluation

  • 72.1% C2R: EWAM achieves the highest C2R and average scores after adaptation on clean demonstrations, exceeding GigaBrain-0.7 by 4.2 C2R points and π0.5 by 26.1.The comparison does not isolate the attention architecture because pretraining data differ across methods.Source: §7.2 · p. 17, Table 4 · p. 17

7.3 RoboTwin In-Domain Results

  • 92.9% average: EWAM leads in-domain RoboTwin results, scoring 93.0% on clean scenes and 92.8% under randomization.It exceeds the strongest baseline in each paradigm by 0.7–2.0 average-success points, with a 0.2-point clean/randomized gap.Source: §7.3 · p. 17, Table 5 · p. 18

7.4 LIBERO Simulation Results

  • 98.8% overall average: EWAM achieves the highest compared-policy score across LIBERO-Spatial, Object, Goal, and Long suites.Its suite scores are 98.6%, 99.8%, 98.6%, and 98.2%, respectively.Source: §7.4 · pp. 17–18, Table 6 · p. 19

7.5 Real-World Evaluation

  • EWAM attains the highest success rate on every reported real-world task across Franka, Dobot, and Unitree G1-D embodiments.Against π0.5, it scores higher on six of seven tasks and ties on the Dobot towel task.Source: §7.5 · p. 18, Table 7 · p. 18, Table 8 · p. 18

7.6 Real-Robot Generalization and Failure Localization

  • Across 30 held-out pour-beans trials, success is 66.7%, 13.3 points below the in-domain baseline; failures cluster around alignment and pouring.Across 40 trials, alignment succeeds in 29 and pouring in 28; spills associated with inaccurate alignment annotate 10 of 12 incomplete-success trials.Source: §7.6 · p. 19, Table 9 · p. 19

7.7 Subtask-Phase Supervision

  • Subtask-phase supervision raises full-episode success on all three simulation block tasks and increases mean objects placed in dual-basket sorting from 1.0/4 to 2.2/4.The simulation gains are instructed ranking 83.5% to 91.0%, stacking 44.5% to 58.5%, and ranking & stacking 1.5% to 16.0%.Source: §7.7 · p. 20, Table 10 · p. 20

7.8.1 Human Pretraining for Cross-Embodiment Transfer

  • At 120K robot-training steps, human-pretrained initialization raises held-out cross-embodiment success from 36.2% to 66.9%, a 30.7-point gain.The compared policies share architecture, downstream robot data, and optimization, differing in initialization.Source: §7.8.1 · p. 20, Figure 8 · p. 21

7.8.2 Human Pretraining for Real-Robot Co-Training; 7.8.3 Effect of Egocentric Co-Training Data

  • With matched 60-robot/30-human co-training, human pretraining raises real-robot success from 40% to 60%; without pretraining, adding 30–1,000 egocentric episodes raises robustness success from 10% to 60%.The latter change is measured under tablecloth and cup changes and does not saturate.Source: §7.8.2 · p. 21, Table 11 · p. 21, Figure 9 · p. 21

8 Discussion

The discussion treats EWAM’s depth-wise handoff as a starting point for adaptive computation and counterfactual planning, while identifying limits in interpretability and computational cost.

From a fixed handoff to adaptive computation

  • Task- and state-conditioned routing could vary the transition between semantic grounding, visual foresight, and action refinement rather than enforcing a fixed schedule.The discussion suggests ambiguous instructions may call for more semantic grounding, uncertain interactions for more prediction, and well-specified motions for earlier action refinement.Source: §8 · p. 22

From visual foresight to counterfactual planning

  • Changing intermediate future-feature content alters spatial targeting and arm selection, suggesting those representations carry control-relevant information.The current model predicts one future trajectory jointly with action rather than comparing alternative outcomes.Source: §8 · p. 22

Limitations and future directions

  • Attention routing and interventions do not provide a complete causal decomposition, and maintaining three large experts remains computationally expensive despite real-time chunked execution.The discussion proposes representation probing and expert-depth computation optimization as future directions.Source: §8 · pp. 22–23

9 Conclusion

EWAM combines semantic and predictive experts through asymmetric attention, and action computation develops a depth-wise progression toward control. The appendix details the architecture and training, then examines cross-task attention enrichment and its evolution during training.

9 Conclusion

  • EWAM’s action queries read from vision-language and video experts through asymmetric attention, with action generation depending on a depth-wise handoff from semantic grounding through predicted futures to action refinement.The conclusion reports performance above VLA, WAM, and hybrid baselines across simulation and real-robot embodiments.Source: §9 · p. 23

A Cross-Paradigm Attention Measurement

  • The cross-paradigm attention comparison uses one randomized RoboTwin episode at denoising step 0, with source groupings determined by each policy’s available streams.Attention allocation is reported stable across tasks and denoising steps; inference steps on the horizontal axis are successive action-chunk queries.Source: A Cross-Paradigm Attention Measurement · p. 31

B.1 Architecture

  • Three depth-aligned transformer experts process semantic input, future-video prediction, and action generation; asymmetric masking lets action queries read all three streams while perceptual queries remain within their own streams.The described shallow, intermediate, and deep block ranges are equal-width descriptive partitions, not hard-coded modules.Source: §B.1 · pp. 31–32

B.2 Training Objectives

  • Pretraining jointly optimizes future-video and action flow-matching losses; task-specific post-training retains both and adds active-subtask text and normalized phase-index supervision.The phase index is regressed from the action-side representation, while the text label identifies the current semantic phase.Source: §B.2 · p. 32

C Token-Count-Normalized Attention Analysis

  • Across 50 RoboTwin tasks, future-frame attention overtakes VL-image attention at layer 13 by raw mass and layer 14 after token-count normalization.The normalized handoff persists after accounting for future frames having twice as many tokens as VL-image inputs.Source: C Token-Count-Normalized Attention… · p. 32, Table 12 · p. 33

C.1 Checkpoint Evolution of the Visual Handoff

  • The attention handoff develops non-monotonically: early checkpoints show changing image and text allocation, while by 40,000 steps the layer-14 future-over-image crossover persists through every later layer.At 5,000 steps no stable crossover is detected; all reported checkpoints come from the same thirty-two-GPU run.Source: §C.1 · p. 33

C.2 Cross-Task Mechanism Replication

Across 50 tasks, action attention follows a shallow vision-language, intermediate visual-handoff, and deep-action progression. The pattern is consistent across tasks and denoising steps, with one task showing a later normalized crossover.

C.2 Cross-Task Mechanism Replication

  • The stable crossover is the first of three consecutive layers where future-frame attention exceeds VL-image attention; its median location and IQR summarize task-level variation.Stage-level uncertainty uses 95% task-bootstrap intervals, while mechanism prevalence uses 95% Wilson intervals.Source: §C.2 · p. 33, Table 13 · p. 34, Table 14 · p. 35
  • 75.38% of shallow attention goes to VL tokens, versus 14.05% to video and 10.58% to action; deep-stage action attention rises to 59.46%.The intermediate stage is more distributed: 36.55% VL, 24.31% video, and 39.14% action.Source: §C.2 · p. 33, Table 14 · p. 35, Figure 12 · p. 35
  • All 50 tasks show shallow VL dominance, an intermediate raw image-to-future handoff, and deep action dominance; the raw crossover clusters around layers 12–13.After token-count control, 49 of 50 tasks show the handoff over layers 13–15; all reach a stable normalized crossover, median layer 14.Source: §C.2 · p. 34, Table 14 · p. 35, Figure 12 · p. 35
  • Denoising-step profiles correlate at 0.998 on average, while STAMP SEAL’s normalized crossover occurs later, at layer 17.STAMP SEAL is the only task without a late-middle normalized handoff; its future-frame enrichment is 0.020 below VL-image enrichment over layers 13–15, yet it retains the raw progression.Source: §C.2 · pp. 34–35

D Attention Routing under a Fully Bidirectional Mask

The fully bidirectional mask retains coarse depth-wise action routing, but loses the asymmetric model’s depth-specific use of the observed image. It also changes what the vision-language stream attends to, and the measurements are limited to routing rather than representational content.

D Attention Routing under a Fully Bidirectional Mask

  • Under bidirectional attention, deep vision-language queries attend 82.0% to video; this routing capture grows during training as deep self-attention declines.The deep video share reaches 82.0% in the converged run, while deep self-attention is 15.3%; the shallow stream instead recovers self-attention early.Source: D Attention Routing under a Fully… · pp. 36, 38, Figure 14 · p. 38, Figure 15 · p. 39
  • The bidirectional mask preserves shallow-VL, intermediate-mixed, and deep-action routing, but image enrichment stays 0.81, 0.70, and 0.67 across stages rather than dropping from shallow reliance.The asymmetric model’s image enrichment is 1.75 shallow and 0.04 deep; the bidirectional model’s image-enrichment ratio is 1.2.Source: D Attention Routing under a Fully… · pp. 36–37, Table 15 · p. 38
  • Future-frame enrichment still peaks at layer 14, but future-over-image ordering persists in only 24% of eligible layers, unlike the asymmetric model’s sustained ordering.The bidirectional profile rises from 0.14 shallow to 0.50 intermediate, then falls to 0.31.Source: D Attention Routing under a Fully… · pp. 36–37, Figure 13 · p. 37
  • These measurements characterize routing, not representational content: residual connections may preserve shallow semantics in deep activations.Determining whether routing changes accompany representational changes requires directly decoding semantic targets from deep vision-language activations.Source: D Attention Routing under a Fully… · p. 38

E.1 RoboTwin 2.0

  • RoboTwin 2.0 evaluation covers 50 bimanual tasks in Clean and Randomized scenes, averaging per-task success rates over 100 episodes per task and setting.The clean-to-random protocol uses no target-embodiment randomized data during downstream adaptation; in-domain results use both clean and randomized adaptation data.Source: §E.1 · p. 39, Table 16 · p. 39

E.2 LIBERO

  • LIBERO measures average task success across Spatial, Object, Goal, and Long suites using one policy and the comparison methods’ observation and action interface.Source: §E.2 · p. 39

E.3 Real-World Evaluation Details

  • Real-robot evaluation spans seven tasks on Franka, Dobot, and Unitree G1-D, with 10 trials per task and success requiring the terminal condition.Initial object poses vary within the reachable workspace, and no manual intervention occurs after execution begins.Source: §E.3 · p. 41, Table 18 · p. 41, Figure 16 · p. 42

E.4.1 Instruction-Conditioned Block Tasks

  • Long-horizon evaluation includes instruction-conditioned block tasks and physical Dual-Basket Sorting; block variants alter the language interface to specify changing active subtasks.Ranking and stacking orders can be instructed separately; the combined task sequences ranking and restacking within one episode.Source: §E.4 · p. 41, §E.4.1 · pp. 41, 43

E.4.2 Real-World Dual-Basket Sorting

Subtask-phase supervision is associated with better completion of long-horizon sequences, while egocentric demonstrations improve robustness to tabletop and cup changes. Without supervision, the physical sorting policy often retries failed grasps instead of switching targets.

E.4.2 Real-World Dual-Basket Sorting

  • Without subtask-phase supervision, the ten sorting trials average 1.0 of four objects placed, with no full clearance; failed grasps typically trigger retries on the same object.The supervised policy clears the table in the illustrated rollout, while unsupervised errors often stall progress.Source: §E.4.2 · p. 43, Figure 17 · p. 44
  • Subtask-phase labels supply a current-goal signal and improve later-stage completion and full-sequence success on instructed block tasks and the physical multi-object sequence.Source: §E.4.2 · p. 43

E.5 Generalization under Tabletop and Cup Changes

  • With robot demonstrations fixed at 60, adding 1,000 egocentric demonstrations raises success from 10% to 60%, a gain of 50 percentage points.Thirty egocentric demonstrations plus 60 robot demonstrations match the 20% success of 90 robot-only demonstrations; success rises from 20% to 60% as egocentric data increase from 30 to 1,000, with no sign of saturation in this range.Source: §E.5 · p. 44

F Inference Latency and Real-Time Deployment

EWAM generates 16-action chunks in 258 ms on one RTX 5090 using five denoising steps. Its separate human and robot pretraining regimes use large egocentric-video and cross-embodiment trajectory datasets, with human hand targets standardized across corpora.

F Inference Latency and Real-Time Deployment

G Pretraining Data

  • The two separate pretraining regimes use approximately 2,084 hours of human egocentric video and 300K cross-embodiment robot trajectories comprising approximately 1,800 hours.The robot data provide synchronized observations, language, states, actions, and future visual observations; heterogeneous spaces use shared padded representations and validity masks.Source: G Pretraining Data · pp. 45–46

G.1 Human Egocentric Data

  • Human pretraining combines VITRA-1M, EgoDex, and Xperience, whose datasets offer complementary coverage and per-frame hand pose and camera geometry.VITRA-1M uses four of five subsets; EgoDex observations below 0.3 confidence are discarded, and Xperience episodes lacking SLAM poses or stereo calibration are dropped.Source: §G.1 · p. 46, Table 21 · p. 46
  • Retained human episodes become world-frame wrist poses plus an anchor camera pose, then camera-relative targets with consistent physical meaning across the mixture.Corpus sampling is weighted by indexed windows, and no action normalization is applied.Source: §G.1 · p. 46

G.2 Robotic Manipulation Data

EWAM’s robotic pretraining corpus is curated from diverse manipulation recordings through filtering and staged quality checks. The appendix also details how egocentric hand reconstructions are mapped and retargeted into robot reference trajectories.

G.2 Robotic Manipulation Data

  • The corpus spans robot platforms, tasks, and objects; platform joint-state dimensionalities vary between single-arm and bimanual setups.Tasks include grasping, placing, moving, organizing, switching, and cleaning.Source: §G.2 · pp. 46, 48, Table 22 · p. 48
  • Dataset defects include corrupted or occluded videos, inactivity, state-action misalignment, and extreme numerical outliers that can impede optimization.A coarse-to-fine pipeline removes trajectory-level visual and alignment defects, then frame-level numerical anomalies while preserving continuity and consistency.Source: §G.2 · p. 48
  • Approximately 1,800 hours of high-quality robotic manipulation data remain after filtering roughly 3,250 hours of raw recordings for tabletop relevance and quality.The corpus is assembled from multiple open-source datasets and simulation data.Source: §G.2 · pp. 46, 48

H Egocentric Reconstruction and Retargeting Details

  • A shared episode-level rigid transform maps both hands from the reconstruction world into the robot base, while a fixed basis maps wrist orientation to tool orientation.The transform aligns axes and sets placement using a real-robot reference trajectory; tool targets use the nominal pinch-center TCP.Source: H Egocentric Reconstruction and… · p. 49
  • Retargeting minimizes grouped TCP-position, orientation-axis, posture, and temporal costs subject to joint, velocity, and acceleration constraints.Workspace and collision conditions are also checked on the returned trajectory; the grouped objective need not have a global minimizer.Source: H Egocentric Reconstruction and… · p. 49
  • Hand detections and associations are tracked in rectified images, with short gaps interpolated and valid articulation segments smoothed before reconstruction outputs are retained.The retained outputs include world-frame wrist poses, wrist-local geometry, and reconstruction-source and validity indicators.Source: H Egocentric Reconstruction and… · pp. 49–50
  • Trajectory refinement uses trust-region quadratic subproblems, multiple inverse-kinematics initializations, staged warm starts, and an offline prefix refinement of the first 1.2 seconds.Targets, gripper commands, and the original timeline remain fixed during refinement.Source: H Egocentric Reconstruction and… · p. 50
  • Articulated hands receive palm-frame and scale alignment plus fingertip inverse kinematics, while exported reference actions are audited separately from physical task performance.The audit checks limits, workspace, collision clearance, forward-kinematics consistency, and signal preservation; grasp states do not measure contact forces.Source: H Egocentric Reconstruction and… · p. 50

Related papers

Loading 2609.39973v1…