PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

Meng Cao, Can Zhao, Jinjie Wei, Haonan Zhao, Yulong Zhang, Duo Han, Xiaofan Li, Ziwu Song, Rongtao Xu, Ruyi Gan, Hao Wang, Qian Wang, Daolin Ma, Ian Reid, Xiaodan Liang

arXiv:2610.20904v1cs.ROcs.CV

TL;DR

Robot learning from costly demonstrations motivates using scalable egocentric human videos, but transferring their manipulation priors without embodiment-specific retargeting remains challenging. EgoJEPA learns effect-aware latent actions, jointly models visual dynamics and actions, and adapts the shared representation with robot-specific adapters. Across five tasks and three embodiments, it reports a 9% absolute success-rate margin over retargeting and 47.2% lower cumulative cross-embodiment training time.

  • Problem

    Robot demonstrations are costly to collect, while retargeting human hand motion requires embodiment-specific mappings and is sensitive to annotation noise.

    Source: §1 · pp. 1–2

  • Method

    EgoJEPA learns effect-aware latent actions from human videos, jointly models latent actions and future visual dynamics, then adapts the shared model with lightweight robot-specific action adapters.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    Across five real-world tasks and three embodiments, EgoJEPA outperforms retargeting by an absolute 9% under out-of-domain evaluation while reducing cumulative cross-embodiment training time by 47.2%.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results support reusing embodiment-agnostic human-video pre-training across robots through lightweight action adapters.

    Source: §1 · p. 2

    Severe temporal misalignment or prolonged occlusion in egocentric video can distort latent representations, and fine-grained contact tasks remain challenging.

    Source: §A.4 · p. 18

Abstract

from arXiv · show

Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.

1 INTRODUCTION

Robot demonstrations are costly to collect, while egocentric human videos offer a scalable source of manipulation supervision. EgoJEPA replaces embodiment-specific retargeting with shared latent actions and reports stronger out-of-domain performance alongside lower cross-embodiment training cost.

1 INTRODUCTION

  • Robot demonstrations gathered through teleoperation or specialized capture systems are costly, labor-intensive, and time-consuming, limiting scalable robot learning.Egocentric human videos are presented as a scalable source of supervision for transferring manipulation skills.Source: §1 · p. 1
  • Retargeting requires morphology-specific geometric mappings and repeated recalibration, while hand-pose noise can propagate into infeasible robot actions.With σ = 10 mm per-axis position noise, bottle-cap-unscrewing success falls from 60% to 30%.Source: §1 · p. 2
  • 74% average out-of-domain success across five real-world tasks and three robot embodiments, with cumulative wall-clock cost reduced from 743.2 to 392.7 hours versus retargeting.EgoJEPA also outperforms LAPA and DexWild on average success rate.Source: §1 · p. 2
  • EgoJEPA learns effect-aware, embodiment-agnostic latent actions from human videos, models visual dynamics and latent actions jointly, then adapts to robot action spaces with lightweight adapters.This three-stage framework avoids rule-based human-to-robot retargeting.Source: §1 · p. 2

2 RELATED WORK

Related work includes latent-based learning from video, retargeting-based transfer from explicit hand motion, and action-conditioned latent world models. EgoJEPA is positioned as combining visual grounding of human motion with transfer across robot embodiments.

WAMs for Dexterous Manipulation

  • Dexterous world-action models predict future state alongside action, but prior approaches rely on embodiment-specific robot demonstrations, limiting cross-embodiment transfer and scalability.EgoJEPA learns transferable manipulation priors from egocentric human videos.Source: §2 · p. 3

Robot Learning from Human Videos

  • Action-free latent methods learn from inter-frame changes or future visual dynamics, but may conflate controllable hand motion with camera motion and passive scene dynamics.This can make their representations less interpretable as action representations.Source: §2 · p. 3
  • Retargeting methods map captured wrist and finger poses into robot-specific action spaces, but remain tied to embodiment-specific geometry and sensitive to hand-tracking noise.EgoJEPA instead grounds explicit human hand motion in visual effects to support transfer without rule-based retargeting.Source: §2 · p. 3

Joint-Embedding Predictive Architectures

  • JEPA work predicts future latent representations, and EgoJEPA explores human motion as a physical condition for latent world modeling across robot embodiments.Prior JEPA studies include self-supervised representation learning and action-conditioned latent world modeling.Source: §2 · p. 3

3 EGOJEPA

EgoJEPA transfers manipulation priors from egocentric human videos through three stages: learning effect-aware latent actions, modeling visual dynamics and latent actions jointly, and grounding the shared representation in a target robot.

3 EGOJEPA

  • EgoJEPA has three stages: human-action pre-training, latent-action world modeling, and embodiment-specific tuning for downstream robot manipulation.The framework learns embodiment-agnostic action representations from egocentric human videos.Source: §3 · p. 3

3.1 STAGE 1: HUMAN-ACTION PRE-TRAINING

  • Stage 1 defines human action as adjacent-frame hand-annotation differences and trains a JEPA objective to predict future visual representations from context and hand-motion differences.The action representation is thereby grounded in visual state transitions rather than rule-based embodiment-specific retargeting.Source: §3.1 · pp. 3–4

3.2 STAGE 2: LATENT-ACTION-BASED WORLD MODELING

  • Stage 2 jointly models future visual dynamics and latent actions using flow matching, with the action branch supervised by frozen Stage 1 JEPA action embeddings.The model predicts the velocity field from a noisy joint state toward the clean video-action target.Source: §3.2 · pp. 4–5

3.3 STAGE 3: EMBODIMENT-SPECIFIC TUNING

  • Stage 3 initializes the Stage 2 world-action model and inserts embodiment-specific action adapters that map robot action tokens to and from the shared action-token space.Robot supervision uses relative action chunks aligned with the hand-motion representation learned in Stage 1.Source: §3.3 · p. 5

4 EXPERIMENTS

Across five manipulation tasks and three robot embodiments, EgoJEPA achieves the highest reported in-domain and out-of-domain success rates. The experiments also examine human-video pre-training, data efficiency, annotation robustness, and cross-embodiment training cost.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers five real-world tasks across Sharpa, Wuji, and Xense Tac UMI, with object and spatial shifts tested without fine-tuning.Each setting uses 20 trials per task, and success means full task completion.Source: §4.1 · p. 6

4.2 REAL-WORLD PERFORMANCE

  • 74% out-of-domain average success rate exceeds retargeting’s 65%, while EgoJEPA reaches 94% in-domain and leads on all five tasks.All methods score lower out of domain; EgoJEPA’s out-of-domain average exceeds DexWild’s 40%.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 ABLATION STUDIES

  • Human pre-training raises average out-of-domain success from 40% for robot-only training to 74%, with gains on all five tasks.On bottle cap unscrewing, success rises from 30% to 70%; EgoJEPA also converges faster with lower Stage 3 action loss.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
  • EgoJEPA’s JEPA pre-training reaches 74% average out-of-domain success versus 63% for the VQ-VAE variant, with gains across all five tasks.The comparison replaces Stage 1 predictive learning with action reconstruction while keeping data and downstream training protocol the same.Source: Figure 3 · p. 7, §4.3 · p. 8
  • With 10% of robot demonstrations, EgoJEPA achieves 35% success on bottle cap unscrewing, versus 30% for robot-only training with 100%.At 50% of demonstrations, EgoJEPA reaches 65%, exceeding retargeting with the full dataset at 60%.Source: §4.3 · p. 8, Figure 5 · p. 8
  • At 10 mm position noise, EgoJEPA’s success drops 10 percentage points, from 70% to 60%, while retargeting drops 30 points, from 60% to 30%.Temporal jitter causes the greatest decay; at its maximum, EgoJEPA reaches 50%, compared with 65% under 40% annotation dropout.Source: Figure 6 · p. 8, §4.3 · p. 9
  • Across five tasks, EgoJEPA cuts cumulative cross-embodiment wall-clock training time from 743.2 to 392.7 hours, a 47.2% reduction.For cardboard box folding with the Xense Tac UMI gripper, Stage 3 tuning takes 37.5 hours versus 193.4 hours for retargeting.Source: §4.3 · p. 9, Figure 7 · p. 9

5 CONCLUSION

EgoJEPA transfers manipulation priors from egocentric human videos to robot control without rule-based retargeting, improving out-of-domain success and reducing cross-embodiment training time. It also shows greater robustness to human annotation noise.

5 CONCLUSION

  • EgoJEPA outperforms the retargeting pipeline by 9% under out-of-domain evaluation and reduces cumulative cross-embodiment training time by 47.2%.The paper also reports greater robustness to human annotation noise.Source: §5 · p. 10
  • EgoJEPA grounds latent actions in visual effects and integrates them into world modeling, decoupling human pre-training from embodiment-specific control.Shared pretrained models can be reused across robots through lightweight action adapters.Source: §5 · p. 10
  • The experiments cover five real-world tasks across three robot embodiments.The paper also reports greater robustness to human annotation noise.Source: §5 · p. 10
  • The authors used generative AI to assist literature searches, manuscript polishing, and experimental design, but not to generate synthetic datasets or implement the method.They state that AI-assisted content was reviewed against experimental records and that they take responsibility for the final work.Source: AI USE STATEMENT · p. 10

A.1 DATASET DETAILS

The appendix describes human pre-training and robot demonstration datasets, their coverage and scope, and the privacy and licensing safeguards applied to purchased data.

A.1 DATASET DETAILS

  • The human pre-training data spans varied episode durations and frame counts, and includes egocentric sequences with aligned hand-keypoint annotations.The appendix also describes frequent tasks and scenes across household activities and industrial operations.Source: §A.1 · p. 14, Figure A2 · p. 15
  • Robot demonstrations cover five manipulation tasks across the Sharpa hand, Wuji hand, and Xense Tac UMI gripper, with action dimensions of 56, 52, and 14, respectively.Demonstration durations range from brief grasping interactions to longer coordinated manipulation sequences.Source: §A.1 · p. 14, Table A2 · p. 15
  • Commercial datasets were licensed for noncommercial academic research and algorithmic evaluation, and providers anonymized human subjects and removed or blurred personally identifiable information.Purchased assets are stored on restricted-access servers; no biometric identifiers or sensitive personal characteristics were collected, inferred, or processed.Source: §A.1 · pp. 14–15

A.2 IMPLEMENTATION DETAILS

The implementation appendix outlines the three-stage training setup and details a VQ-VAE comparison that replaces Stage 1 predictive learning with hand-motion reconstruction. It also describes embodiment-specific retargeting implementations.

A.2 IMPLEMENTATION DETAILS

  • Stages 1–2 use egocentric human data, while Stage 3 initializes from the Stage 2 checkpoint and tunes video and action Transformers with embodiment-specific adapters on robot data.Stage 3 also uses three camera views and proprioceptive conditioning.Source: §A.2 · p. 15, Table A3 · p. 17
  • The VQ-VAE alternative encodes and quantizes hand annotations, then decodes them back to the annotation space for reconstruction.Its objective learns discrete per-frame hand representations without visual-effect supervision; Stages 2 and 3 retain the same training protocols.Source: §A.2 · pp. 15–17
  • Retargeting converts human wrist poses and 3D hand keypoints into embodiment-specific robot targets using calibrated coordinate transforms or a calibrated gripper-opening mapping.Retargeted trajectories are aligned with video frames and converted into relative actions consistent with robot demonstrations.Source: §A.2 · p. 17

A.3 ROBUSTNESS EVALUATION PROTOCOL

The robustness protocol perturbs human hand-motion annotations with position noise, temporal jitter, or annotation dropout during pre-training. Both methods use identical perturbations and are adapted and evaluated with clean robot data on out-of-domain bottle cap unscrewing.

A.3 ROBUSTNESS EVALUATION PROTOCOL

  • The protocol independently applies position noise, temporal jitter, and annotation dropout to human pre-training annotations, while leaving Stage 3 training unchanged.EgoJEPA and the retargeting pipeline receive identical perturbations and are evaluated on bottle cap unscrewing under out-of-domain conditions.Source: §A.3 · p. 17
  • Position noise adds an independently sampled Gaussian translation to all keypoints of a hand at each frame, with σ values of 0, 2, 5, or 10 mm.The same translation is applied to the wrist, preserving hand geometry and leaving rotation annotations unchanged; adaptation and evaluation use clean robot data.Source: §A.3 · p. 17
  • Temporal jitter shifts annotations relative to video by a clip-level offset, with maximum offsets of 0, 100, 200, or 400 ms at 30 Hz.The offset is shared across both hands and held constant within each clip, preserving motion continuity while disrupting visual–motion alignment.Source: §A.3 · pp. 17–18
  • Annotation dropout masks contiguous segments, then reconstructs positions by linear interpolation and rotations by spherical linear interpolation.Both methods use the same masks and interpolation rules for human pre-training, followed by adaptation and evaluation on clean robot data.Source: §A.3 · p. 18

A.4 BROADER IMPACT AND LIMITATIONS

EgoJEPA is presented as lowering labor and computational barriers to multi-embodiment dexterous deployment, while carrying risks from inherited data bias and physical hazards. The authors identify limits under severely degraded egocentric video and contact-intensive tasks.

Broader Impact

  • EgoJEPA is presented as lowering computational and labor barriers to deploying multi-embodiment dexterous systems in real-world settings.The broader-impact discussion names domestic assistance and industrial automation as example settings.Source: §A.4 · p. 18
  • Scaling learning from uncurated video datasets risks inheriting or amplifying biases in human demonstration environments.Source: §A.4 · p. 18
  • Physical deployment of more capable autonomous dexterous systems requires safety, alignment, and strict containment protocols to prevent collisions or operational hazards.Source: §A.4 · p. 18

Limitations

  • Severe temporal misalignment or prolonged occlusion in egocentric videos can distort latent representations and bottleneck downstream task performance.The authors note that EgoJEPA exhibits noise robustness, but extreme action-quality degradation remains a limitation.Source: §A.4 · p. 18
  • Performance drops on out-of-domain tasks requiring fine-grained contact dynamics, such as bottle-cap unscrewing.The authors identify explicit interaction modeling or force-feedback priors in the latent action space as future directions.Source: §A.4 · p. 18

Related papers

Loading 2610.20904v1…