PaperlayerView exact revision on arXiv

Source-linked AI summary

Ego-CEVR: Learning Cross-Embodiment Visual Representations from Human Egocentric Videos

Keru Zhou, Zhirui Fang, Xinrun Xu, Yaopeng Lou, Zeyuan He, Zihao Zhu, Xinze Niu, Chen Wang, Tianfu Li, Samith Va, Bin Qian, Junwei Li, Wenjia Ding, Yicheng Xu, Chang Zhou, Ziwei Wang, Li Liu, Xiu Li, Maoqing Yao

arXiv:2610.37509v1cs.RO

TL;DR

Robot manipulation needs to make use of scalable egocentric video alongside robot demonstrations that provide fine-grained action supervision. Ego-CEVR assigns video prediction to global task representations and robot action supervision to wrist-based interaction representations. Across simulation and real-robot experiments it reports competitive manipulation performance and improved robustness under visual perturbations.

  • Problem

    A central question is how to turn scalable egocentric video into robot manipulation capabilities while using diverse embodied data with different forms of supervision.

    Source: §1 · p. 2

  • Method

    Ego-CEVR learns global task context through future prediction on egocentric and head-camera video, then learns wrist-based interaction representations under robot action supervision.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across simulation and real-robot experiments, Ego-CEVR achieves competitive manipulation performance and improved robustness under visual perturbations.

    Source: §6 · p. 10

  • Takeaways & Limitations

    On real robots, additional human egocentric videos improve manipulation performance and support transfer to a new task, while few-shot demonstrations align video-learned knowledge with fine-grained actions on a new robot.

    Source: §6 · p. 10

    Large-scale human egocentric-video pretraining and experiments across different data and model sizes were not conducted, leaving scaling behavior an open question.

    Source: §5 · p. 10

Abstract

from arXiv · show

As egocentric video becomes increasingly available for embodied learning, an important research question is how to turn this low-cost, scalable source of data into robot manipulation capabilities. Egocentric videos naturally provide rich context for high-level task understanding, while real-robot demonstrations with wrist observations and action annotations provide supervision for fine-grained action alignment. We present Ego-CEVR, a data-native framework for hierarchical visual representation learning that assigns distinct learning objectives to global and interaction representations. The global module learns object relations, task state, and temporal progress through future prediction on egocentric video. The Interaction module interprets wrist observations in global context and learns local alignment and contact cues through robot action supervision. Separate residual paths expose both representations to the action policy, preserving direct access to task context alongside fine-grained execution information. Ego-CEVR supports independent pretraining of its global video module on large-scale human egocentric videos and videos from diverse robot embodiments, without requiring robot action labels or paired wrist observations.

1 Introduction

The introduction asks how heterogeneous embodied data can be organized for manipulation, given that task-level video context and fine-grained action supervision serve different roles. Ego-CEVR assigns those roles to hierarchical representations and reports improvements in task success and robustness across simulation and real-robot experiments.

1 Introduction

  • The motivating challenge is designing data-native models that use diverse embodied sources with different forms of supervision.The introduction notes that the role of different data types in robot learning remains an open design question.Source: §1 · p. 2
  • Semantic generalization may not ensure faithful execution, and video prediction alone may not ground local alignment and contact cues in robot actions.The paper motivates combining semantic understanding, world modeling, and action generation while specifying which representations can be learned without actions.Source: §1 · p. 2
  • Ego-CEVR learns global task context from egocentric and robot head-camera video, while robot action supervision trains local interaction representations from wrist observations.Wrist features query the global representation, while the global representation is computed independently of the interaction module.Source: §1 · p. 2
  • Experiments on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real robots show improved task success and robustness.Ablations examine hierarchical representation learning and multi-source video pretraining.Source: §1 · p. 3
  • The framework combines multi-source egocentric video and action-labeled robot demonstrations at different training stages to learn task-level and local execution knowledge.Source: §1 · p. 3
  • Ego-CEVR includes a few-shot cross-embodiment adaptation procedure that combines video-learned task knowledge with a few target-robot demonstrations.Source: §1 · p. 3

2 Related Work

Related work covers predictive video and vision-language methods for manipulation, alongside approaches that transfer knowledge from human egocentric videos. It motivates organizing video-derived task knowledge and robot-supervised local interaction according to the observations and supervision available.

Vision-language-action policies

  • Vision-language-action models support semantic generalization and multitask manipulation, yet image-text knowledge does not directly specify manipulation state changes or how actions realize instructions.Source: §2 · p. 3

Predictive world models for robot control

  • Prior predictive-control methods use future visual states or video-model representations, but plausible predictions may still lack the spatial precision required for fine manipulation.The cited discussion notes that prediction errors can propagate into action errors on unseen tasks and scenes.Source: §2 · pp. 3–4

Integrating vision-language understanding and world modeling

  • Prior work integrates vision-language understanding, world prediction, and action generation, but Head and Wrist observations raise distinct learning-role questions.Ego-CEVR uses head-video prediction for global representations, conditions interaction representations on them, and gives the action expert separate paths to both.Source: §2 · p. 4

Cross-embodiment learning from heterogeneous egocentric data

  • Human egocentric video offers diverse interaction coverage, while some transfer approaches rely on motion estimation or alignment that affects usable data and supervision.Ego-CEVR’s shared Head/Ego video interface supports independent global learning without a common action space or paired Wrist observations; robot demonstrations ground local interaction and action mapping.Source: §2 · p. 4

3 Method

Ego-CEVR separates task-level context learned from egocentric video from local interaction features learned using wrist observations and robot action supervision. Its action expert receives both representations alongside the base policy’s existing conditioning.

3.1 Overview and Data Formulation

  • The framework combines first-person video data, which lacks paired wrist observations and action labels, with robot demonstrations that provide head- and wrist-camera views and action annotations.The global module can be pretrained independently of wrist observations and action annotations.Source: §3.1 · pp. 4–5

3.2 Predictive Global Representation Learning

  • Future-video prediction trains the global module to produce task-level representations from current egocentric observations, and inference extracts them without generating future video.The prediction loss models visual changes during manipulation; video pretraining needs video and text conditions but no shared robot action space.Source: §3.2 · p. 5

3.3 Global-Conditioned Interaction and Action

  • The interaction module combines wrist features with global context, while separate attention branches expose global and interaction representations to the action expert.Action supervision trains the expert and branches for action prediction and encourages the interaction representation to retain information relevant to local execution.Source: §3.3 · pp. 5–6

4 Experiments

Experiments evaluate Ego-CEVR on simulation benchmarks and real robots, testing manipulation performance, robustness, hierarchical representation learning, and transfer across video sources, tasks, and embodiments.

4.1 Evaluation Setup

  • Evaluations cover LIBERO, LIBERO-Plus, and RoboTwin 2.0, using task success rate and controlled comparisons that hold key training and evaluation settings consistent.Real-robot experiments use AgiBot G1 and G2 platforms for transfer and investigate human egocentric and robot head-camera videos.Source: §4.1 · p. 6, §4 · p. 6

4.2 Manipulation and Robustness

  • Ego-CEVR achieves 97.7% average success on LIBERO, 90.2% overall on LIBERO-Plus, and 72.2% average on RoboTwin 2.0, leading listed methods on the latter two benchmarks.On RoboTwin 2.0, it also leads the Hard setting; LIBERO-Plus results are highest under camera, background, and noise perturbations.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Table 1 · p. 7

4.3 Ablation Study on Hierarchical Representation

  • Compared with flat feature fusion, hierarchical representations raise success from 73.8% to 90.2% on LIBERO-Plus and from 66.7% to 91.1% under camera perturbations.The authors attribute the gains to global task context combined with interaction representations that learn local execution cues conditioned on that context.Source: §4.3 · p. 8, Table 4 · p. 8
  • Jointly predicting Head and Wrist video yields 97.9% on LIBERO but 87.4% on LIBERO-Plus, versus Ego-CEVR’s 97.7% and 90.2%, respectively.The authors suggest wrist-camera motion may encourage overfitting and support learning global context from Head video while using robot action supervision for Wrist-based interaction representations.Source: §4.3 · pp. 8–9

4.4 Ego and Cross-Embodiment Knowledge Transfer

  • With the same video duration, human egocentric videos improve success over the robot-head-only baseline on Plug Insertion, Water Pouring, and Package Sorting.Human-video results are 30.0%, 75.0%, and 80.0%, respectively; on Package Sorting with a five-finger dexterous hand, this is close to the 85.0% achieved with robot head-camera videos.Source: §4.4 · p. 9, Table 5 · p. 9
  • For transfer from Water Pouring to Block Stacking, human egocentric videos raise Task 2 success from 0.0% to 10.0%, while G2 head-camera videos raise it to 20.0%.Both video settings also raise Task 1 success from 60.0% to 65.0%.Source: Table 6 · p. 9, §4.4 · p. 10
  • With five G1 demonstrations fixed, human egocentric videos raise G1 success from 5.0% to 15.0%, while G1 head-camera videos raise it to 35.0%; 50 G1 demonstrations reach 60.0%.The five target-robot demonstrations provide action supervision for aligning video-learned representations with fine-grained actions.Source: Table 6 · p. 9, §4.4 · p. 10

5 Discussion and Limitations

The discussion argues that diverse embodied data call for models that assign learning objectives suited to different observations and supervision. Whether Ego-CEVR benefits from much larger video collections or scales with larger models remains open.

5 Discussion and Limitations

  • The authors argue that data-native models should learn from different observations and forms of supervision because embodied data sources provide different kinds of information.They suggest that how a model organizes data and assigns learning objectives may limit what it learns as data scale grows.Source: §5 · p. 10
  • Effective scaling across increasingly diverse embodied data remains unresolved, and the authors propose multistage hierarchical training as an important direction.They note that transferring knowledge from egocentric video to fine-grained robot actions would still require a distinct action-alignment stage if video generation scales successfully.Source: §5 · p. 10
  • Resource constraints prevented large-scale human egocentric-video pretraining and large-scale experiments across data and model sizes, leaving scaling behavior untested.Whether the framework continues to benefit from substantially larger video collections remains open.Source: §5 · p. 10

6 Conclusion

Ego-CEVR learns complementary global and interaction representations through video prediction and robot action supervision. Experiments report competitive manipulation performance, improved robustness to visual perturbations, and benefits from human-video data and few-shot transfer.

6 Conclusion

  • Ego-CEVR learns complementary global and interaction representations from video prediction and robot action supervision.Source: §6 · p. 10
  • Ego-CEVR achieves competitive manipulation performance and improved robustness under visual perturbations across simulation and real-robot experiments.Source: §6 · p. 10
  • Ablations show benefits from hierarchical representations and assigning different learning objectives to head and wrist observations.Source: §6 · p. 10
  • On real robots, additional human egocentric videos improve manipulation performance and support transfer to a new task.Source: §6 · p. 10
  • Few-shot demonstrations align video-learned knowledge with fine-grained actions on a new robot, reflecting cross-embodiment transfer.The conclusion frames human-to-robot learning as sharing this principle with transfer between robot embodiments.Source: §6 · p. 10

Related papers

Loading 2610.37509v1…