PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World

Ryan Punamiya, Simar Kareer, Zeyi Liu, Josh Citron, Ri-Zhao Qiu, Xiongyi Cai, Alexey Gavryushin, Jiaqi Chen, Davide Liconti, Lawrence Y. Zhu, Patcharapong Aphiwetsa, Baoyu Li, Aniketh Cheluva, Pranav Kuppili, Yangcen Liu, Dhruv Patel, Aidan Gao, Hye-Young Chung, Ryan Co, Renee Zbizika, Jeff Liu, Xiaomeng Xu, Haoyu Xiong, Geng Chen, Sebastiano Oliani, Wenkai Xuan, Chenyu Yang, Xi Wang, James Fort, Richard Newcombe, Josh Gao, Jason Chong, Garrett Matsuda, Aseem Doriwala, Marc Pollefeys, Robert Katzschmann, Xiaolong Wang, Shuran Song, Judy Hoffman, Danfei Xu

arXiv:2604.07607v2cs.ROcs.CV

TL;DR

Robot learning needs scalable, diverse data, but robot demonstrations are expensive to collect and existing human datasets are limited and difficult to extend. EgoVerse provides a collaborative dataset and ecosystem with standardized processing, annotations, and access, alongside replicated cross-embodiment transfer experiments. The study finds that human data improves robot performance, while effective scaling depends on aligned human–robot data and diversity affects generalization by axis.

  • Problem

    Robot demonstrations are expensive and difficult to scale, while existing human datasets are limited, static, and fragmented for human-to-robot transfer.

  • Method

    EgoVerse combines a collaborative egocentric human dataset, the continuously growing EgoDB ecosystem, and standardized cross-lab experiments across tasks and robot embodiments.

  • Results

    Human data improves robot performance, with EgoVerse-A increasing in-domain and out-of-domain performance by up to 30%; positive scaling requires aligned human–robot data.

  • Takeaways & Limitations

    EgoVerse establishes a shared foundation for reproducible human data-driven robot learning in which alignment and diversity shape transfer effectiveness.

  • Takeaways & Limitations

    The study mainly evaluates human-and-robot co-training, and diversity effects rely on offline metrics without confirming improved robotic-manipulation rollouts.

Abstract

from arXiv · show

Robot learning increasingly depends on large and diverse data, yet robot data collection remains expensive and difficult to scale. Egocentric human data offer a promising alternative by capturing rich manipulation behavior across everyday environments. However, existing human datasets are often limited in scope, difficult to extend, and fragmented across institutions. We introduce EgoVerse, a collaborative platform for human data-driven robot learning that unifies data collection, processing, and access under a shared framework, enabling contributions from individual researchers, academic labs, and industry partners. The current release includes 1,362 hours (80k episodes) of human demonstrations spanning 1,965 tasks, 240 scenes, and 2,087 unique demonstrators, with standardized formats, manipulation-relevant annotations, and tooling for downstream learning. Beyond the dataset, we conduct a large-scale study of human-to-robot transfer with experiments replicated across multiple labs, tasks, and robot embodiments under shared protocols. We find that policy performance generally improves with increased human data, but that effective scaling depends on alignment between human data and robot learning objectives. Together, the dataset, platform, and study establish a foundation for reproducible progress in human data-driven robot learning. Videos and additional information can be found at https://egoverse.ai/

I. INTRODUCTION

Robot demonstration collection is expensive and difficult to scale, motivating egocentric human data as a richer, more accessible source for robot learning. EgoVerse combines a continuously growing dataset and ecosystem with a replicated cross-embodiment study of human-to-robot transfer.

  • Robot demonstration collection requires physical hardware, expert teleoperation, and controlled setups, making larger and more diverse datasets slow, expensive, and difficult to sustain.
  • Egocentric human data capture manipulation across diverse everyday environments and provides a robot-independent abstraction for downstream embodiment decisions.
  • EgoVerse addresses unresolved human-to-robot transfer and static human datasets through a continuously growing ecosystem for new contributors and durable transfer insights.
  • 1,362 hours across 240 scenes, 1,965 tasks, and 2,087 demonstrators combine controlled reproducibility with industry-sourced scale, diversity, and manipulation-relevant annotations.
  • EgoDB unifies heterogeneous contributions through standardized processing, storage, access, visualization, and downstream-learning interfaces, including a phone-based capture pipeline.
  • Cross-lab experiments across three robot embodiments show improved policy performance with co-trained human data, while scaling requires aligned human–robot data and diversity affects generalization differently.

III. THE EGOVERSE DATASET: A HUMAN DATASET FOR ROBOT LEARNING FROM AROUND THE WORLD

The EgoVerse dataset combines standardized egocentric capture with scalable cloud processing and controlled diversity across tasks, scenes, objects, and demonstrators. Its collection design preserves shared task semantics while enabling heterogeneous contributions and downstream use.

  • Capture systems: Project Aria glasses standardize academic capture, while industry rigs and a phone-based system support scalable and broadly accessible egocentric recording.
  • Annotations: Each frame includes 3D hand pose from 21 keypoints per hand and calibrated 6-DoF head pose, with source-specific tracking and post-processing pipelines.
  • EgoDB: EgoDB continuously ingests heterogeneous human and robot data, converts it into a unified training-ready format, and supports standardized preprocessing, validation, indexing, and structured queries.
  • Collection protocol: Dataset units use common instructions, approximately 5 minutes of recording, 5–10 demonstrations per task, visibility constraints, and traceable metadata.
  • Flagship tasks: Six shared flagship tasks span single-arm and bimanual coordination, fine-grained placement, and longer-horizon manipulation while remaining feasible for common robot platforms.
  • Diversity axes: Scenes, objects, and demonstrators vary systematically across sites, introducing differences in environment, object properties, motion patterns, timing, coordination, morphology, and viewpoints.
  • Controlled diversity: The controlled-diversity subset independently scales scene and demonstrator diversity, separating these factors under fixed data budgets and structured assignments.

E. The EgoVerse-I Dataset

EgoVerse-I complements the controlled EgoVerse-A subset with broad open-ended task diversity and substantially wider visual coverage. This expansion targets generalist policy learning beyond lab-specific appearance statistics.

  • EgoVerse-A densely covers controlled flagship tasks but lacks the task diversity required to train generalist policies.
  • Nearly 1,400 hours across nearly 2,000 tasks, 240 scenes, and 2,087 demonstrators make EgoVerse-I a large action-labeled egocentric dataset.
  • EgoVerse-I substantially expands visual coverage over single-lab robot data and EgoVerse-A for the fold-clothes task despite shared task semantics.
  • The expanded visual diversity is identified as critical for policies intended to generalize beyond lab-specific appearance statistics.

IV. THE EGOVERSE STUDY: A CONSORTIUM-SCALE EXAMINATION OF HUMAN-TO-ROBOT TRANSFER

The study examines human-to-robot transfer across multiple labs, robot platforms, controllers, and environments using shared protocols. It aligns human and robot representations through camera-centered frames, normalized actions, and embodiment-specific action mappings.

  • Study Design: The experimental design prioritizes reproducibility by replicating robot experiments across multiple labs with distinct platforms, controllers, and environments.Shared protocols and evaluation criteria are used to identify findings that extend beyond system-specific effects.
  • Robot Platforms: Three robot platforms span varied kinematics, sensing configurations, and control interfaces for cross-system evaluation.The platforms include two ARX5-based systems and a Unitree G1 with dexterous hands.
  • Robot Action Representations: Robot B uses quaternion orientations with gripper states, while Robot C combines base-frame SE(3) wrist trajectories with hand keypoints mapped through inverse kinematics.These representations accommodate different embodiment-specific action spaces.

C. Learning Architecture and Algorithm

The learning system uses a transformer-based cross-embodiment policy with modality-specific input processing and shared token representations. Human and robot demonstrations are jointly optimized with a behavior-cloning co-training objective.

  • Architecture: The policy uses an encoder–decoder architecture with shallow modality-specific stems for images, proprioception, and robot-specific wrist cameras.A ResNet-18 processes images, an MLP encodes proprioception, and learned query attention tokenizes inputs into a shared space.
  • Architecture: A shared vision stem processes egocentric RGB observations from human and robot embodiments before tokens are concatenated and decoded into actions.Action decoders may be shared or embodiment-specific depending on the output action space.
  • Training Objective: The encoder and decoder are jointly optimized with a BC co-training loss computed on aggregated human and robot data.This objective supports cross-embodiment policy learning using combined demonstrations.
  • Training Objective: Conditional flow matching loss is computed per embodiment on mini-batches containing human and robot samples.The training procedure applies the loss separately for each embodiment e ∈ {robot, human}.

D. Evaluation Setup

Evaluation tests human-data co-training across representative manipulation tasks, robot embodiments, and in-domain or out-of-domain settings. The protocol uses randomized rollouts and normalized task-specific performance scores.

  • Evaluation Protocol: Evaluation covers four representative Flagship tasks in in-domain settings with matched layouts and out-of-domain settings with unseen objects and environments.Each method receives randomized initial conditions and task-specific subtask measurements.
  • Evaluation Protocol: Each method is evaluated with 20 in-domain and 20 out-of-domain rollouts per task.Reported normalized scores aggregate performance across rollouts using grasps, placements, intermediate manipulations, and full task completion.
  • Co-training Results: Co-training with EgoVerse-A improves in-domain and out-of-domain robot performance by up to 30%.Relative improvements are emphasized because demonstration speed and reset range differ across manipulation settings.
  • Co-training Results: EgoVerse-A co-training is robust across most embodiments and tasks, but Robot B declines on bag-grocery while Robots A and C improve.The authors hypothesize that Robot B’s embodiment limitations caused robot demonstrations to diverge from human strategies in EgoVerse-A.
  • Data Scaling: Scaling experiments keep robot data fixed while increasing in-domain and diverse human data to test whether transferable structure supports performance growth.The study finds that effective scaling depends critically on aligned human–robot data.

F. How Does Human Data Diversity Affect Generalization?

Controlled experiments separate scene and demonstrator diversity while evaluating generalization to unseen scenes and demonstrators. Scene coverage improves generalization, with diminishing returns from greater data density after moderate quantities.

  • Scene Diversity: Increasing scene diversity improves generalization to unseen scenes across data budgets, with stronger gains under limited data.After a moderate data quantity, expanding scene coverage provides measurable gains while increasing within-scene density yields diminishing returns.
  • Combined Diversity: Jointly increasing scene and demonstrator diversity produces complementary improvements under controlled data budgets.The experiments vary training scenes from 4 to 8 and demonstrators from 4 to 8 under a fixed 4-hour budget.
  • Combined Diversity: The marginal benefit of adding demonstrators decreases as scene coverage grows.Scene diversity improves generalization under both tested demonstrator budgets.
  • Limitations: The diversity experiments rely exclusively on offline metrics, so additional robot rollouts are needed to test whether the observed effects transfer to robotic manipulation.The offline metrics capture generalization across human demonstrators and environments but do not establish rollout performance.
  • Demonstrator Diversity: UMAP embeddings show greater overlap between training and validation demonstrators when demonstrator diversity increases from 4 to 12.The visualization concerns the multi-scene demonstrator-scaling setting.

VIII. APPENDIX

The appendix details EgoVerse’s task-diverse data composition and capture systems, combining controlled protocols with broad contributor and environment diversity.

  • Dataset design: EgoVerse-A uses controlled task semantics and data-quality protocols, while EgoVerse-I emphasizes broad demonstration-style diversity and rich annotations.
  • Scenario and object diversity: Each flagship task is collected across 8–12 scenes per lab, with randomized object positions and up to 30 sampled objects per task.
  • Demonstrator diversity: Data collection spans 1–8 demonstrators per lab, preserving variation in motion habits, timing, morphology, viewpoints, and workspace configurations.
  • Capture systems: EgoVerse combines Project Aria glasses, commodity smartphone capture, and customized stereoscopic rigs to support diverse human data sources.

1) Data Collection and Uploading:

EgoDB organizes uploaded episodes through metadata, cloud processing, SQL indexing, and a unified training interface for filtered dataset access.

  • SQL database: EgoDB stores episode metadata in a PostgreSQL table whose rows correspond to individual files and support filtering.
  • Processing pipeline: Uploaded files and metadata enter S3 through a unified script, after which automated processing updates database fields such as paths, errors, and frame counts.
  • Dataset access: EgoVerseDataset queries metadata filters, retrieves processed episodes from S3, and exposes them as training-ready PyTorch datasets.
  • Dataset construction: Episodes can be split into training and validation subsets or subsampled by percentage, with modes for train, valid, total, and percent.

G. Data Alignment and Post Processing

The pipeline temporally aligns human and robot actions, encodes multimodal observations with a cross-embodiment transformer, and decodes actions through flow matching.

  • Data alignment: Human actions use 1-second windows and robot actions 1.5-second windows, with both resampled to sequences of length T = 100.
  • Data alignment: Linear interpolation handles 3D positions, while SLERP handles quaternion and Euler-angle rotations during action resampling.
  • Observation encoding: RGB and proprioceptive inputs become query tokens that are concatenated with learnable context tokens for a non-masked cross-embodiment transformer encoder.
  • Action decoding: The decoder conditions a diffusion transformer on encoder context tokens and predicts the action sequence through iterative flow-matching integration.
  • Inference: 10 fixed-step Euler updates are used during evaluation to produce the final action sequence.

3) Co-training with Flow Matching.:

The co-training setup combines human and robot demonstrations using flow matching and evaluates transfer across tasks, scenes, objects, and robot platforms.

  • Flow matching: For each embodiment, training samples a Beta(1.5, 1.0) timestep and minimizes error in the predicted vector field along a noise-to-action path.
  • Training mixtures: The flagship mixture combines 8 hours of EgoVerse-A, 2 hours of in-domain human data, and task-matched robot demonstrations.
  • Scaling experiments: Scaling experiments compare EgoVerse-A and in-domain human-data mixtures against robot-only training under a fixed 1:1 human-to-robot batch ratio.
  • Evaluation protocol: Evaluation varies workspace positions, object combinations, scenes, planar rotations, and in-domain versus out-of-domain settings across multiple tasks.
  • Observed limitation: Robot B shows a human–robot strategy mismatch for bag-grocery, potentially weakening cross-embodiment alignment during co-training.

3) Additional Analysis:

The additional analysis identifies persistent manipulation difficulties in specific workspace regions and during handover, despite more robust grasping in co-trained policies.

  • Picking primitives remain difficult in certain workspace regions for object-in-container and bag-grocery.
  • Co-trained policies generally exhibit more robust grasping primitives, but improvement remains necessary.
  • Object handover is difficult for cup-on-saucer, especially for Robot C with a dexterous hand.

4) Task Failure Modes:

Controlled diversity experiments show that scene diversity consistently improves generalization, while demonstrator diversity provides task-dependent benefits and scaling gains under fixed data budgets.

  • Single-scene Demonstrator Scaling: Increasing demonstrators improves fixed-budget generalization to unseen demonstrators in single-scene experiments.For fold-clothes, Avg-MSE decreases monotonically; cup-on-saucer improves at larger demonstrator counts after slight low-count non-monotonicity.
  • Multi-scene Demonstrator Scaling: Increasing demonstrators from 4 to 12 consistently improves generalization across eight scenes and both tasks.The evaluation uses unseen demonstrators within the same scenes under a fixed 8-hour training budget.
  • Scene Diversity Scaling: Increasing scene count consistently reduces Avg-MSE on unseen scenes, with task-specific dependence on per-scene data quantity.Fold-clothes benefits especially under low budgets, whereas cup-on-saucer benefits from both sufficient per-scene data and broader scene coverage.
  • Mixed Diversity Scaling: Under a fixed 4-hour budget, scene diversity consistently improves generalization, while demonstrator diversity varies by task.Eight demonstrators outperform four for fold-clothes, but four slightly outperform eight for cup-on-saucer.
  • Visualization: The controlled-diversity visualizations use UMAP to reduce representations to two dimensions.EgoVerse-I uses DinoV3 image embeddings, while demonstrator-diversity visualization uses HPT action-conditioned tokens.
Loading 2604.07607v2…