PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoMimic: Scaling Imitation Learning via Egocentric Video

Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, Danfei Xu

arXiv:2410.24221v1cs.ROcs.CV

TL;DR

Imitation learning needs much larger and more diverse demonstrations, but robotics lacks an Internet-scale data source and prior approaches often handle human videos separately from robot data. EgoMimic captures egocentric human embodiment data and teleoperated robot data, aligns their domains, and co-trains a unified policy. It improves performance across challenging real-world tasks, generalizes to new scenes, and gains more from an additional hour of human hand data than from an additional hour of robot data.

  • Problem

    Robotics lacks an Internet-scale equivalent of the data sources that support scaling in computer vision and natural language processing, while existing policies remain brittle in new scenarios and tasks.

  • Method

    EgoMimic combines Project Aria human capture, a low-cost bimanual robot, cross-domain alignment, and a unified architecture that co-trains on human and robot data.

  • Results

    EgoMimic significantly improves performance across three long-horizon real-world tasks, generalizes to objects and scenes present only in human data, and shows favorable scaling with human data.

  • Takeaways & Limitations

    An additional hour of hand data is significantly more valuable for EgoMimic than an additional hour of robot data, supporting passive human embodiment data as a scalable resource.

Abstract

from arXiv · show

The scale and diversity of demonstration data required for imitation learning is a significant challenge. We present EgoMimic, a full-stack framework which scales manipulation via human embodiment data, specifically egocentric human videos paired with 3D hand tracking. EgoMimic achieves this through: (1) a system to capture human embodiment data using the ergonomic Project Aria glasses, (2) a low-cost bimanual manipulator that minimizes the kinematic gap to human data, (3) cross-domain data alignment techniques, and (4) an imitation learning architecture that co-trains on human and robot data. Compared to prior works that only extract high-level intent from human videos, our approach treats human and robot data equally as embodied demonstration data and learns a unified policy from both data sources. EgoMimic achieves significant improvement on a diverse set of long-horizon, single-arm and bimanual manipulation tasks over state-of-the-art imitation learning methods and enables generalization to entirely new scenes. Finally, we show a favorable scaling trend for EgoMimic, where adding 1 hour of additional hand data is significantly more valuable than 1 hour of additional robot data. Videos and additional information can be found at https://egomimic.github.io/

I. INTRODUCTION

EgoMimic addresses the scarcity and brittleness of robot demonstration data by treating egocentric human data as an equal embodied source and co-training a unified manipulation policy. Its full-stack design combines scalable capture, hardware alignment, cross-domain processing, and joint learning, improving performance across real-world tasks.

  • Robotics lacks an Internet-scale equivalent to the data available in computer vision and natural language processing, limiting scalable imitation learning.Existing end-to-end policies remain brittle in new scenarios and tasks.
  • Egocentric human videos provide scalable embodied demonstrations with visual observations, 3D hand-tracked actions, and onboard localization.Wearable XR devices and smart glasses enable passive data collection at scale.
  • EgoMimic treats human data as a first-class source and learns jointly from human egocentric videos and teleoperated robot data.The framework treats both sources as equal parts of a continuous spectrum of embodied data.
  • Project Aria glasses capture egocentric video, 3D hand tracking, and device SLAM, while a low-cost bimanual robot reduces kinematic and camera-to-camera gaps.The robot uses Project Aria glasses as its main sensor to align human and robot observations.
  • EgoMimic normalizes action distributions, masks visual differences, and co-trains a shared encoder and policy despite distinct human and robot action spaces.The shared representation is intended to support scaling with human embodiment data.
  • Up to 200% relative improvement was observed across continuous object-in-bowl, clothes folding, and grocery packing, with generalization to objects and scenes found only in human data.An additional hour of hand data significantly outperformed an additional hour of robot data.

II. RELATED WORKS

Prior work scales robot data through teleoperation, human-video processing, and cross-embodiment transfer, but these approaches retain hardware, labor, or abstraction gaps. EgoMimic positions human data as another embodied source for robot policy learning.

  • Imitation Learning: Pixel-to-action imitation learning maps visual inputs directly to low-level robot control and has shown generalization when trained at scale.However, such scaling can remain labor- and resource-intensive.
  • Imitation Learning: RT1 required 17 months of data collection and 13 robots, motivating scalable human embodiment demonstrations as a potentially larger and more diverse source.
  • Learning from Video Demonstrations: Human videos have been used for visual pretraining, scene-dynamics prediction, affordance prediction, and other intermediate representations for imitation learning.
  • Data Collection Systems: Space Mouse, virtual reality, ALOHA, and GELLO provide increasingly intuitive or ergonomic systems for collecting teleoperated robot demonstrations.
  • Cross-embodiment Policy Learning: Cross-embodiment methods bridge robot differences through observation reprojection, action abstractions, embodiment conditioning, or domain adaptation.EgoMimic extends this framing by treating human data as another embodiment.
  • EgoMimic’s Position: EgoMimic’s hardware uses Aria glasses for human capture and the robot’s main vision sensor, aligning egocentric RGB, hand tracking, and robot data.

III. EGOMIMIC

EgoMimic is a full-stack pipeline designed to make human embodiment data suitable for robot learning alongside teleoperated robot data. It spans collection hardware, cross-source data processing, and a unified policy architecture.

  • EgoMimic captures and learns from both egocentric human embodiment data and robot data.The pipeline covers hardware setup, data processing and alignment, and unified policy learning.

A. Data Collection Systems and Hardware Design

The data-collection system combines wearable Aria glasses with a lightweight bimanual manipulator to capture human-compatible observations and actions. Hardware choices reduce visual, kinematic, and camera-domain differences between human and robot demonstrations.

  • Human Data Collection: Project Aria glasses provide wearable, passive capture of wide-FOV RGB observations, hand positions, and device pose.The glasses weigh 75g, and side-facing cameras track hands outside the main RGB camera’s view.
  • Human Data Collection: Large-scale Project Aria collection efforts and broad academic availability support future merging of passively collected human data with these datasets.
  • Robot Hardware: The purpose-built bimanual manipulator is lightweight, agile, and cost-effective to better match human movement than heavy table-mounted systems.The design responds to kinematic differences and the slow, cautious movements induced by safety concerns in heavier systems.
  • Camera Alignment: The robot mounts a second pair of Aria glasses near human-eye height as its main sensor, reducing differences in field of view, exposure, and dynamic range.

B. Data Processing and Domain Alignment

EgoMimic aligns human and robot demonstrations by unifying coordinate frames, normalizing pose distributions, and reducing visual appearance differences before joint policy learning.

  • EgoMimic bridges three human-robot gaps: action coordinate frames, action distributions, and visual appearance.These alignment steps prepare both embodiments for unified imitation learning.
  • Human and robot trajectories are transformed into camera-centered reference frames so action supervision uses a unified coordinate system.Human camera motion is handled using Aria pose estimates from visual-inertial SLAM, while robot data uses a calibrated fixed camera frame.
  • Gaussian normalization is applied separately to human and robot proprioception and action data to align their pose distributions.The distributions differ particularly along the y (left-right) dimension.
  • EgoMimic masks hands and robot arms with SAM and overlays a red end-effector direction line to reduce visual appearance gaps.The visual treatment is applied after hardware and data-processing alignment.

C. Training Human-Robot Joint Policies

EgoMimic replaces hierarchical human-to-robot transfer with a unified policy that shares representations across human and robot data while retaining robot-specific action prediction.

  • EgoMimic uses a shared policy architecture so human demonstrations can directly contribute to the learned representation rather than only conditioning a low-level robot policy.The architecture builds on ACT and is intended to generalize to other transformer-based imitation learning algorithms.
  • The training algorithm samples human and robot datasets separately, predicts pose actions for both, and predicts joint actions for robot data.A shared transformer encoder and separate pose and joint decoders support the two action spaces.
  • EgoMimic uses robot joint-space control while retaining pose-space prediction because the ViperX arms can encounter singularities and nonsmooth solutions under Cartesian control.More capable robots may eliminate the need for joint-space action prediction.
  • All policy parameters are shared except shallow embodiment-specific input and output heads, encouraging a joint human-robot representation.Pose loss supervises both domains, whereas joint-action loss supervises robot data only.

IV. EXPERIMENTS

The experiments test whether EgoMimic improves complex manipulation, generalizes to new objects and scenes, and benefits more from additional human than robot data.

  • EgoMimic is evaluated on long-horizon real-world tasks requiring precise alignment, complex motions, and bimanual coordination.The experiments are designed around three hypotheses concerning in-domain performance, scene generalization, and data-scaling value.

A. Experiment Setup

EgoMimic is evaluated on long-horizon real-world tasks requiring precise manipulation, complex motions, spatial variation, and bimanual coordination. The experiments compare performance across three tasks using success rates, task scores, and bag-grabbing rate.

  • Tasks: The evaluation covers continuous object-in-bowl, laundry folding, and grocery packing tasks in real-world settings.These tasks stress long-horizon execution, precise alignment, complex motions, and bimanual coordination.
  • Tasks: Continuous Object-in-Bowl requires repeated toy placement and bowl emptying across randomized bowls, toys, and positions.The task uses 45 rollouts across 9 bowl-toy-position combinations.
  • Tasks: Laundry requires folding randomly positioned shirts through right-sleeve, left-sleeve, and whole-shirt stages using both arms.Success Rate is the percentage of runs completing every stage; evaluation includes 40 rollouts across 8 shirt-position combinations.
  • Tasks: Groceries requires opening a deformable bag handle with one arm and placing three chip packs inside with the other.The evaluation reports task success rate and Open Bag rate across 50 evaluations and 10 bag positions.
  • Baselines: EgoMimic is benchmarked against ACT and Mimicplay to assess whether human data improves in-domain success rates.Robot positions are randomly perturbed during data collection to improve robustness.

B. Results

EgoMimic improves performance across the evaluated tasks, generalizes to unseen shirt colors and scenes, and benefits strongly from additional human data. Ablations further show that normalization, visual alignment, and hand-robot co-training each contribute to performance.

  • In-domain performance: 34-228% relative score improvement and 8-33% absolute success-rate improvement over ACT were observed across tasks.The largest gain was a 228% task-score improvement on Continuous Object-in-Bowl.
  • Policy generalization: EgoMimic retains 85% Success Rate on shirts with unseen colors, compared with ACT’s 25% Success Rate.This evaluates generalization beyond the shirt colors seen during training.
  • Policy generalization: In an entirely new Object-in-Bowl scene, EgoMimic scores 63 points without additional robot data, versus 4 points for Mimicplay.The shift includes unseen background and lighting.
  • Scaling human vs. robot data: One hour of human data yields 1400 demonstrations, compared with 135 demonstrations from one hour of robot data.EgoMimic trained on 2 hours of robot data plus 1 hour of hand data strongly outperforms ACT trained on 3 hours of robot data.
  • Ablations: Removing action normalization, visual masking, or the red overlay caused task-score drops of 38%, 13%, and 26%, respectively.Removing hand data caused a 47% drop on Object-in-Bowl.

V. CONCLUSIONS

EgoMimic co-trains manipulation policies from human egocentric and teleoperated robot data using cross-domain alignment and a unified architecture. Its alignment pipeline normalizes embodiment-specific signals and visually aligns hands and robot arms through masking and overlays.

  • Conclusions: EgoMimic co-trains manipulation policies from human egocentric videos and teleoperated robot data.The framework uses Project Aria glasses, a low-cost bimanual robot setup, alignment techniques, and unified policy learning.
  • Data Processing and Domain Alignment: Human and robot data are temporally aligned by slowing human data by a factor of 4 and using different action horizons.Robot actions use a four-second horizon, human actions a one-second horizon, with action chunks of 100.
  • Data Processing and Domain Alignment: Proprioception and actions are normalized separately for each embodiment using dataset means and standard deviations.The same normalization procedure is applied to action sequences.
  • Data Processing and Domain Alignment: Visual alignment masks human hands and robot arms and adds red-line overlays to their RGB representations.Robot masks are generated from projected forward-kinematics keypoints, while human masks use tracked hand coordinates.
  • Data Collection: Aria sensor data is processed by MPS to estimate device pose, environmental point clouds, hand tracking, and eye gaze.The raw stream includes RGB, SLAM, IMU, eye-tracking, and microphone data.

C. Training Human-Robot Joint Policies

EgoMimic co-trains a unified policy on human hand data and robot data through shared normalization, visual encoding, and modality-aware transformer inputs. Its objective combines robot and hand losses, while grasping remains supervised only from robot data.

  • EgoMimic Training: Human and robot batches are sampled separately and passed through a unified architecture for joint training.The method applies separate Z-score normalization to proprioception and actions from each modality.
  • EgoMimic Training: Shared visual processing converts masked top-down views and robot wrist views into visual tokens alongside proprioception tokens.Top-down images pass through a SAM-based masking module, followed by a shared ResNet-18 visual encoder.
  • Training Objective: The joint objective is L = Lrobot + Lhand, with the hand loss combining an L1 action term and a CVAE latent KL regularizer.The same KL regularizer is used as in ACT.
  • EgoMimic Training: A flexible transformer input sequence accommodates modality-specific observations, adding wrist-image tokens for robot data but not hand data.The authors report that this strategy was sufficient for co-training both data sources.
  • Data Limitations: Human hand pose lacks grasping information, so the gripper action is supervised only through the robot joint-prediction loss.The gripper is represented as another robot joint.
Loading 2410.24221v1…