PaperlayerView exact revision on arXiv

Source-linked AI summary

Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

Ye Wang, Pei Lin, Xiong-Hui Chen, Haoqi Yuan, Zhixuan Liang, Yiyang Huang, Anzhe Chen, Zixing Lei, Jie Zhang, Tao Zhang, Haoyang Li, Tong Zhang, Chenxi Xiao, Ziyuan Jiao, Qin Jin

arXiv:2608.02580v1cs.RO

TL;DR

Robot policy generalization is limited by the scale and diversity of robot demonstrations, while the pretraining value of large-scale ego-to-robot synthesis remains unclear. Ego2Robot converts egocentric manipulation videos into robot data and, when combined with robot data, improves out-of-distribution generalization across perturbation types, including real-robot deployment.

  • Problem

    The pretraining value of large-scale egocentric-video-to-robot synthesis for generalizable robot policies remains unclear amid costly and limited robot data collection.

  • Method

    Ego2Robot converts egocentric manipulation videos into embodiment-specific robot training data through action alignment, visual alignment, and multi-level quality curation.

  • Results

    53.5% on RoboTwin Randomized (+2.6 over robot-only) is achieved by 1:1 Ego2R+Robot co-training, which improves out-of-distribution generalization across perturbation types.

  • Takeaways & Limitations

    Ego2Robot-synthesized data complements robot data, with gains most pronounced under visual, embodiment, and semantic perturbations and benefits observed in real-robot deployment.

  • Takeaways & Limitations

    The evaluation is limited to the task scope of RoboTwin2.0, leaving broader task and embodiment generality untested.

Abstract

from arXiv · show

Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored. We present \textbf{Ego2Robot}, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment. Project page: https://www-ye.github.io/ego2robot_blog/

1 Introduction

Ego2Robot addresses the limited scale and diversity of robot demonstrations by converting egocentric human manipulation videos into embodiment-specific robot data. It evaluates whether combining synthesized and robot data improves generalization across disentangled distribution shifts.

  • Robot manipulation generalization remains constrained because collecting large, diverse robot demonstrations is expensive, labor-intensive, and hardware-limited.
  • Egocentric human videos provide manipulation data at massive scale across diverse objects, environments, and task variations, but human–robot embodiment differences complicate direct conversion.
  • Ego2Robot converts egocentric manipulation data into embodiment-specific robot training data through action alignment, visual alignment, and multi-level quality curation.The pipeline supports both annotated egocentric datasets and pure egocentric videos.
  • The evaluation protocol separates distribution shifts into visual appearance, scene layout, embodiment morphology, and task semantics to analyze combined robot and synthesized-data pretraining.
  • 18,561 hours of synthesized robot training data span 15 robot morphologies, making Ego2Robot the largest ego-to-robot dataset to date.

2 Related Work

Prior work scales robot data, learns from human videos, and augments robot learning, but collection cost, embodiment gaps, limited ego-to-robot scale, and incomplete generalization benchmarks remain challenges.

  • Robot Data Scaling and Learning from Human Data: Large-scale robot datasets and portable collection systems expand demonstrations, yet diversity remains constrained by collection cost and hardware scalability.Human manipulation videos offer broader diversity and motivate visual pretraining, reward learning, motion priors, and point tracking, but these methods still rely on robot data to bridge embodiment gaps.
  • Robot Data Scaling and Learning from Human Data: Co-training human videos with robot data and converting human demonstrations through retargeting, rendering, or embodiment masking provide direct routes to robot policy learning.Retarget-and-render methods have only been studied at limited scale or for individual tasks.
  • Data Augmentation for Robot Learning: Simulation-based and cross-embodiment augmentation methods increase diversity or bridge robot-to-robot visual gaps, whereas this work targets the larger ego-to-robot domain gap.Prior methods include digital-twin demonstrations, inpainting, and image editing.
  • Benchmarks for Robot Generalization: Existing manipulation benchmarks bundle perturbations, complicating failure attribution, while multi-configuration platforms lack standardized cross-embodiment evaluation.Decomposed benchmarks have recently been introduced to address these limitations.

3 Ego2Robot Pipeline

Ego2Robot converts egocentric human manipulation videos into embodiment-specific robot training data through action alignment, visual alignment, and quality curation. The pipeline retargets hand motion into robot trajectories and replaces human arms with rendered robot arms using geometric and depth-aware processing.

  • Pipeline overview: The pipeline produces embodiment-specific robot training data through action alignment, visual alignment, and quality curation.These are the three stages described for processing egocentric videos of human hand manipulation.
  • Action alignment: Action alignment converts human hand poses into robot end-effector trajectories through retargeting and temporal smoothing.This stage aligns the demonstrated human motion with the target robot embodiment.
  • Visual alignment: Visual alignment replaces human arms with rendered robot arms using arm segmentation, hand removal, robot base pose search with IK solving, and depth-aware rendering.The process combines human-arm removal with inverse-kinematics-based robot placement and depth-aware synthesis.

1. Action Alignment

The action alignment stage retargets detected human hand poses into parallel-gripper end-effector trajectories, then smooths and speed-aligns them for robot training. It constructs a consistent gripper frame from hand geometry and adjusts source video frame rates to match robot motion speeds.

  • Action Alignment: Hand poses are converted into parallel-gripper end-effector trajectories through retargeting and temporal smoothing.The pipeline operates on detected hand poses and produces robot-compatible trajectories.
  • Hand-to-Gripper Retargeting: A compact gripper representation is extracted from 21 hand keypoints, including a virtual fingertip formed from the index and middle fingertips.The virtual fingertip is defined as a weighted blend of those two fingertip locations.
  • Hand-to-Gripper Retargeting: The grasp orientation uses a right-handed orthonormal frame whose axes encode approach direction, gripper normal, and grasp axis.The grasp axis follows the jaw line, while the remaining axes are derived from the wrist-to-fingertip direction and cross products.
  • Temporal Smoothing: Savitzky-Golay filtering smooths positions and widths, while Gaussian-weighted SLERP smooths orientations without discarding motion structure.These filters address high-frequency noise from per-frame hand detection.
  • Action Speed Alignment: ANT and EgoDex are downsampled to 60%, EgoVerse to 45%, and ViTRA to 25% of their original frame rates to align action speeds.These settings correspond to approximately 1.7×, 2.2×, and 4× slower motion, respectively.

2. Visual Alignment

Visual alignment converts egocentric videos from human-hand demonstrations into robot-arm scenes through segmentation, hand removal, base-pose optimization, inverse-kinematics rendering, and depth-aware compositing. The pipeline generates parallel data streams across 15 robot morphologies and applies three-level quality filtering.

  • Visual alignment transforms ego videos depicting human hands into scenes showing a robot arm operating in the same environment.
  • SAM 3 segments temporally consistent human-arm masks, and ProPainter removes the arms while reconstructing the background through temporally consistent inpainting.
  • Robot base placement is optimized by searching morphology-constrained candidates around the trajectory centroid and selecting the placement with the highest keyframe IK feasibility.Keyframes cover spatial extremes, and each candidate is validated by solving inverse kinematics in MuJoCo.
  • Given the optimized base pose, frame-by-frame MuJoCo IK produces robot configurations rendered from the original camera viewpoint and composited into the inpainted scene using depth ordering.Depth ordering selects the robot render where its depth is nearer than the scene depth and the robot mask is active.
  • 15 robot morphologies receive independent processing, generating parallel training streams from each egocentric video.The listed morphologies include Panda, UR5e, ARX-L5, xArm7, Sawyer, Kinova Gen3, IIWA, Jaco, FR3, UR10e, ViperX, WidowX, Piper, YAM, and Aloha-Agilex.
  • Three-level quality filtering removes failures and statistical outliers, then uses a vision-language model to audit semantic consistency between rendered actions and original manipulation intents.L1 flags IK failures, self-collisions, action outliers, and insufficient workspace coverage; L2 removes discontinuous or invalid trajectories; L3 performs VLM consistency auditing.

3. Quality Curation

Quality curation processes egocentric data from four sources, adjusts sampling to match robot motion speed, and produces 18,561 hours of synthetic robot training data across 15 robot morphologies. Each sample combines robot-composited visual data with camera-frame action, camera parameters, and a text instruction.

  • Data Sources: ∼1,940 hours of annotated ego data come from ANT, EgoDex, ViTRA, and EgoVerse.The sources contribute 7h, 732h, 249h, and 954h, respectively.
  • Speed Matching: Frame subsampling slows egocentric videos to match the robot speed distribution, using 60% for ANT and EgoDex and 45% for EgoVerse.ANT and EgoDex are ∼1.7× slower, while EgoVerse is ∼2.2× slower after subsampling.
  • Synthetic Data Scale: 18,561 hours of synthetic robot training data result after processing across 15 robot morphologies and quality curation.The dataset is produced from the curated egocentric sources through the pipeline.
  • Sample Representation: Each sample contains a robot-composited video frame, camera-frame EEF action, camera parameters, and a text instruction.Camera-frame relative EEF actions avoid requiring per-video calibration for diverse, unknown egocentric camera placements.

4 Evaluation Framework

The evaluation framework addresses the limitations of aggregate OOD testing by disentangling generalization factors and adding egocentric-perspective task coverage. It extends RoboTwin 2.0 with independent perturbation settings spanning visual appearance and scene layout, alongside EBench precision tabletop tasks.

  • Motivation: Existing RoboTwin 2.0 protocols conflate visual, spatial, embodiment, and semantic shifts into one aggregate OOD metric.This limits analysis of which dimensions benefit from Ego2Robot-synthesized data and which remain constrained by domain gaps.
  • Benchmark Extension: The framework adds 11 independent perturbation settings and camera-frame relative EEF support to RoboTwin 2.0.It also complements RoboTwin 2.0 with EBench, which contains 7 precision tabletop tasks in Isaac Sim using a higher-mounted head camera closer to the egocentric perspective.
  • Perturbation Dimensions: Visual-appearance tests vary background texture, lighting, and robot color, while scene-layout tests vary table height, distractors, and camera viewpoint.Table height varies by ±4cm and camera viewpoint offset by ±5cm; these factors are decoupled for independent evaluation, and robot color applies random hue shifts to all robot links.

5 Experiments

Experiments show that Ego2Robot co-training improves out-of-distribution generalization across visual, scene, embodiment, and task-semantic perturbations, with gains also transferring to real-robot deployment. Pipeline alignment and multi-morphology scaling are important contributors, while casually recorded egocentric videos provide additional effective training signal.

  • Co-training results: Ego2R+Robot (1:1) leads in five of seven columns, reaching 53.5% on RoboTwin Randomized (+2.6 over robot-only) while maintaining 68.1% on Clean.All configurations process ∼19.2M frames under identical pretraining steps and batch sizes, enabling fair comparison.
  • Perturbation generalization: At a 1:1 ratio, Ego2Robot improves all three visual factors: background (+4), lighting (+8), and robot color (+6).The gains are attributed to scene diversity in egocentric videos and multi-morphology rendering across 15 configurations.
  • Perturbation generalization: Co-training improves scene, embodiment, and task-semantic robustness, including camera offset (+6) at 1:1, ARX from 44 to 51, and unseen-object generalization from 29% to 40% (+11) at 3:1.UR5 peaks at 31 (3:1), paraphrased-instruction robustness reaches 69% at 1:1, and Franka remains below 7%.
  • Viewpoint transfer: EBench reaches 51.7 (+12.1 over robot-only) with the 3:1 ratio, indicating stronger co-training benefits when its higher-mounted viewpoint is closer to the egocentric perspective.EBench uses a separate training-set finetuning procedure.
  • Real-robot deployment: Mix + Ego2R Play achieves the best results across all five ARX ACone tasks, with the largest gains on Put Blocks (+14 over Robot-only) and Insert Screw (+13).The evaluation uses 20 teleoperated demonstrations per task plus approximately 7 minutes of egocentric play videos per scene processed into ACone-specific demonstrations.

6 Limitations

Ego2Robot’s current limitations are restricted hand dexterity and potentially imperfect visual synthesis. Future work could support multi-finger hands and improve rendering fidelity under difficult visual conditions.

  • Limitations: Retargeting maps hand poses to parallel-jaw grippers, discarding fine-grained finger articulation and limiting transferable skills.Extending the pipeline to dexterous multi-finger hands could broaden the range of transferable skills.
  • Limitations: Inpainting and depth-aware compositing may introduce visual artifacts under heavy occlusion or complex lighting.Generative models could improve rendering fidelity and reduce these artifacts.

7 Conclusion … D.1 Pretraining.

Ego2Robot presents a scalable pipeline for converting egocentric manipulation videos into robot training data across 15 morphologies, with synthesized data complementing robot data for out-of-distribution generalization. The paper details retargeting, visual synthesis, quality curation, disentangled evaluation, model design, and pretraining procedures, including real-robot multi-task deployment.

  • 7 Conclusion: Ego2Robot converts egocentric hand-manipulation videos into robot training data across 15 robot morphologies.Its synthesized data complements robot data, with the largest gains under visual, embodiment, and semantic perturbations; combining both enables effective multi-task deployment on real robots.
  • A.1 Hand Pose Estimation (Path B).: Hand pose estimation reconstructs per-frame MANO parameters and keypoints, associates detections into temporally consistent tracks, and refines pose and shape across full sequences.Detections are filtered using projected-keypoint masks, while temporal optimization enforces data consistency, smoothness, biomechanical limits, and a 0.05–0.4 m hand-depth constraint.
  • A.2 Subtask Segmentation (Path B).: Subtask segmentation splits long recordings at 60-second boundaries and uses Qwen3.5 to produce complete-goal boundaries and 5–12-word training instructions.The VLM outputs a JSON array containing each subtask’s description, start time, and end time.
  • A.3 Action Alignment Details.: Action alignment standardizes handedness and gripper axes, handles degenerate orientations, filters velocity outliers, and smooths position, gripper width, and orientation trajectories.The procedure uses nearest-neighbor interpolation for detected jumps, two filtering rounds, Savitzky–Golay smoothing, and Gaussian-weighted SLERP with σ=10 frames.
  • A.5 Quality Curation Details.: Quality curation combines pipeline validity checks, statistical outlier filters, sudden-change detection, and VLM semantic-consistency auditing before retaining synthesized episodes.Invalid conditions include excessive IK error, invisible rendering, self-collision, cross-arm contacts, and excessive robot-mask area; inconsistent episodes are discarded.
  • B. Evaluation Framework Details: Evaluation disentangles visual appearance, scene layout, embodiment, and task semantics, including randomized backgrounds, lighting, robot color, table height, clutter, camera offset, paraphrased instructions, and unseen objects.The protocol uses 50 episodes per task with binary success and aggregates Visual, Scene, Embodiment, and Task scores; unseen-object evaluation covers 50 new tasks.

D.2 Pretraining Data. … F.2 Per-Task Results.

Ego2Robot combines large-scale robot and egocentric data for pretraining, with ablations and real-robot experiments evaluating morphology scaling, task transfer, and deployment. Additional results compare Ego2R+Robot against Pi0.5 and report per-task RoboTwin success rates.

  • D.2 Pretraining Data.: 18,561 h of Ego2R data is produced from ∼1,940 h of egocentric sources, rendered across 15 morphologies and mixed with ∼6,565 h of robot data.Robot data comes from DROID, AgibotWorld, and InternData; Ego2R sources are ANT, EgoDex, ViTRA, and EgoVerse.
  • D.2 Pretraining Data.: Pretraining uses Ego2R-to-Robot mixing ratios of 1:3, 3:1, and 1:1, implemented through per-source sampling weights while processing equal total frame counts.Figure 7 presents the sampling weights within each data group.
  • D.3 Finetuning.: Finetuning evaluates RoboTwin Clean, EBench Table Top, and real-robot settings using 50K steps, with 1:1 teleoperation-to-Ego2R mixing for real-robot training.RoboTwin uses 50 tasks and 2,500 episodes; EBench uses 7 tasks and 2,800 episodes.
  • D.4 Ablation Configurations.: The ablation isolates pipeline effects by pretraining only on ego-sourced data, comparing raw ego with Ego2R rendered using 1, 5, 10, or 15 morphologies.The largest configuration contains ∼18,561 h, while adding raw ego reaches ∼20,501 h; all models are finetuned on RoboTwin Clean.
  • E Real Robot Experiment Details; E.1 Platform and Tasks.: The real-robot study uses an ARX ACone dual-arm platform across five manipulation tasks: Put Fruits, Put Blocks, Fold Towel, Sweep Trash, and Insert Screw.The platform has 6-DOF arms, a parallel gripper, head and wrist RGB cameras, and 15 Hz control.
  • E.2 Data.: 100 teleoperation demonstrations and ∼35 minutes of ego play across five scenes yield 675 ACone synthetic episodes, with teleoperation and Ego2R mixed 1:1 during finetuning.Ego play is processed through Path B using WiLoR, DynHaMR, Qwen3.5, and Ego2R.
  • E.3 Ego Play to Ego2R Visualization.: Ego2R visualization overlays the synthesized ACone robot onto inpainted real-world scenes, contrasting original egocentric manipulation with the pipeline output.Figure 8 shows the original human manipulation in the top row and the robot-overlay synthesis in the bottom row.
  • E.4 Scoring; F.1 Comparison with Pi0.5; F.2 Per-Task Results.: 100-point, 20-trial sub-step scoring supports real-robot evaluation, while Ego2R+Robot (1:1) outperforms the Robot-only baseline across nearly all RoboTwin settings and Table 5 reports all 50-task success rates.Pi0.5 is evaluated under the same RoboTwin EEF settings, and Table 5 covers Clean and Randomized conditions for five models.
Loading 2608.02580v1…