Source-linked AI summary
HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
Zhi Wang, Botao He, Kelin Yu, Seungjae Lee, Ruohan Gao, Furong Huang, Yiannis Aloimonos
TL;DR
Robot learning from human egocentric video must overcome differences in visual appearance and kinematics while learning from minutes of data. HumanEgo addresses this with an entity-level hand–object representation, flow matching, and dense auxiliary objectives, achieving strong zero-shot transfer without robot data. With 30 minutes per task, it reaches 92.5% average success across four real-world tasks and also transfers across novel robots, cameras, and environments.
Problem
Human-to-robot transfer from egocentric video is limited by visual and kinematic embodiment gaps and by sparse, multimodal supervision in minutes-scale demonstrations.
Method
HumanEgo uses embodiment- and viewpoint-invariant Interaction-Centric Tokens with flow matching and dense object-motion, 2D-trace, and latent-consistency objectives.
Results
92.5% average success across 4 real-world tasks is achieved with 30 minutes of human data per task, with zero-shot transfer to novel robots, cameras, and environments.
Takeaways & Limitations
Human egocentric video can serve as a robot-data-free, hardware-agnostic source for zero-shot robot policy learning.
Takeaways & Limitations
The framework relies on Aria stereo hand tracking, while monocular substitutes sharply reduce real-world success.
Abstract
from arXiv · showhide
Human egocentric video captures rich manipulation demonstrations without any robot hardware, yet transferring these skills to robots remains challenging due to the embodiment gap between human and robot in both visual appearance and kinematics. We present HumanEgo, a framework that bridges the embodiment gap by lifting each human demonstration to an entity-level representation of hand-object interaction, and training a flow matching policy with dense auxiliary objectives that amplify supervision from every trajectory. HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable. With only 30 minutes of human videos per task, HumanEgo achieves 92.5% average success across four real-world tasks (75% with just 15 minutes), outperforms matched-time robot teleoperation by 41%, and robustly transfers zero-shot across novel robots, cameras, and environments. We release HumanEgo as an easy-to-use, open-source framework for learning robot policies directly from human data: https://github.com/TX-Leo/HumanEgo
1 Introduction
HumanEgo targets the embodiment and minimal-data challenges of learning robot manipulation from human egocentric videos. It combines interaction-centric representations, flow matching, and dense auxiliary supervision to enable zero-shot transfer without robot demonstrations.
- Robot manipulation policies typically require hundreds to thousands of task-specific robot demonstrations, whereas human egocentric videos can be collected cheaply in minutes.
- Human and robot differences in visual appearance and kinematics make direct transfer difficult, motivating representations centered on task-relevant hand–object interaction geometry.
- Minimal-data learning is challenged by multimodal strategies and sparse action signals, because each trajectory contains object motion, visual traces, and hand–object state beyond the hand action.
- HumanEgo uses arm inpainting, virtual grippers, object keypoints, Interaction-Centric Tokens, flow matching, and three dense auxiliary objectives to address these gaps.The auxiliary objectives are 2D trace, object motion, and latent consistency.
- 92.5% average success across 4 real-world tasks follows from 30 minutes of human video per task, while 75% is reached at half that budget.The framework is robot-data-free, hardware-agnostic, and designed for zero-shot human-to-robot transfer.
2 Related Work
Prior approaches either rely on large robot datasets, massive pretraining, paired human–robot data, or robot-trained low-level controllers. HumanEgo instead pursues embodiment-agnostic representations for zero-shot transfer while retaining rich interaction information.
- Large-scale egocentric-policy approaches require enormous compute and per-task robot post-training, while co-training methods depend on paired human and robot data.
- HumanEgo’s system overview combines visual preprocessing, Interaction-Centric Tokens, flow matching, and dense auxiliary objectives to learn bimanual actions from human data.
- Hierarchical methods learn high-level plans from human video but still require robot data for low-level control.
3 HumanEgo
HumanEgo converts egocentric demonstrations into embodiment-agnostic visual and spatial observations, then learns bimanual actions with flow matching and dense auxiliary objectives. Its representation captures entity-level hand–object relations while its shared encoder forecasts scene evolution across complementary spaces.
- HumanEgo records human demonstrations, removes the human arm, renders a virtual gripper and keypoints, encodes entity relations, and generates bimanual actions.
- Each object and hand is treated as an entity whose 6-DoF pose and relative relations are encoded into Interaction-Centric Tokens.
- Hand tracking smooths Aria 3D keypoints and models the thumb–index pair as a virtual parallel-jaw gripper with an end-effector pose and grasp state.
- Object pose estimation combines text-prompted detection, segmentation, contour-keypoint tracking, triangulation, centroid averaging, and orientation estimation.Kinematic latching ties the occluded object pose to the hand from grasp onset.
- ICTs encode entity type, shared-frame pose, hand poses in the entity frame, and grasp state using off-the-shelf perception outputs.Entity anchoring makes changing hand relations reflect approaching, grasping, and transporting states.
- The policy conditions on ICT tokens and an RGB image to generate a K-step bimanual action trajectory containing both hands’ poses and binary grasps.
- Flow matching trains a transformer velocity field to transport Gaussian prior samples toward ground-truth actions and integrates the learned ODE with fixed-step Euler inference.
- Three auxiliary objectives predict object motion, 2D entity traces, and latent consistency, producing dense supervision across 3D physical, 2D visual, and latent spaces.Their automatically derived targets share the context encoder and act as a multi-task regularizer.
4 Experiments
HumanEgo is evaluated across four real-world manipulation tasks for performance, data efficiency, robustness, and component contributions. It achieves strong task-wide performance from minutes of human video, transfers across conditions without retraining, and benefits from spatial and auxiliary representations.
- Evaluation setup: Experiments evaluate HumanEgo across four real-world manipulation tasks and report success rates over randomized trials.The evaluation spans pick-and-place, multi-step coordination, contact-rich manipulation, and sustained rotational control.
- Overall performance: 92.5% average success across four tasks makes HumanEgo the highest-performing method on every individual task.The five human-video baselines range from 1.9% to 45.0%.
- Data efficiency: 75.0% success with 15 minutes of human data surpasses ACT’s 51.2% success with 30 minutes of robot teleoperation.This comparison uses half the human-data collection time against matched robot teleoperation.
- Data efficiency: At approximately 7 minutes, HumanEgo reaches 50% success and scales to 95% at 30 minutes of human demonstrations.The reported scaling curve rises steeply and monotonically.
- Ablations: Auxiliary losses produce the largest reported gain at 8 minutes, improving success from 37.5% to 57.5%, while both variants reach 95% at 30 minutes.The benefit is strongest when demonstrations are scarce and diminishes as data grows.
- Generalization: HumanEgo maintains 85–91.25% success across varied visual conditions and transfers across novel object placements, cameras, and robot arms without retraining.The evaluation covers background, lighting, viewpoint, distractors, object placement, RealSense or ZED cameras, and Trossen, Franka, or UR10 arms.
- Representation ablation: Adding ICT raises success from 7.5% to 85% with raw human RGB, while the full system reaches 95%.Visual preprocessing alone yields only modest gains, whereas explicit spatial representation supplies the major improvement.
- Auxiliary-objective ablation: At 15 minutes, object-motion, latent-consistency, and 2D-trace objectives contribute +17.5, +12.5, and +5 percentage points individually, combining for +25 pp.The objectives supervise forward dynamics in complementary spaces.
5 Conclusion
HumanEgo learns robot manipulation policies from minutes of human egocentric video without robot data or large-scale pretraining. It combines embodiment-agnostic perception, ICT spatial encoding, flow matching, and dense auxiliary objectives, while remaining limited by several perception and precision constraints.
- The framework bridges embodiment differences through arm inpainting, keypoint rendering, and Interaction-Centric Tokens, producing invariance to embodiment, viewpoint, and environment.
- 92.5% average success across four real-world tasks accompanies zero-shot generalization to novel robots, cameras, and environments without retraining.
- The pipeline depends on Aria stereo hand tracking, robust object tracking, and cascaded perception modules, while few-shot learning plateaus near 1 cm precision.
A Data Collection Details
The data-collection system uses Project Aria Gen1 glasses to produce synchronized multimodal recordings and metric hand and camera signals for HumanEgo.
- Project Aria Gen1 glasses record RGB, stereo SLAM, eye tracking, IMUs, and environmental metadata during demonstrations.
- All streams are hardware-timestamped and synchronized to a common Aria clock at millisecond-level alignment.
- Aria Machine Perception Services post-process recordings into metric, ready-to-use signals for the pipeline.
- Closed-loop SLAM supplies drift-corrected 6-DoF camera trajectories, while hand tracking provides 3D hand skeletons for triangulation and ICT.
A.2 Task Details
The appendix specifies the evaluated manipulation tasks, human-data collection protocol, and perception preprocessing used to construct training inputs. Human demonstrations are shorter than matched teleoperation demonstrations, while object geometry is recovered through a brief static scene sweep and multi-view triangulation.
- Task Details: The evaluation covers four real-world tasks spanning pick-and-place, multi-step bimanual coordination, contact-rich reasoning, and sustained rotational control.
- Serve Bread: Serve Bread randomizes object offsets and depths, requiring the robot to grasp a croissant and place it on a plate.
- Downstack Cups: Downstack Cups varies table height and cup positions while requiring toppling, grasping, and covering in a three-step sequence.
- Water Flowers: Water Flowers varies pot and faucet placement and requires temporally ordered bimanual faucet-and-nozzle control.
- Collection Protocol: Human demonstrations take approximately 30–40 seconds, compared with 60–70 seconds for matched teleoperated demonstrations.
- Perception Preprocessing: A 1–2 second scene sweep creates viewpoint diversity for multi-view triangulation before manipulation begins.
- Perception Preprocessing: Automatic phase detection uses head and hand motion to retain manipulation frames and exclude walking, scene sweeps, transitions, and final holds.
- Hand-to-Gripper Transfer: MCP-joint geometry defines a gripper orientation that remains stable throughout pinch grasps because the MCP points do not collapse.
C Training Details
HumanEgo trains a conditional flow-matching policy with dense auxiliary objectives and stabilization techniques, then deploys it on a bimanual WidowX platform using RGB observations.
- Flow Matching Policy: The flow-matching policy maps Gaussian noise to bimanual action chunks along a linear interpolation and reweights position, rotation, and grasp losses.
- Policy Network: A transformer velocity field conditions action chunks on RGB features, time embeddings, and per-entity ICT state tokens.
- Auxiliary Objectives: Auxiliary heads predict object dynamics, future image keypoints, and temporally consistent hand tokens alongside action velocity.
- Training Stabilization: Region attention focuses image processing on active manipulation anchors, while state-noise injection improves robustness to perception noise.
- Optimization: Training uses AdamW with a 10^-4 base learning rate, cosine decay, 200-step warmup, batch size 32, and 400 epochs.
- Inference Setup: Main experiments use two side-by-side 6-DoF WidowX AI arms and a top-down Intel RealSense D405 RGB camera, excluding the arms' wrist cameras.
- Inference: At test time, a fixed-step Euler solver uses 20 inference steps to generate a 50-step bimanual action chunk.
- Control: The controller replans at 10 Hz, executes subsampled actions at 5 Hz, and applies smoothing and displacement limits for safer motion.
E.1 Hand Tracking Method Study
Hand-tracking quality strongly determines HumanEgo performance: stereo depth and persistent, smooth tracking matter more for downstream success than residual pose accuracy alone.
- Stereo depth: 95 % success with Aria-MPS falls to at most 45 % with monocular RGB because depth ambiguity creates a 5–11 cm systematic offset.The offset propagates into the ICT reference frame and prevents consistent grasp learning.
- Tracking quality: WiLoR reaches 45 % versus HaMeR’s 32.5 % despite nearly identical 1.4 cm per-keypoint residual errors after Procrustes alignment.The comparison attributes the gap to tracking smoothness and persistence rather than pose accuracy alone.
- Tracking quality: MediaPipe achieves 0 % success with 66.5 % detection because camera-depth lifting flattens hand thickness to approximately 7 cm versus 16 cm for Aria.This removes pose information that ICT relies on.
- Practical implication: The practical recommendation is to prioritize the perception frontend, especially stereo or learned-depth hand tracking, for hand-derived spatial-token policies.The study identifies frontend tracking as the highest-leverage upgrade.
E.2 Human-Robot Co-Training Study
With total collection time fixed, increasing the human-video fraction consistently improves performance, and pure human data outperforms pure robot teleoperation.
- Results: 65 → 72.5 →77.5 →90 →95 % success as the human-data ratio increases from 0/25/50/75/100 %.The study uses 30 minutes total and evaluates 40 real-world Serve Bread trials per condition.
- Mixture trend: The pure-human policy reaches 95 % versus 65 % for pure robot teleoperation, a +30 pp improvement with no observed co-training sweet spot.The 75/25 mixture reaches 90 %, five points below the human-only condition.
- Human-data contribution: Replacing 25 % of robot teleoperation with egocentric video raises success from 65 % to 72.5 %, despite reducing robot data from 30 to 22.5 minutes.The result indicates higher marginal signal from the human demonstrations in this comparison.
E.3 Reference Frame Study
ICT reference frames trade off low-data learning speed against large-data reliability and deployment invariance: anchor frames help early, while camera frames catch up with more data.
- Trade-off: ICT’s reference-frame choice is therefore a key design decision with a clear trade-off that shifts with training-data volume.The study directly compares anchor-frame and camera-frame representations.
- Low-data regime: The anchor frame substantially accelerates policy learning with few demonstrations by tying spatial reasoning to the task-relevant object.This helps recover hand–target geometry at contact from fewer examples.
- Large-data regime: With more training data, the camera frame catches up and surpasses the anchor frame because it is grounded in the raw sensor and avoids inherited object-estimation noise.The comparison describes a data-dependent reversal between the two representations.
- Deployment property: The anchor frame remains invariant to camera placement because entity coordinates are expressed relative to the object.This supports deployment with cameras at different reasonable angles, heights, or distances.