EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks

• Egocentric data offers natural first-person perception, unrestricted in-the-wild movement, and low-cost scalability through simple wearable hardware.The introduction identifies limited environmental diversity and poor extensibility as constraints on existing robot datasets; VLA progress alone cannot overcome them.Source: §1 · pp. 1–2
• EgoLive comprises 1,680 hours of stereo video at 60 frames per second, covering 65,866 episodes and 346 real-world tasks.Source: §1 · p. 2
• To the authors’ knowledge, EgoLive is the largest open-source annotated egocentric dataset for real-world human tasks to date.Source: §1 · p. 2
• Its custom head-mounted device captures human-like stereo vision at 60 frames per second and 2160×2160 resolution per camera, with a 130° × 130° field of view.Annotations include 6-DoF motion tracking, fine-grained semantic segmentation, and 3D scene reconstruction, all described as high-accuracy.Source: §1 · p. 2
• EgoLive is collected entirely in unconstrained real-world environments, providing scene diversity and ecological validity for robot learning.Source: §1 · p. 2

2.1 Manipulation Data Collection

• Teleoperation can yield high-fidelity trajectories but scales poorly, while UMI reduces visual gaps yet remains embodiment-specific and difficult to adapt to dexterous hands.Source: §2.1 · p. 3
• Wearable-camera egocentric collection removes hardware and spatial constraints while preserving natural hand morphology for scalable, high-DoF dexterous manipulation data.Source: §2.1 · p. 3

2.2 Human Egocentric Dataset

• Generalist datasets offer broad video and language coverage but lack unified manipulation geometry, while manipulation-centric datasets add rich interaction signals yet mostly center on household, laboratory, or tabletop settings.Source: §2.2 · pp. 3–4
• Deployment-scale datasets expand operational capture: Egocentric-10K records factory footage with sparse annotations, while Xperience-10M adds multimodal sensing and hierarchical task and sub-task language.Source: §2.2 · p. 4
• EgoLive targets underexplored real service scenarios with multimodal geometric and sub-task annotations; Table 1 reports the second-longest duration and superior spatiotemporal resolution and annotation comprehensiveness.Source: Table 1 · p. 3, §2.2 · p. 4

2.3 Learning from Egocentric Data

• Capability-learning work uses egocentric demonstrations for action-level learning and larger-scale pretraining, showing their value as scalable sources of action representations and manipulation priors.Source: §2.3 · p. 4
• Human-to-robot transfer methods address embodiment gaps through image or representation alignment, retargeting, shared action spaces, and staged adaptation.Source: §2.3 · p. 4
• Whole-body-control studies extend egocentric learning beyond hand-centered manipulation to humanoid control, coordinated active vision and manipulation, and loco-manipulation.Source: §2.3 · pp. 4–5

3 Dataset

• EgoLive is a large-scale in-the-wild egocentric manipulation dataset with rich, high-quality annotations, wide coverage, and high local density.Its annotations cover action trajectories, semantic understanding, and 3D reconstruction.Source: §3 · p. 5

3.1 Data Collection

• JoyEgoCam captures stereo RGB at 2160 × 2160 resolution and 60Hz, synchronized with 200Hz IMU measurements, using a minimally intrusive design for natural real-world actions.Videos typically last 1 to 3 minutes and record a single continuous mid- to long-duration manipulation activity.Source: §3.1 · pp. 5–6

3.2 Automated Data Annotation

• The pipeline produces synchronized 6D wrist and hand-joint trajectories, hand-object interaction states and action descriptions, plus stereo-optimized 3D hand keypoints and 1152 × 1152-resolution depth.Hands and interacted objects are detected using the model proposed in [38], then tracked via BoT-SORT [2]; SAM2 [34] generates masks, and fine-tuned Qwen3-VL-32B [3] generates descriptions.Source: §3.2 · pp. 6–7

3.3 Dataset Distribution

• EgoLive shows broader semantic coverage and longer-tailed object, action, and attribute distributions than EgoDex and Xperience-10M.The comparison uses label-frequency distributions; the word clouds also show varied task categories and semantic labels.Source: §3.3 · pp. 7–8, Figure 5 · p. 8
• In continuous feature space, EgoLive spans a broader representation manifold with more locally coherent clusters than EgoDex and Xperience-10M.The analysis embeds images with Cosmos-Embed1-448p and visualizes them using t-SNE.Source: §3.3 · p. 8

4.1 Hand Reconstruction

• EgoLive’s 2D hand keypoints align more closely with hands than EgoDex’s, while its 3D skeletons align with hand point clouds and maintain spatial consistency.The framework also addresses depth drift and preserves absolute physical scale under severe dynamic occlusions.Source: §4.1 · pp. 8, 10, Figure 7 · p. 9, Figure 8 · p. 10

4.2 Depth Reconstruction

• Depth reconstruction shows low mean errors within the typical human-operation range, approximately below 0.9 m, and recovers spatial structures in representative real-world scenes.Evaluation uses checkerboard-based references captured from 0.5 m to 3.5 m; the quantitative results are reported in Table 2.Source: §4.2 · pp. 10–11, Table 2 · p. 10, Figure 10 · p. 12

4.3 Instruction Captioning

• Instruction captions segment episodes into 1–20-second atomic-action clips and explicitly describe hands, manipulated objects, and actions, following observations [12] [25].Caption quality is assessed with LLM-as-a-Judge for hand, object, action, and global consistency; examples are presented for the cases in Figure 11.Source: §4.3 · pp. 11–13, Table 3 · p. 12, Figure 11 · p. 13

5 Conclusion

• EgoLive is the world’s largest open-source egocentric dataset dedicated to human daily operational activities in realistic, utility-driven working scenarios.It uses purpose-built hardware, a scalable collection pipeline, and a high-accuracy annotation framework to deliver industrial-grade demonstration data for robot learning.Source: §5 · p. 13
• EgoLive offers qualitative improvements over prior public egocentric datasets in data scale, quality, and diversity.Its deployment-friendly devices and streamlined production pipeline are designed to support continuous growth in scale and coverage.Source: §5 · p. 13
• EgoLive is intended to support human-to-robot alignment and humanoid robot policy learning, and ultimately bridge human behavior and robot action.The dataset is designed as an ever-growing knowledge base of natural human behavioral priors.Source: §5 · p. 13