Source-linked AI summary
Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning
Zishuo Li, Bowen Yang, Changtao Miao, Kai Zhu, Hao Chen, Qingze Guan, Zhengxing Wu, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan, Zhenchao Jin, Jiadong Hong, Taowen Wang, Yushi Feng, You Liu, Yibo Wang, Yifan Yang, Zhaowen Zhou, Man Luo, Hao Cheng, Bo Zhang, Jianshu Li, Jiansheng Cai, Guocai Yao, Jize Zhang, Chenhao Lin, Renjing Xu, Lequan Yu, Chao Shen, Chunhua Shen, Zhe Li
TL;DR
Embodied learning needs large, diverse, structured real-world manipulation data, but existing resources rarely combine scalable capture, rich annotations, and reusable robot-learning tools. Open-AoE provides a smartphone-based dataset and toolchain spanning capture, structured processing, downstream adaptation, and model training, with broad semantic coverage and near-complete temporal annotation coverage.
Problem
Embodied learning requires large, diverse, geometrically, temporally, and semantically structured physical demonstrations, yet existing resources rarely provide these together.
Method
Open-AoE couples consumer-smartphone egocentric capture with processing that aligns vision, language, hand motion, camera trajectories, and action boundaries, plus tools for visualization, retargeting, and model training.
Results
99.99% of the evaluated timeline is covered by Open-AoE action annotations, including 32,407 distinct natural-language descriptions and structured object, verb, and scene fields.
Takeaways & Limitations
Open-AoE provides open infrastructure connecting low-cost real-world collection with multiple embodied-learning paradigms, including VLA policies, World Action Models, and World Models.
Takeaways & Limitations
Cross-dataset annotation-consistency results are an audit under a shared evaluator rather than a fully controlled benchmark because datasets differ in annotation schemas and textual detail.
Abstract
from arXiv · showhide
Egocentric videos of human manipulation provide scalable supervision for embodied intelligence, yet existing resources rarely combine low-cost continuous capture, manipulation-level structured annotations, and reusable tools for robot learning. We present Open-AoE, an open, community-oriented egocentric manipulation dataset and toolchain spanning the full pipeline from smartphone capture to model training. Its first release contains approximately 2,000 hours of manipulation video collected in natural environments by 500+ contributors using 400+ smartphones. The dataset provides text annotations, MANO-based hand poses, camera trajectories, and temporally localized atomic actions. Open-AoE further includes a data processing pipeline that transforms raw recordings into structured samples through temporal action segmentation, semantic annotation, hand reconstruction, and camera trajectory reconstruction. Meanwhile, we provide a separate downstream toolchain supports visualization, cross-embodiment retargeting, model-specific data conversion, and training recipes for VLA policies, WAMs, and World Models. By integrating scalable capture, structured processing, and downstream adaptation, Open-AoE reduces the barriers to both data contribution and reuse, providing practical open infrastructure for embodied model training, human-to-robot transfer, and world modeling.
1. Introduction
Open-AoE addresses the need for large, diverse, geometrically, temporally, and semantically structured manipulation data by providing an open egocentric capture-to-training infrastructure. Its first release combines approximately 2,000 hours of smartphone video with structured annotations, processing tools, and downstream embodied-learning support.
- Motivation: Embodied foundation models require large, diverse physical demonstrations that encode perception, hand motion, object contact, and temporally extended task completion.Manipulation-learning data must also be structured in geometry, time, and semantics.
- Motivation: Egocentric video captures hands, objects, scenes, and action progress in one stream while enabling natural, continuous, and cheap collection in everyday environments.Its viewpoint is closer to robot execution than third-person video and is more natural to collect than robot teleoperation.
- Open-AoE: Open-AoE releases an end-to-end pipeline spanning consumer-smartphone capture, systematic processing, structured reconstruction, and model training.The project targets open infrastructure rather than simply adding more egocentric video.
- Dataset: Approximately 2,000 hours of egocentric human manipulation data were collected in natural environments by 500+ contributors using 400+ smartphone models.The first release covers diverse participants, devices, and everyday manipulation scenarios.
- Dataset: The dataset provides videos, text descriptions, MANO hand poses, camera trajectories, and temporally localized atomic action segments for structured training samples.The processing pipeline converts raw smartphone videos into these structured signals.
- Toolchain: The downstream toolchain supports data visualization, 4D interaction reconstruction, robot motion replay, vision-language-action training, world action models, and world model learning.Researchers can reuse the corpus without rebuilding a private post-processing stack.
2. Related Work
Prior egocentric datasets vary in coverage, sensing fidelity, geometric supervision, and scale, while recent work extends human video toward reconstruction, robot transfer, VLA policies, World Action Models, and predictive world models. Open-AoE complements these efforts with a large smartphone-captured release and structured manipulation supervision.
- Egocentric datasets: Egocentric corpora range from broad daily-activity coverage to geometry-rich supervision with hand, object, and camera information.EPIC-KITCHENS-100 and Ego4D provide broad activity, environment, and language coverage; HOI4D and HOT3D add geometric and motion-capture supervision.
- Egocentric datasets: OpenEgo standardizes 1,107 hours from six public datasets into hand poses and localized action primitives.This line of work addresses dataset normalization and reuse across existing public resources.
- Egocentric datasets: Open-AoE complements large-scale and high-fidelity releases with 2,000 hours of smartphone video from 500+ contributors, 400+ device types, 400+ scenes, and 8,000+ tasks.The release also includes atomic action descriptions, MANO hand motion, and camera trajectories.
- Reconstruction and cross-embodiment transfer: Reconstruction and transfer methods convert egocentric observations into world-space hand motion, metric hand-object trajectories, contact-consistent trajectories, or robotized observations.HaWoR, EgoInfinity, EgoAERO, and EgoEngine represent complementary approaches to geometric reconstruction and cross-embodiment transfer.
- Downstream embodied models: Recent model families use human video for action-aware policy learning, action-conditioned observation dynamics, or reusable scene and interaction prediction.VLA methods include Being-H0, EgoVLA, and VITRA; WAMs include latent-action learning and LaWAM; world models include DreamDojo, iVideoGPT, and Ctrl-World.
3. Open-AoE Dataset · 3.1. Data Processing Pipeline · 3.2. Edge-side Online Detection
Open-AoE combines online edge capture with offline processing to produce an anonymized training corpus. Its device-side front end selectively records meaningful hand-object interactions while adapting capture conditions and enforcing operational constraints.
- 3.1. Data Processing Pipeline: Offline processing performs quality checking and scene labeling before reconstruction, atomic-action annotation, and three-gate quality inspection.The pipeline uses image-based detection, video slicing, and large-model detection during scene labeling.
- 3.1. Data Processing Pipeline: The pipeline combines online capture and offline processing to produce an anonymized training corpus.It comprises four stages: edge-side online detection; offline quality checking with scene labeling; reconstruction and annotation; and quality inspection with data delivery.
- 3.2. Edge-side Online Detection: Edge-side vision models gate recording in real time so that only clips containing meaningful hand-object interactions are retained.The lightweight models govern capture quality directly on the device.
- 3.2. Edge-side Online Detection: A hand-visibility check starts recording when hands appear, stops it when they leave the frame, and prompts users by voice to readjust.
- 3.2. Edge-side Online Detection: Under dim lighting, the device enables constant fill light and global auto-exposure to adapt capture conditions.
- 3.2. Edge-side Online Detection: Fixed focal length, motion stabilization, deblurring, and adaptive frame rate sustain real-time performance during capture.
- 3.2. Edge-side Online Detection: Recording halts when storage becomes insufficient or the device overheats, while only valid and permitted clips are uploaded for offline processing.The front end protects privacy at the source and saves storage and bandwidth.
3.3. Offline Quality Check & Scene Labeling · 3.4. Reconstruction & Annotation
The pipeline first filters and segments recordings using image-based quality checks, frame-rate fixation, lineage tracking, and semantic validation. Qualified segments then undergo joint camera, hand, and atomic-action reconstruction and annotation.
- 3.3.1. Image-based Detection: Rule-based detectors flag defective frames and streams, including rotation, integrity failures, exposure problems, insufficient duration, and decoding errors.The first gate uses sampled frames, image-analysis algorithms, and lightweight detectors without deep models.
- 3.3.2. Video Slicing and Frame-Rate Fixation: Qualified and unqualified spans are split into semantically independent Parts, with overly short qualified segments flagged separately.The pipeline also fixes frame rate before creating Parts.
- 3.3.2. Video Slicing and Frame-Rate Fixation: A unified process-tag file links each Part to its source video and accumulates multi-level labels throughout processing.This preserves data lineage from raw video to each Part.
- 3.3.3. Large-Model Detection: Qwen3.7-Plus performs semantic compliance and valid-action detection, identifying specification violations and incomplete or purposeless manipulations.These checks complement image-level screening by evaluating higher-level scene semantics.
- 3.4. Reconstruction & Annotation: Each qualified Part jointly produces camera-trajectory estimation, hand reconstruction, and atomic-action annotation.This reconstruction stage combines the three outputs rather than treating them as separate downstream products.
- 3.4. Reconstruction & Annotation: DROID-W recovers stable 6-DoF camera trajectories after robust-kernel retuning for handheld and wearable capture under severe shake and torso-motion interference.The method is adapted specifically to egocentric capture conditions.
3.5. Quality Inspection & Data Delivery
Reconstructed and annotated Parts undergo a three-gate quality inspection before archiving, yielding approximately 2,000 hours curated for open-source release.
- Quality Inspection & Data Delivery: Approximately 2,000 hours are curated from the full data pool for open-source release after three-gate quality inspection.The gates assess hand-reconstruction completeness, retargeting correctness, and consistency before archiving.
3.6. Data Distribution Analysis
Open-AoE’s 100-hour distribution audit reveals broad semantic, contributor, device, and optical diversity, with coverage distributed across many collectors and consumer-camera configurations. These distributions support compositional and visually robust manipulation learning, while the downstream benefit of camera diversity remains a hypothesis requiring controlled ablations.
- Semantic and temporal diversity: 32,407 distinct action descriptions, 175 verbs, 8,030 object strings, and 135 scene labels characterize the updated 100-hour audit.Temporal annotations achieve 99.99% coverage, with 9.64-second mean segments and 13.97 segments per minute.
- Semantic and temporal diversity: The joint diversity of scenes, actions, and interaction objects provides a substrate for compositional generalization across manipulation contexts.The sample spans living rooms, workshops, bathrooms, outdoor areas, laundry spaces, and retail or service settings.
- Contributor diversity: 2.6 minutes is the median contribution per anonymous collector ID, while the ten largest contributors account for only 13.7% of sampled duration.This long-tailed coverage preserves variation in manipulation habits, hand appearance, and recording environments.
- Camera and optical diversity: 400+ consumer-smartphone market models are represented in the random 100-hour sample, broadening the reported device coverage for egocentric manipulation data.Using the same random sample makes device, semantic, context, and contributor distributions directly comparable.
- Camera and optical diversity: 30.7%, 30.2%, and 31.7% are the shares of the three dominant 5-degree horizontal-FOV bins, concentrated around 65°–75° and 90°–95°.The multimodal profile reflects recurring camera and lens configurations, requiring policies to handle varied projections, crops, distortions, and imaging pipelines.
- Scope and limitations: All reported distributions use the same random 100-hour sample, while the downstream benefit of camera-domain diversity remains a training hypothesis requiring controlled ablations.The common sampling scope supports direct comparison across semantic, contextual, contributor, device, and optical distributions.
4. Open-AoE Toolchain
The Open-AoE Toolchain turns synchronized egocentric evidence into inspectable, reconstructable, retargetable, and training-ready research assets. Its components form a traceable path from data diagnosis through interaction reconstruction and cross-embodiment conversion to model learning.
- Toolchain foundation: AoE samples align undistorted RGB, camera intrinsics and trajectories, bilateral MANO reconstruction, validity masks, and atomic-action annotations on one timeline.The released implementation, documentation, and model-integration recipes are available in the Open-AoE GitHub repository.
- AoE-Visualization: AoE-Visualization overlays MANO meshes, 21-keypoint skeletons, wrist trajectories, and atomic actions while exposing reprojection error, missing hands, SLAM drift, and semantic misalignment.It combines the image canvas with world-frame 3D and timeline views for synchronized inspection and error diagnosis.
- AoE-Reconstruct-Retarget: AoE-Reconstruct-Retarget produces 4D hand-object assets, robot-executable trajectories, and robotized video, coupling reconstruction, retargeting, and visual-context preservation.The repository integrates EgoInfinity and Do-as-I-Do reconstruction, multiple robot retargeting backends, and simulation and egoview synthesis for validation.
- Robotized video: Robotized video preserves the real environment, manipulated object, illumination, and task progression while changing actor appearance for aligned human-to-robot transfer signals.It is intended for visual-domain transfer, retargeting diagnosis, and robot-scene consistency learning, not as a substitute for real robot execution.
- AoE-Training-Ready: AoE-Training-Ready provides model-specific representations for VLA policies, World Action Models, and World Models rather than a universal training format.Recipes span dense MANO, robot-facing, hand-plus-camera, and latent or weak action representations, supporting policy learning, cross-embodiment control, and visual-dynamics modeling.
- Integrated workflow: The three toolchain components reuse the same egocentric evidence at geometric, action, and visual-dynamics levels in one traceable path.This connects data inspection, interaction reconstruction, cross-embodiment conversion, and model learning.
5. Dataset Analysis
Open-AoE exhibits broad visual coverage, extensive and temporally continuous semantic supervision, strong image-annotation consistency, and dense downstream training-sample availability. These results characterize its data profile across complementary visual, semantic, annotation, and utility dimensions, with cross-schema evaluation caveats.
- Evaluation scope: The analysis compares Open-AoE with OpenEgo, EgoDex, and EgoXtreme across visual diversity, semantic and temporal supervision, annotation consistency, and downstream training utility.Approximately 100 hours are evaluated for Open-AoE, OpenEgo, and EgoDex, while EgoXtreme uses its entire available release.
- Visual diversity: Open-AoE achieves the highest mean in all six visual-distribution metrics: Effective Rank 97.43, Participation Ratio 40.47, Meaningful Coverage 19.89, Normalized Cluster Entropy 0.712, Effective Clusters 16.29, and kNN Domain Mixing 0.0775.The largest advantages are reported for Effective Rank and kNN Domain Mixing, indicating broader active feature directions and stronger cross-dataset local connectivity.
- Semantic breadth and temporal coverage: Open-AoE provides 32,407 distinct natural-language action descriptions and near-complete temporal coverage, alongside 8,030 object strings, 175 action verbs, and 135 scene labels.OpenEgo has 26,864 native labels and EgoDex has 111 categorical action classes; Open-AoE’s structured fields support multiple semantic representation levels.
- Annotation consistency: Open-AoE reaches a sequence-macro image-annotation consistency score of 4.583/5, exceeding OpenEgo (3.029), EgoDex (2.916), and EgoXtreme (2.015).Its 95% bootstrap confidence interval is [4.557, 4.608], and 85.4% of sequences score at least 4; interpretation is limited by differing annotation schemas and textual detail.
- Downstream training utility: Open-AoE yields approximately 1,760 candidate history-future windows per hour, corresponding to 97.8% of the theoretical ceiling, while 98.93% of evaluated entries contain at least one valid hand signal.Candidate windows use 2-second history, future, and stride horizons; the dense extraction reflects temporal continuity and minimal sequence-boundary loss.
6. Conclusion
Open-AoE couples approximately 2,000 hours of real-world egocentric manipulation data with a complete path from smartphone capture to embodied models. Its open ecosystem integrates structured processing, downstream tools, and community contributions so data, tools, and models can evolve together.
- Contribution: Approximately 2,000 hours of real-world egocentric manipulation data are captured with consumer smartphones and connected to a complete path from data to models.The infrastructure links scalable data capture with downstream model use.
- Contribution: The production pipeline converts raw smartphone video into structured samples aligned across vision, language, hand motion, camera trajectories, and action boundaries.These aligned modalities support manipulation-oriented reuse of captured videos.
- Contribution: The open-source toolchain supports synchronized visualization, 4D reconstruction, cross-embodiment retargeting, and training representations for Vision-Langu.The toolchain addresses downstream consumption and adaptation of the structured data.
- Open ecosystem: Open-AoE is designed as an open ecosystem where contributors, institutions, robot developers, and model teams jointly evolve data, tools, and models.The project invites contributions of real-world scenes, tasks, reconstruction and retargeting methods, training adapters, and evaluation results.
Organizations
The paper’s affiliations span Ant Digital Technology and universities in Hong Kong, Guangzhou, Singapore, Beijing, mainland China, and Xi’an.
- Ant Digital Technology, Ant Group is listed as an author affiliation.
- The affiliations include The University of Hong Kong, HKUST (Guangzhou), and The Hong Kong University of Science and Technology.
- Additional affiliations are the National University of Singapore, Beijing Academy of Artificial Intelligence, University of Chinese Academy of Sciences, and Xi’an Jiaotong University.
Appendix · Privacy and Ethical Statement
Open-AoE’s collection and release were based on informed, voluntary contributor authorization and confirmed uploads. Privacy-preserving processing, restricted research purposes, and user obligations were used to protect contributors and support ethical open release.
- Privacy and Ethical Statement: All contributors received information about collection purposes, data types, scientific use, and public release before voluntarily consenting.Consent covered collection, processing, research use, and open-source distribution.
- Privacy and Ethical Statement: Contributors reviewed collected data locally on their mobile devices and explicitly confirmed each upload.Upload confirmation occurred before the data entered the controlled cloud environment.
- Privacy and Ethical Statement: Controlled cloud processing masked visual regions that might reveal personal or sensitive information.The privacy-preserving processing was performed after confirmed upload.
- Privacy and Ethical Statement: The processing removed or desensitized metadata unrelated to embodied-intelligence research.This measure was part of the dataset’s privacy-preserving processing.
- Privacy and Ethical Statement: The dataset is released exclusively for scientific research and technological development in embodied intelligence, robotics, and related fields.It is not intended for identity recognition, personal profiling, surveillance, re-identification, or other rights-infringing uses.
- Privacy and Ethical Statement: Users must comply with the applicable license, privacy regulations, and generally accepted research-ethics principles.The supplied passage also states that users must not attempt to recover or infer contributor identities.
- Privacy and Ethical Statement: The authors identified no unresolved material privacy or ethical concerns associated with research use and open release.This assessment is based on informed consent, authorization, local review, upload confirmation, controlled privacy processing, data minimization, and anonymization.