Source-linked AI summary
Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception
Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, Carl Yuheng Ren
TL;DR
Egocentric AR perception lacks a comprehensive 3D dataset covering humans, objects, environments, and realistic interactions. The paper introduces ADT, a photorealistic Aria-based benchmark with extensive synchronized ground truth and evaluates it across several perception tasks. ADT establishes a broad resource for benchmarking egocentric machine perception, while single-image 3D detection remains limited by depth-estimation failures.
Problem
Existing egocentric and AR datasets provide limited coverage of 3D space, object motion, sensing challenges, or comprehensive ground truth for humans, objects, and environments.
Method
ADT reconstructs two indoor spaces and their objects photorealistically, synchronizes Aria glasses with motion capture, and generates calibrated device, object, human, gaze, segmentation, depth, and synthetic-image data.
Results
ADT provides 200 Aria sequences in two digitized spaces and supports benchmarking object detection, segmentation, and image translation through comprehensive egocentric ground truth.
Takeaways & Limitations
ADT broadens egocentric machine-perception evaluation by combining photorealistic synthetic-real twins, extensive 3D annotations, and system-accuracy analysis.
Takeaways & Limitations
Single-image 3D detection models frequently fail at depth estimation, producing significantly erroneous predicted poses despite good 2D reprojections.
Abstract
from arXiv · showhide
We introduce the Aria Digital Twin (ADT) - an egocentric dataset captured using Aria glasses with extensive object, environment, and human level ground truth. This ADT release contains 200 sequences of real-world activities conducted by Aria wearers in two real indoor scenes with 398 object instances (324 stationary and 74 dynamic). Each sequence consists of: a) raw data of two monochrome camera streams, one RGB camera stream, two IMU streams; b) complete sensor calibration; c) ground truth data including continuous 6-degree-of-freedom (6DoF) poses of the Aria devices, object 6DoF poses, 3D eye gaze vectors, 3D human poses, 2D image segmentations, image depth maps; and d) photo-realistic synthetic renderings. To the best of our knowledge, there is no existing egocentric dataset with a level of accuracy, photo-realism and comprehensiveness comparable to ADT. By contributing ADT to the research community, our mission is to set a new standard for evaluation in the egocentric machine perception domain, which includes very challenging research problems such as 3D object detection and tracking, scene reconstruction and understanding, sim-to-real learning, human pose prediction - while also inspiring new machine perception tasks for augmented reality (AR) applications. To kick start exploration of the ADT research use cases, we evaluated several existing state-of-the-art methods for object detection, segmentation and image translation tasks that demonstrate the usefulness of ADT as a benchmarking dataset.
1. Introduction
ADT addresses the lack of holistic egocentric 3D data for AR by combining realistic activities, comprehensive ground truth, and photorealistic digitization. It is designed to support core perception problems including 3D object detection, human pose estimation, and scene reconstruction.
- Motivation: AR applications require accurate 3D digitization of humans, objects, and environments to maintain contextual awareness across physical and virtual spaces.The paper identifies dynamic object tracking and human-state estimation as additional requirements for updating and intersecting the digital space.
- Gap in existing datasets: Existing datasets typically target specific sub-problems, omit object motion from hand interactions, or provide only narrative and 2D annotations.The paper also notes that existing egocentric datasets do not address the challenges of 3D space.
- Sensing platform: Aria glasses provide RGB, monochrome, eye-tracking, and IMU sensors that support a broad range of machine perception tasks in real-world activities.The paper presents Aria as a sensor-rich platform motivating the development and use of ADT.
- Dataset contribution: ADT contains 200 sequences from an apartment and an office, with 350 stationary and 50 dynamic object instances and comprehensive device, object, human, gaze, segmentation, depth, and synthetic-image ground truth.The dataset includes calibrated sensors and visualizations of object and human ground truth in the recorded spaces.
- Dataset construction and evaluation: The dataset reconstructs every object and both environments in a metric, photorealistic pipeline synchronized with motion capture and Aria glasses.The authors evaluate reconstruction quality and object tracking accuracy, then benchmark object detection, segmentation, and image translation methods.
2. Related Work
Prior datasets cover static scenes, dynamic objects, or egocentric activities, but they differ in interaction coverage, sensing setup, and ground-truth validation. ADT is positioned against these limitations through richer 3D and human-interaction annotations.
- Object-centric datasets: Static-object datasets provide 6DoF poses but generally avoid rapid camera motion, object movement, or human interactions.Examples include Objectron, BOP datasets, T-LESS, and YCB-V, which focus on stationary or controlled object settings.
- Dynamic-object datasets: Dynamic-object datasets annotate moving objects and sometimes hand poses, but use limited object counts or constrained RGB-D sensing setups.FPHA, ECVA, Ho-3D, and TUD-L represent this category with different sensor and annotation limitations.
- Ground-truth accuracy: Many prior datasets do not quantify ground-truth accuracy, while manual validation can introduce subjectivity, annotator variability, and sensor error.The paper specifically contrasts annotator comparisons and manually annotated point clouds with the need for more reliable accuracy analysis.
- Egocentric video datasets: Egocentric video datasets capture realistic activities but commonly provide action, gaze, or 2D object annotations rather than comprehensive 3D supervision.Ego4D adds audio, environment meshes, and gaze, while TREK-150 and EgoTracks focus on hand-object interactions.
- Scene datasets: Scene datasets offer scene-level reconstruction and 3D annotations, but their objects are static and their capture devices are not egocentric.SUN-RGB-D, ScanNet, and Matterport3D are described as large-scale indoor-scene datasets recorded with RGB-D cameras.
3. Dataset Generation Methodology
ADT constructs a photorealistic, metrically aligned digital scene with tracked objects, Aria devices, and humans, then validates object-pose ground truth against Aria imagery. The resulting generative model supports synthetic renderings and per-frame annotations including segmentation and depth.
- ADT generation begins with a stationary, photorealistic digital scene and enables tracking of Aria glasses, objects, and humans within it.
- Stationary Scene Digitization: Room digitization uses FARO scanning, plane-based meshing, and physically based materials, achieving P50 and P80 meshing errors of 0.688mm and 4.68mm.
- Stationary Scene Digitization: Object digitization combines industrial 3D scanning with photogrammetric material reconstruction and real-versus-synthetic comparisons to match object appearance.
- Stationary Scene Digitization: The furnished layout is rescanned, initialized by manually placing object models, and refined with Iterative Closest Point alignment.
- Pose Generation: A common scene frame expresses object, device, and human poses across sequences, while Optitrack-based procedures derive dynamic object, Aria device, and human pose data.
- System Accuracy: The system evaluates object-pose fidelity through reprojection, rotation, and translation errors, reporting averages of 6.78 pixels, 1.29 deg, and 6.83 mm.
- Data Annotations: Combining poses with calibration completes a generative model for synthetic frame rendering, including per-pixel object IDs, metric depth, and instance-level segmentation.
4. Dataset Content
ADT records realistic activities across an apartment and office, with diverse annotated objects and image-level ground truth designed for egocentric perception research.
- The dataset contains 324 stationary object instances in the apartment, 20 in the office, and 74 shared dynamic objects.The apartment includes 281 unique stationary objects, while the office includes 15 unique stationary objects.
- The dataset emphasizes plausible real-life scenarios and diverse objects rather than contrived laboratory settings.
- Per-pixel rendering provides instance-level segmentation and metric depth, while Figure 5 accounts for chair-table occlusion in segmentation and 2D bounding boxes.
- Object categories are annotated and summarized using the top 15 categories under the COCO definition.
- ADT covers 200 sequences: 150 apartment sequences and 50 office sequences spanning single-person and dual-person activities.Apartment activities include decoration, meal preparation, work, examination, cleaning, partying, and dining-table cleaning.
5. Benchmarking
ADT benchmarks existing methods across 2D and 3D detection, segmentation, and synthetic-to-real image translation, using standardized metrics and paired data.
- The benchmark covers 2D detection, 2D segmentation, 3D detection, and image-to-image translation for AR-related perception tasks.
- 2D Detection and Segmentation: FPN and VIT-Det are evaluated on rectified Aria RGB images using COCO AP-Box and AP-Mask metrics.ADT objects are mapped into relevant COCO categories for evaluation.
- 3D Detection: Total3D and Cube R-CNN are evaluated for 3D object detection using average precision across seven common object categories.Total3D receives MaskRCNN 2D bounding boxes as input for a fair comparison.
- Image Translation: Figure 7 presents synthetic RGB, LDM output, and real RGB from left to right as an example of domain transfer.
- Image Translation: Pix2Pix, TSIT, and LDM are trained on paired synthetic-real images from 43 sequences and evaluated on 102 unseen sequences.The task measures synthetic-to-real image translation with pixel-level distance.
6. Conclusion
ADT combines photo-realistic digitization with precise multimodal ground truth and benchmarks several AR-related perception tasks. The authors conclude that its breadth and quality support new egocentric machine perception research opportunities.
- ADT provides photo-realistic synthetic-real twins for 200 Aria sequences recorded in fully digitized apartment and office spaces.
- The dataset includes analyzed ground truth for object and Aria 6DoF poses, human poses, and eye gaze, alongside benchmarks for detection, segmentation, and image translation.
7. Supplementary Material
The supplementary analyses describe ADT’s system-accuracy evaluation, benchmark existing detection and segmentation methods, analyze 3D detection failures, and assess manual 3D annotation quality. Results show domain and scale-related weaknesses in current models, while ADT supports quantitative validation of tracking and annotation pipelines.
- System Accuracy: The system-accuracy pipeline estimates object pose by optimizing alignment between projected 3D markers and manually labeled image centers.Figures 8 and 9 illustrate the optimization and its final alignment results.
- System Accuracy: The measured system error combines 3D object reconstruction, motion capture, Aria device poses, and Aria device calibration.Accuracy is slightly better in the office than in the apartment, possibly because the apartment’s higher ceilings reduce motion-capture accuracy.
- 2D Detection and Segmentation: ADT performance for FPN and VIT-Det 2D detection and segmentation is significantly lower than their performance on COCO, consistent with the datasets’ domain difference.The egocentric nature of ADT remains challenging even after rectifying fisheye RGB images.
- 2D Detection and Segmentation: Large furniture and appliances are easier to detect than small objects such as potted plants, mice, and remotes, highlighting the difficulty of indexing everyday objects in real-world recordings.Qualitative comparisons show FPN detecting large objects and objects under viewpoint variance better overall, while VIT-Det appears better on some small objects.
- 3D Object Detection: Cube-RCNN and Total3D perform significantly worse on ADT because single-image models frequently fail to estimate depth and therefore predict erroneous 3D poses.Cube R-CNN can roughly localize chairs while still failing the 0.25 IoU threshold for accurate 3D pose prediction.
- Manual Annotation: ADT ground-truth 3D bounding boxes enable quantitative evaluation and training feedback for semi-automatic manual annotation pipelines.The supplementary experiment samples 20 videos and compares annotations against accurate 6DoF ground truth; smaller objects are harder to annotate accurately.