Source-linked AI summary
Project Aria: A New Tool for Egocentric Multi-Modal AI Research
Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, Cheng Peng, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, David Caruso, Derek Valleroy, Dinesh Ginjupalli, Duncan Frost, Edward Miller, Elias Mueggler, Evgeniy Oleinik, Fan Zhang, Guruprasad Somasundaram, Gustavo Solaira, Harry Lanaras, Henry Howard-Jenkins, Huixuan Tang, Hyo Jin Kim, Jaime Rivera, Ji Luo, Jing Dong, Julian Straub, Kevin Bailey, Kevin Eckenhoff, Lingni Ma, Luis Pesqueira, Mark Schwesinger, Maurizio Monge, Nan Yang, Nick Charron, Nikhil Raina, Omkar Parkhi, Peter Borschowa, Pierre Moulon, Prince Gupta, Raul Mur-Artal, Robbie Pennington, Sachin Kulkarni, Sagar Miglani, Santosh Gondi, Saransh Solanki, Sean Diener, Shangyi Cheng, Simon Green, Steve Saarinen, Suvam Patra, Tassos Mourikis, Thomas Whelan, Tripti Singh, Vasileios Balntas, Vijay Baiyya, Wilson Dreewes, Xiaqing Pan, Yang Lou, Yipu Zhao, Yusuf Mansour, Yuyang Zou, Zhaoyang Lv, Zijian Wang, Mingfei Yan, Carl Ren, Renzo De Nardi, Richard Newcombe
TL;DR
Future AR devices require context-aware, personalized machine perception from ecologically valid egocentric data, but capturing such data on wearable devices remains challenging. Project Aria addresses this need with a lightweight glasses-form-factor multimodal capture system, associated machine-perception services, and research software. The resulting platform supports egocentric data and derived spatial-AI outputs for research, while recording profiles and privacy guidance constrain sensor use and recording conditions.
Problem
Future AR applications need context-aware, personalized perception from unconstrained wearable data, whose capture and interpretation are challenging under form-factor and power constraints.
Method
Project Aria combines a lightweight glasses-form-factor multimodal data-capture device with software tools and machine-perception services for recording and processing egocentric data.
Results
Project Aria provides ecologically valid recordings and spatial-AI machine-perception results as foundational resources for egocentric perception research and contextualized AI applications.
Takeaways & Limitations
The platform gives researchers a tool for studying future egocentric devices and developing compute and interaction paradigms for always-on contextualized AI.
Takeaways & Limitations
Maximum-resolution, maximum-rate recording of all sensors is infeasible because of power and bandwidth limitations, and recording profiles may reduce streams for privacy or longer recording time.
Abstract
from arXiv · showhide
Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception. These future devices will need to be all-day wearable in a socially acceptable form-factor to support always available, context-aware and personalized AI applications. Our team at Meta Reality Labs Research built the Aria device, an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research in this area. In this paper, we describe the Aria device hardware including its sensor configuration and the corresponding software tools that enable recording and processing of such data.
1. Introduction
Project Aria addresses the research gap between curated allocentric data and the egocentric, multimodal data needed for context-aware, personalized AI. The report introduces a wearable sensing platform and accompanying software to support this research.
- Existing machine-perception methods often perform comparatively poorly on egocentric data and tasks requiring structured reasoning in 3D/4D space.
- Project Aria is a lightweight, unobtrusive glasses-form-factor data-capture system designed to record ecologically valid multimodal data.
- The device provides 1–2 hours of recording with its nominal profile, with longer recordings possible using an external power bank.
- The report introduces the Aria device, software, machine-perception services, and privacy principles for research institutions worldwide.
2. Device
Project Aria is a lightweight glasses-form-factor data-collection device that records tightly calibrated, time-aligned egocentric multimodal streams through a broad sensor suite designed for future wearable machine perception.
- Design goals: The device emulates future machine-perception wearables by integrating a rich suite of egocentric multimodal sensors rather than targeting on-device computation.It is designed as a data-collection and streaming device, not to handle on-device computation workloads.
- Form factor and fit: Around 75g, the glasses form factor is intended to remain light, socially acceptable, and wearable across a broad range of users.Two device sizes, adjustable nose pads, and flexible temples support fit.
- Sensors: The sensor suite combines stereo-overlapping monochrome scene cameras, a forward-facing high-resolution RGB camera, eye-tracking cameras, dual IMUs, microphones, and a magnetometer.The scene cameras support machine perception such as Visual SLAM, while the other sensors provide visual, inertial, audio, and magnetic signals.
- Recording configuration: Recording profiles configure sensor enablement, frame rate, resolution, and output format because maximum-rate recording of all sensors exceeds power and bandwidth limits.Reducing streams or rates can also support privacy and extend recording time.
- Calibration and time alignment: Tight calibration, rigid sensor mounting, and accurate timestamps support combining multiple modalities in downstream machine perception tasks.Factory and online calibration parameters are available, and timestamps can be aligned across devices using shared time domains.
3. Recording Tools
Project Aria provides configurable recording and open-source tools for capturing, storing, accessing, visualizing, and processing multi-sensor recordings.
- The companion app configures sensor selection, frequency, resolution, and output format through research-specific recording profiles.Recordings can be started or stopped with the app or device capture button.
- Researchers can download recordings over USB or upload them to Machine Perception Services for trajectory, calibration, point-cloud, and gaze processing.MPS applies state-of-the-art processing to recover these machine-perception outputs.
- Aria stores all sensor streams in VRS, an open container supporting time-sorted records and very large files.Each stream contains records generated by a corresponding sensor set.
- The open-source C++/Python toolkit supports reading, visualizing, manipulating, and processing Aria recordings and machine-perception outputs.It includes VRS Data Provider and Viewer interfaces, plus access to device calibration data.
4. Machine Perception Services
Project Aria provides foundational Machine Perception Services for trajectories, calibration, spatial reconstruction, and eye gaze. These outputs are designed to support robust downstream 3D perception from challenging real-world recordings.
- Overview: MPS provides foundational machine perception capabilities that research partners can build upon for future AR devices.The services are enabled by proprietary algorithms designed for Project Aria data.
- Trajectories: Trajectory estimation combines cameras, IMUs, GNSS, Wi-Fi, and barometer data with sensor-model, timing, and rigidity information for robust localization.The method is intended to handle fast motion, changing lighting, and partial or temporary camera occlusion.
- Trajectories: The system outputs open-loop and closed-loop 1 kHz trajectories, with open-loop translation drift no more than 0.4% of distance traveled.Closed-loop trajectories are optimized in post-processing and expressed in a single frame of reference.
- Trajectories: Closed-loop trajectories support jointly processing multiple recordings in a common coordinate frame, with typical room-scale global RMSE translation error no more than 1.5 cm.This enables alignment across recordings such as the 18 sequences shown in Figure 8.
5. Privacy Considerations
Project Aria treats privacy as a central requirement for egocentric recording, covering both wearers and bystanders. It combines responsible-research principles with hardware and software features that facilitate privacy protection.
- Motivation: Egocentric recording devices create privacy challenges for both wearers and bystanders, especially when their form factor is unobtrusive.The paper frames privacy preservation as increasingly important as devices blend into everyday activities.
- Responsible innovation: Project Aria aims to establish privacy guidelines and principles while embedding privacy-facilitating features directly into the device where possible.These efforts are presented as part of responsible innovation for egocentric-data research.
- Device safeguards: An LED indicator signals to bystanders when the device is recording raw data.The indicator is a hardware feature intended to make recording status visible.
- Device safeguards: A privacy switch immediately stops and deletes the current recording when activated during a recording session.This lets the wearer respond immediately to a bystander’s request to delete the recording.
- Partner responsibilities: Project Aria partners are required to follow community guidance and practice responsible research protecting the privacy of wearers and non-wearers.The guidance emphasizes protecting people who do not use the research devices.
6. Example Research Applications
Project Aria supports research spanning low-level perception and high-level understanding of users and environments. Its multimodal recordings and perception outputs address challenges involving natural motion, changing environments, and long-term personal context.
- Overview: Project Aria connects low-level machine perception with high-level user- and environment-understanding research.The report presents applications that leverage calibrated sensors, raw data, multiple modalities, and MPS capabilities.
- Egocentric scene reconstruction and understanding: Scene reconstruction and semantic understanding are challenging when wearable data is collected during natural, unconstrained human motion rather than deliberate scanning.NeRF results compare carefully curated handheld recordings with egocentric recordings during natural activity.
- Object interaction: Hand tracking combined with MPS trajectories and point clouds can identify when the wearer’s hand is near a static object.The example uses all three cameras together with MPS outputs.
- Activity and attention understanding: Spatial audio, motion, and eye gaze provide additional signals that can disambiguate the wearer’s activity or attention when egocentric views are ambiguous.Examples include distinguishing playing from holding a guitar and differentiating looking at a clock from reading a book.
- Summarization and question answering: Longitudinal summarization and question answering require datasets spanning minutes to years, which currently do not exist.The paper argues that Aria’s form factor and machine perception capabilities enable research toward this setting.
7. Conclusion
Project Aria provides ecologically valid egocentric data and spatial AI technologies as foundations for contextualized AI and future AR research.
- Project Aria captures ecologically valid data in forms expected from future egocentric devices.
- The project makes spatial AI machine perception technologies available as foundational building blocks for higher-level contextualized AI applications.
- Its integrated sensor suite supports research into new compute, interaction, application, and use-case paradigms for always-on contextualized AI.