Source-linked AI summary
LookOut: Real-World Humanoid Egocentric Navigation
Boxiao Pan, Adam W. Harley, C. Karen Liu, Leonidas J. Guibas
TL;DR
Real-world egocentric navigation remains challenging because dynamic obstacles, active head-turning behavior, and scalable training data are not adequately addressed. LookOut predicts future 6D head poses using temporally aggregated 3D visual features and introduces an Aria-glasses collection pipeline and dataset. The paper reports that the model is effective for the task, while identifying limits when futures are multimodal or scenarios are absent from training data.
Problem
Real-world humanoid egocentric navigation lacks methods for dynamic environments and active head-turning behavior, as well as scalable multimodal training data.
Source: §1 · p. 1Method
LookOut aggregates unprojected per-frame DINO features in 3D to predict future 6D head poses, and the authors collect training data with Project Aria glasses.
Source: §1 · pp. 1–2, §3.2 · p. 4Results
The authors report that LookOut’s temporally aggregated 3D DINO features are effective for the 6D head-pose prediction task.
Source: §1 · p. 2Takeaways & Limitations
The task formulation represents both collision-free movement and human-like information gathering through predicted head rotations.
Source: §6 · p. 8Without generative modeling, the model may regress toward the mean when multiple future paths are possible, and it misses scenarios such as rails absent from training.
Source: §5.3 · p. 8
Abstract
from arXiv · showhide
The ability to predict collision-free future trajectories from egocentric observations is crucial in applications such as humanoid robotics, VR / AR, and assistive navigation. In this work, we introduce the challenging problem of predicting a sequence of future 6D head poses from an egocentric video. In particular, we predict both head translations and rotations to learn the active information-gathering behavior expressed through head-turning events. To solve this task, we propose a framework that reasons over temporally aggregated 3D latent features, which models the geometric and semantic constraints for both the static and dynamic parts of the environment. Motivated by the lack of training data in this space, we further contribute a data collection pipeline using the Project Aria glasses, and present a dataset collected through this approach. Our dataset, dubbed Aria Navigation Dataset (AND), consists of 4 hours of recording of users navigating in real-world scenarios. It includes diverse situations and navigation behaviors, providing a valuable resource for learning real-world egocentric navigation policies. Extensive experiments show that our model learns human-like navigation behaviors such as waiting / slowing down, rerouting, and looking around for traffic while generalizing to unseen environments. Check out our project webpage at https://sites.google.com/stanford.edu/lookout.
1. Introduction
Real-world humanoid navigation from egocentric observations remains difficult, particularly in dynamic environments and when human-like head-turning behavior is needed. LookOut formulates future 6D head-pose prediction and pairs a 3D temporal-reasoning model with a Project Aria data pipeline and the AND dataset.
1. Introduction
- Real-world humanoid navigation still lacks methods for dynamic environments, active head-turn information gathering, and scalable collection of multimodal training data.Head turns can help people check for vehicles or curbs, particularly given cameras’ limited field of view.Source: §1 · p. 1
- LookOut unprojects per-frame DINO features into 3D and aggregates them over time to model environmental geometry, while training on data with pedestrians and vehicles.It predicts head rotations as well as translations to model active information gathering.Source: §1 · p. 1
- The Project Aria pipeline collects naturalistic navigation demonstrations with minimal setup and multiple data modalities, including RGB, gaze, head poses, and point clouds.The collection uses glasses and takes only a few seconds to set up at the start of a session.Source: §1 · p. 2
- The paper introduces prediction of future 6D head-pose trajectories from posed egocentric observations in the presence of static and dynamic obstacles.The task predicts both translations and rotations, representing head-turning behavior as well as movement.Source: §1 · p. 2
- AND contains four hours of real-world navigation recordings from 18 places with dense and diverse traffic.The dataset is collected using the Project Aria pipeline.Source: §1 · p. 2
2. Related Work
Prior navigation research addresses goal-directed planning, robot social navigation, or human egocentric motion, but these lines of work differ in environment dynamics, embodiment, or data coverage. LookOut focuses on short-term collision-free human navigation in real-world settings.
2. Related Work
- Vision-language navigation typically plans long-term goal-directed paths in simulation, where dynamic obstacles are absent or simple.This paper instead focuses on short-term collision-free locomotion.Source: §2 · p. 2
- Robotic social navigation predicts collision-free paths or higher-level instructions, but its wheeled or legged embodiments differ from humanoids in scale, speed, and morphology.Humanoid robot methods are described as using classical methods with laser and stereo vision in simplified environments.Source: §2 · p. 3
- Prior human egocentric methods forecast collisions, trajectories, or full-body poses, but assume static environments and do not learn active information-gathering behavior.EgoNav and EgoCast predict trajectories and full-body poses, respectively.Source: §2 · p. 3
- Existing human datasets lack the combination of real-world navigation, egocentric camera poses, and substantial static and dynamic obstacles needed for this task.Monocular datasets lack camera pose annotations, while synthetic walking data can be simple and unnatural.Source: §2 · p. 3
- Project Aria datasets record diverse activities, but the cited datasets do not capture real-world navigation with potentially colliding agents.AEA covers single-human activities, while Nymeria includes multiple actors in collaborative activities.Source: §2 · p. 3
3. Method
The method predicts a short future sequence of 6D head poses from posed egocentric video. It builds semantic and geometric features by lifting image features into 3D, aggregating across time, and processing a bird’s-eye-view representation.
3.1. Problem Formulation
- Given posed egocentric video and past head poses, the task predicts future 6D head poses, with both input and prediction windows set to eight time steps in the experiments.Head pose combines translation with a 6D continuous rotation representation and is defined in a head-centered canonical frame.Source: §3.1 · p. 3
3.2. Model
- LookOut extracts per-frame DINO features, unprojects them to a 3D feature volume, averages across time, and squeezes the up axis into a BEV map for prediction.The BEV representation is processed by a network before a three-layer MLP predicts the future pose sequence.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
- The model is supervised with combined L1 losses on translations and rotations, weighted equally in the reported experiments.The rotation term uses a rotation matrix converted from the 6D representation.Source: §3.2 · p. 4
3.3. Implementation Details
- A head-centered canonical frame lets the model predict poses relative to the current heading, while past head poses are used only during unprojection.Training uses AdamW for 700,000 steps with batch size four and takes about four days on one NVIDIA RTX A6000 GPU.Source: §3.3 · p. 4
4. Aria Navigation Dataset (AND)
The Aria Navigation Dataset (AND) was created to provide real-world human navigation data with static and dynamic obstacles. Its pipeline uses Project Aria glasses, and the resulting recordings span 18 densely populated locations.
4. Aria Navigation Dataset (AND)
- AND is the authors’ dataset for studying real-world human navigation with substantial static and dynamic obstacles.The dataset is designed to include egocentric video and head-pose data, with point clouds available for collision checking.Source: §4 · p. 4
4.1. Data Collection Pipeline
- The collection pipeline uses Project Aria glasses to record RGB, SLAM cameras, eye tracking, IMU, barometer, and GPS, then processes recordings into head poses and point clouds.Subjects navigate without predefined scripts but are instructed to check for passing vehicles before crossing roads.Source: §4.1 · p. 5
4.2. Dataset Statistics
- AND recordings cover 18 densely populated places and total about four hours, yielding 274,000 RGB frames and 36,000 clips after processing.Locations include campuses, downtown areas, and parks, with recording times chosen to capture dense traffic.Source: §4.2 · p. 5
5. Experimental Results
On held-out environments unseen during training, LookOut is evaluated for head-pose accuracy and obstacle avoidance, alongside ablations, qualitative behavior, and failure cases. The results describe collision avoidance and human-like information gathering, while identifying limits with multimodal futures and unseen obstacles.
5.1. Quantitative Evaluation
- Evaluation measures translation and rotation with L1 losses and uses non-collision scores for static and dynamic obstacles as a rough proxy for collision avoidance.The score is the percentage of predictions at least k centimeters from the closest obstacle; ground-truth sequences are reported for reference.Source: §5.1 · p. 5
5.1.1. Comparison with Baselines
- LookOut achieves the best performance across all metrics in the no-goal setting, while A∗+Lin Ext nearly avoids static obstacles but performs poorly around dynamic ones.The static-obstacle advantage comes from explicit scene occupancy; dynamic obstacles are absent from the point cloud.Source: Table 1 · p. 6, §5.1.1 · p. 6
5.1.2. Ablation Study
- Adding obstacle-proximity modalities improves non-collision metrics; DINO features and an intermediate 3D feature space improve performance, while 3D Conv matches LookOut at greater computational cost.Depth particularly helps with dynamic obstacles because the SLAM point cloud contains only static objects.Source: §5.1.2 · p. 6, Table 2 · p. 6
5.2. Qualitative Evaluation
- Qualitative examples show collision-free predictions, head turns toward useful navigation cues, waiting when paths are not readily traversable, and path shifts after new visual cues appear.The visualizations include five held-out examples with predicted and ground-truth trajectories; BEV views show translation components over the scene point cloud.Source: Figure 3 · p. 7, §5.2 · p. 8, Figure 4 · p. 8
5.3. Failure Cases and Limitations
- A lack of generative modeling may make LookOut struggle when multiple futures are possible, such as choosing either side to avoid incoming pedestrians.The authors suggest generative models as a next step for learning multimodal distributions.Source: §5.3 · p. 8
- The model also fails to predict looking down to avoid a rail that never appeared in its training set.The authors identify broader training-set coverage as a promising response.Source: §5.3 · p. 8
6. Conclusion
The paper formulates future 6D head-pose prediction for humanoid navigation among static and dynamic obstacles, and presents a model, data pipeline, and real-world dataset for the task. Experiments show diverse useful navigation behaviors and performance above baselines across all metrics.
6. Conclusion
- The task predicts future 6D head-pose trajectories from past posed egocentric video amid static and dynamic obstacles, capturing collision-free planning and human-like information gathering.The formulation includes both head translation and rotation.Source: §6 · p. 8
- LookOut uses pre-trained DINO feature encoders and parameter-free unprojection, and aggregates unprojected 3D features over time for semantic and geometric understanding.Source: §6 · p. 8
- The Project Aria glasses pipeline supports an easily scalable 4-hour real-world dataset spanning 18 places with diverse and dense traffic.Source: §6 · p. 8
- The model learns diverse behaviors useful for real-world navigation and surpasses baselines across all metrics.The conclusion also notes that the paper discusses failure cases, limitations, and future directions.Source: §6 · p. 8