PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoNav: Egocentric Scene-aware Human Trajectory Prediction

Weizhuo Wang, C. Karen Liu, Monroe Kennedy III

arXiv:2403.19026v3cs.CV

TL;DR

Egocentric prediction for human walking is challenged by occlusion, limited field of view, and unstructured environments where scene context matters. EgoNav combines past motion with visual memory in a diffusion model that generates multiple future trajectories. The authors report effective results in ablations and comparisons with current state-of-the-art baselines.

  • Problem

    Egocentric walking prediction must account for clutter, occlusion, limited camera views, and environmental features that past-trajectory-only approaches may ignore.

    Source: §I · p. 1

  • Method

    The method conditions a diffusion model on past egocentric motion and a compact visual memory built from aligned color, depth, and semantic observations.

    Source: §I · pp. 1–2, B. Method Overview · p. 4, C. Construction of Visual Memory · p. 4

  • Results

    The authors report that the method is highly effective in ablations and comparisons with current state-of-the-art baselines.

    Source: §VI · p. 11

  • Takeaways & Limitations

    The paper contributes an egocentric walking dataset, visual-memory representation, and faster sampling scheme for trajectory prediction research.

    Source: §VI · p. 11

Abstract

from arXiv · show

Wearable collaborative robots stand to assist human wearers who need fall prevention assistance or wear exoskeletons. Such a robot needs to be able to constantly adapt to the surrounding scene based on egocentric vision, and predict the ego motion of the wearer. In this work, we leveraged body-mounted cameras and sensors to anticipate the trajectory of human wearers through complex surroundings. To facilitate research in ego-motion prediction, we have collected a comprehensive walking scene navigation dataset centered on the user's perspective. We then present a method to predict human motion conditioning on the surrounding static scene. Our method leverages a diffusion model to produce a distribution of potential future trajectories, taking into account the user's observation of the environment. To that end, we introduce a compact representation to encode the user's visual memory of the surroundings, as well as an efficient sample-generating technique to speed up real-time inference of a diffusion model. We ablate our model and compare it to baselines, and results show that our model outperforms existing methods on key metrics of collision avoidance and trajectory mode coverage.

I. INTRODUCTION

Egocentric trajectory prediction matters for collaborative robots, but clutter, occlusion, limited camera views, and unstructured walking environments complicate prediction. The paper proposes a scene-conditioned diffusion model, visual memory, and rapid sampling, and evaluates them on a user-perspective navigation dataset.

I. INTRODUCTION

  • Trajectory prediction can inform both separate motion planners and imitation-learning approaches, which depend on predicting future trajectories from the agent’s perspective.The paper positions its model as a component for both planning paradigms.Source: §I · p. 1
  • The paper uses semantic segmentation to add dense scene labels so predicted behaviors can be related to environmental labels and preferences.This is presented as a way to improve scene understanding in learning-based motion prediction.Source: §I · p. 1
  • Egocentric prediction is difficult because wearable cameras face occlusion in cluttered 3D scenes, limited field of view, and large volumes of raw data.Prior methods often assume fully observed bird’s-eye views, while many walking predictors ignore environmental features and use only past trajectories.Source: §I · p. 1
  • The model predicts a distribution of future trajectories from egocentric observations and past walking, using diffusion, a compact visual-memory representation, and hybrid sampling for real-time generation.The visual memory captures appearance and semantics from a short history of observations; multiple sampled paths provide downstream applications with possible trajectory modes.Source: §I · pp. 1–2
  • The authors collect a user-perspective walking-navigation dataset across diverse indoor and outdoor scenarios and report that their model outperforms baselines on multiple metrics.They also conduct ablations of visual memory and hybrid generation; the dataset and trained model are to be made publicly available.Source: §I · p. 2

II. RELATED WORK

Related work spans prediction for vehicles, sports, and walking, with different assumptions about scene structure and interaction. The paper distinguishes its focus on egocentric walking prediction conditioned on environmental features.

II. RELATED WORK

  • Autonomous-vehicle methods include recurrent and transformer predictors and often use bird’s-eye or occupancy-grid representations, which can restrict predictions to a small set of modes and struggle when structured-environment assumptions fail [17], [18], [19],, [21], [22],.The literature also includes methods that extract motion from demonstrations [6] and integrate prediction with planning.Source: §II · p. 2
  • Sports trajectory prediction focuses more on game strategy and interactions than environmental features; sporting scenarios usually have few, if any, static objects or obstacles, and flat playing grounds, [24].Source: §II · pp. 2–3
  • Walking predictors have often relied on past trajectories, social context, or simplified open-space assumptions, while newer first-person and depth-based methods explore richer scene representations [8],,,, [26],, [9],,,.Prior datasets such as UCY/ETH and Stanford Drone [29] predominantly provide bird’s-eye views.Source: §II · p. 3
  • Diffusion models evolved from DDPM noise prediction to methods such as latent diffusion and DDIM for complex generation and faster processing [32], [33], [34], [35], [36].The passage notes applications across image, video, and action generation.Source: §II · p. 3

III. METHOD

The task is to model likely future paths as a distribution, using a person’s past motion and observations of the surrounding scene. Trajectories are represented as sequences of 6D poses.

III. METHOD

  • A trajectory τ is a sequence of 6D poses containing translation and orientation, and the model predicts future poses from past poses and observations of the surroundings.The observation S encodes scene appearance, geometry, and semantics from wearable visual and depth sensors.Source: A. Problem Formulation · p. 3
  • The goal is to model a distribution of future trajectories, allowing multiple likely paths in settings such as a fork where more than one route may be viable.The model approximates the future trajectory distribution conditioned on the environment.Source: A. Problem Formulation · pp. 3–4
  • The visual memory combines depth, color, and an intensity-encoded 8-class semantic channel in a panorama assembled from multiple time steps.The figure depicts four channels.Source: Figure 3 · p. 4

B. Method Overview

The method conditions a diffusion model on the wearer’s past ego trajectory and a panoramic visual memory assembled from aligned scene images to predict future trajectories. It jointly predicts encoded visual observations and trajectory states, with attention relating the visual memory to trajectory parts.

B. Method Overview

  • A diffusion model conditions future-trajectory prediction on the wearer’s past ego trajectory and visual memory, while also predicting encoded visual observations as auxiliary outputs.The inputs include a short history of RGBD images and a past trajectory transformed into an egocentric frame.Source: B. Method Overview · p. 4

C. Construction of Visual Memory

  • Visual memory is an ego-perspective panoramic representation built by aligning color, depth, and semantic images across frames in a shared scene representation.A distance-based filter removes points too far from the current pose before projection into the ego frame.Source: C. Construction of Visual Memory · p. 4
  • Unlike a single narrow-FOV depth frame, visual memory stitches past frames and multimodal inputs into one representation of scene information relevant to prediction.The comparison figure shows that a single frame can miss stairs, a right-turn path, and a nearby wall; black regions mark undiscovered areas.Source: C. Construction of Visual Memory · p. 4, Figure 4 · p. 5

D. Diffusion Pipeline for Trajectory Prediction

  • A UNet diffusion model with multi-head self-attention relates visual-memory encodings to trajectory parts, and conditions are applied across its down, up, and attention blocks.Classifier-free guidance removes the input conditions during 10% of training.Source: D. Diffusion Pipeline for Trajectory… · pp. 4–5
  • A pre-trained VAE compresses visual memory from 1x180x360x5 to a 64-dimensional embedding, while trajectory states are normalized as 100x9.The nine trajectory channels represent x-y-z positions and ortho6d orientation.Source: D. Diffusion Pipeline for Trajectory… · p. 4
  • The model uses the full 100-step past trajectory because prior motion may affect scene visibility and exploration preferences, potentially violating a pure Markovian assumption.The paper compares this choice with a pure Markovian assumption in an ablation.Source: D. Diffusion Pipeline for Trajectory… · p. 4
  • At inference, the model generates 15 trajectory samples as a discrete distribution of future ego-motion modes; authors report more stable samples when visual-memory prediction is included.The denoised sequence contains trajectory states and visual-memory encodings, with the VAE decoder reconstructing the visual memory.Source: D. Diffusion Pipeline for Trajectory… · p. 5

E. Hybrid Generation

DDPM can produce high-quality trajectory predictions but is too slow for immediate navigation, while DDIM is faster at the expense of sample quality. The proposed hybrid starts with DDIM-like generation and then refines with DDPM.

E. Hybrid Generation

  • DDPM sampling is impractical for immediate responses, especially when many denoising sequences are needed to generate a trajectory distribution.Generating more samples exacerbates the time constraints.Source: E. Hybrid Generation · p. 5
  • DDIM accelerates prediction but compromises sample quality, a trade-off the authors consider untenable for safety-relevant trajectory fidelity.Source: E. Hybrid Generation · p. 5
  • Hybrid generation first approximates the trajectory distribution with a DDIM-like approach, then refines it using DDPM.The authors state this retains the multimodal gradient landscape and nuanced variations of traditional DDPM without its latency.Source: E. Hybrid Generation · p. 5

F. Diffusion Model Detail

The diffusion model uses transformer encoder blocks in a UNet and conditions generation on past states and visual memory; overlapping sub-trajectories expand its training set. The campus dataset pairs diverse walking scenarios with synchronized state and visual data, and the authors plan to release it after face de-identification.

F. Diffusion Model Detail

  • The UNet uses transformer encoder blocks with multi-head self-attention between its Down and Up blocks, conditioned on past trajectory and recent visual memory.Those conditions are embedded before being passed into the Down and Up blocks.Source: F. Diffusion Model Detail · p. 5
  • Overlapping rather than unique sub-trajectories augment training data, yielding more than 220,000 diverse training sub-trajectories.Source: F. Diffusion Model Detail · pp. 5–6
  • The hybrid-generation figure depicts compressed visual memory and past states as conditions, with DDIM steps followed by DDPM steps.The caption describes stripped DDIM steps from 1000 to 0, followed by DDPM steps from n to 0.Source: Figure 5 · p. 6

IV. EGOCENTRIC NAVIGATION DATASET

  • The campus dataset comprises 34 collections, each approximately seven minutes and spanning over 600 meters, with diverse conditions and interactions including pedestrians.Collections cover varied weather, surfaces, terrain, and dynamic obstacles; pedestrians were not deliberately avoided.Source: §IV · p. 6
  • The dataset records torso pose, leg joint angles, torso velocities, and gait frequency alongside aligned color and depth images, segmentation masks, and visual-memory frames.The full collection contains 198 minutes of data and exceeds 400GB.Source: §IV · pp. 6–7
  • A pilot set of roughly 15% of the full dataset supports faster training and iteration with an acceptable amount of performance degradation.The performance comparison is described as qualitative.Source: §IV · p. 7
  • The dataset is intended for open release after faces are de-identified.The authors contrast it with AEA, emphasizing dense depth for outdoor navigation while AEA focuses on indoor daily manipulation and SLAM key points.Source: §IV · p. 7

A. Dataset Detail

The dataset combines 20Hz state and visual records with a keyframing strategy for raw depth and separate 20Hz visual-memory generation. Its semantic classes also define which scene elements count in collision evaluation.

A. Dataset Detail

  • At 20Hz, the dataset records torso localization and motion, leg angles, gait frequency, RGB, aligned 848x480 depth, segmentation masks, and 360-degree panoramas.The collection totals 198 minutes and over 400GB.Source: A. Dataset Detail · p. 7
  • Raw depth is retained when camera movement exceeds 15 degrees of rotation or 1 meter of translation, reducing point-cloud processing to about one keyframe per second.Visual memories are generated separately at 20Hz and center-aligned with the camera pose.Source: A. Dataset Detail · p. 7
  • A tuned Canny filter removes up to 10 pixels around disjoint depth edges to prepare frames for panorama construction.The authors report that stereo-camera raw data can connect disjoint scene edges.Source: A. Dataset Detail · pp. 7–8
  • Collision evaluation considers ground, stairs, walls, obstacles, and rough terrain; CFS marks subsequent trajectory points as collided after a collision.CFS and smoothness are higher-is-better, unlike Best of N.Source: A. Dataset Detail · p. 8, Figure 7 · p. 8

V. EVALUATION

The data-collection setup uses SLAM-based tracking and stereo depth sensing without GPS or LiDAR. Evaluation metrics cover collision-free trajectories, smoothness, and best-of-N alignment with potential human preferences.

A. Hardware Setup

  • The collection rig uses T265 SLAM tracking and a D455 stereo camera, avoiding GPS and pure IMU localization for indoor use and to minimize long-term drift.LiDAR was not selected because of variable lighting, especially outdoors.Source: A. Hardware Setup · p. 8

B. Evaluation Metrics

  • The evaluation introduces collision-free score, smoothness, and best of N, including best of 1 for comparison with unimodal predictions.Source: B. Evaluation Metrics · p. 8
  • The metrics are intended to capture physical feasibility, motion quality, and alignment with potential human preferences.An infographic summarizes the metrics.Source: B. Evaluation Metrics · p. 8

1) Collision-free Score:

The section defines three complementary trajectory metrics: collision avoidance, movement smoothness, and coverage of plausible future paths alongside single-prediction accuracy.

1) Collision-free Score:

  • Collision-free Score measures consecutive prediction steps without collision with static obstacles, using the first collided time step as the score.A collision is more than 10 scene points within 16 cm; doors and movable objects are excluded.Source: 1) Collision-free Score · p. 8

2) Smoothness:

  • Smoothness is the reciprocal of the summed mean absolute errors for predicted speed and acceleration against ground truth, so higher values mean smoother trajectories.The metric is intended to align higher scores with more desirable movement continuity.Source: 2) Smoothness · p. 8

3) Best of N & Best of 1:

  • Best of N measures whether a set of predictions covers the ground truth, while Best of 1 measures the accuracy of a single prediction.Best of N is minADE-K; Best of 1 is ADE and supports comparison with single-output models.Source: 3) Best of N & Best of 1 · p. 9

C. Prediction Evaluation

The evaluation examines five-second trajectory predictions and anticipated visual memory, then tests design choices through ablations. Predictions were visually judged diverse and feasible, while the ablations expose trade-offs in speed, collision avoidance, smoothness, and mode coverage.

C. Prediction Evaluation

  • Over a 100-step (5-second) horizon, the model produced five diverse trajectories judged visually feasible for the scene.The BEV visualization shows predicted trajectories in red against the ground truth in green and semantic-colored point clouds.Source: C. Prediction Evaluation · p. 9, Figure 8 · p. 9
  • Predicted visual memory can reveal failure modes: occlusions or view-angle gaps led the model to hallucinate an unexplored hallway extension and veer into it.In the example, the actual hallway gradually shifted from the center of the predicted path.Source: C. Prediction Evaluation · p. 9, Figure 9 · p. 10

D. Ablations

  • Removing semantic segmentation reduced collision avoidance (-5.1), as the model could no longer distinguish penetrating a door from penetrating a wall.Conflicting behavior in the training data tended to make the model ignore geometric constraints in visual memory.Source: D. Ablations · pp. 9–10
  • Removing visual input produced the worst-performing model, with many samples memorizing training trajectories and an incorrect distribution.This variant also had the worst Best of 1 score, as described in the evaluation metrics.Source: D. Ablations · p. 10
  • Training on the full dataset improved all metrics over the pilot dataset, with Collision +0.6, Best of n -0.06, and Smoothness +2.72.The authors attribute the small collision gap partly to point-cloud accuracy and the larger gains to improved generation variance and sample quality.Source: D. Ablations · p. 10
  • Adding multi-head self-attention improved collision avoidance (+4.0), while inference and training speeds remained largely unchanged despite a larger model.The authors suggest the attention layers help relate encoded visual memory to trajectory samples.Source: D. Ablations · p. 10
  • 33 times faster than full 1000-step DDPM, hybrid generation had only slight metric changes (Collision -0.5, Best of n +0.02); 30-step all-DDIM scored worse (Collision -3.9, smoothness -1.58).The hybrid used 20 DDIM steps followed by 10 DDPM refinement steps and generated 50 trajectories per second on one Nvidia A5000 GPU.Source: D. Ablations · p. 10
  • Using only three past trajectory steps changed Collision (-0.4), Smoothness (-0.48), and Best of N (+0.05); the authors say the Markovian assumption largely holds for most scenarios.They suggest this shorter-history variant could speed deployment.Source: D. Ablations · p. 11

E. Comparison with Baselines

The paper compares its diffusion approach with LSTM and CXA Transformer baselines trained on the pilot dataset using the same pretrained VAE. The comparisons highlight differences in collision avoidance, smoothness, and whether predictions are multimodal.

E. Comparison with Baselines

  • The LSTM baseline achieved a smoothness score (+7.05) and avoided obstacles surprisingly well despite having fewer than 1 million parameters.Its deterministic prediction excels at smooth motion, but provides no guarantee of temporal consistency between consecutive time steps.Source: E. Comparison with Baselines · p. 11
  • CXA Transformer’s collision score was significantly lower (-8.9) and its Best of 1 score was worse than the proposed baseline (+0.38), although smoothness was stronger (+1.66).CXA returns a single autoregressive path; the authors suggest its direct use of stereo frames and limited field of view may contribute to the collision result.Source: E. Comparison with Baselines · p. 11
  • The authors identify diffusion’s multimodal prediction and Visual Memory’s situational awareness as advantages over single-path approaches.The comparison states that all three methods use the same pretrained VAE and are trained and evaluated on the pilot dataset.Source: E. Comparison with Baselines · p. 11

VI. CONCLUSION

The paper presents a diffusion-based pipeline conditioned on past trajectory and scene semantics, alongside Visual Memory, a faster sampling scheme, and an egocentric walking dataset. It reports effectiveness through ablations and comparisons with state-of-the-art baselines, while identifying stereo-depth replacement and faster mobile deployment as future work.

VI. CONCLUSION

  • The diffusion-based pipeline predicts future trajectories from past trajectory and scene semantics, and the paper reports effectiveness through ablations and comparisons with state-of-the-art baselines.Visual Memory, a faster sampling scheme, and a diverse egocentric human walking dataset are also contributions.Source: §VI · p. 11
  • Future work includes monocular depth estimation to relax the need for a stereo depth camera.Source: §VI · p. 11
  • Distilling the latent diffusion model to a consistency model is proposed as future work for real-time generation, especially on mobile platforms.Source: §VI · p. 11
Loading 2403.19026v3…