PaperlayerView exact revision on arXiv

Source-linked AI summary

From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting

Karttikeya Mangalam, Yang An, Harshayu Girase, Jitendra Malik

arXiv:2012.01526v1cs.CVcs.AIcs.RO

TL;DR

Human trajectory forecasting must represent uncertainty from unknown goals and unpredictable path choices, particularly over long horizons. The paper factorizes these sources into goal and path multimodality, implements them in Y-net, and evaluates a forecasting setting extending up to a minute. Y-net improves prior state-of-the-art results in both short- and long-horizon benchmarks.

  • Problem

    Future human trajectories are multimodal because long-term goals may be unknown to the model and path variation can arise from environmental factors and random decisions.

  • Method

    Y-net hierarchically models epistemic uncertainty through long-term goals and aleatoric uncertainty through waypoints and paths, using explicit probability maps.

  • Results

    Y-net improves previous state-of-the-art performance on short-horizon Stanford Drone and ETH/UCY benchmarks and exceeds it by over 77.1% and 55.9% on the long-horizon Stanford Drone and Intersection Drone datasets, respectively.

  • Takeaways & Limitations

    The results support factorized stochasticity modeling for diverse, scene-compliant trajectory prediction across short and long horizons.

  • Takeaways & Limitations

    The waypoint model assumes a waypoint near the straight-line segment between the sampled goal and past trajectory, relaxed with a Gaussian prior and fixed scaling choices.

Abstract

from arXiv · show

Human trajectory forecasting is an inherently multi-modal problem. Uncertainty in future trajectories stems from two sources: (a) sources that are known to the agent but unknown to the model, such as long term goals and (b)sources that are unknown to both the agent & the model, such as intent of other agents & irreducible randomness indecisions. We propose to factorize this uncertainty into its epistemic & aleatoric sources. We model the epistemic un-certainty through multimodality in long term goals and the aleatoric uncertainty through multimodality in waypoints& paths. To exemplify this dichotomy, we also propose a novel long term trajectory forecasting setting, with prediction horizons upto a minute, an order of magnitude longer than prior works. Finally, we presentY-net, a scene com-pliant trajectory forecasting network that exploits the pro-posed epistemic & aleatoric structure for diverse trajectory predictions across long prediction horizons.Y-net significantly improves previous state-of-the-art performance on both (a) The well studied short prediction horizon settings on the Stanford Drone & ETH/UCY datasets and (b) The proposed long prediction horizon setting on the re-purposed Stanford Drone & Intersection Drone datasets.

1. Introduction

Human trajectory forecasting is difficult because people are goal-directed yet future motion remains stochastic, especially over long horizons. The paper factorizes this uncertainty into latent goals and stochastic paths, introducing Y-net and a minute-long forecasting setting.

  • Motivation: Human motion differs from physical-object forecasting because people act to achieve desired outcomes.This makes anticipating motion important for humans, robots, and self-driving vehicles.
  • Motivation: Future trajectories remain stochastic even when past motion, scene semantics, and long-term goals are known.The paper attributes this to environmental factors and other random decisions.
  • Approach: The proposed factorization models long-term goals as epistemic uncertainty and paths leading to those goals as aleatoric uncertainty.The hierarchy first estimates goals, then models path variability conditioned on those estimates.
  • Contributions: Y-net introduces a long-term prediction setting extending up to a minute, about an order of magnitude beyond previous literature.It also evaluates prior short-horizon models and simple baselines in this setting.
  • Contributions: 26.9% and 5.6% ADE improvements, plus 34.0% and 51.9% FDE improvements, are reported on Stanford Drone and ETH/UCY, respectively.The passage reports these as improvements over previous approaches in short-horizon settings.

2. Related Works

Prior trajectory-forecasting work varies in its multimodality formulation, input signals, and prediction outputs. Generative methods provide diversity but limited mode coverage and control, motivating explicit probability-map approaches such as Y-net.

  • Research landscape: Prior methods use varied inputs, including motion history, pose, RGB scenes, semantic cues, location, gaze, and nearby vehicles.The literature is organized around input signals, multimodality formulations, and prediction outputs.
  • Unimodal forecasting: Early approaches modeled future trajectories as unimodal or deterministic paths using social forces, recurrent pooling, attention, pose, gaze, or ego-motion.These methods capture interactions or contextual cues but do not represent the full multimodal formulation developed here.
  • Generative modeling: Latent-variable approaches such as DESIRE and CF-VAE represent multimodality through variational or flow-based generative models.These methods define priors or latent sampling mechanisms for diverse sequence predictions.
  • Generative modeling: Social GAN and SoPhie incorporate multimodality through adversarial losses and attention to agent–environment interactions.They represent a separate generative line of trajectory-forecasting work.
  • Motivation for Y-net: Generative approaches may produce diverse trajectories, but critical-mode coverage and control over direction or sample count are not guaranteed.Y-net instead estimates explicit probability maps that can incorporate spatial constraints downstream.

3. Proposed Method

Y-net factorizes multimodal forecasting into destination and path variability, using scene-aligned representations, U-net sub-networks, explicit heatmaps, and sampling procedures to generate diverse futures.

  • Problem formulation: Given an RGB scene and past positions, the model predicts multiple trajectories by separating final-destination modes from path modes.It produces Ke destination predictions and Ka path predictions per destination; longer horizons permit Ka > 1.
  • Trajectory on Scene Heatmap Representation: The model represents scene semantics and motion history in aligned spatial heatmaps before encoding them with Ue.Semantic segmentation produces scene affordance classes, while past positions become trajectory heatmap channels.
  • Model architecture: Y-net uses Ue, Ug, and Ut, with Ug estimating goal and waypoint heatmaps and Ut predicting paths conditioned on sampled goals and waypoints.All three sub-networks follow U-net-style encoder or decoder structures.
  • Goal and waypoint prediction: Ug outputs Nw + 1 spatial heatmaps representing the final goal and intermediate waypoint distributions.The goal is represented as P̂(u_np+nf), while each waypoint is represented as P̂(u_wi).
  • Non-parametric distribution sampling: Softargmax robustly estimates positions from noisy probability maps, while categorical sampling supports multiple position samples.The paper notes that naive argmax is not robust for noisy distributions.
  • Test-Time Sampling Trick: TTST draws 10,000 candidate goal points, suppresses low-probability outliers, and clusters them into evaluation samples based on Ke.Temperature adjusts the diversity–precision tradeoff before sampling.

3.3. Loss Function

The networks are trained end to end against Gaussian heatmap targets using weighted binary cross-entropy losses for goals, waypoints, and trajectories.

  • Loss Function: Ue, Ug, and Ut are trained jointly with weighted binary cross-entropy losses on predicted goal, waypoint, and trajectory distributions.The ground-truth future is represented as a Gaussian heatmap centered on observed future points with predetermined variance σH.

4.1. Datasets

The study evaluates trajectory forecasting on SDD, InD, and ETH/UCY using separate short- and long-horizon settings, with testing performed on unseen scenes or locations.

  • Stanford Drone Dataset: SDD contains more than 11,000 pedestrians across 20 top-down campus scenes, with over 40,000 agent-scene interactions.The short-term setup follows TrajNet preprocessing at 2.5 FPS.
  • Stanford Drone Dataset: The long-term SDD setup downsamples raw 30 FPS data to 1 FPS, using 5 seconds of history to predict up to one minute ahead.Evaluation uses the standard scene split, with test scenes unseen during training.
  • Intersection Drone Dataset: InD provides over 10 hours of measurements from four urban intersections, with location ID 4 reserved for testing on an unseen location.Its processing follows the long-term SDD procedure.
  • ETH/UCY datasets: ETH/UCY comprises five world-coordinate scenes evaluated with leave-one-out validation for short-term forecasting.The setup uses 3.2 seconds of observed history and predicts 4.8 seconds into the future at 2.5 FPS.
  • ETH/UCY datasets: ETH/UCY coordinates are projected from meters into pixel space using provided or constructed homographies, then converted back for comparison.The scene affordances are represented with two classes: road and not road.

4.2. Implementation Details

Implementation combines semantic scene segmentation with Y-net’s goal, waypoint, and trajectory subnetworks, while training and evaluation use scene-aware preprocessing and augmentation.

  • Segmentation Model: A U-net with a ResNet101 backbone is pretrained for semantic segmentation to incorporate scene constraints and agent interactions.The encoder is ImageNet-pretrained, while the decoder and segmentation head are randomly initialized; the InD model reuses the SDD model.
  • Evaluation setup: Experiments report short-horizon SDD and ETH/UCY results alongside long-horizon evaluation summarized in Tables 1, 2, and 3.The cited table captions specify short-horizon SDD and ETH/UCY comparisons and a 30-second long-horizon setting.
  • Preprocessing and augmentation: Spatial flipping and 90° rotations augment all scene images and trajectories, increasing the training data by a factor of eight.ETH/UCY inputs are padded for divisibility by 32 and predictions are upsampled before comparison.
  • Y-net: Y-net trains goal and waypoint distributions as auxiliary tasks, then uses sampled goals and waypoints during inference.Its trajectory subnetwork is trained with ground-truth goals and waypoints to predict paths leading toward goals through waypoints.

4.3. Metrics

Performance is measured with Average Displacement Error and Final Displacement Error, using the minimum error across multiple predicted futures.

  • Metrics: Average Displacement Error (ADE) is the averaged ℓ2 distance between predicted and ground-truth positions over the entire future trajectory.It measures trajectory-wide positional accuracy.
  • Metrics: Final Displacement Error (FDE) is the ℓ2 distance between the predicted and ground-truth final points.For multiple predictions, the reported error is the minimum across predicted futures.

4.4. Baseline models

The baselines span adversarial, recurrent graph, flow-based variational, inverse-reinforcement-learning, goal-conditioned, and augmentation-based approaches to multimodal trajectory forecasting.

  • Generative models: Social GAN and SoPhie use GAN-based forecasting, with SoPhie adding attention modules for other agents and scene context.Both methods model multimodal trajectories autoregressively or with contextual attention.
  • Latent-variable models: CF-VAE models future uncertainty with a conditional normalizing-flow VAE without disentangling its underlying factors.CGNS instead learns feasible future-trajectory regions through variational divergence minimization in latent space.
  • Planning and augmentation models: P2TIRL and DESIRE use inverse reinforcement learning, while SimAug adds adversarially generated 3D multiview data to adapt forecasting to novel viewpoints.These baselines represent planning-based and data-augmentation-based alternatives.
  • Goal-conditioned models: PECNet uses goal conditioning but does not model multimodality in the path to the goal.It is identified as the prior state of the art on short-term SDD forecasting.
  • Goal-conditioned models: TNT improves PECNet for K = 5 samples on SDD and is the prior state of the art in that setting.This baseline is used for comparison with Y-net’s short-term SDD results.
  • Interaction-aware models: Trajectron++ uses a recurrent graph-based model incorporating other moving agents and scene information, and previously led ETH/UCY short-term forecasting.Its prior state-of-the-art status is specific to the ETH/UCY benchmark.

4.5. Short Term Forecasting Results

Y-net improves short-term trajectory forecasting on SDD and ETH/UCY, outperforming prior state-of-the-art methods under the reported evaluation settings.

  • Ablation Setting: Table 4 evaluates CWS and TTST on the 30-second long-horizon setting with Nw = 1, Ke = 20, and Ka = 1, using pixel errors where lower is better.
  • Stanford Drone Results: At K = 5, Y-net achieves 11.49 ADE and 20.23 FDE on SDD, outperforming TNT.
  • ETH & UCY Results: On ETH/UCY, Y-net achieves 0.18 ADE and 0.27 FDE, improving over Trajectron++ by about 5.6% ADE and 51.9% FDE.The comparison uses Ke = 20 and Ka = 1.

4.6. Long Term Forecasting Results

Y-net is evaluated for long-horizon forecasting up to one minute, where factorized goal and path multimodality improves accuracy, diversity, and robustness relative to baselines.

  • Forecasting Setting: The long-term setting extends prediction horizons to one minute, about ten times longer than prior works, and benchmarks PECNet, Social GAN, and recurrent PECNet baselines.The setting is evaluated on SDD and InD, with models retrained for each long horizon where applicable.
  • Forecasting Results: At tf = 30 seconds on SDD, Y-net achieves 47.94 ADE and 66.72 FDE, improving over PECNet by over 50%.The evaluation conditions on 5 seconds of past motion and uses Ke = 20 with one intermediate waypoint.
  • Forecasting Results: On InD, Y-net improves ADE from 20.25 to 14.99 and FDE from 32.95 to 21.13 relative to PECNet.
  • Conditioned Waypoint Sampling: Removing Conditioned Waypoint Sampling decreases ADE by 24.3% without affecting goal sampling or FDE.
  • Test-Time Sampling Trick: TTST reduces ADE by 9.1% on SDD and 18.5% on InD, while reducing FDE by 30.4% and 35.0%, respectively.
  • Varying Prediction Horizon: The ADE gap between Y-net and PECNet grows as the prediction horizon increases from 5 to 60 seconds.The comparison includes PECNet retrained for each horizon and PECNet recurrently evaluated from a 5-second model.
  • Varying Ka: Increasing Ka consistently improves ADE across datasets and fixed Ke settings, while PECNet remains significantly worse even with 20 times more samples.This supports the reported benefit of factorizing goal multimodality from path multimodality.
  • Qualitative Results: Qualitative results show diverse, scene-compliant trajectories with multimodality in both future goals and paths over a 30-second horizon.The visualizations use SDD test scenes and include heatmaps and sampled trajectories.

5. Conclusion

The paper presents Y-net as a scene-compliant network that factorizes goal and path multimodality, improving short- and long-term trajectory forecasting performance.

  • Conclusion: Y-net uses a U-net structure to model probability heatmaps for epistemic and aleatoric uncertainties through factorized goal and path multimodalities.
  • Conclusion: In short-term settings, Y-net improves previous state-of-the-art performance by 34.0% on SDD and 51.9% on ETH/UCY.
  • Conclusion: In the proposed long-term setting, Y-net exceeds previous state-of-the-art by over 77.1% on SDD and 55.9% on InD.The setting uses prediction horizons up to one minute.
Loading 2012.01526v1…