PaperlayerView exact revision on arXiv

Source-linked AI summary

Stepwise Goal-Driven Networks for Trajectory Prediction

Chuhua Wang, Yuchen Wang, Mingze Xu, David J. Crandall

arXiv:2103.14107v3cs.CV

TL;DR

Trajectory prediction is difficult because agents’ actions depend on environmental and internal factors, while many goal-driven methods represent intention with only one long-term goal. SGNet estimates goals at multiple temporal scales and integrates them into recurrent encoding and decoding, achieving state-of-the-art performance across first-person and bird’s-eye-view benchmarks.

  • Problem

    Many goal-driven trajectory predictors model only a single long-term goal, although intentions may involve goals at multiple time scales.

  • Method

    SGNet predicts successive coarse goals with a stepwise goal estimator and incorporates them into an encoder-decoder framework using attention-based aggregation.

  • Results

    SGNet surpasses or matches state-of-the-art performance across multiple benchmarks covering first- and third-person views and both cars and pedestrians.

  • Takeaways & Limitations

    Modeling both long- and short-term goals provides a supported direction for representing evolving intention in trajectory prediction.

Abstract

from arXiv · show

We propose to predict the future trajectories of observed agents (e.g., pedestrians or vehicles) by estimating and using their goals at multiple time scales. We argue that the goal of a moving agent may change over time, and modeling goals continuously provides more accurate and detailed information for future trajectory estimation. To this end, we present a recurrent network for trajectory prediction, called Stepwise Goal-Driven Network (SGNet). Unlike prior work that models only a single, long-term goal, SGNet estimates and uses goals at multiple temporal scales. In particular, it incorporates an encoder that captures historical information, a stepwise goal estimator that predicts successive goals into the future, and a decoder that predicts future trajectory. We evaluate our model on three first-person traffic datasets (HEV-I, JAAD, and PIE) as well as on three bird's eye view datasets (NuScenes, ETH, and UCY), and show that our model achieves state-of-the-art results on all datasets. Code has been made available at: https://github.com/ChuhuaW/SGNet.pytorch.

I. INTRODUCTION

SGNet addresses trajectory prediction by representing changing intentions with goals at multiple temporal scales rather than only a single distant destination. It combines stepwise goal estimation with encoder-decoder prediction and achieves strong results across viewpoints and agent types.

  • Motivation: Goal-driven methods commonly estimate only a single long-term goal, although intentions may consist of goals at different time scales.The paper motivates stepwise goals as a more comprehensive representation of intention.
  • Approach: SGNet predicts coarse future goals progressively, incorporates them into an encoder and decoder, and uses attention to weight their relative relevance.The encoder uses goals to enrich hidden representations, while the decoder uses them to guide future trajectory prediction.
  • Evaluation: The model is evaluated on first- and third-person datasets involving vehicles and pedestrians, against deterministic and stochastic approaches.The evaluation spans multiple viewpoints and agent categories.
  • Results: SGNet surpasses or matches state-of-the-art performance across multiple benchmarks covering cars, pedestrians, and different viewpoints.The reported comparison includes both first-person and bird’s-eye-view trajectory prediction settings.

III. STEPWISE GOAL-DRIVEN NETWORK (SGNET)

SGNet predicts future positions by generating stepwise goals at successive horizons and using them incrementally in a recurrent encoder-decoder pipeline. Historical trajectories and predicted goals jointly support both current trajectory prediction and future goal generation.

  • Problem formulation: Given an observed trajectory and motion features, SGNet predicts the agent’s future positions over the next ℓd frames.The observed input contains bounding-box information and may include optical flow, velocity, or acceleration.
  • Overview: The stepwise goal estimator predicts future locations from t + 1 to t + ℓd as coarse goals and embeds them incrementally into the decoder.Incremental embedding guides prediction without introducing redundant information.
  • Overview: Predicted goals are fused and fed into the encoder at the next time step to improve current-information encoding and subsequent goal prediction.The encoder therefore uses historical trajectory information together with estimated future states.
  • Overview: The overview distinguishes training, inference, and shared connections while showing encoder time vertically and decoder prediction horizontally.The decoder predicts trajectory from t + 1 through t + ℓd.

B. Encoder

The encoder embeds historical motion and prior aggregated goals into recurrent hidden states, while the goal aggregator compresses multiple stepwise goals using attention. These representations support both trajectory guidance and generation of later goals.

  • B. Encoder: The encoder embeds historical trajectories and optional motion features, concatenates them with aggregated prior goals, and updates a recurrent hidden state.Separate fully connected layers embed trajectory and additional motion features before concatenation.
  • C. Stepwise Goal Estimator (SGE): SGE generates coarse stepwise goals that are conveyed to both encoder and decoder for coarse-to-fine trajectory prediction.For the encoder, goals from the previous time step serve as supplementary features for learning a more discriminative representation.
  • C. Stepwise Goal Estimator (SGE): Goal aggregators combine and compress multiple goals into a single representation by adaptively weighting their relative relevance with attention.The re-weighted goal features are fed into the encoder or decoder.
  • C. Stepwise Goal Estimator (SGE): At decoder time t + i, a subset of stepwise goals provides coarse guidance, while goals before t + i are ignored as redundant historical information.This design supplies future-relevant goals without reusing already encoded earlier goals.

D. Conditional Variational Autoencoder (CVAE)

The CVAE learns future trajectory distributions conditioned on observed trajectories by combining recognition, prior, and generation networks with a latent variable.

  • The CVAE models the distribution of future trajectory Yt conditioned on observed trajectory Xt using a latent variable z.
  • Recognition, prior, and generation networks form the CVAE and are implemented with fully-connected layers.
  • During training, the target encoder processes ground-truth future trajectory Yt to produce hidden state hYt for recognition.
  • The recognition network uses observed and ground-truth hidden states, whereas the prior network uses only the observed hidden state.
  • At test time, the generation network samples z from the prior distribution because the ground-truth future trajectory is unavailable.

E. Decoder

The recurrent decoder uses encoded trajectory information and estimated stepwise goals to generate future positions sequentially.

  • The decoder outputs the future trajectory Yt through a recurrent decoder and trajectory regressor.
  • At each step, the recurrent cell combines the current hidden state with the estimated goal input for the next time step.
  • A single fully-connected trajectory regressor computes the predicted position from the decoder hidden state at each time step.

F. Loss Functions

Training combines trajectory, stepwise-goal, and KL-divergence objectives to supervise prediction quality, goal accuracy, and the CVAE prior.

  • RMSE supervises trajectory prediction from the decoder.
  • The best-of-many approach minimizes distance between the best CVAE trajectory hypothesis and the target to encourage accurate and diverse predictions.
  • The stepwise goal estimator is optimized with RMSE between predicted goal locations and ground-truth locations.
  • KL-divergence loss optimizes the prior network in the CVAE.

IV. EXPERIMENTS

Experiments cover first-person traffic datasets and bird’s-eye-view datasets involving pedestrian and vehicle trajectories, using dataset-specific observation and prediction horizons.

  • First-person datasets: JAAD and PIE provide egocentric pedestrian videos at 30 fps, with 2,800 and 1,835 trajectories, respectively.
  • First-person datasets: HEV-I contains 230 videos divided into 40,000 training and 17,000 test samples.
  • Third-person datasets: ETH and UCY contain 1,536 pedestrians across five data sets and four unique scenes, evaluated with leave-one-out splits.
  • Third-person datasets: NuScenes follows its prediction-challenge splits and settings for autonomous-driving trajectory prediction.
  • Third-person datasets: The experiments use 3.2 seconds of observation to predict 4.8 seconds on ETH and UCY, and 2 seconds to predict 6 seconds on NuScenes.

B. Implementation Details

The model uses GRU-based encoder-decoder components and a stepwise goal estimator, with experiments examining inputs, losses, goal aggregation, and goal connections.

  • Architecture and Inputs: SGNet uses GRUs with 512 hidden units for both encoder and decoder, with benchmark-specific observation lengths and dataset-dependent motion inputs.Inputs include bounding boxes for JAAD and PIE, optical flow for HEV-I, and centroids, velocities, and accelerations for ETH and UCY.
  • Design Trade-offs: GRU, MLP, and convolution-deconvolution implementations of the stepwise goal estimator all achieve state-of-the-art results, while GRU is selected for effectiveness and robustness.The GRU estimator is lightweight, using hidden size 128.
  • Training Strategy: Supervising the stepwise goal estimator improves MSE by 13/68/208 on JAAD and 15/57/181 on PIE for 0.5s, 1.0s, and 1.5s predictions.The result supports using an explicit goal loss to improve goal representations.
  • Design Trade-offs: Removing goal aggregation attention significantly reduces performance, showing that different future goals contribute unequally to prediction.The attention mechanism estimates the relative importance of subsets of future goals.
  • Architecture and Inputs: The full SGNet-ED model feeds predicted stepwise goals into both encoder and decoder and combines goal, BoM trajectory, and KLD losses.The model uses GRU as the stepwise goal estimator backbone.
  • Multimodal Inputs: NuScenes experiments rasterize agents onto maps, extract features with a four-layer CNN, and combine them with the decoder state to integrate map and interaction information.The submitted model ranked third in the NuScenes prediction challenge at the 6th AI Driving Olympics.

E. Comparison with the State-of-the-art

The comparison evaluates SGNet under deterministic and stochastic settings, distinguishing single-trajectory prediction from selecting the best among multiple samples.

  • Evaluation Settings: Deterministic prediction returns one trajectory, whereas stochastic prediction reports the best-performing sample among K generated trajectories.This comparison covers both single-output and multimodal prediction settings.

1) Deterministic Results on First-person Benchmarks:

SGNet achieves strong deterministic results on first-person pedestrian and vehicle benchmarks, with improvements reported across prediction horizons and evaluation metrics.

  • JAAD and PIE: 13%, 15%, and 15% MSE reductions on JAAD occur at 0.5s, 1.0s, and 1.5s prediction horizons compared with.The reported gains become especially relevant as the prediction horizon increases.
  • JAAD and PIE: 3%, 9%, and 11% MSE reductions on PIE occur at 0.5s, 1.0s, and 1.5s prediction horizons.The paper reports similar improvements on additional evaluation metrics.
  • Qualitative Results: Qualitative comparisons visualize observed trajectories in yellow, ground truth futures in red, and SGNet-ED predictions in cyan.The figure presents qualitative deterministic trajectory predictions.
  • Third-person Benchmarks: SGNet-ED outperforms the state of the art by more than 10% on average in ADE and FDE for ETH and UCY deterministic results.ADE measures whole-trajectory accuracy, while FDE measures endpoint accuracy.

4) Stochastic Results on Third-person Benchmarks:

SGNet’s stochastic predictions remain close to observed third-person trajectories across several pedestrian scenes, while its performance varies with scene complexity. The model’s stepwise goals provide temporal information for longer-horizon prediction, but missing interaction and context limit performance in crowded or ambiguous cases.

  • Benchmark results: SGNet-ED outperforms the current state-of-the-art stochastic model by 5% on average across ETH, ZARA1, and ZARA2.It achieves comparable results on HOTEL and UNIV under the ETH-UCY leave-one-out protocol with K = 20.
  • Benchmark results: The model shows larger improvements as prediction length increases, suggesting stepwise goals provide useful temporal information for future locations and object magnitudes.This pattern is reported for first- and third-person datasets.
  • Qualitative results: Across stochastic examples from JAAD, PIE, ETH, and UCY, most of the 20 generated trajectories remain close to the ground-truth trajectory and bounding box.The examples include anticipating pedestrians crossing streets in JAAD and PIE.
  • Failure cases: The model fails when predicting a pedestrian descending stairs because it lacks sufficient context and interaction information.The failure case highlights a boundary for scenes requiring pedestrian interaction and richer contextual cues.
Loading 2103.14107v3…