PaperlayerView exact revision on arXiv

Source-linked AI summary

Stepwise Goal-Driven Networks for Trajectory Prediction

Chuhua Wang, Yuchen Wang, Mingze Xu, David J. Crandall

arXiv:2103.14107v3cs.CV

TL;DR

Trajectory prediction is difficult because agents’ actions depend on environmental and internal factors, while many goal-driven methods represent intention with only one long-term goal. SGNet estimates goals at multiple temporal scales and integrates them into recurrent encoding and decoding, achieving state-of-the-art performance across first-person and bird’s-eye-view benchmarks.

  • Problem

    Many goal-driven trajectory predictors model only a single long-term goal, although intentions may involve goals at multiple time scales.

    Source: §I · p. 1

  • Method

    SGNet predicts successive coarse goals with a stepwise goal estimator and incorporates them into an encoder-decoder framework using attention-based aggregation.

    Source: §I · pp. 1–2, C. Stepwise Goal Estimator (SGE) · p. 3

  • Results

    SGNet surpasses or matches state-of-the-art performance across multiple benchmarks covering first- and third-person views and both cars and pedestrians.

    Source: §I · p. 2, §V · pp. 7–8

  • Takeaways & Limitations

    Modeling both long- and short-term goals provides a supported direction for representing evolving intention in trajectory prediction.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

We propose to predict the future trajectories of observed agents (e.g., pedestrians or vehicles) by estimating and using their goals at multiple time scales. We argue that the goal of a moving agent may change over time, and modeling goals continuously provides more accurate and detailed information for future trajectory estimation. To this end, we present a recurrent network for trajectory prediction, called Stepwise Goal-Driven Network (SGNet). Unlike prior work that models only a single, long-term goal, SGNet estimates and uses goals at multiple temporal scales. In particular, it incorporates an encoder that captures historical information, a stepwise goal estimator that predicts successive goals into the future, and a decoder that predicts future trajectory. We evaluate our model on three first-person traffic datasets (HEV-I, JAAD, and PIE) as well as on three bird's eye view datasets (NuScenes, ETH, and UCY), and show that our model achieves state-of-the-art results on all datasets. Code has been made available at: https://github.com/ChuhuaW/SGNet.pytorch.

I. INTRODUCTION

SGNet addresses trajectory prediction by representing changing intentions with goals at multiple temporal scales rather than only a single distant destination. It combines stepwise goal estimation with encoder-decoder prediction and achieves strong results across viewpoints and agent types.

Motivation

  • Goal-driven methods commonly estimate only a single long-term goal, although intentions may consist of goals at different time scales.The paper motivates stepwise goals as a more comprehensive representation of intention.Source: §I · p. 1, §II · p. 2

Approach

  • SGNet predicts coarse future goals progressively, incorporates them into an encoder and decoder, and uses attention to weight their relative relevance.The encoder uses goals to enrich hidden representations, while the decoder uses them to guide future trajectory prediction.Source: §I · pp. 1–2

Evaluation

  • The model is evaluated on first- and third-person datasets involving vehicles and pedestrians, against deterministic and stochastic approaches.The evaluation spans multiple viewpoints and agent categories.Source: §I · p. 2

Results

  • SGNet surpasses or matches state-of-the-art performance across multiple benchmarks covering cars, pedestrians, and different viewpoints.The reported comparison includes both first-person and bird’s-eye-view trajectory prediction settings.Source: §I · p. 2

III. STEPWISE GOAL-DRIVEN NETWORK (SGNET)

SGNet predicts future positions by generating stepwise goals at successive horizons and using them incrementally in a recurrent encoder-decoder pipeline. Historical trajectories and predicted goals jointly support both current trajectory prediction and future goal generation.

Problem formulation

  • Given an observed trajectory and motion features, SGNet predicts the agent’s future positions over the next ℓd frames.The observed input contains bounding-box information and may include optical flow, velocity, or acceleration.Source: §III · p. 2

Overview

  • The stepwise goal estimator predicts future locations from t + 1 to t + ℓd as coarse goals and embeds them incrementally into the decoder.Incremental embedding guides prediction without introducing redundant information.Source: A. Overview · p. 2
  • Predicted goals are fused and fed into the encoder at the next time step to improve current-information encoding and subsequent goal prediction.The encoder therefore uses historical trajectory information together with estimated future states.Source: A. Overview · p. 2
  • The overview distinguishes training, inference, and shared connections while showing encoder time vertically and decoder prediction horizontally.The decoder predicts trajectory from t + 1 through t + ℓd.Source: Figure 2 · p. 3

B. Encoder

The encoder embeds historical motion and prior aggregated goals into recurrent hidden states, while the goal aggregator compresses multiple stepwise goals using attention. These representations support both trajectory guidance and generation of later goals.

B. Encoder

  • The encoder embeds historical trajectories and optional motion features, concatenates them with aggregated prior goals, and updates a recurrent hidden state.Separate fully connected layers embed trajectory and additional motion features before concatenation.Source: B. Encoder · p. 3

C. Stepwise Goal Estimator (SGE)

  • SGE generates coarse stepwise goals that are conveyed to both encoder and decoder for coarse-to-fine trajectory prediction.For the encoder, goals from the previous time step serve as supplementary features for learning a more discriminative representation.Source: C. Stepwise Goal Estimator (SGE) · p. 3
  • Goal aggregators combine and compress multiple goals into a single representation by adaptively weighting their relative relevance with attention.The re-weighted goal features are fed into the encoder or decoder.Source: C. Stepwise Goal Estimator (SGE) · p. 3, Figure 3 · p. 3
  • At decoder time t + i, a subset of stepwise goals provides coarse guidance, while goals before t + i are ignored as redundant historical information.This design supplies future-relevant goals without reusing already encoded earlier goals.Source: C. Stepwise Goal Estimator (SGE) · p. 4

D. Conditional Variational Autoencoder (CVAE)

The CVAE learns future trajectory distributions conditioned on observed trajectories by combining recognition, prior, and generation networks with a latent variable.

E. Decoder

The recurrent decoder uses encoded trajectory information and estimated stepwise goals to generate future positions sequentially.

  • The decoder outputs the future trajectory Yt through a recurrent decoder and trajectory regressor.Source: E. Decoder · p. 4
  • At each step, the recurrent cell combines the current hidden state with the estimated goal input for the next time step.Source: E. Decoder · p. 4
  • A single fully-connected trajectory regressor computes the predicted position from the decoder hidden state at each time step.Source: E. Decoder · p. 4

F. Loss Functions

Training combines trajectory, stepwise-goal, and KL-divergence objectives to supervise prediction quality, goal accuracy, and the CVAE prior.

  • RMSE supervises trajectory prediction from the decoder.Source: F. Loss Functions · p. 4
  • The best-of-many approach minimizes distance between the best CVAE trajectory hypothesis and the target to encourage accurate and diverse predictions.Source: F. Loss Functions · p. 4
  • The stepwise goal estimator is optimized with RMSE between predicted goal locations and ground-truth locations.Source: F. Loss Functions · p. 4
  • KL-divergence loss optimizes the prior network in the CVAE.Source: F. Loss Functions · p. 4

IV. EXPERIMENTS

Experiments cover first-person traffic datasets and bird’s-eye-view datasets involving pedestrian and vehicle trajectories, using dataset-specific observation and prediction horizons.

First-person datasets

  • JAAD and PIE provide egocentric pedestrian videos at 30 fps, with 2,800 and 1,835 trajectories, respectively.Source: A. Datasets · p. 4
  • HEV-I contains 230 videos divided into 40,000 training and 17,000 test samples.Source: A. Datasets · p. 4

Third-person datasets

  • ETH and UCY contain 1,536 pedestrians across five data sets and four unique scenes, evaluated with leave-one-out splits.Source: A. Datasets · p. 4
  • NuScenes follows its prediction-challenge splits and settings for autonomous-driving trajectory prediction.Source: A. Datasets · p. 4
  • The experiments use 3.2 seconds of observation to predict 4.8 seconds on ETH and UCY, and 2 seconds to predict 6 seconds on NuScenes.Source: A. Datasets · p. 4

B. Implementation Details

The model uses GRU-based encoder-decoder components and a stepwise goal estimator, with experiments examining inputs, losses, goal aggregation, and goal connections.

Architecture and Inputs

  • SGNet uses GRUs with 512 hidden units for both encoder and decoder, with benchmark-specific observation lengths and dataset-dependent motion inputs.Inputs include bounding boxes for JAAD and PIE, optical flow for HEV-I, and centroids, velocities, and accelerations for ETH and UCY.Source: B. Implementation Details · p. 4

Design Trade-offs

  • GRU, MLP, and convolution-deconvolution implementations of the stepwise goal estimator all achieve state-of-the-art results, while GRU is selected for effectiveness and robustness.The GRU estimator is lightweight, using hidden size 128.Source: D. Exploration Study · p. 5

Training Strategy

  • Supervising the stepwise goal estimator improves MSE by 13/68/208 on JAAD and 15/57/181 on PIE for 0.5s, 1.0s, and 1.5s predictions.The result supports using an explicit goal loss to improve goal representations.Source: D. Exploration Study · p. 5

Design Trade-offs

  • Removing goal aggregation attention significantly reduces performance, showing that different future goals contribute unequally to prediction.The attention mechanism estimates the relative importance of subsets of future goals.Source: D. Exploration Study · p. 6

Architecture and Inputs

  • The full SGNet-ED model feeds predicted stepwise goals into both encoder and decoder and combines goal, BoM trajectory, and KLD losses.The model uses GRU as the stepwise goal estimator backbone.Source: D. Exploration Study · p. 6

Multimodal Inputs

  • NuScenes experiments rasterize agents onto maps, extract features with a four-layer CNN, and combine them with the decoder state to integrate map and interaction information.The submitted model ranked third in the NuScenes prediction challenge at the 6th AI Driving Olympics.Source: D. Exploration Study · p. 6, Table VI · p. 6

E. Comparison with the State-of-the-art

The comparison evaluates SGNet under deterministic and stochastic settings, distinguishing single-trajectory prediction from selecting the best among multiple samples.

Evaluation Settings

  • Deterministic prediction returns one trajectory, whereas stochastic prediction reports the best-performing sample among K generated trajectories.This comparison covers both single-output and multimodal prediction settings.Source: E. Comparison with the State-of-the-art · p. 6

1) Deterministic Results on First-person Benchmarks:

SGNet achieves strong deterministic results on first-person pedestrian and vehicle benchmarks, with improvements reported across prediction horizons and evaluation metrics.

JAAD and PIE

  • 13%, 15%, and 15% MSE reductions on JAAD occur at 0.5s, 1.0s, and 1.5s prediction horizons compared with.The reported gains become especially relevant as the prediction horizon increases.Source: 1) Deterministic Results on… · p. 6
  • 3%, 9%, and 11% MSE reductions on PIE occur at 0.5s, 1.0s, and 1.5s prediction horizons.The paper reports similar improvements on additional evaluation metrics.Source: 1) Deterministic Results on… · p. 6

Qualitative Results

  • Qualitative comparisons visualize observed trajectories in yellow, ground truth futures in red, and SGNet-ED predictions in cyan.The figure presents qualitative deterministic trajectory predictions.Source: Figure 4 · p. 7

Third-person Benchmarks

  • SGNet-ED outperforms the state of the art by more than 10% on average in ADE and FDE for ETH and UCY deterministic results.ADE measures whole-trajectory accuracy, while FDE measures endpoint accuracy.Source: 2) Deterministic Results on… · p. 7

4) Stochastic Results on Third-person Benchmarks:

SGNet’s stochastic predictions remain close to observed third-person trajectories across several pedestrian scenes, while its performance varies with scene complexity. The model’s stepwise goals provide temporal information for longer-horizon prediction, but missing interaction and context limit performance in crowded or ambiguous cases.

Benchmark results

  • SGNet-ED outperforms the current state-of-the-art stochastic model by 5% on average across ETH, ZARA1, and ZARA2.It achieves comparable results on HOTEL and UNIV under the ETH-UCY leave-one-out protocol with K = 20.Source: 4) Stochastic Results on Third-person… · p. 7
  • The model shows larger improvements as prediction length increases, suggesting stepwise goals provide useful temporal information for future locations and object magnitudes.This pattern is reported for first- and third-person datasets.Source: 4) Stochastic Results on Third-person… · p. 7

Qualitative results

  • Across stochastic examples from JAAD, PIE, ETH, and UCY, most of the 20 generated trajectories remain close to the ground-truth trajectory and bounding box.The examples include anticipating pedestrians crossing streets in JAAD and PIE.Source: F. Qualitative Results · p. 7

Failure cases

  • The model fails when predicting a pedestrian descending stairs because it lacks sufficient context and interaction information.The failure case highlights a boundary for scenes requiring pedestrian interaction and richer contextual cues.Source: G. Failure Cases · p. 7

Related papers

Loading 2103.14107v3…