PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer Networks for Trajectory Forecasting

Francesco Giuliari, Irtiza Hasan, Marco Cristani, Fabio Galasso

arXiv:2003.08111v3cs.CV

TL;DR

Trajectory forecasting has largely relied on LSTMs and explicit social or scene interactions, motivating a simpler attention-based alternative. The paper applies TF and BERT to individual pedestrian trajectories, finding strong forecasting performance, multimodal predictions, and robustness to missing observations.

  • Problem

    Recent trajectory forecasting progress has centered on LSTMs and modeling social and scene interactions, prompting a test of attention-based sequence models for pedestrian motion.

  • Method

    The paper models each person’s trajectory separately with Transformer Networks, considering the original TF and the larger bidirectional BERT architecture.

  • Results

    TF outperforms the other techniques on TrajNet, while TFq produces multiple plausible futures and Transformers cope with missing observations.

  • Takeaways & Limitations

    Attention-based Transformers can forecast individual pedestrian trajectories without complex human-human or scene interaction terms while supporting multimodal and incomplete-input predictions.

Abstract

from arXiv · show

Most recent successes on forecasting the people motion are based on LSTM models and all most recent progress has been achieved by modelling the social interaction among people and the people interaction with the scene. We question the use of the LSTM models and propose the novel use of Transformer Networks for trajectory forecasting. This is a fundamental switch from the sequential step-by-step processing of LSTMs to the only-attention-based memory mechanisms of Transformers. In particular, we consider both the original Transformer Network (TF) and the larger Bidirectional Transformer (BERT), state-of-the-art on all natural language processing tasks. Our proposed Transformers predict the trajectories of the individual people in the scene. These are "simple" model because each person is modelled separately without any complex human-human nor scene interaction terms. In particular, the TF model without bells and whistles yields the best score on the largest and most challenging trajectory forecasting benchmark of TrajNet. Additionally, its extension which predicts multiple plausible future trajectories performs on par with more engineered techniques on the 5 datasets of ETH + UCY. Finally, we show that Transformers may deal with missing observations, as it may be the case with real sensor data. Code is available at https://github.com/FGiuliari/Trajectory-Transformer.

I. INTRODUCTION

The paper replaces sequential LSTM trajectory forecasting with attention-based Transformer models that forecast each person independently. Despite omitting social and scene mechanisms, TF achieves strong benchmark performance and supports missing observations.

  • Pedestrian forecasting predicts future motion from past trajectories and supports trackers, early action recognition, surveillance, and automotive systems.
  • LSTMs have driven recent progress by sequentially processing trajectories and adding social pooling to model pedestrian interactions.
  • Transformers replace sequential processing with attention and analyze all available observations while weighting which positions matter for prediction.
  • The proposed models forecast each person independently, sidestepping social and map mechanisms used by many recent approaches.
  • TF and BERT are evaluated on TrajNet against 42 forecasting approaches and on ETH+UCY datasets under a unified benchmark protocol.
  • Transformers can predict from missing observations through attention, whereas the paper states that LSTM cannot do so.

III. THE TRANSFORMER MODEL

The model embeds each person’s observed trajectory, adds timestamps, and uses Transformer architectures to predict future positions. Its attention-based design is parallelizable, handles missing observations, and can produce probabilistic multimodal futures.

  • Each Transformer instance predicts one person’s future motion from that person’s current and prior positions.
  • Observed Cartesian positions are projected into a higher-dimensional embedding, and predicted embeddings are back-projected to Cartesian coordinates.
  • Positional encoding timestamps past and future instants so the Transformer can represent temporal order without recurrent processing.
  • TF uses encoder-decoder attention architecture with six-layer encoder and decoder stacks containing attention, feed-forward, and residual blocks.
  • Unlike LSTM’s sequential training, TF training is parallelizable because observations are processed through attention rather than recurrent unrolling.
  • Positional encoding lets TF neglect missing samples while preserving their relative timestamps, avoiding explicit data filling.
  • Regression directly predicts coordinates, while TFq quantizes motions to provide probabilistic outputs for sampling multiple future trajectories.

B. Encoder-decoder Transformer (TF)

TF encodes the observation sequence into keys and values, then autoregressively decodes future positions using encoder-decoder and self-attention. Its separate encoder memory distinguishes it from LSTM hidden-state accumulation.

  • TF uses six-layer encoder and decoder modules, each combining attention, feed-forward, and residual-connection blocks.
  • The encoder transforms the observation sequence into key and value vectors passed to the decoder.
  • The decoder predicts future track positions autoregressively, comparing each query with encoder keys and values and previous decoder predictions.
  • TF keeps the encoded observation memory separate from the decoded sequence, unlike LSTM, which accumulates both in its hidden state.

C. BERT

BERT uses an encoder-only masking scheme for trajectory prediction, hiding target positions during attention and predicting them from the remaining sequence. The paper also evaluates transfer from a pretrained NLP model.

  • BERT consists only of an encoder and uses masking to hide positions targeted for prediction.
  • During training, BERT learns to predict masked positions; during inference, its outputs provide predictions for those positions.
  • BERT is approximately 2.2 times larger than TF, and the paper tests a version pretrained on an NLP task for trajectory forecasting.
  • The pretrained model converts word-embedding inputs and outputs into trajectory (x,y) positions through substituted linear modules.

D. Implementation details

The TF implementation follows the original Transformer configuration, trains with L2 loss and Adam, and normalizes input speeds using training-set statistics.

  • TF uses dmodel = 512, 6 layers, and 8 attention heads, following the original Transformer configuration.
  • Training uses L2 loss, backpropagation with Adam, five epochs of linear warm-up, decaying learning rate, and dropout of 0.1.
  • Input speeds are normalized by subtracting the training mean and dividing by the training standard deviation.

IV. EXPERIMENTAL EVALUATION

The evaluation tests Transformer trajectory forecasting on TrajNet and ETH+UCY, with ablations examining robustness, observation horizons, and missing data.

  • The study evaluates Transformer networks on the TrajNet Challenge and ETH+UCY datasets.
  • An ablation study compares model robustness with the widely adopted LSTM.
  • The experiments vary the observation horizon and test missing observations caused by frame-rate drops or excessive conditions.

A. The Trajnet Challenge

On the TrajNet benchmark, TF is evaluated against diverse forecasting approaches under a unified protocol and achieves the strongest reported performance without requiring social context.

  • Dataset and metrics: TrajNet evaluates 3161 human trajectories using 8 observed values and 12 forecast values in world-plane coordinates.The benchmark is a multi-scenario dataset assembled from four trajectory families.
  • Dataset and metrics: MAD measures average displacement over forecast steps, FAD measures final-step displacement, and their average ranks approaches.
  • Results: TF achieves the new best TrajNet result, improving both MAD and FAD over REDv3 and reducing total error across 3161 tracks by ∼145 meters.
  • Results: The top four approaches are individual predictors without social context, while social approaches appear at lower ranks.
  • Results: BERT ranks 25th when trained from scratch, and its NLP-pretrained version follows immediately.The authors suggest that BERT may require substantially more training data than currently available for trajectories.

B. The ETH+UCY Benchmark

On ETH+UCY, the proposed individual Transformer predictors are competitive with or better than approaches using social or map information, including under both multi-trajectory and deterministic protocols.

  • Benchmark setup: ETH+UCY contains five videos from four scenes, with 3.2 seconds of observations followed by 4.8 seconds of ground-truth forecasting.Evaluation uses leave-one-dataset-out training and testing.
  • Best-of-20 results: Under best-of-20 evaluation, TFq achieves the second-best performance, only 0.10 behind in MAD and 0.10 in FAD.TFq is the only proposed model in Table II that samples trajectory distributions.
  • Best-of-20 results: TFq performs ahead of or comparably to the strongest social and map-based techniques despite forecasting individuals without those additional mechanisms.
  • Deterministic results: In the single-trajectory deterministic regime, TF outperforms every individual LSTM-based approach and beats Social Attention in FAD by a large margin.The only dataset where an LSTM compares favorably is Zara1, which mostly contains straight-line trajectories.

C. Ablation study and qualitative results

The ablation study compares Transformer and LSTM forecasting across longer horizons and finds that TF retains a consistent advantage as prediction length increases.

  • Study design: The study compares TF and LSTM models and includes experiments on prediction horizons and qualitative behavior.
  • Changing the prediction lengths: Table IV compares MAD and FAD from 12 to 32 prediction steps on ETH-univ and Zara1 sequences excluded from TrajNet training.Both models were trained on the TrajNet training set.
  • Changing the prediction lengths: TF maintains a consistent advantage over LSTM at every tested horizon.
  • Changing the prediction lengths: LSTM MAD increases from 0.78 to 4.13, whereas TF MAD increases from 0.71 to 2.98 as the horizon lengthens.The authors report the LSTM degradation as approximately 25% worse.

2) Missing and noisy data :

The study examines how missing observations affect Transformer-based trajectory forecasting and illustrates how TF and TFq behave on challenging motion patterns. Missing recent frames hurt performance, especially when the current frame is absent, while TFq can represent both concentrated and multimodal future distributions.

  • Missing and noisy data :: Missing coordinate observations in long-term coordinate-based forecasting had not previously been addressed, although missing data is common in tracking and heterogeneous-data forecasting.The Transformer uses remaining observations and positional encoding rather than requiring missing-value imputation; linear interpolation gives slight improvements when available.
  • Missing and noisy data :: 16% degradation, from 0.356 to 0.406 MAD, occurs when six observed frames are dropped but the current frame is retained.In this setting, TF can leverage only two observations: the farthest and closest in time.
  • Qualitative results :: TF follows bending motion more precisely than LSTM in one qualitative case, while capturing dynamics better in another despite an incorrect final direction.The comparisons are shown by superposing predicted LSTM and TF trajectories on TrajNet examples.
  • Qualitative results :: LSTM produces more regular trajectories than TF, which is associated with its unrolling and contrasts with TF’s encoder-plus-decoder architecture.This regularity helps LSTM on Zara1’s mostly straight trajectories but makes it weaker on Hotel and TrajNet relative to TF.
  • Qualitative results :: TFq learns both concentrated and multimodal trajectory distributions, including samples around the ground truth and at least three distinct directional modes.The distributions are illustrated using 100 sampled trajectories on Zara1.

V. CONCLUSIONS

The paper proposes attention-based Transformers for individual pedestrian trajectory prediction instead of LSTM-based and explicitly modeled social or environmental interactions. It reports strong forecasting performance, better long-term behavior, plausible multiple futures, and handling of missing observations, while noting that larger datasets may be needed for longer sequences.

  • V. CONCLUSIONS: Attention-based Transformer Networks are proposed for predicting people’s future trajectories without requiring complex social or environmental interaction terms for best performance.The conclusion frames this as a challenge to the widespread use of LSTMs for modeling people motion.
  • V. CONCLUSIONS: The proposed Transformers achieve the best performance on people forecasting datasets and show better long-term prediction behavior.The conclusion also reports sensible multiple future trajectories and coping with missing input observations.
  • V. CONCLUSIONS: Transformers can predict sensible multiple future trajectories and cope with missing input observations, as may occur with real sensor data.The paper connects these capabilities to trajectory forecasting with longer sequences and imperfect observations.
  • V. CONCLUSIONS: Larger datasets of long-term sequences may be needed to realize the proposed temporal models’ potential.The conclusion presents this as a future direction rather than an established result.
Loading 2003.08111v3…