PaperlayerView exact revision on arXiv

Source-linked AI summary

LG-Traj: LLM Guided Pedestrian Trajectory Prediction

Pranav Singh Chib, Pravendra Singh

arXiv:2403.08032v1cs.CVcs.AI

TL;DR

Pedestrian trajectory prediction must account for varied motion patterns, but existing methods leave room to leverage motion cues more fully. LG-Traj combines LLM-generated past cues, clustered future cues, observed trajectories, and social interactions; it outperforms compared methods on ETH-UCY and SDD.

  • Problem

    Pedestrian futures vary, and existing methods need further exploration to leverage motion cues effectively while accounting for motion patterns and neighbouring pedestrians.

    Source: §1 · pp. 1–2

  • Method

    LG-Traj uses LLM-generated cues from past trajectories, Gaussian-mixture clusters of training futures, SVD-augmented observations, a motion encoder, and a social decoder.

    Source: §1 · pp. 2–3

  • Results

    LG-Traj outperforms compared methods on ETH-UCY, improving average ADE/FDE by 10%/3% relative to SMEMO, and on SDD with ADE/FDE of 7.80/12.79.

    Source: Quantitative Results · pp. 11, 13

  • Takeaways & Limitations

    The experiments illustrate the feasibility of integrating LLM-derived motion cues into pedestrian trajectory prediction.

    Source: §5 · p. 14

Abstract

from arXiv · show

Accurate pedestrian trajectory prediction is crucial for various applications, and it requires a deep understanding of pedestrian motion patterns in dynamic environments. However, existing pedestrian trajectory prediction methods still need more exploration to fully leverage these motion patterns. This paper investigates the possibilities of using Large Language Models (LLMs) to improve pedestrian trajectory prediction tasks by inducing motion cues. We introduce LG-Traj, a novel approach incorporating LLMs to generate motion cues present in pedestrian past/observed trajectories. Our approach also incorporates motion cues present in pedestrian future trajectories by clustering future trajectories of training data using a mixture of Gaussians. These motion cues, along with pedestrian coordinates, facilitate a better understanding of the underlying representation. Furthermore, we utilize singular value decomposition to augment the observed trajectories, incorporating them into the model learning process to further enhance representation learning. Our method employs a transformer-based architecture comprising a motion encoder to model motion patterns and a social decoder to capture social interactions among pedestrians. We demonstrate the effectiveness of our approach on popular pedestrian trajectory prediction benchmarks, namely ETH-UCY and SDD, and present various ablation experiments to validate our approach.

1 Introduction

Pedestrian prediction must account for varied motion patterns and interactions with nearby pedestrians. LG-Traj combines LLM-generated past motion cues, clustered future motion cues, observed trajectories, and social decoding to predict future trajectories.

Motivation

  • Pedestrian futures can vary widely, and prediction must account for both individual motion characteristics and neighbouring pedestrians’ social interactions.The introduction identifies learning varied spatio-temporal trajectory representations as a major challenge.Source: §1 · p. 1

Contributions

  • LG-Traj combines LLM-generated past motion cues and observed trajectories with future-trajectory clusters to model motion patterns, then uses neighbour interactions to predict socially plausible futures.Future cues come from clustering training-set future trajectories with a mixture of Gaussians; observed trajectories may also be augmented using SVD.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The model uses a transformer motion encoder for motion patterns and a social decoder that combines its embedding with neighbour information.The overview also shows positional encoding added to motion-cue, observed-trajectory, and cluster embeddings before motion encoding.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 Related Work

Prior trajectory-prediction work models temporal patterns, uncertain futures, and pedestrian interactions using sequence, generative, transformer, and graph-based approaches. The paper positions LLM-generated motion cues as a relatively unexplored direction for pedestrian trajectory prediction.

2.1 Trajectory Prediction

  • Trajectory-prediction methods include recurrent and generative models, transformers for long-range relationships, and graph-based approaches for pedestrian interactions.The related-work discussion notes possible gradient vanishing or explosion in RNN-based models and the use of generative models to capture future variation.Source: §2.1 · p. 3

2.2 Large Language Model

  • Although LLMs have been used in autonomous driving and robot navigation, their use for pedestrian trajectory prediction has seen little effort.LG-Traj uses LLM-generated motion cues to train a trajectory-prediction model.Source: §2.2 · pp. 3–4

3 Method

The method predicts future pedestrian trajectories from past observations. It also augments observed trajectories through an SVD-based rank-k approximation intended to preserve critical information with minimal loss.

3.1 Problem Definition

  • Trajectory prediction generates future trajectories that approximate ground truth, given past observations of pedestrians.The problem definition denotes the observation length as T_ob, the prediction length as T_pred, and the number of pedestrians as N.Source: §3.1 · p. 4

3.2 Trajectories Augmentation

  • The method stacks training-batch observations into a trajectory matrix and applies SVD followed by rank-k approximation to obtain augmented trajectories.The approximation retains the first k singular values and corresponding singular vectors, preserving critical information with minimal loss; the paper presents formulations using original observations for simplicity while applying them equally to augmented observations.Source: §3.2 · pp. 4–5
  • Fig. 2 illustrates a mixture of Gaussians in which each Gaussian represents a diverse trajectory cluster.The method uses this mixture to represent future-motion cues, as described in the clustering section.Source: Figure 2 · p. 5

3.3 Clustering using Mixture of Gaussians

The method represents diverse future motion cues by clustering training trajectories with a Gaussian mixture and assigns soft probabilities to predicted trajectories based on their proximity to ground truth.

Gaussian-mixture clustering

  • Training-set future trajectories are translated to a common origin, rotated to the positive zero-degree direction, and clustered by similar motion behavior into Gaussian components.The clusters represent varied future-motion cues, including linear and curved motion.Source: §3.3 · p. 5
  • The mixture assigns learnable weights to Gaussian clusters, whose means and variances characterize their components.The Gaussian components represent diverse motion cues in future trajectories.Source: §3.3 · p. 5

Motion-cue examples

  • The LLM examples show motion cues for linear, curved, and standing-still trajectories inferred from input coordinates.The examples are presented as cases where the LLM correctly identifies the underlying pattern.Source: Figure 3 · p. 6

Soft Trajectory Probability

  • The soft probability for each pedestrian is derived from negative squared distances between ground truth and cluster centres, followed by normalization and softmax.Under the nearest-neighbour hypothesis, the cluster closest to ground truth receives the maximum probability.Source: Soft Trajectory Probability · p. 6

3.4 Past Motion Cues generated by LLM

LG-Traj uses a guided prompt to have an LLM identify motion patterns from an observed pedestrian trajectory. The generated cues are tokenized for use as past-motion embeddings.

3.4 Past Motion Cues generated by LLM

  • The prompt guides the LLM to identify motion patterns from an observed pedestrian trajectory.It specifies the system’s role, the task, and the format of the input data.Source: Prompt Engineering · pp. 6–7
  • The LLM generates past motion cues M_i from the observed coordinates X_i of pedestrian i.The cues describe the pedestrian’s motion patterns.Source: Generation · p. 7
  • A tokenizer transforms each pedestrian’s generated past motion cues into an embedding vector T_i.The paper denotes the tokenizer as g_ST.Source: Generation · p. 7

3.5 Motion Embedding

The motion embedding combines embeddings of past LLM-generated cues, future-trajectory clusters, and observed trajectories. Positional encoding is added before the combined representation enters the motion encoder.

3.5 Motion Embedding

  • A linear layer maps the tokenized LLM output into the past motion cues embedding Z_m.The linear layer has trainable parameter matrix W_m.Source: §3.5 · p. 7
  • A linear layer produces cluster embedding Z_c, whose rows correspond to the trajectory clusters.The cluster embedding has output dimension O_c and |C| clusters.Source: §3.5 · p. 7
  • The model embeds observed trajectories as Z_p, concatenates Z_m, Z_c, and Z_p, then adds positional encoding to form Z_f for the motion encoder.The observed trajectory matrix is X; positional encoding represents temporal information.Source: §3.5 · pp. 7–8

3.6 Motion Encoder

The motion encoder models spatio-temporal motion patterns using multiple identical layers, each with multi-head self-attention and a feed-forward network.

3.6 Motion Encoder

  • The motion encoder consists of L identical layers, each containing multi-head self-attention and a feed-forward network.Its stated role is to model spatio-temporal motion patterns in pedestrian trajectories.Source: §3.6 · p. 8
  • Multi-head self-attention combines attention heads, with each head attending over query, key, and value matrices.The attention weights are computed from QK^T scaled by the square root of d_k and normalized with softmax.Source: §3.6 · p. 8
  • The encoder passes the motion embedding through L layers to obtain Z_e, whose stated shape is B × 1 × embed_size.The feed-forward network uses learnable weight matrices and bias vectors.Source: §3.6 · p. 8

3.7 Social Decoder

The social decoder combines the encoded motion of the current pedestrian with embeddings of neighboring pedestrians. It uses attention to predict future trajectories and associated probabilities.

3.7 Social Decoder

  • The decoder receives the motion encoder output and a linear embedding of neighbors’ observed trajectories.The current pedestrian supplies the query, while neighboring pedestrians supply the keys and values.Source: §3.7 · p. 8
  • Decoder self-attention weighs neighbor interactions relative to the current pedestrian and produces predicted trajectories with associated probabilities.The probabilities estimate uncertainty for each prediction.Source: §3.7 · p. 9
  • At test time, LG-Traj generates past motion cues from observed test trajectories and reuses the trajectory clusters formed during training as future motion cues.The model combines these cues with the test observations to predict future trajectories.Source: §3.7 · p. 9

3.8 Training Loss

Training uses a trajectory prediction loss and a trajectory-probability loss, combined into one total loss.

3.8 Training Loss

  • Trajectory prediction loss averages the Huber loss between each predicted trajectory and its ground truth.The Huber loss uses a δ threshold, with quadratic and linear cases.Source: §3.8 · p. 9
  • The trajectory-probability loss is cross-entropy between ground-truth and predicted probabilities.The equation averages the probability loss across N trajectories.Source: §3.8 · p. 9
  • The total training loss is the sum of trajectory prediction loss and trajectory-probability loss.The probability loss compares ground-truth and predicted trajectory probabilities.Source: §3.8 · p. 9

4 Experiments

Experiments evaluate LG-Traj on ETH-UCY and SDD using standard ADE/FDE metrics and 8 observed frames to predict 12. The method leads the reported comparisons, while ablations identify motion cues and dataset-dependent augmentation settings as important.

4.1 Experimental settings

  • Experiments use ETH-UCY and SDD with 8 observed frames and 12 predicted frames; ADE averages future-step position error, while FDE measures endpoint error.The observation and prediction periods are 3.2 and 4.8 seconds, respectively.Source: Datasets · p. 10, Evaluation Metrics · p. 10

4.2 Comparison with State-of-art Methods

  • LG-Traj outperforms all compared methods on ETH-UCY, improving average ADE/FDE by 10%/3% versus SMEMO, and records SDD ADE/FDE of 7.80/12.79.Tables 2 and 3 cover ETH, HOTEL, UNIV, ZARA1, ZARA2, and SDD, using 8 observed and 12 predicted time steps.Source: Table 2 · p. 11, Table 3 · p. 11, Quantitative Results · pp. 11, 13
  • Across ETH, UNIV, HOTEL, and ZARA, visualizations show predicted trajectories closely matching ground truth, with observed paths also displayed.Predictions are yellow, observations orange, and ground truth green.Source: Figure 4 · p. 12

4.3 Ablation Studies

  • Ablations report all three components—motion cues, position encoding, and trajectory augmentation—as important, with the largest improvement when motion cues are used.The supplied results do not specify the ablation values.Source: Effect of various components in… · pp. 13–14
  • Tokenizer size 16 is optimal for ETH and HOTEL, while size 32 gives the best results for UNIV, ZARA1, ZARA2, and SDD.Source: Different token sizes for the tokenizer · p. 14
  • On SDD, k=3 gives the best ADE/FDE; lower k loses more trajectory information, whereas k=3 yields minor information loss and motion patterns similar to the originals.The study varies k in rank-k trajectory augmentation.Source: Table 6 · p. 12, Figure 6 · p. 13, Figure 5 · p. 13

5 Conclusion

LG-Traj combines LLM-derived past motion cues with clustered future motion cues to model pedestrian trajectories. Its motion encoder models motion patterns, while its social decoder incorporates interactions among neighbouring pedestrians.

5 Conclusion

  • LG-Traj combines LLM-derived past cues and mixture-of-Gaussians clusters of training-set future trajectories to represent pedestrian motion.The motion encoder uses both cue types alongside observed trajectories.Source: §5 · p. 14
  • A social decoder combines neighbouring pedestrians’ interactions with the motion-encoder embedding to generate socially plausible future trajectories.Source: §5 · p. 14
  • Experiments on ETH-UCY and SDD, together with ablation studies, demonstrate the approach’s effectiveness on pedestrian trajectory prediction benchmarks.The findings also illustrate the feasibility of integrating LLMs into trajectory prediction.Source: §5 · p. 14
Loading 2403.08032v1…