PaperlayerView exact revision on arXiv

Source-linked AI summary

Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models

Zhengxing Lan, Hongbo Li, Lingshan Liu, Bo Fan, Yisheng Lv, Yilong Ren, Zhiyong Cui

arXiv:2405.04909v1cs.CVcs.AI

TL;DR

Trajectory predictors still lack scene cognition and understanding of complex traffic semantics. Traj-LLM processes agent and scene features with pre-trained LLMs without explicit prompt engineering, alongside lane-aware probability learning. It reports outperforming state-of-the-art methods across metrics and in few-shot evaluation.

  • Problem

    Existing trajectory-prediction methods have improved performance, but a gap remains in scene cognition and understanding of complex traffic semantics.

    Source: §I · p. 1

  • Method

    Traj-LLM encodes agent and scene features for pre-trained LLMs to model high-level knowledge and interactions, and adds Mamba-powered lane-aware probability learning.

    Source: §I · p. 2

  • Results

    Traj-LLM generally ranks first or second across reported metrics and is reported to outperform state-of-the-art methods overall and in few-shot evaluation.

    Source: §I · p. 2, B. Results and Computational Performance · p. 7, §VI · p. 10

  • Takeaways & Limitations

    The study presents pre-trained LLMs without explicit prompt engineering as a versatile and easily adaptable approach to agent-motion forecasting.

    Source: §I · p. 2

Abstract

from arXiv · show

Predicting the future trajectories of dynamic traffic actors is a cornerstone task in autonomous driving. Though existing notable efforts have resulted in impressive performance improvements, a gap persists in scene cognitive and understanding of the complex traffic semantics. This paper proposes Traj-LLM, the first to investigate the potential of using Large Language Models (LLMs) without explicit prompt engineering to generate future motion from agents' past/observed trajectories and scene semantics. Traj-LLM starts with sparse context joint coding to dissect the agent and scene features into a form that LLMs understand. On this basis, we innovatively explore LLMs' powerful comprehension abilities to capture a spectrum of high-level scene knowledge and interactive information. Emulating the human-like lane focus cognitive function and enhancing Traj-LLM's scene comprehension, we introduce lane-aware probabilistic learning powered by the pioneering Mamba module. Finally, a multi-modal Laplace decoder is designed to achieve scene-compliant multi-modal predictions. Extensive experiments manifest that Traj-LLM, fortified by LLMs' strong prior knowledge and understanding prowess, together with lane-aware probability learning, outstrips state-of-the-art methods across evaluation metrics. Moreover, the few-shot analysis further substantiates Traj-LLM's performance, wherein with just 50% of the dataset, it outperforms the majority of benchmarks relying on complete data utilization. This study explores equipping the trajectory prediction task with advanced capabilities inherent in LLMs, furnishing a more universal and adaptable solution for forecasting agent motion in a new way.

I. INTRODUCTION

Traj-LLM targets the gap in scene cognition and understanding in trajectory prediction by applying pre-trained LLMs without explicit prompt engineering. It combines agent-scene encoding, LLM interaction modeling, lane-aware learning, and multimodal decoding, and reports improvements over prior methods, including in few-shot settings.

I. INTRODUCTION

  • Existing trajectory predictors model temporal dependencies and spatial interactions, but a gap remains in scene cognition and high-level semantic understanding.Prior approaches include RNNs, LSTMs, GRUs, Transformers, pooling, and GCNs for spatiotemporal modeling –.Source: §I · p. 1
  • Prompt-based LLM trajectory methods require careful prompt design, which can introduce modeling bias and complicate integration with existing predictors.Earlier LLM-based trajectory prediction work focused on pedestrians [15], [16].Source: §I · p. 2
  • Traj-LLM encodes agent and scene features for LLMs, then uses them to model high-level scene knowledge and social interactions without explicit prompt engineering.The approach also uses parameter-efficient fine-tuning and a multimodal Laplace decoder for scene-compliant predictions.Source: §I · p. 2
  • A Mamba-powered lane-aware probability-learning method imitates human lane focus, enhances scene understanding, and guides motion states toward potential lane segments.The paper presents it as a component of Traj-LLM's prediction framework.Source: §I · p. 2
  • Traj-LLM outperforms state-of-the-art methods overall and in few-shot challenges, according to experiments on real-world datasets.The framework's few-shot results are reported alongside its overall predictions.Source: §I · p. 2

A. Trajectory Prediction

  • Lane-based trajectory methods use goals in credible lanes or anchors aligned with trajectory modes, yet the paper identifies a continuing gap in scene cognition.The paper calls for stronger modeling of high-level scene knowledge and interactive information.Source: A. Trajectory Prediction · p. 2

B. Large Language Models

This section reviews LLMs' reasoning and comprehension capabilities and prior prompted applications to trajectory prediction. It also formulates prediction as forecasting a target agent's future coordinates from agent histories and map information.

B. Large Language Models

  • LLMs have been applied to autonomous-driving tasks, while prompted trajectory-prediction work uses them to model pedestrian behavior, scene context, and social relationships.The cited trajectory methods include LG-Traj [16] and LMTraj [15].Source: B. Large Language Models · p. 3
  • Prompt-based trajectory methods require well-structured prompts, which may introduce modeling bias and integration hurdles.The paper contrasts these methods with its exploration of LLMs without explicit prompt engineering.Source: B. Large Language Models · p. 3

III. PROBLEM FORMULATION

  • Trajectory prediction is defined as forecasting the target agent's future coordinates over a specified horizon, conditional on agent histories and an HD map.Agent trajectories and lane centerlines are represented as vectorized entities, and coordinates are normalized around the target agent's latest position.Source: §III · p. 3
  • Lane centerlines are divided into predefined segments represented by a variable number of vectors.This vectorized lane representation captures lane information alongside agent trajectories.Source: §III · p. 3
  • The formulation assumes a Laplace conditional distribution and generates K future trajectories for each target agent with a probability score for each prediction.Source: §III · p. 3

IV. PROPOSED MODEL

Traj-LLM has four components, from sparse context joint encoding to a multimodal Laplace decoder. Its encoding stage fuses local agent and lane information into representations intended for LLM processing.

IV. PROPOSED MODEL

  • Traj-LLM comprises sparse context joint encoding, high-level interaction modeling, lane-aware probability learning, and a multimodal Laplace decoder.The components encode agent and scene features, model interactions, and generate predictions.Source: §IV · p. 3

A. Sparse Context Joint Encoding

  • GRU and MLP embeddings extract agent-state and lane features before fusion exchanges information between them in localized regions.The token-embedding-like process aligns the representations with how LLMs work.Source: A. Sparse Context Joint Encoding · p. 3
  • Multi-head self-attention fuses agent features, while cross-attention updates lane and agent representations with skip connections.The agent-agent fusion also uses gated linear units.Source: A. Sparse Context Joint Encoding · p. 3
  • Concatenated agent and lane representations form sparse context joint encodings that carry local dependencies and are designed to help LLMs understand trajectory data.Source: A. Sparse Context Joint Encoding · p. 4

B. High-level Interaction Modeling

Traj-LLM uses parameter-efficient fine-tuning to adapt a pre-trained LLM to sparse trajectory encodings without retraining all of its parameters. The resulting interaction representations are projected to the dimensions used by the model's subsequent stages.

B. High-level Interaction Modeling

  • Because generic pre-trained LLMs are primarily tailored to text, directly using them on sparse trajectory encodings is unsuitable, while full retraining is computationally demanding.The paper presents PEFT as a more efficient alternative for downstream adaptation,.Source: B. High-level Interaction Modeling · p. 4
  • Traj-LLM uses GPT-2 and LoRA, freezing pre-trained parameters while adding trainable parameters to the attention layers' Query and Key.The method applies LoRA to the Query and Key of the LLM attention layers.Source: B. High-level Interaction Modeling · p. 4
  • The LoRA update keeps the pre-trained weight matrix W frozen and trains low-rank matrices B and A instead.The rank r is much smaller than the weight matrix dimensions.Source: B. High-level Interaction Modeling · p. 4
  • Sparse context encodings enter the LoRA-equipped LLM to produce interaction representations, which an MLP transforms to match the encoding dimensions.The MLP output is the high-level interaction state.Source: B. High-level Interaction Modeling · p. 4

C. Lane-aware Probability Learning

Lane-aware probability learning uses Mamba to estimate which lane segments align with an agent’s future motion, reflecting drivers’ focus on a few influential lanes. It supplies lane-guided features to Traj-LLM’s scene understanding and prediction.

C. Lane-aware Probability Learning

  • Lane-aware probability learning estimates motion-state alignment with lane segments at each future time step, using Mamba to refine pertinent information.The design is motivated by experienced drivers attending to a few potential lane segments that influence future motion.Source: C. Lane-aware Probability Learning · pp. 4–5
  • The Mamba layer normalizes concatenated agent and lane features, processes them with a Mamba block, then applies a position-wise feed-forward network.The input F combines s_i and e_f^l; the block produces lane-aware features, with instance normalization and residual connections used around processing stages.Source: C. Lane-aware Probability Learning · p. 5
  • The model selects the top-c lane segments by predicted score and concatenates them to form M′ for lane-guided interaction.The selected candidate lanes are used to guide the target agent toward potential lane segments.Source: C. Lane-aware Probability Learning · p. 5
  • A binary cross-entropy loss trains lane probabilities, labeling the lane segment closest to the ground-truth position as 1 and the others as 0.The lane-segment distance to the ground-truth position is determined using Euclidean distance.Source: C. Lane-aware Probability Learning · pp. 5–6

D. Multi-modal Laplace Decoder

The multi-modal Laplace decoder predicts a distribution over trajectories from sparse-context, lane-guided interaction, and sampled latent features. It trains trajectory modes and their mixture coefficients with separate regression and classification losses.

D. Multi-modal Laplace Decoder

  • The decoder uses a mixture model whose components follow Laplace distributions to represent multi-modal future motion.For each instance, it outputs trajectory-mixture coefficients and component parameters from representation e_i.Source: D. Multi-modal Laplace Decoder · p. 6
  • Each decoder representation combines sparse context encodings, lane-aware interaction features, and a multivariate-normal latent vector.Cross-attention between s_i and M′ generates the lane-aware feature; an MLP with softmax predicts mixture coefficients, while two MLPs generate location parameters.Source: D. Multi-modal Laplace Decoder · p. 6
  • Training combines a Winner-Takes-All regression loss for the mode with minimum L2 error and a cross-entropy classification loss for mixture coefficients.The overall loss includes a hyperparameter λ that controls the relative importance of the lane loss.Source: D. Multi-modal Laplace Decoder · p. 6

V. EXPERIMENT

The experiments assess Traj-LLM through benchmark comparisons, component ablations, few-shot evaluation, and qualitative prediction analysis.

V. EXPERIMENT

  • The experiments compare Traj-LLM’s results with cutting-edge approaches after describing the experimental setup.The paper says the comparison is intended to substantiate the model’s efficacy.Source: §V · p. 6
  • Ablation experiments evaluate how modifications to Traj-LLM components affect overall performance.Source: §V · p. 6
  • The evaluation also includes a few-shot study of generalization and a qualitative analysis of predictions.Source: §V · p. 6

A. Experimental Setup

Traj-LLM is evaluated on nuScenes using the official prediction scheme and standard displacement and miss-rate metrics. The setup also specifies the prediction modes and model training configuration.

A. Experimental Setup

  • The nuScenes evaluation uses 2-second observed segments to forecast the next 6 seconds, with K set to 5 or 10 prediction modes.The dataset comprises 1,000 driving scenarios collected in multiple cities using vehicle-mounted cameras and lidar at 2 Hz.Source: A. Experimental Setup · p. 6
  • Performance is measured with minADE, minFDE, and MR, where lower values indicate better performance.MR counts instances whose predicted endpoint is more than 2.0 meters from the actual endpoint.Source: A. Experimental Setup · p. 6
  • Training uses AdamW on six RTX4090 GPUs with batch size 132 and initial learning rate 0.001.The architecture has sparse context joint encoding, high-level interaction modeling, and three lane-aware probability learning layers; feature dimensions are 128.Source: A. Experimental Setup · p. 6
  • Table I compares Traj-LLM with state-of-the-art methods for K = 5, marking best values in bold and second-best values with underlining.Source: Table I · p. 7

B. Results and Computational Performance

Traj-LLM ranks first or second across reported metrics and improves on leading baselines. It uses 7.72M trainable parameters and reports 98 ms inference time for an average of 12 agents.

B. Results and Computational Performance

C. Ablation Studies

The ablations report contributions from the LLM and lane-aware learning components, while few-shot trials test performance with reduced training data. With 50% of the data, Traj-LLM outperforms a majority of baselines trained on 100%.

C. Ablation Studies

  • Removing the LLM reduces minFDE5 accuracy by 7.52% and minFDE10 by 9.42%; removing lane-aware learning decreases MR5 by 12.77% and MR10 by 14.81%.The ablations compare Traj-LLM variants for K=5 and K=10; the w/o LLM variant removes the high-level interaction module, while w/o Lane removes lane-aware learning.Source: C. Ablation Studies · p. 8, Table IV · p. 8, Table V · p. 8
  • The w/o LoRA variant freezes the LLM, and the ablation discussion reports that fine-tuning is necessary to transfer cross-modal inference capabilities to complex traffic-scene comprehension.The w/o Lane variant removes lane-aware learning, which selects candidate lanes to guide motion.Source: C. Ablation Studies · p. 8

D. Few-shot Study

  • With 50% of training data, Traj-LLM outperforms a majority of 100%-data baselines across evaluated metrics, and retains commendable performance with 10%–30%.The few-shot trials keep the evaluation sets consistent with full-sample experiments; Tables VI and VII report K=5 and K=10 results.Source: D. Few-shot Study · pp. 8–9, Table VI · p. 9, Table VII · p. 9

E. Qualitative Analysis

Qualitative results show predicted trajectories matching ground-truth trends while remaining within roadway boundaries in full-sample scenarios. Under 50% data, predictions are described as plausible across diverse scenarios and capture acceleration, deceleration, and motion uncertainty.

1) Visualization of motion prediction under full-sample training:

  • For full-sample training, one predicted modality closely follows the ground-truth trend across straight paths and turns, while predictions remain within roadway boundaries.Fig. 8 shows the HD map beside predictions and ground truth, with K=5 and K=10 examples.Source: 1) Visualization of motion prediction… · p. 9, Figure 8 · p. 10

2) Visualization of Motion Prediction under few-shot settings with 50% data samples:

  • With 50% of training data, predictions remain plausible across intersections and straight driving, capturing acceleration, deceleration, and motion uncertainty.Fig. 9 presents K=5 and K=10 examples, with ground truth and predicted trajectories shown alongside the HD map.Source: 2) Visualization of Motion Prediction… · p. 9, Figure 9 · p. 10

VI. CONCLUSIONS

  • The conclusion reports that Traj-LLM avoids explicit prompt engineering and outperforms state-of-the-art methods across various metrics, with few-shot study results further validating its performance.Its pipeline combines sparse context coding, high-level scene and social-interaction knowledge, lane-aware probabilistic learning, and a multi-modal Laplace decoder.Source: §VI · pp. 9–10

Related papers

Loading 2405.04909v1…