PaperlayerView exact revision on arXiv

Source-linked AI summary

CoT-Drive: Efficient Motion Forecasting for Autonomous Driving with LLMs and Chain-of-Thought Prompting

Haicheng Liao, Hanlin Kong, Bonan Wang, Chengyue Wang, Wang Ye, Zhengbing He, Chengzhong Xu, Zhenning Li

arXiv:2503.07234v1cs.CVcs.AIcs.RO

TL;DR

Motion forecasting for autonomous driving requires robust contextual understanding while remaining efficient on edge devices. CoT-Drive uses GPT-4 Turbo, CoT prompting, and teacher-student distillation to train lightweight models, and reports improvements across five real-world datasets, while noting that important challenges remain.

  • Problem

    Motion-forecasting models often struggle with unseen or corner-case traffic because their generalizability and contextual understanding remain limited.

  • Method

    CoT-Drive uses GPT-4 Turbo with CoT prompting to generate semantic annotations, then distills its scene understanding into lightweight edge language models.

  • Results

    CoT-Drive surpasses state-of-the-art models across five real-world driving datasets and establishes a new benchmark for efficient motion forecasting.

  • Takeaways & Limitations

    The study demonstrates a lightweight surrogate approach for transferring LLM scene interpretation to real-time motion forecasting on edge devices.

  • Takeaways & Limitations

    The authors state that significant challenges remain despite CoT-Drive’s advances in prediction accuracy and efficiency.

Abstract

from arXiv · show

Accurate motion forecasting is crucial for safe autonomous driving (AD). This study proposes CoT-Drive, a novel approach that enhances motion forecasting by leveraging large language models (LLMs) and a chain-of-thought (CoT) prompting method. We introduce a teacher-student knowledge distillation strategy to effectively transfer LLMs' advanced scene understanding capabilities to lightweight language models (LMs), ensuring that CoT-Drive operates in real-time on edge devices while maintaining comprehensive scene understanding and generalization capabilities. By leveraging CoT prompting techniques for LLMs without additional training, CoT-Drive generates semantic annotations that significantly improve the understanding of complex traffic environments, thereby boosting the accuracy and robustness of predictions. Additionally, we present two new scene description datasets, Highway-Text and Urban-Text, designed for fine-tuning lightweight LMs to generate context-specific semantic annotations. Comprehensive evaluations of five real-world datasets demonstrate that CoT-Drive outperforms existing models, highlighting its effectiveness and efficiency in handling complex traffic scenarios. Overall, this study is the first to consider the practical application of LLMs in this field. It pioneers the training and use of a lightweight LLM surrogate for motion forecasting, setting a new benchmark and showcasing the potential of integrating LLMs into AD systems.

I. INTRODUCTION

Motion forecasting must capture complex traffic context while remaining robust in unseen scenarios and practical for edge deployment. CoT-Drive addresses this tension by distilling LLM scene understanding into lightweight models and using CoT prompting for contextual interpretation.

  • Accurate motion forecasting supports safe autonomous-vehicle decision-making and planning in dynamic, heterogeneous traffic environments.
  • Existing deep-learning models often perform poorly in unseen or corner-case scenarios because they have limited generalizability and contextual understanding.
  • Online LLMs offer comprehensive scene understanding but face communication latency, cost, and data-security risks in autonomous vehicles.
  • Offline LLMs reduce privacy and transmission risks but remain difficult to deploy on resource-constrained edge devices because of their storage and computational requirements.
  • CoT-Drive uses GPT-4 Turbo as a teacher and lightweight language models as students to transfer scene understanding while reducing the overhead of direct local-LLM deployment.
  • The paper investigates efficient forecasting in challenging highway, urban, and intersection scenes, knowledge transfer to edge devices, and CoT prompting without additional fine-tuning.

III. PROPOSED DATASETS

The proposed Highway-Text and Urban-Text datasets provide large-scale, context-specific scene descriptions for training lightweight language models. Their CoT prompting workflow progresses from scene background through interactions and risks to maneuver and trajectory prediction.

  • Highway-Text and Urban-Text contain over 10 million words describing traffic scenarios for lightweight-model fine-tuning.
  • Highway-Text includes 4,327 NGSIM scenarios and 2,279 HighD scenarios, while Urban-Text draws on MoCAD and ApolloScape samples.
  • The CoT prompting technique uses sequential language instructions to generate context-aware semantic annotations and reduce hallucinations.
  • Background and Statistics identifies agents, states, road conditions, traffic density, incidents, and potential agent behaviors.
  • Interaction Analysis examines how vehicles, pedestrians, and cyclists influence one another and affect future maneuvers.
  • Risk Assessment evaluates collision likelihood and urgency using agent distribution, speed, road conditions, behavior, type, number, and proximity.
  • Prediction produces target-vehicle maneuver forecasts, future trajectory coordinates, justifications, and a summary of the reasoning process.
  • Generated annotations are manually validated and cross-checked against traffic rules and legal standards, including GDPR requirements.

B. Language-Instructed Encoder

The Language-Instructed Encoder uses GPT-4 Turbo with zero-shot CoT prompting to generate semantic answers, then distills those answers into a lightweight student model for edge inference.

  • The teacher model generates semantic answers that encode scene understanding for subsequent student-model training.
  • GPT-4 Turbo receives structured prompts and questions that guide progressive traffic-scene interpretation without additional fine-tuning.
  • CoT reasoning decomposes interpretation into sequential steps that mirror interaction, risk assessment, and prediction.
  • A lightweight edge LM takes historical agent states as input and produces semantic annotations under supervision from the teacher’s informative scene answers.
  • Knowledge distillation progressively aligns the student outputs with teacher answers through an error-based training objective.
  • The study evaluates GPT-Neo, Qwen 1.5, TinyLlama, and Phi 1.5 as candidate student models.

3) Multimodal Fusion:

CoT-Drive fuses semantic annotations with historical agent states, then models spatial interactions and cross-modal dependencies before decoding multimodal trajectories. Its decoder combines maneuver-based Gaussian mixture predictions with deep ensembles to represent aleatoric and epistemic uncertainty.

  • Semantic and temporal fusion: Semantic annotations are processed with DistilBERT and max pooling, while historical target-agent states pass through a Linear-ELU-LSTM network before MLP fusion.
  • Spatial encoding: The transformer-based encoder uses multi-head attention and normalization with shared weights across time frames to model spatial interactions among agents.
  • Cross-modal encoding: An attention mechanism dynamically weights semantic, multimodal, and spatial features to capture cross-modal interactions before decoding.
  • Uncertainty-aware decoding: A Gaussian Mixture Model predicts maneuver-based multimodal trajectories, covering lateral and longitudinal movements to model aleatoric uncertainty.
  • Uncertainty-aware decoding: Deep ensembles of LSTMs, TCNs, and multi-head self-attention models quantify epistemic uncertainty and improve robustness for multimodal future trajectories.
  • Trajectory decoding: The decoder transforms fused features across time and derives bivariate Gaussian parameters for future trajectory predictions.

F. Training and Inference

CoT-Drive trains in two sequential stages: fine-tuning edge language models on distilled scene descriptions, followed by multitask motion-forecasting training. The losses combine autoregressive semantic learning with trajectory and maneuver objectives.

  • Stage 1: LM fine-tuning: Edge language models are first fine-tuned on Highway-Text and Urban-Text to learn semantic scene information distilled from GPT-4 Turbo.
  • Stage 1: LM fine-tuning: The first-stage autoregressive loss trains the student model on merged scene prompts and teacher answers, supporting vocabulary- and semantic-level alignment.
  • Stage 2: motion forecasting: The second-stage loss combines trajectory prediction and maneuver prediction through a multitask objective weighted by hyperparameter α.
  • Stage 2: motion forecasting: The maneuver loss evaluates predicted trajectories against intended maneuvers, while the trajectory loss is defined using a bivariate Gaussian distribution.
  • Stage 2: motion forecasting: The combined losses aim to keep predicted trajectories accurate and aligned with realistic driving maneuvers.

2) Inference:

During inference, the lightweight language model generates scene descriptions from historical agent states, and CoT-Drive combines those annotations with motion history to predict multimodal futures. Evaluation spans five real-world motion-forecasting datasets, with dataset-specific history and prediction horizons.

  • Inference pipeline: Historical agent states are converted into text inputs for fine-tuned edge language models, which generate semantic scene annotations through prompt engineering.
  • Inference pipeline: The forecasting model combines generated semantic annotations with historical agent states to produce multimodal future trajectories.
  • Inference pipeline: Only the knowledge-distilled lightweight language model generates scene descriptions during inference, supporting efficient AV predictions.
  • Language-model evaluation: Validation loss curves compare Phi-1.5, TinyLlama, Qwen-1.5, and GPT-Neo on Urban-Text and Highway-Text.
  • Evaluation datasets: CoT-Drive evaluates language-model scene understanding on Highway-Text and Urban-Text and motion forecasting on NGSIM, HighD, MoCAD, ApolloScape, and nuScenes.
  • Evaluation settings: NGSIM, HighD, and MoCAD use 3-second histories to predict 5-second futures, while ApolloScape and nuScenes follow separate dataset-specific horizons.

3) Evaluation Metrics:

The evaluation uses text-alignment metrics for language-model outputs and displacement-based metrics for motion forecasting. Implementation settings include four A100 GPUs, separate LM and forecasting training schedules, and an ensemble decoder.

  • Scene-understanding metrics: BERT-Score evaluates alignment between generated scene annotations and GPT-4 Turbo references using Precision, Recall, and F1, where values closer to 1 indicate better alignment.
  • Motion-forecasting metrics: NGSIM, HighD, and MoCAD motion forecasts are evaluated with Root Mean Square Error (RMSE).
  • Motion-forecasting metrics: ApolloScape uses Average Displacement Error (ADE) and Final Displacement Error (FDE), including weighted versions for vehicles, pedestrians, and bicycles.
  • Motion-forecasting metrics: nuScenes uses minADEk, minFDEk, and Miss Rate at 2 meters over k to assess predicted trajectory quality.
  • Implementation: CoT-Drive is trained on four NVIDIA A100 40GB GPUs, with language-model fine-tuning using bf16 precision, supervised fine-tuning, and 10 epochs.
  • Implementation: The motion-forecasting framework trains for 16 epochs with Adam and cosine annealing warm restarts, using an ensemble decoder of multi-scale LSTMs, TCNs, and multi-head self-attention.

C. Evaluations on Five Real-world Datasets

CoT-Drive is evaluated against state-of-the-art baselines across five real-world driving datasets, covering highway, urban, and map-free multi-agent forecasting. It reports improvements across ApolloScape, NGSIM, HighD, MoCAD, and nuScenes.

  • ApolloScape: CoT-Drive surpasses AI-TP in ApolloScape WSADE by 5.1% and MSTG in WSFDE by 4.9%.The evaluation targets multi-agent predictions for vehicles, pedestrians, and bicycles in urban scenes.
  • NGSIM: 15.59% higher long-term (5s) accuracy and 12.07% higher average prediction accuracy are reported on NGSIM.The dataset emphasizes high-speed, dense highway scenarios involving lane changes and merging.
  • HighD: 28.7% higher long-term accuracy and 23.08% higher average accuracy are reported on HighD.HighD provides larger sample sizes and more accurate trajectories than NGSIM.
  • MoCAD: CoT-Drive improves average performance by 11.33% across all prediction horizons on MoCAD.MoCAD includes campus streets, intersections, and complex right-hand-drive urban roads.
  • nuScenes: On nuScenes, CoT-Drive improves minADE5, minFDE5, and MR5 by at least 8.34%, 5.16%, and 2.50%, respectively.The map-free framework achieves these gains over non-map-based models without requiring costly HD maps.

D. Comparison of Model Performance and Efficiency

The study compares lightweight language models and CoT-Drive components by balancing scene-understanding quality, inference efficiency, and forecasting performance. Results indicate that moderate model size, knowledge distillation, and the proposed encoder components are important for this balance.

  • Inference Efficiency: Vicuna-13B improves accuracy by only 3.92% while increasing inference time 12.7-fold, making it impractical for real-time systems.The efficiency comparison covers Vicuna-13B, Llama2-7B, and Mistral-7B variants on NGSIM and nuScenes.
  • Lightweight LMs: GPT-4 Turbo distills complex-scene understanding into lightweight student LMs evaluated on Highway-Text and Urban-Text.The compared students include GPT-Neo, Qwen 1.5, TinyLlama, and Phi 1.5.
  • Lightweight LMs: Figure 5 compares parameter count with F1 Score on Urban-Text and Highway-Text, while Figure 6 visualizes semantic annotations for a nuScenes scene.The figures support comparison of model size, scene-understanding performance, and generated descriptions.
  • Lightweight LMs: Increasing parameters from 0.13B to 0.62B raises F1 scores from 0.87 and 0.89 to 0.94 and 0.97, while expansion to 1.5B yields only 0.27% and 0.58% gains.The results show diminishing scene-understanding benefits from further parameter growth.
  • Component Ablations: Removing knowledge distillation reduces performance by up to 22.34%, while removing interaction-aware encoding reduces it by 15.93%.These ablations test the Language-Instructed Encoder and Interaction-aware Encoder components.
  • Component Ablations: Replacing cross-attention with an MLP reduces performance by 9.47%, and replacing the decoder with an MLP reduces it by 18.62%.The results identify cross-modal attention and the proposed decoder as contributors to forecasting performance.

4) Ablation Studies on the CoT Prompting (A3):

Ablation and qualitative analyses examine CoT prompting, ensemble decoding, and the Language-Instructed Encoder across highway and urban scenarios. The results associate these components with improved semantic interpretation and trajectory predictions.

  • 5) Ablation Studies on the Decoder:: Removing all decoder submodels causes an 8.3% drop in CoT-Drive predictions.The decoder ablation evaluates ensemble architectures and submodel counts.
  • 5) Ablation Studies on the Decoder:: Six submodels combining MS-LSTM, self-attention, and TCN outperform nine-submodel configurations, indicating diminishing returns from further expansion.The study frames the result as a balance between model complexity and efficiency.
  • 5) Ablation Studies on the Decoder:: Decoder ensembles improve prediction accuracy by combining strategies for handling aleatoric and epistemic uncertainty.The cited ablation describes the proposed dual-strategy framework for traffic scenarios.
  • 1) Qualitative Results in Highway Scenes:: On NGSIM, the Language-Instructed Encoder identifies a target vehicle’s lane-change intent, while WSiP and its ablated variant predict continuing straight.The qualitative comparison includes CoT-Drive, Ours (-LI), and WSiP in complex highway scenes.
  • 1) Qualitative Results in Highway Scenes:: CoT prompting generates semantic annotations from interaction and risk perspectives, supporting the improvements illustrated in complex highway scenarios.The qualitative analysis attributes the comparison to the Language-Instructed Encoder and CoT prompting.
  • 2) Qualitative Results in Complex Urban Scenes:: In nuScenes urban scenes, CoT-Drive predicts deceleration and stopping at a changing signal, whereas Ours (-LI) predicts a turn.The comparison uses no explicit traffic-light states or HD maps in the described case.
  • 2) Qualitative Results in Complex Urban Scenes:: For a left-turn scenario, CoT-Drive predicts deceleration before turning to account for crossing pedestrians and straight-moving vehicles, unlike Ours (-LI).The qualitative result connects context-aware scene descriptions with surrounding-agent interactions.
  • 2) Qualitative Results in Complex Urban Scenes:: The urban qualitative results are presented on nuScenes by comparing CoT-Drive with its variant without the Language-Instructed Encoder.The visualizations show one predicted trajectory per model and are described as supporting real-world applicability.

VI. DISCUSSIONS

CoT-Drive combines lightweight language models, knowledge distillation, and semantic annotations to improve motion forecasting across diverse traffic scenes. Its remaining weaknesses center on rare behavioral shifts and dependence on high-quality annotated data.

  • Contributions: The Language-Instructed Encoder distills GPT-4 Turbo’s scene understanding into a lightweight language model for real-time edge-device motion forecasting.The paper identifies this as the first trained lightweight LLM surrogate used for motion forecasting.
  • Contributions: Highway-Text and Urban-Text provide CoT-enriched scene descriptions for fine-tuning lightweight language models to generate context-specific semantic annotations.The datasets are derived from real-world data and intended as resources for AD and LLM research.
  • Performance: CoT-Drive surpasses most state-of-the-art baselines across NGSIM, HighD, MoCAD, ApolloScape, and nuScenes.The evaluations cover highways, intersections, and dense urban areas.
  • Limitations: CoT-Drive remains inaccurate when launch, U-turn, lane-changing, or reverse-parking behavior departs sharply from observed trajectories.The model can be closer to ground truth than the baseline in launch scenes, but still struggles with intent and rapid behavioral shifts.
  • Limitations: High-quality, well-annotated datasets are costly and difficult to obtain for rare or complex scenarios, limiting adaptation to unseen conditions.The paper identifies congested streets, adverse weather, privacy concerns, and dataset bias as related data challenges.

C. Discussion of Unsupervised and Semi-supervised Approaches for Scene Description Generation

The discussion contrasts LLM-based scene description with unsupervised, semi-supervised, reinforcement, and evolutionary learning approaches. CoT-Drive uses CoT prompting and knowledge distillation to retain semantic understanding while targeting lightweight edge deployment.

  • Unsupervised and Semi-supervised Approaches: Unsupervised learning can exploit large unlabeled datasets, while semi-supervised learning uses labeled seed data to create pseudo-labels for additional unlabeled data.Both approaches aim to reduce dependence on expensive manual annotation.
  • Comparative Discussion: LLM-based methods depend on curated, labor-intensive annotations, whereas reinforcement and evolutionary learning can use simulated environments with dynamic obstacles.Simulation reduces data preparation but introduces a reality gap that can limit real-world generalization.
  • CoT-Drive: CoT-Drive combines CoT-enhanced scene understanding with knowledge distillation to make LLM functionality feasible for lightweight edge-level deployment.The framework is presented as addressing computational and latency issues associated with larger LLM-based methods.
  • Comparative Discussion: LLMs provide context-aware reasoning and semantic depth, while reinforcement and evolutionary learning primarily optimize policies and may lack broader environmental interpretation.The comparison concerns semantic adaptability and explanatory understanding in complex, unpredictable situations.
  • Results: Comprehensive evaluations on five real-world driving datasets report that CoT-Drive surpasses state-of-the-art models by significant margins.The conclusion characterizes the framework as a practical solution for embedding LLMs in edge autonomous-driving devices.
Loading 2503.07234v1…