PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-Driver: Learning to Drive with GPT

Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, Yue Wang

arXiv:2310.01415v3cs.CVcs.AIcs.CLcs.RO

TL;DR

Autonomous-driving motion planners need to generalize across scenarios while producing safe, comfortable, precise trajectories, yet existing methods face generalization and representation challenges. GPT-Driver reformulates motion planning as language modeling with GPT-3.5, using prompting, reasoning, and fine-tuning; experiments on nuScenes report superior performance, generalization, and interpretability.

  • Problem

    Motion planners must handle diverse driving scenarios and heterogeneous inputs while predicting precise waypoint coordinates, capabilities not established for LLMs.

  • Method

    GPT-Driver converts planner inputs and trajectory coordinates into language tokens and uses GPT-3.5 with prompting, chain-of-thought reasoning, and fine-tuning.

  • Results

    GPT-Driver demonstrates superior motion planning performance, few-shot generalization ability, and interpretability compared with state-of-the-art planners on nuScenes.

  • Takeaways & Limitations

    The approach shows that GPT-3.5 can serve as a motion planner while generating precise trajectory coordinates and explicit natural-language decision reasoning.

  • Takeaways & Limitations

    Inference time remains uncertain because OpenAI API limitations prevent its measurement, and evaluation is limited to open-loop planning rather than closed-loop motion planning.

Abstract

from arXiv · show

We present a simple yet effective approach that can transform the OpenAI GPT-3.5 model into a reliable motion planner for autonomous vehicles. Motion planning is a core challenge in autonomous driving, aiming to plan a driving trajectory that is safe and comfortable. Existing motion planners predominantly leverage heuristic methods to forecast driving trajectories, yet these approaches demonstrate insufficient generalization capabilities in the face of novel and unseen driving scenarios. In this paper, we propose a novel approach to motion planning that capitalizes on the strong reasoning capabilities and generalization potential inherent to Large Language Models (LLMs). The fundamental insight of our approach is the reformulation of motion planning as a language modeling problem, a perspective not previously explored. Specifically, we represent the planner inputs and outputs as language tokens, and leverage the LLM to generate driving trajectories through a language description of coordinate positions. Furthermore, we propose a novel prompting-reasoning-finetuning strategy to stimulate the numerical reasoning potential of the LLM. With this strategy, the LLM can describe highly precise trajectory coordinates and also its internal decision-making process in natural language. We evaluate our approach on the large-scale nuScenes dataset, and extensive experiments substantiate the effectiveness, generalization ability, and interpretability of our GPT-based motion planner. Code is now available at https://github.com/PointsCoder/GPT-Driver.

1 INTRODUCTION

Motion planning must produce safe, comfortable trajectories across diverse and unpredictable scenarios, but existing approaches struggle with uncovered extreme cases and heterogeneous inputs. GPT-Driver reformulates planning as language modeling and combines prompting, reasoning, and fine-tuning to generate precise, interpretable trajectories.

  • Motion planning seeks safe and comfortable trajectories while handling diverse scenarios, unpredictable drivers, and the need for robust, explainable decisions.
  • Rule-based planners are interpretable but generally fail in extreme scenarios not covered by predefined rules.
  • LLMs offer generalization and common-sense reasoning, but their use for motion planning is challenged by heterogeneous inputs and precise coordinate prediction.
  • GPT-Driver converts planner inputs into language tokens and asks GPT-3.5 to generate waypoint coordinates and natural-language decision reasoning.
  • The prompting-reasoning-finetuning strategy targets precise numerical reasoning and transparent decision-making.
  • GPT-Driver demonstrates superior motion planning performance, few-shot generalization ability, and interpretability on nuScenes against state-of-the-art planners.

2 RELATED WORKS

Prior autonomous-driving planners include rule-based, optimization-based, and learning-based methods, while LLM research has mainly addressed language-oriented understanding or high-level embodied-agent planning. GPT-Driver applies GPT-3.5 to low-level waypoint trajectories by converting driving information and outputs into language.

  • Autonomous-driving motion planners comprise rule-based, optimization-based, and learning-based approaches with different formulations and limitations.
  • GPT-Driver is a learning-based planner that uses GPT-3.5 generalization and reasoning to address long-tailed driving scenarios and improve interpretability.
  • GPT-Driver converts observations and ego-states into language prompts, generates trajectories and decision reasoning, then restores trajectories to numerical form.
  • LLMs have shown language understanding and reasoning abilities, but autonomous-driving inputs and outputs are not naturally represented as language.
  • Unlike task-level LLM planning, GPT-Driver targets low-level waypoint trajectories represented as numerical coordinates.

3 GPT-DRIVER

GPT-Driver reformulates autonomous-driving motion planning as language modeling by converting planner inputs and trajectories into language tokens. A prompting-reasoning-finetuning strategy enables GPT-3.5 to generate precise waypoints alongside explicit decision-making descriptions.

  • 3.1 PROBLEM DEFINITION: Motion planning seeks safe and comfortable vehicle trajectories from observations and ego-states, represented as waypoint coordinates across future timesteps.Observations include perception and prediction outputs, while ego-states include historical trajectory, velocity, and acceleration.
  • 3.1 PROBLEM DEFINITION: Traditional learning-based planners imitate human trajectories with L1 regression, but jointly regressing coordinates across scales can make distant waypoints imprecise.Coordinate values may range from 0 to over 50, motivating replacement of trajectory regression with language modeling.
  • 3.2 MOTION PLANNING AS LANGUAGE MODELING: GPT-Driver converts trajectory coordinates into token sequences, allowing GPT-3.5 to generate future waypoints through natural-language descriptions.For example, GPT-3.5 tokenizes 23.17 as “23”, “.”, and “17”.
  • 3.2 MOTION PLANNING AS LANGUAGE MODELING: Token prediction estimates coordinates progressively, combining coarse meter-level integer tokens with fine centimeter-level decimal tokens through vocabulary classification rather than absolute-value regression.This provides the paper’s interpretation of how standard language tokenization can represent precise numerical coordinates.
  • 3.3 PROMPTING-REASONING-FINETUNING: The method uses universal motion-planning context and frame-specific language descriptions of observations and ego-states as LLM inputs.Parameterized inputs are represented in language, addressing the mismatch between heterogeneous planner data and language-model inputs.
  • 3.3 PROMPTING-REASONING-FINETUNING: Prompting, chain-of-thought reasoning, and fine-tuning produce both planned trajectories and explicit reasoning aligned with human driving behavior.The reasoning process identifies critical objects and their effects, while fine-tuning uses human driving trajectories and generated reasoning guidance.

4 EXPERIMENTS

Experiments on nuScenes evaluate GPT-Driver’s planning performance, few-shot generalization, fine-tuning choices, and interpretability, while also identifying open deployment limitations.

  • Experimental Settings: The evaluation uses nuScenes, a real-world dataset containing 1000 driving scenarios and approximately 40000 key frames.The model is fine-tuned on training data and evaluated on the validation set.
  • Comparison Against the State-of-the-Art Methods: GPT-Driver significantly outperforms prior works in L2 error while performing on par with top methods in collision rate.The authors associate L2 with imitation ability and collision rate with planning safety.
  • Few-Shot Motion Planning: With only 10% of the training scenarios, GPT-Driver attains decent validation performance while UniAD fails to obtain good performance under limited data.Both approaches use the same pretrained detection and prediction modules as inputs for fair comparison.
  • Interpretability: GPT-Driver identifies critical objects, assesses their potential effects, and generates both high-level actions and sensible trajectories with natural-language reasoning.The visualization includes planned trajectories and notable objects, while the images themselves are used only for illustration.
  • Few-Shot Motion Planning: Fine-tuning performs significantly better than in-context learning for instructing the LLM in motion planning.The comparison uses training inputs and expected outputs as in-context exemplars for the latter strategy.
  • Limitations: The real-time suitability of GPT-Driver remains uncertain because the OpenAI APIs do not provide its inference time.The authors note that GPT-based planning would typically take longer than MLP-based planning and leave optimization for future work.
  • Limitations: Open-loop evaluation does not fully emulate error accumulation, so close-loop evaluation of GPT-Driver is left for future work.The authors identify this as another limitation of the motion-planning evaluation.

5 CONCLUSION

The conclusion presents GPT-Driver as a GPT-3.5-based autonomous-driving motion planner built by reformulating motion planning as language modeling. Experiments demonstrate planning performance, generalization, and interpretability, while future work targets inference time and additional sensor observations.

  • 5 CONCLUSION: GPT-Driver transforms OpenAI GPT-3.5 into a motion planner for autonomous driving.The method reformulates motion planning as a language modeling problem and uses prompting-reasoning-finetuning.
  • 5 CONCLUSION: Extensive experiments demonstrate GPT-Driver’s planning performance, generalization, and interpretability compared to existing works.
  • 5 CONCLUSION: Future work includes optimizing inference time and incorporating additional sensor observations such as high-definition maps into input prompts.
Loading 2310.01415v3…