PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee. K. Wong, Zhenguo Li, Hengshuang Zhao

arXiv:2310.01412v5cs.CVcs.RO

TL;DR

End-to-end driving systems can predict controls but remain difficult to interpret, motivating DriveGPT4, a multimodal LLM trained for both driving explanations and control prediction. Using BDD-X data, ChatGPT-generated instruction data, and mix-finetuning, it outperforms baselines across multiple driving tasks and supports zero-shot adaptation.

  • Problem

    End-to-end autonomous driving can predict low-level controls directly, but its black-box decisions are difficult for humans to interpret.

    Source: §I · p. 1

  • Method

    DriveGPT4 combines video and text inputs with a ChatGPT-assisted visual instruction dataset and mix-finetuning to generate explanations, answer questions, and predict controls.

    Source: §I · pp. 1–2, A. Model Architecture · p. 3

  • Results

    DriveGPT4 outperforms baseline models across vehicle action description, action justification, general question answering, and control signal prediction tasks on BDD-X.

    Source: §I · p. 2, §VI · p. 7

  • Takeaways & Limitations

    DriveGPT4 provides an interpretable end-to-end autonomous driving system with multimodal interaction and zero-shot adaptation within the reported evaluation scope.

    Source: §VI · pp. 7–8

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion.These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.

I. INTRODUCTION

End-to-end driving can directly predict controls but is difficult to interpret, motivating DriveGPT4 as a multimodal LLM that explains vehicle behavior while generating low-level controls. Its visual instruction dataset and mix-finetuning strategy support multimodal interaction and performance across BDD-X tasks.

  • End-to-end systems directly predict paths or low-level controls, but their black-box decisions raise ethical and legal concerns.Source: §I · p. 1
  • Small language models support explainable driving responses but often produce rigid answers to predefined questions.Source: §I · p. 1
  • DriveGPT4 processes video sequences and textual queries to explain vehicle actions, reason about behavior, and predict the next-step speed and turning angle.Source: §I · pp. 1–2
  • The system uses a visual instruction tuning dataset created with ChatGPT and a mix-finetuning strategy to generate text responses and low-level control signals.Source: §I · p. 2
  • Evaluations on BDD-X across multiple tasks report that DriveGPT4 outperforms all baselines.Source: §I · p. 2

II. RELATED WORKS

Prior end-to-end driving methods predict paths or controls from visual inputs but lack interpretability, while related systems address narrower perception or scene-understanding problems. BDD-X provides labeled action descriptions and justifications relevant to interpretable driving.

End-to-end Autonomous Driving

  • End-to-end autonomous driving directly predicts vehicle paths and low-level controls from visual inputs, but existing approaches lack interpretability.Source: §II · p. 2
  • BDD-X examples pair vehicle actions with textual justifications, supporting interpretation of driving behavior.Source: Figure 1 · p. 2
  • NuPrompt focuses on object tracking and does not address end-to-end driving or vehicle action reasoning.Source: §II · p. 2
  • DriveLM is described as a large benchmark for driving scene understanding.Source: §II · p. 2

III. DATA GENERATION

The data-generation process converts BDD-X annotations into question-answer pairs and augments them with ChatGPT-generated conversations. The resulting 56K samples cover both driving-specific labels and diverse multimodal instruction-following interactions.

BDD-X Dataset

  • BDD-X contains approximately 20,000 samples, including 16,803 training clips and 2,123 testing clips, with eight images per clip.Source: A. BDD-X Dataset · p. 2
  • BDD-X supplies vehicle action descriptions, action justifications, and control signals such as speed and turning angle.Source: A. BDD-X Dataset · p. 2
  • Synonymous questions are paired with action descriptions, action justifications, or next-frame speed and turning-angle labels to create task-specific QA data.Source: A. BDD-X Dataset · p. 2

Additional QAs generated by ChatGPT

  • ChatGPT generates conversations about traffic lights, turning directions, lane changes, surrounding objects, and spatial relations using detected object information.Source: A. BDD-X Dataset · p. 3
  • 56K video-text instruction-following samples combine 16K BDD-X QAs with 40K ChatGPT-generated QAs.Source: A. BDD-X Dataset · p. 3

A. Model Architecture

DriveGPT4 converts multi-frame video and text inputs into language responses and next-step vehicle controls. Its architecture combines visual tokenization, temporal and spatial feature processing, and shared text/control decoding.

Input and output

  • DriveGPT4 accepts videos and texts, then produces human-language responses and low-level vehicle control signals.The system predicts next-step speed and turning angle while supporting conversational descriptions and explanations.Source: A. Model Architecture · pp. 3, 5

Visual processing

  • Video frames are encoded with a pretrained CLIP visual encoder, whose global and patch features represent each image.Each frame yields one global feature and 256 patch features.Source: A. Model Architecture · p. 3

Instruction data

  • The instruction-tuning data combines BDD-X question-answer examples with ChatGPT-generated conversations and multimodal driving information.Examples include vehicle actions, justifications, control predictions, and additional questions about scene objects and interactions.Source: A. Model Architecture · p. 4, Table I · p. 4

Visual processing

  • Pooling converts frame-level patch features into a single tensor for memory-efficient temporal processing.The architecture separately forms temporal and spatial visual features before projecting them into the text domain.Source: A. Model Architecture · pp. 3–4

Decoding

  • Text and control signals share the same text-domain decoder, allowing vehicle controls to be represented and processed as language.Decoded outputs contain control signals in a fixed format.Source: A. Model Architecture · pp. 3, 5

B. Training

DriveGPT4 is trained in two stages: broad video-text alignment pretraining followed by mix-finetuning for interpretable autonomous-driving question answering.

Training stages

  • Training first aligns video and text, then mix-finetunes the LLM to answer interpretable end-to-end autonomous-driving questions.The two stages are pretraining and mix-finetuning.Source: B. Training · p. 5

Pretraining

  • Pretraining uses 593K CC3M image-text pairs and 703K WebVid-2M video-text pairs.These datasets cover varied topics rather than being specifically designed for autonomous driving.Source: B. Training · p. 5

V. EXPERIMENT

Experiments primarily use the BDD-X dataset because few datasets provide vehicle-camera videos with both textual interpretations and control-signal annotations. The dataset is filtered for consistency before evaluation.

Dataset

  • The BDD-X testing set is divided into designated testing splits for the experiment.The testing-set split is documented in Table III.Source: Table III · p. 5, §V · p. 5
  • BDD-X is the main evaluation dataset because few existing datasets pair vehicle-mounted-camera videos with text interpretations and control signals.The evaluation focuses on the available dataset suitable for the task.Source: §V · p. 5

Data preparation

  • Samples with inconsistent control signals and text reasoning are removed before evaluation.This filtering is applied to the BDD-X data used in the experiments.Source: §V · p. 5

A. Interpretable Autonomous Driving

DriveGPT4 is evaluated on interpretable driving language generation, additional question answering, and control prediction using BDD-X. It reports strong performance across these capabilities, including flexible answers beyond fixed action descriptions and justifications.

Evaluation setup

  • DriveGPT4 is compared with ADAPT and multimodal video-understanding LLMs using 8-frame videos for interpretation generation.The evaluated interpretation tasks include action descriptions, action justifications, and additional vehicle-status questions.Source: A. Interpretable Autonomous Driving · p. 5

Interpretation generation

  • The interpretation evaluation covers action description, action justification, and combined full-text generation on BDD-X.Tables IV and V report quantitative comparisons using these text-generation tasks and BLEU4.Source: Table IV · p. 6, Table V · p. 6

Control prediction

  • DriveGPT4 achieves superior results for both speed and turning-angle predictions on the whole BDD-X testing dataset.These control-signal results are reported in Table VI.Source: Table VI · p. 6

Additional question answering

  • Additional questions are more diverse and flexible than the fixed action-description and justification formats.This evaluation tests questions generated by ChatGPT.Source: A. Interpretable Autonomous Driving · p. 6
  • DriveGPT4 outperforms all baselines on additional question answering, while ADAPT cannot answer those questions beyond action descriptions and justifications.Other video-understanding LLMs can answer the questions but lack autonomous-driving domain knowledge.Source: A. Interpretable Autonomous Driving · p. 6, Table VII · p. 6

B. End-to-end Control

DriveGPT4 is evaluated for next-step speed and turning-angle prediction using RMSE and threshold accuracy, and achieves superior results over the previous state-of-the-art method.

  • The evaluation measures next-step speed and turning-angle predictions using RMSE and threshold accuracies Aτ at τ values of 0.1, 0.5, 1.0, and 5.0.Aτ is the proportion of test samples whose prediction errors are below τ.Source: B. End-to-end Control · p. 6
  • The study also presents zero-shot generalization evaluations on NuScenes and video games.Source: Figure 4 · p. 7, Figure 5 · p. 7
  • DriveGPT4 achieves superior results for both speed and turning-angle predictions compared with the previous state-of-the-art method.Source: B. End-to-end Control · p. 7

C. Qualitative Results.

Qualitative evaluations show DriveGPT4 generating text and control predictions from video, with tests extending to NuScenes and video games for zero-shot generalization.

  • DriveGPT4 generates high-quality textual responses and control predictions from four visualized frames of BDD-X testing videos.Source: C. Qualitative Results · p. 7
  • The qualitative evaluation tests DriveGPT4 on NuScenes for zero-shot question answering and on video games for further generalization assessment.Source: C. Qualitative Results · p. 7
  • GPT4-V is compared with DriveGPT4 after being prompted with BDD-X question-answer pairs.Source: Figure 6 · p. 7
  • An example comparison indicates that DriveGPT4 fails to correctly understand some dynamic vehicle actions, including turning and accelerating.Source: C. Qualitative Results · p. 7

D. Ablation Studies

Ablations show that BDD-X QAs, ChatGPT QAs, and mix-finetuning each contribute to DriveGPT4’s flexible question answering and interpretable autonomous-driving capabilities.

  • Removing either BDD-X QAs or ChatGPT QAs during fine-tuning decreases the corresponding performance.Source: D. Ablation Studies · p. 7
  • ChatGPT-generated QA pairs enable more flexible answer patterns and enhance DriveGPT4’s ability to answer BDD-X questions.Source: D. Ablation Studies · p. 7
  • Removing general image and video instruction-following data from mix-finetuning causes severe performance degradation.Source: D. Ablation Studies · p. 7
  • DriveGPT4 outperforms baselines across vehicle action description, action justification, general question answering, and control signal prediction.Source: §VI · pp. 7–8
  • DriveGPT4 exhibits zero-shot adaptation in addition to its performance across interpretation and control tasks.Source: §VI · p. 8

Related papers

Loading 2310.01412v5…