Source-linked AI summary
DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee. K. Wong, Zhenguo Li, Hengshuang Zhao
TL;DR
End-to-end driving systems can predict controls but remain difficult to interpret, motivating DriveGPT4, a multimodal LLM trained for both driving explanations and control prediction. Using BDD-X data, ChatGPT-generated instruction data, and mix-finetuning, it outperforms baselines across multiple driving tasks and supports zero-shot adaptation.
Problem
End-to-end autonomous driving can predict low-level controls directly, but its black-box decisions are difficult for humans to interpret.
Method
DriveGPT4 combines video and text inputs with a ChatGPT-assisted visual instruction dataset and mix-finetuning to generate explanations, answer questions, and predict controls.
Results
DriveGPT4 outperforms baseline models across vehicle action description, action justification, general question answering, and control signal prediction tasks on BDD-X.
Takeaways & Limitations
DriveGPT4 provides an interpretable end-to-end autonomous driving system with multimodal interaction and zero-shot adaptation within the reported evaluation scope.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion.These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.
I. INTRODUCTION
End-to-end driving can directly predict controls but is difficult to interpret, motivating DriveGPT4 as a multimodal LLM that explains vehicle behavior while generating low-level controls. Its visual instruction dataset and mix-finetuning strategy support multimodal interaction and performance across BDD-X tasks.
- End-to-end systems directly predict paths or low-level controls, but their black-box decisions raise ethical and legal concerns.
- Small language models support explainable driving responses but often produce rigid answers to predefined questions.
- DriveGPT4 processes video sequences and textual queries to explain vehicle actions, reason about behavior, and predict the next-step speed and turning angle.
- The system uses a visual instruction tuning dataset created with ChatGPT and a mix-finetuning strategy to generate text responses and low-level control signals.
- Evaluations on BDD-X across multiple tasks report that DriveGPT4 outperforms all baselines.
II. RELATED WORKS
Prior end-to-end driving methods predict paths or controls from visual inputs but lack interpretability, while related systems address narrower perception or scene-understanding problems. BDD-X provides labeled action descriptions and justifications relevant to interpretable driving.
- End-to-end Autonomous Driving: End-to-end autonomous driving directly predicts vehicle paths and low-level controls from visual inputs, but existing approaches lack interpretability.
- BDD-X examples pair vehicle actions with textual justifications, supporting interpretation of driving behavior.
- NuPrompt focuses on object tracking and does not address end-to-end driving or vehicle action reasoning.
- DriveLM is described as a large benchmark for driving scene understanding.
III. DATA GENERATION
The data-generation process converts BDD-X annotations into question-answer pairs and augments them with ChatGPT-generated conversations. The resulting 56K samples cover both driving-specific labels and diverse multimodal instruction-following interactions.
- BDD-X Dataset: BDD-X contains approximately 20,000 samples, including 16,803 training clips and 2,123 testing clips, with eight images per clip.
- BDD-X Dataset: BDD-X supplies vehicle action descriptions, action justifications, and control signals such as speed and turning angle.
- BDD-X Dataset: Synonymous questions are paired with action descriptions, action justifications, or next-frame speed and turning-angle labels to create task-specific QA data.
- Additional QAs generated by ChatGPT: ChatGPT generates conversations about traffic lights, turning directions, lane changes, surrounding objects, and spatial relations using detected object information.
- Additional QAs generated by ChatGPT: 56K video-text instruction-following samples combine 16K BDD-X QAs with 40K ChatGPT-generated QAs.
A. Model Architecture
DriveGPT4 converts multi-frame video and text inputs into language responses and next-step vehicle controls. Its architecture combines visual tokenization, temporal and spatial feature processing, and shared text/control decoding.
- Input and output: DriveGPT4 accepts videos and texts, then produces human-language responses and low-level vehicle control signals.The system predicts next-step speed and turning angle while supporting conversational descriptions and explanations.
- Visual processing: Video frames are encoded with a pretrained CLIP visual encoder, whose global and patch features represent each image.Each frame yields one global feature and 256 patch features.
- Instruction data: The instruction-tuning data combines BDD-X question-answer examples with ChatGPT-generated conversations and multimodal driving information.Examples include vehicle actions, justifications, control predictions, and additional questions about scene objects and interactions.
- Visual processing: Pooling converts frame-level patch features into a single tensor for memory-efficient temporal processing.The architecture separately forms temporal and spatial visual features before projecting them into the text domain.
- Decoding: Text and control signals share the same text-domain decoder, allowing vehicle controls to be represented and processed as language.Decoded outputs contain control signals in a fixed format.
B. Training
DriveGPT4 is trained in two stages: broad video-text alignment pretraining followed by mix-finetuning for interpretable autonomous-driving question answering.
- Training stages: Training first aligns video and text, then mix-finetunes the LLM to answer interpretable end-to-end autonomous-driving questions.The two stages are pretraining and mix-finetuning.
- Pretraining: Pretraining uses 593K CC3M image-text pairs and 703K WebVid-2M video-text pairs.These datasets cover varied topics rather than being specifically designed for autonomous driving.
V. EXPERIMENT
Experiments primarily use the BDD-X dataset because few datasets provide vehicle-camera videos with both textual interpretations and control-signal annotations. The dataset is filtered for consistency before evaluation.
- Dataset: The BDD-X testing set is divided into designated testing splits for the experiment.The testing-set split is documented in Table III.
- Dataset: BDD-X is the main evaluation dataset because few existing datasets pair vehicle-mounted-camera videos with text interpretations and control signals.The evaluation focuses on the available dataset suitable for the task.
- Data preparation: Samples with inconsistent control signals and text reasoning are removed before evaluation.This filtering is applied to the BDD-X data used in the experiments.
A. Interpretable Autonomous Driving
DriveGPT4 is evaluated on interpretable driving language generation, additional question answering, and control prediction using BDD-X. It reports strong performance across these capabilities, including flexible answers beyond fixed action descriptions and justifications.
- Evaluation setup: DriveGPT4 is compared with ADAPT and multimodal video-understanding LLMs using 8-frame videos for interpretation generation.The evaluated interpretation tasks include action descriptions, action justifications, and additional vehicle-status questions.
- Interpretation generation: The interpretation evaluation covers action description, action justification, and combined full-text generation on BDD-X.Tables IV and V report quantitative comparisons using these text-generation tasks and BLEU4.
- Control prediction: DriveGPT4 achieves superior results for both speed and turning-angle predictions on the whole BDD-X testing dataset.These control-signal results are reported in Table VI.
- Additional question answering: Additional questions are more diverse and flexible than the fixed action-description and justification formats.This evaluation tests questions generated by ChatGPT.
- Additional question answering: DriveGPT4 outperforms all baselines on additional question answering, while ADAPT cannot answer those questions beyond action descriptions and justifications.Other video-understanding LLMs can answer the questions but lack autonomous-driving domain knowledge.
B. End-to-end Control
DriveGPT4 is evaluated for next-step speed and turning-angle prediction using RMSE and threshold accuracy, and achieves superior results over the previous state-of-the-art method.
- The evaluation measures next-step speed and turning-angle predictions using RMSE and threshold accuracies Aτ at τ values of 0.1, 0.5, 1.0, and 5.0.Aτ is the proportion of test samples whose prediction errors are below τ.
- The study also presents zero-shot generalization evaluations on NuScenes and video games.
- DriveGPT4 achieves superior results for both speed and turning-angle predictions compared with the previous state-of-the-art method.
C. Qualitative Results.
Qualitative evaluations show DriveGPT4 generating text and control predictions from video, with tests extending to NuScenes and video games for zero-shot generalization.
- DriveGPT4 generates high-quality textual responses and control predictions from four visualized frames of BDD-X testing videos.
- The qualitative evaluation tests DriveGPT4 on NuScenes for zero-shot question answering and on video games for further generalization assessment.
- GPT4-V is compared with DriveGPT4 after being prompted with BDD-X question-answer pairs.
- An example comparison indicates that DriveGPT4 fails to correctly understand some dynamic vehicle actions, including turning and accelerating.
D. Ablation Studies
Ablations show that BDD-X QAs, ChatGPT QAs, and mix-finetuning each contribute to DriveGPT4’s flexible question answering and interpretable autonomous-driving capabilities.
- Removing either BDD-X QAs or ChatGPT QAs during fine-tuning decreases the corresponding performance.
- ChatGPT-generated QA pairs enable more flexible answer patterns and enhance DriveGPT4’s ability to answer BDD-X questions.
- Removing general image and video instruction-following data from mix-finetuning causes severe performance degradation.
- DriveGPT4 outperforms baselines across vehicle action description, action justification, general question answering, and control signal prediction.
- DriveGPT4 exhibits zero-shot adaptation in addition to its performance across interpretation and control tasks.