Source-linked AI summary
LLM4Drive: A Survey of Large Language Models for Autonomous Driving
Zhenjie Yang, Xiaosong Jia, Hongyang Li, Junchi Yan
TL;DR
Autonomous driving must balance data-driven capability with the transparency limitations of end-to-end systems and the coverage gaps caused by simulation mismatch and long-tailed real-world conditions. This paper surveys LLM4AD applications, evaluation resources, challenges, and future directions, concluding that the field spans planning, perception, question answering, and generation while raising important ethical risks.
Problem
End-to-end driving reduces module integration errors but remains difficult to interpret and validate, while simulation mismatch and long-tailed real-world data limit progress toward expert-level driving.
Method
The paper systematically surveys LLM4AD applications, organizing existing work into planning, perception, question answering, and generation while summarizing related datasets.
Results
The survey classifies representative LLM applications, summarizes LLM4AD datasets, and identifies challenges and future research directions.
Takeaways & Limitations
LLM4AD is a broad research field spanning multiple autonomous-driving tasks and application roles.
Takeaways & Limitations
Applying LLMs to autonomous driving raises risks from hallucinations, bias, reasoning errors, adversarial interference, and privacy leakage.
Abstract
from arXiv · showhide
Autonomous driving technology, a catalyst for revolutionizing transportation and urban mobility, has the tend to transition from rule-based systems to data-driven strategies. Traditional module-based systems are constrained by cumulative errors among cascaded modules and inflexible pre-set rules. In contrast, end-to-end autonomous driving systems have the potential to avoid error accumulation due to their fully data-driven training process, although they often lack transparency due to their "black box" nature, complicating the validation and traceability of decisions. Recently, large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. A natural thought is to utilize these abilities to empower autonomous driving. By combining LLM with foundation vision models, it could open the door to open-world understanding, reasoning, and few-shot learning, which current autonomous driving systems are lacking. In this paper, we systematically review a research line about \textit{Large Language Models for Autonomous Driving (LLM4AD)}. This study evaluates the current state of technological advancements, distinctly outlining the principal challenges and prospective directions for the field. For the convenience of researchers in academia and industry, we provide real-time updates on the latest advances in the field as well as relevant open-source resources via the designated link: https://github.com/Thinklab-SJTU/Awesome-LLM4AD.
1 Introduction
Autonomous driving is shifting from modular, rule-based systems toward end-to-end, data-driven approaches, but end-to-end models introduce transparency challenges. The survey examines how LLMs and multimodal models may support interpretable, generalizable, open-world autonomous driving.
- Current autonomous driving paradigm: Conventional autonomous driving uses separate perception, prediction, and planning components for distinct tasks.Perception includes detection, tracking, and semantic segmentation; prediction estimates surrounding agents’ future states; planning often relies on rule-based decisions.
- End-to-end transition: End-to-end systems reduce integration errors and redundant computations through data-driven training, while improving the expression of visual and sensory information.
- End-to-end transition: End-to-end autonomous driving still has a black-box problem that complicates interpretation and validation of decisions.
- LLM4AD motivation: Large language models provide contextual understanding, answer generation, and complex-task handling, while multimodal integration creates a unified feature space across images, text, videos, and point clouds.
- LLM4AD motivation: Visual-language models may support open-world understanding and in-context learning for long-tail perception, decision assistance, and intuitive explanations.
- Survey scope: The paper surveys LLM4AD principles, methods, implementation processes, challenges, and future research directions.
2 Motivation of LLM4AD
LLM4AD is motivated by the gap between current driving competence and expert-level performance under simulation mismatch and long-tailed real-world conditions. The survey presents LLM application in autonomous driving as spanning multiple task types and pipeline roles.
- Motivation of LLM4AD: Autonomous driving seeks to progress from basic driving ability toward an advanced level comparable to experienced human drivers.
- Motivation of LLM4AD: Simulation-based learning faces a sim2real gap, while offline real-world data cannot fully cover the long-tailed distribution of autonomous driving tasks.
- Motivation of LLM4AD: High data collection and annotation costs, together with simulation–real-world differences, leave a gap before expert-level driving skills are reached.
- Scope of LLM4AD: The paper describes LLM applications in autonomous driving as covering a wide range of task types across the driving pipeline.
3 Application of LLM4AD
The survey organizes LLM4AD research into planning, perception, question answering, and generation, while reviewing fine-tuning and prompt-engineering approaches. It also covers evaluation benchmarks, datasets, and metrics used across these applications.
- LLM4AD research is organized into planning, perception, question answering, and generation according to the survey’s taxonomy.
- Fine-tuning pre-trained models: Fine-tuned models translate driving decision-making or motion planning into language-modeling problems, with some approaches outperforming task-specific baselines.MTD-GPT exceeds single-task reinforcement-learning experts, while GPT-Driver exceeds UniAD on L2 but faces comparison and metric limitations.
- Prompt engineering: Prompt-engineering methods use memory, reflection, reasoning, and raw sensor inputs to improve closed-loop driving and generalization.Their inference speed remains unresolved because reasoning and reflection can require multiple question-answering rounds.
- Perception: Perception applications exploit few-shot learning for fast learning and reasoning, using tracking, language-generation, trajectory, and displacement metrics.Examples include AMOTA, AMOTP, IDS, BLEU-4, METEOR, CIDEr, SPICE, minADEk, and minFDEk.
- Question Answering and Generation: Question-answering and generation applications support interactive vehicle assistance, traffic analysis, synthetic scenarios, video prediction, and editable driving simulations.TransGPT supports single-modal and multimodal transportation tasks, while other systems generate trajectories, HDMaps, and natural-language-controlled scenes.
- Evaluation & Benchmark: Benchmarks evaluate instruction following, safety, spatial awareness, traffic-rule adherence, and language responses through simulators, datasets, evaluators, and multiple scoring methods.LangAuto covers eight towns and sixteen environmental conditions, with route completion, violation score, and driving score indicators.
4 Datasets in LLM4AD
LLM4AD datasets extend conventional autonomous-driving data with language, reasoning, and multimodal annotations for perception, prediction, planning, and interaction. The surveyed datasets vary substantially in scale, modality, annotation type, and task complexity.
- Traditional datasets such as nuScenes lack action descriptions, detailed captions, and question-answering pairs needed for interaction with LLMs.
- Specialized datasets: BDD-X, HAD, Talk2Car, DriveLM, DRAMA, Rank2Tell, and NuPrompt add behavior explanations, advice, commands, reasoning graphs, risk labels, sensor data, and object-level language prompts.
- Question-answering datasets: 459,941 question-answer pairs from 34,149 visual scenes make NuScenes-QA a large, balanced resource spanning multiple question types and complexity levels.
- Multimodal reasoning datasets: 600,000 video-text pairs across nuScenes, Waymo, and ONCE provide object, kinematic, relational, question-answer, and reasoning annotations in Reason2Drive.
- Multimodal reasoning datasets: Approximately 419,000 question-answer pairs in LingoQA cover driving actions, scenery, environmental perception, road conditions, and free-form answers.
- Multimodal reasoning datasets: 91K multi-view video-QA pairs in NuInstruct span 17 subtasks requiring temporal, multi-view, and spatial information through automatically generated instruction-response pairs.
- Video datasets: OpenDV-2K contains 2059 hours of curated driving videos with automatically generated language annotations for generalized video prediction model training.
5 Conclusion
The paper surveys LLM4AD applications, representative approaches, and related datasets, while identifying future research directions. It also commits to monitoring continued developments in the field.
- The survey classifies LLM applications for autonomous driving and summarizes representative approaches in each category.
- It summarizes the latest datasets related to LLM4AD and highlights future research directions.
- The authors state that they will continue monitoring developments in the field.
Ethical Statement
Applying LLMs to autonomous driving raises ethical concerns involving safety, fairness, and reasoning reliability. These concerns arise when models misunderstand environments, exhibit bias, or produce false information and reasoning errors.
- Model illusions may cause vehicles to misunderstand external environments or traffic conditions, creating safety hazards.
- Model discrimination and bias may produce unfair or biased vehicle decisions across environments or groups.
- False information and reasoning errors may cause problematic vehicle behavior.