PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Multimodal Large Language Models for Autonomous Driving

Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun Tang, Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, Ziran Wang, Chao Zheng

arXiv:2311.12320v1cs.AI

TL;DR

Autonomous-driving research lacks a comprehensive understanding of how MLLMs can be applied and what challenges remain. This paper systematically surveys MLLM development, driving applications, datasets, benchmarks, and the first LLVM-AD workshop, concluding that the field offers promising directions while requiring further work on data and integration challenges.

  • Problem

    Applying MLLMs to autonomous driving remains insufficiently understood despite their potential to integrate diverse sensory inputs with language reasoning.

  • Method

    The paper reviews MLLM and autonomous-driving foundations, applications, datasets, benchmarks, industry systems, workshop papers, and future research themes.

  • Results

    The survey synthesizes current research and LLVM-AD insights, including new datasets and efforts to improve MLLM algorithms for autonomous driving.

  • Takeaways & Limitations

    MLLMs offer potential for more intuitive, efficient, human-centric, safer, transparent, and personalized autonomous vehicles.

  • Takeaways & Limitations

    Traffic MLLM datasets remain limited in scale and quality and must represent modalities including 3D point clouds, panoramic imagery, HD maps, and regulations.

Abstract

from arXiv · show

With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors to apply in LLM driving systems. In this paper, we present a systematic investigation in this field. We first introduce the background of Multimodal Large Language Models (MLLMs), the multimodal models development using LLMs, and the history of autonomous driving. Then, we overview existing MLLM tools for driving, transportation, and map systems together with existing datasets and benchmarks. Moreover, we summarized the works in The 1st WACV Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD), which is the first workshop of its kind regarding LLMs in autonomous driving. To further promote the development of this field, we also discuss several important problems regarding using MLLMs in autonomous driving systems that need to be solved by both academia and industry.

1. Introduction

MLLMs combine language reasoning with visual, auditory, and other sensory inputs, creating opportunities across autonomous-driving perception, planning, control, and interaction. This survey reviews the field’s foundations, applications, datasets, workshop contributions, and unresolved challenges.

  • MLLM Foundations: MLLMs integrate LLM reasoning with image, video, and audio data for multimodal understanding.This alignment supports tasks including image categorization, text-video matching, and speech detection.
  • Challenges: GPT-4V identifies objects in driving scenes but struggles to recognize critical traffic elements such as lane information.The example illustrates limitations relevant to reliable autonomous-vehicle navigation.
  • Challenges: 27?
  • Survey Scope: The survey covers MLLM and autonomous-driving backgrounds, applications, datasets, industry systems, and future directions.It also reviews papers from the first WACV LLVM-AD workshop and discusses research challenges.
  • Applications: In autonomous driving, LLMs are being explored for perception, motion planning, motion control, navigation, and human-vehicle interaction.External information sources such as HD maps, traffic reports, and weather updates can complement onboard sensing.

2. Development of Autonomous Driving

Autonomous driving has progressed from early sensor-based experiments to standardized automation levels, deep-learning systems, and increasingly deployed platforms. The field’s development has also emphasized control improvements, safety, explainability, and verification.

  • Early Development: Early autonomous-driving research used stereo cameras, sonars, and laser scanning for lane keeping and obstacle avoidance.The Carnegie Mellon ALV project exemplified this early sensor-based approach.
  • Standardization: The SAE classification introduced six automation levels, providing widely acknowledged milestones for autonomous-driving research and development.The classification was published in 2014.
  • Learning-Based Systems: Deep neural networks advanced object detection, scene understanding, and vehicle localization, while deep reinforcement learning improved control and motion planning.These methods support adaptation to dynamic and uncertain driving conditions.
  • Platforms: Open-source and commercial platforms include Tesla Autopilot, NVIDIA DRIVE, Autoware.AI, Baidu Apollo, and PonyAlpha.These platforms reflect the diversification of autonomous-driving software systems.
  • Safety and Trust: Recent research increasingly targets safer, more reliable, explainable, and verifiable autonomous systems.Trustworthy AI addresses concerns associated with black-box deep neural networks.

3. Development of Multimodal Language Models

Multimodal modeling evolved from handcrafted vision-language tasks to deep, pretrained models that connect diverse data modalities. LLM-based agents now extend this trajectory toward embodied reasoning and autonomous driving, although driving applications require integration of complex sensory and regulatory data.

  • Embodied Agents: LLM-based autonomous agents use pretrained knowledge to generate action plans and executable policies for embodied tasks.Examples include lifelong learning, skill libraries, iterative prompting, and language-guided robot trajectories.
  • Driving Applications: Autonomous-driving studies convert visual information into text or decompose user commands into intermediate reasoning steps and action descriptions.
  • Early Multimodal Models: Early multimodal AI focused on image or video captioning and text-based image retrieval using rule-based methods and handcrafted features.
  • Deep Learning Era: Deep learning introduced CNN- and RNN-based vision-language models that mapped image and text features into shared semantic spaces.Researchers also extended multimodal learning to audio, speech, and 3D data.
  • Pretraining: Pretraining multimodal models on extensive datasets before task-specific fine-tuning produced performance improvements across applications.This direction was inspired by pretrained NLP models such as BERT, T5, and GPT.
  • MLLMs: MLLMs use LLMs to process multiple modalities and exhibit emergent abilities such as image-based storytelling and OCR-free mathematical reasoning.These capabilities distinguish them from many traditional multimodal methods.

4. Multimodal Language Models for Autonomous Driving

MLLMs extend language models with multimodal inputs to support traffic-scene understanding, planning, control, explanation, and human-vehicle interaction. The surveyed work spans foundational models, autonomous-driving applications, industry initiatives, and generative world models.

  • Perception: MLLMs analyze images and point clouds through text, improving zero-shot and few-shot classification, segmentation, and object detection.This multimodal text-based analysis extends perception beyond predefined visual categories.
  • Perception: Driving-oriented systems apply multimodal inputs to textual driving responses, hazard identification, intention prediction, and BEV-language reasoning.Examples include DriveGPT4, HiLM-D, and Talk2BEV.
  • Generative models: Generative models such as GAIA-1 and UniSim combine video, text, actions, and diverse driving data to generate or replicate realistic driving interactions.WOSAC introduced a public challenge for realistic, interactive-agent simulation.
  • Planning, control, and interaction: LLMs and MLLMs are also used for language-mediated planning, robot control, human-machine interaction, explainability, and controller-parameter adaptation.Reported applications include translating natural-language commands into executable actions and generating explanations aligned with downstream controllers.
  • MLLM capabilities: MLLMs process and fuse visual signals with linguistic instructions to support navigation and broader autonomous-driving tasks.Their potential includes understanding traffic scenes and improving driving decision-making.
  • Industry initiatives: Industry initiatives integrate vision, language, and action, generate map and traffic context, predict multi-agent motion, and support scene understanding and behavior prediction.Examples include LINGO-1, GAIA-1, MAPLM, MotionLM, and related systems.

5. Datasets and Benchmarks

Autonomous-driving datasets have progressed from primarily 2D camera annotations toward multimodal 3D sensing and language-grounded traffic understanding. However, existing traffic language datasets remain limited in scale and quality, and MLLM performance is still below human level.

  • Vision datasets: Earlier autonomous-driving datasets focused mainly on 2D bounding boxes and masks for RGB images, while human-level driving requires 3D perception and localization.The section contrasts conventional 2D annotation with the requirements of real-world autonomous driving.
  • Vision datasets: KITTI, nuScenes, and Waymo Open provide multimodal sensor data and large collections of scenes for 3D perception, mapping, detection, tracking, and related tasks.KITTI combines stereo cameras and LiDAR with 3D-box annotations and several benchmark tasks.
  • Multimodal-language datasets: Multimodal-language driving datasets comprise texts added to existing nuScenes data and independently collected datasets such as BDD-X and DRAMA.Talk2Car, nuScenes-QA, DriveLM, and NuPrompt belong to the first category.
  • Limitations: Traffic-scene language understanding remains below human performance because data-text pairs combine 3D point clouds, panoramic imagery, HD maps, and traffic regulations.The surveyed datasets are described as limited in scale and quality for these challenges.

6. LLVM-AD Workshop Summary

LLVM-AD brought academic and industry participants together to examine MLLMs for autonomous driving and launched open-source real-world traffic language datasets with associated challenges. Its accepted papers covered language integration, vision systems, computational efficiency, data handling, and evaluation.

  • Workshop and datasets: LLVM-AD was held with WACV 2024 as a collaborative forum for applying MLLMs to autonomous driving.The half-day workshop included paper presentations, demonstrations, and invited talks.
  • Workshop and datasets: The workshop launched two open-source real-world traffic language understanding datasets and two challenges evaluating language and computer-vision models.These initiatives targeted practical autonomous-driving challenges.
  • Workshop datasets: MAPLM uses HD-map labels to generate descriptive traffic-scene paragraphs across camera images, point clouds, and BEV images.The dataset connects computer vision, mapping, and natural-language processing.
  • Workshop datasets: The UCU Dataset targets natural-language understanding of safety, driving, comfort, and vehicle-specific commands.Examples include emergency braking, cruise control, climate settings, and sunroof operation.
  • Accepted papers: The nine accepted papers addressed user-vehicle interaction, motion planning, vehicle control, vision-based planning, 3D reconstruction, optimization, localization, annotation, and evaluation.The workshop therefore covered both MLLM integration and supporting autonomous-system technologies.
  • Workshop significance: Collectively, the workshop contributions advanced the integration of language models and related technologies into autonomous systems.The reported direction emphasizes more intuitive, efficient, and human-centric autonomous vehicles.

7. Discussion

The discussion identifies opportunities for LLMs and MLLMs in autonomous driving, including personalization, user interaction, map understanding, and transparent decision-making. It also emphasizes constraints involving data coverage, real-time hardware, latency, and the limited maturity of real-vehicle deployment.

  • Datasets: New large-scale datasets should combine panoramic images, 3D point clouds, HD map annotations, diverse traffic scenarios, and corner cases.Existing datasets are limited in scale and quality and may not robustly benchmark visual-language understanding in driving scenes.
  • Hardware Support: Real-time autonomous-driving planners require low-latency reasoning and control, while LLM FLOPs increase latency and power consumption.Remote deployment additionally faces bandwidth challenges when transferring perception information and control decisions.
  • Hardware Support: Navigation planning tolerates higher response times and fewer queries, making remote-server deployment more practical than latency-sensitive driving planning.The hardware performance demand is therefore easier to meet for navigation planners.
  • User-Vehicle Interaction: LLMs can interpret driver intentions as vehicle commands, but takeover-related intentions require planner-like low-latency hardware.Higher latency may be acceptable for unrelated functions such as entertainment control.
  • Personalized Autonomous Driving: LLM personalization can adapt to driving patterns, user preferences, spoken commands, and changing road conditions, but real-vehicle research remains limited.Most recent personalization studies use simulations rather than actual vehicles.
  • Trustworthy and Safe Autonomous Driving: LLM-based systems can explain complex decisions in understandable language, while zero-shot adaptation supports responses to previously unseen driving circumstances.The paper connects these capabilities with improved transparency, user trust, safety, and driving experience.

8. Conclusion

The survey examines how MLLMs can be integrated into next-generation autonomous driving systems through historical review, literature synthesis, benchmark analysis, and workshop findings. It concludes by identifying challenges, opportunities, and research directions for further exploration.

  • The survey connects the previously distinct developments of MLLMs and autonomous driving to examine their integration in next-generation driving systems.
  • It reviews algorithms, applications, current research, and benchmarking datasets for applying multimodal language models to autonomous driving.
  • It synthesizes findings from the first LLVM-AD workshop, including proposed datasets and improvements to MLLM algorithms for autonomous driving.
  • The survey discusses future research themes, challenges, opportunities, and the potential of MLLMs in autonomous driving.
Loading 2311.12320v1…