Source-linked AI summary
DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
Erfei Cui, Wenhai Wang, Zhiqi Li, Jiangwei Xie, Haoming Zou, Hanming Deng, Gen Luo, Lewei Lu, Xizhou Zhu, Jifeng Dai
TL;DR
Autonomous-driving models can struggle with corner cases and complex reasoning, while language-based driving decisions remain difficult to convert into reliable closed-loop control. DriveMLM addresses this gap by aligning multimodal LLM outputs with behavioral-planning states and training an MLLM planner with generated decision-state explanations. It achieves 76.1 DS and 0.955 MPI on CARLA Town05 Long, reported as 4.7 points and 1.25 times better than Apollo.
Problem
Existing driving models have limited real-world understanding for corner cases, while LLM decisions are difficult to convert into reliable vehicle-control commands for closed-loop driving.
Method
DriveMLM aligns LLM outputs with behavioral-planning states, uses an MLLM with multimodal inputs and language instructions, and generates annotated training data through an efficient data engine.
Results
76.1 DS and 0.955 MPI were achieved on CARLA Town05 Long, reported as 4.7 points and 1.25 times better than Apollo.
Takeaways & Limitations
DriveMLM demonstrates closed-loop autonomous driving in realistic simulators while providing natural-language explanations and plug-and-play integration with modular AD systems.
Abstract
from arXiv · showhide
Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multimodal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (e.g., camera, lidar) as input and makes driving decisions and provide explanations; This model can plug-and-play in existing AD systems such as Autopilot and Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that replacing the decision-making modules of the Autopilot and Apollo with DriveMLM resulted in significant improvements of 3.2 and 4.7 points on the CARLA Town05 Long respectively, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs.
1 Introduction
DriveMLM addresses the difficulty of converting language-based driving decisions into reliable closed-loop vehicle control by aligning LLM outputs with behavioral-planning states. It combines a multimodal planner, data engine, and integration strategy for modular autonomous-driving systems.
- Existing end-to-end driving models often lack sufficient real-world understanding for corner cases and scenarios requiring complex reasoning.
- Language-based LLM driving decisions are difficult to convert into vehicle-control commands for closed-loop driving.
- DriveMLM aligns LLM outputs with behavioral-planning decision states and predicts them from multimodal sensor inputs, traffic rules, system messages, and user instructions.
- The framework uses a data engine to generate decision states and corresponding explanations for model training and evaluation.
- DriveMLM integrates with modular systems such as Apollo and AutoPilot without significant changes, enabling closed-loop driving.
- 76.1 DS and 0.955 MPI on CARLA Town05 Long were reported, with results 4.7 points and 1.25 times better than Apollo.
2 Related Work
Related work spans multimodal language models, agent interaction, autonomous-driving architectures, and closed-loop simulators. The paper identifies a gap between LLM driving outputs and reliable real-time control in realistic closed-loop evaluation.
- 2.1 Multi-modal Large Language Models: Multimodal LLMs extend language models with the ability to analyze and comprehend information from diverse modalities.
- 2.2 LLM-based Agents: LLM-based agent systems integrate language models with modal APIs to support complex open-world tasks such as image, video, and audio processing.
- 2.3 Autonomous Driving Models: Autonomous-driving research includes modular frameworks such as Apollo and Autoware, end-to-end models, finite-state decision systems, and open-source simulators.
- 2.3 Autonomous Driving Models: CARLA provides comprehensive sensor simulation and realistic environments and is widely used to evaluate closed-loop driving performance.
- 2.3 Autonomous Driving Models: Prior LLM driving studies predict decisions, driving signals, trajectories, or control signals, while DriveGPT4 is limited by monocular-camera input.
- 2.3 Autonomous Driving Models: Existing LLM-based driving works lack realistic-simulator closed-loop evaluation because linguistic outputs are hard to transform into reliable control signals or direct control prediction remains difficult for real-time driving.
3 Proposed Method
DriveMLM aligns multimodal language-model outputs with executable behavioral-planning decisions, then uses an MLLM planner and an efficient data engine to support closed-loop driving and explanations.
- 3.1 System Overview: The framework has three components: behavioral-planning state alignment, an MLLM planner, and an efficient data-collection strategy.
- 3.2 Behavioral Planning States Alignment: Behavioral-planning alignment converts linguistic decisions into outputs that existing modular systems can transform into vehicle-control signals.
- 3.3 MLLM Planner: The MLLM planner tokenizes multi-view images, LiDAR, system messages, and user instructions before decoding decisions and explanations.
- 3.2 Behavioral Planning States Alignment: Speed states are KEEP, ACCELERATE, DECELERATE, and STOP, while path states are FOLLOW, LEFT CHANGE, RIGHT CHANGE, LEFT BORROW, and RIGHT BORROW.
- 3.2 Behavioral Planning States Alignment: At each time step, one speed decision and one path decision are mutually inferred and sent to the motion-planning framework.
- 3.3 MLLM Planner: An image domain intermediary aligns LiDAR features with image features, using a frozen ViT-L/14 image encoder and an SST LiDAR encoder trained with cosine similarity.
- 3.3 MLLM Planner: The system message specifies the driving task, traffic rules, decision-state definitions, and modality placeholders, while outputs contain decisions and explanations.
- 3.4 Efficient Data Engine: The data pipeline collects varied realistic scenarios and expert driving interactions, then generates decision states and explanation annotations.
4 Experiments
DriveMLM is evaluated through data analysis, open-loop understanding, closed-loop driving, qualitative scenarios, and ablations. The experiments show stronger decision prediction, explanations, and closed-loop performance, while also identifying computational and sensor-modality trade-offs.
- 4.1 Data Analysis: The dataset contains 280 hours of data across 50k routes, 30 scenarios, eight CARLA maps, varied weather and lighting, four cameras, and LiDAR.Each recorded frame includes corresponding explanations and accurate decisions for safely completing scenarios.
- 4.1 Data Analysis: Behavioral-planning-state alignment converts MLLM outputs into control signals for closed-loop driving, while human interaction annotations pair instructions with responding decisions and explanations.These features target both vehicle control and customized language-instruction understanding.
- 4.3 Evaluation of Driving Knowledge: DriveMLM achieves the highest open-loop decision accuracy, surpassing LLaVA with 40.97% accuracy, and its higher F1-score than Apollo indicates stronger performance across path and speed decisions.The evaluation also measures explanation quality with BLEU-4, CIDEr, and METEOR.
- 4.5 Ablation Study: Multi-view images improve path and speed F1-scores and increase accuracy by 18.19%, while temporal QFormer improves performance by 7.4% and point clouds show little enhancement.Temporal QFormer also produces a 0.05 improvement in average speed-decision F1-score; accelerating the network remains relevant for real-time decision-making.
- 4.4 Evaluation in Closed-Loop Driving: DriveMLM handles unknown obstacles, emergency vehicles, and conditional overtaking by using borrow-lane, yielding, or refusal decisions based on traffic conditions.These qualitative cases contrast with methods that overlook obstacles, stop, or fail to yield to emergency vehicles.
5 Conclusion
DriveMLM uses a multimodal LLM to model behavioral planning in modular autonomous-driving systems, enabling closed-loop driving and natural-language explanations. It outperforms the Apollo baseline on CARLA Town05 Long.
- DriveMLM enables closed-loop autonomous driving in realistic simulators by modeling a modular system’s behavior planning with an MLLM.
- DriveMLM generates natural-language explanations for driving decisions, supporting greater transparency and trustworthiness.
- DriveMLM outperforms the Apollo baseline on the CARLA Town05 Long benchmark.
Declarations
The paper reports author affiliations, data-availability information, competing interests, and author contributions.
- Data Availability: The authors state that datasets generated and analyzed in the study are available from the corresponding author on reasonable request.
- Competing Interests: Wenhai Wang’s editorial role is disclosed, and the authors report no other competing interests.
- Authors’ Contributions: All authors contributed to the study’s conception and design, with contributions spanning methodology, analysis, writing, resources, and supervision.
- Author Details: The listed affiliations include institutions in China, Hong Kong, and the United States.