Source-linked AI summary
LMDrive: Closed-Loop End-to-End Driving with Large Language Models
Hao Shao, Yuxuan Hu, Letian Wang, Steven L. Waslander, Yu Liu, Hongsheng Li
TL;DR
Autonomous-driving systems commonly use fixed-format inputs and remain challenged by unforeseen events and complex urban scenarios. LMDrive combines natural-language instructions with camera-LiDAR data in an end-to-end closed-loop model, alongside a 64K-clip dataset and LangAuto benchmark. Closed-loop experiments demonstrate its effectiveness, while ablations show substantial dependence on key design components.
Problem
Fixed-format sensor and waypoint inputs limit autonomous vehicles’ language comprehension and interaction with humans and the environment.
Method
LMDrive combines camera-LiDAR data and natural-language navigation or notice instructions in an end-to-end closed-loop driving framework.
Results
Extensive closed-loop experiments demonstrate LMDrive’s effectiveness, with the paper also releasing about 64K clips and the LangAuto benchmark.
Takeaways & Limitations
LMDrive provides a language-based closed-loop autonomous-driving framework, dataset, and benchmark for studying instruction-following in complex scenarios.
Abstract
from arXiv · showhide
Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive
1. Introduction
LMDrive addresses the limitations of fixed-format autonomous-driving inputs by combining natural-language instructions with multimodal sensor data in a closed-loop, end-to-end framework. It also introduces a language-guided dataset and benchmark for realistic instructional driving.
- Existing modular and end-to-end systems rely on fixed-format inputs, limiting language understanding and interaction with humans and the environment.
- Language-aware autonomous vehicles are intended to handle long-tail urban events and adapt to passengers’ sudden notices through natural language.
- LMDrive processes camera-LiDAR data and natural-language driving instructions to directly generate vehicle control signals for closed-loop driving.
- The framework integrates a frozen pre-trained LLM with multimodal encoders and learnable input/output adapters for autonomous driving.
- The paper contributes about 64K instruction-following clips, the LangAuto benchmark, and extensive closed-loop experiments.
2. Related Works
Prior work has integrated LLMs into autonomous driving mainly through open-loop perception-to-text-to-control pipelines. LMDrive targets the unresolved closed-loop setting, where cumulative error, temporal consistency, and end-to-end trainability matter.
- Recent end-to-end driving methods fuse multimodal sensors or combine trajectory planning with direct control, while related LLM systems explore language and visual reasoning.
- Many LLM-based driving approaches convert scene perceptions and navigation commands into text, generate textual decisions, and translate them into executable controls.
- DRIVEGPT4 predicts control signals from frame sequences but lacks an input command for following specific navigation instructions.
- Existing LLM driving research largely evaluates open-loop settings and leaves cumulative error, temporal action consistency, and end-to-end trainability unaddressed.
- LMDrive is presented as the first language-based end-to-end autonomous-driving method evaluated in a closed-loop setting, with associated datasets, benchmarks, and models released.
3. Dataset generation
The dataset is built by collecting expert multimodal driving trajectories, segmenting them into instruction-aligned clips, and annotating navigation and optional notice instructions. Its instruction design covers varied natural-language and safety-relevant scenarios.
- The agent uses three input sources—camera-LiDAR data, navigation instructions, and human notice instructions—to generate scene-compliant, instruction-following actions.
- Dataset construction first collects sensor data and control signals with an expert agent, then parses and labels the trajectories with instructions.
- The expert-generated raw dataset contains about 3M driving frames across 2.5k routes, 8 towns, and 21 environmental conditions.
- Parsing segments frame sequences into clips aligned with individual navigation instructions and adds notice instructions when adversarial events occur.
- Instruction design includes follow, turn, and other navigation types plus notice instructions, totaling 56 instruction types with diversified language variants.
4. LMDrive methodology
LMDrive uses a multimodal vision encoder to convert camera-LiDAR observations into visual tokens and an LLM to combine those tokens with language instructions. The model is pretrained for driving perception, then instruction-finetuned to predict controls and instruction completion.
- Architecture: The architecture combines a vision encoder for camera-LiDAR scene understanding with an LLM, tokenizer, Q-Former, and adapters for control prediction.
- Vision encoder: The vision encoder fuses image and LiDAR features in a BEV decoder, producing BEV, waypoint, and traffic-light visual tokens for the language model.
- Vision encoder: Image views are encoded with a 2D backbone and transformer, while LiDAR is encoded with a 3D PointPillars backbone before multimodal fusion.
- Training: Vision pretraining uses object detection, future waypoint prediction, and traffic-light status classification before prediction heads are removed and the encoder is frozen.
- Action prediction: The LLM processes historic visual tokens and language instructions to predict action tokens, future waypoints, and whether each instruction is complete.
- Training: Training has separate vision-encoder pretraining and instruction-finetuning stages that align visual and instructional inputs with control signals.
5. LangAuto Benchmark
LangAuto is a closed-loop CARLA benchmark that evaluates autonomous driving under natural-language navigation and notice instructions across diverse scenarios and conditions.
- LangAuto is the first benchmark to evaluate closed-loop driving performance under language instructions.
- Unlike prior CARLA benchmarks using discrete commands or target waypoints, LangAuto provides navigation and optional notice instructions in natural language.
- The benchmark covers all 8 publicly available CARLA towns, including highways, intersections, and roundabouts.
- LangAuto includes 16 environmental conditions formed from 7 weather and 3 daylight conditions.
- Driving score combines route completion and infraction score, jointly measuring progress and safety.
6. Experiments
Experiments evaluate LMDrive’s backbone choices, module design, notice handling, and sequential instruction following in closed-loop CARLA settings.
- 6.2. Quantitative Results: LLaVA-v1.5 surpasses the other tested 7B LLM backbones, while a randomly initialized model struggles to drive properly.
- 6.2. Quantitative Results: Removing Q-Former reduces average driving score from 36.2 to 31.7, while removing BEV tokens lowers infraction score from 0.81 to 0.72.
- 6.2. Quantitative Results: Without visual pre-training, driving score drops to 16.9, demonstrating the importance of pre-training the vision encoder.
- 6.2. Quantitative Results: Notice instructions supplied during adversarial events significantly decrease collisions and traffic-rule violations.
- 6.2. Quantitative Results: Figure 6 shows that LMDrive changes predicted waypoints for different instructions and rejects a misleading right-turn instruction with slower speed and a safe path.
- 6.2. Quantitative Results: Merging 2 to 3 navigation instructions into longer commands decreases both driving score and route completion ratio for LLaVA- and Vicuna-based agents.
7. Conclusion
The paper concludes that LMDrive combines natural-language instructions with multimodal sensor data for closed-loop autonomous driving and supports further language-based driving research.
- LMDrive integrates natural-language instructions with multimodal sensor data for interaction in complex driving scenarios.
- The work releases around 64K multimodal data clips with corresponding navigation instructions and establishes the LangAuto language-guided driving benchmark.
- Extensive closed-loop experiments demonstrate LMDrive’s effectiveness and motivate further exploration of language-based closed-loop end-to-end autonomous driving.
Supplementary Material
The supplementary material visualizes weather and daylight conditions used in the LangAuto benchmark.
- Figure 7 visualizes weather and daylight conditions included in the LangAuto Benchmark.
A. Implementation Details
The implementation uses multi-view camera and LiDAR inputs with a vision encoder, Q-Former, and specified training settings for LMDrive.
- Sensor configuration: The sensor setup includes four RGB cameras and one LiDAR, with 800 × 600 camera resolution, 100° horizontal FOV, and 10Hz LiDAR rotation.The side cameras are angled at 60°; the LiDAR field of view is 10°/-30° with 64 channels.
- Training details: Vision-encoder pretraining uses AdamW, cosine scheduling, 35 epochs, five warm-up epochs, random scaling, and color jittering.The learning rates are BatchSize 512 ×5e−4 for the transformer encoder and 3D backbone, and BatchSize 512 ×2e−4 for the 2D backbone.
- Training details: Instruction finetuning uses a 1e−4 learning rate with batch size 32, 15 epochs, 2,000 warm-up steps, weight decay 0.07, and a 40-frame history horizon.Clips exceeding 40 frames are truncated to retain the most recent frames.
B. Additional Experiments
Additional experiments examine temporal sampling and notice-instruction usage, identifying narrower horizons and omitted notices as detrimental settings.
- Notice instructions: Removing all notice instructions produces worse driving and infraction scores, suggesting that training with notices improves performance.The method randomly removes 75% of notice instructions to avoid overfitting, and evaluation uses LangAuto-Short.
C. Benchmark Details
LangAuto evaluates language-guided driving across multiple benchmark tracks, maps, environmental conditions, and diverse navigation and notice instructions, including misleading and connected commands.
- Benchmark scope: LangAuto includes 12 shown environmental conditions, 16 benchmark conditions overall, and eight town maps.The benchmark statistics are reported across LangAuto, LangAuto-Short, and LangAuto-Tiny tracks.
- Instruction types: Misleading instructions are paired with driving scenarios and can violate traffic rules, while connected instructions provide chained navigation commands.The paper gives a single-lane example in which changing to the left-hand lane violates traffic rules.