Source-linked AI summary
LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, Mingyu Ding
TL;DR
Learning-based AD systems face difficulties with high-level understanding, rare events, and interpretability. LanguageMPC uses an LLM for high-level reasoning and translates its decisions into MPC guidance through parameter adaptation, with experiments reporting improved single-vehicle performance and complex coordinated behaviors.
Problem
Existing learning-based AD systems rely on predefined rules or reward designs, struggle with rare scenarios, and provide limited interpretability.
Method
LanguageMPC integrates an LLM for high-level scenario reasoning and decision-making with MPC, converting textual decisions into mathematical controller guidance.
Results
LanguageMPC surpasses existing learning-based and optimization-based methods in single-vehicle tasks and manages multi-vehicle coordination and text-guided driving behavior.
Takeaways & Limitations
The results support using LLMs as high-level decision-makers for complex autonomous-driving scenarios involving safety, efficiency, adaptability, and interpretability.
Takeaways & Limitations
LLMs may respond too slowly for sudden, highly time-critical events occurring on sub-second timescales.
Abstract
from arXiv · showhide
Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability. To address these problems, this work employs Large Language Models (LLMs) as a decision-making component for complex AD scenarios that require human commonsense understanding. We devise cognitive pathways to enable comprehensive reasoning with LLMs, and develop algorithms for translating LLM decisions into actionable driving commands. Through this approach, LLM decisions are seamlessly integrated with low-level controllers by guided parameter matrix adaptation. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination, thanks to the commonsense reasoning capabilities of LLMs. This paper presents an initial step toward leveraging LLMs as effective decision-makers for intricate AD scenarios in terms of safety, efficiency, generalizability, and interoperability. We aspire for it to serve as inspiration for future research in this field. Project page: https://sites.google.com/view/llm-mpc
I. INTRODUCTION
Existing learning-based AD systems struggle with predefined rules, rare scenarios, and opaque decision processes. LanguageMPC uses an LLM for high-level reasoning and translates its decisions into controller guidance, improving reported performance and interpretability.
- Predefined rules and reward designs limit existing learning-based AD systems’ ability to generalize across situations.
- Limited datasets and sampling efficiency make rare real-world driving decisions difficult to learn.Chauffeurnet reportedly found that even 30 million state-action samples were insufficient for an optimal policy.
- Black-box decision processes hinder interpretability, targeted optimization, regulatory recognition, and practical implementation.
- LanguageMPC employs an LLM as the AD system’s high-level reasoning component and converts its decisions into mathematical guidance for MPC.
- The system also manages multi-vehicle coordination and driving-behavior adjustment from textual inputs.
II. RELATED WORK
LanguageMPC combines an LLM’s high-level reasoning with MPC’s real-time control to address the gap between language-based decision-making and executable driving actions. Its pipeline encodes scenarios, guides actions, and adjusts controller parameters.
- Prior autonomous-driving research uses modular or end-to-end learning approaches, but modular systems face compatibility, error accumulation, and inefficiency challenges.
- Existing LLM-based work demonstrates perception, reasoning, and high-level planning, yet often lacks direct generation of final control actions.
- LanguageMPC directly combines the LLM with MPC so LLM reasoning guides final control-action generation without relying on a learned trajectory decoder.
- A prompt generator converts real-time scenario data and traffic-rule instructions into descriptive text for the LLM.
- The LLM performs scenario encoding, action guidance, and weight adjustment before textual decisions are converted into mathematical representations.
- MPC executes LLM decisions under soft constraints in a dual-frequency system balancing slower inference with real-time control.
A. MPC Controller
The MPC controller optimizes a weighted cost over a finite horizon while LanguageMPC adapts weights, action bias, and observations to scenario-specific LLM decisions. This preserves real-time control while changing trade-offs among tracking, action, smoothness, and safety.
- MPC solves a finite-time open-loop optimization online and applies the first control action from the optimal sequence.
- The controller minimizes expected cumulative cost over an action sequence in an MDP defined by states, actions, costs, transitions, and an initial distribution.
- The cost function uses non-negative weights and twice-differentiable norm residuals, but universal weights are impractical across diverse driving scenarios.
- LanguageMPC constructs overall cost as tracking, action, and safety terms, then adapts weight matrices using LLM high-level decisions.
- The reference trajectory is generated without signals or other traffic participants, and the controller aims to follow it.
- Acceleration and steering-angle limits provide smooth-control constraints, while the LLM supplies an action bias for deviations from nominal actions.
- A safety cost penalizes insufficient distance from other vehicles using actual and desired safety distances.
B. Scenario Encoding
LanguageMPC encodes static and dynamic scenario information for MPC, using LLM-selected trajectories, attention masks, action biases, and adaptive weights to improve context-sensitive planning.
- The observation matrix combines static environmental elements with dynamic information about surrounding vehicles for MPC cost computation.
- LanguageMPC generates three Bessel-curve reference trajectories from nearby lane centerlines and selects one using scenario context.The selected trajectory becomes the ego vehicle’s tracking target.
- The LLM allocates attention with a binary mask that filters irrelevant vehicles from the MPC dynamic observation matrix.This focuses MPC on relevant traffic participants and reduces unnecessary computation or logical conflicts.
- The midpoint of a discretized LLM-selected action interval becomes an action bias guiding MPC control actions.
- LanguageMPC dynamically selects MPC weights from a predefined pool to adapt tracking, action penalty, smoothness, and safety trade-offs across scenarios.The approach avoids relying on direct action commands from the LLM while retaining robust low-level control.
- In an unsignalized-intersection example, the LLM recognizes that vehicle 32 is yielding and focuses on the vehicle ahead.The attended vehicles are included in the MPC dynamic observation matrix.
C. Dual-frequency System
LanguageMPC uses a dual-frequency system because LLM decision-making is relatively slow: strategic high-level decisions run less frequently, while trajectory planning and control run more frequently.
- The dual-frequency system separates low-frequency strategic decisions from high-frequency trajectory planning and control.The LLM makes strategic decisions every few seconds to adapt to driving changes.
IV. EXPERIMENTS
The experiments evaluate LanguageMPC with GPT-3.5 and LangChain across single-vehicle decision-making and more complex driving behaviors, including multi-vehicle coordination.
- LanguageMPC is evaluated with GPT-3.5 as the base LLM and LangChain managing LLM text output.
- Experiments measure reasoning and decision-making across single-vehicle tasks, driving-behavior modulation, and multi-vehicle coordination.Qualitative analyses also examine challenging and rare situations.
A. Settings
The study uses SUMO with IDSim to generate dynamic traffic scenarios and evaluates five driving settings with varied road structures, traffic densities, and participant behaviors.
- Simulation uses SUMO and IDSim to generate traffic involving vehicles, bicycles, and pedestrians.
- Five evaluated scenarios are Signalized Intersections, Unsignalized Intersections, Lanes, Roundabouts, and Emergency Obstacle Avoidance.
- Ablation experiments distinguish scenario-encoder modification of Ody from LLM-generated action bias ˆA and weight matrix W.
- The traditional MPC baseline uses a fixed weight matrix and incorporates all traffic participants within sensing range.It was tuned and validated through real-world vehicle experiments at iDLab, Tsinghua University.
- The study also compares against four reinforcement-learning-based planning methods.
B. Metrics
LanguageMPC evaluates driving through failure and collision counts, traffic-flow inefficiency, travel time, safety penalties, and a weighted overall cost. The metrics are combined so their contributions are balanced by matching magnitudes.
- Failure cases count trips that do not reach the target within 200 seconds, while collisions are tracked separately as safety incidents.
- Time efficiency records how long the ego vehicle takes to reach its target area, while the safety penalty reflects surrounding-vehicle distance and deceleration.
- Overall cost combines the evaluation metrics with weighted values, whose magnitudes are matched so each metric contributes equally.
C. Single-vehicle Decision-making
LanguageMPC improves decision-making in complex single-vehicle scenarios by combining scenario understanding, attention allocation, and high-level reasoning with regulation-aware driving. Its evaluations report lower overall cost, minimal failures, no collisions, and strong performance in dynamic settings.
- Significant reductions in overall cost occur across all evaluated scenarios, including Emergency Obstacle Avoidance, with minimal failures and no collisions.
- In intersections and roundabouts, LanguageMPC slightly increases travel time while improving traffic-flow efficiency and reducing safety penalties.The passage characterizes this behavior as cautious and regulation-compliant.
- In lane-changing and Emergency Obstacle Avoidance scenarios, LanguageMPC outperforms RL and MPC across all reported metrics.
- LanguageMPC predicts surrounding vehicles’ intentions and allocates attention to the vehicle affecting the ego vehicle’s path.
- LanguageMPC uses textual guidance to change lanes and avoid a blocked road-construction area.
- Unlike RL’s intersection overtake, LanguageMPC follows the vehicle until exiting the intersection before overtaking, complying with traffic regulations.
D. Text-modulated Driving
LanguageMPC lets users modulate driving style and guide behavior with text. The system responds conservatively by following the vehicle ahead and aggressively by safely overtaking, while also adapting to road construction.
- Driving Style Adjustment: Textual instructions adjust driving style: conservative commands slow the vehicle and maintain following, whereas aggressive commands produce safe overtaking.
- Textual Guidance for Complex Scenarios: Textual guidance enables LanguageMPC to change lanes and avoid a blocked area in road-construction scenarios.
E. Multi-vehicle Joint Control
LanguageMPC combines distributed vehicle-level LLMs with a central coordinating LLM to manage complex multi-vehicle encounters. In a narrow-road case, the system assigns yielding and proceeding actions, while soft MPC constraints preserve robustness despite low-frequency LLM decisions.
- E. Multi-vehicle Joint Control: Distributed LLMs share local vehicle status with a central LLM, which issues coordinated commands for adaptive multi-vehicle decision-making.The architecture uses distributed control at each vehicle and centralized coordination for the convoy.
- E. Multi-vehicle Joint Control: In a narrow-road encounter, distributed LLMs recognize the situation and report vehicle statuses to the central LLM for coordination.This reduces the decision-making burden for a single-vehicle system in the corner case.
- E. Multi-vehicle Joint Control: One vehicle decelerates while the other proceeds, demonstrating collaboration between centralized decision-making and distributed control.The coordinated behavior is presented as effective multi-vehicle coordination in the narrow-road scenario.
- E. Multi-vehicle Joint Control: LLM constraints act as soft cost-function terms rather than directly modifying actions, allowing MPC to remain robust under low-frequency decisions.The paper reports that high-level driving decisions usually evolve slowly and coherently, while emergency pre-evaluation reduces delay.
- E. Multi-vehicle Joint Control: The conclusion reports improved safety, efficiency, and adaptability in complex and dynamic driving scenarios through LLM–MPC integration.It attributes these improvements to the reasoning capabilities and interpretability of LLMs within the integrated system.