PaperlayerView exact revision on arXiv

Source-linked AI summary

VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Driving

Keke Long, Haotian Shi, Jiaxi Liu, Xiaopeng Li

arXiv:2408.04821v2cs.RO

TL;DR

Autonomous driving systems face challenges from rare out-of-distribution situations and unreliable VLM responses, while the contribution of model-based control to VLM-based driving needs further evaluation. VLM-MPC pairs an upper-level VLM with a real-time MPC controller; experiments report safe PET and improved smoothness compared with VLM-based control and real-world trajectories.

  • Problem

    Autonomous driving faces rare out-of-distribution situations and VLM hallucinations, while systematic evaluation of methods for reliable FM responses remains relatively rare.

    Source: §1 · pp. 2, 4

  • Method

    VLM-MPC uses an upper-level VLM to generate control parameters from driving inputs and a lower-level MPC that controls the vehicle in real time while accounting for vehicle dynamics and engine lag.

    Source: §1 · p. 2

  • Results

    VLM-MPC consistently maintains PET above safe thresholds, improves smoothness over real-world trajectories and VLM-based control, and shows more collision risks in some VLM-based scenarios.

    Source: §4 · p. 21

  • Takeaways & Limitations

    Experiments support combining model-based control with language-model controllers for more accurate and reliable decisions in dynamic, complex driving environments.

    Source: §4 · p. 21

    The study does not account for errors in traffic-participant perception.

    Source: §2.2.2 · p. 7

Abstract

from arXiv · show

Motivated by the emergent reasoning capabilities of Vision Language Models (VLMs) and their potential to improve the comprehensibility of autonomous driving systems, this paper introduces a closed-loop autonomous driving controller called VLM-MPC, which combines the Model Predictive Controller (MPC) with VLM to evaluate how model-based control could enhance VLM decision-making. The proposed VLM-MPC is structured into two asynchronous components: The upper layer VLM generates driving parameters (e.g., desired speed, desired headway) for lower-level control based on front camera images, ego vehicle state, traffic environment conditions, and reference memory; The lower-level MPC controls the vehicle in real-time using these parameters, considering engine lag and providing state feedback to the entire system. Experiments based on the nuScenes dataset validated the effectiveness of the proposed VLM-MPC across various environments (e.g., night, rain, and intersections). The results demonstrate that the VLM-MPC consistently maintains Post Encroachment Time (PET) above safe thresholds, in contrast to some scenarios where the VLM-based control posed collision risks. Additionally, the VLM-MPC enhances smoothness compared to the real-world trajectories and VLM-based control. By comparing behaviors under different environmental settings, we highlight the VLM-MPC's capability to understand the environment and make reasoned inferences. Moreover, we validate the contributions of two key components, the reference memory and the environment encoder, to the stability of responses through ablation tests.

1 INTRODUCTION

Autonomous driving systems face limits from predefined rules, rare out-of-distribution situations, and unreliable VLM responses, while the contribution of model-based control to VLM driving remains underexplored. The paper proposes VLM-MPC and evaluates its safety, smoothness, and response stability.

1 INTRODUCTION

  • Predefined rules and reward designs can limit adaptability across traffic scenarios, while limited datasets may not ensure reliable decisions in rare real-world situations.Foundation models are presented as a potential approach because their reasoning capabilities may improve interpretability and performance in complex environments.Source: §1 · p. 2
  • FM-to-action systems directly output vehicle actions but may omit vehicle dynamics and struggle with the response speed needed for near-collision control.The paper contrasts this with asynchronous FM-to-command systems, which leave high-frequency control to established lower-level schemes such as MPC.Source: §1 · p. 2
  • VLM-MPC uses an upper-level VLM to generate driving parameters and a lower-level MPC to control the vehicle in real time, accounting for vehicle dynamics and engine lag.The VLM receives front-camera images, textual environment descriptions, and reference memory; the MPC provides state feedback to the VLM.Source: §1 · p. 2
  • The paper identifies a gap in evaluating whether model-based controllers improve FM-based driving and how to make FM responses more reliable.Hallucinations can produce inconsistent or misleading driving responses, and systematic evaluation of reliability solutions is described as rare.Source: §1 · p. 4
  • VLM-MPC maintains PET above safe thresholds and improves smoothness over real-world trajectories and VLM-based control, while some VLM-based scenarios pose collision risks.The evaluation also reports appropriate behavior changes across environmental conditions and improved stability from the reference memory and environment encoder.Source: §1 · p. 4
  • Reference memory supplies scene-matched average parameters, while the environment encoder extracts weather, lighting, and road conditions to add context to VLM responses.Ablation tests report that both components significantly enhance response stability.Source: §1 · p. 4

2 METHODOLOGY

The methodology section introduces VLM-MPC and outlines its architecture, upper-layer VLM, and lower-layer MPC controller.

2 METHODOLOGY

  • The method is organized around an overall VLM-MPC architecture, an upper-layer VLM, and a lower-layer MPC controller.The section proceeds from the architecture to the two controller levels.Source: §2 · p. 5

2.1 Overall Architecture

VLM-MPC separates planning and vehicle control into an upper-level VLM and a lower-level MPC. The components operate asynchronously, allowing the lower-level controller to continue using the latest VLM parameters between updates.

2.1 Overall Architecture

  • The upper-level VLM combines environment, scene, and reference-memory information into a prompt that produces key driving parameters.Its components generate camera-based environment descriptions, extract ego and preceding-vehicle information, and supply historical scenario parameters.Source: §2.1 · p. 5
  • The lower-level rule-based MPC uses the upper-level parameters and vehicle dynamics to determine actions for execution.The architecture separates VLM planning from vehicle control.Source: §2.1 · p. 5
  • The two levels run at different frequencies, and the MPC continues to use the latest VLM instruction and parameters during asynchronous intervals.This setup means the VLM need not operate at the MPC's control frequency.Source: §2.1 · p. 5

2.2 Upper-layer: VLM

The upper layer assembles image-derived environment descriptions, scene states, and scenario-matched reference parameters, then prompts the VLM to generate driving parameters for the MPC.

2.2.1 Environment Encoder

  • CLIP compares front-camera image features with predefined descriptions to produce linguistic environment information, including weather, lighting, and road conditions.Descriptions for some attributes are emitted only when confidence exceeds specified thresholds.Source: §2.2.1 · p. 6

2.2.2 Scene Encoder

  • The scene encoder supplies historical ego-vehicle states and information about the preceding vehicle and obstacles, including pedestrians and relevant stop lines.Preceding-vehicle and stop-line positions come from nuScenes data and maps; perception errors are not modeled.Source: §2.2.2 · pp. 6–7

2.2.3 Reference Memory

  • Reference memory maps environment characteristics to averaged MPC parameters, including prediction horizon, control weights, desired speed, and desired headway.The memory is built from nuScenes trajectories and groups scenes by conditions such as intersections, rain, and night.Source: §2.2.3 · pp. 7–8

2.2 Upper-layer: VLM

  • The upper layer collects information from its modules to prompt the VLM to generate six driving parameters.The section organizes this process around reference memory, environment and scene encoders, and prompt generation.Source: §2.2 · p. 6, §2.2.4 · p. 8

2.2.4 Prompt Generator

  • Initial VLM calls combine reference memory, a front-camera image, an environment description, and ego and preceding-vehicle states; later calls also use the previous parameter set.The prompt generator also applies chain-of-thought guidance as linked reasoning steps for complex driving decisions.Source: §2.2.4 · p. 8

2.3 Lower-layer: MPC

The lower-layer MPC uses VLM-generated parameters to optimize vehicle motion in real time, accounting for longitudinal dynamics and updating from vehicle-state feedback.

2.3 Lower-layer: MPC

  • MPC optimizes the vehicle trajectory in real time using VLM-generated parameters, accounting for engine lag and updating from the latest vehicle state.The controller aims to improve traffic efficiency and safety while respecting traffic and vehicle constraints.Source: §2.3 · p. 9

2.3.1 CAV Dynamic Model

  • The longitudinal vehicle model describes position, speed, and acceleration dynamics, with engine input and a feedback-linearizing control structure.The control law assumes model parameters are known and uses desired acceleration determined by the upper controller.Source: §2.3.1 · p. 9
  • The MPC minimizes speed error and control effort when no preceding vehicle or stop line is available, and includes a safe-headway term when one is available.A constant-time-headway strategy specifies desired spacing from the preceding vehicle.Source: §2.3.1 · pp. 9–10

3.1 Experiment settings

The experiments use nuScenes scenes to evaluate the proposed controller across weather and traffic settings, with safety, comfort, and stability metrics. The baselines compare fixed-parameter MPC and language-model control approaches with VLM-MPC.

3.1 Experiment settings

  • Llava 1.6 is the main VLM; GPT-4-o and GPT-4-mini are also tested in sensitivity analysis, with Llava run locally on an RTX 4090.The Llava setup used Ubuntu, 32 GB DDR5 RAM, temperature 0, and default settings for other parameters; GPT models were accessed by API.Source: §3.1 · p. 10

3.1.1 Dataset

  • The evaluation covers 303 nuScenes scenes, including Singapore nighttime scenes despite the algorithm being designed for right-hand traffic.The dataset spans locations and weather conditions; scenario parameters are grouped by rain, intersection, and night, with significant differences indicated by asterisks in Figure 4.Source: §3.1.1 · p. 10, Figure 4 · p. 11, Table 2 · p. 11

3.1.2 Closed-loop Evaluation Metrics

  • Closed-loop evaluation measures safety with PET, comfort with RMSa, and stability with scene completion rate.Higher PET indicates a longer time gap between potential conflicts; completion rate reflects whether the VLM/LLM consistently produces valid, reasonable parameters, since no MPC failures were observed.Source: §3.1.2 · p. 11

3.1.3 Baseline Models

  • The baselines are fixed-parameter MPC, direct LLM-to-Action, and LLM-MPC without camera input or memory-based parameter aggregation.Table 3 compares these approaches with VLM-MPC, which uses image inputs and reference memory.Source: §3.1.3 · p. 12, Table 3 · p. 12

3.2 Experiment Results

Across the tested scenarios, VLM-MPC maintains PET above the 1-second safety threshold and improves smoothness relative to baseline methods. Results also report completion rates and model response times.

3.2.1 Safety

  • Across tested traffic conditions, VLM-MPC maintains PET above 1 second; the baseline LLM controller falls below it, with values 0.37 and 0.05 in dry non-intersections and 0.64 in rainy intersections.Source: §3.2.1 · pp. 12–13
  • At 5 seconds in a Boston intersection example, desired speed drops from 6.5 to 2.0 m/s; the text says retaining GPT's 5-second parameters would not stop smoothly before the stop line.The example describes changing conditions as the vehicle approaches the intersection.Source: §3.2.1 · p. 13, Figure 5 · p. 14

3.2.2 Smoothness

  • −23.2 m/s² was the LLM-to-Action extreme deceleration, versus −3.5 to 3 m/s² for MPC-based methods; VLM-MPC has among the lowest average RMSa in most scenarios.The MPC-based speed and acceleration distributions closely match the real-world trajectory; the real-world trajectory includes an extreme deceleration of −10.53 m/s².Source: §3.2.2 · pp. 15–16, Table 5 · p. 16, Figure 7 · p. 16

3.2.3 Completion Rate

  • Across all scenarios, proposed VLM-MPC achieved a 99.7% completion rate with Llava 1.6, while GPT-3.5-turbo, GPT-4-o, and GPT-4-mini each achieved a 100% completion rate.Source: §3.2.3 · p. 17

3.2.4 Response Time

  • 3.42 seconds average for Llava meets the 0.2 Hz requirement; GPT-4-o and GPT-4-mini average around 10 seconds, and local Llava/Llama models are faster than cloud GPT models.The baseline Llama 3.1 averages 1.5 seconds, compared with 3.9 seconds for GPT-3.5-turbo.Source: §3.2.4 · p. 17, Figure 8 · p. 17, Figure 9 · p. 18

3.3 Sensitivity analysis

The section tests how reference memory and environment descriptions affect VLM-MPC, then examines whether the model’s image attention aligns with its generated descriptions.

3.3.1 Impact of Reference Memory

  • Without reference memory, completion remains 99.3%, but generated parameters favor higher speeds and larger headways, and smoothness worsens in rain and at night.The ablation also produces two invalid responses; planned trajectories improve smoothness in clear weather.Source: §3.3.1 · p. 18

3.3.2 Impact of Environment Description

  • Without environment information, completion falls to 87.5%, with failures attributed to missing or malformed parameters when the VLM cannot understand the driving task.The authors identify road information as important context for interpreting the task and responding accurately.Source: §3.3.2 · pp. 18–19
  • Excluding environment information noticeably slows Llama responses, with some taking more than 200 seconds.In successful-response scenarios, safety is maintained and response time remains shorter than for real-world trajectories and end-to-end methods.Source: §3.3.2 · p. 19

3.3.3 Interpretability analysis

  • Across two scenarios, about 14% of attention goes to vision tokens during the first 50–60 generated tokens, when the model describes the environment.Later tokens set prediction horizon, cost weights, desired speed, and desired headway, relying less on images and more on text.Source: §3.3.3 · p. 19, Figure 10 · p. 20
  • Image attention shifts toward the road, parked vehicles, wet road, or traffic light as the corresponding descriptions are generated.The attention-map procedure combines model attention with vision-encoder attention averaged across layers to locate image regions associated with generated phrases.Source: §3.3.3 · p. 19

4 CONCLUSION AND FUTURE RESEARCH

VLM-MPC pairs VLM upper-level decisions with real-time MPC vehicle control. In nuScenes experiments, it maintained safe PET and improved smoothness against VLM-based control and real-world trajectories.

4 CONCLUSION AND FUTURE RESEARCH

  • The closed-loop controller assigns adaptive upper-level decisions to the VLM and real-time vehicle control to MPC using ego and surrounding-vehicle state.The experiments use the nuScenes dataset and compare the approach with an LLM controlling the vehicle independently.Source: §4 · p. 21
  • VLM-MPC consistently keeps PET above safe thresholds and improves smoothness versus real-world trajectories and VLM-based control.Some VLM-based control scenarios posed collision risks; the authors also report that VLM-MPC with Llava meets the proposed response-time requirements.Source: §4 · p. 21
  • Ablation tests support contributions from reference memory and the environment encoder to response stability.Source: §4 · p. 21
  • Future work calls for more ablations and real-vehicle experiments, including tests of asynchronous operating frequency and out-of-distribution conditions.The authors note that nuScenes and Waymo often lack the same scene under different driving conditions; real-world closed-loop tests would include cumulative perception, decision-making, and control uncertainties.Source: §4 · p. 21
Loading 2408.04821v2…