Source-linked AI summary
LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
Zhijie Qiao, Haowei Li, Zhong Cao, Henry X. Liu
TL;DR
LightEMMA examines whether successive VLM generations improve autonomous-driving performance, addressing limited evidence across model families and generations. It applies a unified, adaptation-free evaluation protocol to 15 models on nuScenes and finds that newer models and stronger general reasoning do not consistently yield better driving performance. Scenario analysis identifies recurring failure modes and supports domain-specific adaptation.
Problem
VLMs are increasingly used for autonomous driving, but evidence remains limited on whether successive generations improve driving performance across model families.
Method
LightEMMA evaluates VLMs under a lightweight, unified protocol using front-view images, driving history, generated control actions, and trajectory comparison without model-specific adaptations.
Results
Successive VLM generations do not consistently improve autonomous-driving performance despite stronger general reasoning and larger model scale.
Takeaways & Limitations
Recurring limitations in spatial reasoning and dynamic scene understanding highlight the need for domain-specific adaptation in VLM-based autonomous driving.
Abstract
from arXiv · showhide
Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. To systematically examine this assumption, we introduce LightEMMA, a longitudinal framework for evaluating the autonomous driving performance of VLMs. LightEMMA uses a lightweight, unified evaluation protocol that assesses each model's intrinsic driving capability without model-specific fine-tuning, architectural changes, or prompt engineering. Using this protocol, we evaluate 15 models from five major families on the challenging nuScenes prediction benchmark. Empirical findings show that, despite increased model scale and enhanced general reasoning capabilities, successive VLM generations do not consistently achieve better driving performance. Further analysis of driving scenarios reveals recurring failure modes, including overreliance on historical actions and difficulty reconciling conflicting visual cues. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems. The source code is available at https://github.com/michigan-traffic-lab/LightEMMA.
I. INTRODUCTION
LightEMMA addresses limited evidence about whether advancing VLMs improve autonomous driving by enabling unified longitudinal evaluation across model families and generations. Its results challenge the assumption of continual driving-performance gains.
- VLMs offer open-world reasoning and contextual scene understanding, but their autonomous-driving efficacy remains unresolved.
- Prior work demonstrated unified and open-source VLM driving approaches, yet existing evaluations covered only a small set of models.
- LightEMMA evaluates VLMs across five major families and three years under a unified protocol without model-specific fine-tuning, architectural changes, or prompt engineering.
- Successive VLM generations do not consistently improve driving performance, and scenario analysis identifies recurring failure modes requiring domain-specific adaptation.
II. RELATED WORK
Related work includes VLM-based systems that map visual driving inputs to controls while combining language explanations, vehicle states, and driving datasets.
- DriveGPT4 fine-tunes LLaMA-2 to predict next-step speed and turning angle from monocular driving video and ego speed.
B. Efficient and Hybrid VLM Systems
Efficient and hybrid driving VLMs address the computational burden of inference through compact models, gated attention, or slow-fast combinations with conventional driving pipelines.
- EM-VLM4AD combines six camera views with gated attention before feeding features to a compact T5 model for driving-scene question answering.
- EM-VLM4AD requires less than one-tenth the memory and computation of other evaluated driving VLMs.
- DriveVLM-Dual couples a low-frequency VLM branch for complex-scene analysis with a conventional branch for 3D spatial grounding and trajectory refinement.
- nuScenes provides surround-view cameras, radar, and LiDAR for 3D detection, tracking, and motion prediction, while related datasets target language grounding or long-tail driving.
- LightEMMA uses front-view images and driving history, prompts VLMs for control actions, integrates those actions into trajectories, and compares them with ground truth.
A. VLM Selection
The study selects comparable VLM variants across major families while standardizing visual inputs and representing driving history as speed-curvature actions.
- The evaluation covers GPT, Claude, Gemini, Qwen, and LLaMA, selecting three generational variants per family to balance capability, inference time, and cost.
- LLaMA lacks vision-language support before version 3.2 and retains limited multimodal options afterward, constraining its available generational variants.
- The framework supplies raw front-view images without preprocessing or auxiliary perception overlays, and uses only the current frame.
- Driving history is encoded as six speed-curvature pairs sampled every 0.5 seconds over three seconds to match nuScenes temporal resolution.
D. VLM Prompting
LightEMMA guides each VLM through a three-stage Chain-of-Thought pipeline that moves from scene description to driving intent and structured low-level commands. The resulting commands are used for kinematic trajectory generation over a fixed prediction horizon.
- D. VLM Prompting: The pipeline sequentially feeds each stage’s output into the next, providing cumulative context across three stages.
- D. VLM Prompting: Scenario description uses the front-view image to identify lanes, traffic signals, vehicles, pedestrians, and other relevant scene elements.
- D. VLM Prompting: High-level driving intent combines the scene description with historical ego actions to predict the next high-level driving action.
- D. VLM Prompting: Low-level command generation converts the scene description and high-level action into six speed-curvature pairs, without supplementary text or explanations.
- E. Trajectory Generation: Kinematic integration propagates future vehicle states in an egocentric reference frame from initial position (0, 0) and orientation 0.
- E. Trajectory Generation: The VLM’s command sequence supplies speed and curvature values that iteratively update the vehicle’s position and orientation.
- E. Trajectory Generation: A fixed timestep of 0.5 s matches the nuScenes sampling interval for trajectory generation.
- E. Trajectory Generation: The predicted trajectory contains six positions sampled at 0.5 s intervals across a 3 s horizon.
F. Example Workflow
The example workflow applies LightEMMA’s staged prompting to an urban construction scene with an officer directing traffic. The model recommends slowing substantially while remaining prepared to stop.
- The example begins with a front-view urban scene containing construction activity, work vehicles, and a police officer directing traffic.
- The high-level intent calls for significant deceleration of approximately 2–3 m/s2 while following the lane.
- The vehicle should remain ready to come to a complete stop if directed by the officer.
IV. EXPERIMENTS
The experiments evaluate LightEMMA across model efficiency and autonomous-driving performance using a longitudinal set of VLMs on nuScenes. The protocol emphasizes comparable inference measurements across evaluated models.
- The evaluation uses 150 nuScenes test scenarios comprising 3,908 frames for trajectory prediction.
- The study measures inference time, token usage and cost, response reliability, and trajectory prediction accuracy.
- Table I compares model efficiency and computational cost across the evaluated models.
- Gemini-2.0-Flash achieves the fastest complete Chain-of-Thought inference at 4.5 seconds per frame.
- The Qwen family has consistently low latency, whereas the LLaMA family has the highest latency overall.
B. Token Usage and Cost
The study measures token usage and cost alongside response reliability, identifying broad token-count patterns and anomalies in reported billing or tokenization. It also describes format errors and their correction for downstream analysis.
- B. Token Usage and Cost: Commercial costs are computed from token usage, billing records, and official pricing schedules, and reported in cents per frame.
- B. Token Usage and Cost: Most models use 4,000–6,000 input tokens per frame, while output counts remain in the low hundreds.
- B. Token Usage and Cost: Claude is the most expensive commercial family, while Gemini is the least expensive family overall.
- B. Token Usage and Cost: Gemini-2.5-Flash shows anomalously low input-token counts and billing inconsistencies that are resolved in Gemini-3.0-Flash.
- B. Token Usage and Cost: LLaMA’s unexpectedly low input-token counts likely result from omitting image tokens in its official tokenizer.
- C. Response Reliability: Format errors include missing punctuation, explanatory text, and incorrect output lengths despite strict output-format instructions.
- C. Response Reliability: Error handling extracts twelve values and organizes them into six speed-curvature pairs for downstream analysis.
- C. Response Reliability: Corrected format-error rates remain below 0.1% for all models, although the errors indicate difficulty adhering strictly to the prescribed format.
D. Trajectory Prediction Accuracy
Trajectory accuracy varies substantially across model families, and newer generations do not consistently improve autonomous-driving performance. GPT-4o performs best overall, while the LLaMA family performs worst across prediction horizons.
- GPT-4o achieves the lowest ADE and FDE values, followed closely by GPT-5.0 and Claude-3.7-Sonnet.
- Gemini-3.0-Flash is the strongest-performing variant within the Gemini family.
- Qwen-3.0-VL-8B rivals much larger commercial offerings despite its smaller scale.
- The LLaMA family performs worst overall, with the highest errors across all prediction horizons.
- Successive VLM generations do not consistently improve driving performance, and larger model scale does not reliably guarantee better results.
V. SCENARIO-BASED FAILURE ANALYSIS
Scenario analysis exposes recurring failures that average displacement metrics alone cannot explain, including historical-action bias, missed contextual cues, abrupt braking, and conflicting visual interpretations.
- Case 1: Trajectory Bias from Historical Actions: Historical right-turn actions can bias models toward predicting a right turn even when the current scene requires continuing straight.Similar errors also occur after left turns, but predictions correct shortly after the vehicle resumes a straight path.
- Case 2: Implicit Turn Cues: Models may predict continuing straight when a left turn is implied by lane positioning but lacks explicit turn markings or dedicated signals.Additional contextual information, such as explicit navigation instructions, may be needed for reliable handling.
- Cases 3 & 4: Divergent Responses to Stop Signals: At red signals, some models brake abruptly despite recognizing hazards, while others ignore the stopped vehicle and signal entirely.The first behavior suggests insufficient spatial reasoning for distance estimation and controlled deceleration; the second has higher ADE.
- Case 5: Divergent Responses to Green Signals: Models responding to green signals may either accelerate appropriately or remain stationary, revealing difficulty translating dynamic cues into timely control actions.Delayed responses could obstruct traffic flow and increase rear-end-collision risk.
- Case 6: Conflicting Visual Cues: Even low-ADE models disagree when a green signal conflicts with a vehicle blocking the path, with some accelerating and others correctly remaining stopped.The case requires reconciling competing scene elements.
VI. CONCLUSION
LightEMMA evaluates VLMs across families and generations, finding that stronger general reasoning does not necessarily improve driving performance. Its failure analysis points to spatial-reasoning and dynamic-scene-understanding limitations, motivating domain-specific adaptation.
- LightEMMA provides a lightweight, unified framework for evaluating VLMs in autonomous driving.
- Across five model families, newer generations and stronger general reasoning do not necessarily improve driving performance.
- Scenario-based analysis reveals recurring limitations in spatial reasoning and dynamic scene understanding.
- The findings highlight the need for domain-specific adaptation in VLM-based autonomous driving.