PaperlayerView exact revision on arXiv

Source-linked AI summary

SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment

Katrin Renz, Long Chen, Elahe Arani, Oleg Sinavski

arXiv:2503.09594v1cs.CVcs.RO

TL;DR

Existing driving systems often separate strong driving from broad language understanding, while VQA may not align with the actions a vehicle takes. SimLingo combines camera-only closed-loop driving, vision-language understanding, and language-action alignment, achieving state-of-the-art CARLA results and strong language-task performance. The paper’s official Leaderboard evaluation is limited because the final model could not be submitted after the leaderboard closed.

  • Problem

    Existing methods struggle to achieve high driving performance and extensive language understanding together, and language-only VQA may be inconsistent with driving actions.

  • Method

    SimLingo is a camera-only VLM-based model that jointly handles closed-loop driving, vision-language understanding, and language-action alignment through Action Dreaming.

  • Results

    SimLingo achieves state-of-the-art driving results on CARLA Leaderboard 2.0 and Bench2Drive while maintaining strong performance across language-related tasks.

  • Takeaways & Limitations

    Action Dreaming provides a way to evaluate and train whether diverse language instructions correspond to predicted driving actions.

  • Takeaways & Limitations

    Only SimLingo-BASE could be tested on the official CARLA Leaderboard because the leaderboard closed in June 2024.

Abstract

from arXiv · show

Integrating large language models (LLMs) into autonomous driving has attracted significant attention with the hope of improving generalization and explainability. However, existing methods often focus on either driving or vision-language understanding but achieving both high driving performance and extensive language understanding remains challenging. In addition, the dominant approach to tackle vision-language understanding is using visual question answering. However, for autonomous driving, this is only useful if it is aligned with the action space. Otherwise, the model's answers could be inconsistent with its behavior. Therefore, we propose a model that can handle three different tasks: (1) closed-loop driving, (2) vision-language understanding, and (3) language-action alignment. Our model SimLingo is based on a vision language model (VLM) and works using only camera, excluding expensive sensors like LiDAR. SimLingo obtains state-of-the-art performance on the widely used CARLA simulator on the Bench2Drive benchmark and is the winning entry at the CARLA challenge 2024. Additionally, we achieve strong results in a wide variety of language-related tasks while maintaining high driving performance.

1 Wayve 2 University of T¨ubingen 3 T¨ubingen AI Center

SimLingo unifies autonomous driving, vision-language understanding, and language-action alignment in a camera-only vision-language-action model. It is reported as state of the art on CARLA Leaderboard 2.0 and Bench2Drive, with Action Dreaming introduced to align language and action.

  • SimLingo unifies autonomous driving, vision-language understanding, and language-action alignment in one vision-language-action model.
  • The model uses only camera images rather than expensive sensors such as LiDAR.
  • Action Dreaming is introduced as an instruction-following task to improve alignment between language and action.
  • SimLingo is reported as state of the art on the official CARLA Leaderboard 2.0 and Bench2Drive.

1. Introduction

The introduction motivates integrating VLM capabilities into autonomous driving while identifying a gap between language-space understanding and driving actions. SimLingo addresses this gap with closed-loop driving, language understanding, and instruction-action alignment.

  • VLMs offer broad world knowledge and generalization to unseen prompts, motivating their integration into autonomous driving.
  • Evaluating driving understanding only through language can leave answers disconnected from the model’s actual driving decisions.
  • The proposed system combines closed-loop driving, vision-language understanding, and language-action alignment.
  • Action Dreaming evaluates instruction following by pairing diverse language instructions with corresponding predicted actions without executing unsafe instructions.
  • The authors rigorously test the models on a challenging closed-loop benchmark rather than relying only on simplified or open-loop settings.

2. Related Work

Prior work applies end-to-end learning and language understanding to driving, but much evaluation remains open-loop or simplified. The paper emphasizes the unresolved need to test instruction-action alignment in challenging closed-loop settings.

  • End-to-end autonomous driving: CARLA Leaderboard 2.0 makes driving substantially harder, with TransFuser’s score dropping from 66.32 to 0.58 when transferred zero-shot from Leaderboard 1.0.
  • End-to-end autonomous driving: Many leading end-to-end methods use auxiliary outputs and multiple sensors, while camera-first models seek competitive performance using only images.
  • Language models for driving: Existing language-driving approaches are often evaluated qualitatively, in open-loop settings, or in simplified environments such as HighwayEnv.
  • Vision-language understanding: VQA-based systems frequently omit action prediction or use open-loop evaluation, and open-loop results lack evidence of transfer to closed-loop driving.
  • Vision-language understanding: The field lacks systematic evaluation of whether driving instructions are aligned with the actions produced by the model.

3. Method

SimLingo uses camera images, navigation information, language prompts, and vehicle speed in a VLM-based driving pipeline. It predicts language and disentangled temporal-speed and geometric-path actions, and trains on expert and simulated instruction-action trajectories.

  • Task overview: The driving task requires reaching target locations through diverse CARLA environments, weather conditions, and complex scenarios.
  • Task overview: During inference, Commentary makes the model predict an action and its reason in language before predicting the action conditioned on that commentary.
  • Task overview: Action Dreaming supplies normal and out-of-distribution instructions, then tests whether the model predicts the corresponding action without executing unsafe behavior.
  • Data collection: The driving dataset is collected in CARLA with the privileged rule-based expert PDM-lite across multiple route sets and scenarios.
  • Data collection: Longer routes with three scenarios address a distribution shift caused by Leaderboard 2.0’s increased maximum distance between navigational target points.
  • Data collection: Multiple alternative instructions and actions for one visual context force the model to attend to language rather than infer actions from visual cues alone.
  • Architecture: SimLingo encodes a camera image, GPS target points or a high-level language command, ego speed, and a task prompt.
  • Architecture: The action representation separates temporal speed waypoints from geometric path waypoints, representing future positions by time and by distance-independent geometry.

4. Experiments

The experiments evaluate SimLingo on CARLA Leaderboard 2.0, Bench2Drive, language understanding, instruction following, Action Dreaming, and training-mixture ablations. Results indicate strong closed-loop driving alongside broad language capabilities, with aligned Action Dreaming data slightly improving driving.

  • Driving performance: SimLingo-BASE achieves state-of-the-art performance on the official CARLA Leaderboard 2.0 using camera-only input.It is reported as the only leaderboard model using only camera images among entries with method reports.
  • Driving performance: 4.6x higher driving performance than CaRINA hybrid and 33% higher than TF++ are reported for SimLingo-BASE on the Sensor track.The comparison is reported for the official CARLA Leaderboard 2.0 results.
  • Driving performance: 39.9 percent higher driving score and zero static-object collisions result from disentangled path-and-speed waypoints in the ablation.The ablation compares disentangled path+speed waypoints with an entangled waypoint representation.
  • Language understanding: SimLingo outperforms zero-shot InternVL2 models on the balanced DriveLM and Commentary evaluation split.The language evaluation covers VQA and Commentary, while the baseline is tested zero-shot.
  • Driving performance: 63.45 DS is achieved after changing dataset composition and tuning the controller, versus 49.30 DS before those changes on Bench2Drive.The authors state that dataset composition contributes to the improvement while additional advancements remain essential for state-of-the-art performance.
  • Language and driving: SimLingo preserves the driving performance of SimLingo-BASE while adding several language-related capabilities.Bench2Drive results compare the full vision-language-action model with the pure driving model.
  • Instruction following: Language-command conditioning reaches driving performance within the variance of GPS target-point conditioning.This supports following basic navigational commands without relying on target-point recovery as a shortcut in this setting.
  • Action Dreaming: Action Dreaming enables SimLingo to follow a wide range of instructions beyond the commands natively supported by CARLA.The evaluation measures instruction-following success rate and aligns language understanding with the action space.

5. Conclusion and Limitations

SimLingo combines state-of-the-art CARLA driving with language understanding and Action Dreaming for language-action alignment. The authors note that the full model was not tested on the official leaderboard and that CoT had not produced statistically significant driving improvements.

  • SimLingo achieves state-of-the-art results on the official CARLA Leaderboard 2.0 and Bench2Drive while demonstrating language understanding.
  • The model measures language performance on driving Commentary and VQA, with finetuned InternVL2 excelling in the specialized driving domain.
  • Action Dreaming aligns vision-language understanding with the action space by evaluating predictions for diverse language instructions.
  • The full SimLingo model was not tested on the official CARLA Leaderboard because it closed in June 2024; only SimLingo-BASE was tested there.
  • Although the final model conditions driving actions on intermediate commentary through CoT, statistically significant driving improvements were not observed.

A. Datasets

The paper provides a more detailed description of the collected datasets and the procedures used to generate labels for language-related datasets.

  • The appendix details the collected datasets and how labels are generated for each language-related dataset.

A.1. Driving dataset - Scenarios

The driving and language datasets cover diverse scenarios and use structured labels, augmented questions, and offline trajectory simulation to support evaluation and language-action alignment.

  • Driving scenarios: The dataset includes diverse CARLA scenarios such as control loss, unprotected turns, crossing traffic, obstacles, door hazards, and slow-moving hazards.
  • VQA data: VQA data contain 28M training question-answer pairs for 1M Town 12 frames, with keyframe extraction used for evaluation.
  • VQA data: GPT-4 generates 20 alternative sentences for each question and answer to reduce overfitting to fixed phrasing.
  • Commentary labels: Commentary labels describe route actions, speed actions, and their reasons using simulator-state-derived templates.
  • Action Dreaming: Action Dreaming uses an offline simulation with a Kinematic Bicycle Model and PDM-lite PID controllers to generate alternative trajectories and assess collisions and traffic-rule adherence.

B.2. Training buckets

Training emphasizes challenging driving examples through bucketed sampling, while SimLingo-BASE and SimLingo differ in their encoders, language models, and loss functions.

  • B.2. Training buckets: Most straight, hazard-free driving is learned early, so later training on such data wastes computation.
  • B.2. Training buckets: The training procedure samples buckets for acceleration, steering, vehicle hazards, traffic controls, and other interesting driving events instead of the entire dataset.
  • Model variants: SimLingo-BASE uses a lightweight architecture with a CLIP-ViT encoder and a 50M-parameter LLaMA-based decoder trained from scratch.
  • Model variants: SimLingo uses InternVL2's original image encoder and pretrained language model, with LoRA finetuning for the language model and a changed loss after adding Action Dreaming data.

B.4. Metric descriptions

The evaluation reports route completion, infraction penalties, and driving scores for Leaderboard 2.0 and Bench2Drive, with additional measures for speed and comfort. These metrics expose trade-offs in penalized route performance, including the possibility that early stopping can increase Leaderboard scores on long routes.

  • Leaderboard 2.0: Leaderboard 2.0 averages Route Completion and Infraction Score across routes, while global Driving Score is the primary ranking metric.Driving Score combines route completion with infraction penalties.
  • Infraction Penalty: Infraction Score multiplies penalty coefficients for each infraction type and occurrence, with collisions and traffic violations assigned severity-dependent coefficients.Examples include 0.50 for pedestrian collisions, 0.60 for vehicle collisions, and 0.70 for red-light violations.
  • Leaderboard 2.0 metric discussion: Driving Score can favor stopping before route completion when further infractions reduce the score more than continued progress increases it.This trade-off occurs on long routes.
  • Bench2Drive: Bench2Drive modifies Driving Score by ignoring the minimum-speed penalty and adds Efficiency, which measures ego speed relative to surrounding actors.Because Bench2Drive routes are short, the long-route stopping trade-off does not occur there.
  • Comfort: Comfort metrics include limits for jerk, lateral and longitudinal acceleration, yaw acceleration, longitudinal jerk, and yaw rate.Reported thresholds include 8.37 m/s^3 maximum jerk and 0.95 rad/s maximum yaw rate.
  • Language-action metrics: Dreamer-mode success is assessed from predicted path and speed behavior, including waypoint-derived slope for slowing and acceleration.Object-collision success compares predicted-path displacement to expert and dreamer trajectories.

C.1. Ablations on CARLA Leaderboard 2.0

Ablations show that SimLingo’s disentangled output representation, visual pretraining, and early-stopping threshold materially affect Leaderboard performance. Chain-of-thought inference yields only a small improvement, while repeated submissions reveal evaluation variance.

  • Output representation: Reducing layout collisions from 0.68 to 0 with disentangled outputs demonstrates the benefit of additional path prediction for lateral control.The comparison is reported for SimLingo-BASE on Leaderboard results.
  • Inference mode: Chain-of-thought inference produces a small improvement over inference without CoT, but omitting CoT remains feasible when inference speed matters.Commentary is used as the default CoT task.
  • Early stopping: Early-stopping distance has a large impact on the final Leaderboard Driving Score because the metric favors avoiding penalties on long routes.The optimal threshold is difficult to determine because routes and scenario density are secret.
  • Leaderboard variance: At a 2100-meter threshold, three submissions scored 6.9, 5.5, and 5.3 DS, yielding a mean of 5.9 and standard deviation 0.87.A 2400-meter threshold produced a mean DS of 5.83 with standard deviation 0.90.

C.2. Bench2Drive Multi-Ability Results

SimLingo outperforms existing methods across Bench2Drive’s multi-ability evaluation, with especially large gains in several abilities. Chain-of-thought inference improves results only slightly and is therefore optional when speed is prioritized.

  • Multi-Ability Results: SimLingo outperforms existing methods in all Bench2Drive abilities and improves the average by 25 percentage points.The largest gains are reported for Merging, Overtaking, Emergency Brake, and Traffic Sign; Give way remains challenging.
  • Chain-of-Thought Inference: Commentary-based chain-of-thought inference produces a small improvement, while inference without CoT causes only a limited performance decrease.The authors use Commentary as the default inference mode but consider no-CoT inference feasible for faster execution.

D. Qualitative Results

Qualitative closed-loop results show that SimLingo adapts path and speed predictions to navigation and dreamer-mode language commands. It handles many paraphrases and contextual instructions, but unseen concepts and some commands remain difficult.

  • Navigational Commands: Closed-loop navigation examples show predicted path and speed waypoints changing with the navigation command.The model differentiates situations and adapts behavior to the requested command.
  • Navigational Commands: For “Turn left,” the vehicle changes into the left lane and stays there rather than interpreting the instruction as an automatic 90-degree turn.The example reflects lane semantics and avoidance of an oncoming lane.
  • Navigational Commands: With a misleading “Turn right” instruction where no right-turn lane exists, the vehicle remains on the road and continues straight during closed-loop rollout.The authors describe the model as slightly confused in this case.
  • Navigational Commands: Out-of-distribution commands can still yield valid paths: irrelevant text preserves road behavior, while contextual wording can select a left turn without relying on isolated direction words.The model fails to follow the unseen “Do a U-Turn” concept.
  • Dreamer Mode: Dreamer-mode examples evaluate language-to-action behavior for faster, slower, target-speed, collision-object, and lane-change commands.Figures visualize path waypoints in red and speed waypoints in green; speed curves are inferred from speed waypoints in several modes.
  • Dreamer Mode: Lane-change prompts specify direction, start distance, transition length, or target lane, providing varied language forms for the same action family.Examples include transitions to the left or right and locating the vehicle in a specified lane.
Loading 2503.09594v1…