PaperlayerView exact revision on arXiv

Source-linked AI summary

SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment

Katrin Renz, Long Chen, Elahe Arani, Oleg Sinavski

arXiv:2503.09594v1cs.CVcs.RO

TL;DR

Autonomous-driving language models are often evaluated without closed-loop action prediction, leaving systematic evidence about instruction-action alignment limited. SimLingo uses a vision-language model for closed-loop driving, language understanding, and action-aligned instruction following. It reports state-of-the-art CARLA Leaderboard 2.0 and Bench2Drive results alongside language understanding and high action-prediction success across varied instructions.

  • Problem

    Many language-understanding approaches for autonomous driving use open-loop evaluation or omit action prediction, and systematic evaluation of instruction-action alignment remains limited.

    Source: §2 · p. 2

  • Method

    SimLingo is a vision-language driving model that combines closed-loop driving and language understanding with Action Dreaming, which pairs varied instructions with driving actions.

    Source: §3.1 · p. 3, §5 · p. 8

  • Results

    SimLingo achieves state-of-the-art results on CARLA Leaderboard 2.0 and Bench2Drive, with language understanding and high success predicting actions for varied instructions.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The Action Dreaming benchmark evaluates how language inputs influence actions without executing dangerous instructions.

    Source: §1 · p. 2

    Because the CARLA Leaderboard closed in June 2024, the authors could not submit the final SimLingo model and tested only SimLingo-BASE there.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Integrating large language models (LLMs) into autonomous driving has attracted significant attention with the hope of improving generalization and explainability. However, existing methods often focus on either driving or vision-language understanding but achieving both high driving performance and extensive language understanding remains challenging. In addition, the dominant approach to tackle vision-language understanding is using visual question answering. However, for autonomous driving, this is only useful if it is aligned with the action space. Otherwise, the model's answers could be inconsistent with its behavior. Therefore, we propose a model that can handle three different tasks: (1) closed-loop driving, (2) vision-language understanding, and (3) language-action alignment. Our model SimLingo is based on a vision language model (VLM) and works using only camera, excluding expensive sensors like LiDAR. SimLingo obtains state-of-the-art performance on the widely used CARLA simulator on the Bench2Drive benchmark and is the winning entry at the CARLA challenge 2024. Additionally, we achieve strong results in a wide variety of language-related tasks while maintaining high driving performance.

1. Introduction

The introduction argues that driving-scene VQA can be disconnected from behavior and proposes aligning language inputs with actions while evaluating driving in a challenging closed-loop setting. It presents Action Dreaming and a VLM-based generalist driving model as responses to these challenges.

1. Introduction

  • Because VQA answers can be disconnected from driving decisions, the authors argue that actions adapting to language cues provide causal evidence of language understanding.For example, a model could identify a red light in language while its actions indicate acceleration.Source: §1 · p. 1
  • The VLM-based model achieves state-of-the-art driving performance on CARLA Leaderboard 2.0 and Bench2Drive while including several language-related tasks in one model.The authors emphasize evaluation on a challenging closed-loop benchmark, rather than only simplistic environments or open-loop planning.Source: §1 · p. 2
  • Action Dreaming uses simulated possible futures to create diverse instruction-action pairs for the same visual context and benchmarks language-action alignment without executing unsafe instructions.This design addresses the risk that post-hoc instruction-action pairs can let a model infer actions from visual cues alone without attending to the instruction.Source: §1 · p. 2

2. Related Work

Related work spans end-to-end driving, language-model-based driving, and visual question answering, but the supplied passages identify gaps in camera-only driving, closed-loop performance, and systematic language-action evaluation.

End-to-end autonomous driving

  • Imitation-learning end-to-end training dominates CARLA Leaderboard 1.0, but TransFuser’s zero-shot driving score falls from 66.32 to 0.58 on harder Leaderboard 2.0.Most state-of-the-art end-to-end methods use auxiliary outputs and multiple sensors; camera-only DriveCoT [59] and TCP [62] report competitive results.Source: §2 · p. 2

Language models for driving

  • Driving language-model work includes multimodal foundation-model frameworks, text-based motion planners, and common-sense approaches, but much is evaluated open-loop or in simplified environments.FED uses language feedback during training; LMDrive [51] follows a limited set of instructions but trails state-of-the-art benchmark models [34], while DriveMLM uses an off-the-shelf planner rather than end-to-end training.Source: §2 · p. 2

Visual question answering

  • VQA-based driving work often uses open-loop evaluation or omits action prediction, and open-loop results have no demonstrated transfer to closed-loop benchmarks.The passages also identify unreliable open-loop ablations and claims, and a lack of systematic instruction-following evaluation, especially for instruction-action alignment.Source: §2 · p. 2

3. Method

SimLingo combines camera-based driving with language understanding and language–action alignment. Its method defines three tasks, builds paired driving and instruction-action data, and trains a vision-language model to predict language and actions.

3.1. Task overview

  • SimLingo’s three tasks are closed-loop driving, vision-language understanding, and Action Dreaming, which tests whether language instructions align with predicted actions.Commentary explains what the ego vehicle should do next and why, and is enabled by default at inference, conditioning action prediction on generated language. Action Dreaming includes instructions that should be rejected, such as crashing into objects.Source: §3.1 · pp. 2–3

3.3. Architecture

  • SimLingo uses camera images, navigation, speed, and task prompts with InternVL2-1B to generate language and disentangled temporal-speed and geometric-path waypoints.The waypoints feed PID controllers for acceleration and steering; geometric path waypoints provide denser steering supervision, including when stationary. Images are tiled at 448×448 resolution, with two tiles yielding 512 visual tokens; training switches between GPS target points and language commands.Source: Figure 2 · p. 3, §3.3 · pp. 4–5

3.2. Datasets

  • SimLingo jointly trains for expert driving and simulated alternative actions, with language supervision for VQA, commentary, and deciding whether an instruction is safe to execute.Expert driving data comes from PDM-lite [53] on CARLA routes, with balancing and weather augmentation, totaling 3.1 million samples at 4 fps. For Action Dreaming, simulated alternative trajectories and paired instructions ensure the model must attend to language; the data labels safety and reasons for rejection.Source: §3.2 · pp. 3–4

3.4. Training

  • Training samples expert and dream trajectories equally, varies language supervision for expert data, and uses weighted data buckets to emphasize challenging scenarios.Expert samples use VQA 50% of the time, commentary prediction 35%, commentary in the prompt 7.5%, and no language supervision 7.5%; dream samples split evenly between accepting and rejecting instructions. Bucket sampling reduces samples per epoch to 650,000.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate camera-only driving, vision-language understanding, and language-action alignment across CARLA and language-based benchmarks. SimLingo achieves strong driving and language results while retaining driving performance when language capabilities are added.

4.1. Benchmarks and Metrics

  • Evaluation spans CARLA Leaderboard 2.0 secret routes, 220-route Bench2Drive, balanced DriveLM-hard and Commentary validation sets, and open-loop Action Dreaming.Reported measures include Driving Score, Route Completion, and Infraction Score on the leaderboard; Bench2Drive driving score, success rate, efficiency, and comfortness; GPT and SPICE for language; and success rate for Action Dreaming.Source: §4.1 · p. 5

4.3. Results

  • SimLingo-BASE outperforms CaRINA hybrid by 4.6x and TF++ by 33% on the CARLA Leaderboard 2.0 Sensor track, using camera images alone.The authors also report that disentangled path-plus-speed waypoints raise driving score by 39.9% and reduce static-object collisions to zero.Source: Table 1 · p. 6, §4.3 · p. 7

4.2. Implementation Details

  • The model is trained for 14 epochs on eight A100 80GB GPUs, and inference predicts action and reason in language before conditioning action prediction on generated commentary.Training uses LoRA for the LLM and SmoothL1 loss; the leaderboard model uses inference early stopping to address the Driving Score metric’s behavior.Source: §4.2 · p. 6

4.3. Results

  • On Bench2Drive, SimLingo and SimLingo-BASE outperform prior methods on driving score and success rate; adapting TCP-traj raises DS from 49.30 to 63.45.The adapted baseline uses PDM-lite data composition and a controller tuned for that dataset to improve comparison fairness.Source: Table 2 · p. 7, §4.3 · p. 7
  • On the balanced DriveLM and Commentary evaluation split, SimLingo outperforms zero-shot InternVL2 models; the DriveLM-only fine-tuned InternVL2-1B-DriveLM reaches GPT-score 63.85, only slightly above SimLingo.SimLingo especially struggles with answer types rarely seen during training.Source: Table 3 · p. 7, §4.3 · pp. 7–8
  • Using language commands for navigation yields driving performance within the variance of GPS target-point conditioning, enabling instruction-based control without the target-point recovery shortcut.The comparison uses the same model with GPS target points or high-level language commands.Source: Table 4 · p. 8, §4.3 · p. 8
  • Training on synthetic instruction-action pairs increases Action Dreaming success rate from 28.22 to 72.96 compared with training only on CARLA simulator instructions.Qualitative examples show the model adapting its predicted path to both instruction and scene context.Source: Table 5 · p. 8, §4.3 · p. 8
  • Adding VQA and Commentary training leaves driving performance similar to a pure driving model, while Action Dreaming training slightly improves Bench2Drive performance.The ablation compares a model with non-aligned language understanding against the pure driving model.Source: Table 6 · p. 8, §4.3 · p. 8

5. Conclusion and Limitations

SimLingo combines strong simulated driving performance with driving-language understanding and action prediction. Its evaluation is limited by the untested full model on the official leaderboard, the lack of observed significant driving gains from CoT, and simulation-only testing.

Conclusion

  • SimLingo achieves state-of-the-art results on the official CARLA Leaderboard 2.0 and Bench2Drive while demonstrating language understanding.The conclusion reports language performance on driving Commentary and VQA after finetuning InternVL2 for the driving domain.Source: §5 · p. 8
  • Action Dreaming aligns vision-language understanding with actions by predicting actions for a wide variety of language instructions with a high success rate.The paper introduces Action Dreaming as a task for aligning language understanding with the driving action space.Source: §5 · p. 8

Limitations

  • Only SimLingo-BASE was tested on the official CARLA Leaderboard 2.0 because it closed in June 2024, though both SimLingo models were rigorously checked on local Bench2Drive.Testing the full SimLingo model on the official leaderboard is left for future work.Source: §5 · p. 8
  • CoT conditions driving actions on intermediate commentary, but the authors observed no statistically significant driving improvements and hypothesize that CoT-specific data and training recipes are needed.The reported lack of significant improvement is referenced in Appendix C.3.Source: §5 · p. 8
  • Driving and language understanding were evaluated only in simulation, and VLM inference latency remains a real-world challenge that smaller models and engineering efforts such as may address.The authors identify real-time in-car VLM testing as a future possibility.Source: §5 · p. 8

A. Datasets

The section describes the collected datasets and how labels are generated for each language-related dataset.

A. Datasets

  • The section details the collected datasets and label-generation process for each language-related dataset.Source: A. Datasets · p. 12

A.1. Driving dataset - Scenarios … A.4. Action Dreamer

The appendix details the scenario coverage and language-label generation used for SimLingo’s dataset, then describes Action Dreamer’s offline simulation for generating and evaluating alternative ego trajectories.

A.1. Driving dataset - Scenarios

  • The collected driving data covers intersection and highway interactions, including turns, merges, cut-ins, red-light running, and yielding to emergency vehicles.The scenario set also includes obstacles, hazards, pedestrians, and parking maneuvers, with responses such as braking, yielding, or changing lanes.Source: §A.1 · p. 12

A.2. VQA - DriveLM

  • VQA training uses 28M generated question-answer pairs for 1M Town 12 frames, with DriveLM keyframes for evaluation and GPT-4-generated wording alternatives to reduce phrase overfitting.The validation set is balanced across answer types; the generated labels use heuristic-based procedures and consistent sentence structures.Source: §A.2 · pp. 12–13

A.3. Commentary

  • Commentary labels combine route action and justification, speed action, and a reason derived from the leading object or traffic condition.For route deviations, labels distinguish before, during, and after the maneuver; the return-to-route wording is generic because only front-view cameras are used.Source: §A.3 · p. 13
  • Speed descriptions use current and target speed plus the speed limit, while IDM features identify leading objects and inform object-specific reasons and junction notices.The junction notices describe whether other vehicles are stopped with the junction clear or moving away, or advise attention to approaching or junction vehicles.Source: §A.3 · pp. 13–14

A.4. Action Dreamer

  • Action Dreamer forecasts ego poses with a Kinematic Bicycle Model and PDM-lite PID controllers, then checks trajectories against non-reactive object paths over the next 10 timesteps.Ground-truth actions and expert-planner waypoints spaced every 10 cm are the defaults; alternative trajectories modify actions, paths, or target speeds.Source: §A.4 · p. 14
  • Its alternatives include reachable-object collision scenarios, faster and slower driving, random target speeds from 0 to 35 m/s, and conditioned lane changes.Object mode filters for objects within 15 meters of the ego route and at least 3 meters ahead; lane changes exclude junctions and ongoing lane changes.Source: §A.4 · p. 14

B. Implementation details · B.1. Hyperparameter

The implementation-details passage identifies Table 7 as presenting the hyperparameters used to train SimLingo.

B.1. Hyperparameter

  • Table 7 presents the hyperparameters used to train SimLingo.The supplied passage does not include the hyperparameter values.Source: §B.1 · p. 14

B.2. Training buckets · B.3. SimLingo vs. SimLingo-BASE

The sections describe targeted sampling of challenging driving data and the design and training differences between SimLingo-BASE and SimLingo. SimLingo-BASE is a lightweight model for studying driving-specific design choices, while SimLingo adds language capabilities and uses different components and settings.

B.2. Training buckets

  • Because uneventful straight driving is learned early, training on the full dataset later wastes computation, so the authors sample from buckets of interesting scenarios.Some easy, uneventful data remains inevitable, motivating a bucket that samples from the whole dataset.Source: §B.2 · p. 14

B.3. SimLingo vs. SimLingo-BASE

  • SimLingo-BASE independently encodes two image splits, concatenates, downsamples, and projects them before a LLaMA-based transformer decoder; it shares SimLingo’s output representation.The architecture is shown in Figure 5.Source: Figure 5 · p. 15

B.2. Training buckets

  • The training buckets cover acceleration and deceleration, steering, vehicle and other hazards, obstacle swerving, older Leaderboard 1.0 towns, and a sample of the full dataset.There are five acceleration/deceleration buckets, two steering buckets, three vehicle-hazard buckets, and one each for stop signs, red lights, walkers, obstacle swerving, Towns 01–10, and whole-dataset sampling; straight-driving samples are excluded from the steering buckets, and acceleration values between -1 and 1 are excluded from the acceleration buckets.Source: §B.2 · p. 15

B.3. SimLingo vs. SimLingo-BASE

  • SimLingo uses 14 training epochs, compared with 30 for SimLingo-BASE.The authors did not repeat SimLingo-BASE experiments under SimLingo’s new settings because the CARLA Leaderboard had closed and computational overhead was high; they expect the earlier claims and results to hold.Source: §B.3 · p. 15
  • SimLingo replaces SimLingo-BASE’s Clip-ViT image encoder with InternVL2-1B’s original encoder; both models use full finetuning.SimLingo-BASE uses Clip-ViT as the default in the LLaVA VLM.Source: §B.3 · p. 15
  • SimLingo-BASE uses a 50M-parameter LLaMA-based decoder trained from scratch, whereas SimLingo fine-tunes InternVL2-1B’s pretrained LLM with LoRA.The BASE decoder is a transformer trained from scratch.Source: §B.3 · p. 15
  • After Action Dreaming data caused training instability with L2 loss, SimLingo switched to SmoothL1 loss from SimLingo-BASE’s L2 loss.The passage attributes the loss change to observed training instability after adding Action Dreaming data.Source: §B.3 · p. 15

B.4. Metric descriptions · C. Quantitative Results

The section reports SimLingo’s multi-ability Bench2Drive results and defines the driving, language-understanding, and action-dreaming metrics used to evaluate performance.

B.4. Metric descriptions

  • Leaderboard 2.0 ranks methods primarily by overall Driving Score, averaging route metrics and combining route completion with an infraction penalty.The final metrics are arithmetic means across routes; global values determine the overall score.Source: §B.4 · pp. 15–16
  • Infraction penalties reduce a base score of 1.0 by severity, while off-road driving reduces route completion and specified route failures stop evaluation immediately.Penalties include pedestrian, vehicle, and static-object collisions, red-light and stop-sign violations, and emergency-vehicle yielding failures; route stops include deviation over 30 meters, blocking over 180 simulation seconds, communication timeout over 60 seconds, or route timeout.Source: §B.4 · p. 16
  • Bench2Drive omits the minimum-speed penalty from Driving Score, adds Efficiency to reflect speed relative to surrounding actors, and avoids Leaderboard 2.0’s early-stopping tradeoff on short routes.Success Rate measures completed routes without infractions, ignoring the minimum-speed penalty.Source: §B.4 · p. 16
  • Comfortness evaluates jerk, acceleration, yaw acceleration, and yaw rate against repository-defined thresholds; VQA and Commentary use SPICE and a modified DriveLM GPT Score.The GPT Score uses GPT-4 instead of GPT-3.5 and adds a prompt instruction to rate content similarity rather than sentence structure.Source: §B.4 · p. 16
  • Action Dreaming uses category-specific Success Rate criteria for slowing, speeding up, target speed, lane changes, and object collisions.Target-speed success checks predictions against instructed and ground-truth speed ranges; lane-change success compares the predicted endpoint with dreamer and expert paths, while object success depends on path divergence or speed accuracy.Source: §B.4 · pp. 16–17

C. Quantitative Results

  • SimLingo outperforms existing methods across all Bench2Drive abilities, improving the average by 25 percentage points.Table 8 presents the multi-ability results.Source: Table 8 · p. 17
  • The SimLingo-BASE ablations indicate that disentangled output representation is superior and that the early-stopping threshold has a large impact on Leaderboard 2.0 scores.Table 9 reports official Leaderboard 2.0 scores and highlights the default configuration.Source: Table 9 · p. 17

C.1. Ablations on CARLA Leaderboard 2.0

SimLingo-BASE ablations show benefits from disentangled path prediction and vision-language pretraining, while early-stopping thresholds substantially affect Leaderboard DS. Repeated submissions also show score variance, and inference-time chain-of-thought provides a small improvement.

Output representation

  • Disentangled output representation reduces static-layout collisions from 0.68 to 0, supporting the use of additional path predictions for lateral control.The ablation compares output representations on the Leaderboard; static-layout collisions are reported because they are mainly caused by poor steering.Source: §C.1 · p. 17

Vision-Language and CLIP pretraining

  • Vision-language pretraining is essential for strong driving performance, and ImageNet-pretrained ResNet-34 scores 2.71 vs. 6.87 DS, underscoring the larger ViT and internet-scale pretraining.Training from scratch performs worse; the passage notes that longer training may improve it but is unlikely to match the pretrained model.Source: §C.1 · pp. 17–18

Inference Mode

  • Using chain-of-thought during inference yields a small Bench2Drive improvement compared with inference without chain-of-thought.Table 10 compares SimLingo inference with and without CoT; the passage does not provide score values.Source: Table 10 · p. 18

Early stopping

  • Leaderboard DS varies substantially with early-stopping distance, and the secret routes and scenario density make the ideal threshold difficult to determine.The authors calculate a rough expected-DS function to identify a threshold range; the ablation reports DS for given traveled distances.Source: §C.1 · p. 18

Leaderboard variance

  • At a 2100-meter threshold, three submissions score 6.9, 5.5, and 5.3, with mean DS 5.9 and standard deviation 0.87.The repeated submissions estimate Leaderboard evaluation variance.Source: §C.1 · p. 18
  • At a 2400-meter threshold, three submissions score 6.4, 6.3, and 4.8, with mean DS 5.83 and standard deviation 0.90.These repeated submissions provide a second estimate of Leaderboard evaluation variance.Source: §C.1 · p. 18

C.2. Bench2Drive Multi-Ability Results · C.3. Chain-of-Thought Inference Mode

SimLingo outperforms existing works across Bench2Drive multi-ability metrics, with especially strong gains in four abilities while Give way remains challenging. Using Commentary as Chain-of-Thought (CoT) yields a small improvement, but inference without CoT remains feasible when speed matters.

C.2. Bench2Drive Multi-Ability Results

  • SimLingo outperforms existing works on Bench2Drive multi-ability metrics, with large boosts in Merging, Overtaking, Emergency Brake, and Traffic Sign, while Give way remains challenging.The passage points to Tab. 8 but provides no metric values.Source: §C.2 · p. 18

C.3. Chain-of-Thought Inference Mode

  • Using Commentary as Chain-of-Thought (CoT) gives a small improvement and is the default, while inference without CoT remains feasible when speed matters.The performance drop without CoT is described as small.Source: §C.3 · p. 18

D. Qualitative Results

Closed-loop examples show SimLingo adapting to navigational commands and usually following Dreamer instructions, while also revealing confusion on misleading prompts and limitations with unseen or ignored commands.

D. Qualitative Results

  • In closed loop, SimLingo changes lanes when told “Turn left,” then stays in its lane; it handles misleading and nonsensical commands, but cannot execute the unseen “Do a U-Turn.”When asked “Turn right” at an intersection without a right-turn lane, it proceeds straight and stays on the road; a context question about a tree on the right still leads it to choose a left turn.Source: D. Qualitative Results · p. 18, Figure 6 · p. 19
  • Across Dreamer modes, the model usually follows instructions—even against expert behavior such as accelerating at a red light—but red-boxed examples show it sometimes ignores them.The figures cover faster, slower, target-speed, collision, and lane-change modes.Source: D. Qualitative Results · p. 18, Figure 7 · p. 20, Figure 8 · p. 20

Language/Vision -> Action (Dreaming - Lane change)

  • Dreamer lane-change prompts specify varied start distances and transition lengths, including changes right or left with transitions from 2 to 16 meters.The listed commands include starts at 0, 1, and 5 meters, with transition lengths of 2, 5, 7, and 16 meters.Source: D. Qualitative Results · p. 22
  • The lane-change prompts also vary their phrasing, using paraphrases such as “veer,” “glide,” and “navigate” to express directional lane changes.These examples include rightmost-lane instructions and a direction-qualified destination in the same direction of travel.Source: D. Qualitative Results · p. 22
  • Other prompts specify lane positions by ordinal and side, including targets in the same direction and lanes in the opposite direction.Examples name the second lane from the left, the third lane from the right in the opposite direction, the rightmost lane of the opposite direction, and the first lane from the right.Source: D. Qualitative Results · p. 22

Related papers

Loading 2503.09594v1…