Source-linked AI summary
CarLLaVA: Vision language models for camera-only closed-loop driving
Katrin Renz, Long Chen, Ana-Maria Marcu, Jan Hünermann, Benoit Hanotte, Alice Karnsund, Jamie Shotton, Elahe Arani, Oleg Sinavski
TL;DR
CarLLaVA addresses camera-only closed-loop autonomous driving without expensive auxiliary labels. It combines vision-language pretraining with waypoint and path outputs and scenario-focused training; the report presents it as outperforming the previous state of the art on the CARLA leaderboard.
Problem
Method
CarLLaVA uses a pretrained LLaVA vision encoder and LLaMA decoder to predict time-conditioned waypoints and space-conditioned path waypoints from camera input, with training focused on challenging samples.
Source: §1 · p. 1, §3 · pp. 2–3Results
Takeaways & Limitations
The report concludes that vision-language pretraining and camera-only input can support closed-loop driving performance without expensive labels or sensors.
Source: §5 · p. 5
Abstract
from arXiv · showhide
In this technical report, we present CarLLaVA, a Vision Language Model (VLM) for autonomous driving, developed for the CARLA Autonomous Driving Challenge 2.0. CarLLaVA uses the vision encoder of the LLaVA VLM and the LLaMA architecture as backbone, achieving state-of-the-art closed-loop driving performance with only camera input and without the need for complex or expensive labels. Additionally, we show preliminary results on predicting language commentary alongside the driving output. CarLLaVA uses a semi-disentangled output representation of both path predictions and waypoints, getting the advantages of the path for better lateral control and the waypoints for better longitudinal control. We propose an efficient training recipe to train on large driving datasets without wasting compute on easy, trivial data. CarLLaVA ranks 1st place in the sensor track of the CARLA Autonomous Driving Challenge 2.0 outperforming the previous state of the art by 458% and the best concurrent submission by 32.6%.
1. Introduction
CarLLaVA targets camera-only, end-to-end driving without costly auxiliary labels, using vision-language pretraining and a semi-disentangled output representation. Its training recipe prioritizes challenging samples to reduce wasted compute.
1. Introduction
- CarLLaVA uses only camera images and driving trajectories, avoiding expensive BEV, depth, or semantic-segmentation labels that impede scalable transfer from simulation to real-world driving.The introduction contrasts this with methods requiring LiDAR or costly labels.Source: §1 · p. 1
- The model uses the LLaVA-NeXT vision encoder pretrained on internet-scale vision-language data, and the report says this transfers to driving better than training from scratch.The authors also highlight high-resolution image patches and a semi-disentangled representation of time-conditioned waypoints and space-conditioned path waypoints.Source: §1 · p. 1
- The training recipe samples from buckets of interesting scenarios because uneventful straight-driving examples are learned early and otherwise consume later training compute.The paper describes reducing training time by making greater use of interesting samples.Source: §1 · p. 1
2. Related Work
Prior foundation-model driving work has often been assessed qualitatively or in open-loop settings, while closed-loop CARLA systems using multimodal LLMs rely on LiDAR and customized encoders. CarLLaVA instead targets camera-only closed-loop driving performance.
Foundation models for driving
- Many foundation-model driving systems are evaluated mainly qualitatively or open-loop; DriveMLM and LMDrive are closer closed-loop CARLA comparisons but use image and LiDAR with customized encoders.Those systems focus on instruction following, whereas CarLLaVA focuses on closed-loop driving performance as a baseline for VLM driving research.Source: §2 · p. 2
End-to-end closed-loop driving in CARLA
- End-to-end CARLA methods commonly use auxiliary outputs and costly sensors, while CarLLaVA uses camera images and trajectory input with path waypoints for steering and standard waypoints for longitudinal control.This avoids heuristics or predefined speed classes, and predicts waypoints directly from transformer features without a GRU.Source: §2 · p. 2
3. Method
CarLLaVA maps camera observations, target points, and ego speed to waypoint and path predictions. Its architecture uses a pretrained vision encoder and language-model decoder, while its training data and sampling strategy emphasize challenging driving scenarios.
3. Method
- The model receives camera images, the next two target points, and ego speed, with single-view, temporal, and multi-view input configurations.The task is navigating a 10x10 km2 map through target points and varied environments, weather, and traffic scenarios.Source: §3 · p. 2
- Its semi-disentangled output pairs time-conditioned waypoints for PID longitudinal control with space-conditioned path waypoints for PID lateral control.The authors report that entangled waypoints led to steering errors, while path predictions provide denser supervision, including when the vehicle is stationary.Source: §3 · p. 2, Figure 1 · p. 2
- The LLaVA-NeXT CLIPViT-L-336px encoder processes high-resolution images as independently encoded patches, then concatenates their features spatially; a downsampling adapter projects them into the LLaMA decoder.Learnable queries generate path and waypoint outputs, and cumulative sums of predicted waypoint differences form the final waypoints.Source: Figure 1 · p. 2, §3 · p. 3
- For preliminary language explanations, the model autoregressively generates commentary after path and waypoint outputs, training on tokenized explanations with language-model loss.The explanation labels are generated using rule-based expert logic and heuristics; the report leaves detailed label statistics to future work.Source: §3 · p. 3
- They collect 2.9 million samples at 5 fps from shortened official Town 12 and Town 13 CARLA routes centered on scenarios, balancing scenario counts and augmenting weather and scenario distances by ±10%.Because short single-scenario routes often end within 200 meters, they also use routes featuring three scenarios per route.Source: §3 · p. 3
- Training buckets focus on acceleration, steering, hazards, and obstacle swerving, while retaining a bucket sampled from the full dataset for some uneventful driving.This reduces samples per epoch to 650,000 because hazard-free straight driving is learned early and otherwise wastes later training compute.Source: §3 · pp. 3–4
4. Experiments
The experiments report CarLLaVA’s official leaderboard comparison and assess output design, vision pretraining, early stopping, model scale, input extensions, and language output.
Benchmarks and implementation
- Evaluation uses official secret-route Leaderboard 2.0 and local 10xShort routes, with up to 10 randomly sampled routes per scenario; base C1T1 trains in about 27 hours on 8xA100 40GB GPUs.The report lists Driving Score, Route Completion, and Infraction Score as official metrics.Source: §4 · p. 4
Leaderboard state of the art
- 5.18 vs 6.87 DS: base CarLLaVA C1T1 is reported to outperform the state of the art on the official leaderboard.The authors note high leaderboard-score variance and say their reported official numbers were the first submissions.Source: §4 · p. 4
Output representation and vision pretraining
- Semi-disentangled output reduces static-layout collisions from 0.68 to 0.0, while vision pretraining is described as essential for good driving performance.The ResNet-34 ImageNet-pretrained comparison is reported as 2.71 vs 6.87 DS, which the authors associate with the larger ViT and internet-scale image-language pretraining.Source: §4 · p. 4
Early stopping
- Early-stopping distance has a big impact on Leaderboard DS, and the authors describe the trade-off as difficult to tune with secret routes and scenario density.They evaluate Leaderboard DS at different travelled distances.Source: §4 · pp. 4–5
Leaderboard variance
- Three repeat submissions yielded mean DS 5.87 (SD 0.81) at threshold 2100 and mean DS 5.8 (SD 0.87) at threshold 2400.The authors report these repetitions as an estimate of leaderboard evaluation variance.Source: §4 · p. 5
Scale and input extensions
- A 350M model trained from scratch improves performance slightly, whereas 1B LoRA variants perform worse; added temporal or back-camera input improves expected scenarios but not overall score.Table 3 reports further model ablations, with DSS denoting 10xShort benchmark performance.Source: §4 · p. 5, Table 3 · p. 5
Language explanations
- The model generates driving commentary, but the authors caution that it is not always aligned with the actions taken; Figure 2 marks paths red, waypoints green, and target points blue.The commentary results are preliminary.Source: §4 · p. 5, Figure 2 · p. 5
Failure cases
- Rear-end collisions are a common failure case that temporal input can reduce, while merging at high speeds is also identified as a failure mode.The report names the temporal-input C1T2 model in connection with reducing rear-end collisions.Source: §4 · p. 5
5. Conclusion
CarLLaVA is presented as the winning CARLA Autonomous Driving Challenge 2.0 2024 entry, using vision-language pretraining with camera-only input. Its semi-disentangled outputs and efficient training approach are associated with superior lateral and longitudinal control and substantial improvement over previous methods.
Contribution
- CarLLaVA is reported as the winning CARLA Autonomous Driving Challenge 2.0 2024 entry and uses vision-language pretraining with camera images only.The conclusion says it avoids expensive labels or sensors.Source: §5 · p. 5
Approach
- Its semi-disentangled output representation and efficient training approach are associated with superior lateral and longitudinal control.Source: §5 · p. 5
Implications
- The authors characterize CarLLaVA as a scalable, cost-effective solution and report substantial improvement over previous methods.They describe the results as indicating potential for vision-language models in real-world autonomous driving applications.Source: §5 · p. 5