Source-linked AI summary
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, Jamie Shotton
TL;DR
Autonomous driving systems remain difficult to interpret and support with out-of-distribution reasoning, while integrating new numeric modalities into LLMs requires substantial data and computation. This paper fuses object-level vectorized scene representations with a pretrained LLM, using structured-language pretraining and Driving QA, and reports preliminary gains in question answering and action prediction.
Problem
Autonomous driving systems often operate as black boxes, and integrating new vector modalities into pretrained LLMs requires extensive data and computational resources.
Method
The paper integrates object-level vectorized 2D scene representations into a pretrained LLM with adapters, using structured language to bridge vector spaces and language embeddings and formulating driving as Driving Question Answering.
Results
LLM-based policies outperform Perceiver-BC for action prediction under matched training data and trainable parameters, while vector-modality pretraining improves Driving QA grading scores by 9.1% and 10.8% under GPT and human grading.
Takeaways & Limitations
The architecture and its modality-grounding, data-labeling, and evaluation pipelines establish a foundation for research integrating numeric vector modalities with LLMs for autonomous driving.
Takeaways & Limitations
Results are preliminary, with open-loop versus closed-loop discrepancies, lengthy LLM inference, extensive testing requirements, and insufficient precision in baseline driving commands for effective closed-loop operation.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown promise in the autonomous driving sector, particularly in generalization and interpretability. We introduce a unique object-level multimodal LLM architecture that merges vectorized numeric modalities with a pre-trained LLM to improve context understanding in driving situations. We also present a new dataset of 160k QA pairs derived from 10k driving scenarios, paired with high quality control commands collected with RL agent and question answer pairs generated by teacher LLM (GPT-3.5). A distinct pretraining strategy is devised to align numeric vector modalities with static LLM representations using vector captioning language data. We also introduce an evaluation metric for Driving QA and demonstrate our LLM-driver's proficiency in interpreting driving scenarios, answering questions, and decision-making. Our findings highlight the potential of LLM-based driving action generation in comparison to traditional behavioral cloning. We make our benchmark, datasets, and model available for further exploration.
1 Introduction
The paper addresses the interpretability and spatial-understanding limitations of autonomous-driving systems by integrating object-level vector representations with pretrained LLMs. It contributes a multimodal architecture, scalable data pipeline, Driving QA dataset, and evaluation approach for simulation.
- Motivation: Modern autonomous-driving systems often remain black boxes, limiting out-of-distribution reasoning and interpretability.These limitations persist across end-to-end and component-based system designs.
- Motivation: Text and symbolic inputs support reasoning and communication but have limited nuanced spatial understanding for autonomous navigation.The paper positions multimodal integration as a way to combine language capabilities with spatial scene information.
- Approach: The proposed architecture fuses vectorized object-level 2D scene representations into pretrained LLMs using adapters.The numeric modality represents information such as speed, actuator positions, and distance measurements.
- Approach: Training data are collected with a custom 2D simulator and an RL driving agent, while structured language grounds numerical vectors for LLM pretraining.A teacher LLM generates question-answer data conditioned on language descriptions of driving scenarios.
- Contributions: The work provides a 160k-pair Driving QA dataset covering 10k driving situations with control commands.The dataset is paired with RL-agent driving behavior and expert-LLM-generated question-answer labels.
- Contributions: The paper introduces a Driving QA evaluation method using an expert LLM grader and reports an end-to-end multimodal baseline.The framework encompasses simulation, automatic data collection, vector-modality integration, and GPT-based evaluation.
2 Related Works
Related work spans interpretability methods, multimodal foundation models, vision-language driving systems, and language-grounded robotics. These lines of research motivate combining pretrained language models with structured driving modalities.
- End-to-End Autonomous Driving Systems: Autonomous-driving research has advanced end-to-end deep learning, but interpretability of decision making remains a fundamental challenge.Interpretability matters for uncertainty identification, trust, human-AI collaboration, and safety.
- Explainability Methods: Model-agnostic, gradient-based, saliency-map, and attention-map methods have been used to explain neural-network predictions and internal operations.The passage places these approaches within a broader body of explainability methods.
- Multimodal Models: Vision-language models unify images and language for tasks including image captioning, visual question answering, and image-text similarity.Other multimodal models combine information from multiple sensor and media types.
- Vision-Language Driving: Driving research has used visual observations and language to summarize scenes, follow directions, and predict vehicle controls.These systems include policies trained from images and language directions and models involving human assistance.
- Language-Grounded Robotics: Robotics research has fused language with point clouds, 3D object candidates, and low-level control through LLM-based approaches.These efforts provide nearby examples of language grounding beyond autonomous driving.
3 Method
The method builds a Driving QA pipeline that converts object-level vector scenes into structured language, generates GPT-labeled questions, and trains an LLM agent through vector grounding and DQA finetuning.
- Data Generation: The framework uses a custom 2D simulator and an RL agent to collect continuous driving data from diverse virtual environments.The RL policy operates on object-level ground-truth scene representations and is optimized with PPO.
- Structured Language Generation: Structured language generation converts semantically rich object vectors into human-readable captions describing observations, state, and planned actions.The vectors include object counts, locations, orientations, speeds, bounding boxes, and related attributes.
- Driving QA Dataset: GPT generates 16 question-answer sets per scenario from the structured descriptions, including driving, action-related, unanswerable, and unrelated questions.Guardrails require observation-grounded responses when information is absent and scope-limited responses to unrelated questions.
- Training: Training uses two stages: vector representation pretraining with the LLM frozen, followed by finetuning for driving questions and actions.Vector encoder and transformer weights are optimized during grounding so their embeddings can be decoded by the LLM.
- Driving LLM Agent: The Driving LLM Agent fuses vector inputs with a frozen LLM through a Vector Encoder, Vector Former, and LoRA module.The Vector Encoder uses MLP layers and cross-attention, adding ego features to learned input latents.
- Training: The resulting agent is refined on 10,000 scenarios and can reason about actions and answer driving-related questions from current object-level vectors.The training scenarios incorporate 16 driving QA entities, captioning data, and an action-prediction QA pair.
4 Evaluation and Results
The evaluation examines perception, action prediction, and open-ended Driving QA using separate scenarios and LLM- or human-based grading. Pretraining improves perception, action prediction, and QA scores, while comparisons with Perceiver-BC have methodological limitations.
- Evaluation setup: The evaluation uses 1,000 distinct driving scenarios and prompts the model with vector captions plus an action-prediction question.Numerical outputs are extracted from the language response using regular expressions.
- Evaluation setup: Perceiver-BC replaces the pretrained LLM with a non-pretrained Perceiver while matching approximately 25 million trainable parameters.It is trained on the same 10k dataset but without VQA data.
- Metrics: The reported metrics cover perception errors, traffic-light detection and distance, control prediction errors, and weighted token cross-entropy loss.The metrics include Ecar, Eped, AccTL, DTL, Elon., steering error, and Ltoken.
- Perception and action prediction: Pretraining significantly enhances the model’s perception and action prediction capabilities and lowers Ltoken.The reported improvement includes more accurate car and pedestrian quantification and more effective token prediction.
- Perception and action prediction: LLM-based policies outperform Perceiver-BC on the reasoning-intensive action prediction task with the same training data and trainable parameters.The paper attributes this to knowledge acquired during general pretraining, including driving responses such as stopping at red lights.
- Perception and action prediction: The Perceiver-BC comparison is not entirely equitable because the LLMs use cross-entropy training and 16x more driving QA pairs.The authors therefore describe the comparison as a point of reference.
- Driving QA: Driving QA responses are graded by GPT-3.5 on a 0–10 scale, with the final score averaged across questions.Hand-scoring 230 randomly sampled QA pairs produced comparable results, although GPT grading can be lenient toward semantically close incorrect answers.
- Driving QA: 9.1% and 10.8%: pretraining improves grading scores over models without pretraining under GPT and human grading, respectively.The vector-only model also scores substantially above constant or randomly shuffled answer baselines.
5 Conclusion and Limitations
The paper presents a promising foundation for integrating vector modalities with LLMs in autonomous driving, but its preliminary system still requires substantial refinement for effective closed-loop navigation and real-world adaptation.
- Limitations: Preliminary results indicate that the LLM does not yet fully navigate in simulation.The authors identify open-loop versus closed-loop discrepancies and ongoing challenges in evaluating the system efficiently in closed loop.
- Limitations: Closed-loop operation is constrained by lengthy LLM inference, extensive testing requirements, and insufficient precision in direct driving commands.The authors attribute these issues to task intricacy, model architecture, and the scale and quality of the training dataset.
- Limitations: Numeric inaccuracies and the system’s early developmental stage can create discrepancies in explanations and reasoning, preventing full implementation in closed-loop settings.
- Conclusion: The architecture, modality-grounding method, data auto-labelling pipeline, and LLM-based evaluation pipeline establish a starting point for future research.
- Future Work: Adapting the methodology to real-world driving would require sufficient real-world perception labels for pretraining and fine-tuning a VLM.Improving the architecture’s handling of numeric-vector nuances is identified as another future direction.
6 Ethical Implications and Broader Impact
Introducing LLMs into autonomous driving brings inherited ethical implications and reinforces the need for systems that safely handle all possible driving scenarios.
- Ethical Implications: LLM-based autonomous driving inherits ethical implications associated with LLMs.
- Broader Impact: The authors stress that autonomous-driving systems must handle all possible driving scenarios safely.
- Broader Impact: Improved interpretability could help build trust, accelerate adoption, and support safer and more efficient transportation.
Appendix A Perceiver-BC model
The appendix compares the LLM agent with a Perceiver-based behavioral-cloning model designed with shared vector-processing modules and a similar trainable-parameter budget.
- Motivation: The Perceiver-BC model was introduced to compare pretrained LLM reasoning proficiency with a simpler action-prediction baseline.
- Architecture: The baseline shares the LLM agent’s Vector Encoder and Vector Former but replaces the LLM with adapters by a perceiver-based policy module.
- Architecture: The comparison was calibrated to approximately 25 million trainable parameters for the Perceiver-BC model and the LLM agent.
- Training: The Perceiver-BC model used the same 10k dataset and five training epochs as the LLM agent, with only captioning and action data.
- Architecture: Figure 5 presents the architecture of the behaviour cloning baseline.
Appendix B Additional Results
The appendix provides additional DrivingQA results and includes failure cases to supplement the main evaluation.
- Additional Results: Additional DrivingQA results are presented in Figure 6.
- Additional Results: Figure 7 presents further DrivingQA results together with failure cases, including failure cases in its final two examples.
B.1 Design of the Object-level Vector Representation
The paper represents driving conditions as compact object-level vectors containing semantically meaningful information about routes, nearby vehicles, pedestrians, and the ego vehicle. Structured language labels derived from these vectors make complex situations interpretable to an LLM for reasoning, decision-making, and explainable outputs.
- Object-level vectors encode the intended route, nearby vehicles, pedestrians, and ego-vehicle state as compact driving-condition representations.The schema includes route, nearby-vehicle, pedestrian, and ego-vehicle descriptors.
- Semantically significant vector attributes are converted into structured language labels that an LLM can understand.The conversion provides a language-format representation of complex driving situations.
- The language-form representation supports understanding driving conditions, making decisions, and generating explainable outputs from internal reasoning.
B.2 RL expert
The RL expert is trained to map object-level driving vectors to executable vehicle actions in simulation. It combines heuristic desired-speed and stopping signals with PPO-based closed-loop learning and predicts acceleration and steering distributions.
- The RL agent maps object-level vectors to executable actions, using independent beta distributions for acceleration and steering-wheel angle.Acceleration is partitioned into accelerator pedal and brake pressure before vehicle-dynamics simulation.
- Proximal Policy Optimization trains the action distributions through closed-loop simulation and replay-buffer optimization.Multiple vehicles are controlled simultaneously to generate 100 to 200 thousand replay-buffer samples efficiently.
- Perceiver IO preprocesses simulator vectors into tokens and uses separate output queries for acceleration, steering, and value estimation.
- The reward incorporates desired speed, lane-centering and heading terms, steering smoothness, and a failed-to-stop penalty.The reward variables include vehicle speed, desired speed, lateral lane distance, lane angle, consecutive steering actions, and scaling hyper-parameters.
- Desired speed is selected as the minimum of heuristics for junctions, nearby vehicles, traffic lights, pedestrians, and lead vehicles.
- The failed-to-stop heuristic is intended to ensure appropriate stopping, including a give-way condition when speed exceeds 2m/s.