PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveLM: Driving with Graph Visual Question Answering

Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, Hongyang Li

arXiv:2312.14150v3cs.CV

TL;DR

Autonomous-driving systems need better generalization and human-friendly interaction, but existing VQA tasks do not model humans’ multi-object, multi-step reasoning. The paper introduces Graph VQA, DriveLM datasets, and DriveLM-Agent; the baseline is competitive with driving-specific models and shows stronger zero-shot generalization on unseen objects or sensor configurations.

  • Problem

    Existing driving VQA paradigms do not provide a suitable proxy for humans’ multi-object, multi-step perception, prediction, and planning reasoning.

  • Method

    The paper formulates Graph VQA as logically dependent QA pairs in a graph, builds DriveLM datasets, and uses DriveLM-Agent for Graph VQA and end-to-end driving.

  • Results

    DriveLM-Agent performs competitively with state-of-the-art driving-specific models and improves zero-shot generalization to unseen objects or sensor configurations.

  • Takeaways & Limitations

    Graph VQA provides a framework for structured driving-scene reasoning, while DriveLM offers a benchmark for evaluating this capability and its connection to driving.

  • Takeaways & Limitations

    DriveLM-Agent has long inference times, running roughly 10× slower than UniAD because graph reasoning requires multiple prediction rounds.

Abstract

from arXiv · show

We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems to boost generalization and enable interactivity with human users. While recent approaches adapt VLMs to driving via single-round visual question answering (VQA), human drivers reason about decisions in multiple steps. Starting from the localization of key objects, humans estimate object interactions before taking actions. The key insight is that with our proposed task, Graph VQA, where we model graph-structured reasoning through perception, prediction and planning question-answer pairs, we obtain a suitable proxy task to mimic the human reasoning process. We instantiate datasets (DriveLM-Data) built upon nuScenes and CARLA, and propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving. The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides a challenging benchmark for this task. Our DriveLM-Agent baseline performs end-to-end autonomous driving competitively in comparison to state-of-the-art driving-specific architectures. Notably, its benefits are pronounced when it is evaluated zero-shot on unseen objects or sensor configurations. We hope this work can be the starting point to shed new light on how to apply VLMs for autonomous driving. To facilitate future research, all code, data, and models are available to the public.

1 Introduction

The paper argues that existing autonomous-driving systems lack generalization and human-friendly interaction, while existing VQA paradigms do not capture humans’ multi-object, multi-step reasoning. It proposes Graph VQA, DriveLM datasets, and DriveLM-Agent to address this gap.

  • Motivation: Current autonomous-driving stacks lack crucial capabilities, especially generalization to unseen scenarios or sensor configurations and human-friendly interaction.Humans instead perform object-centric perception, prediction, and planning before selecting a driving action.
  • Motivation: VLMs offer world knowledge, language-based interaction, and multi-step logical reasoning that can support autonomous driving.Their end-to-end differentiability is also desirable for autonomous-driving systems.
  • Research gap: Scene-level and single-object VQA do not mimic human reasoning because humans consider multiple objects across multiple reasoning steps.The paper therefore introduces a new task, datasets, and baseline architecture.
  • Contributions: Graph VQA formulates perception, prediction, and planning as question-answer pairs in a directed graph with logical dependencies guiding answers.It also includes behavior and motion-planning questions with dedicated metrics.
  • Contributions: DriveLM-nuScenes and DriveLM-CARLA provide graph-structured annotations linking images to driving behavior, with more text annotations per frame and challenging zero-shot test data.The CARLA dataset includes the PDM-Lite expert algorithm and uses a fully rule-based labeling pipeline.
  • Contributions: DriveLM-Agent combines a general VLM, trajectory tokenization, and graph prompting to repurpose VLMs for end-to-end autonomous driving.It performs competitively with driving-specific models, generalizes from nuScenes to unseen Waymo deployment, and benefits most from prediction and planning QAs.

2 DriveLM: Task, Data, Metrics

DriveLM defines Graph VQA as graph-structured reasoning over perception, prediction, planning, behavior, and motion, and instantiates it through datasets, expert-generated data, and dedicated metrics.

  • GVQA formulation: GVQA organizes each frame’s question-answer pairs into a directed acyclic graph whose vertices represent object-linked QAs and whose edges encode logical dependencies.A current question may use context from multiple parent or grandparent nodes.
  • GVQA formulation: Object-level edges represent interactions between objects, while task-level edges connect the reasoning stages.The stages cover perception, prediction, planning, behavior, and motion.
  • Representations: Motion represents the ego vehicle’s future BEV trajectory as N waypoint coordinates, with successive position offsets computed at fixed time intervals.Behavior provides an interface from perception, prediction, and planning to motion through discretized speed and steering categories.
  • Dataset construction: DriveLM-nuScenes uses selected video frames, key objects, ground-truth and manually annotated QAs, while DriveLM-CARLA generates graph-linked QAs from privileged expert information.The datasets differ because nuScenes and CARLA have substantially different collection settings.
  • Dataset construction: PDM-Lite is a lightweight CARLA expert using IDM-based target speeds, two proposals, and a simplified cost function for scalable QA generation.It obtains an improved DS of 44% on official CARLA validation routes.
  • Evaluation: DriveLM-Metrics evaluate motion with ADE, FDE, and collision rate, behavior with classification accuracy, and GVQA semantic accuracy using parsed language-model scores.Higher semantic scores indicate better semantic accuracy.

3 DriveLM-Agent: A GVQA Baseline

DriveLM-Agent uses a general VLM to translate images into driving trajectories through staged GVQA reasoning. It propagates graph-linked answers as context, aggregates them into behavior, and tokenizes trajectories for motion output.

  • Architecture: DriveLM-Agent uses BLIP-2 as a general VLM to translate an image into ego motion through perception, prediction, planning, behavior, and motion stages.The architecture is designed to exploit knowledge acquired during pre-training.
  • Architecture: Perception, prediction, and planning build scene understanding, behavior converts their information into a driving-action description, and motion produces an executable trajectory.This staged design mirrors multi-step human reasoning and supports explainability.
  • Graph prompting: For each graph edge, the model appends parent-node answers as context to the child question, concatenating information from multiple preceding nodes when needed.This passes forward information according to the graph’s logical dependencies.
  • Graph prompting: The inference graph can vary in size and structure according to the task or available compute budget, with specific subgraphs selected using heuristics.The model is trained on all available QAs but can infer on selected subgraphs.
  • Motion output: Trajectory tokenization converts waypoint coordinates into 256 bins so a general VLM can output fine-grained motion sequences.The bins are defined empirically from training-set trajectory statistics.

4 Experiments

The experiments evaluate VLM-based end-to-end driving, zero-shot transfer across sensor configurations, question-wise context effects, and GVQA performance. DriveLM-Agent is competitive for planning, benefits from graph context under Waymo transfer, and supports interpretable multi-stage reasoning, while GVQA remains challenging.

  • Experimental Setup: The experiments test VLM repurposing for end-to-end driving, unseen sensor configurations, question-wise context effects, and P1−3 GVQA performance.The evaluation covers DriveLM-nuScenes, Waymo, and both planning and graph-based question answering.
  • 4.1 VLMs for End-to-End Driving: Behavior context enables VLM motion planning on nuScenes to perform competitively with UniAD-Single, while BLIP-RT-2 without reasoning does not match it.DriveLM-Agent predicts behavior as an intermediate step before motion, surpassing UniAD-Single according to the reported results.
  • 4.2 Generalization Across Sensor Configurations: 54.29 speed accuracy with the full graph exceeds 43.90 with no context and 41.28 with the chain approach on zero-shot Waymo transfer.The models are trained on DriveLM-nuScenes and evaluated on Waymo without adaptation, despite the changed camera configuration.
  • Qualitative Results: DriveLM-Agent outputs generally provide intuitive answers on nuScenes and Waymo, although exceptions occur in nuScenes planning and Waymo perception.Waymo examples include meaningful prediction and planning answers despite imperfect perception, supporting interactive use of GVQA.
  • 4.3 Question-wise Analysis in DriveLM-nuScenes: Prediction and planning QA context improves behavior performance over perception-only context, while adding planning after prediction does not significantly improve it further.The authors suggest that other vehicles’ future status may already contain the information needed for behavior decisions.
  • 4.4 Performance for P1−3 via GVQA: DriveLM-Agent outperforms zero-shot BLIP-2 on both datasets, and both systems show potential gains from graph-based step-wise reasoning.The reported P1−3 evaluation compares no-context and ground-truth-context settings across DriveLM-nuScenes and DriveLM-CARLA.

5 Related Work

Related work addresses generalization in autonomous driving through data-driven and semantic approaches, while emerging language-grounded methods use LLMs or VLMs for planning, prediction, and explanations. DriveLM combines VLMs with graph-structured question-answer training to address this underexplored direction.

  • Generalization in Autonomous Driving: Prior autonomous-driving research targets long-tail generalization through additional data and semantic supervision for unseen or anomalous objects.Examples include simulation-based data collection for safety-critical cases and semantic supervision for detecting unfamiliar objects.
  • Language-grounded Driving: Language-grounded driving methods encode scene states or sensor data into language-model inputs for planning, control prediction, and explanations.GPT-Driver and LLM-Driver prompt LLMs with perceived scene states, while DriveGPT4 maps raw sensor data to tokens for end-to-end outputs.
  • Language-grounded Driving: DriveLM combines VLMs with graph-structured QA training to investigate generalization in autonomous driving through logically connected perception, prediction, and planning.The related-work discussion identifies this as an underexplored application of language models to driving generalization.

6 Discussion

DriveLM-Agent has practical limitations in efficiency, driving-specific inputs, and evaluation scope. The baseline is slower than UniAD, uses limited visual input, and is evaluated in an open-loop setting.

  • Efficiency Constraints: DriveLM-Agent has higher inference cost than UniAD-Single despite using fewer trainable parameters.Table 5 identifies the trade-off between parameter count and inference cost.
  • Efficiency Constraints: 10× slower inference than UniAD results from multiple graph-based prediction rounds and low throughput of 8.5 tokens/s.The authors suggest faster language models and inference optimizations may alleviate this constraint.
  • Driving-specific Inputs: The baseline takes a low-resolution front-view image as input, limiting temporal information and 360-degree scene understanding.Driving-specific sensors such as LiDAR are not processed as well, while multi-view and multimodal inputs remain future work.
  • Closed-loop Planning: The approach is evaluated under an open-loop scheme rather than closed-loop planning.The authors identify affordable closed-loop training and computation as a promising direction, with CARLA providing a foundation for further research.

Overview

The appendices expand the paper with dataset, metric, prompting, experimental, qualitative, and related-work details. They also organize supplementary material around motivating questions and empirical answers.

  • Supplementary Materials: The supplementary material includes additional discussions and details on annotation, GVQA metrics, context setting, trajectory tokenization, and ablations.These materials cover DriveLM-nuScenes and DriveLM-CARLA as well as further DriveLM-Agent results on nuScenes and Waymo.
  • Datasets and Annotation: Appendix sections document dataset composition, annotation pipelines, QA-category statistics, expert generation, graph construction, and data annotation.Separate descriptions cover DriveLM-nuScenes and the PDM-Lite expert and dataset-generation process.
  • Metrics and Agent Design: The appendices explain GVQA metrics and GPT-score, prompting with context, trajectory tokenization, and training–evaluation context differences.They also report tokenizer hyperparameters and trajectory sentence patterns.
  • Additional Experiments: Additional experiments examine unseen-object generalization, context effects on zero-shot ability, conventional VQA metrics, and model efficiency.The supplementary results extend the evaluation beyond the main experiments.
  • Qualitative Results: Qualitative analyses compare questions, answers, predictions, ground truth, SPICE, and GPT Score across nuScenes, Waymo, and CARLA.The examples provide intuition for the reported metrics and model outputs.
  • Related Work: Further related work covers graph-structure reasoning and vision-language benchmarks for autonomous driving.These perspectives complement the paper’s discussion of graph-structured reasoning and driving applications.

A Motivating Questions

The paper motivates planning with general VLMs by combining their web-scale knowledge and language interaction with graph-structured, human-like driving reasoning. It presents DriveLM as a benchmark and baseline framework while acknowledging runtime and question-design limitations.

  • Motivation: VLMs may improve driving generalization through internet-scale world knowledge and support human-friendly language interaction.General VLMs can be adapted to driving through fine-tuning on small autonomous-driving datasets.
  • Limitations: Runtime efficiency remains a limitation: the unoptimized approach is around 1 order of magnitude slower than UniAD, although optimization may enable practical use.Suggested directions include distillation, quantization, caching, and running only the final motion stage at high frame rate.
  • Limitations: The paper does not recommend a detailed question-collection protocol, identifying question design as dependent on domain expertise and future work.It frames the work as a pioneering study for driving with VLMs.

B.2 Collection Methodology

DriveLM-nuScenes is collected through a staged annotation workflow that moves from key-frame and key-object selection to frame-level perception, prediction, and planning QAs. Quality checks and task- and object-level analyses organize the resulting graph-structured benchmark.

  • Annotation workflow: Annotators select key frames, identify relevant key objects, and then label frame-level QAs in three consecutive steps.The process uses stitched six-camera nuScenes inputs and pre-annotated bounding boxes based on ground-truth categories.
  • Annotation workflow: Quality checks and post-processing divide annotations into batches of eight video clips and produce two DriveLM-nuScenes dataset versions.The workflow includes multiple rounds of quality control after annotation.
  • Statistics and facts: Task- and object-level distributions show rich QA categories and sufficient logical relationships to construct graph-structured questions.The benchmark connects the stages of human driving logic and analyzes questions associated with key objects and traffic elements.
  • Question design: Examples span scene description, object relations, future motion, safe and dangerous actions, object importance, and the ego vehicle’s target action.These templates link object-centered perception and prediction to planning decisions.

C.3 Collection Methodology

DriveLM-CARLA is generated in CARLA Leaderboard 2.0 using routes, expert trajectories, simulator sensors, and privileged state information. Hand-crafted templates convert this information into language labels for the dataset.

  • Simulator settings: CARLA Leaderboard 2.0 provides Town 12 for training and Town 13 for evaluation across diverse rural, residential, and urban environments.Each town covers 10 x 10 square kilometers, with 90 training routes and 20 evaluation routes.
  • Data collection: The expert is executed on routes to collect RGB, LiDAR, semantic, depth, and BEV semantic data, although DriveLM-Agent uses only RGB images.Additional modalities support auxiliary tasks for retraining TransFuser++.
  • Data collection: CARLA supplies privileged attributes for the ego vehicle, other vehicles, pedestrians, traffic controls, landmarks, and weather.These attributes include geometry, motion, lane information, traffic states, and distances relevant to driving decisions.
  • Language labels: Language labels are generated with hand-crafted sentence templates, without GPT-4 augmentation in the dataset version used here.GPT-based augmentation is described as an option for greater linguistic diversity and reduced template overfitting.

D.1 P1−3 VQA Metrics

DriveLM evaluates perception, prediction, and planning outputs with standard VQA metrics plus semantic and graph-aware measures. Behavior predictions are additionally assessed by discretizing trajectory changes into speed and steering categories.

  • P1−3 VQA metrics: P1−3 VQA evaluation uses BLEU, ROUGE_L, METEOR, CIDEr, and GPT score to compare generated answers with references.GPT score is introduced for more semantically comprehensive evaluation than lexical matching alone.
  • P1−3 VQA metrics: Completeness measures the proportion of frame-associated ground-truth questions whose predicted answers exceed a correctness threshold.The implementation uses SPICE to determine whether each QA is correctly answered.
  • Behavior metrics: Behavior prediction is evaluated with classification accuracy, including separate speed and steering components.Trajectory point differences are averaged and mapped into predefined speed and steering bins before comparison.

D.3 Motion Task Metrics

DriveLM evaluates motion prediction with displacement and collision metrics, and converts continuous trajectories into language-model-compatible discrete tokens.

  • Motion performance is measured with ADE, FDE, and collision rate over predicted trajectories.ADE averages L2 displacement errors across predicted horizons, while FDE measures final displacement error.
  • ADE, FDE, and collision-rate calculations follow UniAD’s setting rather than ST-P3’s, and DriveLM’s validation split is not directly comparable to the full nuScenes validation set.The DriveLM split contains keyframes with intention changes.
  • DriveLM-Agent uses trajectory tokenization to generate ego future trajectories directly with the language model.The process discretizes and tokenizes continuous trajectories following the RT-2 approach.
  • Each trajectory coordinate axis is partitioned into 256 discrete intervals to balance action granularity and token efficiency.The discretized coordinate values are mapped to numeric language-model tokens.

F.1 Implementation Details

The implementation evaluates graph-based VLM reasoning across metrics, contexts, models, sensor configurations, and unseen-object settings. Results emphasize context-dependent gains while documenting metric inconsistency and practical limitations.

  • F.2 Generalization to Unseen Objects: DriveLM-CARLA-ped evaluates zero-shot generalization to pedestrians absent from the training and validation splits.The experiment compares DriveLM-Agent with TransFuser++ and tests pedestrian-specific graph questions.
  • F.3 Results with More Metrics in VQA: Conventional VQA metrics show inconsistent improvement trends, motivating GPT Score as the main evaluation metric.The reported metrics include BLEU-4, METEOR, CIDEr and ROUGE-L.
  • F.5 Multi-frame Input: Multi-frame inputs provide a slight improvement for the evaluated LLaMA-Adapter V2 model, whose performance is similar to BLIP-2.The comparison is reported within nine training epochs.
  • F.3 Results with More Metrics in VQA: Improving P1−3 QA performance is non-trivial and has little effect on final behavior and motion performance under the current question setting.The authors identify question design as an area needing further exploration.
  • F.6 Computational Complexity: Inference efficiency remains a concern because DriveLM-Agent is compared computationally with UniAD-Single and could benefit from caching vision tokens and batching question patterns.These are proposed directions for fundamentally speeding up inference.

G.3 DriveLM-CARLA

DriveLM-Agent is qualitatively evaluated on CARLA and related driving scenarios for scene understanding, graph reasoning, and generalization. The examples show both successful prioritization of relevant obstacles and pedestrian-related failures.

  • Generalization to the unseen pedestrian scenario: DriveLM-Agent correctly recognizes a crossing pedestrian, infers that the vehicle should stop, and generates a complete stop in the successful CARLA example.Other examples show failures either translating detection into behavior or detecting the pedestrian at all.
  • Generalization across sensor configurations: Waymo qualitative examples are used to illustrate generalization to new sensor configurations after training on DriveLM-nuScenes.The Waymo questions are manually defined and lack annotated ground truth.
  • Graph Visual Question Answering: The CARLA VQA examples show DriveLM-Agent identifying important objects and reasoning about following a leading vehicle and adjusting speed.The examples cover both vehicle-following and stop-sign intersection situations.
  • Graph Visual Question Answering: In the stop-sign example, DriveLM-Agent identifies that the motorcycle ahead, rather than merely the stop sign, is the primary reason to stop.This demonstrates prioritization of a dynamic obstacle over a traffic-control device in that circumstance.
  • The paper frames graph reasoning as a structured approach connecting object interactions and task stages for explainable autonomous-driving decisions.The related-work discussion situates this within graph-based reasoning and driving VQA benchmarks.
Loading 2312.14150v3…