PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveLM: Driving with Graph Visual Question Answering

Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, Hongyang Li

arXiv:2312.14150v3cs.CV

TL;DR

Existing driving VQA tasks do not capture the multi-step object-centric reasoning humans use. DriveLM introduces Graph VQA, datasets, and a VLM agent for joint reasoning and driving, achieving competitive open-loop performance and improved zero-shot generalization.

  • Problem

    Existing driving VQA paradigms do not provide a suitable proxy for humans’ multi-step object-centric perception, prediction, and planning reasoning.

    Source: §1 · pp. 2–3

  • Method

    The paper formulates Graph VQA as logically dependent question-answer pairs, builds DriveLM datasets, and trains DriveLM-Agent with graph prompting and trajectory tokenization.

    Source: §1 · p. 3, §2.1 · p. 4

  • Results

    DriveLM-Agent performs competitively for open-loop driving and improves zero-shot generalization, including handling Waymo data after training only on nuScenes.

    Source: §1 · p. 3, §4.1 · p. 10

  • Takeaways & Limitations

    Graph-structured reasoning offers a principled framework for driving-scene reasoning and supports VLM-based autonomous driving beyond task-specific stacks.

    Source: §1 · p. 3, §6 · p. 14

    DriveLM-Agent has long inference times, running roughly 10× slower than UniAD because graph reasoning requires multiple prediction rounds.

    Source: §6 · p. 14

Abstract

from arXiv · show

We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems to boost generalization and enable interactivity with human users. While recent approaches adapt VLMs to driving via single-round visual question answering (VQA), human drivers reason about decisions in multiple steps. Starting from the localization of key objects, humans estimate object interactions before taking actions. The key insight is that with our proposed task, Graph VQA, where we model graph-structured reasoning through perception, prediction and planning question-answer pairs, we obtain a suitable proxy task to mimic the human reasoning process. We instantiate datasets (DriveLM-Data) built upon nuScenes and CARLA, and propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving. The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides a challenging benchmark for this task. Our DriveLM-Agent baseline performs end-to-end autonomous driving competitively in comparison to state-of-the-art driving-specific architectures. Notably, its benefits are pronounced when it is evaluated zero-shot on unseen objects or sensor configurations. We hope this work can be the starting point to shed new light on how to apply VLMs for autonomous driving. To facilitate future research, all code, data, and models are available to the public.

1 Introduction

The paper introduces Graph VQA as a graph-structured proxy for human object-centric perception, prediction, and planning, addressing limitations in current autonomous-driving systems and existing VQA paradigms. It contributes DriveLM datasets and DriveLM-Agent, which achieve competitive open-loop planning and improve zero-shot generalization to unseen deployment settings.

Motivation

  • Current autonomous-driving stacks lack generalization and human interaction, while humans use object-centric perception, prediction, and planning rather than geometrically precise BEV representations.VLMs offer internet-scale world knowledge, language-based interaction, multi-step logical reasoning, and end-to-end differentiability, motivating their use for autonomous driving.Source: §1 · pp. 2–3

Task

  • Graph Visual Question Answering represents perception, prediction, and planning reasoning as directed-graph question-answer pairs with logical dependencies and dedicated behavior and motion-planning metrics.Unlike scene-level or single-object VQA, GVQA models reasoning across multiple objects and multiple linked steps.Source: §1 · p. 3

Datasets

  • DriveLM-nuScenes and DriveLM-CARLA provide graph-arranged annotated QAs linking images to driving behavior, substantially more text annotations per frame, and challenging zero-shot test data.DriveLM-CARLA also includes PDM-Lite, described as the first working rule-based expert algorithm for CARLA Leaderboard 2.0.Source: §1 · p. 3

Baseline

  • DriveLM-Agent combines a trajectory tokenizer applicable to general VLMs with graph prompting that supplies logical dependencies as VLM context for end-to-end autonomous driving.The approach is presented as a simple methodology for repurposing VLMs without requiring a driving-specific architecture.Source: §1 · p. 3

Experiments

  • DriveLM-Agent performs competitively with state-of-the-art driving-specific models in open-loop planning and handles unseen Waymo deployment after training only on nuScenes.The experiments also find that graph structure improves zero-shot generalization, while prediction- and planning-stage QA pairs most help the final driving decision.Source: §1 · p. 3

2 DriveLM: Task, Data, Metrics

DriveLM introduces Graph VQA (GVQA), a dependency-structured task that mirrors human driving reasoning from perception through prediction and planning. It instantiates GVQA in nuScenes and CARLA datasets and evaluates motion, behavior, and P1–3 predictions with dedicated metrics.

2 DriveLM: Task, Data, Metrics

  • GVQA is motivated as a proxy for human decision-making, which progresses from locating key objects to estimating interactions and selecting ego actions.The section also introduces an official evaluation server and public leaderboard to benchmark methods combining language models with autonomous driving.Source: §2 · p. 4

2.1 DriveLM-Task: GVQA

  • GVQA organizes perception, prediction, planning, behavior, and motion question-answer pairs into a directed acyclic graph with object-level and task-level dependencies.Each vertex represents a question-answer pair associated with key objects, while edges encode object interactions and reasoning-stage progression.Source: §2.1 · pp. 4–5

2.2 DriveLM-Data

  • DriveLM-Data provides comprehensive graph-structured driving annotations: nuScenes uses expert-designed, quality-checked manual labeling, while CARLA generates scalable rule-based QAs from simulator information.DriveLM-CARLA contains 1.6M QAs, and its PDM-Lite expert reaches a 44% Driving Score on official Leaderboard 2.0 validation routes.Source: §2.2 · pp. 6–7

2.3 DriveLM-Metrics

  • DriveLM-Metrics evaluate motion with ADE, FDE, and collision rate, behavior with overall, steering, and speed accuracy, and P1–3 with SPICE and GPT Score.GPT Score uses ChatGPT-3.5 to assess semantic alignment between predicted and ground-truth answers, complementing SPICE’s structural comparison.Source: §2.3 · pp. 7–8

3 DriveLM-Agent: A GVQA Baseline

DriveLM-Agent uses BLIP-2 to translate images into motion through perception, prediction, planning, behavior, and motion stages, while GVQA context connects reasoning steps. It uses behavior aggregation and trajectory tokenization to convert graph-derived information into executable trajectories.

3 DriveLM-Agent: A GVQA Baseline

  • DriveLM-Agent uses BLIP-2 to map images to ego motion through perception, prediction, planning, behavior, and motion stages.The behavior stage summarizes graph information in language before the motion stage generates an executable trajectory.Source: §3 · p. 8

3.1 Prompting with Context

  • Context prompts append parent-node question-answer pairs to child questions, forwarding GVQA dependencies through multi-step reasoning.Inference graph size and structure are adaptable; DriveLM-Agent trains on all available QAs but samples task-specific subgraphs at inference.Source: §3.1 · pp. 8–9
  • The behavior stage aggregates all perception, prediction, and planning QAs into a natural-language description of the intended vehicle movement.This reflective summary captures crucial graph information and is reported as empirically crucial for VLM-based driving.Source: §3 · p. 8, §3.1 · p. 9

3.2 Trajectory Tokenization for Motion

  • DriveLM-Agent tokenizes waypoint coordinates into 256 bins so a BLIP-2 motion model can generate fine-grained trajectories from images and behavior descriptions.The motion model uses redefined tokenizer vocabulary, independent LoRA weights, and training only on motion-stage QAs.Source: §3.2 · p. 9

4 Experiments

Experiments evaluate DriveLM-Agent for end-to-end driving, zero-shot sensor generalization, question-wise context effects, and GVQA performance across perception, prediction, and planning. Results show that intermediate behavior reasoning and graph-structured context improve VLM-based driving, especially under changed sensor configurations.

4 Experiments

  • The experiments jointly test VLM end-to-end driving, unseen sensor generalization, question-wise context effects, and GVQA perception, prediction, and planning performance.Additional VLM comparisons on DriveLM-nuScenes and unseen-object generalization on DriveLM-CARLA are deferred to the supplementary material.Source: §4 · p. 9

4.1 VLMs for End-to-End Driving

  • DriveLM-Agent surpasses UniAD-Single on nuScenes, while its graph context improves zero-shot Waymo performance after training on DriveLM-nuScenes.BLIP-RT-2 without context or VQA training cannot match UniAD-Single, whereas behavior prediction as an intermediate step significantly boosts performance; single-frame VLMs remain below video-based UniAD.Source: §4.1 · p. 10, Table 2 · p. 11

4.2 Generalization Across Sensor Configurations

  • On unseen Waymo sensor configurations, DriveLM-Agent improves speed accuracy from 43.90 without context to 54.29 with the full graph, while Chain reaches only 41.28.UniAD-Single drops below BLIP-RT-2, whereas DriveLM-Agent uses only front-view input without adaptation and provides further multistage improvements.Source: §4.2 · p. 11

4.3 Question-wise Analysis in DriveLM-nuScenes

  • Prediction and planning QA context improves behavior over perception-only context, while adding planning QA after prediction does not significantly improve performance.The representative questions cover object perception, motion and appearance, future states, moving-direction relevance, attention ordering, possible actions, collision risks, and safe actions.Source: §4.3 · p. 12

4.4 Performance for P1−3 via GVQA

  • DriveLM-Agent significantly outperforms zero-shot BLIP-2 on both datasets, and graph-based step-wise reasoning benefits both zero-shot and fine-tuned settings.DriveLM-nuScenes is more challenging than DriveLM-CARLA, likely because its human answers are more diverse than CARLA’s rule-based answers.Source: §4.4 · pp. 12–13, Table 4 · p. 13

5 Related Work

Prior autonomous-driving research addresses long-tail generalization mainly through data-driven and semantic-supervision methods, while emerging language-grounded approaches use multimodal inputs for planning and control. DriveLM instead combines VLMs with graph-structured QAs to target logical reasoning and demonstrate zero-shot end-to-end planning benefits.

Generalization in Autonomous Driving

  • Graph VQA offers a new approach to improving autonomous-driving generalization by learning logical reasoning, addressing the unsatisfactory zero-shot performance of existing systems.Prior work emphasizes data-driven methods such as TrafficSim and semantic supervision for unseen or anomalous objects [14] [63] [89].Source: §5 · p. 13

Language-grounded Driving

  • Concurrent language-grounded methods prompt LLMs with scene states or sensor tokens for planning, control prediction, and explanations [57] [11] [102].These approaches include GPT-Driver [57], LLM-Driver [11], and DriveGPT4 [102], but their potential for improving autonomous-driving generalization remains underexplored.Source: §5 · pp. 13–14
  • DriveLM combines VLMs with graph-structured QAs and demonstrates zero-shot end-to-end planning benefits not shown by concurrent language-grounded studies.The approach is presented as a way to address autonomous-driving generalization through structured reasoning over DriveLM training data.Source: §5 · p. 14

6 Discussion

DriveLM shows promising generalization, but its baseline has efficiency, sensor-modality, temporal-coverage, and evaluation-setting limitations. The work nevertheless contributes Graph VQA, datasets, metrics, and a simple baseline for end-to-end autonomous driving.

Conclusion

  • The work proposes Graph VQA, new datasets and metrics, and a simple baseline that uses VLMs as end-to-end autonomous driving agents with improved generalization.The discussion presents these contributions as producing promising results while acknowledging the work’s limitations.Source: §6 · p. 14

Efficiency Constraints

  • DriveLM-Agent is roughly 10× slower than UniAD and has an 8.5 tokens/s throughput despite using fewer trainable parameters than UniAD-Single.The authors note that faster LLMs may alleviate this limitation as the field progresses.Source: §6 · p. 14, Table 5 · p. 14

Driving-specific Inputs

  • DriveLM-Agent uses only low-resolution front-view images, limiting LiDAR compatibility, temporal information, and 360-degree scene understanding.The graph formulation could support multiple views and frames, while multimodal and multi-frame inputs remain future work.Source: §6 · p. 14

Closed-loop Planning

  • The approach is evaluated open-loop, so the reported benefit of incorporating ego-vehicle status may not translate reliably to real-world closed-loop driving.The authors identify affordable closed-loop training and computation as a future direction, with CARLA providing a foundation for further research.Source: §6 · p. 14

Overview … Appendix H – Other Related Work

The appendices expand the paper with motivating discussions, dataset annotation details, GVQA metrics, DriveLM-Agent implementation specifics, further experiments, qualitative results, and related work.

Overview

  • The appendices provide additional discussions, annotation details, metric explanations, agent design information, ablations, experiments, visualizations, and related work complementing the main text.They cover DriveLM-nuScenes and DriveLM-CARLA annotation, GVQA metrics, context and trajectory tokenization, DriveLM-Agent ablations, and supplementary results.Source: Overview · p. 21

Appendix A – Motivating Questions

  • Appendix A indexes motivating questions, including why general VLMs should be adapted to driving, and answers them intuitively and empirically.Source: Appendix A · p. 21

Appendix B – DriveLM-nuScenes

  • Appendix B describes DriveLM-nuScenes composition, its detailed annotation pipeline, and statistics across question-answer categories.Source: Appendix B · p. 21

Appendix C – DriveLM-CARLA

  • Appendix C details the PDM-Lite expert, DriveLM-CARLA dataset composition, graph structure, and data-generation and annotation processes.Source: Appendix C · p. 21

Appendix D – DriveLM-Metrics

  • Appendix D explains task-specific GVQA metrics, their differences, and the rationale for using GPT-score in P1−3 VQA tasks.Source: Appendix D · p. 21

Appendix E – DriveLM-Agent

  • Appendix E specifies context-based prompting and trajectory tokenization, including training–evaluation context differences, sentence patterns, and tokenizer hyperparameters.Source: Appendix E · p. 21

Appendix F – Experiments

  • Appendix F reports experiments on unseen-object generalization, context effects on zero-shot ability, conventional VQA metrics, and model efficiency.Source: Appendix F · p. 22

Appendix G – Qualitative Results; Appendix H – Other Related Work

  • Appendices G and H provide qualitative examples with metric comparisons and discuss related work on graph reasoning and vision-language benchmarks for autonomous driving.The qualitative examples span nuScenes, Waymo, and CARLA, while metric illustrations include SPICE and GPT Score on nuScenes.Source: Appendix G · p. 22, Appendix H · p. 22

A Motivating Questions · B DriveLM-nuScenes · B.1 Dataset Composition

The section motivates VLM-based driving for long-tail generalization while identifying runtime, evaluation, and reasoning limitations. It then defines DriveLM-nuScenes as a scene- and frame-level QA dataset organized around perception, prediction, and planning.

A Motivating Questions

  • VLM planning is expected to help most on novel or unseen driving situations that resemble knowledge acquired during web-scale pre-training.This targets the rare long-tail scenarios that are critical for safe autonomous driving and commercialization, while novel-object generalization remains an active research area.Source: A Motivating Questions · pp. 23–25
  • General VLMs combine billion-scale vision-language pre-training with adaptation on small driving datasets, whereas injecting external language data into driving-specific models is non-trivial.Combining general VLMs with driving-specific models is identified as an important future direction.Source: A Motivating Questions · p. 23
  • Open-loop evaluation uses key frames where ego history is weakly predictive, excludes ego history in some baselines, and introduces DriveLM-CARLA for future closed-loop evaluation.Running 4B-parameter models at CARLA’s required 20 FPS remains an engineering challenge, with distillation, quantization, caching, or faster final-motion execution proposed as solutions.Source: A Motivating Questions · p. 23
  • Without optimization, DriveLM-Agent is around 1 order of magnitude slower than UniAD, but proposed CARLA optimizations could make VLM driving practical.Runtime efficiency is acknowledged as a limitation, while faster LLM inference and BLIP-2 acceleration are cited as reasons it may improve; prior work such as is also noted.Source: A Motivating Questions · pp. 23–24
  • Graph reasoning does not yet yield major VQA improvements, potentially because of simple prompting, small base VLMs, or weak logical dependencies in the dataset.DriveLM-CARLA is proposed as a platform for studying these factors and improving future GVQA annotation.Source: A Motivating Questions · p. 24
  • DriveLM’s technical contribution is a standardized driving-language task, dataset, and evaluation framework rather than a new algorithmic method.The data and task are designed to adapt general VLMs to driving and can produce promising results with existing models.Source: A Motivating Questions · p. 25

B.1 Dataset Composition

  • DriveLM-nuScenes contains 4072 training frames and 799 validation frames with scene descriptions, frame-level QA, and 2D boxes across multi-view nuScenes images.Frame-level QA covers perception, prediction, and planning; perception uses nuScenes and OpenLane-V2 ground truth, while prediction and planning answers are manually annotated.Source: §B.1 · pp. 25–26

B DriveLM-nuScenes

  • Key QA objects use <c, CAM, x, y> tags for identity, camera, and box-center coordinates, supplemented by dictionaries containing size, category, motion state, and visual description.Figure 5 summarizes the dataset organization into scene descriptions and perception, prediction, and planning QA.Source: §B.1 · p. 26, Figure 5 · p. 26

B.2 Collection Methodology … C.2 Expert Algorithm: PDM-Lite

The paper builds DriveLM datasets through structured human annotation and scalable CARLA generation, organizing perception, prediction, and planning questions into graph-based reasoning tasks. It also defines PDM-Lite as a six-stage CARLA expert algorithm for forecasting agents, proposing safe speeds, simulating trajectories, and controlling the vehicle.

B.2 Collection Methodology

  • DriveLM-nuScenes uses experienced drivers to select informative frames, identify relevant objects, and annotate frame-level factual and open-ended questions, followed by rigorous quality checks and post-processing.The pipeline includes rule-based answers for factual questions, manual annotation for open-ended questions, batch-level re-annotation or correction, and translation from Chinese into English with manual checks.Source: §B.2 · pp. 26–27, Figure 6 · p. 27

B.3 Statistics and Facts

  • DriveLM-nuScenes organizes interconnected QA categories across perception, prediction, and planning, providing sufficient logical relationships for graph-structured reasoning.Its templates cover scene description, object localization and status, occlusion, action sequences, motion prediction, object interaction, attention, signal meaning, safety, prioritization, and goal action.Source: §B.3 · pp. 28–31, Figure 8 · p. 29
  • DriveLM-nuScenes further includes object-level QA distributions and free-form scene comments, extending the benchmark from structured categories to flexible driving-related reasoning.Examples include identifying safe or dangerous actions, estimating collision likelihood, selecting target actions, and commenting on complex scenes or pedestrians at intersections.Source: §B.3 · pp. 32–33, Figure 9 · p. 33

C DriveLM-CARLA

  • DriveLM-CARLA provides automatically generated frame-level QA pairs connected in an interconnected graph for road layouts, stop signs, traffic lights, and vehicles.The current composition can be extended to additional categories such as static objects, weather, and traffic signs.Source: C DriveLM-CARLA · p. 33, §C.1 · p. 34, Figure 10 · p. 35

C.1 Dataset Composition

  • CARLA generation enables scalable annotation without manual effort while supporting semantic segmentation, depth, LiDAR, and other sensor outputs for diverse architectures.Questions are designed for situational reasoning and categorized as perception, prediction, or planning; object IDs persist over time, and parent-child QA relations support temporal reasoning and graph traversal.Source: §C.1 · p. 34

C.2 Expert Algorithm: PDM-Lite

  • PDM-Lite targets all 38 scenarios in CARLA Leaderboard 2.0 through six stages beginning with A*-based dense path planning and 2-second, 20Hz forecasting of nearby agents.It augments routes with speed limits and traffic-control distances, shifts route segments for scenarios requiring adjacent lanes, and forecasts actors within 50m using a kinematic bicycle model and constant-action assumption.Source: §C.2 · p. 34
  • PDM-Lite uses IDM to derive a minimum target speed across relevant leading actors, simulates the ego trajectory in closed loop, rejects collision-prone proposals, and predicts controls with steering and throttle-brake controllers.The system applies IDM to vehicles, pedestrians, non-green traffic lights, and stop signs; it simulates 2s at 20Hz, scores bounding-box intersections, uses lateral PID steering, and applies linear regression for throttle and brake.Source: §C.2 · p. 35, Table 8 · p. 36
  • PDM-Lite is evaluated on 20 official Leaderboard 2.0 validation routes across three seeds, with paired routes differing only in weather to assess performance variance.Because the expert uses privileged information, the reported variance is attributed to random initialization of surrounding traffic rather than weather parameters.Source: §C.2 · p. 36

C.3 Collection Methodology · D DriveLM-Metrics

DriveLM-Data is collected in CARLA using distinct training and evaluation towns, diverse routes, objects, weather, and multimodal simulator data. DriveLM-Metrics evaluates VQA, behavior, and motion tasks through three metric groups.

C.3 Collection Methodology

  • CARLA 0.9.14 with Leaderboard 2.0 generates training data in Town 12 and evaluation data in Town 13 across varied rural, residential, and urban environments.The dataset uses 90 training routes spanning 780.6 kilometers and 20 evaluation routes spanning 247.6 kilometers, segmented into approximately 150-meter samples.Source: §C.3 · pp. 36, 38
  • PDM-Lite is a six-stage planner that solves all 38 scenarios of CARLA Leaderboard 2.0.This planner is detailed in Section C.2.Source: Figure 11 · p. 36
  • Route diversity is expanded with additional trucks, vans, bicycles, and motorcycles, randomized weather, and multiple scenarios, while night-time settings are excluded because low illumination harms automatic labeling.Background traffic initially contains only cars; night scenes are omitted due to insufficient visibility for object labeling.Source: §C.3 · p. 38
  • The collection pipeline executes an expert over each route and records RGB, LiDAR, semantic segmentation, depth, and bird’s-eye-view semantic segmentation data.DriveLM-Agent uses only RGB images, whereas TransFuser++ retraining requires additional modalities for auxiliary tasks.Source: §C.3 · p. 38
  • Simulator-derived privileged annotations describe ego and other vehicles, pedestrians, traffic controls, static objects, landmarks, and weather parameters.Annotations include geometry, motion, distances, lane information, object attributes, traffic-light and sign effects, and landmark text or values.Source: §C.3 · p. 38
  • Questions and answers are generated from simulator information with hand-crafted sentence templates, without language-model augmentation in the reported dataset version.The authors note that models such as GPT-4 could increase linguistic diversity and reduce template overfitting, but do not use that augmentation here.Source: §C.3 · p. 38

D DriveLM-Metrics

  • DriveLM-Metrics comprises three metric groups: P1−3 VQA metrics, behavior-task metrics, and motion-task metrics.The metric suite is introduced as the evaluation framework for these three task categories.Source: D DriveLM-Metrics · p. 39

D.1 P1−3 VQA Metrics · D.2 Behavior Task Metrics

The paper evaluates P1−3 VQA with conventional text-overlap and semantic metrics, augmented by GPT Score and graph-aware Completeness. Behavior prediction is measured by overall classification accuracy with separate speed and steering components derived from future-trajectory displacements.

D.1 P1−3 VQA Metrics

  • P1−3 VQA uses BLEU, ROUGE_L, METEOR, CIDEr, and SPICE to assess similarity, recall, alignment, semantic consistency, and scene-graph agreement.These metrics differ in their sensitivity to n-gram overlap, word order, synonymy, TF-IDF weighting, and generated-versus-reference scene graphs.Source: §D.1 · p. 39
  • GPT Score supplements traditional metrics by using ChatGPT to assign a 0–100 numerical rating of prediction quality based on semantic reasoning.The evaluation prompt is provided in Table 10, whose score is reported as similar to DriveGPT4 [102].Source: §D.1 · pp. 39–40, Table 10 · p. 40
  • Completeness measures the fraction of ground-truth questions correctly answered for a frame, counting a QA as correct when its SPICE score exceeds 0.5.The score is computed as correct predictions divided by total predictions.Source: §D.1 · p. 40

D.2 Behavior Task Metrics

  • Behavior prediction is evaluated with classification accuracy for the overall behavior category and its behavior speed and behavior steer components.The ground-truth future ego trajectory contains N bird’s-eye-view points represented by (x, y) coordinates at fixed time intervals.Source: §D.2 · p. 40
  • Trajectory displacement is computed independently for x and y at each interval by subtracting each point from its predecessor.These xdist and ydist sequences provide the quantities used to derive behavior categories.Source: §D.2 · p. 40
  • The mean xdist and ydist values are mapped to predefined speed and steering bins, whose pair forms the behavior category compared against DriveLM-Agent outputs.The resulting category is represented as (Bspeed, Bsteer), and related accuracy is calculated from the comparison.Source: §D.2 · p. 40

D.3 Motion Task Metrics … F Experiments

This block defines motion-stage evaluation, specifies DriveLM-Agent’s graph-context prompting and trajectory-token generation, and introduces the supplementary experimental analyses. It also clarifies metric conventions and dataset-split comparability.

D.3 Motion Task Metrics

  • ADE, FDE, and collision rate evaluate predicted trajectories using standard nuScenes and Waymo motion metrics.ADE averages L2 displacement errors at 1, 2, and 3 seconds; FDE measures endpoint error at 3 seconds, while collision rate averages collisions across those horizons.Source: §D.3 · p. 41
  • DriveLM motion metrics follow UniAD rather than ST-P3, and DriveLM-nuScenes validation results are not directly comparable with prior full-validation results.UniAD evaluates 3-second FDE and collision rate at that timestep, whereas ST-P3 averages measurements over six timestamps; DriveLM-nuScenes contains keyframes with intention changes.Source: §D.3 · p. 41

E DriveLM-Agent

  • DriveLM-Agent combines graph prompting with trajectory tokenization to generate ego-vehicle future trajectories through a language model.Its architecture includes an inverse mapping from real-world ego trajectories into the model’s token space.Source: E DriveLM-Agent · p. 41, Figure 12 · p. 42

E.1 Prompting with Context

  • During inference, DriveLM-Agent interactively predicts context across question stages in the order P1, P2, P3, B, and M.Training uses teacher-forced ground-truth parent answers as context, whereas inference feeds preceding predicted answers into succeeding questions.Source: §E.1 · p. 41

E.2 Trajectory Tokenization Details

  • Continuous ego trajectories are discretized and tokenized into action sequences using the RT-2 method.The model generates token sequences that are converted back into continuous coordinate trajectories.Source: §E.2 · p. 42
  • Each trajectory coordinate axis is divided into 256 intervals to balance trajectory detail with a manageable language-model vocabulary.The discretization is based on the future-trajectory distribution in nuScenes.Source: §E.2 · p. 42
  • The tokenization maps 256 numeric vocabulary tokens to coordinate bins and adds startof-trajectory and end-of-trajectory markers.Single-digit numeric tokens are excluded before selecting the 256 numeric token IDs.Source: §E.2 · p. 42

F Experiments

  • The supplementary experiments expand the main-paper evaluation with implementation details, additional VQA metrics, ablations, comparisons, and computational-complexity analysis.These analyses cover the experimental subsections associated with Section 4.Source: F Experiments · p. 42

F.1 Implementation Details … G.2 Waymo

The supplementary sections detail DriveLM-Agent’s training and evaluation procedures, then report generalization, metric, model, complexity, and qualitative analyses across driving datasets and sensor configurations. Together, they show strong zero-shot and qualitative generalization while identifying limitations in question answering, inference efficiency, and single-frame motion estimation.

F.1 Implementation Details

  • Training uses learning rate 0.0001, random seed 1234, default LoRA settings, BLIP-2 maximal sequence length 400, and teacher-forced ground-truth contexts for some training questions.At inference, highly imbalanced per-frame P1–3 QA counts, with variance over 260 on DriveLM-nuScenes, are balanced by computing GVQA scores on stage-specific QA subsets selected by designed patterns.Source: §F.1 · p. 43

F.2 Generalization to Unseen Objects

  • 20.92 →92.35: providing pedestrian-specific QA raises privileged DriveLM-Agent performance on DriveLM-CARLA-ped, while baseline accuracy drops from 59.63 to 4.59 and TransFuser++ from 70.19 to 8.72.Adding pedestrian QA improves performance to 27.04, whereas the privileged setting reaches 92.35; all models achieve 100% steering accuracy because pedestrians cross straight roads.Source: §F.2 · pp. 44–45, Table 11 · p. 44

F.3 Results with More Metrics in VQA

  • Different conventional VQA metrics show inconsistent improvement trends, motivating GPT Score as the primary metric for comprehensive evaluation.Table 12 reports BLEU-4, METEOR, CIDEr, and ROUGE-L alongside the GPT Score.Source: §F.3 · p. 45, Table 12 · p. 45

F.4 Stage-wise Ablation on Zero-shot Generalization across Sensor Configurations

  • Higher behavior-task accuracy improves motion-task performance, chiefly because additional behavior context raises speeding accuracy and thereby affects FDE.This stage-wise analysis evaluates zero-shot generalization across sensor configurations on Waymo frames.Source: §F.4 · p. 45, Table 14 · p. 46

F.5 More VLMs evaluated on DriveLM-nuScenes

  • Improving QA performance under the current question setting is non-trivial, and it has little effect on final task performance, including behavior and motion.Because DriveLM-nuScenes contains many question types, the analysis uses stage ablations and representative-question-wise analysis; the authors suggest question setup merits further exploration.Source: §F.5 · p. 46

F.6 Computational Complexity

  • DriveLM-Agent’s inference complexity is compared with UniAD-Single, with vision-token caching and batching question patterns proposed to fundamentally accelerate inference.The computational-complexity comparison appears in Table 5 of the main paper.Source: §F.6 · p. 46

G Qualitative Results / G.1 DriveLM-nuScenes

  • Qualitative examples show graph-based VQA reasoning across perception, prediction, planning, and behavior, but single-frame inputs make object movement status difficult to determine even for humans.The qualitative results cover DriveLM-nuScenes, Waymo sensor generalization, and unseen-object generalization; DriveLM-nuScenes examples compare predictions with ground truth using SPICE and GPT scores.Source: G Qualitative Results · p. 46, §G.1 · p. 47, Figure 13 · p. 47

G.2 Waymo / F.1 Implementation Details

  • Waymo inference uses DriveLM-nuScenes-trained models with rule-based questions generated from matched object answers, demonstrating qualitative generalization without Waymo annotations or ground truth.General-purpose perception questions seed the graph, and matched objects such as pedestrians, cars, and trucks generate subsequent prediction and planning questions.Source: §F.1 · p. 43, §G.2 · p. 48

G.3 DriveLM-CARLA · H More Related Work · I Broader Impact

DriveLM-Agent shows strong qualitative scene understanding and some generalization in CARLA and new sensor configurations, while still failing in some pedestrian cases. The paper situates Graph VQA among graph reasoning, embodied planning, and driving vision-language benchmarks, and discusses both its promise and ethical risks.

G.3 DriveLM-CARLA

  • In an unseen pedestrian scenario, DriveLM-Agent correctly recognizes the pedestrian and stops the vehicle in a successful example.Source: §G.3 · p. 48, Figure 16 · p. 50
  • DriveLM-Agent generalizes to new sensor configurations in Waymo examples, demonstrating strong qualitative generalization capability.Source: Figure 15 · p. 49
  • DriveLM-Agent’s pedestrian failures include detecting but mishandling pedestrians and completely overlooking them, preventing evasive or stopping maneuvers.The latter failure poses a significant real-world safety risk.Source: §G.3 · pp. 48–49, Figure 16 · p. 50
  • In CARLA Graph VQA, DriveLM-Agent identifies important objects, follows a leading vehicle, and reasons about speed adjustment despite occasional simulator color-label inaccuracies.Source: §G.3 · p. 49, Figure 17 · p. 51
  • At a stop-sign intersection, DriveLM-Agent identifies the scene and prioritizes a motorcycle ahead as the primary reason to stop over the traffic-control device.Source: §G.3 · p. 49, Figure 17 · p. 51

H More Related Work

  • DriveLM connects perception, prediction, and planning through graph-structured reasoning, drawing on prior graph reasoning and embodied-planning work with LLMs, [5], [83].Related methods include scene-graph and knowledge-graph reasoning, arbitrary graph modeling, and language-model-based robotic planning.Source: H More Related Work · p. 52
  • Existing driving vision-language benchmarks mainly describe scene-level or individual-object information, whereas DriveLM organizes object- and task-level annotations as a graph [20] [59] [67] [71] [98] [107].Source: H More Related Work · p. 52

I Broader Impact

  • The authors argue that VLMs could improve autonomous driving and human-system interactivity, while acknowledging risks from hallucinations and high resource use.They suggest greater interactivity may build confidence, hasten acceptance, and support safer transportation long term.Source: I Broader Impact · p. 52

Related papers

Loading 2312.14150v3…