PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution

Chengkai Xu, Jiaqi Liu, Yicheng Guo, Peng Hang, Jian Sun

arXiv:2609.31814v1cs.AI

TL;DR

Existing VLM driving benchmarks often assess open-loop understanding or closed-loop performance without enough structure to explain capability relationships and diagnose models. DriveHierarchy organizes evaluation into four ranks and combines public-dataset question answering with interactive closed-loop scenarios. Its studies report meaningful relationships across capability levels and present the benchmark as a practical basis for diagnosis and targeted intervention.

  • Problem

    Existing open-loop benchmarks can entangle capabilities in task-level scores, while closed-loop outcomes rarely reveal which ability underlies observed behavior, limiting diagnosis and targeted improvement.

    Source: §1 · pp. 1–2

  • Method

    DriveHierarchy organizes VLM driving into four ranks and combines a public-dataset open-loop corpus with a closed-loop simulator and 100 interactive scenarios.

    Source: §1 · p. 2

  • Results

    The study reports meaningful relationships across capability levels and a useful basis for targeted analysis and intervention design.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The benchmark provides a practical framework for understanding VLM driving and studying how open-loop capabilities relate to closed-loop performance.

    Source: §5 · pp. 9–10

    R4 remains simulator-based and cannot be interpreted as a direct proxy for real-world deployment performance.

    Source: A Limitations, Broader impacts and… · p. 14

Abstract

from arXiv · show

Evaluating VLM-based autonomous driving remains difficult because driving competence is composite, where a capable system must ground traffic participants and hazards, integrate context across views and time, reason about future evolution, and act appropriately under closed-loop interaction. Existing benchmarks usually assess either open-loop understanding or closed-loop driving but provide limited structure for explaining how these abilities are organized, how they relate, and how they may inform model diagnosis and improvement. We present \textsc{DriveHierarchy}, a hierarchical benchmark that organizes VLM-based autonomous driving into four ranks, spanning perceptual grounding, contextual memory, mental reasoning, and closed-loop execution. To instantiate this hierarchy, we integrate multiple open-source autonomous-driving datasets into a unified open-loop benchmark with 76,798 question-answer pairs over 84,279 frames and develop a closed-loop simulation platform with interactive scenario construction on a real-world road network, from which 100 driving scenarios are curated for embodied evaluation. Experiments on 15 VLMs show that \textsc{DriveHierarchy} captures structured but non-redundant capability variation, relates open-loop understanding to closed-loop driving, and provides a practical basis for diagnosis and benchmark-guided optimization. \textsc{DriveHierarchy} therefore serves as a unified framework for evaluating and improving VLM-based autonomous driving systems. An anonymized project has been released on https://github.com/PerfectXu88/DriveHierarchy

1 Introduction

VLM driving competence spans perception, context, reasoning, and action, but existing open- and closed-loop evaluations offer limited diagnostic structure. DriveHierarchy organizes these abilities into four ranks and combines open-loop data with interactive closed-loop scenarios to study model capabilities and their relationships.

1 Introduction

  • Open-loop task scores entangle capabilities, while route-level closed-loop outcomes rarely identify which ability underlies behavior, limiting diagnosis and targeted improvement.The paper frames driving competence as grounding hazards, integrating context across views and time, reasoning about future evolution, and acting under interaction.Source: §1 · p. 2
  • DriveHierarchy organizes VLM driving into R1 Perceptual Grounding, R2 Contextual Memory, R3 Mental Reasoning, and R4 Closed-loop Execution.The hierarchy is intended to support fine-grained capability analysis alongside system-level assessment.Source: §1 · p. 2, Figure 1 · p. 2
  • The benchmark integrates public driving datasets into a corpus of 84,279 frames and 76,798 question-answer pairs, and curates 100 scenarios for closed-loop testing.The paper also reports a comprehensive study of 15 open-source VLMs and a targeted supervised fine-tuning case study.Source: §1 · p. 2
  • The study reports meaningful relationships across capability levels and presents the benchmark as a basis for targeted analysis and intervention design.Its contributions include a hierarchical benchmark and a joint simulation platform for interactive evaluation under diverse functional driving scenarios.Source: §1 · p. 3

2 Related Work

Prior work has developed VLM-based end-to-end driving and both open- and closed-loop benchmarks, but evaluation remains fragmented in objective and granularity. The paper situates DriveHierarchy within work on hierarchical capability decomposition to address the limited structure for connecting abilities to driving outcomes.

VLM-Based End-to-End Autonomous Driving

  • VLM-based driving has evolved from language-assisted agents toward end-to-end and Vision-Language-Action systems, where perception, memory, reasoning, and decisions are increasingly entangled.This trend creates evaluation challenges even as VLMs offer unified representation and interpretability.Source: §2 · p. 3

Autonomous Driving Benchmarks

  • Open-loop benchmarks often report task-specific scores, while closed-loop benchmarks commonly emphasize completion, collisions, or aggregate success, limiting capability diagnosis and links to driving outcomes.The paper characterizes both types as useful for comparison but less informative for explaining competence and weaknesses.Source: §2 · p. 3

Hierarchical Capability Decomposition for Embodied Intelligence

  • Hierarchical capability decomposition treats intelligence as organized across partially separable abilities, but autonomous-driving evaluations still offer limited structure linking lower-level abilities to higher-level behavior.The paper connects this approach to work moving multimodal evaluation beyond flat aggregate scores.Source: §2 · p. 3

3 DriveHierarchy

DriveHierarchy structures evaluation across three open-loop ranks and a closed-loop rank, using capability-oriented tasks and interactive scenarios. Its design pairs local diagnosis with analysis of how open-loop abilities relate to embodied driving behavior.

3.1 Overview and Design Principle

  • The design divides driving evaluation into R1–R3 open-loop tasks and R4 closed-loop execution, using 14,000 question-answer pairs over 31,945 frames and 100 R4 scenarios.The open-loop ranks probe scene understanding and cognitive processing; R4 evaluates whether those capabilities are associated with driving behavior under continuous interaction.Source: §3.1 · p. 4

3.2 Open-Loop Benchmark Construction

  • R1 assesses single-frame scene representation through recognition, distance perception, visual grounding, and situation description; R2 evaluates memory across views, time, and spatial relations; R3 tests future behavior prediction and temporal sequence planning.Each open-loop sample is a tuple of visual input, task-specific natural-language query, and ground-truth target; the visual input may be a single image, synchronized multi-view set, or short clip, and predictions receive bounded scores from 0 to 100.Source: §3.2 · pp. 4–5

3.3 Closed-loop Benchmark Construction

  • R4 uses CARLA-SUMO co-simulation on a real-world road network and an interactive scenario editor to curate 100 scenarios; its score jointly measures route completion, safety, and efficiency.The multiplicative scoring design prevents route progress alone from masking unsafe or inefficient behavior.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

4 Experiments

Across 15 VLMs, the experiments find differentiated open-loop profiles, associations between open-loop capability and closed-loop driving, and gains from benchmark-guided supervision on non-target abilities and R4.

4.1 Experimental Setup

  • The evaluation covers 15 general-purpose and driving-specialized VLMs across open-loop ranks R1–R3 and closed-loop R4, with SFT targets selected from Qwen3-VL-8B-Instruct deficits.The target groups are those with the largest gaps to task-wise state-of-the-art results.Source: §4.1 · p. 6

4.2 Open-Loop Benchmark Results

  • Qwen2.5-VL(72B) leads the open-loop weighted score at 54.47, ahead of Qwen3-VL(32B) at 49.64 and Gemma-3-it(27B) at 49.53.Leading models are competitive across all three open-loop ranks, while different models peak on different subtasks.Source: §4.2 · p. 6
  • Figure 4 profiles show Qwen-series scaling improving balance, while similarly sized models retain distinct strengths across distance, temporal reasoning, spatial relations, and R3_2.Gemma-3-it(12B) is described as the most balanced and as leading on R3_2; Qwen3-VL(8B) offers one of the best overall trade-offs at that scale.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.4 From Open-Loop Capability to Closed-loop Driving

  • R1, R2, and R3 correlate with R4 at 0.664, 0.596, and 0.418, respectively, associating open-loop ability with—but not determining—closed-loop performance.The paper describes lower- and mid-level abilities as more predictive of closed-loop success than the current R3 tasks.Source: Table 2 · p. 7, §4.4 · p. 8

4.3 Capability Correlation and Weight Allocation

  • Subtask correlations range from ρ = 0.961 for nearest- and referred-object distance to near-independence, and fixed weights favor less redundant subtasks within rank budgets of 3:3:4.Rank correlations also show R1–R2 association (ρ = 0.843) exceeding the links between R1/R2 and R3 (ρ = 0.346 and ρ = 0.425).Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.5 Benchmark-Guided SFT

  • After R1_2_B supervision, 10 of 13 non-target abilities increase, while the largest non-target decrease is −2.09, indicating structured cross-ability transfer.Other interventions also show positive spillovers without a rank-wide collapse.Source: Table 3 · p. 9, §4.5 · p. 9
  • Joint optimization of benchmark-identified weak capabilities raises the closed-loop score from 0.902 to 8.021 (+7.119), the largest reported R4 gain.R4 was not a supervision target; the reported change transfers from open-loop capability supervision.Source: Table 4 · p. 9, §4.5 · p. 9

5 Conclusion

DriveHierarchy shifts VLM driving evaluation from flat model comparison toward capability-structured diagnosis, relating open-loop competence to downstream behavior. Its targeted SFT case study illustrates how these diagnoses may inform model intervention.

5 Conclusion

  • DriveHierarchy makes driving evaluation capability-structured rather than a flat model comparison.The conclusion frames the benchmark around making the internal organization of driving ability explicit.Source: §5 · pp. 9–10
  • The benchmark relates open-loop competence to downstream driving behavior.The paper presents this relationship as a central value of the benchmark.Source: §5 · p. 10
  • A targeted SFT case study illustrates how capability diagnoses may inform model intervention.The conclusion presents this as an illustration, not a universal characterization.Source: §5 · p. 10

A Limitations, Broader impacts and Future work

The authors bound DriveHierarchy's claims to simulation and its current task coverage, scoring choices, evaluation runs, and single-model optimization study. They frame it as a diagnostic research instrument and identify infrastructure, methodology, and generality as areas for further work.

Limitations

  • R4 is simulator-based, so it is not a direct proxy for real-world deployment performance.The current hierarchy also leaves some higher-level reasoning capabilities under-covered; its composite score depends on a modeling choice, and deterministic single-run scores warrant caution for minor model differences.Source: A Limitations, Broader impacts and… · p. 14

Broader Impacts

  • DriveHierarchy is intended for capability analysis and controlled comparison, not as a certification standard for safety-critical deployment.Strong benchmark performance does not imply real-world deployment readiness; benchmark optimization may also overfit represented task distributions, annotations, or scenario designs.Source: A Limitations, Broader impacts and… · p. 14

Future Work

  • Future work includes maturing R4 infrastructure, strengthening virtual-real validation, and testing methods across more models and training recipes.The authors also plan robustness analysis, alternative composite scores, and uncertainty estimates over repeated scenario executions.Source: A Limitations, Broader impacts and… · p. 14

B Benchmark Construction Details

The benchmark combines an open-loop corpus with a refined evaluation set and a closed-loop co-simulation platform. Supporting materials enable auditing and reproduction while respecting upstream licensing constraints.

Benchmark Construction Details

  • The open-loop corpus contains 76,798 question-answer pairs over 84,279 frames, while the refined set contains 14,000 pairs over 31,945 frames.The corpus spans recognition, grounding, memory, reasoning, and planning; the closed-loop component uses CARLA-SUMO co-simulation and 100 curated driving scenarios.Source: B Benchmark Construction Details · p. 14
  • The release separates redistributable assets from upstream resources governed by their original licenses.It includes metadata, permitted annotations, prompts, scoring code, scenario configurations, reproduction scripts, and component-level logs; restricted raw data is represented by samples and reconstruction scripts.Source: B Benchmark Construction Details · pp. 14–15
  • The refined-set manifest stores source identifiers and answer labels to support audits of source proportions and answer distributions by task.The benchmark comparison table defines “Hier.” as explicit capability hierarchy and “Diag.” as capability-level diagnostic outputs.Source: Table 6 · p. 15, Table 5 · p. 15
  • Re-running the full benchmark requires obtaining the listed upstream datasets and model checkpoints.The package supplies reconstruction and scoring code, scenario configurations, pinned software environments, and commands for rebuilding manifests and replaying scenarios.Source: B Benchmark Construction Details · p. 15

C Experiment details and reproducibility

The open-loop evaluation uses official open-source model releases and reports aliases for their full checkpoints. Its inference setup uses tensor parallelism on NVIDIA A800 GPUs.

C.1 Model Configurations

  • The evaluation's short model names are aliases for the full checkpoints listed in Table 7.Unless otherwise stated, models are loaded from their official open-source releases.Source: §C.1 · p. 15
  • Reported GPU counts correspond to the tensor-parallel inference configuration on NVIDIA A800 GPUs.Source: §C.1 · p. 15
  • Table 7 lists the exact checkpoints used in the open-loop evaluation.The main paper's short names refer to those full checkpoint names.Source: §C.1 · p. 15

C.2 Inference and Evaluation Details

Open-loop samples use task-specific visual inputs and prompts, with constrained response formats designed to make evaluation consistent across task types. A shared deterministic protocol supports reproducible, restartable evaluation.

C.2 Inference and Evaluation Details

  • Each sample pairs a fixed system prompt with a task-specific query and may use a single image, synchronized multi-view inputs, or a short temporal sequence.Output formats are task-specific, including yes/no, numeric, multiple-choice, bounding-box, and ordered-label responses.Source: §C.2 · pp. 15–16
  • Multi-image inputs follow a fixed order, and temporal-order inputs are explicitly indexed to standardize the visual-textual interface.The protocol also constrains output formats to support deterministic parsing across heterogeneous tasks.Source: §C.2 · p. 16
  • Open-loop inference uses a unified deterministic decoding protocol, with settings summarized in Table 8.The evaluation stores per-model results, resumes partial runs, and logs, excludes, and reruns items with missing inputs, malformed records, or failed generations.Source: §C.2 · p. 16, Table 8 · p. 16

C.3 Closed-Loop Evaluation Setting

R4 evaluates each model in a controlled, independent CARLA-SUMO rollout for every scenario. A shared model-to-control interface supplies visual and route context and turns model outputs into vehicle actions.

C.3 Closed-Loop Evaluation Setting

  • R4 runs each model as an online VLM service in a fresh CARLA-SUMO instance per scenario, holding the scenario loop, simulator configuration, observation interface, and termination rules fixed.Only model-specific serving configuration changes; scenarios run independently so simulator state and traffic history do not carry across runs.Source: §C.3 · pp. 16–17, Table 9 · p. 17

C.4 Model-to-Control Interface for R4

  • At each decision step, the interface supplies a CARLA front-view image, bird’s-eye route image, and structured driving state, including speed, route, destination distance, navigation command, and future route points.The default chain requests an action decision; an optional waypoint-prediction question supports waypoint-style intermediate outputs.Source: §C.4 · p. 17
  • The action prompt asks for steering and target speed while directing the model to consider route geometry, traffic, rules, and immediate collision or off-road risk.Responses are parsed into steering and target-speed values, clipped to allowed ranges, and passed through the actuation bridge; all models share the same inputs and interface.Source: §C.4 · pp. 17–18

C.5 Benchmark-Guided SFT Details

Benchmark-guided SFT uses a consistent LoRA training setup to study transfer beyond supervised task scope, including whether open-loop supervision transfers to R4. No R4 scenario is used for training or fine-tuning.

C.5 Benchmark-Guided SFT Details

  • SFT experiments use Qwen/Qwen3-VL-8B-Instruct with LoRA, a 5% validation split, frozen vision tower and aligner, and adapters on all linear layers.The same optimization recipe is used across runs, and analysis emphasizes changes on non-target open-loop question types and transfer to R4.Source: §C.5 · p. 18, Table 10 · p. 18
  • No R4 scenario is used for training or fine-tuning in the reported experiments.The evaluation covers transfer from open-loop supervision to closed-loop R4 driving.Source: §C.5 · p. 18
  • Closed-loop evaluation over 100 scenarios takes approximately 20 hours per model on average and 36 to 72 A800 GPU-hours.Open-loop evaluation requires approximately 24 hours per model on average and roughly 48 to 144 A800 GPU-hours.Source: §C.5 · p. 18

D Open-Loop Evaluation Metrics Designs

The evaluation defines task- and rank-level open-loop scores, a weighted headline score, and a composite closed-loop score. Correlation-aware weighting is calibrated once and frozen, while the appendices document example questions, R4 scenarios, and resource provenance.

D Open-Loop Evaluation Metrics Designs

  • The headline open-loop score is a fixed weighted sum over 14 task scores, alongside task-level and rank-level aggregates.The rank score is the arithmetic mean of its task scores; task weights are frozen calibration weights intended to reduce repeated reward for redundant subtasks.Source: D Open-Loop Evaluation Metrics Designs · pp. 19–20

E Closed-Loop Scoring Details

  • R4 combines route completion, safety, and efficiency in a multiplicative scene score so route progress alone cannot mask unsafe or inefficient behavior.The route term tracks normalized progress toward the destination, safety combines event penalties and lane-keeping quality, and efficiency measures excess distance or time.Source: E Closed-Loop Scoring Details · p. 20

F Weight Allocation Details

  • Correlation-aware weights are calibrated once after the first benchmark-testing round and then frozen, with fixed rank-level budgets distributed according to subtask uniqueness.The allocation reduces the effective weight shared by highly correlated subtasks and gives more allocation to less redundant subtasks within a rank.Source: F Weight Allocation Details · p. 21

G Examples for QAs in DriveHierarchy

H Description of scenarios in R4

  • The current R4 benchmark includes 10 categories of typical scenarios constructed with the scenario editor, with future scenarios planned to expand this foundation.Examples include pedestrian encounters, static obstacles, intersections, traffic disturbances, abrupt lead-vehicle deceleration, and roundabout navigation.Source: H Description of scenarios in R4 · pp. 29–31

I Public Resources Used

  • The benchmark uses public datasets, simulators, checkpoints, and frameworks under their respective licenses, and releases only assets permitted for redistribution.The authors also state that part of the closed-loop scoring implementation is adapted from Bench2Drive.Source: I Public Resources Used · p. 32, Table 11 · p. 32

Related papers

Loading 2609.31814v1…