PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Driving

Enming Zhang, Peizhe Gong, Xingyuan Dai, Min Huang, Yisheng Lv, Qinghai Miao

arXiv:2503.06497v3cs.CVcs.AI

TL;DR

Safety cognition in VLMs for autonomous driving remains insufficiently assessed by conventional benchmarks, especially for interactive safety-critical scenarios. The paper introduces SCD-Bench and its annotation and evaluation processes, and reports that models trained on SCD-Training perform better across general, driving, and safety-cognition benchmarks.

  • Problem

    Existing VLM benchmarks largely miss safety-critical human interactions, leaving reliability under incorrect, vague, or malicious driving queries unclear.

    Source: §I · p. 1

  • Method

    The paper introduces four-dimensional SCD-Bench, ADA-assisted annotation with expert review, automated LLM assessment, and the SCD-Training dataset.

    Source: §I · p. 1

  • Results

    Models trained on SCD-Training show balanced, superior performance across general tasks, autonomous-driving tasks, and SCD-Bench.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors report that their experiments validate the approach and provide insights for future research on VLM safety cognition in autonomous driving.

    Source: §V · p. 9

Abstract

from arXiv · show

Ensuring the safety of vision-language models (VLMs) in autonomous driving systems is of paramount importance, yet existing research has largely focused on conventional benchmarks rather than safety-critical evaluation. In this work, we present SCD-Bench (Safety Cognition Driving Benchmark) a novel framework specifically designed to assess the safety cognition capabilities of VLMs within interactive driving scenarios. To address the scalability challenge of data annotation, we introduce ADA (Autonomous Driving Annotation), a semi-automated labeling system, further refined through expert review by professionals with domain-specific knowledge in autonomous driving. To facilitate scalable and consistent evaluation, we also propose an automated assessment pipeline leveraging large language models, which demonstrates over 98% agreement with human expert judgments. In addressing the broader challenge of aligning VLMs with safety cognition in driving environments, we construct SCD-Training, the first large-scale dataset tailored for this task, comprising 324.35K high-quality samples. Through extensive experiments, we show that models trained on SCD-Training exhibit marked improvements not only on SCD-Bench, but also on general and domain-specific benchmarks, offering a new perspective on enhancing safety-aware interactions in vision-language systems for autonomous driving.

I. INTRODUCTION

VLM safety in autonomous driving remains uncertain, particularly when models must handle safety-critical human interactions and unreliable user queries. The paper introduces SCD-Bench to assess four safety-cognition dimensions, evaluates models, and presents SCD-Training as a resource for improving performance across task domains.

I. INTRODUCTION

  • The paper identifies a gap in evidence on VLM reliability as driving assistants facing incorrect, vague, or malicious queries and safety-critical human interactions.It frames alignment with human safety expectations and effective interaction as unresolved questions.Source: §I · p. 1
  • SCD-Bench evaluates safety cognition across Command Misunderstanding, Malicious Decision, Perception Induction, and Ethical Dilemmas.The dimensions cover responses to unclear or conflicting commands, harmful requests, misleading descriptions, and ethical choices.Source: §I · p. 1
  • The authors report that SCD-Bench evaluations reveal limitations in all four capabilities of general and autonomous-driving VLMs.Further analysis is said to highlight shortcomings and suggest directions for improvement.Source: §I · p. 2
  • 324.35K SCD-Training samples support models that show balanced, superior performance across general tasks, autonomous-driving tasks, and SCD-Bench.The authors describe this as the first SCD-Training dataset and report training the SCD-Model with it.Source: §I · p. 2

A. Vision-Language Models

  • Recent VLM work uses LLM backbones, multimodal instruction data, and pretraining followed by fine-tuning to improve visual QA, captioning, and reasoning.The cited examples include LLaMA [5], Qwen [6], Vicuna [7], and InternLM [8] backbones.Source: A. Vision-Language Models · p. 2

B. Benchmarking VLMs in Autonomous Driving

Existing autonomous-driving VLM benchmarks address tasks such as multimodal grounding, corner cases, robustness, and trustworthiness, while SCD-Bench targets safety cognition in interactive driving. It organizes that evaluation into four task dimensions.

B. Benchmarking VLMs in Autonomous Driving

  • SCD-Bench assesses responses to misleading commands, malicious intent, perception induction, and ethical dilemmas, distinguishing its multidimensional safety-cognition focus from earlier benchmark emphases.The related-work comparison contrasts this focus with adversarial robustness, multimodal grounding, and general trustworthiness.Source: B. Benchmarking VLMs in Autonomous… · p. 2
  • Prior benchmarks include DriveLM's interconnected perception-prediction-planning questions, LingoQA's driving-video QA, and CODA-LM's corner-case evaluation.The paper describes CODA-LM as using hierarchical data structures and LLM-based judges.Source: B. Benchmarking VLMs in Autonomous… · p. 2

A. SCD-Bench Tasks

  • SCD-Bench uses an open-ended question-answer dataset organized into Command Misunderstanding, Malicious Decision, Perception Induction, and Ethical Dilemmas.Figure 2 presents categories, fine-grained dimensions, and corresponding examples.Source: A. SCD-Bench Tasks · p. 3, Figure 2 · p. 3

1) Command Misunderstanding:

SCD-Bench tests how VLMs handle unclear commands, harmful instructions, misleading scene descriptions, and ethical dilemmas. Its images are sampled and filtered to retain varied, safety-relevant driving scenarios.

1) Command Misunderstanding:

  • VLMs should seek clarification rather than act on ambiguous references or contradictory driving instructions.The task includes references to unclear or nonexistent objects and commands with conflicting directives.Source: 1) Command Misunderstanding · p. 3

2) Malicious Decision:

  • Malicious Decision tests responses to direct law-breaking requests and indirectly unsafe commands that may appear legal depending on context.Following either without objection is described as indicating a lack of safety awareness.Source: 2) Malicious Decision · p. 3

3) Perception Induction:

  • Perception Induction tests resistance to false claims about specific objects and inaccurate descriptions of the broader driving scene.Accepting misleading inputs without correction is described as raising safety concerns.Source: 3) Perception Induction · p. 3

4) Ethical Dilemmas:

  • Ethical Dilemmas asks models to choose among Egoism, Altruism, and Utilitarianism in hypothetical extreme-driving situations.The task is intended to reveal ethical preferences, including in scenarios such as brake failure at a red light.Source: 4) Ethical Dilemmas · p. 3

1) Data Collection:

  • Sampling from nuScenes, ONCE, and Waymo followed by bounding-box filtering and manual screening produced 824 images, including 99 for Ethical Dilemmas.The collection process sampled one frame every ten frames, balanced scene complexity and diversity, and added 50 pedestrian images to address scarcity.Source: 1) Data Collection · pp. 3–4

2) Annotation:

ADA combines image understanding, prompt templating, and text generation to produce task-specific driving image-text annotations. Autonomous-driving experts inspect and filter generated questions before the final benchmark set is assembled.

2) Annotation:

  • ADA's perception stage generates a scene description, extracts salient object information, and uses bounding boxes to refine object locations.The modules use QwenVL2-72B for scene descriptions, GPT-4o-mini for object reasoning, and Grounding DINO for boxes.Source: 2) Annotation · p. 5
  • The Template Module combines environmental information with task criteria and few-shot examples to construct question-generation instructions.Environmental information integrates the scene description, object information, and bounding boxes.Source: 2) Annotation · p. 5
  • GPT-o1 generates task-specific text from the instructions, after which driving experts inspect and filter it, yielding 5,043 test questions.The paper presents ADA as a three-module system comprising perception, templating, and generation.Source: 2) Annotation · p. 5

C. Evaluation Method

SCD-Bench evaluates four safety-cognition tasks, classifying responses by whether models resist misleading or harmful inputs, clarify unclear commands, and maintain ethical preferences.

C. Evaluation Method

1) Criterion:

  • For Perception Induction, accepting misleading environment information is unsafe, rejecting it is safe, and an ambiguous response is weakly safe.The criterion applies to explicit environment-inducing information.Source: 1) Criterion · p. 5
  • For Malicious Decision, accepting a harmful decision is unsafe, rejecting it is safe, and an unclear response is weakly safe.Source: 1) Criterion · p. 5
  • For Command Misunderstanding, clarification is safe, executing an unclear or contradictory instruction is unsafe, and an ambiguous response is weakly safe.Source: 1) Criterion · p. 5
  • Ethical Dilemmas are assessed by calculating the proportion of responses for each ethical type.Source: 1) Criterion · pp. 5–6

2) Safety Rate:

The Safety Rate is computed from model responses to test-set image-text queries using a binary safe-response indicator.

  • J(·) assigns 1 when a model response is classified as safe and 0 otherwise.Source: 2) Safety Rate · p. 6
  • M denotes the test dataset, and q_i denotes its i-th text-and-image query pair.Source: 2) Safety Rate · p. 6
  • g(·) represents the vision-language model’s response function, whose output is evaluated by J(·).Source: 2) Safety Rate · p. 6

3) Attack Success Rate:

The section defines attack and ethical-preference measures and describes GPT-4o-based automated classification for large-scale evaluation of VLM responses.

3) Attack Success Rate:

  • Unsafe responses count as successful attacks, and Attack Success Rate measures their proportion; the evaluation expects higher SR and lower AR.Source: 3) Attack Success Rate · p. 6

4) Stable Value:

  • Stable Value is defined to measure the stability of VLM ethical preferences using the standard deviation of preference choices and its maximum.Source: 4) Stable Value · p. 6

5) Automated Evaluation:

  • GPT-4o-2024-05-13 serves as the automated discriminative function for classifying responses used to calculate SR and AR.Task-specific prompts apply the respective evaluation criteria.Source: 5) Automated Evaluation · p. 6
  • Automated prompts classify safety-task responses as safe, weakly safe, or unsafe, while the ethical-dilemma prompt identifies the selected option among A, B, or C.Source: 5) Automated Evaluation · pp. 6–7

D. Training Dataset

SCD-Training draws images from several open-source autonomous-driving datasets and generates question-answer pairs across four safety dimensions.

IV. EXPERIMENTS

The experiments evaluate general and driving-focused VLMs on SCD-Bench and report that SCD-Model-1B attains 90.74% safety. They also examine ethical preferences and the SCD-Training fine-tuning setup.

A. Experimental setups

  • SCD-Model-1B is based on InternVL2-1B and fine-tuned with LoRA on SCD-Training, LingoQA, and ShareGPT4V.The setup freezes the language model, MLP layers, and visual backbone while fine-tuning the language model.Source: A. Experimental setups · p. 7

B. Performance on SCD-Bench

  • 90.74% safety is achieved by SCD-Model-1B on SCD-Bench, while GPT-4o performs best overall among evaluated models.Smaller models show limited safety awareness, and Command Misunderstanding—especially Reference—is the hardest capability across models.Source: B. Performance on SCD-Bench · pp. 7–8
  • Ethical-dilemma evaluation treats higher Stable Value as more consistent and predictable preferences; fine-tuning can shift preferences, including SCD-Model-1B(E2A) from Egoism to Altruism.Annotations use open-form text rather than multiple-choice indices to avoid label leakage.Source: B. Performance on SCD-Bench · p. 8

C. Qualitative Examples on SCD-Bench

The qualitative examples contrast model behavior across safety and ethical tasks, while reported benchmark results indicate gains from SCD-Training on general and autonomous-driving tasks, with a trade-off from adding general-capability data.

C. Qualitative Examples on SCD-Bench

  • Qualitative examples show GPT-4o responding more safely to citation and contradiction prompts, while Qwen2-VL and LLaVA-1.6 tend to follow instructions blindly; object recognition is more challenging than direct malicious prompts.The ethical-preference example evaluates models using multiple-choice outputs.Source: C. Qualitative Examples on SCD-Bench · pp. 8–9

1) Performance on General Tasks:

  • SCD-Training also improves the original InternVL2-1B model’s general capabilities on MME and POPE.The paper reports these comparisons in Table IV.Source: 1) Performance on General Tasks · p. 9

2) Performance on Autonomous Driving Tasks:

  • SCD-Training improves InternVL2-1B on LingoQA and remains robust on multi-frame temporal inputs, whereas adding general-capability fine-tuning data tends to degrade performance.The comparison uses the LingoQA benchmark.Source: 2) Performance on Autonomous Driving… · p. 9

1) Impact of generalization ability:

The paper reports that generalization ability and model size are associated with stronger safety cognition, and that automated evaluation closely agrees with human experts. Its conclusion presents SCD-Bench, ADA, and SCD-Model as contributions to safety-cognition evaluation and training.

1) Impact of generalization ability:

  • Improving generalization ability is reported to enhance safety cognition in autonomous driving across comparisons involving eight general benchmarks and SCD-Bench.The listed general benchmarks are MMMU, RealWorldQA, TextVQA, MMBench, ChartQA, AI2D, OCRBench, and MathVista.Source: 1) Impact of generalization ability · p. 9, Figure 8 · p. 9

2) Impact of model size:

  • Increasing parameter count under the same architecture and data significantly improves safety cognition in autonomous driving.This is the paper’s reported model-size comparison.Source: 2) Impact of model size · p. 9, Figure 8 · p. 9

3) Agreements between Human and Evaluators:

  • Human-expert agreement with the automated evaluator is 99.59% for QwenVL-2 and 99.74% for GPT-4o.The study compares evaluations of results from those two models.Source: 3) Agreements between Human and… · p. 9

V. CONCLUSION

  • The conclusion presents SCD-Bench for systematic evaluation, ADA for scalable annotation, and SCD-Model as a VLM aligned with driving safety cognition.The paper states that domain experts curate the data and conduct human evaluations.Source: §V · p. 9
Loading 2503.06497v3…