PaperlayerView exact revision on arXiv

Source-linked AI summary

BD-HazardVLM: Probing Vision-Language Models for Latent Defensive-Driving Hazards in Bangladesh Road Scenes

Abdullah Al Maruf, Md. Sajedul Islam, Tanmoy Mridha, Irfan Hossain Bhuiyan

arXiv:2610.20627v1cs.CVcs.LGcs.RO

TL;DR

Defensive reasoning by vision-language models in weakly structured Global South traffic remains unclear, despite their growing use in driving-scene understanding. BD-HazardVLM benchmarks image-level defensive reasoning in Bangladesh scenes and evaluates three open models. Their fine-grained performance is weak, showing that coarse hazard recognition does not ensure reliable defensive reasoning.

  • Problem

    How reliably vision-language models reason defensively in weakly structured Global South traffic remains unclear, despite their growing use for driving-scene understanding.

    Source: §1 · p. 1

  • Method

    BD-HazardVLM is a 500-image Bangladesh-centric benchmark with structured hazard, timing, severity, action, evidence, and trap annotations, used to evaluate three models under a shared zero-shot task definition.

    Source: §1 · p. 2

  • Results

    Across three VLMs, fine-grained reasoning remains weak: category accuracy is below 40%, timing accuracy ranges from 1.6% to 12.4%, and exact action-set matching remains below 2%.

    Source: §8 · p. 14

  • Takeaways & Limitations

    For the three evaluated open VLMs, coarse hazard recognition does not ensure reliable fine-grained defensive reasoning.

    Source: §8 · p. 14

    BD-HazardVLM is not a validated vehicle-safety component.

    Source: §7 · p. 14

Abstract

from arXiv · show

Vision-language models are increasingly used for driving-scene understanding, yet their defensive reasoning under weakly structured Global South traffic remains unclear. We introduce BD-HazardVLM, a diagnostic benchmark of 500 Bangladesh road-scene images, comprising 375 from RSUD20K and 125 from TFP-BD, annotated for hazard presence, primary hazard category, risk timing, severity, defensive actions, visible evidence, and hallucination- and Western-road-bias traps. We evaluate Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6 using a shared zero-shot task definition and structured output schema. Binary hazard-presence accuracy ranges from 88.0 percent to 89.2 percent, essentially matching the 89.0 percent majority-class baseline. Fine-grained reasoning is substantially weaker: category accuracy remains below 40 percent, risk-timing accuracy ranges from 1.6 percent to 12.4 percent, and exact defensive-action matching remains below 2 percent. Although InternVL3-8B achieves the highest severity accuracy at 61.2 percent, it has zero recall on the 58 high-severity cases. Qwen exhibits severe category mode collapse, while all models show weak latent-hazard anticipation and near-zero self-reported trap recognition. For the evaluated models, these findings show that coarse hazard recognition does not imply reliable fine-grained defensive-driving reasoning and motivate geographically diverse safety evaluation.

1 Introduction

Defensive driving requires more than describing visible road users: drivers must identify hazards, judge their timing and severity, and choose preventive actions. BD-HazardVLM evaluates these capabilities in Bangladesh mixed traffic, where latent risks and weakly structured roads remain underexplored.

1 Introduction

  • BD-HazardVLM evaluates defensive-driving reasoning in Bangladesh scenes, where hazards can require preventive action before an immediate collision threat becomes explicit.Examples include pedestrians obscured by vehicles, buses blocking the view, unstable mixed-traffic interactions, and roadside crowds.Source: §1 · p. 2
  • Bangladesh scenes include heterogeneous traffic, informal crossings, roadside markets, occlusion, damaged surfaces, and limited lane organization, while local datasets lack direct labels for defensive reasoning.Geographic mismatch can also prompt unsupported assumptions about infrastructure or organized road behavior.Source: §1 · p. 2
  • Category accuracy remains below 40%, risk-timing accuracy ranges from 1.6% to 12.4%, and exact defensive-action matching remains below 2%.Binary hazard-presence accuracy is also misleading: the 89.0% majority-class baseline essentially matches the evaluated models; InternVL3-8B misses all 58 high-severity cases despite its highest aggregate severity accuracy.Source: §1 · p. 2
  • The benchmark uses a structured taxonomy of twelve locally relevant hazard categories, temporal risk, severity, defensive actions, visible evidence, and hallucination- and Western-road-bias traps.The contributions also describe a controlled zero-shot evaluation using the same images, task definition, and output schema.Source: §1 · p. 3

2 Related Work

Prior driving-VLM work spans scene understanding, planning, control, and reliability evaluation, while South Asian datasets provide geographically relevant imagery. BD-HazardVLM complements these efforts with image-level defensive reasoning and explicit hazard, timing, severity, action, and trap labels.

2.1 Vision-Language Models for Autonomous Driving

  • BD-HazardVLM isolates image-level defensive reasoning rather than full-stack driving, trajectory prediction, question answering, or closed-loop control.It tests identification of the primary hazard, latent versus immediate risk, severity, and a defensive response in Bangladesh mixed traffic.Source: §2.1 · p. 3

2.2 Hazard, Corner-Case, and Reliability Evaluation

  • Corner-case and reliability benchmarks examine perception, driving suggestions, robustness, and trustfulness, while BD-HazardVLM targets locally relevant hazards before they become immediate.Its labels separate hazard identity, temporal risk, severity, and defensive action.Source: §2.2 · pp. 3–4

2.3 South Asian and Bangladesh Road-Scene Datasets

  • RSUD20K and TFP-BD provide geographically relevant Bangladesh imagery, but their original annotations target object detection, traffic flow, or pedestrian analysis rather than defensive-driving reasoning.BD-HazardVLM adds image-level hazard categories, timing, severity, recommended actions, and scene-level traps.Source: §2.3 · p. 4

2.4 Hallucination and Geographic Bias in VLMs

  • Hallucination and geographic-bias benchmarks motivate trap labels for unsupported assumptions about infrastructure, but BD-HazardVLM measures self-reported scene susceptibility, not verified hallucinations in full responses.Direct hallucination measurement would require separate evidence-based human adjudication.Source: §2.4 · pp. 4–5

3 The BD-HazardVLM Benchmark

BD-HazardVLM is an image-level diagnostic benchmark built from Bangladesh road scenes, selected for scenario diversity rather than to estimate natural hazard prevalence. Its annotations capture scene-level hazards and defensive assessments, with centralized review and an explicit limitation on subjective labels.

3.1 Benchmark Scope and Image Sources

  • The 500-image benchmark combines 375 RSUD20K images and 125 TFP-BD images for scene-level defensive-hazard reasoning, not detection, segmentation, or trajectory prediction.The source datasets contribute complementary road-level and dense mixed-traffic views.Source: §3.1 · p. 5

3.2 Scenario-Driven Image Selection

  • Images were selected to maximize scenario diversity across difficult mixed-traffic hazards, so the sample is not a representative estimate of hazard frequency across Bangladesh roads.Selection considered road type, traffic density, weather, time, visibility, vehicle composition, pedestrian activity, and competing hazards.Source: §3.2 · pp. 5–6

3.3 Annotation Taxonomy and Schema

  • Annotations encode hazard presence, primary category, immediate or latent risk timing, severity, defensive actions, visible evidence, and hallucination- and Western-road-bias traps.The taxonomy has twelve mutually exclusive categories, and recommended actions are evaluated as unordered sets; timing and severity are image-level assessments, not physical collision estimates.Source: §3.3 · p. 6

3.4 Annotation and Quality Control

  • Four annotators labeled disjoint 125-image subsets, and two reviewers checked all 500 annotations for consistency, taxonomy compliance, evidence quality, and formatting.Because images were not initially annotated independently by multiple annotators, no formal agreement statistic is reported; subjective labels should be read relative to adjudicated annotations, not unique human-consensus ground truth.Source: §3.4 · p. 7

3.5 Dataset Characteristics

  • 445 images contain a major hazard, while risk timing comprises 404 latent, 41 immediate, and 55 none cases; none are labeled both.This intentional class imbalance prioritizes fine-grained category, timing, severity, and action evaluation over raw hazard-presence accuracy.Source: §3.5 · p. 7

3.6 Release and Licensing

  • The planned release includes manifests, annotations, guidelines, prompts, predictions, and evaluation utilities, while users must obtain source images from their original repositories.Annotations and model outputs are intended for CC BY-NC 4.0 release, and evaluation and preparation code for MIT licensing.Source: §3.6 · p. 8

4 Evaluation Protocol

Three open vision-language models are evaluated zero-shot on the same 500 images with a shared semantic task and structured output schema. The protocol scores exact and partial task performance and uses paired tests with Holm adjustment for model comparisons.

4.1 Models and Inference Setup

  • Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6 process the same 500 images zero-shot, without task-specific training or fine-tuning, using greedy decoding.Native processors, visual representations, and numerical precision differ across models because of GPU-memory constraints, so small performance differences warrant care.Source: §4.1 · p. 8

4.2 Prompt and Output Schema

  • All models receive the same Bangladesh mixed-traffic context, taxonomy, action vocabulary, and output schema, and are instructed to rely only on visible evidence rather than assume unsupported Western-road infrastructure.Responses contain eight structured fields; MiniCPM-V-4.6 can receive one formatting-only retry, while evidence text is retained for qualitative audit rather than scored quantitatively.Source: §4.2 · pp. 8–9

4.3 Evaluation Metrics

  • Hazard presence is scored by accuracy against the majority-class baseline, while category, timing, and severity use exact-match accuracy and actions use exact set match, overlap, and Jaccard similarity.Trap recognition is evaluated against human annotations using positive-flag rate and trap recall, which measure self-reported susceptibility rather than actual hallucination frequency.Source: §4.3 · p. 9

4.4 Statistical Analysis

  • Pairwise correctness differences for category, timing, and severity use exact McNemar tests, action Jaccard scores use paired Wilcoxon tests, and Holm adjustment is applied within each task.Corrected p-values below 0.05 are considered statistically significant.Source: §4.4 · p. 9

5 Results

No model leads every task, and binary hazard-presence accuracy is essentially matched by the majority-class baseline. Fine-grained category, timing, severity, action, and trap-recognition results reveal substantial weaknesses, including failures on high-severity cases.

5.1 Overall Performance

  • 88.0%–89.2% hazard-presence accuracy is essentially matched by the 89.0% majority-class baseline, while category, timing, severity, and action tasks distinguish performance more meaningfully.No model dominates every task: Qwen leads category accuracy, InternVL leads timing and severity accuracy, and MiniCPM leads defensive-action Jaccard.Source: §5.1 · pp. 9–10, Table 4 · p. 10

5.2 Hazard-Category Recognition

  • 38.2% category accuracy for Qwen2.5-VL-7B is highest, while category macro-F1 ranges from 9.4% to 10.5% across models.Qwen assigns the rickshaw/easybike category to 426 images (85.2%) despite 205 ground-truth cases (41.0%), reflecting substantial category mode collapse.Source: §5.2 · p. 10

5.3 Risk-Timing and Severity Estimation

  • 12.4% timing accuracy for InternVL3-8B is highest, while MiniCPM-V-4.6 scores 1.6%; InternVL also has 61.2% severity accuracy but 0.0% recall on 58 high-severity cases.MiniCPM predicts no latent or none timing cases and is correct on only eight images; an audit found its low score was not caused by parsing mismatch. High-severity recall is 5.2% for Qwen and 22.4% for MiniCPM.Source: §5.3 · pp. 10–11

5.4 Defensive-Action Recommendation

  • 1.6%, 0.4%, and 1.0% exact action-set matching for Qwen, InternVL, and MiniCPM contrasts with respective mean Jaccard scores of 22.0%, 25.8%, and 29.9%.The models often recommend broadly safe actions but struggle with context-specific responses such as monitoring occlusion, avoiding overtaking, or yielding.Source: §5.4 · p. 11

5.5 Hallucination- and Western-Road-Bias Trap Recognition

  • 64.8% of images are annotated hallucination-prone and 57.0% Western-road-bias-prone, yet trap recall is near zero across all models.These measures reflect self-reported trap recognition, not actual hallucination frequency; determining actual unsupported claims requires separate human adjudication of raw responses.Source: §5.5 · pp. 11–12

6 Qualitative Error Analysis

The examples show that isolated object or category recognition does not ensure coherent defensive reasoning across risk timing, severity, and actions.

6 Qualitative Error Analysis

  • Across four cases, models trigger hazards in a control scene, convert latent pedestrian risk to immediate or combined risk, understate pedestrian severity, or misjudge timing and severity despite a correct occlusion category.The cases illustrate that correct object or category recognition does not guarantee coherent reasoning across category, timing, severity, and action fields.Source: §6 · p. 12

7 Limitations and Ethical Considerations

BD-HazardVLM is a focused, image-level diagnostic benchmark, with annotation and model-comparison choices that constrain how its results should be interpreted. Its outputs are intended for research, not safety-critical deployment.

Scope and annotation

  • The 500-image benchmark is not representative of all Bangladesh roads, and single frames cannot assess motion, trajectories, or time-to-collision.Risk-timing and severity labels reflect visible image-level evidence; unobserved ego speed, acceleration, and object motion could change the dynamic interpretation.Source: §7 · p. 12
  • Without independent duplicate annotation, the benchmark reports no formal inter-annotator agreement or independent human-agreement ceiling for subjective attributes.Each reviewer reviewed their own 125-image subset.Source: §7 · pp. 12–13

Taxonomy and model comparability

  • The taxonomy assigns one primary category to scenes with interacting risks, while model-specific processors and precision settings make small performance differences uncertain.Results characterize three evaluated open models under disclosed inference configurations, not larger or proprietary VLMs.Source: §7 · p. 13

Hallucination interpretation

  • Trap outputs are model self-reports of recognizing possible unsupported reasoning, not measurements of actual hallucination rates in complete responses.Direct measurement would require separate evidence-based human adjudication.Source: §7 · p. 13

Release and responsible use

  • The original images will not be redistributed; users must obtain them from their source repositories and follow their licenses, and the benchmark is not a validated vehicle-safety component.Annotations are intended for multimodal-reasoning and driving-safety research, not identification, surveillance, or safety-critical deployment.Source: §7 · pp. 13–14

8 Conclusion

BD-HazardVLM evaluates image-level defensive-driving reasoning in Bangladesh scenes across three vision-language models. Fine-grained performance is weak, and the results show that coarse hazard recognition does not ensure reliable defensive reasoning.

8 Conclusion

  • The benchmark evaluates image-level defensive-driving reasoning across hazard category, risk timing, severity, defensive actions, and reasoning traps.It is a 500-image Bangladesh-centric diagnostic benchmark.Source: §8 · p. 14
  • Across three VLMs, category accuracy remains below 40%, timing accuracy ranges from 1.6% to 12.4%, and exact action-set matching remains below 2%.InternVL3-8B misses all 58 high-severity cases despite its leading aggregate severity accuracy; Qwen2.5-VL-7B exhibits severe category collapse.Source: §8 · p. 14
  • For these three evaluated open VLMs, coarse hazard recognition does not ensure reliable fine-grained defensive reasoning.All models show weak latent-risk anticipation and near-zero self-reported trap recognition.Source: §8 · p. 14
Loading 2610.20627v1…