PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases

Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, Dit-Yan Yeung, Huchuan Lu, Xu Jia

arXiv:2404.10595v5cs.CV

TL;DR

Existing LVLM evaluations lack automated, quantifiable assessment of severe road corner cases. The paper introduces CODA-LM and CODA-VLM, and reports that text-only judges align better with human judgments while CODA-VLM performs comparably to GPT-4V and exceeds it on regional perception.

  • Problem

    Existing LVLM evaluations focus primarily on natural images, leaving automated and quantifiable assessment of self-driving corner cases lacking.

    Source: §1 · p. 1

  • Method

    The paper builds CODA-LM with hierarchical GPT-4V pre-annotations verified by humans, evaluates LVLMs with text-only LLM judges, and develops CODA-VLM.

    Source: §1 · p. 2

  • Results

    +21.42%: CODA-VLM surpasses GPT-4V on regional perception while performing comparably overall and surpassing open-source counterparts on CODA-LM.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors position CODA-LM as a catalyst for developing reliable and interpretable autonomous-driving systems.

    Source: §6 · p. 8

    CODA-LM is built on CODA corner cases, which may not cover all possible unexpected driving conditions.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) have received widespread attention for advancing the interpretable self-driving. Existing evaluations of LVLMs primarily focus on multi-faceted capabilities in natural circumstances, lacking automated and quantifiable assessment for self-driving, let alone the severe road corner cases. In this work, we propose CODA-LM, the very first benchmark for the automatic evaluation of LVLMs for self-driving corner cases. We adopt a hierarchical data structure and prompt powerful LVLMs to analyze complex driving scenes and generate high-quality pre-annotations for the human annotators, while for LVLM evaluation, we show that using the text-only large language models (LLMs) as judges reveals even better alignment with human preferences than the LVLM judges. Moreover, with our CODA-LM, we build CODA-VLM, a new driving LVLM surpassing all open-sourced counterparts on CODA-LM. Our CODA-VLM performs comparably with GPT-4V, even surpassing GPT-4V by +21.42% on the regional perception task. We hope CODA-LM can become the catalyst to promote interpretable self-driving empowered by LVLMs.

1. Introduction

Existing self-driving systems and LVLM evaluations are inadequate for severe road corner cases, while manual evaluation limits scale. CODA-LM provides an automated benchmark, and the paper also reports stronger human alignment from text-only judges and strong CODA-VLM results.

1. Introduction

  • Traditional modular self-driving systems remain inadequate at generalizing to open-domain scenarios, especially severe real-world corner cases.A preliminary LVLM study on CODA samples relied on redundant manual inspections, hindering larger-scale evaluation.Source: §1 · p. 1
  • +21.42%: CODA-VLM surpasses GPT-4V on regional perception while remaining comparable overall and surpassing open-source counterparts.The paper presents CODA-VLM as a new driving LVLM evaluated on CODA-LM.Source: §1 · p. 2
  • CODA-LM is the first benchmark for automated, systematic LVLM evaluation on self-driving corner cases, with general perception, regional perception, and driving-suggestion tasks.The benchmark uses hierarchical data to guide GPT-4V pre-annotations that human annotators verify.Source: §1 · p. 2
  • Text-only LLM judges align more closely with human judgments than LVLM judges when evaluating LVLM responses.Source: §1 · p. 2

2. Related Work

Prior LVLM benchmarks focus on natural images, while driving datasets vary in task scope and coverage. The paper positions CODA-LM at the intersection of multimodal evaluation and real-world road corner cases.

2. Related Work

  • Natural-image LVLM benchmarks use manually curated datasets and are difficult to apply to complicated driving scenarios.Examples include MME [15], MMBench [33], SEED-Bench-2 [26], Auto-Bench [24], and Tri-HE [49].Source: §2 · p. 2
  • Driving datasets cover perception, behavior, regional perception, or advice, while CODA provides real-world corner cases across 10K scenes and more than 40 classes.NuScenes-QA [41], BDD-X [25], DRAMA [37], and DriveLM [42] address different driving tasks; StreetHazards [21] simulates corner cases.Source: §2 · p. 2
  • Existing corner-case datasets lack language, whereas vision-language datasets do not cover road corner cases, motivating CODA-LM.CODA-LM is presented as a large-scale multimodal road corner-case dataset with hierarchical automatic evaluation.Source: §2 · p. 2

3. CODA-LM Dataset

CODA-LM organizes real-world road corner cases into a hierarchy of perception and driving-advice tasks. Its tasks assess scene understanding, target-focused interpretation, and advice for the ego car.

3. CODA-LM Dataset

  • CODA-LM comprises 9,768 real-world driving scenarios and organizes assessment into general perception, regional perception, and driving suggestions.The dataset includes annotations for critical road entities and road corner cases, which are grouped into seven entity categories.Source: §3 · pp. 2–3, Figure 2 · p. 3

3.1. General Perception

  • General perception assesses critical entities’ appearance, location, and influence on ego-car behavior across varied time and weather conditions.The conditions include day and night, and clear, cloudy, and rainy weather.Source: §3.1 · p. 3

3.2. Regional Perception

  • Regional perception tests whether LVLMs can describe and explain corner-case objects inside specified bounding boxes.The task focuses on unusual elements that may be overlooked or misinterpreted by traditional models.Source: §3.2 · p. 3

3.3. Driving Suggestions

  • Driving suggestions assesses advice for the ego car after the model perceives the general and regional aspects of its environment.The task is related to autonomous-driving planning and evaluates the formulation of driving strategies.Source: §3.3 · p. 3

4. CODA-LM Construction

CODA-LM construction combines structured GPT-4V pre-annotations with human revision, text-only LLM judging, and a driving-specific LVLM approach. The authors report stronger results for visualized regional prompts and improved consistency from text-only judging.

4.1. Data Collection

  • GPT-4V first describes objects and their effects within a category-to-object-to-element JSON hierarchy, then converts the structured text into coherent pre-annotations for human verification and revision.The paper reports that this structure-coherence pipeline achieves significant consistency with humans.Source: §4.1 · pp. 3–4
  • Red-rectangle visualization of bounding boxes performs significantly better than textual grounding and is used as the default regional-perception prompt.Source: §4.1 · p. 4

4.2. Evaluation Framework

  • A text-only GPT-4 judge scores response correctness from 1 to 10 using reference answers and few-shot examples, and aligns better with human judgments than a GPT-4V judge.The evaluation criteria include accuracy, hallucination penalty, and consistency for general perception, and rationality, relevance, and detail for driving suggestions.Source: §4.2 · pp. 4–5

4.3. CODA-VLM

  • CODA-VLM combines driving-specific instruction fine-tuning with high-resolution sub-image encoding and token pooling to improve visual representation and knowledge transfer.It uses LoRA to fine-tune the language model and visual encoder; a 2×2 MaxPool on sub-image tokens reduces redundancy and offers a better efficiency-performance trade-off.Source: §4.3 · p. 5

5. CODA-LM Benchmark

The benchmark section compares 10 LVLMs across three self-driving corner-case tasks, then analyzes model designs, judges, annotation structure, and visual prompts. It also reports component ablations and identifies limits in CODA-LM’s scenario coverage.

5.1. Main Results

  • CODA-VLM performs best or second best across evaluated dimensions, surpasses all open-source counterparts, and exceeds GPT-4V by +21.42% on regional perception.Figure 4 marks response errors in red and accurate parts in green.Source: Table 2 · p. 6, §5.1 · p. 6, Figure 4 · p. 8

5.2. Analysis

  • AnyRes improves regional perception, while 2×2 MaxPool balances performance and efficiency by reducing redundant visual tokens.Increasing LLaVA1.5-7B input resolution from 224 to 336 lets it outperform Shikra-7B by 20%; LLaVA-NeXT-13B surpasses LLaVA1.5-13B by over 11% with AnyRes.Source: §5.2 · p. 6
  • MiniCPM-V-2.5-8B exceeds LLaVA-NeXT-13B by 12% in general perception and 17% in driving suggestions, while open models lag GPT-4V on those tasks.The authors attribute the smaller model’s results to LLaMA3-8B and adopt it for CODA-VLM with driving-scene fine-tuning.Source: §5.2 · pp. 6–7

5.3.1 Human Consistency of Judges

  • Text-only GPT-4 with reference answers achieves more than 80% human consistency across all three tasks and outperforms GPT-4V judge variants.The comparison evaluates judge rankings against human judgments on sampled model responses.Source: §5.3.1 · p. 7, Table 3 · p. 7

5.3.2 Hierarchical Data Structure for General Perception

  • Structured responses followed by coherence yield the best consistency with human judgments among the tested pre-annotation approaches.The alternatives are plain-text prompting and structured prompting followed by concatenating category annotations.Source: §5.3.2 · p. 7, Table 4 · p. 7

5.3.3 Visual Prompts for Regional Perception

  • Red-rectangle visualization prompts consistently improve regional perception over grounding prompts across evaluated LVLMs, including Shikra-7B.Shikra-7B had been pretrained with grounding data.Source: §5.3.3 · p. 7, Table 5 · p. 7

5.3.4 CODA-VLM Components

  • Ablations on a 200-image test subset find CODA-VLM achieves a better trade-off between efficiency and performance across its tested components.The ablated components are domain-specific fine-tuning, CLIP encoder LoRA training, AnyRes, and 2×2 MaxPool; training time is estimated using 8× A800 GPUs.Source: §5.3.4 · p. 7, Table 6 · p. 7

5.4. Limitations

  • CODA-LM’s CODA-derived corner cases may not cover every unexpected driving condition, and the authors identify controllable generation as future work.The paper also identifies action-level annotations, automated data calibration, and better use of visual pretraining as open directions.Source: §5.4 · p. 8

6. Conclusion

The conclusion presents CODA-LM as a real-world corner-case dataset with a hierarchy of perception and driving-suggestion tasks for automated LVLM evaluation. It also frames the benchmark as a catalyst for reliable, interpretable autonomous driving, while noting that fully intelligent driving remains out of reach.

6. Conclusion

  • CODA-LM provides real-world multimodal road corner cases with a hierarchical framework spanning general perception, regional perception, and driving suggestions.The paper reports a comprehensive evaluation of representative LVLMs and introduces CODA-VLM for driving perception and suggestions.Source: §6 · p. 8
  • The authors say the field remains far from a fully intelligent driving agent and position CODA-LM as a catalyst for reliable, interpretable autonomous driving.This is the paper’s stated intended contribution, not a claim that the benchmark itself delivers a complete driving agent.Source: §6 · p. 8

A. More on Dataset Construction

The appendix documents materials for annotation and evaluation, including prompts, a labeling interface, and few-shot evaluation examples.

A. More on Dataset Construction

  • GPT-4V pre-annotation prompts are provided, and the Gradio interface helps human annotators refine general-perception pre-annotations by merging, modifying, or deleting them.The prompts include system and regional-perception prompts.Source: A. More on Dataset Construction · p. 11, Figure 6 · p. 11
  • Task-specific evaluation prompts use few-shot in-context examples to improve accuracy for general perception and driving suggestions.The appendix points to example sets for both tasks.Source: A. More on Dataset Construction · p. 11

B. More Experiments

The additional experiments compare keyword-based metrics with Text-Score for regional perception and provide an evaluation-framework overview. They report that SPICE tracks model trends better than the other tested keyword metrics, while Text-Score remains the default.

B. More Experiments

  • For regional perception, BLEU-4 and CIDEr do not accurately reflect performance, and METEOR scores do not reliably reflect semantics; SPICE indicates model trends, with InternLM2-vl leading among open-source models.The authors retain Text-Score as the primary metric unless otherwise specified.Source: B. More Experiments · pp. 11–12
  • The evaluation framework uses text-only GPT-4 judges with in-context few-shot examples to assess LVLMs on CODA-LM.The framework overview is shown in Figure 7.Source: Figure 7 · p. 12

C. Qualitative Comparison

This section presents CODA-LM data examples and qualitative comparisons of LVLM responses across three tasks. It also identifies Text-Score as the primary metric for regional perception.

C. Qualitative Comparison

  • Figures 12–14 show CODA-LM data examples, while Figures 15–20 present LVLM response comparisons across general perception, regional perception, and driving suggestions.The section introduction identifies these figure ranges and task coverage.Source: C. Qualitative Comparison · p. 12, Figure 12 · p. 15, Figure 13 · p. 16
  • The regional perception comparison uses Text-Score as its primary metric because traditional metrics may not distinguish LVLM capabilities on complicated autonomous-driving tasks.The table caption does not provide specific model scores or identify a winner.Source: Table 7 · p. 13
  • The general-perception examples describe a green bus ahead in the ego lane and a black SUV ahead, illustrating scene descriptions that include nearby vehicles.The supplied model-answer excerpts identify those lead vehicles but do not provide a full comparison of model quality.Source: C. Qualitative Comparison · p. 18

CODA-VLM

CODA-VLM’s qualitative driving-suggestion examples address following distance, pedestrians, cyclists, and lane constraints. Additional figures present comparisons for regional perception and driving suggestions.

CODA-VLM

  • Figures 17–18 provide additional qualitative comparisons for regional perception, and Figures 19–20 do so for driving suggestions.The captions identify comparison scope but do not report which model performs better.Source: Figure 17 · p. 19, Figure 18 · p. 20, Figure 19 · p. 20
  • A CODA-VLM suggestion for a scene with a black SUV ahead addresses following distance, pedestrians, a no-parking sign, and traffic cones.The provided excerpt also mentions staying in the current lane, though the sentence is truncated.Source: CODA-VLM · p. 20
  • CODA-VLM advises maintaining distance behind a bus, avoiding the bike lane, and watching for pedestrians at a crosswalk while following the green light.The example combines vehicle-following advice with attention to cyclists, pedestrians, and traffic signals.Source: CODA-VLM · p. 21

Related papers

Loading 2404.10595v5…