PaperlayerView exact revision on arXiv

Source-linked AI summary

LingoQA: Visual Question Answering for Autonomous Driving

Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, Oleg Sinavski

arXiv:2312.14115v4cs.ROcs.AIcs.CV

TL;DR

Autonomous-driving VQA needs broader reasoning-focused benchmarks and evaluation methods that better reflect human judgments than conventional language metrics. LingoQA introduces a diverse dataset, benchmark, learned text classifier, and baseline study; its released toolkit supports fast, human-correlated evaluation, while its scope remains limited by short videos, front-camera data, and restricted model testing.

  • Problem

    Autonomous-driving VQA lacks diverse reasoning-focused data and automated free-form answer metrics that evaluate correctness independently of phrasing while aligning with human ratings.

    Source: §1 · p. 2, §2.2 · p. 4, §3.1 · p. 5

  • Method

    LingoQA combines a free-form autonomous-driving QA dataset and evaluation suite with Lingo-Judge, a text classifier that estimates answer correctness from questions and human and model answers.

    Source: §1 · pp. 2–3, §3.1 · p. 6

  • Results

    Lingo-Judge reaches 0.950 Spearman correlation with human scores and outperforms BLEU, METEOR, CIDEr, and GPT-4 on the benchmark.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The released benchmark provides a fast, human-correlated evaluation method for autonomous-driving visual question answering and supports empirical study of vision-language model combinations.

    Source: §1 · p. 2, §7 · p. 14, §6 · p. 14

    The benchmark uses short videos and few frames from a single front-facing camera, omits driving-decision and attention evaluation, and focuses on 7B-parameter models.

    Source: §6 · p. 14

Abstract

from arXiv · show

We introduce LingoQA, a novel dataset and benchmark for visual question answering in autonomous driving. The dataset contains 28K unique short video scenarios, and 419K annotations. Evaluating state-of-the-art vision-language models on our benchmark shows that their performance is below human capabilities, with GPT-4V responding truthfully to 59.6% of the questions compared to 96.6% for humans. For evaluation, we propose a truthfulness classifier, called Lingo-Judge, that achieves a 0.95 Spearman correlation coefficient to human evaluations, surpassing existing techniques like METEOR, BLEU, CIDEr, and GPT-4. We establish a baseline vision-language model and run extensive ablation studies to understand its performance. We release our dataset and benchmark as an evaluation platform for vision-language models in autonomous driving.

1 Introduction

LingoQA targets trust-sensitive visual question answering for vision-only autonomous driving by combining diverse reasoning questions, an evaluation suite, and a learned text-based metric. The paper also establishes a baseline through empirical studies of vision-language model components and video-fusion choices.

  • LingoQA addresses autonomous-driving VQA across scene description, future action prediction, justification, and attention.Source: Figure 1 · p. 2, §1 · p. 2
  • Lingo-Judge uses a learned text classifier to evaluate answer correctness and correlates strongly with human judgments.Source: §1 · pp. 2–3
  • The baseline study finds partial attention-layer fine-tuning with Vicuna-1.5-7B and late video fusion most effective on Action and Scenery datasets.The setup uses 5 video frames over 4 seconds.Source: §1 · pp. 2–3
  • 419.9k free-form QA pairs broaden autonomous-driving datasets beyond perception to driving reasoning and action justifications.Source: §1 · pp. 2–3
  • 0.950 Spearman correlation enables Lingo-Judge to outperform METEOR, BLEU, CIDEr, and GPT-4 as an automated evaluation method.The metric is designed for fast feedback during training and development.Source: §1 · pp. 2–3

2 Related Work

Prior autonomous-driving research has explored language and multimodal models, but progress is constrained by limited quantitative evaluation and weak alignment between common text metrics and human judgments. LingoQA contributes broader reasoning-focused data and an evaluation approach intended to address these gaps.

  • Autonomous-driving research uses visual attention and textual explanations to improve trust, interpretability, and human-AI collaboration.Source: §2.1 · p. 3
  • Existing work includes GPT-based driving studies, multitask language-control frameworks, and multimodal vision-language-action models.Source: §2.1 · pp. 3–4
  • Progress remains slow because few works quantitatively improve prior autonomous-driving vision-language models.Source: §2.2 · p. 4

Evaluation Metrics

  • BLEU, METEOR, ROUGE, and CIDEr rely substantially on n-gram similarity and align weakly with human feedback or answer meaning.Source: §2.2 · p. 4

Datasets for Autonomous Driving

  • LingoQA expands autonomous-driving QA with questions about perception, scenery, driving behaviour, anticipation, attention, and action justification.Source: §2.3 · p. 4

3 LingoQA Benchmark

LingoQA is a benchmark for evaluating autonomous-driving reasoning through free-form video question answering, combining diverse datasets with an automated correctness metric. Lingo-Judge closely matches human judgments while enabling faster, sample-level evaluation.

  • LingoQA combines an automated evaluation metric with datasets for evaluating autonomous-driving reasoning through video question answering.Source: §3 · p. 5

Evaluation Metric

  • The benchmark targets free-form answers whose correctness is difficult for n-gram metrics to assess against underlying meaning.Source: §3.1 · p. 5
  • Lingo-Judge takes a question, human answer, and model answer, then estimates answer correctness using a learned text classifier.Source: §3.1 · p. 6
  • 95% binary classification accuracy was achieved on a held-out test set, while full-dataset evaluation takes 10 seconds on an A100 GPU.Source: §3.1 · p. 6
  • 0.95 Spearman correlation and 0.993 Pearson correlation show Lingo-Judge closely tracks human scores and exceeds existing metrics and GPT-4.Source: §3.1 · p. 7

Datasets

  • 419.9k QA pairs span nine competencies, including action, justification, localisation, counting, anticipation, and counterfactual reasoning.Source: §3.2 · pp. 7–8
  • The Action and Scenery datasets complement each other by covering driving behaviours and perception-focused questions, respectively.Source: §3.2 · p. 8, Figure 2 · p. 9

4 Model Methodology

LingoQA Baseline is a Vicuna-based vision-language model that answers autonomous-driving reasoning questions from short video segments. It encodes frames separately, fuses their visual tokens with textual context, and trains in two stages for video QA.

  • LingoQA Baseline uses a 7B-parameter Vicuna v1.5 model to answer autonomous-driving reasoning questions grounded in video.Source: §4 · p. 9

Architecture

  • CLIP and Q-Former encode individual frames, whose output tokens are passed with chat history and questions to the language model.Source: Figure 3 · p. 10, §4.1 · p. 10
  • The language model is an autoregressive Vicuna v1.5 LLM conditioned on textual inputs and image tokens to predict the next language token.Source: §4.1 · p. 10

Training

  • Training first aligns image features with the pretrained language model on GQA and SVIT, then fine-tunes on Action and Scenery video QA.Source: §4.2 · p. 11
  • Each fine-tuning sample contains five frames drawn from a four-second video span and one question-answer pair.Source: §4.2 · p. 11

5 Empirical Evaluation on LingoQA

The evaluation studies how training choices, data composition, temporal inputs, fusion strategies, and language models affect LingoQA performance, then compares vision-language systems with humans. Results show strong dependence on task-specific training and temporal reasoning, while current models remain below human capability.

Training Recipe and Dataset Mixture

  • Fine-tuning on LingoQA doubles performance over generic VQA pre-training, with both Action and Scenery datasets improving results.Source: §5.1 · p. 11

Impact of Frame Count

  • Using 5 frames over 4 seconds preserves multi-frame performance, while single-image input performs slightly worse but remains effective for some scenarios.Source: §5.1 · p. 12

Impact of Video Fusion Strategy

  • Mid-fusion and late-fusion effectively incorporate video content, supporting temporal information in the vision-language model.Source: §5.1 · p. 12

Impact of Large Language Model

  • Vicuna-1.5-7B achieves the best language-model score, while Llama-2-7B is slightly lower and OPT-7B substantially underperforms.Source: §5.1 · p. 12

Human Study

  • Human performance falls from 96.6% with video to 81.8% with a single frame, highlighting the importance of temporal context and single-frame reasoning.Source: §5.2 · p. 13

Model Evaluation

  • A 23% gap separates single-frame LLaVA from single-frame human capability, while GPT-4V remains 37% below multi-frame human performance.Source: §5.2 · p. 13, Table 5 · p. 13

6 Discussion and Limitations

The discussion presents Lingo-Judge as an efficient, human-correlated evaluation tool while identifying limits in its generalization and in the benchmark’s data, sensing, model, and task scope.

Strengths of Lingo-Judge

  • Lingo-Judge is presented as a fast, accurate, and interpretable classifier for benchmarking autonomous-driving VQA against human evaluations.Source: §6 · p. 14

Limitations of Lingo-Judge

  • Lingo-Judge is tailored to LingoQA and may generalize poorly to new questions or response styles, while distinguishing only factual correctness rather than human preference.Source: §6 · p. 14

Dataset and Model Limitations

  • The benchmark uses short videos, few frames, one front-facing camera, and 7B models, and evaluates question answering rather than driving decisions or attention.Source: §6 · p. 14

7 Conclusion

The paper concludes that LingoQA combines an evaluation dataset, the Lingo-Judge metric, and a comprehensive training dataset to support autonomous-driving VQA research.

  • LingoQA provides a learned classifier-based metric, a high-quality training dataset, and an evaluation benchmark highly correlated with human ratings.Source: §7 · p. 14
  • Fast Lingo-Judge feedback supports empirical exploration, while the released benchmark and model-combination experiments provide a foundation for further research.Source: §7 · p. 14

A LingoQA Dataset Examples

The LingoQA examples span scenery, action, and evaluation datasets, combining descriptive scene questions with questions about driving competencies and interactions.

Scenery Dataset

  • The scenery dataset covers descriptive elements such as object colours, junction types, construction zones, traffic lights, and road layout.Source: A LingoQA Dataset Examples · p. 20

Action Dataset

  • The action dataset focuses on driving competencies, including traffic-light effects on driving and interactions with other road agents.Source: A LingoQA Dataset Examples · p. 20

Dataset Examples

B Lingo-Judge Examples

Lingo-Judge is evaluated through qualitative examples against GPT-4 and language-based metrics, highlighting its handling of semantically equivalent answers and crucial contradictions. The section also describes GPT-4 evaluation procedures with and without intermediate reasoning.

  • Lingo-Judge handles semantically equivalent answers that n-gram metrics may judge dissimilar, such as “None” and “There are no cars.”Source: B Lingo-Judge Examples · p. 21
  • Lingo-Judge identifies crucial answer differences, such as green versus red traffic lights, where CIDEr can still show high agreement.Source: B Lingo-Judge Examples · p. 21
  • GPT-4 can be misled by partially correct answers and incorrectly affirm details such as a traffic-light colour.Source: B Lingo-Judge Examples · p. 21

GPT-4 evaluation

  • With Chain-of-Thought prompting, GPT-4 proposes an evaluation strategy, applies it to a model answer, and assigns a grade from 0 to 5.Source: C GPT-4 Grading · pp. 21–22
  • Without Chain-of-Thought prompting, GPT-4 receives the question, valid answers, and model prediction before directly assigning a grade from 0 to 5.Source: C GPT-4 Grading · p. 22
  • GPT-4 evaluation required more than 13 minutes without Chain-of-Thought prompting and more than 50 minutes with it.Source: C GPT-4 Grading · p. 22

D Lingo-Judge Correlation Study

The correlation study compares Lingo-Judge, GPT-4, and standard language metrics against human evaluations across model outputs. Lingo-Judge shows the strongest correlation and tighter confidence intervals, while the study also examines generalisation across answer styles and model architectures.

F Training Parameters

The LingoQA Baseline is trained in separate pre-training and fine-tuning stages, using equally weighted datasets in both stages. Training takes 20 hours for pre-training and 5 hours for fine-tuning on an NVIDIA A100 8GPU 80GB machine.

  • The baseline training process consists of pre-training followed by fine-tuning, with datasets sampled at equal weight in both stages.Source: F Training Parameters · p. 25
  • 20h of pre-training and 5h of fine-tuning are performed on an NVIDIA A100 8GPU 80GB machine.Source: F Training Parameters · p. 25

G LingoQA Baseline Examples

The LingoQA baseline combines action and scenery training to answer both action-justification and perception questions. The section also situates qualitative metric comparisons alongside robustness and failure-case analyses.

LingoQA Baseline Examples

Metric comparisons

  • Lingo-Judge can detect crucial contradictions such as green versus red traffic lights, while CIDEr may still show high agreement.Source: Table 6 · p. 29
  • Lingo-Judge has failure cases in which its correctness judgement disagrees with human judgement.Source: Table 7 · p. 30
  • Lingo-Judge accuracy is limited for mostly incorrect long-form answers, which can receive higher ratings than warranted by human evaluation.Source: Table 8 · p. 30

Related papers

Loading 2312.14115v4…