Source-linked AI summary
LingoQA: Visual Question Answering for Autonomous Driving
Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, Oleg Sinavski
TL;DR
Autonomous-driving VQA needs broader reasoning-focused benchmarks and evaluation methods that better reflect human judgments than conventional language metrics. LingoQA introduces a diverse dataset, benchmark, learned text classifier, and baseline study; its released toolkit supports fast, human-correlated evaluation, while its scope remains limited by short videos, front-camera data, and restricted model testing.
Problem
Autonomous-driving VQA lacks diverse reasoning-focused data and automated free-form answer metrics that evaluate correctness independently of phrasing while aligning with human ratings.
Method
LingoQA combines a free-form autonomous-driving QA dataset and evaluation suite with Lingo-Judge, a text classifier that estimates answer correctness from questions and human and model answers.
Results
Lingo-Judge reaches 0.950 Spearman correlation with human scores and outperforms BLEU, METEOR, CIDEr, and GPT-4 on the benchmark.
Takeaways & Limitations
The released benchmark provides a fast, human-correlated evaluation method for autonomous-driving visual question answering and supports empirical study of vision-language model combinations.
Takeaways & Limitations
The benchmark uses short videos and few frames from a single front-facing camera, omits driving-decision and attention evaluation, and focuses on 7B-parameter models.
Abstract
from arXiv · showhide
We introduce LingoQA, a novel dataset and benchmark for visual question answering in autonomous driving. The dataset contains 28K unique short video scenarios, and 419K annotations. Evaluating state-of-the-art vision-language models on our benchmark shows that their performance is below human capabilities, with GPT-4V responding truthfully to 59.6% of the questions compared to 96.6% for humans. For evaluation, we propose a truthfulness classifier, called Lingo-Judge, that achieves a 0.95 Spearman correlation coefficient to human evaluations, surpassing existing techniques like METEOR, BLEU, CIDEr, and GPT-4. We establish a baseline vision-language model and run extensive ablation studies to understand its performance. We release our dataset and benchmark as an evaluation platform for vision-language models in autonomous driving.
1 Introduction
LingoQA targets trust-sensitive visual question answering for vision-only autonomous driving by combining diverse reasoning questions, an evaluation suite, and a learned text-based metric. The paper also establishes a baseline through empirical studies of vision-language model components and video-fusion choices.
- LingoQA addresses autonomous-driving VQA across scene description, future action prediction, justification, and attention.
- Lingo-Judge uses a learned text classifier to evaluate answer correctness and correlates strongly with human judgments.
- The baseline study finds partial attention-layer fine-tuning with Vicuna-1.5-7B and late video fusion most effective on Action and Scenery datasets.The setup uses 5 video frames over 4 seconds.
- 419.9k free-form QA pairs broaden autonomous-driving datasets beyond perception to driving reasoning and action justifications.
- 0.950 Spearman correlation enables Lingo-Judge to outperform METEOR, BLEU, CIDEr, and GPT-4 as an automated evaluation method.The metric is designed for fast feedback during training and development.
2 Related Work
Prior autonomous-driving research has explored language and multimodal models, but progress is constrained by limited quantitative evaluation and weak alignment between common text metrics and human judgments. LingoQA contributes broader reasoning-focused data and an evaluation approach intended to address these gaps.
- Autonomous-driving research uses visual attention and textual explanations to improve trust, interpretability, and human-AI collaboration.
- Existing work includes GPT-based driving studies, multitask language-control frameworks, and multimodal vision-language-action models.
- Progress remains slow because few works quantitatively improve prior autonomous-driving vision-language models.
- Evaluation Metrics: BLEU, METEOR, ROUGE, and CIDEr rely substantially on n-gram similarity and align weakly with human feedback or answer meaning.
- Datasets for Autonomous Driving: LingoQA expands autonomous-driving QA with questions about perception, scenery, driving behaviour, anticipation, attention, and action justification.
3 LingoQA Benchmark
LingoQA is a benchmark for evaluating autonomous-driving reasoning through free-form video question answering, combining diverse datasets with an automated correctness metric. Lingo-Judge closely matches human judgments while enabling faster, sample-level evaluation.
- LingoQA combines an automated evaluation metric with datasets for evaluating autonomous-driving reasoning through video question answering.
- Evaluation Metric: The benchmark targets free-form answers whose correctness is difficult for n-gram metrics to assess against underlying meaning.
- Evaluation Metric: Lingo-Judge takes a question, human answer, and model answer, then estimates answer correctness using a learned text classifier.
- Evaluation Metric: 95% binary classification accuracy was achieved on a held-out test set, while full-dataset evaluation takes 10 seconds on an A100 GPU.
- Evaluation Metric: 0.95 Spearman correlation and 0.993 Pearson correlation show Lingo-Judge closely tracks human scores and exceeds existing metrics and GPT-4.
- Datasets: 419.9k QA pairs span nine competencies, including action, justification, localisation, counting, anticipation, and counterfactual reasoning.
- Datasets: The Action and Scenery datasets complement each other by covering driving behaviours and perception-focused questions, respectively.
4 Model Methodology
LingoQA Baseline is a Vicuna-based vision-language model that answers autonomous-driving reasoning questions from short video segments. It encodes frames separately, fuses their visual tokens with textual context, and trains in two stages for video QA.
- LingoQA Baseline uses a 7B-parameter Vicuna v1.5 model to answer autonomous-driving reasoning questions grounded in video.
- Architecture: CLIP and Q-Former encode individual frames, whose output tokens are passed with chat history and questions to the language model.
- Architecture: The language model is an autoregressive Vicuna v1.5 LLM conditioned on textual inputs and image tokens to predict the next language token.
- Training: Training first aligns image features with the pretrained language model on GQA and SVIT, then fine-tunes on Action and Scenery video QA.
- Training: Each fine-tuning sample contains five frames drawn from a four-second video span and one question-answer pair.
5 Empirical Evaluation on LingoQA
The evaluation studies how training choices, data composition, temporal inputs, fusion strategies, and language models affect LingoQA performance, then compares vision-language systems with humans. Results show strong dependence on task-specific training and temporal reasoning, while current models remain below human capability.
- Training Recipe and Dataset Mixture: Fine-tuning on LingoQA doubles performance over generic VQA pre-training, with both Action and Scenery datasets improving results.
- Impact of Frame Count: Using 5 frames over 4 seconds preserves multi-frame performance, while single-image input performs slightly worse but remains effective for some scenarios.
- Impact of Video Fusion Strategy: Mid-fusion and late-fusion effectively incorporate video content, supporting temporal information in the vision-language model.
- Impact of Large Language Model: Vicuna-1.5-7B achieves the best language-model score, while Llama-2-7B is slightly lower and OPT-7B substantially underperforms.
- Human Study: Human performance falls from 96.6% with video to 81.8% with a single frame, highlighting the importance of temporal context and single-frame reasoning.
- Model Evaluation: A 23% gap separates single-frame LLaVA from single-frame human capability, while GPT-4V remains 37% below multi-frame human performance.
6 Discussion and Limitations
The discussion presents Lingo-Judge as an efficient, human-correlated evaluation tool while identifying limits in its generalization and in the benchmark’s data, sensing, model, and task scope.
- Strengths of Lingo-Judge: Lingo-Judge is presented as a fast, accurate, and interpretable classifier for benchmarking autonomous-driving VQA against human evaluations.
- Limitations of Lingo-Judge: Lingo-Judge is tailored to LingoQA and may generalize poorly to new questions or response styles, while distinguishing only factual correctness rather than human preference.
- Dataset and Model Limitations: The benchmark uses short videos, few frames, one front-facing camera, and 7B models, and evaluates question answering rather than driving decisions or attention.
7 Conclusion
The paper concludes that LingoQA combines an evaluation dataset, the Lingo-Judge metric, and a comprehensive training dataset to support autonomous-driving VQA research.
- LingoQA provides a learned classifier-based metric, a high-quality training dataset, and an evaluation benchmark highly correlated with human ratings.
- Fast Lingo-Judge feedback supports empirical exploration, while the released benchmark and model-combination experiments provide a foundation for further research.
A LingoQA Dataset Examples
The LingoQA examples span scenery, action, and evaluation datasets, combining descriptive scene questions with questions about driving competencies and interactions.
- Scenery Dataset: The scenery dataset covers descriptive elements such as object colours, junction types, construction zones, traffic lights, and road layout.
- Action Dataset: The action dataset focuses on driving competencies, including traffic-light effects on driving and interactions with other road agents.
- Dataset Examples: Figure 4 presents examples from the scenery, action, and evaluation datasets, with further video examples in the supplementary material.
B Lingo-Judge Examples
Lingo-Judge is evaluated through qualitative examples against GPT-4 and language-based metrics, highlighting its handling of semantically equivalent answers and crucial contradictions. The section also describes GPT-4 evaluation procedures with and without intermediate reasoning.
- Lingo-Judge handles semantically equivalent answers that n-gram metrics may judge dissimilar, such as “None” and “There are no cars.”
- Lingo-Judge identifies crucial answer differences, such as green versus red traffic lights, where CIDEr can still show high agreement.
- GPT-4 can be misled by partially correct answers and incorrectly affirm details such as a traffic-light colour.
- GPT-4 evaluation: With Chain-of-Thought prompting, GPT-4 proposes an evaluation strategy, applies it to a model answer, and assigns a grade from 0 to 5.
- GPT-4 evaluation: Without Chain-of-Thought prompting, GPT-4 receives the question, valid answers, and model prediction before directly assigning a grade from 0 to 5.
- GPT-4 evaluation: GPT-4 evaluation required more than 13 minutes without Chain-of-Thought prompting and more than 50 minutes with it.
D Lingo-Judge Correlation Study
The correlation study compares Lingo-Judge, GPT-4, and standard language metrics against human evaluations across model outputs. Lingo-Judge shows the strongest correlation and tighter confidence intervals, while the study also examines generalisation across answer styles and model architectures.
- Lingo-Judge, GPT-4, BLEU4, METEOR, and CIDEr are compared against human evaluations over scores from 15 models and two human-labeller groups.
- Spearman correlation measures whether a metric ranks answers in the same order as human evaluators, while Pearson correlation measures linear association.
- The study uses Fisher transformation to compute 95% confidence intervals for the correlation coefficients.
- Lingo-Judge provides higher correlation with human evaluations and tighter confidence intervals than the other metrics.
- Generalisation is evaluated across GPT4-V, LLaVA, and FUYU, with human labelling providing ground-truth performance comparisons.
F Training Parameters
The LingoQA Baseline is trained in separate pre-training and fine-tuning stages, using equally weighted datasets in both stages. Training takes 20 hours for pre-training and 5 hours for fine-tuning on an NVIDIA A100 8GPU 80GB machine.
- The baseline training process consists of pre-training followed by fine-tuning, with datasets sampled at equal weight in both stages.
- 20h of pre-training and 5h of fine-tuning are performed on an NVIDIA A100 8GPU 80GB machine.
G LingoQA Baseline Examples
The LingoQA baseline combines action and scenery training to answer both action-justification and perception questions. The section also situates qualitative metric comparisons alongside robustness and failure-case analyses.
- LingoQA Baseline Examples: The baseline combines action and scenery datasets, whereas action-only training favors action questions and scenery-only training favors perception.
- LingoQA Baseline Examples: Figure 11 compares models trained with no LingoQA data, action data only, scenery data only, or both datasets.
- Metric comparisons: Lingo-Judge can detect crucial contradictions such as green versus red traffic lights, while CIDEr may still show high agreement.
- Metric comparisons: Lingo-Judge has failure cases in which its correctness judgement disagrees with human judgement.
- Metric comparisons: Lingo-Judge accuracy is limited for mostly incorrect long-form answers, which can receive higher ratings than warranted by human evaluation.