Source-linked AI summary
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, Roozbeh Mottaghi
TL;DR
Existing VQA benchmarks often use repetitive questions with limited outside knowledge and reasoning. A-OKVQA addresses this gap with about 25K crowdsourced questions requiring commonsense and world knowledge, evaluated through dataset analysis and vision-language baselines. The baselines perform impressively but still leave substantial room for improvement in the required reasoning and knowledge.
Problem
Many VQA datasets rely on simplistic, repetitive questions requiring little world knowledge or reasoning, limiting evaluation of these capabilities.
Method
A-OKVQA crowdsources about 25K questions on COCO images requiring diverse outside knowledge and reasoning, with multiple evaluation formats and rationales.
Results
Baseline models show impressive performance but lack sufficient reasoning capability and/or knowledge, leaving substantial room for improvement.
Takeaways & Limitations
A-OKVQA provides a challenging benchmark for evaluating vision-language models' use of commonsense and world knowledge in visual reasoning.
Abstract
from arXiv · showhide
The Visual Question Answering (VQA) task aspires to provide a meaningful testbed for the development of AI models that can jointly reason over visual and natural language inputs. Despite a proliferation of VQA datasets, this goal is hindered by a set of common limitations. These include a reliance on relatively simplistic questions that are repetitive in both concepts and linguistic structure, little world knowledge needed outside of the paired image, and limited reasoning required to arrive at the correct answer. We introduce A-OKVQA, a crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer. In contrast to the existing knowledge-based VQA datasets, the questions generally cannot be answered by simply querying a knowledge base, and instead require some form of commonsense reasoning about the scene depicted in the image. We demonstrate the potential of this new dataset through a detailed analysis of its contents and baseline performance measurements over a variety of state-of-the-art vision-language models. Project page: http://a-okvqa.allenai.org/
1 Introduction
A-OKVQA is introduced to make VQA more challenging by requiring diverse outside knowledge and multi-step reasoning on real-world images. The dataset provides multiple evaluation formats, rationales, and analyses of baseline models.
- A-OKVQA targets VQA's limited reasoning and world-knowledge demands amid increasingly saturated benchmark performance.
- The dataset requires explicit facts, commonsense, physics, and visual knowledge, combined through image recognition, question understanding, knowledge recall, and reasoning.
- About 25K questions support multiple-choice and direct-answer evaluation, with training rationales explaining the knowledge or reasoning behind answers.
- The authors contribute a benchmark, dataset analysis, baseline evaluations, and result analyses covering tail answers and model complementarity.
2 Related Work
Prior VQA datasets often emphasize perception, retrieval, or constrained reasoning, while A-OKVQA is positioned as an open-domain benchmark requiring commonsense reasoning over real images.
- Visual Question Answering: Most traditional VQA datasets focus on recognition tasks and require little outside knowledge or reasoning.
- Knowledge-based VQA datasets: Earlier knowledge-based datasets use fixed knowledge sources, making their questions constrained and often reducible to knowledge retrieval.
- Knowledge-based VQA datasets: OK-VQA broadened knowledge beyond a fixed source but remained biased toward simple lookup questions with limited reasoning.
- Knowledge-based VQA datasets: A-OKVQA instead emphasizes common-sense knowledge, greater reasoning, and rationale annotations for analyzing knowledge acquisition and reasoning.
- Explanation / Reasoning VQA: Unlike simulated reasoning benchmarks, A-OKVQA requires reasoning over real images and free-form language.
3 A-OKVQA Collection
A-OKVQA was collected from visually rich COCO images and refined through crowd annotation, filtering, diversity controls, and rationale collection.
- Image source: The authors selected COCO images because they contain multiple objects and entity types and support reuse of established models.
- Image source: Training and validation images with more than three objects were labeled interesting, while a ResNet-50 classifier selected test images with 78% validation accuracy.
- Question collection & filtering: 437 crowd-workers wrote and refined questions through qualification, manual review, and automated filtering to improve quality.
- Question collection & filtering: CLIP-based image clustering and question-database checks were used to reduce repetitive questions and encourage diversity.
- Rationales: Workers wrote one- to two-sentence rationales explaining why answers were correct, including world knowledge absent from the images.
4 Dataset Statistics
A-OKVQA combines broad knowledge requirements with diverse, reasoning-intensive questions and annotations designed for both multiple-choice and direct-answer evaluation. Its statistics show substantial scale, long-tail answers, held-out answers, and greater question diversity than OK-VQA.
- 24,903 Question/Answer/Rationale triplets are split into 17.1K training, 1.1K validation, and 6.7K test examples.
- Most answers form a long tail, with a few frequent labels and many infrequent answers.
- 87.6% of validation and 82.7% of test ground-truth answers appear in training, leaving a significant portion of held-out answers unseen during training.
- A-OKVQA provides multiple-choice and direct-answer annotations, with rationales supporting evaluation of knowledge-based responses.
- The dataset spans factoid, commonsense, visual, and physical knowledge, with most questions focusing on commonsense and visual knowledge.
- The average minimum distance is 0.311 between OK-VQA training and A-OKVQA test questions, versus 0.256 between OK-VQA training and test questions.
5 Experiments
The experiments benchmark large-scale pretrained, rationale-generating, and specialized VQA models on A-OKVQA using multiple-choice and direct-answer evaluation. Results show that image-informed and rationale-aware approaches outperform weaker baselines, while specialized models—especially GPV-2—perform best overall.
- Experimental Setup: A-OKVQA evaluates pretrained, rationale-generating, and specialized VQA models in multiple-choice and direct-answer settings.Multiple-choice selects among four options, while direct-answer evaluation allows generated text.
- Large-scale Pre-trained Models: ClipCap outperforms the other large-scale pretrained baselines in direct-answer evaluation.The model combines powerful image features with a fine-tuned language model.
- Rationale Generation: Ground-truth rationales outperform ground-truth captions for prompting GPT-3, while generated rationales provide only a smaller improvement over generated captions.This indicates that question- and answer-specific rationale information is useful, but rationale generation remains improvable.
- Specialized Models: GPV-2 performs best across the specialized-model settings, while KRISP is the strongest model trained only on A-OKVQA.GPV-2 beats models without ground-truth rationales in all settings, and KRISP benefits from direct use of outside knowledge graphs.
- Specialized Models: Models trained on VQAv2 or OK-VQA transfer poorly to A-OKVQA, showing a substantial difference between these datasets.Rationales improve performance, even when their answer tokens are replaced by an [answer] token.
6 Analysis of Models
Model analyses reveal that A-OKVQA is difficult in ways that expose distinct strengths and weaknesses. Performance varies with answer rarity and knowledge type, and models often succeed on different questions.
- Answer Frequency: GPV-2 predicts rare answers better than other methods, especially discriminative models such as LXMERT.Its advantage is strongest for answers appearing 1–5 or 6–10 times in training.
- Knowledge Types: GPV-2 is best overall and across knowledge categories, while KRISP performs relatively better on Knowledge Base and Physical Knowledge questions.KRISP’s performance is consistent with its access to explicit knowledge sources.
- Prediction Overlap/Difference: Only 5.85% of test questions are answered correctly by every model, while 30.96% are answered correctly by none.The worst and best models achieve 15.81% and 40.7% direct-answer accuracy, respectively.
- Prediction Overlap/Difference: 29.2% of GR-GPT’s correctly answered questions are answered incorrectly by GPV-2, indicating substantial differences in model prediction patterns.This comparison uses the percentage of one model’s correct answers missed by another.
- Prediction Overlap/Difference: A majority-vote combination reaches 29.5 accuracy versus 27.1 for its best constituent model, but adding GPV-2 lowers the combination to 35.60 versus GPV-2’s 40.7.The results support complementarity among methods but not uniform benefit from combining them.
- Qualitative Analysis: Qualitative failures include questions that all discussed models cannot answer, illustrating missing reasoning across knowledge types.The examples are presented in Figure 3.
7 Conclusion
A-OKVQA targets under-evaluated reasoning in vision-language models through questions requiring commonsense and world knowledge. Baseline analyses show substantial remaining weaknesses and distinct model strengths, motivating combinations of complementary capabilities.
- Conclusion: A-OKVQA is a knowledge-based VQA benchmark centered on commonsense and world knowledge reasoning.The dataset includes rationales that expose the reasoning leading to each answer.
- Conclusion: Baseline models show impressive performance but still lack sufficient reasoning capability and/or knowledge for the task.The authors identify substantial room for improvement.
- Conclusion: Different models exhibit different weaknesses and strengths, so solving A-OKVQA requires combining capabilities from multiple methods.This conclusion follows the paper’s extensive baseline and error analyses.
A.1 Examples of rejected questions
The dataset filters out questions that can be answered through overly simple recognition, counting, image-independent reasoning, or color identification. About 60% of authored questions were removed for these or related flaws.
- Rejected Question Criteria: Around 60% of written questions were removed because they exhibited one or more quality flaws.The filtering prioritized overall question quality.
- Rejected Question Criteria: Rejected questions included simple recognition of common objects, readily specified-object counting, image-independent questions, and simple color questions.Examples from these four categories appear in Figure 4.
A.2 Data collection interface
Crowdworkers used guided interfaces to write questions and rationales, with similarity checking for questions and answer verification before rationale writing.
- Workers received detailed instructions and examples before writing questions in the collection interface.
- A similarity-checking tool returned the five closest existing questions to prompt workers to rewrite overly similar questions.
- Before writing rationales, workers confirmed or supplied the answer, and questions with answer disagreements were removed.
- Workers then wrote a 1-2 sentence explanation of why the answer was correct, including external knowledge when relevant.
B Additional Details for Large-scale Pre-trained Models
The large-scale pretrained-model experiments use multiple answer-generation and rationale-generation setups, including CLIP-based methods and a modified ClipCap architecture.
- The experiment vocabulary contains 10,424 answers selected from correct choices and answers appearing in at least three training questions.
- CLIP-based experiments include zero-shot and trained settings, with the trained setting learning a linear layer using a CLIP-style contrastive loss.
- The contrastive objective forms cosine similarities between question or image embeddings and corresponding answer encodings, then applies row-wise softmax matching probabilities.
- Modified ClipCap maps CLIP image representations to GPT-2 prefix tokens while conditioning on tokenized questions and ground-truth answers.
- During inference, GPT-2 autoregressively generates answer tokens until EOS; beam search performed worse for the short 1-3-word answers.
- Rationale generation replaces ground-truth answer tokens with rationales, omits multiple-choice choices, uses beam search, and selects checkpoints by BLEU and METEOR validation scores.
D Additional Details for Specialized Models
Specialized-model experiments cover discriminative, pretrained vision-language, knowledge-based, and generative systems, with additional tests using rationales.
- Discriminative methods use a fixed answer vocabulary containing 2,133 unigrams or bigrams spanning 1,937 words.
- Pythia is fine-tuned on A-OKVQA after replacing its classification layer with a randomly initialized layer for the answer vocabulary.
- LXMERT is fine-tuned on A-OKVQA after pretraining on image-sentence and visual question-answering corpora.
- ViLBERT is fine-tuned on A-OKVQA, including variants first trained on VQAv2 or OK-VQA.
- KRISP combines multimodal Transformers with graph neural networks over knowledge graphs, using A-OKVQA-filtered knowledge.
- GPV-2 is a generative model built from T5 and VinVL features, fine-tuned to generate common answers and evaluated with beam search or answer ranking.
- Additional GPV-2 experiments append ground-truth rationales or replace answer mentions with an [answer] token during testing.