Source-linked AI summary
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, Roozbeh Mottaghi
TL;DR
Most VQA benchmarks emphasize visual recognition rather than questions requiring reasoning or external knowledge. The paper introduces OK-VQA, a benchmark of knowledge-dependent questions, and shows that state-of-the-art VQA performance drops substantially in this setting.
Problem
Most existing VQA benchmarks provide limited evidence about reasoning with knowledge beyond the image.
Method
OK-VQA is a benchmark containing questions whose answers require external knowledge resources and reasoning beyond the image.
Results
State-of-the-art VQA performance drops on OK-VQA, while background knowledge can improve results and the benchmark remains challenging.
Takeaways & Limitations
The dataset provides a diverse benchmark for evaluating knowledge-based visual question answering, with more than 14,000 questions across knowledge categories.
Abstract
from arXiv · showhide
Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain. See http://okvqa.allenai.org to download and browse the dataset.
1. Introduction
Existing VQA benchmarks mainly test visual recognition, while ideal VQA also requires reasoning and world knowledge. OK-VQA addresses this gap with questions whose images are insufficient and external resources are necessary.
- Most current VQA questions focus on counting, colors, and visual detection rather than logical reasoning or external knowledge.
- Figure 1 illustrates that answering a teddy-bear question requires linking image content to facts about an American president.The image alone is incomplete; external sentences provide the missing connection and answer.
- OK-VQA contains only questions requiring external resources and evaluates models where image inspection alone cannot produce the answer.Solving these questions also requires identifying needed knowledge, retrieving it, and incorporating it into the answer.
- The dataset includes more than 14,000 questions across knowledge categories including science and technology, history, and sports.The authors characterize it as diverse, difficult, and the largest knowledge-based VQA dataset in natural images at the time.
- The authors benchmark state-of-the-art VQA models, report drastically degraded performance, and propose baselines using unstructured knowledge.
2. Related Work
Prior VQA methods generally lack mechanisms for external knowledge, while knowledge-based systems often rely on structured facts and supervised retrieval. OK-VQA instead targets reasoning over unstructured resources.
- Early, attention-based, and modular VQA systems integrate visual and textual inputs but are not designed to leverage external knowledge.They therefore cannot handle questions whose answers are not fully represented in the image.
- Existing knowledge-based approaches commonly use subject-relation-object or visual concept-relation-attribute triplets with supervised fact retrieval.
- OK-VQA differs by requiring systems to retrieve relevant facts from the web or a database rather than selecting facts from a fixed knowledge base.The comparison is made explicitly with FVQA.
- Knowledge-based question answering has also received substantial attention in NLP.
3. OK-VQA Dataset
The authors motivate OK-VQA by showing that common VQA datasets rarely require background knowledge, then construct a filtered dataset from COCO images with knowledge-requiring questions.
- Common VQA datasets mostly ask recognition questions such as counting objects, identifying animals, or naming colors, so they poorly evaluate knowledge-based systems.
- More than 78% of 10,000 VQA questions were judged answerable by people aged 10 or younger, suggesting little background knowledge is required.
- The authors sample random COCO images from the original training and validation splits to create the new dataset.They selected visually complex images as suitable material for knowledge-based question labeling.
- Workers were prompted to write image-related questions requiring outside knowledge rather than counting or direct visual identification.
- Manual filtering reduced 86,700 collected questions to 34,921 questions requiring knowledge.
- The dataset also addresses answer bias by considering skewed answer patterns during construction.The supplied passage begins this discussion but does not state the completed correction procedure.
4. Dataset Statistics
OK-VQA is designed as a diverse, knowledge-focused benchmark whose questions span ten mutually exclusive knowledge categories and trade dataset size for knowledge and difficulty.
- Comparison with other VQA datasets: OK-VQA is described as the largest VQA dataset specifically targeting knowledge-based VQA on natural scenes, to the authors’ knowledge.
- Question and knowledge diversity: Five MTurk workers assigned each question to one of ten mutually exclusive knowledge categories, with “Other” used when no category had a plurality.
- Comparison with other VQA datasets: The dataset trades off size for knowledge and difficulty because manual filtering avoids dataset pitfalls and emphasizes the long tail of answers.
- Question and knowledge diversity: 14,055 total questions include 12,591 unique questions and 7,178 unique question words, indicating substantial question variety.
- Comparison with other VQA datasets: Question and answer lengths are comparable to KB-VQA and FVQA and longer than most other VQA datasets, except DAQUAR and CLEVR.
- Category-specific patterns: Category-specific terms include bus, sandwich, clouds, herbivore, and umpire, illustrating distinct question and answer patterns across knowledge categories.
5. Benchmarking
The benchmark evaluates standard VQA models, knowledge-retrieval baselines, and combinations of both on OK-VQA. Results show that external knowledge is necessary, retrieved knowledge complements VQA models, and visual-feature improvements alone provide limited gains.
- Knowledge-based baselines: ArticleNet retrieves Wikipedia articles from question and image-derived queries, selects relevant sentences, and trains a network to locate answers within the retrieved text.Its predictions and hidden states support answer extraction and integration with VQA models.
- Knowledge-based baselines: MUTAN + AN and BAN + AN incorporate ArticleNet sentence representations into end-to-end VQA pipelines, while oracle variants provide optimistic upper bounds.The augmented systems feed retrieved sentence states through memory networks before final VQA classification.
- Benchmark setup: MUTAN, BAN, MLP, Q-Only, ArticleNet, augmented models, and oracle variants are benchmarked on the full OK-VQA dataset and its knowledge categories.The evaluation uses the common VQA metric and reports category-level results.
- Benchmark results: No method approaches the 72.41 score reported for the best real open-ended standard-VQA result, indicating that OK-VQA requires information beyond the image.The authors argue that clever visual-question modeling alone is insufficient for this benchmark.
- Benchmark results: ArticleNet improves state-of-the-art VQA models despite low standalone performance, showing that retrieved knowledge provides complementary information.The authors identify stronger benefits in brands, science, and cooking, while noting that much of the dataset requires knowledge beyond Wikipedia.
- Ablations: Removing image features causes a large performance drop, whereas replacing ResNet18 with ResNet50 or ResNet152 yields only marginal improvement.This indicates that OK-VQA remains visually grounded, but retrieval and reasoning are major sources of difficulty.
6. Conclusion
The paper introduces OK-VQA as a benchmark for questions whose answers require reasoning over external knowledge rather than image content alone. Experiments show substantial performance degradation for state-of-the-art VQA models and leave considerable room for improvement.
- Conclusion: OK-VQA evaluates knowledge-based visual question answering when the question and image do not contain sufficient information to answer.The benchmark requires reasoning over external knowledge resources.
- Conclusion: State-of-the-art VQA model performance drops significantly on OK-VQA.The conclusion presents this degradation as evidence of the benchmark’s challenge.
- Conclusion: The dataset analysis shows that background knowledge can improve results, while experiments demonstrate substantial difficulty and considerable room for improvement.The conclusion summarizes both the benchmark’s challenge and the potential value of knowledge-based methods.
A. More Dataset Statistics
The paper characterizes OK-VQA through answer, question, and category statistics, while also describing the retrieval pipeline used to construct external-knowledge articles. These analyses expose dataset distributions and the scale of retrieved evidence per question.
- Dataset statistics: Answer-length and answer-frequency distributions are reported, with all five answers per question included in the frequency histogram.Answers appearing six to ten times comprise about 10% of all answers.
- Dataset statistics: Additional OK-VQA dataset statistics are provided in Table 4.The passage identifies the table as a source of further dataset measurements without specifying individual values.
- Dataset statistics: Figure 10 reports the most common and highest-relative-frequency question words and answers for each knowledge category.Relative frequency is computed as category frequency divided by overall frequency.
- Article extraction: Article extraction collects possible queries, retrieves the top Wikipedia article for each query, and selects a small relevant subset from each article.Queries combine question words with visual entities from object, place, and detection classifiers.
- Article extraction: Each final retrieved article contains its title and five most relevant sentences, with roughly 100 articles retrieved per question during the pipeline.Sentence priority depends on unique query-word count, total query-word count, and sentence order.
B.2. ArticleNet Overview
ArticleNet filters retrieved Wikipedia articles by learning where answers occur, then uses its hidden states to encode relevant sentences for VQA. At inference, the highest-scoring answer-bearing sentences are selected for downstream training.
- ArticleNet training: ArticleNet is trained to predict whether and where the ground-truth answer appears in each retrieved article and sentence.Answer matches are labeled at article, title, sentence, and word levels using Porter stemming.
- ArticleNet architecture: The network receives the question, ResNet152 visual features, article title, and five retrieved sentences, then predicts answer presence in the title, sentences, or article.Its hidden states are later used to encode sentences in the VQA pipeline.
- ArticleNet inference: After training, ArticleNet ranks candidate sentences by asent, and the highest-scoring sentences are used in VQA training.For answer extraction, the system selects the word maximizing awi·asent among retrieved sentences.
- ArticleNet evaluation: Figure 1 shows that retrieval@K is evaluated for both words and sentences.The figure provides a retrieval curve without a reported numerical outcome in the supplied passage.
B.3. ArticleNet Performance
ArticleNet retrieves relevant sentences and words from articles, and its retrieved hidden states are incorporated into MUTAN and BAN through a memory network.
- ArticleNet retrieval: ArticleNet retrieves relevant sentences and words from articles with reasonable accuracy.Ranking words by word score multiplied by sentence score produces higher retrieval than using word score alone.
- Base models: The BAN implementation uses a single Bilinear Attention Network with COCO-trained Faster R-CNN features and a 2,000-answer vocabulary.The MUTAN implementation uses the attention version of Multimodal Tucker Fusion with parameters matched to its best-performing reference model.
- Knowledge integration: MUTAN and BAN incorporate hidden states from the top Nart predicted sentences into an end-to-end memory network.Duplicate sentences are ignored during training and testing.
D. Training and Model Details
The models use specified optimization, encoding, and memory-network configurations, including pre-trained question encoding and tuned hidden-layer settings.
- ArticleNet training: ArticleNet uses a pre-trained skip-thought question encoder, batch normalization, ReLU activations, and Sigmoid output layers.It is trained for 10,000 iterations with batch size 64 and ADAM at learning rate 10^-4.
- VQA model training: MUTAN, MLP, and Q-Only models are trained for 500 epochs, while BAN models are trained for 200 epochs.All use batch size 128 and ADAM with learning rate 10^-4.
- Model configuration: The VQA models use two attention glimpses and hidden feature size 512, settings reported to outperform the VQAv2 defaults on this dataset.The memory network uses 20 retrieved articles, two hops, and hidden size 300.
E. Additional Dataset Examples
The paper provides additional OK-VQA dataset examples in Figures 13–15, illustrating further sample questions from the benchmark.
- Additional examples: Figures 13, 14, and 15 provide additional examples of Outside Knowledge VQA.These figures contain more sample questions from OK-VQA.
- Additional examples: Figures 13–15 are dataset-example figures showing more sample questions from OK-VQA.They extend the paper’s visual examples of the benchmark.