PaperlayerView exact revision on arXiv

Source-linked AI summary

Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories

Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, Vittorio Ferrari

arXiv:2306.09224v2cs.CV

TL;DR

Encyclopedic-VQA addresses the difficulty of answering visual questions about detailed properties of fine-grained categories and instances. It introduces a large-scale dataset with a controlled Wikipedia knowledge base and shows that retrieval augmentation substantially improves performance over standard vision-language models.

  • Problem

    Large vision-language models perform poorly on visual questions requiring encyclopedic knowledge about fine-grained categories and instances.

    Source: §1 · p. 1

  • Method

    The paper constructs Encyclopedic-VQA with fine-grained category and instance questions, Wikipedia evidence attributions, single-hop and chained two-hop questions, and matched images.

    Source: §1 · pp. 2–3, §2 · p. 3, §4.2 · p. 5

  • Results

    13.0% accuracy for PaLI rises to 87.0% with oracle retrieval of supporting sections, while an automatic retrieval-augmented prototype reaches 48.8%.

    Source: §1 · p. 3, §5.1 · p. 7, §5.2 · p. 8

  • Takeaways & Limitations

    The dataset enables research on retrieval-augmented vision-language models for encyclopedic knowledge.

    Source: §6 · p. 9

Abstract

from arXiv · show

We propose Encyclopedic-VQA, a large scale visual question answering (VQA) dataset featuring visual questions about detailed properties of fine-grained categories and instances. It contains 221k unique question+answer pairs each matched with (up to) 5 images, resulting in a total of 1M VQA samples. Moreover, our dataset comes with a controlled knowledge base derived from Wikipedia, marking the evidence to support each answer. Empirically, we show that our dataset poses a hard challenge for large vision+language models as they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA [37], yet it only achieves 13.0% accuracy on our dataset. Moreover, we experimentally show that progress on answering our encyclopedic questions can be achieved by augmenting large models with a mechanism that retrieves relevant information from the knowledge base. An oracle experiment with perfect retrieval achieves 87.0% accuracy on the single-hop portion of our dataset, and an automatic retrieval-augmented prototype yields 48.8%. We believe that our dataset enables future research on retrieval-augmented vision+language models. It is available at https://github.com/google-research/google-research/tree/master/encyclopedic_vqa .

1. Introduction

Encyclopedic-VQA targets detailed properties of fine-grained entities, addressing gaps in existing knowledge-based VQA datasets. Experiments show that standard VLMs struggle, while retrieval augmentation substantially improves accuracy.

Motivation

  • Detailed properties of long-tail entities are difficult for VLMs to encode, and incorrect answers may be confidently produced without explanations.The paper hypothesizes that such information occurs rarely in web training data and reports PaLI errors on the motivating examples.Source: §1 · p. 1
  • Retrieval augmentation can provide both the knowledge needed to answer encyclopedic questions and attribution that improves interpretability and supports error diagnosis.The retrieved evidence is intended to help users assess answers and model failures.Source: §1 · pp. 1–2
  • Existing OK-VQA and A-OKVQA primarily test commonsense knowledge, while earlier detailed-property datasets lack attribution, scale, or broad coverage.The authors therefore identify no existing VQA dataset as fully satisfactory for encyclopedic questions.Source: §1 · p. 2

Dataset contribution

  • The dataset asks about detailed properties of fine-grained categories and instances, including subjects from iNaturalist 2021 and Google Landmarks Dataset v2.Questions include properties such as founders and maximum ages, and many are complex two-hop questions.Source: §1 · pp. 2–3
  • Encyclopedic-VQA contains 221k unique question-answer pairs matched with around five images, producing 1M examples with Wikipedia answer attribution.Its controlled knowledge base contains 2M Wikipedia pages, and attribution is marked at section granularity.Source: Figure 2 · p. 2, §1 · p. 3

Empirical results

  • 87.0% oracle retrieval accuracy and 48.8% automatic retrieval accuracy contrast with PaLI’s 13.0% accuracy, showing retrieval substantially improves encyclopedic VQA.The results support retrieval-augmented VLMs while leaving room for further improvement.Source: §1 · p. 3

2. Design principles for our dataset

The dataset is designed to be multimodal, encyclopedic, attributable, large-scale, and capable of testing multi-document reasoning. These principles align the benchmark with detailed visual questions and retrieval-based research.

Multimodal

  • Each question should require both the image and textual question, rather than being answerable from either modality alone.Source: §2 · p. 3

Encyclopedic

  • Questions target detailed properties of fine-grained categories or instances that are problematic for vanilla VLMs.Source: §2 · p. 3

Controlled knowledge base

  • Every answer is attributable to a specific part of a free-form multimodal knowledge base, enabling evaluation of whether models answer for the right reason.The knowledge base is intended to contain both text and images.Source: §2 · p. 3

Scale

  • The dataset should be large because scale matters increasingly for large VLMs.Source: §2 · p. 3

Two-hop

  • A portion of questions should require knowledge from multiple documents, creating two-hop challenges and headroom for future model development.Source: §2 · p. 3

3. Related Work

Prior VQA datasets either emphasize generic or commonsense knowledge, use narrower domains or structured knowledge bases, or lack scale and attribution. Encyclopedic-VQA instead combines fine-grained topics, free-form Wikipedia evidence, large scale, and multi-hop questions.

Knowledge-based VQA

  • OK-VQA and A-OKVQA mostly require commonsense knowledge, whereas Encyclopedic-VQA exclusively targets detailed properties of fine-grained categories and instances.A-OKVQA includes 18% detailed-property questions, but mainly about basic-level categories.Source: §3 · p. 3

Knowledge bases

  • Unlike FVQA and KVQA, which use structured RDF knowledge bases, Encyclopedic-VQA uses free-form Wikipedia text and covers animals, plants, and landmarks.KVQA is exclusively about people, while Encyclopedic-VQA offers broader topical coverage.Source: §3 · pp. 3–4

Detailed-property VQA

  • Encyclopedic-VQA differs from S3VQA through larger scale, multi-hop questions, and released knowledge-base attribution.The comparison is 1M versus 7k VQA triplets.Source: §3 · p. 4

Multi-hop reasoning

  • The dataset’s two-hop construction uses bridge entities to chain questions across related Wikipedia pages.This adapts the bridge-entity idea from HotpotQA to compound visual questions.Source: §3 · p. 4

4. Dataset

Encyclopedic-VQA combines fine-grained visual categories and instances with Wikipedia-grounded questions, evidence annotations, and multi-hop reasoning. Its automated and human-assisted construction yields a large, diverse dataset designed to require visual identification and encyclopedic knowledge.

4. Dataset

  • The dataset defines multimodal triplets (I, Q, A) with textual answers and Wikipedia-section evidence, while referring to subjects by super category so visual recognition is necessary.Each question has a single unambiguous answer given the controlled knowledge base.Source: §4 · p. 4

4.1. Supporting Datasets

  • iNat21 and GLDv2 provide fine-grained species and landmark categories, while a 2M-article English Wikipedia knowledge base supports questions about detailed properties; unique article mappings cover 80% and 50%, respectively.Source: §4.1 · pp. 4–5

4.2. Single-hop questions

  • Single-hop questions are templated or automatically generated from category articles, filtered and human-validated, then paired with images of the corresponding category.Templated questions are answered from the relevant Wikipedia page, while generated questions use their source section as answer evidence; multi-answer questions receive completed and verified answer lists.Source: §4.2 · pp. 5–6

4.3. Two-hop questions

  • Two-hop questions chain two single-hop retrieval steps through a bridge entity with its own Wikipedia page, then retain only questions passing answer-based validation.Bridge entities are identified automatically and artificial candidates are discarded before generation and validation.Source: §4.3 · p. 6

4.4. Dataset statistics

  • The dataset contains 221k textual question-answer pairs and 1M image-question-answer triplets across 16.7k categories, using up to five images per pair and 514k unique images.It includes templated, automatically generated, multi-answer, and two-hop questions, with splits designed to test unseen images, questions, answers, and categories.Source: §4.4 · p. 6, Table 2 · p. 6

4.5. Question Diversity

  • Encyclopedic-VQA has the largest question variety among compared knowledge-based VQA datasets and disparity close to the strongest alternatives.Variety is assessed using unique question bigrams, while disparity is average Sentence-BERT cosine distance between question pairs.Source: §4.5 · p. 7, Table 3 · p. 7

5. Experiments

Experiments show that Encyclopedic-VQA is difficult for large models without retrieval, while oracle and automatic retrieval substantially improve performance. Correct document and section retrieval are especially important, including for attribution and more complex question types.

5.1. Large Models without Retrieval

  • PaLI reaches only 13.0% accuracy without retrieval, despite achieving state-of-the-art performance on OK-VQA.PaLM and GPT-3 achieve 19.7% and 15.5% using text questions alone.Source: §5.1 · pp. 7–8

5.2. Large Models with Oracle Retrieval

  • 87.0% accuracy is achieved by PaLM with oracle KB-section retrieval, compared with 13.0% for PaLI without retrieval.Oracle KB-section retrieval reaches 48.8% for PaLI and 82.1% for GPT-3, showing the value of precise evidence.Source: §5.1 · p. 7, §5.2 · p. 8, Table 5 · p. 8

5.3. Large Models with Visual Retrieval

  • Lens-based retrieval raises PaLM and GPT-3 accuracy to 48.0% and 44.9% with retrieved KB articles.Selecting a relevant section raises PaLI to 28.1% and PaLM to 48.8%, while GPT-3 does not improve further.Source: §5.3 · pp. 8–9

5.4. Attribution for Retrieval-Augmented models

  • Correct KB-section retrieval is essential: PaLM accuracy is 82.3% with the correct section but 20.7% with an incorrect one.Google Lens retrieves the correct article 47.4% of the time and the correct section 45.6% of the time.Source: §5.4 · p. 9, Table 6 · p. 9

5.5. Comparison to PromptCap

  • PromptCap outperforms models used alone but remains substantially worse than the paper’s retrieval-augmented methods.This comparison further supports the difficulty of Encyclopedic-VQA.Source: §5.5 · p. 9

5.6. Multi-answer and two-hop questions

  • Lens KB-section retrieval obtains 9.2% and 14.7% for PaLI on multi-answer and two-hop questions, while PaLM reaches 33.6% and 22.8%.GPT-3 reaches 32.1% and 18.7%, respectively, confirming the challenge of these question types.Source: §5.6 · p. 9

6. Conclusions

The paper introduces Encyclopedic-VQA, a large-scale dataset of detailed questions about fine-grained categories and instances with answer attributions. Experiments show that standard VLMs struggle, while retrieval augmentation improves performance but leaves room for further progress.

6. Conclusions

  • Encyclopedic-VQA combines detailed questions about fine-grained categories and instances with a knowledge base containing answer attributions.The authors show that retrieval augmentation substantially improves results while leaving headroom for future research.Source: §6 · p. 9

A. Correctness of our dataset

A user study evaluates whether Encyclopedic-VQA questions are answerable from their corresponding Wikipedia pages and whether collected answers are correct. Most questions satisfy this criterion, matching the answerability estimate for A-OKVQA.

A. Correctness of our dataset

  • 86% of sampled Encyclopedic-VQA questions are answerable from their corresponding Wikipedia pages with correct collected ground-truth answers.Six experts answered sampled questions, with majority agreement used for validation.Source: A. Correctness of our dataset · p. 13
  • A-OKVQA also has 86% answerable questions under its evaluation criterion, providing context for the Encyclopedic-VQA quality estimate.The comparison uses at least three equivalent answers among ten ground-truth answers.Source: A. Correctness of our dataset · p. 13
  • The user study concludes that Encyclopedic-VQA is of very high quality.This conclusion follows the expert validation procedure and its 86% answerability result.Source: A. Correctness of our dataset · p. 13

B. Quality of BEM [12] evaluation measure

The paper evaluates BEM against human judgments and compares it with exact matching for this dataset. BEM closely tracks human judgments, whereas exact matching is often overly strict.

B. Quality of BEM [12] evaluation measure

  • BEM judgments equal human judgments in 96% of evaluated cases, with humans stricter in 1% and BEM stricter in 3%.This supports BEM as a measure closely aligned with human correctness judgments.Source: B. Quality of BEM [12] evaluation… · p. 13
  • Exact-match judgments agree with human judgments in only 68% of cases and always reject answers that humans judge correct when they differ.The result justifies using BEM rather than exact matching for Encyclopedic-VQA evaluation.Source: B. Quality of BEM [12] evaluation… · p. 13

C. CLIP retrieval in existing retrieval-augmented VQA systems [22, 34]

CLIP-based retrieval from Wikipedia titles and descriptions retrieves the correct article poorly on Encyclopedic-VQA. This limits the expected effectiveness of KAT [22] and REVIVE [34], whereas Lens performs better for identifying subjects.

Implications for existing systems

  • Because KAT [22] and REVIVE [34] depend on retrieving knowledge containing the answer, their CLIP retrieval component is unlikely to work well on Encyclopedic-VQA.The paper evaluates this component in isolation because correct retrieval is crucial for the downstream answer generator.Source: C. CLIP retrieval in existing… · pp. 13–14

CLIP retrieval

  • 3.3% top-1 recall and 16.5% top-20 recall show that CLIP retrieves the correct Wikipedia article infrequently.The retrieval query uses the image embedding, while Wikipedia articles are represented by CLIP embeddings of concatenated titles and descriptions.Source: C. CLIP retrieval in existing… · pp. 13–14, Table 7 · p. 13

Lens retrieval

  • Lens works better for subject identification because it searches billions of visually similar web images and can use attached metadata to identify categories.The identified category is then used to find the relevant Wikipedia page; specialized image classifiers are also suggested as a possible substitute.Source: C. CLIP retrieval in existing… · p. 14

D. Qualitative results

Qualitative examples show that retrieval can turn plausible but incorrect guesses about detailed properties into precise, attributable answers. However, some answers remain incorrect even when the relevant knowledge is retrieved.

Retrieval effects

  • Retrieved context substantially improves answers to detailed questions, producing precise answers attributable to the retrieved knowledge.The examples concern properties such as an individual reptile’s egg count, dates, and a specific fish’s size.Source: D. Qualitative results · p. 14

Remaining failures

  • Even with the correct knowledge retrieved, generated answers can remain incorrect because models may struggle to use the retrieved information accurately.The qualitative results therefore separate retrieval failures from answer-generation failures.Source: D. Qualitative results · p. 14

E. Dataset Statistics

This appendix points readers to Table 8 for a detailed overview of dataset statistics.

Dataset statistics

  • Table 8 provides dataset statistics, with further details available in Section 4 of the main paper.The cited passage does not report the table’s numerical values.Source: E. Dataset Statistics · p. 14

F. Prompts for dataset creation

The appendix describes prompt-based procedures for constructing and evaluating single-hop and two-hop questions. It also documents retrieval prompts, validation steps, qualitative comparisons, and dataset-scale reporting.

F. Prompts for dataset creation

  • FLAN PaLM prompts rephrase automatically generated single-hop questions and chain two single-hop questions into two-hop questions.The prompts use specific instructions, chain-of-thought examples where possible, and difficult cases to guide model behavior.Source: F. Prompts for dataset creation · p. 14

F.1. Rephrasing automatically generated single-hop questions

  • Single-hop rephrasing replaces a category name with its supercategory, such as changing a named church question to “Where is this church located?”The procedure filters unchanged outputs and achieved 90% correctness in a sample of 100 rephrased questions.Source: §F.1 · p. 14

F.2. Chaining single-hop questions into two-hop questions

  • Two-hop questions are formed by using the first question’s answer as a bridge entity while preserving the second question’s answer.For example, an animal’s competitor becomes the entity queried for population size.Source: §F.2 · pp. 14–15
  • Figures 6 and 7 compare PaLI and PaLM answers across no-retrieval, Lens-section, and oracle-section setups, including ground-truth answers.The examples include both successful cases and a case where every setup fails.Source: Figure 6 · p. 15, Figure 7 · p. 15
  • Two-hop outputs are validated by asking the model to answer them from the original question-answer pairs and retaining only matching answers.A sample of 100 chained questions was assessed as 88% correct after filtering common failures.Source: §F.2 · pp. 15–16

G. Prompts for evaluation

  • Evaluation prompts support question-only answering, retrieval with entity names, full articles, or sections, using the image additionally for PaLI.Lens and oracle setups share retrieval prompts, while missing retrieved content falls back to question-only answering.Source: G. Prompts for evaluation · p. 16, §G.1 · p. 16, §G.2 · p. 19

G.2. Prompt for relevant section identification

  • Relevant Wikipedia sections are identified by asking PaLM whether each section contains the answer and concatenating all sections marked relevant.The prompt uses yes/no examples, and unmatched outputs are treated as indicating irrelevance.Source: §G.2 · pp. 16, 19
  • Prompt examples operationalize entity anonymization by replacing names with “this” plus the entity type, and chaining questions so the final answer matches the second question’s answer.The appendix gives examples for towers, fish, buildings, animals, plants, and countries, along with prompt placeholders for questions and answers.Source: §G.2 · pp. 17–18

Related papers

Loading 2306.09224v2…