PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, Ming-Wei Chang

arXiv:2302.11713v5cs.CVcs.AIcs.CL

TL;DR

The paper asks whether pretrained vision-language models can answer visual questions requiring fine-grained knowledge rather than common sense or image-only reasoning. It introduces INFOSEEK and evaluates pretrained and knowledge-base-assisted systems. Models struggle on these questions, but fine-tuning and accurate visual entity recognition with document retrieval improve performance.

  • Problem

    It remains unclear whether pretrained vision-language models can answer knowledge-intensive, information-seeking questions grounded in visual entities.

  • Method

    The paper introduces INFOSEEK, a benchmark with human and large-scale generated visual information-seeking questions, and evaluates No-KB and With-KB model protocols.

  • Results

    Pretrained models struggle with fine-grained visual information-seeking questions, while fine-tuning and visual entity recognition improve performance; entity-recognition improvements raise capability from 18% to 45.6%.

  • Takeaways & Limitations

    Accurate visual entity recognition and access to relevant external documents provide substantial headroom for visual information-seeking question answering.

  • Takeaways & Limitations

    INFOSEEK is limited to English and primarily focuses on knowledge derived from Wikipedia.

Abstract

from arXiv · show

Pre-trained vision and language models have demonstrated state-of-the-art capabilities over existing tasks involving images and texts, including visual question answering. However, it remains unclear whether these models possess the capability to answer questions that are not only querying visual content but knowledge-intensive and information-seeking. In this study, we introduce InfoSeek, a visual question answering dataset tailored for information-seeking questions that cannot be answered with only common sense knowledge. Using InfoSeek, we analyze various pre-trained visual question answering models and gain insights into their characteristics. Our findings reveal that state-of-the-art pre-trained multi-modal models (e.g., PaLI-X, BLIP2, etc.) face challenges in answering visual information-seeking questions, but fine-tuning on the InfoSeek dataset elicits models to use fine-grained knowledge that was learned during their pre-training. Furthermore, we show that accurate visual entity recognition can be used to improve performance on InfoSeek by retrieving relevant documents, showing a significant space for improvement.

1 Introduction

The paper introduces INFOSEEK to test whether pretrained vision-language models can answer visual questions requiring fine-grained, information-seeking knowledge beyond image content or common sense. Models struggle in zero-shot settings, while fine-tuning and visual entity recognition with knowledge-base retrieval substantially improve performance.

  • Benchmark motivation: 70.8% of OK-VQA questions are answerable without search, compared with 4.4% for INFOSEEK’s fine-grained visual questions.This contrast motivates a benchmark focused on knowledge about specific visual entities rather than broadly familiar information.
  • Motivation: INFOSEEK targets visual information-seeking questions whose answers cannot be obtained from visual content alone.The dataset is designed around questions about specific entities, such as a church’s construction date or interior decorator.
  • Contribution: INFOSEEK combines a manually curated test set with over 1 million generated image-question-answer triplets for multimodal fine-tuning.The dataset construction pairs visual entities with fine-grained knowledge and supports evaluation beyond common-sense answering.
  • Findings: Pretrained vision-language models, including PaLI-X and BLIP2, struggle with INFOSEEK questions in zero-shot evaluation.BLIP2 outperforms PaLI-X before fine-tuning, but both face difficulty answering knowledge-intensive queries.
  • Findings: Fine-tuning INFOSEEK data substantially improves PaLI-X and enables it to outperform fine-tuned BLIP2 on questions unseen during fine-tuning.The result suggests that fine-tuning can elicit knowledge already present in pretrained models.
  • Findings: 18% to 45.6%: improving visual entity recognition increased model capability on visual information-seeking questions.The experiment supports retrieving relevant entity documents as a promising route for improving performance.

2 The Need for a New Visual Information-seeking Benchmark

Existing visual knowledge-question answering benchmarks often test common-sense knowledge, allow answers from text alone, or cover too few entity categories. INFOSEEK is introduced to evaluate fine-grained knowledge grounded in visual entities.

  • Limitations of existing benchmarks: 70.8% of OK-VQA questions can be answered without a search engine, indicating emphasis on commonly known, coarse-grained knowledge.Such questions may require identifying only a broad object type rather than the specific visual entity.
  • Limitations of existing benchmarks: OK-VQA is unsuitable for evaluating long-tailed visual-entity knowledge because many questions do not require information beyond the image and common sense.This limits its ability to expose weaknesses in pretrained models’ handling of rare knowledge.
  • Limitations of existing benchmarks: PaLM achieves 31.5% accuracy on ViQuAE using questions alone, outperforming an image-access retrieval model by 9.4%.The result shows that many ViQuAE questions reveal enough textual information to answer without inspecting the image.
  • Limitations of existing benchmarks: ViQuAE and related datasets provide limited entity coverage, with over 43% of ViQuAE questions involving human entities.Narrow coverage can reduce task complexity and restrict evaluation of knowledge across entity categories.
  • INFOSEEK: INFOSEEK benchmarks information-seeking questions about visual entities rather than only testing whether models can identify those entities.It is designed to probe pretrained knowledge beyond visual entity recognition.

3 INFOSEEK: A VQA Benchmark of Visual Information-seeking Questions

INFOSEEK combines human-written and large-scale automatically generated visual information-seeking questions, links visual entities to external knowledge, and evaluates generalization across unseen entities and questions.

  • Dataset composition: INFOSEEK contains 8.9K human-written questions and 1.3M automatically generated examples covering diverse visual entities.The two components support realistic evaluation and large-scale training.
  • Dataset composition: The dataset uses images from nine classification and retrieval datasets, including landmarks, animals, food, and aircraft.Visual entities are linked to corresponding Wikipedia articles to provide knowledge sources.
  • Human data: Human annotators write questions from entity images, short descriptions, and Wikipedia section titles without seeing the answers.A two-stage process is used to encourage genuine information-seeking intent and reduce question-only shortcuts.
  • Annotation: Annotators label concise answers from Wikipedia and classify questions as TIME, NUMERICAL, or STRING.The resulting image-question-answer triples undergo human verification, with 95% annotation accuracy reported as evidence of reliability.
  • Automated data: The automated component converts Wikidata triples for 300 relations into natural-language question-answer pairs using human-authored templates.Filtering ensures answers can be found in Wikipedia, while subsampling balances entities and relations.
  • Evaluation: Evaluation holds out both entities and questions, reporting separate split accuracies and their harmonic mean as overall accuracy.STRING and TIME use VQA accuracy, whereas NUMERICAL uses Relaxed Accuracy.

4 Protocols and Models for INFOSEEK

INFOSEEK evaluates models under No-KB and With-KB protocols, separating direct image-question answering from pipelines that recognize visual entities and use knowledge-base information. The experiments include end-to-end vision-language models, language QA models, and retrieval-based KB readers.

  • Evaluation protocols: INFOSEEK defines No-KB and With-KB protocols according to whether auxiliary visual-entity and knowledge-base data are available during training.No-KB requires predicting answers directly from the image and question; With-KB decomposes the task into visual entity recognition and language QA.
  • Evaluation protocols: No-KB models must store world knowledge in their parameters and use it after fine-tuning to answer image-question pairs.This protocol tests how much knowledge end-to-end models memorize during pre-training and how effectively fine-tuning elicits it.
  • Evaluation protocols: With-KB models explicitly recognize a visual entity and then perform textual reasoning using entity information grounded in a knowledge base.The two-stage design supports separate analysis of visual entity recognition and language reasoning, with improved interpretability through component-level diagnosis.
  • Models: The model suite includes BLIP2, InstructBLIP, PaLI-17B, and PaLI-X, with PaLI models fine-tuned on INFOSEEK because their zero-shot performance is minimal.BLIP2 and InstructBLIP connect a frozen vision encoder to a frozen instruction-tuned language model, whereas PaLI uses non-instruction-tuned language models.
  • With-KB models: The visual entity recognition stage uses CLIP to retrieve the top k=5 similar entities from 100K multimodal Wikipedia entries.Candidates are ranked using weighted cosine similarity between the query and candidate representations.
  • With-KB models: The language QA stage compares PaLM, which relies on memorized pre-training knowledge, with FiD, which reads retrieved Wikipedia articles.FiD takes N=100 retrieved articles as input and is fine-tuned on INFOSEEK.

5 Experiments

Experiments show that pre-trained models struggle with visual information-seeking questions, while INFOSEEK fine-tuning and explicit knowledge access improve performance. Pipeline systems are strong overall, but end-to-end models retain advantages for tail entities and some less fine-grained questions.

  • No-KB results: Pre-trained vision-language models struggle in zero-shot INFOSEEK, while fine-tuned PaLI-X improves significantly and surpasses fine-tuned BLIP2 on unseen queries.BLIP2 nevertheless outperforms PaLI-X before fine-tuning.
  • No-KB results: Fine-tuning improves BLIP2’s country-location accuracy on unseen entity Amberd from 18% to 92%.The entity was not present in the training set, indicating strong generalization on this example.
  • No-KB results: InstructBLIP(0-shot) scores 7.4 versus BLIP2’s 11.3 on INFOSEEK Wikidata and tends to produce coarse-grained answers such as “architect” instead of a person’s name.The authors attribute the behavior to instruction-tuning datasets with less fine-grained answer distributions; INFOSEEK fine-tuning helps close the gap.
  • With-KB results: KB-access pipeline models significantly outperform the best No-KB models on the INFOSEEK Human split.The comparison supports using visual recognition and language reasoning with explicit knowledge access for natural info-seeking questions.
  • With-KB results: An oracle entity recognizer raises FiD accuracy from ∼20% to ∼50%, revealing substantial headroom from improving visual entity recognition.The simulated recognition accuracy increases from 22% using CLIP to 100%.
  • With-KB results: On INFOSEEK Human, FiD outperforms PaLM on TIME questions (21.5 vs 14.6) and NUMERICAL questions (25.6 vs 21.3).The authors suggest Wikipedia articles help with fine-grained information that is rare or difficult for PaLM to memorize.
  • Question types: End-to-end models have a shortfall on fine-grained TIME and NUMERICAL questions, while performing better on attribute and relation queries.The reported human-split results highlight this fine-grained knowledge gap.
  • Entity popularity: End-to-end PaLI-17B outperforms pipeline systems on tail entities, particularly geographical questions, whereas pipelines win by a large margin on popular head entities.End-to-end models can choose alternative reasoning routes, including answering some geographical questions without identifying the entity.

6 Related Work

Prior work covers pre-trained vision-language models and knowledge-based VQA systems that connect visual inputs with language models or external knowledge. INFOSEEK builds on these directions by combining entity recognition, retrieval, and language QA to study visual information-seeking.

  • Pre-trained vision-language models: Pre-trained vision-language research commonly augments language models with visual inputs through mappings from external visual encoders to frozen language models.This line of work uses large-scale image-text pre-training and related architectures.
  • Knowledge-based VQA models: Knowledge-based VQA approaches retrieve external knowledge, transform images into captions, or use language QA models to answer questions.INFOSEEK studies both retrieval-based and language-model-based approaches with the OVEN model.
  • Knowledge-based VQA models: INFOSEEK combines visual entity recognition with Wikidata and Wikipedia-based reasoning to examine the difficulty of visual information-seeking questions.This connects knowledge-based VQA pipelines with the paper’s visual entity grounding setup.

7 Conclusion

INFOSEEK evaluates visual information-seeking questions requiring fine-grained knowledge and shows that current pre-trained visual-language models struggle with them. Pipeline analyses further indicate that incorporating fine-grained knowledge into pre-training could improve end-to-end models.

  • INFOSEEK is a large-scale VQA dataset focused on visual information-seeking questions.
  • Current state-of-the-art pre-trained visual-language models struggle with fine-grained questions about visual entities, including time and numerical information.
  • Pipeline systems grounding visual entities in an external knowledge base suggest that incorporating fine-grained knowledge into pre-training could improve end-to-end models.

8 Limitation

INFOSEEK is limited to English and primarily focuses on knowledge derived from Wikipedia. Future work could extend the dataset to other languages and domains.

  • INFOSEEK is limited to the English language.
  • The dataset primarily focuses on knowledge derived from Wikipedia.
  • Future research could extend INFOSEEK to multilingual settings, medical information, artwork, and emerging Wikipedia updates.

A.1 Human Annotation Quality Control

INFOSEEK combines controlled human annotation with expert verification and filtering against Wikipedia evidence. Its figures illustrate the dataset’s examples, question distributions, entity categories, and annotation workflow.

  • Annotation Procedure: Annotators write non-binary, non-visual-attribute questions from Wikipedia entity descriptions and section titles.
  • Annotation Procedure: Stage 2 labels short Wikipedia answer spans and classifies questions as TIME, NUMERICAL, or OTHERS.
  • Expert Feedback and Correction: Expert annotators review question-answer pairs and image-question-answer triples, rejecting unqualified examples and clarifying ambiguous images.
  • Annotation Quality: 95% accuracy in expert judgments supports the reliability of the INFOSEEK Human annotations.
  • Filtering: QA pairs are filtered by checking answer consistency between Wikidata and Wikipedia, using exact or fuzzy matching and limited time tolerance.

A.3 Evaluation Metric.

INFOSEEK evaluates STRING, TIME, and NUMERICAL questions with type-specific matching rules. Overall accuracy is computed across unseen-question and unseen-entity splits using their harmonic mean.

  • STRING and TIME: STRING and TIME questions use VQA accuracy against multiple answer references.
  • Matching Rules: Exact Match counts a prediction as correct when it matches any reference exactly.
  • TIME: TIME evaluation allows a one-year error tolerance because historical dates are often estimated.
  • NUMERICAL: NUMERICAL questions use Relaxed Accuracy with a 10% tolerance range for single values and range predictions.
  • Overall Evaluation: Overall accuracy is the harmonic mean of results on UNSEEN QUESTION and UNSEEN ENTITY splits.

B Implementation details of the baseline systems

The baseline systems use varied fine-tuning, prompting, decoding, and retrieval configurations for INFOSEEK. Reported results include scores for OFA and mPLUG-owl, while PaLM variants and FID use different information settings.

  • Model configurations: PaLI and PaLI-X are fine-tuned on the INFOSEEK training set, with PaLI-X having 55B parameters.
  • Model configurations: BLIP2 and InstructBLIP are fine-tuned with LAVIS using the prompt “Question: [question] Short answer:”.
  • Model configurations: OFA-large is fine-tuned for 20k steps with beam-search decoding using beam size 5.
  • Reported results: 11.7 and 4.0 are OFA's INFOSEEK Wikidata and Human split scores, respectively.
  • Reported results: 7.7 is mPLUG-owl's score on the INFOSEEK Human split after 10k-step LoRA fine-tuning.
  • Information and retrieval settings: PaLM uses 5-shot in-context learning, while FID is fine-tuned in two stages using entity passages and INFOSEEK data.

C Additional Experiment Results

Additional experiments characterize annotation categories, evaluation splits, and supplementary INFOSEEK results. The dataset separates unseen entities from unseen questions to assess generalization beyond training examples.

  • Additional experiments: Supplementary tables report complete With-KB results, Wikidata validation results, question-type scores, and OK-VQA annotation results.
  • Additional experiments: Five adult annotators categorize 500 sampled OK-VQA training examples into three categories.
Loading 2302.11713v5…