PaperlayerView exact revision on arXiv

Source-linked AI summary

Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering

Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, Soumen Chakrabarti

arXiv:2103.05568v3cs.CVcs.LG

TL;DR

OKVQA questions vary in whether they require integrating image information with outside knowledge, and answer overlap can make frequent-answer guessing effective. The paper introduces annotated and newly built datasets focused on select–substitute–search reasoning, alongside an interpretable system that outperforms strong baselines in classification and open-domain settings.

  • Problem

    OKVQA includes image-independent, speculative, OCR-only, and image-answerable questions, while 48.9% of test answers occur in training.

    Source: §1.1 · p. 2, §2 · p. 3

  • Method

    The paper builds an annotated OKVQA subset and S3VQA around queries that refer to an image entity through a more general type, and presents an interpretable system for this reasoning pattern.

    Source: §1.2 · p. 2, §7 · p. 7

  • Results

    S3 outperforms strong baselines in both existing classification and proposed open-domain settings.

    Source: §7 · p. 7

  • Takeaways & Limitations

    The challenge makes this form of reasoning structurally clear and supports systematic evaluation of how systems handle it.

    Source: §1.2 · p. 2

    Existing OKVQA has substantial train-test answer overlap and heterogeneous question types, including OCR-only and speculative questions.

    Source: §2 · p. 3

Abstract

from arXiv · show

Multimodal IR, spanning text corpus, knowledge graph and images, called outside knowledge visual question answering (OKVQA), is of much recent interest. However, the popular data set has serious limitations. A surprisingly large fraction of queries do not assess the ability to integrate cross-modal information. Instead, some are independent of the image, some depend on speculation, some require OCR or are otherwise answerable from the image alone. To add to the above limitations, frequency-based guessing is very effective because of (unintended) widespread answer overlaps between the train and test folds. Overall, it is hard to determine when state-of-the-art systems exploit these weaknesses rather than really infer the answers, because they are opaque and their 'reasoning' process is uninterpretable. An equally important limitation is that the dataset is designed for the quantitative assessment only of the end-to-end answer retrieval task, with no provision for assessing the correct(semantic) interpretation of the input query. In response, we identify a key structural idiom in OKVQA ,viz., S3 (select, substitute and search), and build a new data set and challenge around it. Specifically, the questioner identifies an entity in the image and asks a question involving that entity which can be answered only by consulting a knowledge graph or corpus passage mentioning the entity. Our challenge consists of (i)OKVQAS3, a subset of OKVQA annotated based on the structural idiom and (ii)S3VQA, a new dataset built from scratch. We also present a neural but structurally transparent OKVQA system, S3, that explicitly addresses our challenge dataset, and outperforms recent competitive baselines.

1 INTRODUCTION

OKVQA requires combining image information with outside knowledge, but its questions can rely on image-only cues, speculation, or answer-frequency shortcuts. The paper introduces a challenge focused on a transparent select–substitute–search pattern and an interpretable system designed for it.

1.1 Discontent with current state of OKVQA

  • Existing OKVQA mixes questions requiring outside knowledge with OCR, speculation, image-only answers, and questions independent of the image.The benchmark also has 48.9% test answers appearing in training, making frequent-answer classification an effective shortcut.Source: §1.1 · p. 2
  • The paper presents a new dataset with clear query structure and an interpretable neural architecture, pursuing both data and algorithm remedies to OKVQA’s pitfalls.The authors suggest these resources may contribute to question answering over video, audio, and knowledge graphs.Source: §1.1 · p. 2

1.2 A new OKVQA Challenge Dataset

  • The challenge pairs an annotated OKVQA subset with S3VQA, whose questions target an image entity referred to through its more general type.The S3VQA questions are designed to require knowledge-graph, image-extracted, and open-domain search information.Source: §1.2 · p. 2

1.3 An interpretable OKVQA system

  • S3 selects a question span and a scene-graph object to substitute, then uses the reformulated query for answer classification or open-domain retrieval.The open-domain option retrieves responses and extracts answer spans in a reading-comprehension style.Source: §1.3 · pp. 2–3

2 LIMITATIONS IN EXISTING OKVQA DATA

The authors identify answer overlap and heterogeneous reasoning requirements as major limitations of OKVQA. Some questions call for object detection followed by external knowledge, while others require OCR, speculation, or different forms of reasoning.

2 LIMITATIONS IN EXISTING OKVQA DATA

  • 48.9% of OKVQA test answers also occur in training, enabling systems to boost accuracy by restricting predictions to frequent training answers.The dataset contains over 14,000 question-image pairs, with 9,000 training and 5,000 test examples.Source: §2 · p. 3
  • Only 18% of questions require object detection followed by reasoning over external knowledge; 7% require OCR alone, and 12% involve opinion or speculation.The remaining questions are described as another category, illustrated in Figure 2.Source: §2 · p. 3
  • Type-1 questions resemble bridging queries and can be reformulated with a scene graph into queries answerable directly through Web search.This observation motivates development of the challenge dataset.Source: §2 · p. 3

3 DESIGN OF A NEW OKVQA CHALLENGE DATA SET

The challenge structures outside-knowledge VQA around selecting a question span, substituting image information, and searching external sources. It provides an annotated OKVQA subset and a new dataset designed to test these operations.

3 DESIGN OF A NEW OKVQA CHALLENGE DATA SET

  • S3 frames each query as Select, Substitute, and Search, with annotations enabling the operations to be evaluated independently.Gold spans support selection evaluation; substitutions and reformulated queries with gold answers support substitution and search evaluation.Source: §3 · p. 3

3.1 OKVQAS3: Annotated subset of OKVQA

  • 2,640 of OKVQA’s 14,000 question-image pairs (18.8%) are Type 1 and form the annotated OKVQAS3 subset.The subset includes gold spans, substitutions, and answers; detected objects omit the gold object for 24.7% of its pairs because of detection errors and vocabulary limitations.Source: §3.1 · pp. 3–4

3.2 S3VQA: New dataset built from scratch

  • S3VQA is built from Open Images entities, their parent labels, and Wikipedia snippets used to generate questions that explicitly mention each entity.Each entity mention is replaced with its parent label, and questions answerable from the image alone or not fact-based are removed; each question has one correct answer.Source: §3.2 · p. 4

4 PRIOR ARCHITECTURES

Prior outside-knowledge VQA systems retrieve evidence from articles or knowledge graphs, but commonly predict from the most frequent training answers.

4 PRIOR ARCHITECTURES

  • ArticleNet retrieves relevant Wikipedia articles, while Gardères et al. integrate ConceptNet with visual and language embeddings; both formulate answer prediction as classification over frequent training answers.ArticleNet encodes articles with a GRU and uses them with question and image features; Gardères et al. use graph convolution networks.Source: §4 · p. 4

5 PROPOSED ARCHITECTURE

S3 reformulates a question by selecting a span and substituting an image object, then searches retrieved text for an answer. Its MRC component supports both open-domain answer prediction and classification for comparison with prior work.

5.1 Select and Substitute

  • S3 selects a question span, matches it to a detected image object, and substitutes that object to produce a reformulated query.The SpanSelector predicts span boundaries; object substitution uses a combined triplet and binary cross-entropy loss, with a hypernym-hyponym scorer drawing on WordNet.Source: §5.1 · pp. 4–5

5.2 Search

  • The search stage retrieves the top 10 Google snippets for each reformulated question and passes them to an MRC module to extract an answer.The MRC module prunes the retrieved text to information relevant to the reformulated query.Source: §5.2 · p. 5
  • MRC enables direct answer prediction from a question and context, while a classification option supports comparisons with prior work.For classification, embeddings of the reformulated question and MRC output are concatenated and fed to a softmax over the top k training answers.Source: §5.2 · p. 5

6 EXPERIMENTS AND RESULTS

The experiments compare S3 with BLOCK on OKVQAS3 and S3VQA under classification and open-domain evaluation. S3’s open-domain OKVQAS3 result slightly exceeds its best classification result, while MRC improves classification performance over omitting it.

6.1 Methods

  • The comparison includes BLOCK, which fuses image-question representations before predicting among frequent training answers, and S3, which uses select, substitute, and search operations.Adding context as an input to BLOCK did not improve its accuracy.Source: §6.1 · p. 5

6.2 Evaluation

  • OKVQAS3 classification predicts from the top-k training answers, with k=1084; S3VQA accuracy uses exact string match against its single ground-truth answer.The OKVQAS3 test set has 42.1% answer overlap with training, while S3VQA has 11.6%.Source: §6.2 · p. 6

6.3 Implementation details

  • S3 selects a question span, substitutes detected entities to identify the matching image object, then uses retrieved text snippets for answer extraction and classification.The implementation uses BERT for select and substitute, SpanBERT for snippet answer extraction, and a BLOCK fusion module for S3 classification.Source: §6.1 · pp. 5–6, §6.3 · p. 6

6.4 Results

  • MRCsing and MRCmult substantially outperform S3 without MRC on OKVQAS3 classification, and MRCmult consistently outperforms MRCsing; BLOCK significantly underperforms S3.The experiments use the top 10 Google snippets, concatenated as T10text.Source: §6.4 · p. 6, Table 1 · p. 6
  • 28.9 vs. 28.57: the best open-domain OKVQAS3 result is slightly better than the best classification result; BLOCK scores close to 0 on S3VQA.S3VQA has exactly one correct ground-truth answer per question, making evaluation stricter.Source: Table 2 · p. 7, §6.4 · p. 7
  • The open-domain MRC comparison covers T5 and SpanBERT, with separate results for pretrained models and models fine-tuned on OKVQAS3.The passage reports that fine-tuning significantly helps performance.Source: Table 3 · p. 7, §6.4 · p. 7

7 CONCLUSION

The paper presents two challenge datasets and a structurally transparent S3 system for OKVQA. The authors report that S3 outperforms strong baselines in classification and open-domain settings.

7 CONCLUSION

  • The paper identifies opacity and uninterpretable reasoning as limitations of existing multimodal QA datasets.Source: §7 · p. 7
  • The challenge comprises OKVQAS3, an annotated subset of OKVQA, and S3VQA, a newly built dataset focused on a recurring structural idiom in VQA.Source: §7 · p. 7
  • S3 is a structurally transparent and interpretable system tailored to the challenge datasets, and the authors report that it outperforms strong baselines in both settings.Source: §7 · p. 7

Related papers

Loading 2103.05568v3…