PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Answer Validation for Knowledge-Based VQA

Jialin Wu, Jiasen Lu, Ashish Sabharwal, Roozbeh Mottaghi

arXiv:2103.12248v3cs.CVcs.CL

TL;DR

Knowledge-based VQA must combine image content with diverse external knowledge, but retrieval often returns noisy or irrelevant facts. MAVEx retrieves multimodal knowledge specific to promising answer candidates and validates those candidates by learning which sources to trust, achieving state-of-the-art performance on OK-VQA.

  • Problem

    Knowledge-based VQA requires external visual, textual, and commonsense knowledge, while retrieving relevant evidence from diverse sources is challenging because directly retrieved facts are often noisy.

  • Method

    MAVEx generates promising answer candidates, retrieves answer-specific knowledge from Wikipedia, ConceptNet, and Google images, and validates candidates using source-specific support.

  • Results

    MAVEx achieves state-of-the-art results on OK-VQA, with a 40.3 score and 41.4 ensemble score.

  • Takeaways & Limitations

    Answer-guided retrieval provides more informative supporting evidence and a more solid knowledge-based VQA system.

  • Takeaways & Limitations

    Failure cases show that MAVEx can over-rely on retrieved facts, ignore retrieved knowledge, or miss key visual clues such as displayed time differences.

Abstract

from arXiv · show

The problem of knowledge-based visual question answering involves answering questions that require external knowledge in addition to the content of the image. Such knowledge typically comes in various forms, including visual, textual, and commonsense knowledge. Using more knowledge sources increases the chance of retrieving more irrelevant or noisy facts, making it challenging to comprehend the facts and find the answer. To address this challenge, we propose Multi-modal Answer Validation using External knowledge (MAVEx), where the idea is to validate a set of promising answer candidates based on answer-specific knowledge retrieval. Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source. Our multi-modal setting is the first to leverage external visual knowledge (images searched using Google), in addition to textual knowledge in the form of Wikipedia sentences and ConceptNet concepts. Our experiments with OK-VQA, a challenging knowledge-based VQA dataset, demonstrate that MAVEx achieves new state-of-the-art results. Our code is available at https://github.com/jialinwu17/MAVEX

1 Introduction

Knowledge-based VQA requires reasoning beyond image content, but retrieving useful evidence from diverse external sources is difficult because direct retrieval is often noisy. MAVEx addresses this by using answer candidates to guide multimodal knowledge retrieval and validate candidates, achieving state-of-the-art results on OK-VQA.

  • Knowledge-based VQA combines visual recognition with logical reasoning and external knowledge from images, encyclopedias, and commonsense knowledge bases.
  • Existing systems typically retrieve knowledge for the question and image before a separate comprehension model predicts the answer.
  • Direct retrieval often produces unhelpful facts, whereas including the correct candidate answer can substantially improve retrieved evidence quality.
  • MAVEx uses promising answer candidates to guide retrieval, incorporates knowledge at multiple granularities, and learns which source to trust for validation.
  • 40.3 score and 41.4 ensemble score establish state-of-the-art results for MAVEx on OK-VQA.

2 Related Work

Related work develops VQA and knowledge-based VQA systems, but conventional VQA models lack explicit external-knowledge mechanisms. Recent approaches increasingly combine multimodal transformers with symbolic or retrieved knowledge.

  • Visual Question Answering: Modern VQA systems use visual detectors and multimodal transformers, but primarily reason from image content without explicitly incorporating external knowledge.
  • Knowledge-Based VQA Datasets: Knowledge-based VQA datasets progress from template-generated or structured-fact questions toward open-domain questions requiring broader knowledge.
  • Knowledge-Based VQA: Earlier knowledge-based approaches retrieve facts, map questions to knowledge-base triplets, or aggregate context-aware knowledge subgraphs.
  • Knowledge-Based VQA: Recent systems use multimodal transformers with implicit pretrained knowledge and, in some cases, combine it with external symbolic knowledge.
  • Knowledge-Based VQA: MAVEx extends this direction by accessing answer-specific external knowledge to assess each candidate's correctness.

3 The MAVEx Framework

MAVEx is a three-stage framework that generates promising answer candidates, retrieves multimodal knowledge guided by those candidates, and validates the candidates using the retrieved evidence.

  • MAVEx generates promising answer candidates before retrieving knowledge guided by each candidate and validating the candidates.
  • The framework mines answer-specific knowledge from Wikipedia and ConceptNet text resources and Google image searches.

3.1 Answer Candidate Generation

MAVEx uses a strong VQA model to generate a small set of high-scoring answer candidates that subsequently guide external knowledge retrieval.

  • A fine-tuned ViLBERT-multi-task model generates candidates by assigning scores to answers collected from the OK-VQA training set.
  • The highest-scoring answers are selected as candidates, while other VQA models or ontology-querying approaches can also serve this generation stage.

3.2 Answer Guided Knowledge Retrieval

MAVEx retrieves answer-specific external knowledge by parsing question and candidate-answer phrases, grounding them in image objects, and matching queries against textual and visual knowledge pools.

  • Query extraction: MAVEx begins with question phrases, answer candidates, and image concepts as retrieval starting points.
  • Query extraction: It extracts target, question, and answer noun phrases using constituency parsing, with answer phrases linking mentioned objects to images.
  • Query extraction: An object linker narrows retrieval to detected objects referred to by question or answer phrases, accepting links with scores above 0.5.
  • Knowledge-pool matching: The retrieval process combines question- and answer-derived phrases with external textual and visual knowledge, as illustrated for a question-answer pair.
  • Knowledge retrieval: For each noun phrase, MAVEx generates head-and-attribute queries, converts each question-answer pair into a statement, and retrieves textual facts from Wikipedia and ConceptNet.
  • Knowledge-pool matching: Textual matching retains sentences with mean recall greater than 0.6 and removes sentences matched to only one query, while visual matching uses linked objects and top-5 Google Images for answer phrases.

3.3 Answer Candidate Validation

The validation module aggregates retrieved knowledge hierarchically across queries and noun phrases, then scores how well each source supports each answer candidate.

  • Multi-granular embedding: MAVEx aggregates knowledge from individual queries into noun-phrase embeddings and then into a whole-question embedding.
  • Knowledge sources: The same embedding design is applied separately to Wikipedia, ConceptNet, and visual knowledge, with distinct learnable parameters for each source.
  • Noun-phrase embedding: For each question noun phrase, multi-head attention uses its ViLBERT features as the query to aggregate retrieved knowledge features.
  • Answer-specific embedding: Answer-specific knowledge is similarly aggregated with attention, using the joint visual-textual representation z as the keys.
  • Source scoring: For each source, MAVEx predicts candidate probabilities by applying a feedforward layer to the knowledge embedding combined with z.
  • Candidate validation: Validation scores combine candidate representations with knowledge features, and consistency criteria require the candidate’s own knowledge to support it more than alternatives.

3.4 Training and Implementation Details

MAVEx is implemented on ViLBERT with standard visual feature extraction and trained in separate candidate-generation and answer-validation stages on OK-VQA.

  • Implementation: The system uses ViLBERT-multi-task with Mask-RCNN and ResNet-152 for object detection and extracts 2,048-dimensional features for up to 100 objects.
  • Data considerations: OK-VQA test images overlap with pre-training and Visual Genome data, creating potential image-level data contamination concerns.
  • Training: The candidate generator is fine-tuned on OK-VQA for 150 epochs and supplies the top 5 answers for each question.
  • Training: The validation module is trained for 75 epochs using binary cross-entropy-based validation objectives, additional VQA losses, and learning rates of 2e-5 and 5e-5.

4 Experiments

MAVEx is evaluated on OK-VQA through intrinsic candidate-generation analysis, main comparisons with prior systems, and ablations of external knowledge sources. The experiments show strong candidate coverage, state-of-the-art performance, and complementary benefits from Wikipedia, ConceptNet, and image knowledge.

  • Experimental Setup: OK-VQA contains 14,031 images and 14,055 questions requiring open-domain knowledge beyond image content.The dataset covers a variety of knowledge categories and lacks a knowledge base covering all questions.
  • Experimental Setup: MAVEx uses the OK-VQA benchmark, with Table 1 comparing systems by external knowledge sources and dataset-version reporting.The table marks pretrained models contaminated by OK-VQA test images and identifies results reported on version 1.1.
  • 4.1 Intrinsic Evaluation: 59.7 VQA soft score is achieved by the best answer among MAVEx’s top-5 candidates, while top-6, top-8, and top-10 sets reach 62.1, 65.1, and 67.1.The top-5 set is selected as a trade-off between efficiency, answer coverage, and accuracy; answers outside the candidate set cannot be produced.
  • 4.2 Main Results: 1.4 points and 6.6 points separate the MAVEx single model from KRISP and ConceptBert, while a three-model ensemble improves 2.47 points over KRISP.Single-model results are averaged across three initialization seeds, with a standard deviation of 0.21.
  • 4.3 Ablation Studies: 3.4, 3.3, and 3.1 points are gained from Wikipedia, ConceptNet, and images, respectively, over the base ViLBERT system.Combining all three sources yields a net 5% gain over the ViLBERT baseline, and decontextualization adds 0.4% compared with Wikipedia alone.
  • 4.3 Ablation Studies: Qualitative examples show MAVEx correcting ViLBERT answers when the base model supplies good candidates but external knowledge is needed to select the correct answer.The framework gathers knowledge from three external sources during validation.

Red (VQA)

MAVEx succeeds on cases where ViLBERT is wrong by using external knowledge, but it also fails when it overweights retrieved facts, visual content, or weak search terms.

  • Red (VQA): Figure 5 contrasts ViLBERT’s incorrect answers with MAVEx’s correct answers using three external knowledge sources.Green boxes mark correct answers, red boxes incorrect answers, and boxes beneath predictions show retrieved knowledge.
  • Red (VQA): Replacing noun-phrase or question-level multi-head attention with average pooling reduces performance to 39.77 and 39.60, respectively.
  • Red (VQA): Without answer validation, MAVEx scores 39.2, 4% higher than ViLBERT base and 1.1% lower than the full model.
  • Red (VQA): The model can over-rely on a retrieved fact and ignore a decisive visual clue, such as wires above a train.
  • Red (VQA): The model can instead rely on visual content, such as a tennis court, while failing to use retrieved knowledge.
  • Red (VQA): A further failure occurs when the model does not recognize the key clue: differing displayed times on clocks.

5 Conclusion

MAVEx addresses knowledge-based VQA by validating candidate answers against answer-specific knowledge from multiple sources. The approach achieves state-of-the-art performance on OK-VQA.

  • 5 Conclusion: MAVEx retrieves answer-specific textual and visual knowledge and learns which sources to trust when validating candidate answers.
  • 5 Conclusion: The method formulates knowledge-based VQA as verifying candidate validity using source-specific support scores.
  • 5 Conclusion: MAVEx achieves state-of-the-art performance on the OK-VQA dataset.
Loading 2103.12248v3…