PaperlayerView exact revision on arXiv

Source-linked AI summary

FVQA: Fact-based Visual Question Answering

Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, Anthony Dick

arXiv:1606.05433v4cs.CV

TL;DR

Existing VQA largely limits questions to information directly available from images and questions, excluding many commonsense and factual queries. FVQA addresses this gap with a dataset containing structural supporting facts and a model that maps questions to knowledge-base queries, achieving 58.76% accuracy with an ensemble.

  • Problem

    Existing VQA focuses mainly on questions answerable from direct image and question analysis, whereas many visual questions require external commonsense knowledge.

  • Method

    FVQA adds structural supporting facts to image-question-answer data and maps questions to knowledge-base queries over image and external knowledge.

  • Results

    58.76% accuracy is achieved by the Ensemble model, which combines complementary conventional LSTM-based and knowledge-based VQA models.

  • Takeaways & Limitations

    The dataset and approach provide answers with supporting facts and avoid limiting test answers to concepts observed in training.

Abstract

from arXiv · show

Visual Question Answering (VQA) has attracted a lot of attention in both Computer Vision and Natural Language Processing communities, not least because it offers insight into the relationships between two important sources of information. Current datasets, and the models built upon them, have focused on questions which are answerable by direct analysis of the question and image alone. The set of such questions that require no external information to answer is interesting, but very limited. It excludes questions which require common sense, or basic factual knowledge to answer, for example. Here we introduce FVQA, a VQA dataset which requires, and supports, much deeper reasoning. FVQA only contains questions which require external information to answer. We thus extend a conventional visual question answering dataset, which contains image-question-answerg triplets, through additional image-question-answer-supporting fact tuples. The supporting fact is represented as a structural triplet, such as <Cat,CapableOf,ClimbingTrees>. We evaluate several baseline models on the FVQA dataset, and describe a novel model which is capable of reasoning about an image on the basis of supporting facts.

I. INTRODUCTION

FVQA targets VQA questions that require external commonsense or factual knowledge beyond direct image and question analysis. It introduces supporting facts and an explicit reasoning approach, achieving higher accuracy than existing baseline VQA methods.

  • Existing VQA datasets largely emphasize questions answerable through direct analysis of images and questions, limiting the depth of required understanding.
  • FVQA focuses on questions requiring external information, because supplying all background knowledge through image-question-answer training examples would be prohibitive.The number of concepts requiring illustration scales combinatorially.
  • Each FVQA question-answer pair includes a supporting fact represented as a structural triplet, enabling answer inference, fact search, and evaluation of inference chains.For example, <Cat,CapableOf,ClimbingTrees> supports answering which pictured animal can climb trees.
  • RNN-based baselines lack explicit reasoning and are limited to concepts observed during training, despite the much larger space of possible concepts and relationships.Without explicit reasoning, their answers may reflect image information or answer prevalence in the training set.
  • The proposed method detects image content, links it to combined knowledge bases, maps questions to queries, and processes retrieved supporting facts into answers.Its reported Top-1 accuracy is 56.91%, outperforming existing baseline VQA methods.

II. RELATED WORK

Prior VQA datasets and models mainly address image-grounded question answering, while related KB-QA work supplies external knowledge through structured or unstructured sources. FVQA instead combines visual grounding with queries over large structured knowledge bases and explicit supporting facts.

  • A. Visual Question Answering Datasets: Existing VQA datasets span visual question types, grounding settings, and answer formats, but many focus primarily on visual knowledge.Examples include object-level grounding, free-form generation, multiple-choice selection, and fill-in-the-blank tasks.
  • B. Visual Question Answering Methods: Dominant VQA methods combine convolutional and recurrent networks, often using spatial attention, but commonly restrict answers to a fixed vocabulary.These methods are collectively referred to as RNN approaches in the paper.
  • C. Knowledge-bases and VQA: General image questions require information not contained in the image, including facts about absent objects or identities such as Mona Lisa.
  • C. Knowledge-bases and VQA: Structured knowledge bases represent external information as triples connecting concepts through relationships, and may be manually annotated or automatically extracted.The paper combines DBpedia, WebChild, and ConceptNet.
  • C. Knowledge-bases and VQA: Compared with the closest prior method, FVQA avoids predefined question templates and expands beyond one manually annotated knowledge source by adding two automatically learned bases.The paper links this expansion to answering more general questions.
  • C. Knowledge-bases and VQA: Unlike approaches that query unstructured Web articles or use image-specific schemas, this work queries structured relational knowledge bases for general knowledge.The paper notes that image-specific schemas prohibit questions relying on general world knowledge.

III. CREATING THE FVQA DATASET

FVQA uses a specialized question-collection system that constrains annotators to create questions requiring both image-grounded visual concepts and external commonsense supporting facts.

  • Annotators choose an image-related visual concept, select a relevant associated fact, and write a question whose answer requires both sources.The system records the annotators’ choices and provides a tutorial restricting questions to those needing visual and external knowledge.

A. Images and Visual Concepts

The dataset is built from images and extracted visual concepts linked to facts from multiple structured knowledge bases. Annotators then use these concepts and facts to construct supported visual questions.

  • A. Images and Visual Concepts: The image pool contains 2,190 samples from Microsoft COCO validation images and ImageNet test images, combining complex scenes with broader object-category coverage.Microsoft COCO contributes 80 object categories and ImageNet contributes 200.
  • A. Images and Visual Concepts: Visual concepts include detected objects, scenes, and actions extracted using visual recognition models and image attributes.The excerpt specifies 326 distinct object classes.
  • B. Knowledge Bases: Knowledge about visual concepts is drawn from DBpedia, ConceptNet, and WebChild.
  • B. Knowledge Bases: Extracted information is stored as RDF triples and accessed through SPARQL queries, with relationships and fact counts summarized in Table I.The table also gives example facts associated with visual concepts.
  • C. Question Collection: Question collection proceeds by selecting a visual concept, selecting a relevant fact, and asking a question whose answer requires both image and fact information.The answer is restricted to one of the two concepts in the supporting fact.

D. Data Statistics

FVQA contains 5,826 questions linked to 4,216 unique supporting facts, organized across visual concepts, relationships, knowledge-base sources, and answer sources.

  • 5,826 questions correspond to 4,216 unique supporting facts, collected by 38 volunteers and divided across five random train/test splits.Each split contains 1,100 training images and 1,090 test images, with roughly 2,927 training and 2,899 test questions.
  • FVQA defines 32 question types using key visual concept, key relationship, KB source, and answer source categories.Key visual concepts are objects, scenes, or actions; answer sources are Image or KB.
  • 80% of questions rely on ConceptNet facts, while DBpedia and WebChild support 14% and 6%, respectively.
  • Questions and facts are distributed across relationships including Category, UsedFor, IsA, RelatedTo, CapableOf, AtLocation, HasProperty, and HasA.The figure identifies UsedFor, Category, IsA, RelatedTo, and CapableOf as the five most frequent relationships.

E. Human Study of Common-sense Knowledge

Human evaluations indicate that FVQA questions generally require external common-sense knowledge and that the associated supporting facts are considered useful for answering them.

  • 97.6% of collected questions were judged to require common-sense knowledge by at least two of three subjects.
  • More than 99% of supporting facts for knowledge-requiring questions were judged valuable for answering those questions.

F. Comparison

FVQA differs from prior VQA datasets by supplying structured supporting facts and enabling reasoning that combines image information with external knowledge bases.

  • FVQA provides supporting facts for visual questions, addressing external commonsense knowledge that existing VQA datasets generally do not supply.
  • The dataset restricts answers to concepts from images and knowledge bases, excluding Yes/No questions.The paper notes that random guessing can achieve approximately 50% accuracy on balanced Yes/No datasets.
  • The proposed approach maps questions to KB queries rather than directly to answers, avoiding a fixed answer vocabulary and exposing supporting facts.
  • The unified KB links image-extracted visual concepts to corresponding concepts across multiple external knowledge bases.
  • LSTM models predict key visual concepts, key relationships, and answer sources, covering 32 combinations in the dataset.

C. Querying the KB

The KB-querying stage searches an image-linked knowledge base for candidate facts, then matches their results to question keywords to select an answer.

  • A query is constructed from the image id, predicted key relationship, and predicted key visual-concept type to retrieve candidate supporting facts.
  • The query searches all visual concepts grounded in the image, restricts their type, and retrieves entities connected by the key relationship.
  • The highest-scoring candidate fact is selected after matching retrieved results to question keywords, and its associated visual concept becomes the answer.In the example, matching “climb trees” selects the Cat fact and returns Cat.
  • The search runs over the entire knowledge base rather than only the 4,216 facts used to construct the collected questions.
  • For the example query CapableOf,Object, candidate pairs include Dog–Playing Frisbee, Cat–Chasing Rats, and Cat–Climbing Trees.

D. Answering

The method selects answers by predicting the answer source and matching question content to candidate knowledge-base facts. It supports both image-side and knowledge-base-side answers through different selection procedures.

  • Candidate selection: The question-to-concept matching score is defined as Jaccard similarity between normalized concept and question-keyword sets.The score measures relevance between each candidate supporting fact and the question.
  • TAS = Image: When TAS = Image, the highest-scoring candidate fact is selected and its visual concept ?X becomes the answer.The matching score compares question keywords with candidate concept ?Y using Jaccard similarity over normalized word sets.
  • TAS = KB: For TAS = KB and TKVC = Scene or Action, the most probable visual concept ?X is selected and its corresponding concept ?Y becomes the answer.The probabilities come from the softmax layer of the visual models.
  • Answer ordering: A visual concept ?X can correspond to multiple concepts ?Y, whose answers are ordered by frequency in the training data.The most frequent answer appears first.

V. EXPERIMENTS

The experiments evaluate question-query mapping, answer accuracy, answer-source and visual-concept conditions, knowledge-base sources, and explicit supporting-fact prediction. Results show strong gains over baselines but expose errors in perception, query mapping, and answer post-processing.

  • Overall accuracy: 58.76% is the best overall accuracy, achieved by the Ensemble model, while top-3-QQmaping is the best-performing single model at Top1.The ensemble combines conventional LSTM-based and knowledge-based models; top-3-QQmaping outperforms top-1-QQmaping but remains below gt-QQmaping.
  • Knowledge-base sources: Top-3-QQmapping outperforms other baselines for DBpedia and ConceptNet questions, but trails Hie-Question+Image+Pre-VQA on Webchild.The comparison notes that Hie-Question+Image+Pre-VQA uses extra data.
  • Visual concepts: Object-related questions substantially outperform scene- and action-related questions, especially when image features are provided.The authors connect this pattern to VggNet pretraining on object classification.
  • Answer sources: Answering Image-source questions is nearly five times more accurate than answering KB-source questions.The authors attribute the gap to the nearly unlimited answer space when answers do not directly appear in the image.
  • Failure analysis: Failure cases arise from incorrect visual concepts, incorrect question-to-query mappings, or post-processing that selects the wrong answer from multiple returned facts.For KB answers, the ground truth may not be first even when it appears among the top-N answers.
  • Supporting-fact prediction: 41% supporting-fact prediction accuracy demonstrates that the model can identify the correct fact from millions of incorporated knowledge-base facts.The approach uniquely reports explicit supporting facts alongside predicted answers.

VI. CONCLUSION

The paper introduces FVQA and a knowledge-base query approach for visual questions requiring external commonsense knowledge. The approach scales question answering by mapping questions to queries and exposes the supporting fact used to produce each answer.

  • VI. CONCLUSION: FVQA extends visual question answering with a supporting fact that is critical for answering each visual question.The supporting fact is drawn from large-scale structured knowledge bases.
  • VI. CONCLUSION: The approach learns mappings from questions to KB queries rather than directly mapping questions to answers.This is presented as more scalable to answer diversity.
  • VI. CONCLUSION: The method provides both an answer and the supporting fact on which the answer is based.This makes the reasoning process visible.

Method

The experiments organize accuracy results by knowledge-base source and visual-concept type, alongside an overall method comparison on FVQA testing splits.

  • Knowledge-base sources: Table VIII reports accuracies for questions based on different knowledge-base sources.The table compares methods across those source-specific question subsets.
  • Visual concepts: Table IX reports accuracies for questions focusing on object, scene, and action visual concepts.The table compares methods across three visual-concept categories.
  • Answer sources: Table X reports overall accuracies for different methods according to answer source.Best single-model results are highlighted, while ground-truth question-query mappings are excluded from rankings.
Loading 1606.05433v4…