Source-linked AI summary
Explicit Knowledge-based Reasoning for Visual Question Answering
Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, Anthony Dick
TL;DR
Most visual question answering methods lack explicit reasoning, explanations, and the capacity to encode the world knowledge needed for complex questions. This paper combines image-derived concepts with a large external knowledge base to answer and explain such questions, substantially outperforming the predominant approach in evaluation.
Problem
Most VQA methods connect CNN image analysis directly to LSTM text processing without explicit reasoning or explanations, while their capacity to encode broad prior knowledge is limited.
Method
The method detects visual concepts, links them to relevant knowledge-base entities, queries structured relationships, and generates answers with explanations based on those connections.
Results
Ahab substantially outperforms the predominant VQA approach when tested, achieving 88.2% accuracy on AnimalRelative questions and outperforming humans on some knowledge-based question types.
Takeaways & Limitations
The approach supports complex questions requiring information absent from images, including relationships between images and taxonomic relatedness between depicted animals.
Takeaways & Limitations
The KB-VQA dataset is still relatively small and does not support training large models; the authors plan to extend it.
Abstract
from arXiv · showhide
We describe a method for visual question answering which is capable of reasoning about contents of an image on the basis of information extracted from a large-scale knowledge base. The method not only answers natural language questions using concepts not contained in the image, but can provide an explanation of the reasoning by which it developed its answer. The method is capable of answering far more complex questions than the predominant long short-term memory-based approach, and outperforms it significantly in the testing. We also provide a dataset and a protocol by which to evaluate such methods, thus addressing one of the key issues in general visual ques- tion answering.
1 Introduction
Visual question answering combines image understanding and natural-language processing, but predominant CNN–LSTM methods mainly handle simple image-grounded questions without explicit reasoning or explanations. Ahab addresses these limits by linking detected visual concepts to large-scale knowledge bases, enabling broader questions and a dataset designed to evaluate external-knowledge reasoning.
- Most VQA methods connect CNN image analysis directly to LSTM text processing without performing explicit reasoning.
- CNN–LSTM systems answer simple questions about image content effectively, such as color and counting questions.
- LSTM-based VQA does not explain answers and cannot practically encode the millions of concepts and relationships needed for broad world knowledge.
- Ahab detects image content, links it to a knowledge base, converts natural-language questions into queries, and supports multiple reasoning steps before producing an answer.
- External knowledge enables questions about relationships between images and taxonomic relatedness between depicted animals.
- Ahab uses DBpedia, requires no VQA training data, provides reasoning explanations, and targets a broader range of image questions than competing methods.
- The KB-VQA dataset labels human-generated questions as Visual, Common-sense, or KB-knowledge and requires more external knowledge than earlier VQA datasets.
2 The KB-VQA Dataset
KB-VQA is designed to evaluate visual question answering that requires visual understanding, common sense, and external knowledge. Its 2,402 questions span predefined templates, three knowledge levels, and concept phrases largely uncommon in COCO-VQA.
- KB-VQA evaluates algorithms on higher-knowledge questions requiring explicit reasoning about image contents with external information.
- Templates: Five questioners generated 3 to 5 question-answer pairs per image by instantiating 23 predefined templates.
- Questions: Questions are labeled Visual, Common-sense, or KB-knowledge according to the estimated information required to answer them.Visual questions use image concepts; Common-sense questions should not require external sources; KB-knowledge questions are expected to require Wikipedia or similar resources.
- Data analysis: 1,256 Visual, 883 Common-sense, and 263 KB-knowledge questions were collected, so around half require external knowledge.IsImgRelate and IsThereAny cover almost half of Common-sense questions, while 18 templates occur among KB-knowledge questions with a more balanced distribution.
- Data analysis: 254 concept phrases appeared in externally informed questions, with 55% mentioned fewer than 20 times in COCO-VQA and over 85% of 67 KB-knowledge phrases equally rare.Examples absent from COCO-VQA include “logistics”, “herbivorous animal”, “animal-powered vehicle”, “road infrastructure” and “portable electronics”.
- Data analysis: Compared with other VQA datasets, KB-VQA contains a larger proportion of questions requiring external knowledge; COCO-VQA has 5.5% adult-level common-sense questions and no observed KB-knowledge questions.DAQUAR is almost exclusively visual, while TORONTO-QA questions are generated from image captions describing visible content.
3 The Ahab VQA approach
Ahab constructs a combined image–knowledge-base representation, parses natural-language questions into templates and queries, and generates answers with logical reasons from discovered paths.
- 3.1 RDF Graph Construction: Ahab detects objects, scenes, and attributes, then links these visual concepts to relevant DBpedia entities.The detectors select 224 object classes, the top three scene scores, and the top ten attributes per image.
- 3.1 RDF Graph Construction: The resulting RDF graph connects image entities to DBpedia concepts through same-concept links and represents relationships as triples.Each arrow corresponds to a triple, with visual concepts linked to DBpedia entities sharing the same semantic meaning.
- 3.2 Answering Questions: Natural-language questions are tagged and matched against regular-expression templates to extract slot-phrases for knowledge-base queries.Quepy uses tokenization, part-of-speech tagging, lemmatization, and flexible regex matching before forwarding extracted phrases.
- 3.2 Answering Questions: Ahab uses transitive categories, Wikilink counts, and infobox predicates to answer different question templates.Hyponymy and shared categories support several templates, while correlation and food-ingredient questions use Wikilinks or infobox relations.
- 3.2 Answering Questions: Search paths from visual concepts to knowledge-base concepts provide logical reasons explaining how answers involving external knowledge are generated.The predicates and entities along each path expose how relationships between image concepts and knowledge-base concepts were established.
4 Experiments
Ahab is evaluated against an LSTM baseline and human reference using human-scored accuracy and answer correctness across question types and knowledge levels. It outperforms LSTM broadly, especially as external-knowledge requirements increase, while also producing mostly valid reasons for a subset of questions.
- Evaluation: A human evaluation protocol scores answers from 1 (“Totally wrong”) to 5 (“Perfect”), treating scores above “Borderline” as correct.Five examiners evaluate answers double-blind because open-ended questions and generated reasons are poorly handled by existing automated metrics.
- Question types: 88.2% accuracy on AnimalRelative questions exceeds human performance, and Ahab also outperforms humans on ListSameYear.LSTM answers for AnimalRelative and Prop questions were all marked wrong by examiners.
- Knowledge levels: Ahab’s advantage over LSTM grows with external-knowledge requirements: LSTM declines as knowledge increases, whereas Ahab improves across Visual, Common-sense, and KB-knowledge levels.Ahab outperforms LSTM at all three knowledge levels, with the largest gap on questions requiring external knowledge.
- Reason accuracy: More than 80% of Ahab’s generated reasons are judged correct among the 45% of questions for which the system provides reasons.Visual questions do not receive coded reasons because they can be answered by directly interrogating pixels.
- Examples and reasoning: The two-image examples use shared transitive categories or visual-concept correlation scores to answer relationship and image-comparison questions.For example, railway station and airport images share the DBpedia concept “transport infrastructure.”
5 Conclusion
Ahab combines structured image representations with external knowledge to answer and explain a wide range of visual questions. The paper also contributes evaluation resources and reports substantially better performance than the predominant approach.
- 5 Conclusion: Ahab reasons about general image content using a structured representation augmented with information from a large external knowledge base.Its reasoning can be explained through knowledge-base entities and the connections between them.
- 5 Conclusion: Ahab can use any knowledge base with a SPARQL interface, including datasets covering domains such as taxonomy, music, and government statistics.Multiple knowledge bases can also be linked through common identifiers to form larger repositories.
- 5 Conclusion: Ahab can draw sensible general conclusions from images when a knowledge base containing common sense is available.
- 5 Conclusion: The paper provides a dataset and evaluation methodology for general visual question answering.
- 5 Conclusion: Ahab substantially outperforms the predominant visual question answering approach in the reported evaluation.
A Mapping ⟨concept⟩Phrases to KB-Entities with Redirections
Ahab maps natural-language concept phrases to knowledge-base entities by matching labels, then uses redirections to accommodate alternate expressions such as abbreviations and synonyms.
- A Mapping ⟨concept⟩Phrases to KB-Entities with Redirections: Ahab searches knowledge-base entities by matching a natural-language phrase to entity labels.For “Religion,” the shown SPARQL query selects entities whose English label matches the phrase.
- A Mapping ⟨concept⟩Phrases to KB-Entities with Redirections: The label query can return multiple entities for one phrase, including category and non-category entities.The example reports two entities matched to “Religion.”
- A Mapping ⟨concept⟩Phrases to KB-Entities with Redirections: Natural-language expressions vary through synonyms, capitalization, punctuation, tense, abbreviation, and misspelling.The paper gives “Frisbee” and “Flying disc” as expressions for the same object.
- A Mapping ⟨concept⟩Phrases to KB-Entities with Redirections: DBpedia redirection entities map alternate concept expressions to a concrete entity.This mechanism allows an abbreviation such as “Relig.” to locate KB:Religion through wikiPageRedirects.
- A Mapping ⟨concept⟩Phrases to KB-Entities with Redirections: Using redirections significantly enriches the vocabulary of concept phrases.
B Query Generation and Post-processing
Ahab generates question-specific SPARQL queries over image and knowledge-base representations, then applies template-specific processing to answer diverse question types.
- B Query Generation and Post-processing: Ahab generates queries tailored to question templates after mapping slot phrases to knowledge-base entities.Some templates additionally require post-processing.
- B Query Generation and Post-processing: ASK queries return whether a pattern has a solution, whereas SELECT queries return variables from all matched solutions.Variables begin with ?, while other terms represent fixed entities or predicates.
- B Query Generation and Post-processing: Image scene information is obtained from detected attributes or scenes, with COCO-trained attributes given higher priority.Separate queries retrieve scene-related attributes and scene names.
- B Query Generation and Post-processing: The query routines support boolean, counting, selection, relational, and comparison questions over detected objects and knowledge-base categories.Examples include checking category membership, finding the largest qualifying object, counting matches, and returning common categories.
- B Query Generation and Post-processing: Transitive category searches are limited to three steps to avoid reaching overly general concepts.
- B Query Generation and Post-processing: The MostRelObj scoring function combines direct or categorical relations with indirect WikiLink connectivity, weighting the first component more heavily.The experiments set α to 50.
C Evaluation Protocol
The evaluation uses a double-blind interface showing one question, one image, and three anonymized answers. Examiners rate correctness and logical reasoning on a five-point scale.
- C Evaluation Protocol: The evaluation presents a question, image, and three answers generated by LSTM, Ahab, and another human subject.Answer sources are withheld from examiners in a double-blind process.
- C Evaluation Protocol: Examiners assign each answer a correctness score from 1 to 5.The interface requires only one score for each evaluation item.
- C Evaluation Protocol: Scores range from totally wrong at 1 to perfect at 5, with intermediate categories for slightly wrong, borderline, and OK answers.
- C Evaluation Protocol: Overall accuracy for a question type is the number of answers with correctness greater than 3 divided by the total number of questions.The same rule is applied to logical-reasoning correctness.
D Visual Concepts
Ahab detects visual concepts across objects, attributes, and scenes, using class vocabularies assembled from established datasets and prior work.
- Ahab detects three visual-concept types: objects, attributes, and scenes.
- Scene classes in Ahab follow the MIT Places205 definition.
- Ahab’s object vocabulary merges 80 MS COCO classes with 200 ImageNet classes.
- Ahab uses a 147-attribute vocabulary trained in prior work.
E Examples
The paper presents examples of Ahab’s KB-VQA questions, generated answers, and reasons, alongside the RDF entities, predicates, and visual-concept inventories used by the system.
- Figure 11 shows KB-VQA questions, Ahab-generated answers, and reasons for those answers.
- Ahab uses RDF entities and predicates defined either by Ahab or by DBpedia.
- Table 4 lists the 147 image attributes used as visual concepts in Ahab.
- Table 5 lists the 224 object classes used as visual concepts in Ahab.