Source-linked AI summary
Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, Anton van den Hengel
TL;DR
The paper addresses VQA questions requiring information not contained in the image by fusing image-derived text with external knowledge. It feeds the combined representation and question to an LSTM, achieving the best reported results on two large-scale VQA datasets while remaining constrained by knowledge-base quality.
Problem
VQA questions can require external or common-sense information that is not present in the image.
Method
The model combines generated image captions and attribute information with text mined from an external knowledge base before VQA LSTM decoding.
Results
The system achieves the best reported results on Toronto COCO-QA and MS COCO-VQA, including 69.73% accuracy on Toronto COCO-QA.
Takeaways & Limitations
External knowledge can significantly improve VQA performance on questions requiring information beyond the image.
Takeaways & Limitations
Available knowledge bases are patchy and inconsistent or narrowly topic-specific, limiting the useful information that can be extracted.
Abstract
from arXiv · showhide
We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible. It particularly allows questions to be asked about the contents of an image, even when the image itself does not contain the whole answer. The method constructs a textual representation of the semantic content of an image, and merges it with textual information sourced from a knowledge base, to develop a deeper understanding of the scene viewed. Priming a recurrent neural network with this combined information, and the submitted question, leads to a very flexible visual question answering approach. We are specifically able to answer questions posed in natural language, that refer to information not contained in the image. We demonstrate the effectiveness of our model on two publicly available datasets, Toronto COCO-QA and MS COCO-VQA and show that it produces the best reported results in both cases.
1. Introduction
VQA must answer natural-language questions whose required operations and information are not fixed by the image alone. The paper combines image-derived textual representations with external knowledge and reports best results on two VQA benchmarks.
- VQA differs from detection and segmentation because both the question form and required operations are unknown at runtime.
- Questions may require information absent from the image, such as knowing that beach umbrellas provide shade on sunny days.
- The method fuses generated image captions with text mined from an external knowledge base, then uses the combined representation to answer general image questions.
- The approach combines knowledge-base generality with the LSTM's generality in question form.
- 69.73% accuracy on Toronto COCO-QA exceeds the latest reported 55.92% state of the art.
2. Related Work
Earlier VQA systems used Bayesian, parsing, segmentation, or CNN-RNN approaches, but knowledge-base integration remained limited. The paper emphasizes a text-based fusion of image information and external knowledge within an LSTM while noting that available knowledge bases are uneven and topic-specific.
- Prior VQA methods included Bayesian nearest-neighbor sampling, joint parse graphs, and CNN-RNN architectures for encoding questions and generating answers.
- Knowledge bases built from Wikipedia are patchy and inconsistent, while hand-curated bases are inevitably topic-specific.
- The proposed model accepts general question forms without manually specified formalization, relying on processing a suitable training set.
3. Extracting, Encoding, and Merging
The system converts image attributes, generated captions, and retrieved knowledge into vector representations, merges them, and conditions a VQA LSTM on the resulting scene representation and question. DBpedia supplies text for the detected attributes through SPARQL, while Doc2Vec encodes the retrieved passages.
- 3.1. Attribute-based Image Representation: The method represents image content as attributes extracted from human-written captions, including objects, actions, and properties.
- 3.1. Attribute-based Image Representation: A region-based multi-label CNN predicts attributes from an arbitrary number of image sub-region proposals.
- 3.2. Caption-based Image Representation: Five beam-search captions form the internal textual representation, whose final caption-LSTM hidden states are average-pooled into the 512-dimensional Vcap(I).
- 3.3. Relating to the Knowledge Base: The top five predicted attributes generate DBpedia SPARQL queries that retrieve informative comment text.
- 3.3. Relating to the Knowledge Base: Doc2Vec encodes the retrieved text into a fixed-length knowledge vector after the five returned paragraphs are combined.
4. A VQA Model with Multiple Inputs
The model generates multi-word answers by encoding image information and the question with an encoder LSTM, then decoding the answer sequence with a decoder LSTM. Training jointly learns embedding and recurrent parameters by minimizing a regularized sequence-generation cost.
- The VQA model maximizes the probability of the correct answer given an image and question by treating answering as word-sequence generation.Questions and answers are represented as sequences, with an END token marking answer completion.
- An encoder LSTM integrates semantic image information and the question, while a decoder LSTM generates the answer; their weights are shared.The decoder conditions each generated word on the image, question, and previously generated answer words.
- During training, the question and answer are concatenated, and each word is represented as a one-hot vector over the word dictionary.The decoder runs through the answer sequence and uses the END token as the final target.
- The initial input combines learned embeddings for image attributes, captions, and external knowledge.The embedding weights for these three information sources are learned jointly in practice.
- The training objective jointly learns the embedding and LSTM parameters by minimizing a cost function with a regularization term.The regularizer is λθ · ||θ||2.
5. Experiments
The experiments evaluate the proposed models on Toronto COCO-QA and VQA, comparing attribute, caption, and external-knowledge representations against baselines and prior methods. The combined model achieves the strongest reported performance, while category results indicate that external knowledge is especially useful for questions requiring general knowledge.
- Datasets and evaluation: The experiments use Toronto COCO-QA and VQA, both based on MS COCO images, with Toronto questions covering object, number, color, and location categories.Toronto COCO-QA contains 78,736 training and 38,948 testing examples generated from 117,684 images.
- Models and evaluation: The final Att+Cap+Know-LSTM combines predicted attributes, image-caption representations, and external knowledge, while ablations isolate each information source.Att+Cap-LSTM, Att+Know-LSTM, and Cap+Know-LSTM evaluate the contributions of internal textual representation and external knowledge.
- Toronto COCO-QA results: All proposed models outperform the baseline and competing methods on Toronto COCO-QA, with Att+Cap+Know-LSTM achieving the best results.The final model surpasses the baseline by nearly 20% and previous state-of-the-art methods by around 15%.
- VQA results: 11% overall improvement over VggNet-LSTM is achieved by the final A+C+K-LSTM model on the VQA validation set.Adding image captions or external knowledge improves the attribute-based model, while the combined model performs best.
- Per-category results: For Why questions, Att+Know-LSTM reaches 11.88% versus 7.77% for Att-LSTM, while A+C+K-LSTM reaches 13.53%.The paper identifies Why questions as requiring general knowledge, unlike Number and Color questions that can be answered directly.
6. Conclusion
The paper extends RNN-based visual question answering with external knowledge to address general, open-ended questions about images. Despite limitations in available knowledge bases, the approach improves performance on questions requiring external knowledge and achieves the best results on two large-scale VQA datasets.
- The approach extends state-of-the-art RNN-based VQA to incorporate external information for general, open-ended image questions.The method is intended to support questions whose answers are not fully contained in the image.
- Available knowledge bases lack much beneficial information, but they still significantly improve performance on questions requiring external knowledge.The paper specifically identifies ‘Why’ questions as an example of this category.
- The system performs best on two large-scale VQA datasets and shows promising results on the VQA evaluation server.
- Future work includes generating knowledge-base queries conditioned on both the question and the image.The goal is to extract information that is more specifically related to each query.
- The authors argue that successful general VQA will require access to a large external knowledge base.They connect this requirement with deeper image-content analysis and common-sense reasoning.