PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Captioning and Visual Question Answering Based on Attributes and External Knowledge

Qi Wu, Chunhua Shen, Anton van den Hengel, Peng Wang, Anthony Dick

arXiv:1603.02814v2cs.CV

TL;DR

The paper addresses the limitation of direct CNN-RNN Vision-to-Language models that omit explicit high-level concepts and may lack information needed for general visual questions. It adds an attribute representation and, for VQA, combines image content with captions and external knowledge; the resulting models achieve best-reported performance on several captioning and VQA benchmarks.

  • Problem

    Direct CNN-RNN Vision-to-Language models do not explicitly represent high-level concepts, while VQA can require information absent from the image.

  • Method

    The paper inserts CNN-predicted semantic attributes into an RNN model and feeds VQA with attributes, generated captions, and knowledge-base information.

  • Results

    The models achieve best-reported results on several image-captioning and VQA benchmarks, including 70.98% accuracy on Toronto COCO-QA versus 61.60% for the latest state of the art.

  • Takeaways & Limitations

    Explicit image-content attributes improve V2L performance, and external knowledge can significantly improve questions requiring information beyond the image.

  • Takeaways & Limitations

    The available knowledge bases contain limited beneficial information and are patchy and inconsistent, motivating improved knowledge bases and question- and image-specific queries.

Abstract

from arXiv · show

Much recent progress in Vision-to-Language problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. In this paper we first propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering. We further show that the same mechanism can be used to incorporate external knowledge, which is critically important for answering high level visual questions. Specifically, we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. It particularly allows questions to be asked about the contents of an image, even when the image itself does not contain a complete answer. Our final model achieves the best reported results on both image captioning and visual question answering on several benchmark datasets.

1 INTRODUCTION

Vision-to-Language systems commonly map CNN image features directly to text, but visual questions may require information beyond the image. This paper inserts human-meaningful attributes into CNN-RNN models and adds mined external knowledge for VQA.

  • CNN-RNN systems had become state-of-the-art for image captioning and visual question answering without explicitly modeling high-level image concepts.
  • VQA accepts an image and a free-form question at runtime, making it more complex than captioning and sometimes requiring common sense or subject-specific knowledge.
  • The proposed attribute-based model predicts semantic attributes with CNN classifiers, then trains an RNN to generate captions or answers from their likelihoods.
  • The VQA extension combines predicted attributes, generated captions, and paragraphs mined from an RDF knowledge base before encoding them with an LSTM.
  • 70.98% accuracy on Toronto COCO-QA exceeded the latest state of the art at 61.60%, while the model also produced a 59.50% state-of-the-art result on the VQA evaluation server.
  • The paper adds experiments on two VQA datasets, additional ablations, and the A+C+Selected-K-LSTM model, which leads to a new VQA state-of-the-art result.

2 RELATED WORK

Earlier V2L work used retrieval, templates, visual concepts, or direct CNN-RNN mappings. This paper positions its framework as a trainable attribute-based CNN-RNN system that also incorporates general-purpose external knowledge.

  • Earlier image annotation methods framed captioning as ranking or retrieval, while other approaches co-embedded images and sentences.
  • Several captioning systems used detected attributes, objects, modifiers, locations, or parsed scenes to construct descriptions through templates or language models.
  • The dominant CNN-RNN approach learned image-to-sentence mappings directly, offering end-to-end training and varied captions while bypassing an intermediate high-level representation.
  • The paper introduces a high-level representation within CNN-RNN models and investigates its value across Vision-to-Language tasks.
  • Unlike prior VQA systems that restricted question forms, the framework uses image content, generated captions, and mined external knowledge to answer questions with an RNN.
  • Instead of a problem-specific knowledge base, the method queries the larger, more general DBpedia using RDF query language and processes natural-language questions through training.
  • The captioning framework maps images to semantic attributes with a CNN and maps the attribute vector to word sequences with an LSTM.

3 IMAGE CAPTIONING USING ATTRIBUTES

The model inserts an explicit, human-meaningful attribute representation between CNN image analysis and LSTM caption generation. Attributes are learned from captions, predicted with a multi-label CNN over image regions, and used to generate sentences.

  • 3.1 Attribute-based Image Representation: The image-analysis stage predicts caption-derived semantic attributes as a multi-label classification problem, producing the fixed-length vector Vatt(I).Each vector dimension contains the prediction probability for one attribute.
  • 3.1 Attribute-based Image Representation: Attribute vocabulary entries are mined directly from training captions and may include nouns, verbs, and adjectives describing image content.This avoids requiring separate hand-labeled attribute data.
  • 3.1 Attribute-based Image Representation: The attribute CNN uses an ImageNet-pretrained VggNet, fine-tunes it on image-attribute data, and aggregates proposal-region outputs with cross-hypothesis max pooling.The whole image is included alongside selected proposal hypotheses.
  • 3.2 Caption Generation Model: The caption generator is an LSTM trained to maximize the probability of the correct description conditioned on Vatt(I), rather than direct CNN image features.The LSTM predicts each next word from the attribute vector and preceding words.
  • 3.2 Caption Generation Model: Training unrolls the LSTM over start, sentence, and END tokens, with word dictionaries formed from words occurring at least five times in each training set.The supplied passage reports dictionary sizes of 2538, 7414, and 8791 for three datasets.
  • 3.2 Caption Generation Model: At inference, beam search retains the five best partial sentences, and Figure 4 presents examples pairing predicted attributes with generated captions.The examples include baseball-related attributes and captions.

4 A VQA MODEL WITH EXTERNAL KNOWLEDGE

The VQA model combines attribute, internally generated caption, and external knowledge representations within an LSTM framework. External knowledge is retrieved from DBpedia, encoded semantically, and selected according to the question before answer generation.

  • 4.3 An Answer Generation Model with Multiple Inputs: The model combines attribute-based image features, caption representations, and knowledge-base embeddings into a single input for the VQA LSTM.Caption-LSTM hidden states are average-pooled, while mined knowledge is encoded with Doc2Vec.
  • 4.1 Relating to the Knowledge Base: DBpedia queries use the five most strongly predicted image attributes and retrieve generally informative comment text through SPARQL.The approach can also use problem-specific knowledge bases or common-sense databases.
  • 4.2 Encoding External Knowledge: Doc2Vec converts variable-length SPARQL-returned text into fixed-length semantic representations for integration with the VQA model.The retrieved text is typically longer than generated captions, motivating paragraph-level semantic encoding.
  • 4.2 Encoding External Knowledge: Question-guided selection ranks mined knowledge paragraphs by cosine similarity and combines the k closest paragraphs before re-encoding them.This selection is intended to reduce irrelevant knowledge, such as bird information for a question about a dog.
  • 4.3 An Answer Generation Model with Multiple Inputs: The VQA objective maximizes the probability of the correct answer, while training learns embedding, word, and LSTM parameters with regularized optimization.The implementation uses mini-batch SGD, gradient clipping, and dropout.

5 EXPERIMENTS

Experiments evaluate the attribute-based CNN–RNN framework for image captioning and VQA, including ablations of captions, external knowledge, and question-guided selection. Across several benchmarks, the proposed models improve over CNN–RNN baselines, with the selected-knowledge model achieving leading VQA results.

  • Evaluation setup: The experiments compare attribute-based models against CNN–RNN baselines across Flickr8k, Flickr30k, MS COCO, and four VQA datasets.Captioning uses BLEU and perplexity, with METEOR and CIDEr additionally reported for MS COCO.
  • Image captioning: An intermediate image-to-word attribute prediction stage brings significant improvements over directly using CNN representations for captioning.The baseline replaces the attribute vector with the CNN’s last hidden layer.
  • Image captioning: 0.73 on B-1 was achieved on the COCO test set, while the model surpassed human performances on 13 of 14 reported metrics.Human evaluation also found the proposed model superior to the VggNet+LSTM baseline on both reported measures.
  • VQA results: The proposed VQA variants outperform the baseline, and A+C+Selected-K-LSTM achieves the best result on the DAQURA-Reduced set.The selected-knowledge model outperforms the unfiltered Att+Cap+Know-LSTM variant, supporting question-based knowledge selection.
  • VQA results: On Toronto COCO-QA, A+C+Selected-K-LSTM surpasses the baseline by nearly 20% and previous state-of-the-art methods by around 10%.Caption representations outperform knowledge-only additions on this automatically caption-derived benchmark.
  • VQA results: On the VQA validation set, A+C+S-K-LSTM outperforms the VggNet-LSTM baseline by 11%, while adding captions or external knowledge further improves performance.The attribute-based Att-LSTM also improves over the VggNet+LSTM baseline.
  • Question categories: For why questions, Att+Know-LSTM rises from 7.77% to 11.88%, while A+C+S-K-LSTM reaches 13.76%.The results associate external knowledge with gains on question types requiring common-sense information, especially why and where questions.

6 CONCLUSIONS

The paper finds that explicit attributes improve vision-to-language performance across image captioning and visual question answering, while external knowledge further supports general visual questions. Its knowledge-based VQA system reaches state-of-the-art results, but future gains depend on more targeted queries and improved knowledge bases.

  • Explicit attribute prediction improves vision-to-language performance in all tested cases.The attribute layer is inserted into the CNN-LSTM framework as an intermediate representation of image content.
  • The image captioning model outperforms the state-of-the-art on several captioning datasets.
  • External knowledge can significantly improve VQA performance on questions requiring information beyond the image, including “Why” questions.The approach extends RNN-based VQA to incorporate information from knowledge bases and uses knowledge selection based on both question and image content.
  • The VQA system achieves state-of-the-art results on three VQA datasets and the best results on the VQA evaluation server.
  • Future work should generate knowledge-base queries reflecting both question and image content and improve the knowledge base itself.The paper notes that broader common-sense knowledge could help answer high-level questions.
Loading 1603.02814v2…