Source-linked AI summary
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, Wei Xu
TL;DR
The paper addresses visual question answering for freestyle questions and variable-length answers. It proposes the mQA model and the FM-IQA multilingual dataset, then evaluates generated answers with human judges. In 64.7% of cases, judges treated model answers as human answers, while model limitations remained in commonsense reasoning and small or similar objects.
Problem
Freestyle answers can be complete sentences with many valid alternatives, making automatic metrics difficult to apply accurately.
Method
The paper combines a four-component mQA model with the FM-IQA dataset of freestyle Chinese question-answer pairs and English translations.
Results
64.7% of mQA answers were treated as human answers in a Visual Turing Test.
Takeaways & Limitations
FM-IQA supports evaluation of freestyle multilingual visual question answering and can also support tasks such as visual machine translation.
Takeaways & Limitations
The model sometimes fails on incorrect commonsense reasoning, very small or visually similar target objects, and questions requiring high-level everyday-life reasoning.
Abstract
from arXiv · showhide
In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: http://idl.baidu.com/FM-IQA.html
1 Introduction
The paper targets freestyle visual question answering, where answers can address specific image content rather than merely describe an image generically. It proposes mQA, combining question, image, and answer representations, and evaluates it with human judges.
- Image captioning provides generic descriptions, whereas visual question answering addresses particular objects or parts of an image.
- mQA uses an LSTM for the question, a CNN for the image, an answer-context LSTM, and a fusing component to predict each answer word.
- FM-IQA contains 158,392 images, 316,193 Chinese question-answer pairs, and corresponding English translations.
- 64.7% of model answers were treated as human answers in a Visual Turing Test, with an average score of 1.454.
2 Related Work
Prior visual question answering work commonly used restricted or predefined questions and smaller datasets. This paper instead combines a larger freestyle multilingual dataset with an mQA architecture that separately models questions and answers.
- Recent vision and language research relies heavily on CNNs, RNNs, LSTMs, and large image datasets with sentence annotations.
- Earlier visual question answering studies often used predefined or template-generated questions.
- FM-IQA is larger than cited earlier datasets, which contained only 2,591 and 1,449 images.
- The paper also contributes English translations and evaluates generated answers with human judges rather than relying only on automatic metrics.
- Unlike compared models using a single LSTM, mQA uses separate LSTMs for questions and answers while sharing word embeddings.
3 The Multimodal QA (mQA) Model
The mQA model encodes a question, image, and evolving answer context, then fuses these representations to generate the next answer word. Its design shares selected word-embedding parameters while fixing the pretrained CNN.
- 3.1 The Four Components of the mQA Model: The model has four components: question LSTM, image CNN, answer-context LSTM, and a fusing component that generates the next answer word.
- 3.1 The Four Components of the mQA Model: The question LSTM represents the question using word embeddings and stores sequence context without feeding the image into this component.
- 3.1 The Four Components of the mQA Model: A pretrained GoogleNet supplies the image representation after its final SoftMax layer is removed.
- 3.1 The Four Components of the mQA Model: The fusing layer combines question-state, image, answer-state, and current-word representations before Softmax predicts the next answer word.
- 3.2 The Weight Sharing Strategy: The word-embedding matrix is shared between question and answer LSTMs and tied transposed to Softmax, reducing nearly half the parameters.
- 3.3 Training: The CNN is fixed during QA training, while the trainable components use log-likelihood optimization for ground-truth answer sequences.
4 The Freestyle Multilingual Image Question Answering (FM-IQA) Dataset
FM-IQA is a large multilingual dataset built from MS COCO with freely authored questions and answers. Its annotations span visual recognition, spatial relations, interactions, attributes, and commonsense reasoning.
- 4.1 The Data Collection: FM-IQA starts from 158,392 MS COCO images and collects question-answer annotations through online crowdsourcing.
- 4.1 The Data Collection: Annotators may ask any image-related question answerable through visual content and commonsense, producing freestyle and diversified questions.
- 4.1 The Data Collection: Quality monitoring selects 195 annotators whose questions are relevant and answers correct before they label the remaining images.
- 4.2 The Statistics of the Dataset: The dataset contains 316,193 Chinese question-answer pairs with English translations, and each image has at least two annotations.
- 4.2 The Statistics of the Dataset: A random sample of 1,000 question-answer pairs and corresponding images forms the test set.
- 4.2 The Statistics of the Dataset: Questions cover object actions, classes, positions, interactions, attributes, and higher-level reasoning based on vision, language, and commonsense.
- 4.2 The Statistics of the Dataset: Answers may be a single word, a phrase, or a complete sentence.
5 Experiments
The experiments evaluate mQA with human-based tests and answer-quality scores, then compare model variants. mQA is treated as human-generated in 64.7% of cases, while component variants perform worse than the complete model.
- Evaluation: Freestyle answers make automatic metrics difficult because multiple answers can be correct and keyword importance is not adequately represented.The paper therefore uses human judges for evaluation.
- The Visual Turing Test: 64.7% of mQA answers are treated by human judges as answers provided by a human in the Visual Turing Test.The evaluation mixes model-generated answers with answers labeled by human annotators.
- The Visual Turing Test: The Visual Turing Test has passing-rate standard deviations of 0.013 for human, 0.019 for blind-mQA, and 0.024 for mQA across judge groups.The authors report this as evidence that the evaluation is stable and reliable for the task.
- The Score of the Generated Answer: Among answers not rated perfectly correct, over half are partially correct; score-evaluation standard deviations are 0.020 for human and 0.041 for mQA.The three judge groups assign the same score in 88.3% of human cases and 83.9% of mQA cases.
- Performance Comparisons of the Different mQA Variants: All three mQA variants perform worse than the complete mQA model on word error rates and losses.The variants modify question embedding, LSTM parameter sharing, or Transposed Weight Sharing.
6 Discussion
The mQA model answers freestyle image questions and is evaluated on the FM-IQA dataset, while failure cases expose challenges in visual grounding and commonsense reasoning.
- Evaluation: 64.7% of mQA answers were treated as human answers in a real Turing Test.The evaluation mixed model-generated and annotator-provided answers and used human judges.
- Dataset: The FM-IQA dataset contains over 310,000 question-answer pairs and supports multilingual freestyle image question answering.The dataset includes Chinese question-answer pairs and corresponding English translations.
- Model: The mQA model can generate a sentence or phrase as an answer to a freestyle image question.A modified multimodal LSTM can also generate freestyle questions about image content.
- Failure cases: Failure cases arise from incorrect commonsense reasoning, very small or visually similar target objects, and questions requiring high-level everyday-life reasoning.One example produces an OOV sign when the needed word is absent from the model's dictionary.
- Future work: Future work targets these issues by incorporating more visual and linguistic information, including object detection or attention models.These additions are proposed as ways to address the observed failure cases.