Source-linked AI summary
Learning to Answer Questions From Image Using Convolutional Neural Network
Lin Ma, Zhengdong Lu, Hang Li
TL;DR
Image QA requires answers that jointly reflect complex images and specific questions, yet CNNs had not been studied for this task. The paper introduces an end-to-end CNN framework that learns image, question, and inter-modal representations, and reports performance surpassing the state of the art on public image QA datasets.
Problem
Image QA demands detailed, joint understanding of image content and questions, while CNNs had not previously been studied for this task.
Method
An end-to-end model uses an image CNN, sentence CNN, and multimodal convolution layer to learn image-question representations and their interactions for answer prediction.
Results
The proposed CNN surpasses state-of-the-art methods on public image QA datasets, including DAQUAR and COCO-QA.
Takeaways & Limitations
Convolutional architectures can model image content, question representations, and their inter-modal interactions within one jointly learned image QA framework.
Abstract
from arXiv · showhide
In this paper, we propose to employ the convolutional neural network (CNN) for the image question answering (QA). Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations, but also their inter-modal interactions to produce the answer. More specifically, our model consists of three CNNs: one image CNN to encode the image content, one sentence CNN to compose the words of the question, and one multimodal convolution layer to learn their joint representation for the classification in the space of candidate answer words. We demonstrate the efficacy of our proposed model on the DAQUAR and COCO-QA datasets, which are two benchmark datasets for the image QA, with the performances significantly outperforming the state-of-the-art.
Introduction
Image question answering requires joint understanding of complex image content and specific questions, but CNNs had not been studied for this task. The paper proposes an end-to-end CNN model that learns image, question, and inter-modal representations, outperforming the state of the art on public datasets.
- Motivation: Image QA requires answers conditioned on both the image and question, including object identification, comparison, and counting.The task involves more image-language interactions than automatic image captioning and requires detailed understanding of potentially non-salient image regions.
- Research gap: CNNs had been widely used for image and sentence representations, but their ability for image QA had not been studied.The paper positions image QA as an unaddressed application of CNN-based multimodal representation learning.
- Contribution: The paper proposes an end-to-end CNN model trained on image-question-answer triplets to answer free-form, natural-language-like questions.The model is designed to learn the answer-prediction function directly from paired image, question, and answer data.
- Results: The proposed CNN surpasses the state of the art on public image QA datasets.The introduction reports this outcome without specifying dataset-level metric values in the supplied passages.
- Contribution: The model uses convolutional architectures to encode image content, represent the question, and learn their interactions jointly for answer prediction.Its architecture explicitly targets both unimodal representations and the relations between the image and question.
Related Work
Prior image QA methods include semantic parsing, image segmentation, sequence generation, and multimodal classification approaches. The paper argues that treating the image as a single appended word cannot capture complicated image-question relations, motivating convolutional architectures for multimodal fusion.
- Task context: Image QA emerged alongside the visual Turing test as a task involving questions and answers about real-world images.The visual Turing test asks judges to distinguish machine-generated answers from human answers, while image QA focuses on producing the answer.
- Prior approaches: Earlier approaches used semantic parsing and image segmentation, or formulated image QA as a question-answer generation problem.The related work describes both structured multi-world reasoning and deep neural network generation-based formulations.
- Prior approaches: Neural-Image-QA appends the image representation to the question as an independent word before learning a joint representation for classification.This approach incorporates image information into the question sequence at the beginning or ending position.
- Motivation for the proposed model: The paper argues that appending the image as one word cannot effectively exploit complicated image-question relations, motivating end-to-end convolutional architectures.The proposed approach is intended to capture inter-modal relationships as well as separate image and question representations.
Proposed CNN for Image QA
The proposed image-QA framework uses CNNs to represent the image and question, model their interactions, and predict an answer end to end. Its multimodal convolution layer combines image and question features before softmax classification.
- The task predicts answer a from question q and related image I, with answers selected from the set Ω.
- The framework couples an image CNN, sentence CNN, multimodal convolution layer, and softmax layer, jointly learning their parameters end to end.The image CNN encodes image content, the sentence CNN generates the question representation, and the multimodal layer generates their joint representation.
- Image CNN: The image CNN maps image content to a fixed-length representation and projects its 4096-dimensional feature vector into a smaller dimension d with nonlinear activation.The projection reduces parameters for multimodal fusion and is described as enhancing nonlinear modeling capacity.
- Sentence CNN: The sentence CNN applies convolution and max-pooling over word embeddings to compose local word relations into a whole-question representation.Three convolution and max-pooling layers are used; convolution windows cover three semantic components and share parameters across the question.
- Multimodal Convolution Layer: The multimodal convolution treats the image representation as a semantic component alongside two consecutive question components to capture cross-modal interactions.The resulting multimodal representation is passed to softmax to produce the answer for the image-question pair.
- Multimodal Convolution Layer: Compared with appending the image as an individual LSTM word, the CNN design is intended to preserve interactions between the image and high-level multiword question semantics.The paper uses “the red bicycle” as an example of a high-level semantic representation that may interact with the image.
Experiments
The experiments evaluate the proposed CNN model on public image QA datasets and analyze its results.
- Experiments: Experiments evaluate the proposed CNN model on public image QA datasets.The paper introduces dataset descriptions, evaluation measurements, and experimental results as the evaluation sequence.
Configurations and Training
The model uses a fixed sentence-CNN architecture and is trained end to end with mini-batch stochastic gradient descent and regularization.
- Sentence CNN: The sentence CNN uses three convolution and max-pooling layers with 300, 400, and 400 feature maps.Questions are limited to 38 words, and 50-dimensional skip-gram embeddings initialize the input.
- Training: Training uses mini-batch stochastic gradient descent with negative log likelihood, tuning all model parameters and word embeddings.Mini batches contain 100 examples.
- Training: Dropout with probability 0.1 is used to prevent overfitting.
Image QA Datasets
The evaluation uses DAQUAR and COCO-QA, which vary in size, image coverage, question types, and answer format.
- DAQUAR-All: DAQUAR-All contains 6,795 training and 5,673 testing samples from 795 and 654 images, respectively.It covers 894 object categories and includes object, color, and counting questions with single- or multiword answers.
- DAQUAR-Reduced: DAQUAR-Reduced contains 3,876 training and 297 testing samples across 37 object categories.Only 25 images are used to generate testing samples, and answers may contain one or multiple words.
- COCO-QA: COCO-QA contains 79,100 training and 39,171 testing samples generated from about 8,000 and 4,000 images.Its questions cover object, number, color, and location categories.
Evaluation Measurements
Image QA performance is measured primarily with accuracy and WUPS, with results organized in dataset-specific tables.
- Metrics: Accuracy measures the proportion of correctly answered testing questions.
- Metrics: WUPS measures word similarity using common subsequences in a taxonomy tree.
- Reported results: Table 1 reports image QA performances on DAQUAR-All.
- Metrics: WUPS@0.0 and WUPS@0.9 use threshold parameters 0.0 and 0.9, respectively.These thresholds match those used in prior work.
Experimental Results and Analysis
Experiments across DAQUAR and COCO-QA compare image-question models and analyze the contributions of image content, sentence composition, and multimodal interactions. The proposed CNN achieves strong performance by jointly modeling these components.
- The proposed CNN significantly outperforms the multi-world approach and Neural-Image-QA on DAQUAR-All.
- 2-VIS+BLSTM outperforms VIS+LSTM by a large margin on DAQUAR-Reduced, with the same observation reported for COCO-QA.
- The proposed CNN outperforms all competitor models on COCO-QA across accuracy, WUPS@0.9, and WUPS@0.0.
- Removing multimodal convolution still outperforms IMG+BOW, VIS+LSTM, and 2-VIS+BLSTM in accuracy, attributed mainly to sentence CNN composition.
- Without image representation, the proposed model’s performance significantly drops; sentence CNN also performs better than LSTM and BOW for question representation.
- Randomly reshuffled questions on COCO-QA achieve 40.74 accuracy, 53.06 WUPS@0.9, and 80.41 WUPS@0.0.
Conclusion
The paper concludes that its CNN model addresses image QA by jointly generating image and question representations and modeling their interactions. Experiments demonstrate superiority over state-of-the-art methods.
- The proposed CNN uses convolutional architectures to represent the image, compose the question, and learn their interactions for answer prediction.
- Experimental results on public image QA datasets demonstrate superiority over state-of-the-art methods.