Source-linked AI summary
Exploring Models and Data for Image Question Answering
Mengye Ren, Ryan Kiros, Richard Zemel
TL;DR
Image question answering requires models that jointly use visual and textual information, while available datasets and evaluation settings are limited. The paper develops end-to-end neural models and automatically generates QA pairs from image descriptions. Its models outperform baselines and the existing approach on answer accuracy and WUPS, while the generated dataset is larger and more evenly distributed.
Problem
Image QA requires answering questions about image details, but existing data are limited and current approaches involve constrained datasets or intermediate vision stages.
Method
The paper develops end-to-end neural networks with visual semantic embeddings and converts image descriptions into question-answer pairs for dataset construction.
Results
The models outperform baselines and the existing approach in answer accuracy and WUPS, while the generated dataset provides more data and more evenly distributed answers.
Takeaways & Limitations
The work provides baseline comparisons and a larger generated dataset for developing end-to-end image question-answering systems.
Takeaways & Limitations
The models classify single-word answers, focus on a limited question domain, and are difficult to interpret.
Abstract
from arXiv · showhide
This work aims to address the problem of image-based question-answering (QA) with new models and datasets. In our work, we propose to use neural networks and visual semantic embeddings, without intermediate stages such as object detection and image segmentation, to predict answers to simple questions about images. Our model performs 1.8 times better than the only published results on an existing image QA dataset. We also present a question generation algorithm that converts image descriptions, which are widely available, into QA form. We used this algorithm to produce an order-of-magnitude larger dataset, with more evenly distributed answers. A suite of baseline results on this new dataset are also presented.
1 Introduction
The paper studies image question answering as joint image-text learning, requiring attention to image details and combining computer-vision subproblems. It contributes an end-to-end visual-semantic QA model and a generated dataset with baseline results.
- 1 Introduction: Image QA jointly learns from images and text while requiring attention to image details rather than vague image description.The task also combines subproblems such as image labeling and object detection.
- 1 Introduction: The paper presents an end-to-end QA model using visual semantic embeddings to connect a CNN and an RNN.It compares this model with other models.
- 1 Introduction: The paper introduces an algorithm that converts description sentences into questions and uses it to create the COCO-QA dataset with baseline results.The dataset is intended to provide more QA data with more evenly distributed answers.
- 1 Introduction: Answers are restricted to a single word, making image QA a classification problem with easier and more robust evaluation.This avoids evaluation issues associated with multi-word generation.
2 Related Work
Prior work established image QA datasets and models, but existing approaches used constrained settings and intermediate vision components. The paper positions its models and datasets against these limitations and related efforts.
- DAQUAR: DAQUAR provides indoor-scene images with question-answer pairs, segmentation, depth, and object-labeling information.Its questions cover object, color, and object-count categories, with 37-class and 894-class configurations.
- Examples: Figure 1 contrasts model responses on DAQUAR and COCO-QA, marking correct answers in green, incorrect answers in red, and showing top-answer probabilities.The leftmost example is from DAQUAR; the remaining examples are from COCO-QA.
- Prior Models: The published DAQUAR approach combines semantic parsing and image segmentation but depends on dataset-specific predicates, segmentation accuracy, and depth information.It also computes possible spatial relations in training images.
- Related Efforts: Related efforts used MS-COCO or other datasets, human-generated questions, and recurrent networks to encode questions and produce answers.The paper distinguishes its classification formulation from approaches producing sentence-form answers.
3 Proposed Methodology
The methodology combines neural image-question models with automatic QA generation from image descriptions. The proposed VIS+LSTM treats image features as sentence input, while the generation pipeline produces constrained object, number, and color questions for COCO-QA.
- Proposed Methodology: The methodology has two parts: neural networks and visual-semantic embeddings for modeling, plus QA synthesis from image descriptions for dataset construction.This separates the model-side and dataset-side contributions.
- 3.1 Models: VIS+LSTM treats the image as one word of the question and compares this recurrent model with simpler alternatives.It builds on an LSTM sentence model.
- 3.1 Models: The model maps frozen 4096-dimensional VGG image features into 300- or 500-dimensional word-embedding space before LSTM processing.The final LSTM outputs pass through a softmax layer to generate answers.
- 3.2 Question-Answer Generation: The existing DAQUAR dataset has approximately 1,500 images and 7,000 questions, limiting training of large models and making mode guessing strong.These limitations motivate a new dataset.
- 3.2 Question-Answer Generation: The proposed generator converts image descriptions into QA pairs to obtain more data and a more even answer distribution.The conversion preserves language variability and targets questions answerable through rough image understanding without logical reasoning.
- 3.2.1 Pre-Processing & Common Strategies: The generation pipeline uses syntactic parsing and transformations including sentence splitting, determiner changes, and constrained wh-movement.For example, it transforms declarative descriptions into questions beginning with interrogative words.
- 3.2.2 Question Generation: The algorithm generates object, number, and color questions using noun categories, extracted numbers, and color-adjective relations.It follows a conservative strategy intended to produce high-quality questions.
- 3.2 Question-Answer Generation: 24.98% to 7.30% was the reduction in frequency of the most common answer words in COCO-QA’s test set after answer rejection.Answers appearing too rarely or too often were rejected.
4 Experimental Results
Experiments evaluate several image-question answering models and baselines on COCO-QA and DAQUAR, finding gains from multimodal representations while exposing weaknesses in counting and spatial reasoning.
- Datasets: COCO-QA questions are easier for humans than DAQUAR, while QA-pair rejection makes mode-guessing perform poorly.The dataset is designed to encourage models to exploit salient object relations rather than exhaustively search all possible relations.
- Models and baselines: The experiments compare multimodal, blind, deaf, prior-based, nearest-neighbor, and ensemble models using answer accuracy and WUPS.The evaluated models include VIS+LSTM, 2-VIS+BLSTM, IMG+BOW, FULL, GUESS, BOW, LSTM, IMG, IMG+PRIOR, and K-NN.
- Object questions: The FULL model improves accuracy by 50% compared to IMG, demonstrating a difference between pure object classification and image question answering.IMG+BOW benefits from single-object recognition, whereas spatial relations among multiple objects and image details remain challenging.
- Counting: Counting remains weak: COCO-QA models sometimes count up to five or six in clean single-object images but often fail with multiple object types.No counting advantage over blind baselines was observed on DAQUAR, and the authors identify substantial room for improvement.
- Category results: On COCO-QA color questions, IMG+BOW and VIS+LSTM significantly outperform blind models, while prior knowledge immediately improves IMG accuracy on Color and Number questions.The models sometimes associate different colors with different objects, but still fail on some easy examples.
5 Conclusion and Current Directions
The paper presents end-to-end neural models for image QA, alongside baselines and an automated method for creating larger datasets from image descriptions. It reports useful but coarse image and question understanding, with important limitations in answer form, question scope, interpretability, and counting.
- The paper presents end-to-end neural network models for image question answering.
- A question generation algorithm converts image descriptions into large-scale image QA data without extensive human effort.
- The models show reasonable question understanding and coarse image understanding but remain naïve in many situations.
- The models are limited to single-word answer classification rather than longer free-form answers requiring text generation and automatic evaluation.
- The approach focuses on a limited question domain, and model predictions remain difficult to interpret.
A.1 Question Generation: Syntax Tree Example
The syntax-tree example illustrates how a declarative image description is transformed into an object question by replacing the object with “what” and moving the interrogative structure.
- “A man is riding a horse” becomes “What is the man riding?” through object replacement and wh-movement.
A.2 Post-Processing of COCO-QA Detail
COCO-QA post-processing filters rare answers and samples QA pairs using a probability that depends on current answer frequency, with constants selected for generation.
- Answers occurring below a frequency threshold are discarded before QA-pair enrollment.
- QA pairs are enrolled one at a time using a probability based on the current count of their ground-truth answer.
- COCO-QA generation uses K = 100 and K2 = 200, with K ≤ K2.
A.3 More Sample Questions and Responses
The examples show model responses across object, number, color, and location questions, including correct predictions, confidence scores, and characteristic counting and spatial errors.
- Object Questions: Object-question examples compare ground-truth answers with predictions from IMG+BOW, VIS+BLSTM, and BOW models.
- The examples cover object, number, color, and location question categories.
- Number Questions: The counting examples include correct, undercounted, and overcounted predictions across COCO-QA and DAQUAR.
- Location Questions: The location examples show both correct predictions, such as “pan” and “oven,” and incorrect alternatives such as “sink,” “chair,” and “kitchen.”