Source-linked AI summary
Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
Mateusz Malinowski, Marcus Rohrbach, Mario Fritz
TL;DR
The paper addresses question answering on real-world images, where answers must be conditioned on both visual and language input. It proposes Neural-Image-QA, an end-to-end CNN-LSTM model trained jointly to generate answers, and reports doubled performance over prior work while analyzing language-only behavior and human consensus.
Problem
Question answering on real-world images requires jointly interpreting image content and question meaning, creating a multimodal challenge for existing approaches.
Method
Neural-Image-QA combines a CNN image representation with an LSTM that receives the image and question and generates answer words end-to-end.
Results
Neural-Image-QA doubles prior performance, while a language-only variant slightly trails the full model and outperforms a human baseline under the same condition.
Takeaways & Limitations
The paper extends neural image-description methods to question answering and uses additional human answers and metrics to analyze consensus and ambiguity.
Takeaways & Limitations
Spatial relations, small objects, negations, and shapes remain challenging, with insufficient training data potentially contributing to errors.
Abstract
from arXiv · showhide
We address a question answering task on real-world images that is set up as a Visual Turing Test. By combining latest advances in image representation and natural language processing, we propose Neural-Image-QA, an end-to-end formulation to this problem for which all parts are trained jointly. In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question). Our approach Neural-Image-QA doubles the performance of the previous best approach on this problem. We provide additional insights into the problem by analyzing how much information is contained only in the language part for which we provide a new human baseline. To study human consensus, which is related to the ambiguities inherent in this challenging task, we propose two novel metrics and collect additional answers which extends the original DAQUAR dataset to DAQUAR-Consensus.
1. Introduction
The paper targets question answering on real-world images by jointly interpreting visual and language input. Neural-Image-QA combines CNN and LSTM components in an end-to-end model trained to generate answers.
- Neural-Image-QA addresses question answering on real-world images by combining question-intent inference, visual scene understanding, and word-sequence prediction.
- Answers depend on jointly interpreting the question's inferred meaning and the image content, unlike approaches conditioned on only one modality.
- The model analyzes images with a CNN and feeds the visual representation and question into an LSTM to generate the answer.
- CNN and LSTM parameters are trained jointly and end-to-end from words and pixels to produce correct answers.
- The approach reports doubled performance over prior work while also introducing human-consensus analyses, metrics, and additional reference answers.
2. Related Work
Related work spans CNNs for visual recognition, LSTMs for sequence modeling, and their combination for visual description. The paper extends these neural ideas to question answering conditioned jointly on images and language.
- Neural architectures: CNNs and LSTMs provide the visual-recognition and variable-length sequence-modeling foundations used by recent neural approaches.
- Visual-language generation: Prior CNN-RNN systems generate image or video descriptions from visual features, whereas this work generates answers from visual and natural language input.
- Grounding: Language grounding associates words with meaning linked to sensory input, but this paper leaves representations of meaning latent and learns them jointly.
- Question answering: Textual question answering commonly uses semantic parsing to obtain logical forms, while recent neural methods formulate related tasks with recurrent models or classification.
- Visual Turing Test: Visual Turing Test systems include binary synthetic-data tasks and semantic-parser approaches, while Neural-Image-QA uses an end-to-end neural architecture without ontological commitment.
3. Approach
Neural-Image-QA models answer prediction as a variable-length sequence-generation problem conditioned on an image and question. A CNN supplies visual features to an LSTM, which jointly learns to generate answer words.
- Problem formulation: The model predicts an answer a from image x and question q using a parametric probability measure p(a|x, q; θ).
- Answer generation: Multiple-word answers are generated recursively from vocabulary V, with an added end token terminating the sequence.
- Answer generation: At each test step, previously predicted answer words are appended to the question representation and the process stops when the end token is predicted.
- Multimodal encoding: The image is encoded by a CNN and provided at every time step alongside the question and prior answers as LSTM input.
- LSTM architecture: The LSTM maintains a memory cell in addition to its hidden state, helping learn long-term dynamics and reducing vanishing and exploding gradients.
- Training: All network weights and biases are learned jointly with cross-entropy loss, while loss excludes predictions made before the question mark.
4. Experiments
The experiments evaluate Neural-Image-QA on DAQUAR using accuracy and WUPS, including reduced datasets and answer-length subsets. WUPS incorporates word-level ambiguity through thresholded taxonomy-based Wu-Palmer similarity.
- The study compares model variants with prior work and analyzes language-only performance, human guessing, and ambiguity-sensitive evaluation.
- DAQUAR contains 12,468 human question-answer pairs on indoor-scene images, evaluated with accuracy and WUPS at 0.9 and 0.0.
- Experiments cover the full dataset, a reduced 37-category set with 25 test images, and subsets containing 1–4-word answers.
- WUPS generalizes accuracy by accounting for word-level ambiguities such as synonymous or closely related answer concepts.The metric uses thresholded taxonomy-based Wu-Palmer similarity; smaller thresholds are more forgiving.
4.1. Evaluation of Neural-Image-QA
Neural-Image-QA is evaluated against prior work on full and reduced DAQUAR, with additional comparisons across answer lengths and single-word training. The model improves substantially over prior work, although performance declines for longer answers and remains below human accuracy.
- Over 9% points in accuracy and over 11% in WUPS, Neural-Image-QA improves on prior work for multiple-word answers on full DAQUAR.The comparison disadvantages Neural-Image-QA because the published prior result uses ground-truth object annotations.
- Single-word training performs best for one-word answers, benefiting from a dataset in which about 90% of answers contain one word.The authors attribute the improvement to joint language-visual training and dataset bias.
- Accuracy drops rapidly as answer length increases, but Neural-Image-QA produces a significant number of correct two-word answers.Figure 4 evaluates multi-word models on 1-, 2-, 3-, and 4-word subsets and compares them with single-word variants on one-word answers.
- A 30% points gap to human accuracy and a 25-point WUPS gap remains on full DAQUAR.
- 34.68% Accuracy and 40.76% WUPS at 0.9 are achieved on reduced DAQUAR, compared with 21.95% Accuracy and 22.6 WUPS for prior work.The reduced set contains 37 object classes, 25 images, and 297 test question-answer pairs.
4.2. Answering questions without looking at images
The study tests how much answer information is available from questions alone by removing visual input and comparing language-only models with vision-based models and humans.
- 17.15% and 32.32% are achieved by the best language-only models on the full and reduced sets, respectively.
- The best vision-based models reach 19.43% and 34.68% accuracy on the full and reduced sets, respectively.
- Humans guess the correct answer in 7.86% of cases without images, while the best language-only model exceeds this baseline by over 9%.The authors associate these guesses with prior knowledge, common sense, and plausible dataset answers.
4.3. Human Consensus
The authors extend DAQUAR with multiple human answers to measure ambiguity and agreement, then introduce consensus-sensitive evaluation metrics. Agreement varies across questions, and models perform substantially better on subsets with stronger human consensus.
- DAQUAR-Consensus: DAQUAR-Consensus collects an average of 5 test answers per image-question pair without imposing additional answer restrictions.The extension enables analysis of inter-human agreement and ambiguity.
- Consensus measures: Average Consensus Metric (ACM) evaluates expected agreement across multiple human answers, weighting answers by their similarity to the generated answer.As the number of human references approaches the total number of humans, ACM measures inter-human agreement for each question.
- Consensus measures: Min Consensus Metric (MCM) replaces averaging with a maximum, rewarding agreement with at least one human answer.The max operator selects the human answer closest to the prediction, representing a minimal form of consensus.
- Human agreement: Human answers agree on many questions, but partial agreement is most common and complete disagreement remains possible.The analysis uses whole-data and test-data consensus categories: 0%, at least 50%, and 100%.
- Consensus results: On stronger-agreement subsets, model accuracy gains reach 10% over the full set and 20% over the no-agreement subset.These subsets act as curated versions of DAQUAR with ambiguities factored out.
- Consensus results: MCM improves human-baseline evaluation and, combined with WUPS at 0.9, yields an approximately 20% higher score.The measure is intended to capture ambiguity in interpreting questions.
4.4. Qualitative results
Qualitative examples compare Neural-Image-QA with language-only predictions and include human-answer consensus results. The examples expose both plausible language-based guesses and differences between single-word and multiple-word variants.
- Qualitative comparisons: Tables 7–9 show predictions from architecture variants, emphasizing differences between Neural-Image-QA and the Language only model.The single-word model is used except in Table 8, which presents the multiple-words approach.
- Qualitative comparisons: Language only can make reasonable guesses from linguistic patterns, such as predicting table for the largest object or pillow or doll for an object on a bed.The examples are chosen to highlight qualitative differences rather than only successful predictions.
- Human-answer evaluation: Table 6 evaluates Min and Average Consensus on human answers using additional DAQUAR-Consensus answers as references.The table reports consensus scores in percent.
4.5. Failure cases
The model handles many question types but struggles with spatial relations and several less-represented cases. Qualitative failures include occlusion, uncovered possible answers, and unusual objects.
- Performance limitations: Spatial relations remain challenging at approximately 21 WUPS at 0.9 despite approximately 35 WUPS at 0.9 for color and counting subsets.Spatial questions constitute a substantial part of DAQUAR.
- Performance limitations: Questions involving small objects, negations, and shapes score below 12 WUPS at 0.9.The authors suggest that too few training examples may contribute to these errors.
- Qualitative failures: Displayed failure cases include strong occlusion, possible answers absent from the ground truth, and unusual instances such as a red toaster.These cases expose limitations in visual evidence and answer coverage.
5. Conclusions
The paper presents Neural-Image-QA as a neural architecture that outperforms prior work while revealing language-only biases and limitations in visual representation. It also extends DAQUAR to support analysis of human consensus and disagreement.
- Neural-Image-QA doubles prior performance on the challenging image question-answering task.
- The language-only variant performs only slightly worse than the image-conditioned model and exceeds a new human no-image baseline.This suggests that dataset biases and prior knowledge provide useful answer cues without visual input.
- Global CNN representations do not adequately capture indoor scene statistics, spatial reasoning, or small objects.The paper notes that larger datasets are needed to explore these representation limits fully.
- DAQUAR-Consensus adds multiple reference answers and introduces Average Consensus and Min Consensus to measure human disagreement.The extension provides an average of five test answers per image-question pair.
- A Supplementary Material
The supplementary material compares Neural-Image-QA with language-only variants through qualitative examples, correlation analysis, and failure cases. It shows that visual content improves average answers, although both models struggle with longer questions and multi-word responses.
- Qualitative examples highlight challenges and differences between Neural-Image-QA and Language only architectures.The supplement also includes failure cases in Table 9.
- Language only can make reasonable guesses by exploiting dataset biases that resemble common-sense knowledge.Examples include stereotyped associations between objects and locations.
- Neural-Image-QA predicts better average answers by exploiting visual image content.The supplement illustrates this with cases where Language only repeats an answer seen for the same question on another image.
- The correlation plot compares question-predicted-answer word correlations for Language only against corresponding human-answer correlations.
- Both models struggle with long questions and questions requiring larger numbers of answer words.They generally predict answer types correctly, such as colors for color questions and numbers for counting questions.