Source-linked AI summary
Stacked Attention Networks for Image Question Answering
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Smola
TL;DR
Image QA models often fail on fine-grained regions, while many questions require multiple reasoning steps. SANs repeatedly query image regions with a semantic question representation, refining attention progressively. Across four benchmarks, the proposed SANs significantly outperform previous state-of-the-art approaches, while error analysis identifies wrong-region focus, wrong answers, ambiguity, and label errors.
Problem
Existing image QA models often struggle with answers tied to fine-grained image regions, and answering many questions requires multi-step reasoning over multiple objects and relationships.
Method
SANs use a semantic question vector to query regional image representations through multiple attention layers, refining the query and attention distribution before predicting the answer.
Results
SANs significantly outperform previous state-of-the-art approaches across all four image QA data sets, with two-layer models achieving the best reported results across data sets and question models.
Takeaways & Limitations
Layer-by-layer attention visualization shows SANs progressively focus on relevant visual clues, supporting the use of repeated image querying for image QA reasoning.
Takeaways & Limitations
In sampled COCO-QA errors, 42% involved the right region but a wrong answer, while 31% were ambiguous cases and 5% reflected clearly wrong labels.
Abstract
from arXiv · showhide
This paper presents stacked attention networks (SANs) that learn to answer natural language questions from images. SANs use semantic representation of a question as query to search for the regions in an image that are related to the answer. We argue that image question answering (QA) often requires multiple steps of reasoning. Thus, we develop a multiple-layer SAN in which we query an image multiple times to infer the answer progressively. Experiments conducted on four image QA data sets demonstrate that the proposed SANs significantly outperform previous state-of-the-art approaches. The visualization of the attention layers illustrates the progress that the SAN locates the relevant visual clues that lead to the answer of the question layer-by-layer.
1. Introduction
Image QA models often struggle with answers tied to fine-grained image regions and questions requiring multi-step reasoning. SANs address this by repeatedly attending to image regions using a question representation, progressively refining the evidence used for prediction.
- Existing neural image QA models typically combine global CNN image features with LSTM question features, limiting precision for fine-grained regional answers.
- Answering questions involving relationships among multiple objects requires locating referenced objects, eliminating irrelevant regions, and identifying the visual evidence most indicative of the answer.
- SANs introduce stacked attention networks that support multi-step reasoning for image question answering.
- The model extracts regional image representations and a semantic question vector, then uses the question to query image regions through successive attention layers.Each refined query combines the question representation with retrieved image features before another image query.
- Comprehensive evaluations on four image QA benchmarks report that multiple-layer SANs outperform previous state-of-the-art approaches by a substantial margin.The study also visualizes attention outputs to show progressive focus on relevant visual clues.
2. Related Work
Prior image QA work used parsing, segmentation, encoder-decoder models, and other neural architectures, while attention had been successful mainly in related image captioning. SANs adapt attention to image QA and extend it with multiple layers for multi-step reasoning.
- Image captioning methods evolved from global image features and recurrent generation toward attention-based caption generation and candidate reranking.
- Image QA differs from captioning because the question is given and the model must learn relevant visual and textual representations to infer an answer.
- Earlier image QA models used semantic parsing and segmentation or encoder-decoder networks that encoded images and questions before decoding answers.
- Before SANs, attention had not been explored for image QA despite its success in image captioning.
- SANs extend attention-based approaches with multiple attention layers that support multi-step reasoning in image QA.
3. Stacked Attention Networks (SANs)
SANs combine regional CNN image features with CNN- or LSTM-based question representations, then repeatedly attend to image regions using progressively refined queries. This design targets fine-grained, multi-step visual reasoning rather than relying on a single global image vector.
- Architecture: SAN comprises an image model, a question model, and a stacked attention model that locates question-relevant regions for answer prediction.The image model uses CNN features, while the question model uses either CNN or LSTM representations.
- Image model: The image model extracts spatially organized CNN features, with each vector corresponding to an image region.The cited implementation uses a 512×14×14 feature map, representing 14×14 regions.
- Question model: The question model represents the question using either the final hidden state of an LSTM or pooled convolutional features from unigram, bigram, and trigram filters.For the LSTM model, the final hidden state is used as vQ; the CNN model concatenates max-pooled features across filter sizes.
- Stacked attention: In an example, the first attention layer identifies regions related to the basket, bicycle, and sitting, while the second focuses more sharply on the dogs that answer the question.The example illustrates progressive localization of relevant visual clues across attention layers.
- Stacked attention: SAN first uses the question representation to generate an attention distribution over image regions and computes a weighted visual feature.The attention distribution assigns probabilities to regions conditioned on the question vector.
- Stacked attention: Each attention layer combines the retrieved visual feature with the preceding query to create a refined query for the next image search.The initial query is the question vector, and the process repeats K times before answer inference.
4. Experiments
The experiments evaluate SANs across four image QA datasets using accuracy and WUPS-based metrics, with two-layer SANs consistently outperforming one-layer models and prior baselines. Analyses also examine question types, attention progression, and error categories.
- Experimental setup: The study evaluates SANs on DAQUAR-ALL, DAQUAR-REDUCED, COCO-QA, and VQA, comparing them with recently proposed image QA baselines.Image QA is formulated as classification, with classification accuracy, WUPS0.9, and WUPS0.0 used across the datasets; VQA additionally uses a human-label matching metric.
- Overall results: Two-layer SANs achieve the best results across all four datasets, while LSTM- and CNN-based question models provide similar performance.On DAQUAR-ALL, the two proposed two-layer SANs exceed the two strongest baselines by 5.9% and 7.6% absolute accuracy.
- Overall results: 6.5%, 10.4%, 11.5%, and 33.5% absolute accuracy gains are reported on DAQUAR-REDUCED over IMG-CNN, 2-VIS+BLSTM, Ask-Your-Neurons, and Multi-World, respectively.On COCO-QA, two-layer SANs improve accuracy over the cited best baselines by 5.1% and 6.6%.
- Overall results: 4.8% absolute accuracy improvement is obtained by SAN(2, CNN) over the LSTM Q+I baseline on VQA.The authors identify VQA as the largest of the four datasets and report this comparison for the official-server results.
- Question-type analysis: Question-type gains are largest for Color on COCO-QA and Other on VQA, whereas VQA Yes/No performance improves only slightly.SAN(2, CNN) improves COCO-QA Color by 7.2% and VQA Other by 9.7% over the respective best baselines.
- Attention visualization: Attention visualizations show scattered first-layer attention becoming more focused in the second layer on regions leading to the correct answer.For the horn-color example, attention shifts from recognizing the woman to focusing on her head, identifying the red horn.
- Error analysis: Among 100 sampled COCO-QA mistakes, 22% involve wrong regions, 42% wrong answers despite correct regions, 31% ambiguous labels, and 5% clearly wrong labels.The error analysis distinguishes localization, prediction, answer ambiguity, and annotation problems.
5. Conclusion
The paper introduces SANs for progressively locating relevant visual regions through multiple attention queries, and reports substantial gains over prior approaches on four image QA datasets.
- SANs use a multiple-layer attention mechanism to query an image repeatedly, progressively locating relevant visual regions and inferring answers.
- Experimental results show that SANs significantly outperform previous state-of-the-art approaches on all four image QA datasets.
- Attention-layer visualizations illustrate how the model progressively focuses on relevant visual clues leading to an answer.