Source-linked AI summary
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Huijuan Xu, Kate Saenko
TL;DR
VQA requires joint image–language understanding, while prior captioning-based approaches did not explicitly model spatial inference. The paper proposes a multi-hop Spatial Memory Network with spatial attention, and reports improved results on DAQUAR and VQA compared with previously published models and iBOWIMG.
Problem
VQA requires answering questions about photographs through joint image and language understanding, but recent captioning-based approaches failed to model spatial inference.
Method
SMem-VQA stores activations from different image regions in memory, aligns question words with image patches, and gathers evidence through repeated attention hops.
Results
2.27% on test-dev and 2.35% on test-standard: SMem-VQA Two-Hop improved over iBOWIMG on VQA.
Takeaways & Limitations
Attention visualizations show learned spatial inference steps, while evaluations on DAQUAR and VQA showed improved results over previously published models.
Abstract
from arXiv · showhide
We address the problem of Visual Question Answering (VQA), which requires joint image and language understanding to answer a question about a given photograph. Recent approaches have applied deep image captioning methods based on convolutional-recurrent networks to this problem, but have failed to model spatial inference. To remedy this, we propose a model we call the Spatial Memory Network and apply it to the VQA task. Memory networks are recurrent neural networks with an explicit attention mechanism that selects certain parts of the information stored in memory. Our Spatial Memory Network stores neuron activations from different spatial regions of the image in its memory, and uses the question to choose relevant regions for computing the answer, a process of which constitutes a single "hop" in the network. We propose a novel spatial attention architecture that aligns words with image patches in the first hop, and obtain improved results by adding a second attention hop which considers the whole question to choose visual evidence based on the results of the first hop. To better understand the inference process learned by the network, we design synthetic questions that specifically require spatial inference and visualize the attention weights. We evaluate our model on two published visual question answering datasets, DAQUAR [1] and VQA [2], and obtain improved results compared to a strong deep baseline model (iBOWIMG) which concatenates image and question features to predict the answer [3].
1 Introduction
VQA requires joint image–language understanding and spatial inference, but prior captioning-based models lacked explicit object-position reasoning. The paper introduces SMem-VQA, a multi-hop spatial memory architecture evaluated with visualizations and benchmark experiments.
- VQA combines computer vision, natural language processing, and artificial intelligence to answer questions about images.The task emphasizes inference over image content and is easier to evaluate than image captioning.
- The first hop aligns individual question-word embeddings with image regions, while later hops gather additional evidence using the question and prior attended regions.The architecture supports multi-step inference and visualizes evidence through attention weights.
- Existing models lack an explicit notion of object position and intermediate computation based on spatial attention.Many questions require locating objects and comparing their contents or positions.
- SMem-VQA stores convolutional outputs from different image regions in memory and uses attention to select relevant spatial evidence.The model adapts end-to-end memory networks to a setting where the question and visual memory come from different modalities.
- The paper uses synthetic spatial questions to analyze learned inference rules and evaluates the model on DAQUAR and VQA datasets.The introduction describes extensive evaluation against existing models on publicly available datasets.
2 Related work
Prior VQA research developed from handcrafted semantic and scene-analysis systems toward end-to-end neural models, while attention and memory networks supplied relevant architectural foundations. The paper positions its approach against captioning-derived VQA systems and the strong iBOWIMG baseline.
- Memory networks use external memory and recurrent attention to select information for multi-step question answering.The end-to-end memory network is described as requiring less supervision than earlier competitive memory networks.
- Soft attention computes weighted combinations of candidate memory items using alignment functions based on concatenation or dot products.
- End-to-end VQA models commonly adapted image-captioning architectures using LSTMs conditioned on question and CNN image features.These approaches learned features directly from data but had limited performance according to the related-work discussion.
- The iBOWIMG baseline concatenates image features with a bag-of-words question representation and feeds them to a softmax classifier.The paper compares SMem-VQA with iBOWIMG and other VQA models having comparable or better results.
3 Spatial Memory Network for VQA
SMem-VQA uses a multi-hop memory architecture that stores spatial CNN features and uses question-guided attention to select visual evidence for answering VQA questions. Its first hop aligns individual question words with image regions, while a second hop uses the updated whole-question representation to gather additional evidence.
- Network overview: SMem-VQA processes question words and spatial CNN features, embedding them into a shared semantic space for attention and evidence selection.The network uses separate attention and evidence embeddings before combining visual evidence with the question representation for answer prediction.
- First-hop attention: The first hop computes word-region correlations so each image location receives attention from its most correlated question word.This word-guided mechanism is intended to provide finer-grained attention than a bag-of-words representation alone.
- Evidence and prediction: Attention-weighted evidence features are summed and combined with the question embedding to predict the answer.The evidence transformation detects semantic concepts or objects, while the attention weights select where to collect that evidence.
- Second-hop inference: A second hop uses the updated question vector to extract additional visual features based on the whole question rather than individual words.The first hop provides fine-grained local evidence, whereas the second hop considers global evidence from the question representation.
- Multi-hop prediction: The final prediction combines the question representation with local first-hop and global second-hop visual evidence, and additional hops are possible.The complete network is differentiable and jointly optimized through stochastic-gradient backpropagation.
4 Experiments
The experiments test whether SMem-VQA learns spatial inference using synthetic tasks and evaluate its performance and attention behavior on DAQUAR and VQA. Two-hop spatial attention improves VQA results over iBOWIMG, while visualizations show how the model selects supplementary evidence.
- Synthetic spatial inference: Synthetic experiments test whether attention supports absolute and relative spatial inference beyond question-only prediction.The study uses questions designed so the answer depends on an object's location or its position relative to another object.
- Relative position recognition: 96% test accuracy on relative-position questions, compared with around 75% for iBOWIMG and around 70% for an absolute-position baseline.Attention focuses on locations around the cat, while evidence activations identify both the cat and red square.
- Results on DAQUAR: On DAQUAR, both SMem-VQA One-Hop and Two-Hop outperform IMG+BOW and the other reported baseline models.The paper evaluates using 0-1 accuracy and highlights the model's ability to visualize the inference process.
- Results on VQA: On VQA, SMem-VQA Two-Hop improves over iBOWIMG by 2.27% on test-dev and 2.35% on test-standard.The Two-Hop model also achieves the best per-answer category accuracy, while adding a third hop does not improve results further.
- Attention visualizations: Two-hop attention collects supplementary evidence, complementing first-hop word-level visual alignment with global visual evidence from the whole question.Correlation vectors indicate how strongly each question word relates to the image features at the selected location.
5 Conclusion
The paper introduces SMem-VQA, combining memory networks with spatial attention for VQA, and evaluates its learned spatial reasoning on synthetic and benchmark questions. It reports improved results and makes inference steps inspectable through attention visualizations.
- SMem-VQA adapts a memory network architecture with spatial attention to visual question answering.
- Synthetic spatial questions and attention visualizations demonstrate that the model learns inference rules based on spatial attention.
- Evaluation on the DAQUAR and VQA datasets shows improved results over previously published models.
- The model provides insight into its processing by visualizing the inference steps learned by the deep network.