Source-linked AI summary
Hierarchical Question-Image Co-Attention for Visual Question Answering
Jiasen Lu, Jianwei Yang, Dhruv Batra, Devi Parikh
TL;DR
VQA attention models typically identify relevant image regions but do not model which question words matter. This paper introduces hierarchical co-attention over image and question representations, improving performance across VQA and COCO-QA benchmarks.
Problem
Prior VQA attention models focused on visual attention, leaving question attention as an unmodeled aspect of understanding image-question pairs.
Method
The model jointly attends to image regions and question fragments at word, phrase, and question levels, using hierarchical representations and co-attention.
Results
The model improves state-of-the-art accuracy from 60.4% to 62.1% on open-ended VQA, from 64.2% to 66.1% on multiple-choice VQA, and from 61.6% to 65.4% on COCO-QA.
Takeaways & Limitations
Ablation studies show that co-attention and question hierarchy contribute to performance, with question-level attention producing the largest reported accuracy drop when removed.
Abstract
from arXiv · showhide
A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.
1 Introduction
The paper argues that VQA requires attention to both relevant image regions and question words. It introduces co-attention and a three-level question hierarchy to jointly represent image-question information for answer prediction.
- Visual attention highlights image regions relevant to answering a question, but prior VQA attention models do not model which question words matter.The paper motivates question attention using linguistically varied questions with the same meaning.
- Co-attention jointly reasons about visual attention and question attention, using image representations to guide question attention and question representations to guide image attention.
- The model represents questions hierarchically at word, phrase, and question levels, constructing image-question co-attention maps at each level.
- 1-dimensional CNNs capture unigram, bigram, and trigram information at the phrase level, while recurrent neural networks encode the entire question.
- Co-attended image and question features are recursively combined from word level to question level for final answer prediction.
2 Related Work
Prior VQA work primarily developed visual attention mechanisms, while question attention had not been explored in VQA. The paper situates its co-attention approach against visual-attention models and related language-attention methods.
- Image attention: Earlier VQA models used image attention to select relevant spatial regions, sometimes through stacked or multi-hop attention, object proposals, or memory-based mechanisms.
- Image attention: Existing VQA attention approaches modeled visual attention alone, while sequential attention mechanisms could suffer from error propagation.
- Language Attention: Although language-attention methods addressed alignment and fixed-vector bottlenecks in NLP, prior VQA work had not explored question attention.
3 Method
The method hierarchically represents questions and jointly attends to image regions and question words at word, phrase, and question levels. It offers parallel and alternating co-attention mechanisms, recursively combining attended features for answer prediction.
- Question Hierarchy: The question is encoded hierarchically at word, phrase, and question levels before co-attention is applied at each level.Words are embedded, phrase features use 1-D convolutions over unigrams, bigrams, and trigrams, and an LSTM encodes the pooled sequence.
- Question Hierarchy: Max-pooling across n-gram responses produces phrase-level features while preserving sequence length and order through adaptive selection at each time step.The pooling operation combines convolution responses from different n-gram sizes at each word location.
- Co-Attention: Parallel co-attention simultaneously generates image and question attention maps from an affinity matrix connecting every image location with every question position.The learned attention maps yield weighted image and question feature vectors, and the mechanism operates at all three hierarchy levels.
- Co-Attention: Alternating co-attention summarizes the question, attends to the image using that summary, and then attends to the question using the attended image feature.This sequence is repeated at each level of the question hierarchy.
- Answer Prediction: An MLP recursively combines co-attended image and question features from all three levels to predict answer probabilities.VQA is treated as a classification task, with concatenated attention features encoded recursively for final answer prediction.
4 Experiment
The experiments evaluate the hierarchical co-attention model on VQA and COCO-QA, comparing attention strategies, feature backbones, and component ablations. The model improves reported state-of-the-art results, while ablations and visualizations examine the roles of hierarchical attention and multimodal alignment.
- Datasets and Evaluation: The model is evaluated on the VQA and COCO-QA datasets using benchmark comparisons and component ablations.VQA includes open-ended and multiple-choice settings, while COCO-QA reports classification accuracy and WUPS.
- VQA Results: 62.1% open-ended VQA accuracy is achieved by Oursa+ResNet, improving on the 60.4% DMN+ baseline.On multiple-choice VQA, Oursa+ResNet reaches 66.1%, compared with 64.2% for FDA.
- COCO-QA Results: 65.4% COCO-QA accuracy is achieved by Oursa+ResNet, improving on the 61.6% SAN(2,CNN) state-of-the-art result.Parallel co-attention performs better than alternating co-attention in this setup.
- Ablation Study: Removing question-level, phrase-level, and word-level co-attention reduces accuracy by 1.7%, 0.3%, and 0.2%, respectively.The reported ordering places question-level attention as the largest contributor among the three hierarchy levels.
- Qualitative Results: The qualitative visualizations show co-attention maps at word, phrase, and question levels, with attention scaled from red for high to blue for low.The model can localize key question phrases and corresponding image regions, such as “what color” with relevant objects.
5 Conclusion
The paper concludes that hierarchical co-attention supports VQA by attending to image regions and question fragments across three levels, with ablations and visualizations supporting the model’s components and interpretability.
- The proposed hierarchical co-attention model attends to different image regions and question fragments for visual question answering.
- Question representations are modeled at word, phrase, and question levels to capture information at different granularities.
- Ablation studies demonstrate the roles of co-attention and question hierarchy in the model’s final performance.
- Visualizations show interpretable co-attention to image regions and question fragments during answer prediction.
- The model was evaluated on visual question answering and may also apply to other vision-and-language tasks.