Source-linked AI summary
ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, Ram Nevatia
TL;DR
VQA requires models to connect natural-language questions with relevant regions of an image, a challenge that standard visual-question integration may not fully address. ABC-CNN learns question-guided attention by using question-derived configurable convolution over spatial image features, and it outperforms state-of-the-art methods across three benchmark datasets while producing semantically relevant attention maps.
Problem
VQA must integrate language semantics with image content, while existing feature integration may not fully exploit question intent to select relevant regions.
Method
ABC-CNN generates a question-guided attention map by convolving a spatial image feature map with a configurable kernel derived from question embeddings.
Results
ABC-CNN outperforms state-of-the-art methods on Toronto COCO-QA, DAQUAR, and VQA, and its attention maps reflect question-relevant regions.
Takeaways & Limitations
Question-guided attention improves VQA performance and helps reveal how question semantics and image contents are integrated.
Takeaways & Limitations
The study restricts VQA pairs to single-word answers, treating answer generation as multi-class classification.
Abstract
from arXiv · showhide
We propose a novel attention based deep learning architecture for visual question answering task (VQA). Given an image and an image related natural language question, VQA generates the natural language answer for the question. Generating the correct answers requires the model's attention to focus on the regions corresponding to the question, because different questions inquire about the attributes of different image regions. We introduce an attention based configurable convolutional neural network (ABC-CNN) to learn such question-guided attention. ABC-CNN determines an attention map for an image-question pair by convolving the image feature map with configurable convolutional kernels derived from the question's semantics. We evaluate the ABC-CNN architecture on three benchmark VQA datasets: Toronto COCO-QA, DAQUAR, and VQA dataset. ABC-CNN model achieves significant improvements over state-of-the-art methods on these datasets. The question-guided attention generated by ABC-CNN is also shown to reflect the regions that are highly relevant to the questions.
1. Introduction
VQA requires integrating image content with natural-language questions, but standard models may not fully exploit question intent to focus on relevant image regions. ABC-CNN addresses this with question-guided attention and improves performance across benchmark datasets.
- VQA answers natural-language questions about semantic content in images or videos and supports applications including image retrieval, early education, and navigation for blind people.
- VQA is challenging because systems must understand language, extract image semantics, and relate both modalities in one framework.
- Existing VQA models combine visual features and question embeddings, but this integration may lose the opportunity to use query intent to focus on differing image regions.
- ABC-CNN introduces question-guided attention that locates informative regions determined jointly by the image and the question.
- ABC-CNN extracts spatial image feature maps with CNNs, question embeddings with an LSTM, and attention through configurable convolution before answer generation.
- ABC-CNN significantly outperforms state-of-the-art methods on Toronto COCO-QA, DAQUAR, and VQA, while its attention maps reflect regions queried by questions.
2. Related Work
Related VQA and vision systems commonly combine CNN-based visual features with recurrent language processing and attention mechanisms. ABC-CNN adapts configurable convolution by deriving its kernels from question embeddings, making question semantics central to attention selection.
- VQA and image captioning both combine visual-content reasoning with natural-language generation, commonly using CNN features and LSTM-based decoders.
- Prior attention models estimate region importance for tasks including object recognition, detection, fine-grained classification, and image captioning.
- ABC-CNN’s framework includes CNN image features, LSTM question understanding, configurable-convolution attention extraction, and multi-class answer generation.
- Dynamic convolutional layers determine kernels from encoded input information; ABC-CNN instead configures convolutional kernels from question embeddings because questions identify relevant VQA attention regions.
3. Attention Based Configurable CNN
ABC-CNN integrates image and question representations through question-guided attention. A configurable convolutional kernel derived from question semantics identifies relevant image regions, which are weighted for single-word answer classification.
- Answer Generation: The answer generator is a softmax multi-class classifier that predicts the highest-probability word from an answer dictionary.The framework is trained end-to-end, and this paper restricts evaluation to question-answer pairs with single-word answers.
- Architecture: ABC-CNN uses CNN-extracted spatial image features, LSTM question embeddings, attention extraction, and answer generation.The framework retains spatial information rather than reducing each image to a single global feature.
- Attention Extraction: The question embedding is projected into a configurable convolutional kernel with the same number of channels as the image feature map.This kernel encodes visual features corresponding to the question’s semantic object information.
- Attention Extraction: The kernel is convolved with the image feature map and softmax-normalized to produce an attention distribution over image regions.The resulting question-guided attention map has the same spatial size as the image feature map and can focus on regions such as an umbrella.
- Attention Extraction: Attention maps spatially weight image features to emphasize queried objects and suppress irrelevant regions before answer prediction.The weighted features are combined with the original image feature map and question embedding for nonlinear projection.
4. Experiments
ABC-CNN is evaluated on three VQA datasets using single-word-answer settings and standard accuracy and WUPS metrics. Across the reported experiments, attention-guided models outperform baselines and attention maps focus on question-relevant regions.
- Datasets and metrics: The evaluation covers Toronto COCO-QA, DAQUAR, and VQA, using QA pairs with single-word answers.The single-word subsets account for 100%, 85%, and 90% of Toronto-QA, VQA, and DAQUAR, respectively.
- Datasets and metrics: Performance is measured with answer accuracy, WUPS 0.9, and WUPS 0.0.Answer accuracy requires an exact ground-truth match, while WUPS measures word similarity using the Wu-Palmer taxonomy-based similarity.
- Results: ABC-CNN with the ATT model surpasses all baseline models on Toronto COCO-QA.It exceeds ENSEMBLE by 0.3% in answer accuracy despite using a single model.
- Results: Removing attention causes losses of 1.34% in accuracy, 0.85% in WUPS 0.9, and 0.35% in WUPS 0.0.The ablation compares ABC-CNN with its NO-ATT variant.
- Results: ABC-CNN outperforms single models across all metrics on DAQUAR-reduced and outperforms state-of-the-art methods on DAQUAR-full and VQA.On DAQUAR-reduced, it is 0.53% below ENSEMBLE on WUPS 0.9.
- Attention analysis: The question-guided attention maps capture different question intents with different regions, while the original feature map remains available for answer prediction.This allows answers even when the attention map does not focus on the queried object because it is the only object in the image.
5. Conclusion
The paper presents ABC-CNN as a unified attention-based framework for VQA, using question-guided attention to connect image content with question semantics. It reports improved performance across three datasets and attention maps related to question semantics.
- Conclusion: ABC-CNN unifies visual feature extraction and question understanding through a question-guided attention map.The map is generated by a configurable convolutional network whose kernel is adaptively determined by question meaning.
- Conclusion: ABC-CNN outperforms state-of-the-art methods on Toronto COCO-QA, DAQUAR, and VQA datasets.
- Conclusion: The produced attention maps are highly related to the semantics of image-related questions.