Source-linked AI summary
Exploring Human-like Attention Supervision in Visual Question Answering
Tingting Qiao, Jianfeng Dong, Duanqing Xu
TL;DR
Human attention labels for VQA v1.0 image-question pairs are limited, and collecting human-labeled attention maps is time-consuming and expensive. HAN is trained on VQA-HAT to generate human-like attention maps for VQA v2.0 image-question pairs, which are used to supervise an attention-based VQA model. 0.15% higher overall accuracy on the VQA v2.0 test-dev set is reported after adding human-like attention supervision.
Problem
Human attention labels for VQA v1.0 image-question pairs are limited, and collecting human-labeled attention maps is time-consuming and expensive.
Source: Introduction · p. 1Method
HAN is trained on VQA-HAT to generate human-like attention maps for VQA v2.0 image-question pairs, which are used to supervise an attention-based VQA model.
Source: Introduction · p. 2Results
0.15% higher overall accuracy on the VQA v2.0 test-dev set is reported after adding human-like attention supervision.
Source: Introduction · p. 2Takeaways & Limitations
The authors conclude that human-focused regions help attention-based VQA models achieve better performance, and report more accurate attention maps and better answers with supervision.
Source: Introduction · p. 2, Conclusion · p. 7
Abstract
from arXiv · showhide
Attention mechanisms have been widely applied in the Visual Question Answering (VQA) task, as they help to focus on the area-of-interest of both visual and textual information. To answer the questions correctly, the model needs to selectively target different areas of an image, which suggests that an attention-based model may benefit from an explicit attention supervision. In this work, we aim to address the problem of adding attention supervision to VQA models. Since there is a lack of human attention data, we first propose a Human Attention Network (HAN) to generate human-like attention maps, training on a recently released dataset called Human ATtention Dataset (VQA-HAT). Then, we apply the pre-trained HAN on the VQA v2.0 dataset to automatically produce the human-like attention maps for all image-question pairs. The generated human-like attention map dataset for the VQA v2.0 dataset is named as Human-Like ATtention (HLAT) dataset. Finally, we apply human-like attention supervision to an attention-based VQA model. The experiments show that adding human-like supervision yields a more accurate attention together with a better performance, showing a promising future for human-like attention supervision in VQA.
Introduction
The paper addresses the limited availability and cost of human attention annotations by proposing a network that generates human-like maps for VQA image-question pairs. On VQA v2.0 test-dev, adding those maps as supervision improves overall accuracy by 0.15%.
Introduction
- The authors report that explicit supervision makes attention maps more accurate and predicted answers better.Figure 1 visualizes images, attention maps, and predicted answers.Source: Figure 1 · p. 1, Introduction · p. 2
- HAN is designed to automatically generate attention maps for image-question pairs by training on existing VQA-HAT, offering an alternative to collecting human-labeled maps, a time-consuming and expensive task (Das et al. 2016; Gan et al. 2017).Source: Introduction · p. 1
- The pre-trained HAN produces maps for all VQA v2.0 image-question pairs, forming the Human-Like ATtention (HLAT) dataset for supervision and evaluation.The paper says HLAT can be used to train attention-based models and evaluate the quality of model-generated attention.Source: Introduction · p. 2
- 0.15% higher overall accuracy on the VQA v2.0 test-dev set is reported after adding human-like attention supervision to the baseline.The authors conclude that regions humans focus on help attention-based VQA models achieve better performance.Source: Introduction · p. 2
Related Work
Prior VQA attention models learn attention implicitly, which may not align with human focus; existing human attention annotations are limited and costly to collect. The paper motivates human-like attention supervision as an alternative to object segmentation for questions requiring broader reasoning.
Related Work
- Earlier VQA approaches include stacked attention, dynamic memory, multimodal compact bilinear pooling, hierarchical co-attention, dual attention, and multimodal low-rank bilinear pooling.These methods use attention to focus on image regions or combine attended visual and textual features.Source: Related Work · p. 2
- Implicitly learned attention may be fortuitous, motivating the use of HLAT maps to supervise VQA models on VQA v2.0.The paper contrasts its approach with attention learned without supervision.Source: Related Work · p. 2
- Object-segmentation supervision can be inaccurate for questions requiring reasoning or global information, motivating human-like attention maps.The paper gives questions about children paying attention to class and whether it is sunny as examples.Source: Related Work · p. 2
- VQA-HAT provides human attention annotations for around 10% of VQA v1.0 image-question pairs, while collecting such datasets takes substantial time and effort.The paper proposes training HAN on VQA-HAT to generate maps automatically for image-question pairs.Source: Related Work · p. 2
Approach
The approach section introduces the Human Attention Network (HAN), which predicts an attention map from an image-question pair.
Approach
- HAN predicts an attention map given an image-question pair.The section overview says the paper presents HAN and a supervised VQA attention model in separate sections.Source: Approach · p. 2, Figure 2 · p. 3
Input representation
Images are represented with spatial ResNet features, while questions are encoded by a GRU. The resulting image regions and final question state provide the model's visual and textual representations.
Image representation
- The image is scaled to 448 × 448 and represented by ResNet-152 features over 14 × 14 regions, with 2048 dimensions per region.The feature map is taken from the last convolutional layer.Source: Image representation · p. 3
Question representation
- Question words are embedded and fed sequentially into a GRU, whose final hidden state is used as the question representation.The word embedding matrix and GRU are trained end-to-end.Source: Question representation · p. 3
Human Attention Network
HAN predicts human-like attention maps for image-question pairs by generating coarse maps and refining them. It is trained to match human attention using mean squared error.
Human Attention Network
- HAN predicts an attention map for each image-question pair by refining coarse maps into a more precise map.Its attention generation module produces several coarse maps; its refinement module combines them.Source: Human Attention Network · p. 3
Attention generation
- Image and question features are projected and fused, then a convolutional layer generates multiple attention maps.The fusion uses element-wise multiplication, and the convolution uses G kernels of size 1.Source: Attention generation · p. 3
Attention refinement
- A GRU processes the sequence of coarse attention maps, and its last hidden state is transformed into a refined attention distribution over the image.A fully connected layer and softmax produce the refined map.Source: Attention refinement · pp. 3–4
Objective function
- HAN is trained with mean squared error to bring its predicted attention map close to the human attention map.The training data consists of image-question pairs with human attention, and the loss is minimized over the training set.Source: Objective function · p. 4
Attention supervision for VQA
The section contrasts unsupervised attention in MLB with a supervised variant using human-like attention maps generated by HAN. The supervised model uses these maps to guide attention while retaining answer classification.
Attention supervision for VQA
- The section presents the unsupervised model first, then introduces the supervised model as a way to let generated attention learn from ground truth.The supervised model adds attention supervision to the unsupervised approach.Source: Attention supervision for VQA · p. 4, Supervised attention model · p. 4
Unsupervised attention model
- MLB generates multiple attention maps, pools image features using them, fuses the attentive visual and question features, and classifies candidate answers.Its attention maps are learned without supervision, which may produce inaccurate maps.Source: Unsupervised attention model · p. 4, Supervised attention model · p. 4
Supervised attention model
- The supervised model uses HAN-generated attention maps for VQA v2.0 pairs and trains its attention to match them while predicting answers.A GRU encodes the sequence of MLB attention maps, then a linear layer and softmax produce attention probabilities; the total loss weights classification and attention-supervision losses.Source: Supervised attention model · pp. 4–5
Experiment
The experiments assess HAN’s similarity to human attention and the effect of GRU fusion, then evaluate human-like attention supervision on VQA v2.0.
Dataset
- The VQA-HAT training set contains 58,475 human attention maps, with 1,374 more in validation, from VQA v1.0 image-question pairs.Maps were collected by 800 annotators; eight zero-valued maps were removed before training.Source: Dataset · p. 5
Implementation details
- The experimental setup follows Kim et al. (2016b); word embeddings and the word-encoding GRU use pre-trained skip-thought initialization, with 300k iterations and batch size 64.HAN uses three glimpses and a 512-dimensional GRU hidden state to refine attention maps.Source: Implementation details · p. 5
Results
- 0.668 mean rank-correlation: HAN attention has the highest reported correlation with human attention, exceeding human performance by 0.045.Higher mean rank-correlation indicates better attention-map quality.Source: Table 1 · p. 5, Results · p. 6
Evaluation metric: rank correlation
- Rank-correlation compares spatial rankings of 14 × 14 machine and human attention maps, averaging correlation over three annotators’ maps.Higher correlation indicates better attention-map quality.Source: Evaluation metric: rank correlation · p. 5
Results
- GRU fusion significantly outperforms averaging attention maps in experiments with two and three glimpses.The comparison tests the GRU against an average map for each glimpse setting.Source: Results · p. 6, Table 2 · p. 6
Attention-based model for VQA
- HLAT attention maps are used as ground truth for the supervised VQA model, which is evaluated on VQA v2.0 test-dev.The evaluation uses the provided answer-accuracy metric; an answer is fully accurate when at least three annotators agree.Source: Attention-based model for VQA · p. 6, Evaluation metric · p. 7
- 0.11% at G=1 and 0.15% at G=2: the supervised model outperforms the unsupervised model on VQA v2.0 test-dev.Table 3 reports the test-dev results.Source: Table 3 · p. 7, Evaluation metric · p. 7
Conclusion
The paper reports that human-like attention supervision improves VQA performance and that the supervised model produces more accurate attention and better answers.
Conclusion
- The supervised model outperforms the unsupervised model, and visualizations show more accurate attention that yields better answers.The authors conclude that explicit attention supervision helps attention-based VQA models perform better.Source: Conclusion · p. 7
Supplementary Material
The supplementary figures compare HAN-generated maps with human attention and show examples of attention supervision in VQA.
Supplementary Material
- Figure 6 presents HAN-generated attention as comparable in quality to human attention.The figure shows examples of human and HAN-generated attention maps.Source: Figure 6 · p. 9
- Figure 7 contrasts coarse, inaccurate unsupervised attention with supervised attention that is more accurate and yields better answers.In some examples both models predict the same answer, but the supervised model has greater confidence.Source: Figure 7 · p. 10