Source-linked AI summary
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
Abhishek Das, Harsh Agrawal, C. Lawrence Zitnick, Devi Parikh, Dhruv Batra
TL;DR
The paper asks whether human and deep-network attention focus on the same image regions in VQA. It collects human attention through deblurring interfaces, releases VQA-HAT, and finds that current VQA attention does not seem to match human attention.
Problem
The paper asks which image regions humans use to answer VQA questions and whether deep VQA models attend to the same regions.
Method
The authors collect human attention maps by having subjects sharpen question-relevant regions of blurred images, then compare them with machine maps qualitatively and by rank-order correlation.
Results
0.26 mean rank-correlation for the most accurate VQA model was lower than 0.49 for task-independent saliency maps, while center-bias-controlled trends remained similar for VQA attention models.
Takeaways & Limitations
VQA-HAT enables evaluation of unsupervised VQA attention maps and can support explicit attention supervision for VQA.
Takeaways & Limitations
Defining a necessary attention map in pixel space is ill-posed because humans may still answer after arbitrary pixels are blacked out.
Abstract
from arXiv · showhide
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.
1. Introduction
The paper studies which image regions humans use to answer VQA questions and whether attention-based VQA models select the same regions. It introduces human attention maps and compares them with machine-generated maps.
- VQA questions target different image areas, including background details and contextual information, unlike image captioning's often coarse visual understanding.
- The study asks which regions humans inspect for visual questions and whether deep VQA models attend to those same regions.
- Human attention varies with the question: people inspect the floor for surface type but the player and racket for the game being played.
- The authors design human attention studies and release the resulting VQA-HAT dataset for evaluating or training attention-based models.
- 0.26 mean rank-correlation was achieved by the most accurate VQA model, versus 0.49 for task-independent saliency maps.
2. Related Work
Prior VQA attention models produce spatial distributions optimized for answering questions, while human attention studies provide a distinct reference for evaluating where systems look.
- The work uses the free-form, open-ended VQA dataset released by Antol et al.
- VQA Models: SAN uses LSTM question encodings to produce spatial attention over convolutional image features, while HieCoAtt attends at word, phrase, and question levels.
- VQA Models: Existing VQA attention models are unsupervised intermediate spatial distributions trained to optimize VQA cross-entropy, not directly supervised human attention.
- Human Studies: Large-scale eye-tracking studies are useful but difficult and expensive, motivating mouse-tracking approaches for collecting attention maps.
- Human Studies: The authors collect ground-truth annotations by instructing subjects to sharpen blurred regions important for answering questions.
3. VQA-HAT (Human ATtention) Dataset
VQA-HAT collects human attention maps through a deblurring task in which subjects sharpen question-relevant image regions. Pilot interface evaluations seek a balance between revealing too little and too much information.
- Subjects click and drag over a blurred image to progressively sharpen regions that help answer the associated visual question.
- The study compares interfaces showing different combinations of the question, answer, original image, and blurred image, then evaluates them through human question answering.
- AMT payment incentives encourage minimal deblurring, while human answer accuracy indicates that subjects can still identify sufficient regions.
- Approximately 20,000 AMT HITs from 800 unique workers produce the publicly released VQA-HAT dataset.
4. Human Attention Maps vs Unsupervised Attention Models
The study compares unsupervised VQA attention maps with human maps using rank-order correlation, finding that model maps correlate positively but generally less strongly than task-independent saliency maps. Removing center bias changes this comparison, with HieCoAtt-Q surpassing Judd saliency.
- Evaluation setup: Human attention maps are collected for VQA image-question pairs and compared against unsupervised attention models.The evaluated models include SAN-2 and word-, phrase-, and question-level HieCoAtt maps.
- Evaluation setup: Rank correlation is computed after scaling maps to 14x14 and ranking pixels, making evaluation invariant to absolute attention probabilities.This avoids sensitivity to peaky or diffuse maps caused by temperature settings.
- Results: Both SAN-2 and HieCoAtt maps positively correlate with human attention, but less strongly than task-independent Judd saliency maps.Table 2 reports mean rank-order correlation averaged over validation image-question pairs.
- Results: 0.623 is the mean interhuman rank-correlation on the validation set, providing a reference for agreement among human attention maps.The estimate uses three human attention maps per image-question pair.
- Center bias: Judd saliency has stronger center bias than human attention, with correlations of 0.877 and 0.458, respectively, against a central attention map.The center bias may partly reflect centrally located objects becoming likely question subjects in the VQA dataset.
- Center bias: After removing positively center-correlated human maps, HieCoAtt-Q correlates more highly with human attention than Judd saliency.Relative trends between SAN-2 and HieCoAtt remain similar to those on the full validation set.
5. Conclusion & Discussion
The paper releases VQA-HAT and uses it to quantify whether VQA models attend to the same image regions as humans. It also identifies the semantic definition of necessary and sufficient attention maps as an open question.
- Conclusion: VQA-HAT is introduced and released as a dataset for evaluating unsupervised attention maps or training models with attention supervision.The dataset targets attention-based VQA models.
- Conclusion: The work quantifies whether attention-based VQA models look at the same image regions as humans when producing answers.This directly operationalizes the paper’s central comparison between machine and human attention.
- Discussion: Pixel-level necessary attention maps are ill-posed because arbitrary pixels can be removed while humans may still answer correctly.The paper therefore leaves the appropriate semantic space for necessary and sufficient maps as a question for future work.