PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

Abhishek Das, Harsh Agrawal, C. Lawrence Zitnick, Devi Parikh, Dhruv Batra

arXiv:1606.03556v2cs.CVcs.CL

TL;DR

The paper asks whether people and VQA models attend to the same image regions when answering visual questions. It collects task-driven human attention maps and compares them with model maps; the most accurate VQA model’s maps correlate less with human maps than task-independent saliency does.

  • Problem

    The study asks which image regions people use to answer visual questions and whether deep VQA models attend to those same regions.

    Source: §1 · p. 1, §4 · p. 5

  • Method

    The authors collect human attention maps by asking subjects to sharpen answer-relevant regions of blurred images, then compare them with VQA-model maps.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    0.26 mean rank-correlation with human attention maps for the most accurate VQA model, versus 0.49 for task-independent saliency maps.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VQA-HAT maps can evaluate unsupervised attention maps or provide attention supervision for VQA models.

    Source: §5 · p. 8

    It is ill-posed to define necessary attention maps at the pixel level, and the appropriate semantic space for necessary and sufficient human attention maps remains an open question.

    Source: §5 · p. 8

Abstract

from arXiv · show

We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.

1 Introduction

Selective attention is a useful model for visual question answering because different questions can call for different image regions. The paper introduces human attention maps and finds that current VQA-model attention does not match human attention as closely as task-independent saliency does.

1 Introduction

  • VQA questions can target different regions: people look at the floor for a surface-type question but at a player and racket for a game question.The paper frames this question-specific attention against selective human perception and computational attention models used across vision and language tasks.Source: §1 · p. 1
  • The most accurate VQA model’s attention maps have a mean rank-correlation of 0.26 with human maps, below task-independent saliency’s 0.49.The paper reports that after controlling for center bias, task-independent saliency correlation is poor while trends for machine-generated VQA-attention maps remain the same.Source: §1 · p. 2
  • The paper introduces VQA-HAT, human attention maps collected to evaluate machine-generated attention and potentially train attention-based VQA models.The study compares maps qualitatively through visualizations and quantitatively through rank-order correlation.Source: §1 · p. 2

2 Related Work

Prior VQA attention models produce spatial distributions as intermediate variables optimized for answering questions, while prior large-scale mouse annotations captured task-independent viewing. This work collects task-driven maps by asking people to sharpen image regions relevant to a VQA question.

2 Related Work

  • Existing VQA attention models produce spatial distributions optimized for downstream VQA cross-entropy; their interpretability is described as fortuitous.The section discusses SAN and HieCoAtt, as well as modular question-parsing approaches.Source: §2 · p. 2
  • The annotation procedure presents a blurred image and asks subjects to sharpen answer-relevant regions with smooth click-and-drag strokes.Figure 3 illustrates the deblurring procedure used to collect attention maps.Source: Figure 3 · p. 3, §2 · p. 3
  • Prior mouse tracking collected task-independent annotations by letting people freely explore blurred images, whereas this work asks them to sharpen regions needed to answer questions.Large-scale eye tracking is described as difficult and expensive to collect.Source: §2 · p. 3

3 VQA-HAT (Human ATtention) Dataset

The VQA-HAT study develops a blurred-image annotation task and tests variants that balance exploration against producing compact, answer-supporting attention maps. The answer-visible blurred-image interface performed best in human evaluation, and the resulting dataset includes tens of thousands of training maps.

3 VQA-HAT (Human ATtention) Dataset

  • The annotation task presents a blurred image and question, then lets subjects gradually sharpen regions through click-and-drag strokes.A low-resolution blurry image provides partial holistic scene information; gradual strokes aim to capture initial exploration followed by focused sharpening.Source: §3 · p. 3, §3.1 · p. 3

3.1.1 Blurred Image without Answer

  • Without the answer, subjects sometimes produced exploratory or incomplete maps, especially for counting and binary questions, and often answered incorrectly.For example, a subject could count only two visible players while missing a third elsewhere in the image.Source: §3.1.1 · p. 3

3.1.2 Blurred Image with Answer

  • With the correct answer visible, subjects were asked to sharpen as few regions as possible; knowing the answer helped them continue exploring for relevant regions in counting and binary questions.This variant was the Blurred Image with Answer interface.Source: §3.1.2 · pp. 3, 5

3.1.3 Blurred and Original Image with Answer

  • The third interface showed the question-answer pair and full-resolution image, but asking subjects to imagine another person answering from the sharpened image was counterintuitive.The design aimed to encourage exploitation rather than exploration.Source: Figure 4 · p. 4, §3.1.3 · p. 5

3.2 Dataset Evaluation

  • The interface evaluation used a second set of subjects to answer questions from attention-sharpened images, testing whether each map exposed enough information.The authors say the answer-visible blurred-image interface balanced highlighting too little versus too much.Source: §3.2 · p. 5, Table 1 · p. 5
  • The Blurred Image with Answer interface yields the highest human answer accuracy among the three tested annotation interfaces.A second set of subjects answered questions using the attention-sharpened images; the table compares their VQA accuracies.Source: Table 1 · p. 5, §3.2 · p. 5
  • The study collected maps for 58,475 training and 1,374 validation question-image pairs, and visualized average maps and example questions for six selected clusters.The training pairs came from 248,349 total, and validation pairs from 121,512 total; the dataset is to be released publicly.Source: §3.2 · p. 5, Figure 5 · p. 5

4 Human Attention Maps vs Unsupervised Attention Models

After removing human maps correlated with center bias, HieCoAtt-Q correlates more strongly with human attention than Judd saliency; on the full validation set, SAN-2 and HieCoAtt correlate positively but less strongly than Judd. Overall, current VQA attention models do not seem to look at the same regions as humans, though higher VQA accuracy is associated with slightly higher human-attention correlation.

4 Human Attention Maps vs Unsupervised Attention Models

  • Figure 6 shows random examples with human attention in column 2 and machine-generated attention in columns 3–5.The figure also includes corresponding rank-correlation coefficients.Source: Figure 6 · p. 6, §4 · p. 7
  • The comparison evaluates SAN-2 and HieCoAtt word-, phrase-, and question-level maps using rank correlation between ranked pixels in 14x14 maps.The order-based metric is chosen to be invariant to absolute spatial probability values.Source: §4 · p. 7
  • On the full validation set, SAN-2 and HieCoAtt maps correlate positively with human attention, but less strongly than task-independent Judd saliency maps.Table 2 reports mean rank-correlation coefficients averaged over image-question pairs.Source: Table 2 · p. 7, §4 · p. 7
  • Current VQA attention models do not seem to look at the same regions as humans, while higher model accuracy is associated with slightly higher human-attention correlation.The reported accuracy rises from 58.9% for SAN to 62.1% for HieCoAtt; inter-human mean rank-correlation is 0.623.Source: §4 · p. 7
  • After removing human maps positively correlated with a central attention map, HieCoAtt-Q has higher correlation with human attention than Judd saliency.On this reduced set, Judd saliency correlation drops significantly because of its strong center bias.Source: §4 · pp. 7–8, Table 3 · p. 8

5 Conclusion & Discussion

The paper releases VQA-HAT and distinguishes sufficient attention regions from the smallest sufficient regions. It argues that necessity is ill-posed at the pixel level and identifies finding a meaningful semantic space as a future question.

5 Conclusion & Discussion

  • VQA-HAT can evaluate machine-generated attention maps or provide attention supervision for VQA models.The dataset is introduced as a resource for both uses.Source: §5 · p. 8
  • Defining a necessary attention map in pixel space is ill-posed because random pixels may be removed while people can still answer the question.The paper leaves open which semantic space would make necessary and sufficient maps meaningful.Source: §5 · p. 8
Loading 1606.03556v2…