Source-linked AI summary
Visual7W: Grounded Question Answering in Images
Yuke Zhu, Oliver Groth, Michael Bernstein, Li Fei-Fei
TL;DR
Visual7W addresses the weak link between textual QA and local image regions by grounding object mentions to bounding boxes. It builds a large seven-category multiple-choice benchmark with textual and visual answers, then evaluates an attention-based LSTM. The model reaches 55.6%, while humans achieve 96.6% and state-of-the-art LSTMs 52.1%.
Problem
Previous visual QA work loosely associated questions and answers with whole images, despite many questions referring to local regions.
Method
The paper constructs Visual7W with object-level groundings, multiple-choice textual and visual answers, and a spatial-attention LSTM for grounded QA.
Results
55.6% is achieved by the attention-based LSTM, compared with 52.1% for state-of-the-art LSTMs and 96.6% for humans.
Takeaways & Limitations
Object grounding supports visual-region answers, resolves coreference ambiguity, and enables grounded QA evaluation beyond global image-level associations.
Abstract
from arXiv · showhide
We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.
1. Introduction
Visual7W addresses the weak image-level link in prior visual QA by grounding object mentions to image regions. It introduces a large, multiple-choice 7W benchmark with textual and visual answers and evaluates an attention-based model.
- Previous visual QA datasets mainly established global associations between sentences and images, while many questions concern local visual regions.
- Object-level grounding links textual object mentions to bounding boxes, resolving coreference ambiguity and enabling visually grounded answers.
- Visual7W collects 327,939 QA pairs on 47,300 COCO images, with 1,311,756 human-generated choices and 561,459 object groundings.
- The dataset covers seven W-question categories and adds pointing questions whose answers are image regions, alongside telling questions with textual answers.
- 55.6% is achieved by the spatial-attention LSTM, compared with 52.1% for state-of-the-art LSTMs and 96.6% for humans.
2. Related Work
Visual QA extends vision-language research by asking interactive questions about images rather than only generating descriptions. Visual7W differs from earlier benchmarks by linking object mentions to visual appearances through grounding.
- Visual question answering is presented as an interactive extension of image captioning with stronger connections to real-world applications.
- Traditional text-based QA uses pipelines for parsing, knowledge-base querying, and answer generation, whereas neural models increasingly learn directly from questions and answers.
- Earlier visual QA datasets evaluated textual answers but omitted links between object mentions and their visual appearances.
- Visual7W establishes object grounding to evaluate visual question answering with grounded image-region answers.
3. Creating the Visual7W Dataset
Visual7W standardizes seven categories of grounded questions in a multiple-choice format, collecting textual answers, visual answers, and object-level annotations. Its grounding addresses ambiguity between language and image objects.
- 3.1. Collecting the 7W Questions: Visual7W uses what, where, when, who, why, how, and which questions in a multiple-choice setting with textual and visual answers.
- 3.1. Collecting the 7W Questions: Workers write concise questions and answers from image content, and each QA pair requires at least two positive quality votes.
- 3.2. Collecting Object-level Groundings: 561,459 object bounding boxes were collected, averaging 12 boxes per image, to ground object mentions and pointing choices.
- 3.2. Collecting Object-level Groundings: Object-level groundings resolve coreference ambiguity, support visual answers, and expose object distributions relevant to answering QA tasks.
- 3.2. Collecting Object-level Groundings: Coreference ambiguity occurs in 1% of accepted questions and 7% of accepted answers, complicating evaluation when multiple image objects match a mention.
4. Comparison and Analysis
Visual7W combines grounding annotations with diverse, information-rich QA pairs and compares human and model performance across datasets.
- Grounding: 561,459 object groundings span 36,579 categories, with 85% of categories having fewer than 5 instances.The annotations provide open-vocabulary coverage rather than focusing only on predefined salient categories.
- Object Distribution: Pointing QA covers an order of magnitude more object categories than telling QA, while the most frequent categories favor persons, daily-life objects, and natural entities.Figure 4 ranks object categories by frequency.
- Human-Machine Performance: Visual7W’s nearly perfect human performance and large human–model gap make it a challenging benchmark with room for future research.The paper compares Visual7W with VQA and Facebook bAbI in Table 2.
- QA Diversity: The dataset averages 6.9-word questions and 2.0-word answers, with 27.6% of answers exceeding two words.Where and why answers average three words, while telling answers show a long-tail length distribution.
5. Attention-based Model for Grounded QA
The model combines an LSTM with spatial attention so grounded QA can focus on relevant image regions while processing questions and answers.
- Motivation: Spatial attention addresses the limitation of global image features by selecting local regions relevant to visual QA.Grounded regions are pertinent to answering many questions, while other image regions may be irrelevant.
- Architecture: The model encodes the image and question into an LSTM hidden state before decoding an answer.The image is fed as the first input token, followed by the question tokens.
- Input Representation: The image is transformed into a 4096-dimensional VGG-16 fc7 feature, while word tokens are mapped into 512-dimensional embeddings.The image and word embeddings are fed sequentially into the LSTM.
- Spatial Attention: At each step, attention produces a weighted average of convolutional features conditioned on the previous hidden state.The attention vector has 196 dimensions over the 14 × 14 feature map, and uniform attention recovers the standard LSTM.
- Training and Inference: For telling QA, the model receives ground-truth answer tokens during training; for pointing QA, it scores candidate regions using the final hidden state.Cross-entropy loss trains the model parameters, and testing selects the candidate with the largest likelihood.
6. Experiments
The experiments evaluate humans and models on multiple-choice telling and pointing QA, showing a substantial human–model gap and benefits from image content and spatial attention.
- Experiment Setup: Multiple-choice evaluation uses an image, a question, and four candidates, with natural-language answers for telling QA and image regions for pointing QA.Accuracy measures whether the selected candidate is correct; the dataset is split 50%/20%/30% into training, validation, and test sets.
- Human Experiments: Human experiments test answering without images and with images using five workers per condition and majority-vote accuracy.The experiments measure both visual QA performance and the use of common sense without visual input.
- Human Experiments: Human response time doubles when images are displayed, and the correlation between response time and accuracy is −0.135.Why questions take longer on average, while longer response times do not imply higher performance.
- Model Experiments: The LSTM reaches 46.2% without images versus 35.3% for humans, while combining questions and images raises baseline accuracy to 52.1%.The authors attribute the image-free LSTM result to learned answer priors and report that both questions and image content contribute to better results.
- Performance: 96.6% human accuracy contrasts with a 52.1% Question + Image baseline, leaving a substantial performance gap.The paper describes this gap as providing room for future research.
- Model Experiments: LSTM-Att achieves the best model performance at 55.6% and outperforms other baselines on every question type except how.The model uses spatial attention to focus on pertinent image regions as it reads question tokens.
- Model Analysis: Attention peaks align with answer-object bounding boxes, whereas examples without answer objects produce more scattered heat maps.The heat-map visualizations show attention changing as the model reads QA words.
- Model Analysis: Accuracy increases with object-category frequency, yet the model exceeds 50% on categories with only a few training instances.The authors interpret this as knowledge transfer from common categories to rare ones.
7. Conclusions
Visual7W links QA text to object regions through grounding and uses this structure to study visually grounded answers and attention-based QA.
- Conclusions: Object grounding links object mentions in QA sentences to image bounding boxes, supporting deeper image understanding beyond object recognition.The annotations also help resolve coreference ambiguity, understand object distributions, and evaluate visually grounded QA.
- Conclusions: The paper proposes an attention-based LSTM for grounded QA and identifies common-sense knowledge as a direction for improving complex reasoning.The stated future direction concerns QA tasks that require complex reasoning.