Source-linked AI summary
Where To Look: Focus Regions for Visual Question Answering
Kevin J. Shih, Saurabh Singh, Derek Hoiem
TL;DR
Visual question answering requires determining which image evidence answers a textual query, because questions may call for focused regions, the whole image, or more specialized reasoning. The paper learns region relevance and candidate-answer scores jointly from question-answer pairs, using a margin-based framework for multiple-choice VQA. The model outperforms the baselines and existing work on the MS COCO VQA multiple-choice task, with substantial gains on questions such as object-color identification.
Problem
VQA needs a method for learning where to look from question-image pairs, since different questions require focused regions, whole-image evidence, or specialized reasoning.
Method
The model maps language and visual region features into a shared latent space, weights regions by relevance, and scores candidate answers using a jointly learned margin-based framework.
Results
The model outperforms all baselines and existing work on the MS COCO VQA multiple-choice task, with substantial gains for questions requiring focused regions.
Takeaways & Limitations
Selective region modeling is especially useful for VQA questions such as identifying object colors that require attention to particular image regions.
Takeaways & Limitations
The direct relevance formulation was the only tested version that successfully accounted for both query and region information.
Abstract
from arXiv · showhide
We present a method that learns to answer visual questions by selecting image regions relevant to the text-based query. Our method exhibits significant improvements in answering questions such as "what color," where it is necessary to evaluate a specific location, and "what room," where it selectively identifies informative image regions. Our model is tested on the VQA dataset which is the largest human-annotated visual question answering dataset to our knowledge.
1. Introduction
The paper frames VQA as a problem of learning where to look, using question-image pairs to select relevant regions and score candidate answers. It presents a region-selection mechanism and a margin-based multiple-choice framework that outperforms the provided baselines.
- VQA requires answering natural-language questions about images while handling language grounding, recognition, relations, relevance, counting, and reading.
- Questions differ in visual scope: some require the full image, others a focused region, and some specialized knowledge or reasoning beyond the paper’s target.
- The proposed method learns a non-linear mapping of language and visual region features into a shared latent space to determine relevance.
- Relevant regions are used to score question-answer pairings, with the embedding and scoring function learned jointly using a margin-based loss from question-answer supervision.
- The study evaluates multiple-choice VQA on the VQA dataset because its scoring is less ambiguous than open-ended answer verification.
- The region-selection model is reported to outperform provided baselines, while uniformly weighting all regions is not helpful.
2. Related Works
The related work situates this approach among image captioning, VQA datasets, memory networks, and language-representation methods. Its distinguishing direction is selecting image evidence from a textual query rather than generating text from image regions.
- Image-captioning systems tie text to images through attention, salient-object detection, or word generation, whereas this model determines relevant regions from a textual query.
- The paper uses the VQA dataset because its questions and answers are open-ended, while evaluating the multiple-choice setting to reduce ambiguity.
- Other VQA datasets based on reformulated object annotations provide visual tasks but limit the scope of their questions.
- The model is inspired by End-to-End Memory Networks, treating image regions like sentences and projecting question and visual features into a shared subspace for relevance.
- Prior VQA work often uses recurrent language models, but bag-of-words, averaging, and normalized CCA representations can perform roughly as well or better.
3. Approach
The approach jointly embeds question-answer text and candidate image regions to learn relevance weights, then scores answers using weighted visual-language features. It uses structured margin training and parse-based language representations to identify informative regions for VQA.
- Region Selection Layer: The model embeds question-answer text and image regions into a shared latent space, using inner products and softmax to compute per-region relevance weights.The weighted visual-language representation is passed through a scoring network to estimate whether the answer is correct.
- QA Objective: Training uses a hinge loss that requires the correct answer score to exceed the highest-scoring incorrect choice by a consensus-scaled margin.With annotator fractions of 0.6 and 0.2, the required margin is 0.4.
- Region Selection Layer: For each question-answer pair, 100 candidate regions are combined with repeated text features, linearly projected, relevance-weighted, and passed through nonlinear layers to produce a score.The candidates consist of 99 top-ranked Edge Boxes after non-max suppression plus a whole-image region.
- Region Selection Layer: The inner-product relevance formulation was the only tested alternative that incorporated both query and region information; direct relevance prediction varied little with language.This comparison motivates computing relevance in a latent embedded space rather than directly from concatenated features.
- Language Representation: Question-answer representations are formed from averaged 300-dimensional word2vec vectors organized into semantic bins, producing a fixed 1500-dimensional input.The bins separate question type, nominal subject, other nouns, remaining words, and candidate-answer words.
- Architecture: The network uses a multi-layer feedforward architecture with word2vec text inputs and CNN region features, including projections, batch normalization, and ReLU layers.VGG-s features concatenate a 4096-dimensional fully connected output with a 1000-dimensional pre-softmax output for each region.
4. Experiments
Experiments compare region selection with language-only, whole-image, and uniformly weighted-region baselines on multiple-choice VQA, then assess localization qualitatively and against manual annotations. The proposed model outperforms the baselines and existing work, while relevant regions generally receive higher weights.
- Experimental setup: The evaluation uses the MS COCO VQA multiple-choice task across validation, test-dev, and test-standard partitions.The dataset contains 82,783 training images, 40,504 validation images, and 81,434 test images.
- Baseline comparisons: The region-selection model outperforms language-only, whole-image, and uniformly averaged region-feature baselines on validation accuracy.Uniform weighting of regions is reported as not helpful.
- Baseline comparisons: The model outperforms the best-performing image-and-text models from the VQA paper on both test-dev and test-standard.The model was retrained on train+val data with a 10% held-out training set for model selection.
- Qualitative analysis: Qualitative results show strong localization for color questions and question-dependent focus on informative objects such as toilets and kites.The system can focus on relevant objects even when it answers counting questions incorrectly.
- Qualitative analysis: Attention varies with question-answer pairings: irrelevant choices tend to produce less-focused weighting, while matching regions alone do not guarantee high confidence.For example, a green region can be identified while the corresponding answer confidence remains below that of correct options.
- Region evaluation: In 148 of 205 manually annotated validation instances (72.2%), mean pixel weighting inside relevant regions exceeds the global average.Weights are computed by cumulatively adding each region’s selection weight to its constituent pixels.
- Language-model analysis: The parsed language representation significantly outperforms a simpler two-bin question-answer representation on the VQA real-images validation subset.The parsed model uses additional semantic bins defined with the Stanford Parser.
5. Conclusion
The model selects image regions to solve visual question answering and outperforms baselines and existing work on the MS COCO VQA multiple-choice task. Gains are substantial for questions such as object-color identification that require focusing on particular regions.
- The model learns to select image regions from images to solve visual question answering problems.
- It outperforms all baselines and existing work on the MS COCO VQA multiple-choice task.
- The model shows substantial gains on questions such as identifying object colors, which require focusing on particular regions.
- Future work includes specialized tasks such as counting and reading, pretrained object and attribute detectors, geometric reasoning, outside knowledge, small objects, and activity recognition.