PaperlayerView exact revision on arXiv

Source-linked AI summary

VQA: Visual Question Answering

Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, Devi Parikh

arXiv:1505.00468v7cs.CLcs.CV

TL;DR

The paper addresses whether visual question answering can demand richer visual understanding and reasoning than generic image captioning. It introduces an open-ended VQA task and large dataset, finding that its best model outperforms vision-only and language-only baselines but remains below human performance.

  • Problem

    Image captioning can rely on coarse scene understanding and word statistics, motivating a task that tests more detailed visual understanding and complex reasoning.

  • Method

    The paper introduces open-ended VQA and constructs a dataset from 204,721 MS COCO images and 50,000 abstract scenes, with questions and answers collected from human subjects.

  • Results

    The best model reaches 58.16% open-ended and 63.09% multiple-choice accuracy, outperforming vision-only and language-only baselines but remaining below human performance.

  • Takeaways & Limitations

    VQA provides a benchmark spanning computer vision, natural language processing, and commonsense reasoning through diverse open-ended questions and answers.

  • Takeaways & Limitations

    The dataset captures perceived rather than objectively required question-answering age and commonsense difficulty from uncontrolled MTurk annotations.

Abstract

from arXiv · show

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ~0.25M images, ~0.76M questions, and ~10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).

1 INTRODUCTION

VQA combines visual understanding with language and knowledge-based reasoning through free-form, open-ended questions targeting diverse image details and contexts. Its standardized evaluation and large dataset support systematic progress on this challenging task.

  • Motivation: VQA advances a multidisciplinary AI direction combining Computer Vision, Natural Language Processing, and Knowledge Representation & Reasoning.The introduction situates VQA within renewed interest in multidisciplinary AI research, including image and video captioning.
  • Task scope: VQA questions target visual details and higher-level activities, knowledge-base reasoning, and commonsense reasoning.Examples include counting bikes, recognizing crying, identifying vegetarian pizza, and inferring vision or social expectations.
  • Evaluation: VQA supports automatic quantitative evaluation, including a multiple-choice variant that selects answers from a predefined list.The open-ended task instead requires a free-form response.
  • Dataset: 204,721 MS COCO images and 50,000 newly created abstract scenes form the presented dataset.The MS COCO images depict diverse, complex scenes, while the abstract scenes support research focused on high-level VQA reasoning.

2 RELATED WORK · 3 VQA DATASET COLLECTION

The VQA dataset combines real MS COCO images and abstract scenes, collecting image-dependent questions and human answers for open-ended and multiple-choice evaluation. Its collection process promotes diverse questions requiring visual evidence, including commonsense reasoning grounded in the image.

  • 3 VQA DATASET COLLECTION: The dataset includes representative questions, image-viewing answers, and answers generated without viewing the image.These examples are presented for numerous representative dataset cases.
  • 3 VQA DATASET COLLECTION: The dataset uses 123,287 MS COCO training and validation images, plus 81,434 test images.Real-image splits follow MS COCO’s train/validation/test strategy, including test-dev, test-standard, test-challenge, and test-reserve.
  • 3 VQA DATASET COLLECTION: Questions target both low-level visual attributes and commonsense knowledge about scenes while requiring the image for correct answers.Examples include identifying a cat’s color, counting chairs, and inferring an animal’s sound from the pictured scene.
  • 3 VQA DATASET COLLECTION: Subjects were instructed to avoid generic image-independent questions by asking questions that require the image to answer.The same interface was used for real images and abstract scenes.
  • 3 VQA DATASET COLLECTION: Three questions from unique workers were gathered for each image or scene, with previous questions shown to increase question diversity.This collection procedure was applied across both real images and abstract scenes.
  • 3 VQA DATASET COLLECTION: The dataset supports open-ended and multiple-choice answer modalities for testing.Open-ended answers are evaluated with an accuracy metric based on agreement among workers.
  • 3 VQA DATASET COLLECTION: An answer receives 100% accuracy when at least 3 workers provide that exact answer.Responses are lowercased, numbers converted to digits, and punctuation and articles removed before comparison.
  • 3 VQA DATASET COLLECTION: Question distributions are visualized by their first four words for real images and abstract scenes.The figure uses a random sample of 60K real-image questions and all abstract-scene questions.

4 VQA DATASET ANALYSIS

The dataset analysis shows that VQA questions have similar structures across real images and abstract scenes, while answers vary by question type and often require image-grounded reasoning. Human studies further indicate substantial agreement, limited answerability from commonsense or captions alone, and a need for deeper image understanding.

  • Question Types and Lengths: Question distributions are similar for real images and abstract scenes, with most questions spanning four to ten words.Question types are clustered by their first four words.
  • Typical Answers: Answer formats depend strongly on question type: yes/no questions favor “yes” and “no,” while other types elicit diverse or specialized responses.“What is...” and “What type...” questions have richer response diversity, whereas color and directional questions are more specialized.
  • Typical Answers: Although answers are brief enough for automatic evaluation, open-ended questions can require complex reasoning despite their deceptively simple responses.The dataset contains 23,234 unique one-word answers.
  • Human Agreement: 83.30% agreement is observed for real images and 87.49% for abstract scenes, with averages of 2.70 and 2.39 unique answers per question, respectively.These figures come from the Question + Image condition and reflect significant inter-human agreement.
  • Image Necessity: 47.43% of questions were judged by at least three subjects to require commonsense, while 18.14% received this judgment from at least six subjects.The study asked subjects to assess whether questions could be answered using commonsense knowledge without an image.
  • Caption Sufficiency: Captions improve accuracy over questions alone but remain significantly worse than showing the image, demonstrating that VQA needs deeper image understanding than generic captions provide.Caption-based results were measured with human-provided captions while withholding the image.

5 VQA BASELINES AND METHODS

The section establishes VQA baselines and a two-channel image–question model with 1,000 answer outputs, then compares language, vision, and fusion choices on open-ended and multiple-choice tasks. The best model uses a deeper LSTM question encoder with normalized image features, while analyses show gains from normalization, element-wise fusion, and a larger answer set.

  • Model architecture: The model combines image and question embeddings, fuses them through element-wise multiplication, and predicts among the 1,000 most frequent answers covering 82.67% of train+val answers.Image features are transformed into the question embedding space before fusion, followed by an MLP and softmax over candidate answers.
  • Ablations: Using K = 1000 possible answers outperforms K = 500 by 0.82% for open-ended and 1.92% for multiple-choice tasks.The model selects among the most frequent answers in the dataset.
  • Baselines and methods: Question-only models outperform the nearest-neighbor baseline, with BoW Q achieving 48.09% open-ended and 53.68% multiple-choice accuracy, and LSTM Q achieving 48.76% and 54.75%.The passage attributes this advantage to statistical priors in question wording and question types.
  • Best model: The best model, deeper LSTM Q + norm I, reaches 58.16% open-ended accuracy on VQA test-standard.It uses a two-layer LSTM for questions, VGGNet image features, ℓ2 normalization, common-space transformation, and element-wise fusion.
  • Ablations: ℓ2-normalizing image features improves accuracy by 0.16% for open-ended and 0.24% for multiple-choice tasks.These gains are reported by comparing the normalized and unnormalized image-feature models.
  • Ablations: Element-wise fusion outperforms concatenation by 0.95% for open-ended and 1.24% for multiple-choice tasks.The concatenation variant doubles the number of parameters in the following fully connected layer.

6 VQA CHALLENGE AND WORKSHOP

The VQA evaluation server supports test-set submissions and accuracy breakdowns, while an annual challenge and workshop are intended to promote systematic progress. Papers reporting VQA results should use test-standard accuracies and compare them with corresponding test2015 leaderboards.

  • Evaluation infrastructure: The evaluation server accepts test-set results and returns an accuracy breakdown.This infrastructure supports standardized evaluation of VQA systems.
  • Challenge and workshop: An annual VQA challenge and workshop are being organized to facilitate systematic progress, with the first workshop scheduled for CVPR 2016.The passage identifies the first workshop as the initial instance of this series.
  • Reporting standards: Papers should report test-standard accuracies from either the test2015 or Challenge test2015 phase.The guidance applies across open-ended and multiple-choice real-image and abstract-image settings.
  • Reporting standards: Reported test-standard accuracies should be compared with the corresponding test2015 leaderboard accuracies.The comparison is specified separately for each evaluation setting and leaderboard.

7 CONCLUSION AND DISCUSSION

The paper introduces open-ended Visual Question Answering, in which systems answer natural-language questions about images. It provides a large dataset and shows that accurate answering requires capabilities spanning vision, language, and commonsense reasoning.

  • VQA asks systems to provide accurate natural-language answers to open-ended questions about images.
  • The dataset contains over 250K images, 760K questions, and around 10M answers.
  • Answering the dataset’s questions accurately requires computer vision, natural language processing, and commonsense reasoning.The paper also demonstrates the wide variety of questions and answers in the dataset.

APPENDIX OVERVIEW

The appendix provides additional analyses, qualitative visualizations, evaluation details, model analyses, dataset documentation, collection interfaces, answer lists, and examples. It also includes VQA Real Image Challenge leaderboards for open-ended and multiple-choice settings.

  • The appendix covers caption–Q&A comparisons, “What is” visualizations, human multiple-choice accuracy, and VQA baseline details.
  • It analyzes the model’s “Age” and “Commonsense,” documents the abstract scene dataset and data-collection interfaces, and lists the dataset’s top answers.
  • Figure 13 shows test-standard-accuracy leaderboards for the VQA Real Image Challenge in open-ended and multiple-choice formats.The snapshot is dated October 28, 2016.
  • The appendix adds examples from the VQA dataset.

APPENDIX I: CAPTIONS vs. QUESTIONS

Questions and answers capture information beyond captions, with distinct word-distribution patterns. In particular, answers emphasize specific spatial relations, whereas captions favor more general spatial relations.

  • Questions and answers provide information beyond captions, as shown by differences in their noun, verb, and adjective word distributions.The comparison extracts nouns, verbs, and adjectives from caption and question-and-answer data and visualizes their shared and distinct words.
  • Answers have a higher proportion of specific spatial relations, such as “in” and “on,” than captions.
  • Captions have a higher proportion of general spatial relations, such as “with” and “near,” than answers.

APPENDIX II: “WHAT IS” ANALYSIS · APPENDIX III: MULTIPLE-CHOICE HUMAN AC-

The appendices analyze “What is” question and answer distributions across real images and abstract scenes. The analysis highlights diverse referenced objects, object relations, and answer types associated with question endings.

  • APPENDIX II: “WHAT IS” ANALYSIS: The analysis compares “What is” questions in real images and abstract scenes.
  • APPENDIX II: “WHAT IS” ANALYSIS: Question distributions are organized by their first five words.
  • APPENDIX II: “WHAT IS” ANALYSIS: The questions reference a diverse range of objects.
  • APPENDIX II: “WHAT IS” ANALYSIS: Some questions express relations between objects, including “holding” and “sitting on”.
  • APPENDIX II: “WHAT IS” ANALYSIS: Answer distributions are examined for “What is” questions ending in different words.
  • APPENDIX II: “WHAT IS” ANALYSIS: Questions ending in “eating” can have answers such as “pizza”, “watermelon” and “hot dog”.

CURACY … OF OUR MODEL

The evaluation reports human accuracy and inter-human agreement, describes question-type and nearest-neighbor baselines, and estimates the model’s age and commonsense using validation questions with annotations.

  • CURACY: 3,000 questions were sampled across real images and abstract scenes, with three human answers collected per question to compute multiple-choice accuracy.Table 6 reports the resulting human accuracies and inter-human agreement for open-ended answers.
  • CURACY: Multiple-choice accuracy is roughly unchanged for yes/no questions but significantly higher for other questions than open-ended accuracy.The passage specifies an approximately 15% increase for real images before truncation.
  • APPENDIX IV: DETAILS ON VQA BASELINES: The per Q-type prior baseline predicts the most popular answer for each question type, with types defined from question prefixes and constrained to at least 30 training questions.Question types and popular answers are computed on the real-images training set.
  • APPENDIX IV: DETAILS ON VQA BASELINES: The nearest-neighbor baseline retrieves k training questions using Skip-Thought cosine similarity, then selects the most similar associated image using fc7 cosine similarity.Skip-Thought performed best among the tested text feature spaces, and fc7 features came from the CaffeNet model in BVLC Caffe.
  • APPENDIX V: “AGE” AND “COMMONSENSE”: The best model, deeper LSTM Q + norm I, was selected using VQA test-dev accuracies for estimating answerable-question age and commonsense.Both estimates use questions from the VQA validation set with the corresponding annotations.
  • OF OUR MODEL: Answerable-question age is estimated as an accuracy-weighted average of the average human age required to answer each annotated validation question correctly.The weighting uses the model’s predicted-answer accuracy for each question.
  • APPENDIX V: “AGE” AND “COMMONSENSE”: The model’s degree of commonsense is estimated using a weighting procedure applied to annotated VQA validation questions.The supplied passage begins this procedure but is truncated before specifying the complete weighting definition.

APPENDIX VI: ABSTRACT SCENES DATASET

The abstract scenes dataset contains generated scenes built through a drag-and-drop interface, with objects that can be flipped and scaled. Object scale determines rendering order, while some objects have attributes representing different poses or states.

  • Scene creation: Subjects created abstract scenes using a drag-and-drop interface.The interface is shown in Fig. 23 (right).
  • Object manipulation: Objects could be flipped horizontally and scaled during scene creation.The passage describes these as available transformations for each object.
  • Rendering: Object scale determined the rendering order of the objects.
  • Object attributes: Many objects had different attributes corresponding to different poses or states.

APPENDIX VII: USER INTERFACES · APPENDIX VIII: ANSWER DISTRIBUTION · APPENDIX IX: ADDITIONAL EXAMPLES

The appendices document the AMT interfaces used to collect diverse questions, image-grounded answers, and plausible incorrect multiple-choice answers. They also visualize question and answer distributions and provide random examples spanning real images, abstract scenes, and multiple-choice questions.

  • APPENDIX VII: USER INTERFACES: The question-collection interface showed workers previously asked questions and instructed them to ask something different, increasing dataset diversity.This interface was used when collecting the third question for an image.
  • APPENDIX VII: USER INTERFACES: The answer-collection interface showed subjects the corresponding image while they answered previously collected questions.A separate interface collected plausible but incorrect multiple-choice answers without showing the image, using commonsense.
  • APPENDIX VIII: ANSWER DISTRIBUTION: Question stems beginning with “What is” are distributed by their first five words for 60K real-image questions and all abstract-scene questions.The visualization uses arc length to represent the number of questions containing each word.
  • APPENDIX VIII: ANSWER DISTRIBUTION: Answers to “What is” questions are distributed according to the different words ending the questions, including “doing?” and “on?”.The figure reports distributions for a random sample of 60K real-image questions and all abstract-scene questions.
  • APPENDIX VII: USER INTERFACES: A separate AMT interface collected plausible but incorrect multiple-choice answers while subjects answered without seeing the image.The collection process relied only on commonsense for these distractor answers.
  • APPENDIX IX: ADDITIONAL EXAMPLES: Random examples illustrate questions and image-seen versus image-unseen answers for MS COCO images and abstract scenes, alongside representative multiple-choice questions.Figures 27–29 cover real-image, abstract-scene, and multiple-choice examples, respectively.
Loading 1505.00468v7…