VQA: Visual Question Answering

1 INTRODUCTION

• VQA takes an image and an open-ended natural-language question as input and produces a natural-language answer, targeting scenarios such as assisting visually impaired users or intelligence analysts.The proposed task is motivated by the goal of advancing AI through multimodal tasks with well-defined quantitative evaluation.Source: §1 · p. 1
• Questions can require fine-grained recognition, object detection, activity recognition, knowledge-base reasoning, and commonsense reasoning, beyond coarse scene-level understanding.Even yes/no answers may require nontrivial reasoning; many questions can nevertheless be answered with one to three words.Source: §1 · p. 1
• The paper presents both open-ended answering and multiple-choice VQA, with short answers enabling quantitative evaluation by counting correctly answered questions.The multiple-choice task asks systems to choose from a predefined answer list.Source: §1 · pp. 1–2
• The dataset combines 204,721 MS COCO images and 50,000 abstract scenes, with three questions per image or scene and ten answers with confidence per question, totaling over 760K questions and around 10M answers.The abstract scenes support research on high-level reasoning without requiring systems to parse real images.Source: §1 · p. 2
• The paper analyzes question and answer types, compares their information content with image captions, provides baselines combining text and visual features, and proposes an annual challenge and workshop.The authors frame VQA as a balance between ambitious image-understanding challenges and accessibility to communities with established component methods.Source: §1 · p. 2

VQA Efforts

• Unlike earlier restricted or synthetic efforts, [36], this work targets human-generated, open-ended questions and answers, with >250,000 images versus 2,591 and 1,449, to broaden knowledge and reasoning.The earlier datasets limited answers to predefined categories or generated questions from templates.Source: §2 · p. 2
• Related work includes video-query answering [50], questions from visually-impaired users, abstract-scene commonsense [34], plausibility judgments,, prompted captions, Chinese COCO Q&A [18], and caption-derived questions [44].[44] automatically generated object, count, color, and location questions from COCO captions.Source: §2 · p. 2
• [37] conditions its LSTM question representation on CNN image features at each step and decodes answers sequentially; this paper computes them independently, multiplies them, then predicts answer classes.The paper describes its approach as “late fusion.”Source: §2 · p. 2

Text-based Q&A

• Text-based Q&A provides inspiration, but VQA grounds questions in images as well as text, and human-generated questions make commonsense knowledge and complex reasoning more essential.The text-Q&A discussion also highlights grounding questions in descriptions and QA pairs from simulated actors and objects [54].Source: §2 · p. 2

Describing Visual Content

• Image and video captions describe visual content but can often be non-specific; VQA questions require detailed specific image information, for which generic image captions are of little use [53],.Source: §2 · p. 2

Other Vision+Language Tasks

• Coreference resolution and referring-expression generation identify objects with concrete concepts such as color and location, while visual questions and answers reveal a richer variety of visual concepts.The related tasks include coreference resolution, and referring expressions,.Source: §2 · p. 2

Real Images / Abstract Scenes

• The dataset uses 123,287 COCO training and validation images and 81,434 test images, plus 50K adjustable abstract scenes built from human models, objects, and animals.The abstract-scene clipart was designed to mirror real images while letting researchers focus on high-level reasoning without complex visual recognition.Source: §3 · p. 3

Splits / Captions

• Real-image splits follow MS COCO, while abstract scenes use 20K/10K/20K train/validation/test splits; both image types have five captions each.For the VQA challenge, test-standard is the default reporting split, test-reserve checks for overfitting, and test-challenge determines winners.Source: §3 · p. 3

Questions

• The smart-robot interface elicited questions spanning visual recognition and commonsense, while requiring the image to answer them.Pilot studies compared prompts about a toddler, alien, or smart robot; workers were asked to stump the robot, which could recognize objects, scenes, expressions, poses, and properties.Source: §3 · p. 4
• Three unique workers contributed questions per image or scene, with prior questions shown to encourage diversity, yielding over 0.76M questions.Source: §3 · p. 4

Answers

• Each question received 10 answers from unique workers who had not asked it, accommodating valid answer variation and disagreement.Workers were instructed to give brief, matter-of-fact phrases and also rate whether they could answer correctly as no, maybe, or yes.Source: §3 · p. 4

Evaluation

• Open-ended answers are scored by exact-response agreement, with full accuracy when at least three workers gave the same answer.Responses are lowercased, numbers converted to digits, and punctuation and articles removed; the paper avoids soft similarity metrics, noting that 89.32% of answers are single words.Source: §3 · p. 4
• Multiple-choice questions offer 18 unique, randomized candidates drawn from correct, plausible, popular, and random answers, and more than one option can score above zero.Plausible distractors come from answers given without the image, helping test whether the image is needed; accuracy reflects the number of workers who supplied an option, divided by three and capped at one.Source: §3 · p. 4

4.1 Questions

• Question-type distributions are similar for real images and abstract scenes, with varied forms, and most questions contain four to ten words.Examples of question types include “What is...”, “Is there...”, “How many...”, and “Does the...”.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 Answers

• 89.32% of real-image answers and 90.51% of abstract-scene answers are one word, yet response patterns range from yes/no to diverse or specialized answers.Yes/no questions account for 38.37% of real-image questions and 40.66% of abstract-scene questions; inter-human agreement is 83.30% and 87.49%, respectively, and rises with confidence. Agreement exceeds 95% for yes/no questions but is below 76% for other questions; exact string matching does not account for synonyms or plurality.Source: §4.2 · pp. 5–7, Figure 5 · p. 6, Figure 6 · p. 7

4.3 Commonsense Knowledge

• About 21% of questions were answered correctly without images, showing that commonsense alone is insufficient and visual information is critical to VQA.For 47.43% of questions, at least three of ten subjects judged commonsense necessary; workers’ perceived average age required was 8.92, and average commonsense degree was 31.01%. The two rankings of question types correlated at 0.58, while the age annotations reflect worker perceptions rather than objective requirements.Source: §4.3 · p. 7

4.4 Captions vs. Questions

• Captions improved answer accuracy over questions alone but remained significantly less accurate than images, indicating that captions miss information needed for correct answers.Caption noun, verb, and adjective distributions differed significantly from those in questions and answers for both real images and abstract scenes (Kolmogorov-Smirnov test, p < .001).Source: §4.4 · pp. 7–8

5.1 Baselines

• Baselines include random answers, a constant “yes” prior, per-question-type priors, and nearest-neighbor question-image pairs; multiple-choice priors and neighbors use Word2Vec cosine similarity.The nearest-neighbor open-ended answer is the most frequent ground-truth answer among retrieved question-image pairs.Source: §5.1 · p. 8

5.2 Methods

• The model uses a two-layer LSTM for questions and ℓ2-normalized VGGNet features for images, fuses their common-space embeddings element-wise, and predicts among 1,000 answers covering 82.67% of train+val answers.The fused representation passes through a two-hidden-layer MLP and softmax; VGGNet parameters are frozen.Source: §5.2 · pp. 8–9, Figure 8 · p. 9

5.3 Results

• On test-dev, LSTM Q reaches 48.76% open-ended and 54.75% multiple-choice accuracy, while BoW Q reaches 48.09% and 53.68%, respectively; both beat nearest neighbor.Nearest neighbor scores 42.70% open-ended and 48.49% multiple-choice. Language-only methods perform surprisingly well, plausibly by exploiting question-type priors such as answering “yellow” to “What color is the banana?” [56].Source: Table 2 · p. 9, §5.3 · pp. 9–10
• The best model, deeper LSTM Q + norm I, achieves 58.16% open-ended and 63.09% multiple-choice accuracy on test-standard, outperforming vision- and language-alone baselines; all methods remain below human performance.The model was selected using test-dev accuracy.Source: §5.3 · p. 10
• The model performs well on common visual-object answers but poorly on counts, especially higher counts; accuracy also falls from 61.07% for age-3–4 questions to 47.83% for age-18+ questions.On annotated validation questions, its estimated age and commonsense levels are 4.74 years and 17.35%, versus averages of 8.98 years and 31.23% required, respectively.Source: §5.3 · p. 10
• Image features help on scene-level questions such as “What sport” and “What animal,” but add no information for reasoning-heavy types such as “Is the” or “How many”; all question-type results trail humans.The analysis reports open-ended test-dev results by question type, with Q+C results on validation.Source: Table 3 · p. 10, §5.3 · p. 10
• On test-dev, ℓ2 image normalization improves open-ended and multiple-choice accuracy by 0.16% and 0.24%, while element-wise fusion beats concatenation by 0.95% and 1.24%, respectively.Using K = 1000 instead of K = 500 improves accuracy by 0.82% open-ended and 1.92% multiple-choice; K = 2000 instead of K = 1000 improves it by 0.40% and 1.16%, respectively.Source: §5.3 · pp. 10–11, Table 4 · p. 12
• Truncating question vocabulary at words occurring at least 5 or 11 times improves accuracy over using all words, while training on the filtered dataset reduces accuracy.Vocabulary cutoffs of 5 and 11 improve open-ended/multiple-choice accuracy by 0.24%/0.17% and 0.06%/0.02%, respectively. The filtered dataset retains 344,600 questions and reduces accuracy by 1.13% open-ended and 1.88% multiple-choice.Source: §5.3 · pp. 11–12, Table 4 · p. 12

6 VQA CHALLENGE AND WORKSHOP

• An evaluation server accepts test-set submissions and returns an accuracy breakdown.The section says results may be uploaded for the test set.Source: §6 · p. 12
• The authors are organizing an annual VQA challenge and workshop, with the first workshop planned for CVPR 2016.They state that these events are intended to facilitate systematic progress in VQA.Source: §6 · p. 12
• Papers reporting VQA results should provide test-standard accuracies and compare them with the corresponding test2015 leaderboards.Accuracies can be calculated using either test2015 or Challenge test2015; the section also says Table 5 compares the best model’s real-image open-ended and multiple-choice accuracies with other entries as of October 28, 2016.Source: §6 · p. 12

• VQA asks systems to provide accurate natural-language answers to open-ended questions about images, using a dataset of over 250K images, 760K questions, and around 10M answers.Source: §7 · p. 12
• The dataset’s open-ended, non-task-specific questions span diverse answers and require computer vision, natural-language processing, and commonsense reasoning.Source: §7 · p. 12
• Task-specific questions, such as those gathered from visually impaired users or within a domain like sports, may help enable practical VQA applications.Bigham et al. created an application in which visually impaired users capture images and ask open-ended questions answered by human subjects; these questions can rarely be answered using generic captions.Source: §7 · p. 12
• The authors frame VQA as a way to push the frontiers of “AI-complete” problems while remaining amenable to automatic evaluation.Source: §7 · p. 12

Appendix contents

• It covers caption-versus-Q&A analysis, “What is” visualizations, human accuracy on multiple-choice questions, and VQA baseline details.These are appendix items I–IV.Source: APPENDIX OVERVIEW · p. 12
• It adds discussion of model “Age” and “Commonsense,” the abstract scene dataset, dataset-collection interfaces, and top dataset answers.These are appendix items V–VIII.Source: APPENDIX OVERVIEW · p. 12
• A leaderboard figure shows test-standard accuracies for the VQA Real Image Challenge in open-ended and multiple-choice formats, in an October 28, 2016 snapshot.Source: Figure 13 · p. 13
• The appendix also includes additional examples from the VQA dataset.This is appendix item IX.Source: APPENDIX OVERVIEW · p. 13

APPENDIX I: CAPTIONS vs. QUESTIONS

• For both real images and abstract scenes, normalized caption and question–answer word distributions differ significantly for nouns, verbs, and adjectives (Kolmogorov–Smirnov test, p < .001), supporting complementary perspectives on visual scenes.The comparison combines questions and answers and normalizes word frequencies per image.Source: Appendix I · p. 13
• Captions use adjectives for visible properties such as material and pattern, whereas question–answer adjectives more often express typicality or commonsense properties such as “edible” and “unsafe”.Source: Appendix I · p. 13
• Question–answer nouns include “ethnicity” and “hairstyle,” while caption nouns include plural visible objects and groups; “man” and “people” occur in both.Source: Appendix I · p. 13
• Questions and answers contain a higher proportion of specific spatial relations such as “in” and “on,” while captions contain more general relations such as “with” and “near.”The proportions are shown for real images and abstract scenes.Source: Appendix I · p. 14, Figure 14 · p. 15

APPENDIX II: “WHAT IS” ANALYSIS

• Fig. 21 groups “What is” questions by their first five words for both real images and abstract scenes.Source: APPENDIX II: “WHAT IS” ANALYSIS · p. 17
• The questions reference a diverse range of objects.Source: APPENDIX II: “WHAT IS” ANALYSIS · p. 17
• Some questions express relations between objects, including “holding” and “sitting on”.Source: APPENDIX II: “WHAT IS” ANALYSIS · p. 17
• Fig. 22 shows answer distributions for “What is” questions ending in different words.Source: APPENDIX II: “WHAT IS” ANALYSIS · p. 17
• Questions ending in “eating” can elicit answers such as “pizza”, “watermelon” and “hot dog”.Source: APPENDIX II: “WHAT IS” ANALYSIS · p. 17
• Answers vary for endings such as “for?” or “picture?”, while “holding?” can elicit the intuitive answer “umbrella”.Source: APPENDIX II: “WHAT IS” ANALYSIS · p. 17

CURACY

• Multiple-choice accuracy was about the same for yes/no questions and higher for “other” questions by ≈15% on real images and ≈11% on abstract scenes.The authors note that “other” questions may be ambiguous, which may explain the increase.Source: CURACY · p. 17
• Human multiple-choice accuracy was measured from three answers per question on a random subset of 3,000 questions for real images and abstract scenes.Table 6 also reports inter-human agreement for open-ended answers.Source: CURACY · p. 17

APPENDIX IV: DETAILS ON VQA BASELINES

• The per-Q-type prior predicts each question type’s most popular answer, with types determined by question openings and requiring at least 30 training questions.Question types and popular answers were computed on the real-image training set.Source: APPENDIX IV: DETAILS ON VQA BASELINES · p. 17
• The nearest-neighbor baseline retrieves training questions by Skip-Thought cosine similarity, then selects the answer associated with the most similar image-question pair.It compares images using fc7 cosine similarity and uses k = 4, selected on test-dev; Skip-Thought outperformed bag-of-words and Word2Vec features.Source: APPENDIX IV: DETAILS ON VQA BASELINES · p. 17

OF OUR MODEL

• The model’s estimated age is the accuracy-weighted average of question ages on validation questions with age annotations.The best model was the deeper LSTM Q + norm I, selected using VQA test-dev accuracies.Source: OF OUR MODEL · p. 17
• The model’s estimated degree of commonsense is the accuracy-weighted average of question-level commonsense ratings on annotated validation questions.The ratings indicate whether a question requires commonsense to answer it.Source: OF OUR MODEL · p. 17

• The appendix shows a subset of dataset objects in Fig. 23 (left), additional generated scenes in Fig. 28, and the creation interface in Fig. 23 (right).Subjects used the interface to create scenes.Source: APPENDIX VI: ABSTRACT SCENES DATASET · p. 17
• Objects could be flipped horizontally and scaled, with scale determining their rendering order.Source: APPENDIX VI: ABSTRACT SCENES DATASET · p. 17
• Many objects had pose or type attributes; most animals had five discrete poses, while humans had eight expressions and continuously adjustable paperdoll poses.Source: APPENDIX VI: ABSTRACT SCENES DATASET · p. 17

APPENDIX VII: USER INTERFACES

• Question collectors saw earlier questions and were asked to ask something different, a design intended to increase question diversity.Fig. 24 shows the interface for collecting a third question after workers had seen previous questions.Source: APPENDIX VII: USER INTERFACES · p. 17, Figure 24 · p. 19
• Answers were collected with the image visible, while a separate no-image interface elicited plausible but incorrect answers using commonsense.The no-image answers also helped assess how accurately questions could be answered using common-sense knowledge alone.Source: APPENDIX VII: USER INTERFACES · p. 17, Figure 25 · p. 20, Figure 26 · p. 20

APPENDIX IX: ADDITIONAL EXAMPLES

• Random examples cover questions and answers for the real-image MS COCO dataset, including answers collected with and without the image shown.The examples include only a subset of answers for each question.Source: APPENDIX IX: ADDITIONAL EXAMPLES · p. 21, Figure 27 · p. 22
• Random examples also show questions and answers for the abstract-scene dataset, with answers collected both while viewing and without viewing the image.The examples show a subset of image-viewing answers alongside answers given without the image.Source: APPENDIX IX: ADDITIONAL EXAMPLES · p. 21, Figure 28 · p. 23
• The appendix includes random multiple-choice question examples from both the real-image and abstract-scene datasets.Source: APPENDIX IX: ADDITIONAL EXAMPLES · p. 21, Figure 29 · p. 24