PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Question Answering: A Survey of Methods and Datasets

Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton van den Hengel

arXiv:1607.05910v1cs.CV

TL;DR

VQA asks systems to answer natural-language questions about images, often using visual reasoning and knowledge beyond the image. This survey synthesizes VQA methods and datasets, analyzes Visual Genome scene graphs and knowledge-base expansion, and discusses future research directions. It identifies broad method and dataset patterns while reporting that scene-graph vocabulary expansion can cover 79.58% of questions.

  • Problem

    VQA requires reasoning over images and external knowledge, while existing datasets often emphasize visual-level questions and require little external knowledge.

  • Method

    The survey classifies VQA methods, reviews datasets, analyzes Visual Genome scene graphs, and expands them with triples retrieved from DBpedia, WebChild, and ConceptNet.

  • Results

    79.58% of questions could potentially be answered from scene-graph vocabularies expanded with external knowledge, compared with about 40% of answers directly matching original scene-graph elements.

  • Takeaways & Limitations

    The survey points toward richer datasets, stronger connections to structured knowledge bases, and continued use of NLP models for VQA.

  • Takeaways & Limitations

    Current knowledge-base VQA approaches handle limited hand-coded question types or may retrieve question-independent and irrelevant information.

Abstract

from arXiv · show

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.

1. Introduction

VQA connects computer vision and NLP by asking systems to answer natural-language questions about images, often requiring visual reasoning and external knowledge. The survey organizes methods, reviews datasets, analyzes Visual Genome scene graphs, and identifies future directions involving richer datasets and knowledge bases.

  • VQA presents an image and textual question, requiring a short answer; variants include binary, multiple-choice, and fill-in-the-blank settings.
  • Unlike image captioning, VQA frequently requires common-sense or encyclopedic information absent from the image.This makes VQA a multimodal reasoning task while retaining relatively simple answer-based evaluation.
  • The survey emerged as VQA attracted attention because mature computer-vision and NLP techniques and large-scale datasets enabled substantial recent progress.It aims to provide a comprehensive overview of models, datasets, and promising future directions.
  • The survey reviews four categories of VQA methods, including joint embeddings, attention mechanisms, compositional models, and additional architectural approaches.Joint embedding methods commonly combine CNNs and RNNs to place images and questions in a shared feature space before answer classification.
  • The dataset review compares resources by size, reasoning demands, and required information beyond the image, finding that most favor visual questions with little external knowledge.The survey concludes that more varied and sophisticated datasets will eventually be required.
  • Visual Genome scene-graph analysis finds that about 40% of answers directly match graph elements, while linking graphs to external knowledge bases raises the matching rate.The survey uses this analysis to motivate stronger connections between structured annotations and external knowledge.

2. Methods for VQA

The survey organizes VQA methods by how they connect visual and textual modalities, from joint embeddings and attention to modular architectures. It also compares their reported performance and identifies limits involving reasoning, question complexity, and external knowledge.

  • 2.1 Joint embedding approaches: VQA methods commonly map image and question representations into a shared feature space using CNNs and RNNs.The resulting features are combined by either a classifier for predefined short answers or a recurrent decoder for variable-length phrases.
  • 2.1 Joint embedding approaches: Joint-embedding systems include encoder–decoder LSTMs, classifiers, and multimodal pooling methods such as MCB and MRN.Neural-Image-QA generates answers with a decoder LSTM, whereas VIS+LSTM formulates answering as classification; MCB uses compact bilinear pooling and MRN learns multimodal residual representations.
  • 2.2 Attention mechanisms: Attention mechanisms select spatial image features using information from the question and image, enabling the output stage to focus on relevant regions.Variants include question-guided attention, multi-hop attention, object-region selection, and hierarchical co-attention.
  • 2.2 Attention mechanisms: Attention improves reported performance over global image features, with multiple attention layers and co-attention providing further gains in several evaluations.The survey reports improvements over global features and describes gains from stacked attention and hierarchical co-attention models.
  • Performance and limitations: The survey identifies limited complex questions, opaque answer mappings, and missing external knowledge as important constraints on current approaches.Attention provides little or no benefit for binary questions, NMNs are constrained by benchmark question complexity, and joint embeddings leave the nature of learned reasoning unclear.
  • 2.3 Compositional Models: Neural Module Networks parse question structure into independent, composable modules whose arrangement varies with each problem instance.Modules can perform operations such as attending to concepts, shifting attention according to relations, intersecting attentions, and measuring whether the result is non-empty.

Motivation

The survey examines VQA methods spanning memory-augmented architectures, external knowledge bases, and approaches that combine visual processing with language representations. It compares their mechanisms, reasoning capabilities, and evaluation results, while identifying dataset-scale and representation limitations.

  • Memory-augmented architectures: Dynamic Memory Networks use modular input, question, episodic-memory, and answer components to retrieve image facts relevant to a question.For VQA, image features are extracted from patches with a VGG CNN and processed sequentially by a GRU; episodic memory attends to image regions.
  • Memory-augmented architectures: Noh et al. supervise multiple memory passes during training but use only one pass during test-time inference.
  • Memory-augmented architectures: Dynamic Memory Networks perform competitively across question types, but counting is weaker, likely because fixed image patches may cross object boundaries.
  • Structured knowledge bases: Knowledge-base methods decouple reasoning from stored knowledge by representing facts as linked triples in machine-readable graphs.Examples include DBpedia, Freebase, YAGO, OpenIE, NELL, WebChild, and ConceptNet; facts use the form (arg1,rel,arg2).
  • Structured knowledge bases: Ahab maps images and questions to queries over a constructed DBpedia graph rather than directly mapping them to answers.Visual concepts are extracted with CNNs and associated with similar DBpedia nodes.
  • Structured knowledge bases: Explicit knowledge representations can expose reasoning chains or supporting facts, unlike monolithic neural networks that provide little computational insight.
  • Evaluation: On small external-knowledge benchmarks, Ahab reaches 69.6% versus 44.5% overall accuracy, while FVQA reaches 58.19% versus 23.37% top-1 accuracy.Both methods target questions requiring external knowledge, but their evaluations use datasets with limited question types and small scale.

3. Datasets and evaluation

VQA datasets differ in scale, reasoning demands, and reliance on information beyond the image. The survey compares major datasets and finds substantial variation in question formats, annotations, and benchmark biases.

  • Dataset dimensions: VQA datasets vary in size, required reasoning, and dependence on non-visual information such as common sense or subject-specific knowledge.Some questions require detecting one object, while others require inference over multiple facts or concepts.
  • Datasets of natural images: DAQUAR contains 12,468 question/answer pairs from 1,449 indoor RGBD images, but restricts answers to 16 colors and 894 object categories.It includes both template-generated and human-collected questions, with 6,794 training and 5,674 testing pairs.
  • Datasets of natural images: COCO-QA contains 123,287 images and categorizes questions by object, number, color, and location, but 23.29% of test questions also occur in training.The questions were automatically generated from image descriptions, contributing to a high repetition rate.
  • Datasets of natural images: FM-IQA uses human-generated questions over 123,287 COCO images, producing greater question diversity than earlier datasets.Annotators could ask any question related to each image.
  • Datasets of natural images: Visual Genome QA contains 1.7 million question/answer pairs and uniquely provides scene-graph annotations describing visual elements, attributes, and relationships.Visual7W extends this resource with four-way multiple-choice answers and bounding boxes grounding mentioned objects.
  • Evaluation and biases: VQA benchmarks exhibit strong language biases: half of all questions can be answered from the question alone, rising above 78% for binary questions.Balanced complementary scenes reduce this shortcut, making blind models perform close to chance.

4. Structured scene annotations for VQA

The survey evaluates whether Visual Genome scene graphs contain enough information to answer VQA questions directly, and tests whether external knowledge can fill the gaps. Scene graphs provide useful structure but omit many attributes and concepts needed for VQA.

  • Scene-graph representation: Scene graphs represent visual elements as nodes and their relationships as directed edges, providing structured annotations for each Visual Genome image.Nodes may encode objects, attributes, and actions.
  • Direct-answer test: The direct-answer test builds an image vocabulary from scene-graph node labels and matches each answer against individual words or word combinations.The procedure assumes a perfect vision system could recover the annotated graph.
  • Direct-answer test: 40.02% of Visual Genome answers can be found directly in scene graphs, while 53% cannot; 7% are counting answers excluded from further analysis.The remaining unanswered fraction is high despite the apparent detail of scene-graph descriptions.
  • Missing information: Unrepresented answers include fine-grained colours and materials, global labels such as locations, and higher-level concepts involving actions, reasons, weather, or time.The question-type analysis reports these categories among answers absent from scene graphs.
  • Implications: Current scene graphs are rich intermediate abstractions but are not comprehensive enough for VQA, so answering some questions still requires access to the image or complementary knowledge.The survey identifies low-level visual attributes and global scene information as important omissions.
  • Knowledge expansion: Connecting scene-graph labels to DBpedia, WebChild, and ConceptNet expands the representation and raises answer coverage from 40.02% to 79.58%.The experiment queries the knowledge bases for triples and appends the resulting facts to scene-graph nodes.

5. Discussion and future directions

The discussion frames VQA as an open-ended reasoning challenge whose datasets and methods remain limited in scope. It highlights external knowledge, dataset design, and NLP integration as central directions for progress.

  • Scope of VQA: Open-world, free-form VQA remains far from its ultimate AI-complete goal, while constrained formats such as multiple choice and short answers are attainable intermediate objectives.The survey notes that reduced formats are easier to evaluate and may better represent current progress.
  • Dataset design: Dataset collection choices affect question complexity, the number of facts involved, and the amount of information that must be inferred beyond the image.The review covers human and semi-automatic collection protocols and constraints on regions, objects, and question types.
  • External knowledge: Many VQA questions require external knowledge, motivating methods that connect visual reasoning with structured knowledge bases.The discussion treats external knowledge as a recurring requirement for questions over images.
  • External knowledge: Existing knowledge-base VQA approaches are limited by hand-coded question templates or question-independent retrieval that may include irrelevant information.The survey identifies memory-augmented neural networks as a potentially scalable way to select relevant knowledge adaptively.
  • NLP integration: Textual question answering offers established information-retrieval and semantic-parsing approaches, while VQA increasingly adopts NLP concepts alongside computer-vision methods.Examples of transferred NLP concepts include word embeddings and related language-processing tools.

6. Conclusion

The article surveys VQA methods and datasets, then identifies external knowledge and NLP tools as promising directions. Its conclusion connects these directions to VQA and broader visual scene understanding.

  • Conclusion: The survey reviews common-space image–question representations, attention, modular and memory-augmented architectures, and datasets differing in question type and difficulty.It combines a state-of-the-art review with discussion of future research directions.
  • Conclusion: The authors suggest scaling structured knowledge-base integration and continuing to explore NLP tools for VQA and visual scene understanding.These are presented as promising directions for ongoing and future work.
Loading 1607.05910v1…