PaperlayerView exact revision on arXiv

Source-linked AI summary

An Analysis of Visual Question Answering Algorithms

Kushal Kafle, Christopher Kanan

arXiv:1703.09684v2cs.CVcs.AIcs.CL

TL;DR

Existing VQA benchmarks make algorithm comparison difficult because biased question distributions and simple accuracy can reward easy, frequent question-types. The paper introduces TDIUC, a 12-category natural-image dataset with absurd questions and bias-compensating metrics, then uses it to analyze VQA models. The resulting evaluation enables deeper understanding of algorithm behavior, while Visual Genome’s non-exhaustive and duplicated annotations constrain its use.

  • Problem

    Existing VQA datasets and evaluation metrics contain biases that make aggregate performance difficult to interpret and algorithms difficult to compare.

  • Method

    The paper creates TDIUC with 12 question-types, absurd questions, and evaluation metrics that compensate for biased question distributions, then evaluates baseline and state-of-the-art VQA algorithms.

  • Results

    TDIUC’s absurd questions and new evaluation metrics enable a deeper understanding of VQA algorithm behavior.

  • Takeaways & Limitations

    The benchmark enables more nuanced comparisons of VQA algorithms and helps identify the benefits of specific techniques.

  • Takeaways & Limitations

    Visual Genome annotations can be non-exhaustive and duplicated, so TDIUC uses them only for color and positional reasoning questions.

Abstract

from arXiv · show

In visual question answering (VQA), an algorithm must answer text-based questions about images. While multiple datasets for VQA have been created since late 2014, they all have flaws in both their content and the way algorithms are evaluated on them. As a result, evaluation scores are inflated and predominantly determined by answering easier questions, making it difficult to compare different methods. In this paper, we analyze existing VQA algorithms using a new dataset. It contains over 1.6 million questions organized into 12 different categories. We also introduce questions that are meaningless for a given image to force a VQA system to reason about image content. We propose new evaluation schemes that compensate for over-represented question-types and make it easier to study the strengths and weaknesses of algorithms. We analyze the performance of both baseline and state-of-the-art VQA models, including multi-modal compact bilinear pooling (MCB), neural module networks, and recurrent answering units. Our experiments establish how attention helps certain categories more than others, determine which models work better than others, and explain how simple models (e.g. MLP) can surpass more complex models (MCB) by simply learning to answer large, easy question categories.

1. Introduction

The paper argues that existing VQA benchmarks inadequately measure broad visual understanding because biased question distributions make aggregate scores misleading. It introduces a 12-category benchmark, bias-compensating metrics, balanced object-presence questions, and absurd questions for more nuanced algorithm analysis.

  • Motivation: A good VQA benchmark should test a wide range of computer vision tasks in an unbiased manner.
  • Motivation: Existing VQA datasets contain biases and lack meaningful question categories, making algorithm abilities difficult to compare.A method can appear strong by excelling on common categories such as color questions while performing poorly on spatial reasoning.
  • Contributions: The paper creates a new VQA benchmark whose questions are divided into 12 task-based categories.
  • Contributions: The proposed evaluation metrics compensate for dataset bias, while balanced object-presence questions test whether distributional balance improves learning.
  • Contributions: Absurd questions require systems to determine whether a question is valid for the image.

2. Background

Prior natural-image VQA datasets exhibit content, answer, and question-type biases, while simple accuracy overweights frequent categories. Earlier synthetic bias-analysis datasets provide useful controls but are limited in scale, realism, or visual demands.

  • Prior natural-image datasets: Six major natural-image VQA datasets were released between 2014 and 2016, but all exhibit important biases.
  • Prior natural-image datasets: Question-only models achieve 50% accuracy on COCO-VQA, suggesting that some answers are predictable without image information.
  • Evaluation bias: Simple accuracy overweights frequent question-types: improving Is/Are accuracy by 15% raises COCO-VQA overall accuracy by over 5%, whereas answering all Why/Where questions correctly raises it by only 4.1%.
  • Evaluation bias: The paper addresses coarse or absent question-type reporting by categorizing questions meaningfully, computing sub-scores, and incorporating them into evaluation metrics.
  • Synthetic datasets: SHAPES and CLEVR study reasoning with synthetic geometric images, but their settings are not representative of natural imagery and downplay visual reasoning.

3. TDIUC for Nuanced VQA Analysis

TDIUC is a natural-image VQA benchmark designed for fine-grained analysis, with 12 explicitly defined question-types and bias-aware evaluation. Its questions combine imported, automatically generated, and human-created examples, including absurd queries.

  • TDIUC benchmark: TDIUC explicitly assigns questions to 12 distinct categories, enabling category-level measurement of which questions are easy or hard.
  • Question-types: The categories span classical and higher-level tasks, including object presence, recognition, counting, attributes, activities, sports, positional reasoning, scenes, sentiment, affordances, and absurd questions.
  • Question construction: Questions come from imported COCO-VQA and Visual Genome examples, automatically generated annotation-based pairs, and human annotators.
  • Question construction: Annotation-based generation uses multiple templates populated from semantic regions, objects, attributes, and relationships; counting questions use eight templates.
  • Absurd questions: Absurd questions pair an image with questions about absent objects and require the answer Does Not Apply when the query is nonsensical for that image.
  • Question construction: Visual Genome is used only for color and positional reasoning because its annotations can be non-exhaustive and duplicated.

4. Proposed Evaluation Metric

The paper addresses skewed question-type distributions by evaluating each of 12 types separately and combining those scores with arithmetic and harmonic mean-per-type metrics. It also normalizes for answer-frequency imbalance within each type.

  • Motivation: Simple accuracy makes rare question-types difficult to assess when test questions are unevenly distributed across types.The paper therefore proposes multiple measures that compensate for bias and skewed distributions.
  • Per-type metrics: Accuracy is computed separately for each of the 12 question-types before producing a unified score.This preserves per-type performance while enabling an overall comparison.
  • Per-type metrics: Arithmetic MPT averages the 12 per-type accuracies, whereas Harmonic MPT measures whether performance is high across all types.The harmonic mean is sensitive to low performance in any question-type.
  • Answer normalization: Normalized metrics address answer imbalance by averaging accuracy across unique answers within each question-type.For example, the answer ‘two’ covers over 35% of counting questions.

5. Algorithms for VQA

The reviewed VQA algorithms generally combine CNN image features with recurrent or bag-of-words question representations, while differing in interaction, attention, and reasoning mechanisms. The section contrasts simple concatenation baselines with spatially attentive and compositional models.

  • Common formulation: Most VQA systems formulate answering as classification over answer categories using CNN image features and RNN or bag-of-words question representations.The input is an image-question pair, and the output is an answer category.
  • Baseline models: Linear and MLP baselines concatenate question and image embeddings, with image features taken from a CNN’s last hidden layer.These models provide simple reference systems for comparing more complex architectures.
  • Baseline models: Simple approaches can remain competitive with complex attentive models.This comparison motivates testing whether architectural complexity consistently improves VQA performance.
  • Attention: Spatial-attention models weight visual features according to question relevance instead of using only global CNN features.For color questions, attention can emphasize image regions containing the queried object and suppress others.
  • Compared architectures: MCB combines spatial attention with an efficient approximation of the image-question outer product so feature elements interact.It uses an LSTM to embed the question and avoids the cost of explicitly computing the high-dimensional outer product.
  • Compared architectures: NMN composes discrete modules arranged from a parsed S-expression to answer compositional questions.Modules such as find(x) produce intermediate outputs such as heat maps for detecting queried entities.
  • Compared architectures: RAU performs multi-step reasoning through answering blocks that use the image, question, and previous LSTM output.Its inference steps form part of a larger LSTM that progressively reasons about the question.

6. Experiments

The experiments retrain baseline and state-of-the-art VQA methods on TDIUC and report performance across its 12 question-types. The setup compares image-only, question-only, joint, MLP, MCB, NMN, and related models under specified visual-feature configurations.

  • Experimental setup: The study trains multiple baseline and state-of-the-art VQA methods on TDIUC.The evaluation reports results for the listed model families under a common benchmark.
  • Compared models: The evaluated systems include question-only, image-only, joint linear, MLP, MCB, attentive MCB, NMN, and RAU variants.QUES and IMG isolate question and image information, while Q+I and MLP use both modalities.
  • Feature configuration: All models use ResNet-152 image features computed from 448×448 images.This provides a shared visual representation across the evaluated systems.
  • Baseline configuration: QUES, Q+I, and MLP use 4800-dimensional skip-thought question vectors and normalized ResNet-152 pool5 image features.The MLP has three ReLU hidden layers with 6000, 4000, and 2000 units, followed by softmax.
  • State-of-the-art configuration: Publicly available code trains MCB, MCB-A, NMN, and RAU with default experimental settings, except that NMN and RAU use ResNet-152 visual representations.The visual-representation upgrade is the stated deviation from the default code configurations.
  • Reported results: Table 3 reports accuracy for each of TDIUC’s 12 question-types, while appendix Table 5 reports scores normalized by mean-per-unique-answer.The two tables provide ordinary per-type and normalized evaluations.

7. Detailed Analysis of VQA Models

Detailed experiments show that evaluation design and model architecture substantially affect apparent VQA performance. Balanced metrics expose answer-distribution bias, while attention and model capacity help selectively across question types.

  • Performance by question type: Scene recognition, sport recognition, and object presence exceed 90% accuracy under MPT, while counting remains low across all methods.Subordinate object recognition exceeds 80%, whereas weaker categories require further analysis of data volume, bias, and difficulty.
  • Effects of evaluation metrics: MLP surpasses MCB and NMN on simple accuracy, but MPT shows NMN and MCB outperforming MLP because simple accuracy favors populous categories.The authors interpret this pattern as evidence that MLP is prone to overfitting.
  • Effects of evaluation metrics: MCB-A’s subordinate object recognition falls from 85.54% unnormalized to 23.22% normalized, revealing reliance on skewed answer distributions.Scene recognition similarly drops from 93.06% to 38.53%; normalized and unnormalized scores therefore diagnose generalization beyond answer statistics.
  • Rare answers: MCB trained only on TDIUC-Tail outperforms MCB trained on all TDIUC across every question type, showing that rare answers can be learned.The authors state that normalized accuracy discourages reliance on answer statistics and may be useful for deployment.
  • Balancing object presence: After TDIUC training, MCB-A reaches 95.02% on “yes” and 92.26% on “no” object-presence questions, compared with 86.3% and 11.2% after COCO-VQA training.The comparison shows how dataset answer bias affects generalization in object-presence questions.
  • Advantages of attentive models: Attention most improves color, attribute, absurd, and counting categories because local spatial features help models locate specified objects.MCB-A computes attention from spatially local features, and RAU shows similar increments.
  • Compositional and modular approaches: NMN performs better than MLP under MPT and N-MPT but shows no substantial question-type advantage, partly because its S-expression parser can produce misleading parses.RAU avoids rigid parses through end-to-end multi-hop inference and performs well on absurd and other categories.

8. Conclusion

The conclusion presents TDIUC as a structured benchmark for rigorous VQA analysis. Its question categories, absurd questions, and bias-compensating metrics provide deeper insight into algorithm behavior.

  • 8. Conclusion: TDIUC defines 12 question types, includes absurd questions, and supports rigorous analysis of recent VQA algorithms.The dataset is paired with evaluation metrics designed to compensate for VQA dataset biases.
  • 8. Conclusion: Absurd questions and the proposed evaluation metrics enable deeper understanding of VQA algorithm behavior.

Appendices

The appendix provides additional details about TDIUC construction and statistics omitted from the main paper because of space constraints.

  • Appendices: The appendix adds TDIUC dataset-creation details and statistics omitted from the main paper.The omission is attributed to inadequate space.

A.1. Questions using Visual Genome Annotations

Visual Genome annotations support Color Attribute and Positional Reasoning questions, but annotation ambiguity and strict filtering limit their use.

  • A.1. Questions using Visual Genome Annotations: Visual Genome annotations were used to create Color Attribute and Positional Reasoning questions.The source contains object, attribute, and relationship metadata for generating these question types.
  • A.1. Questions using Visual Genome Annotations: Color Attribute questions use the template ‘What color is the <object>?’.The construction relies on attribute metadata associated with annotated objects.
  • A.1. Questions using Visual Genome Annotations: The construction process filters objects to avoid multiple attributes, ambiguous names, duplicate instances, and unreliable synset matches.These restrictions address cases such as differently named objects and objects whose names already include a color.
  • A.1. Questions using Visual Genome Annotations: Applying stringent criteria produced only a small number of Visual Genome questions compared with other sources.The source-wise question counts are reported in Table 4.

A.2. Answer Distribution

Answer distributions vary substantially across question types, creating opportunities for inflated scores when common answers dominate; balancing and image-level splitting address evaluation and leakage concerns.

  • A.2. Answer Distribution: Counting, scene recognition, and sentiment understanding concentrate many questions among a few top answers.By contrast, positional reasoning and object utility and affordances have more varied answer distributions.
  • A.2. Answer Distribution: Figure 3 reports relative answer frequencies within each question type, so bar size does not indicate the same number of examples across types.Large bars in less populated categories can represent relatively few training examples.
  • A.2. Answer Distribution: Balancing the top two answers reduces the incentive for mode guessing, including under simple accuracy.For counting, ‘two’ is repeated almost as often as ‘one’.
  • A.2. Answer Distribution: The train and test partitions keep questions associated with the same image together and target a 70% train, 30% test/validation split.Images are clustered by source before applying source-specific splitting rules.

B. Additional Experimental Results

Additional experiments report normalized question-type scores and evaluate models on TDIUC-Tail, isolating uncommon answers and comparing training on the full dataset versus the tail subset.

  • B. Additional Experimental Results: Normalized accuracy is computed separately for each unique answer within a question type and then averaged.These detailed normalized scores are provided for each question type.
  • B. Additional Experimental Results: Table 5 reports normalized accuracy by question type and five overall metrics for the VQA models.Arithmetic and harmonic overall scores average the corresponding reported sub-scores; an asterisk marks training without absurd questions.
  • B. Additional Experimental Results: Tables 6 and 7 compare MCB trained on full TDIUC with MCB trained and evaluated only on TDIUC-Tail.Table 6 presents unnormalized scores, while Table 7 presents normalized scores.
  • B. Additional Experimental Results: TDIUC-Tail contains questions whose answers occur fewer than 1000 times and excludes absurd and object presence questions.The additional results evaluate models on this uncommon-answer subset.
Loading 1703.09684v2…