Source-linked AI summary
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh
TL;DR
Visual Question Answering models can achieve superficial success by relying on language priors rather than image content. This paper balances VQA with complementary image pairs and finds a significant performance drop for state-of-the-art models on the balanced dataset.
Problem
Language priors can produce superficial VQA performance without models truly understanding visual content.
Method
The paper constructs a balanced VQA dataset by pairing similar images whose answers differ for the same question.
Results
State-of-the-art VQA models show a significant performance drop when tested on the balanced dataset.
Takeaways & Limitations
The balanced dataset reduces exploitable language priors and makes image understanding more necessary for successful VQA.
Takeaways & Limitations
Complementary-image collection was impossible for 22% of VQA questions using the available neighboring-image interface.
Abstract
from arXiv · showhide
Problems at the intersection of vision and language are of significant importance both as challenging research questions and for the rich set of applications they enable. However, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at www.visualqa.org as part of the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.
1. Introduction
The paper argues that language priors let VQA models perform well without understanding images, and introduces a balanced dataset pairing similar images with different answers to make visual information necessary. The dataset is approximately twice the size of VQA, and its collection protocol supports counter-example explanations using similar images with different predicted answers.
- Method: The authors balance VQA by collecting a complementary image I′ similar to an original image I but producing a different answer A′ for the same question Q.This construction gives almost every question a pair of similar images with different answers, reducing language bias and requiring image-based discrimination.
- Dataset: 1.1 Million image-question pairs make the balanced dataset almost twice the size of VQA, with approximately 13 Million associated answers across approximately 200k COCO images.The dataset is presented as a benchmark for VQA approaches and is publicly available through the project website.
- Explanation: The collection protocol enables a counter-example explanation model that answers an image-question pair and provides similar images it believes have different answers.These “hard negatives” are intended to help users establish greater trust in the VQA model.
2. Related Work
Prior VQA work introduced widely used datasets and models, while this work builds on Antol et al.’s VQA dataset to reduce language bias through balancing. Related efforts used hard negatives, restricted clipart edits, or explanation mechanisms, whereas this work collects similar real-image pairs and introduces a new explanation approach.
- Visual Question Answering: This work builds on Antol et al.’s widely used VQA dataset, reducing language biases while producing a more balanced dataset about twice its size.The paper also benchmarks a model on the balanced dataset, though the supplied passage truncates the benchmark description.
- Visual Question Answering: Each question in the proposed balanced VQA dataset is paired with two similar images that produce different answers.This construction provides complementary image pairs for balancing the dataset.
- Data Balancing and Augmentation: Unlike Hodosh et al.’s hand-designed similar captions, this work uses a novel annotation interface to collect two similar images for VQA questions.The paper frames the approach as constructing a more rigorous evaluation protocol using hard negatives.
- Data Balancing and Augmentation: Compared with Zhang et al.’s restricted binary clipart setting, this work addresses real images, where fine-grained editing of image content is not possible.Zhang et al. asked annotators to change clipart scenes so the answer changed; the supplied passage truncates the contrast with this work.
- Models with explanation: Existing explanation methods generate natural-language explanations or visual region maps, while this work introduces a further explanation mechanism.The supplied passage identifies prior explanation methods as responses to typically black-box and noninterpretable deep-learning predictions, but truncates the description of this work’s mechanism.
3. Dataset
The dataset balances VQA by pairing each image-question-answer triplet with a similar complementary image yielding a different answer, using a two-stage AMT collection process. The resulting dataset is substantially larger and more balanced than the original, though complementary-image selection was impossible for 22% of questions.
- Complementary image collection: For each original triplet (I, Q, A), workers select a similar image I′ where Q makes sense but produces a different answer A′.Workers choose from 24 nearest-neighbor images after viewing the original image, question, and answer.
- Complementary image collection: The 24 candidate neighbors are retrieved using VGGNet fc7 activations and ℓ2 distances between image representations.Each image is represented by activations from the penultimate layer of a deep CNN.
- Answer annotation: For every selected complementary image, 10 new AMT workers provide answers, with the majority answer designated as A′.This second annotation round mirrors the original VQA answer collection procedure.
- Dataset construction: The two-stage process creates semantically similar image pairs requiring models to recognize subtle visual differences that change the answer to the same question.The paired images are I and I′, with respective answers A and A′.
- Limitations: 22% of VQA questions received a “not possible” selection because no neighbor both supported the question and changed its answer.The main causes were objects too small to appear in globally similar neighbors and rare concepts.
- Dataset statistics: More than 443K train, 214K val, and 453K test question-image pairs make up the complete balanced dataset, which is significantly more balanced than original VQA.Binary questions have a more balanced yes/no distribution, although the dataset is not perfectly balanced.
4. Benchmarking Existing VQA Models
Benchmarking existing VQA models on the balanced dataset shows that their original performance relied substantially on language priors, while additional balanced training data improves accuracy but leaves substantial room for visual understanding.
- Benchmark Setup: Existing state-of-the-art VQA models are retrained on balanced data to test whether impoverished language cues encourage greater reliance on visual information.The benchmark includes d-LSTM+n-I, HieCoAtt, and MCB, alongside prior and language-only baselines.
- Benchmark Results: Models trained on the original unbalanced dataset perform significantly worse on balanced validation than on unbalanced validation, confirming severe language bias.This comparison is reported as UU versus UB in Table 1.
- Benchmark Results: 2-3% accuracy improves when models train on the complete balanced dataset rather than the similarly sized Bhalf set.The comparison is BhalfB versus BB, and the full balanced dataset is approximately twice the size of the original.
- Visual Reasoning: Balanced evaluation also requires models to distinguish subtle differences between highly similar paired images that have different answers to the same question.The paired construction enables analyses of whether complementary images receive both correct, identical, or different predictions.
- Answer-Type Analysis: Yes/no accuracy drops by ∼10.8% for MCB and ∼12.4% for HieCoAtt from UU to UB, exposing strong exploitation of language biases.The balanced validation set reduces the answer regularities present in the unbalanced validation set.
- Answer-Type Analysis: The largest UB-to-BhalfB improvements occur for yes/no, at ∼4.5% for MCB and ∼3% for HieCoAtt, and number, at ∼3% for MCB and ∼2% for HieCoAtt.These answer types had previously shown minimal improvements among existing approaches, making the balanced benchmark more discriminative.
5. Counter-example Explanations
The paper introduces counter-example explanations that answer a visual question and retrieve a similar image expected to produce a different answer. The model learns this behavior from human-selected complementary images and generally produces sensible explanations, though selecting effective counter-examples remains challenging.
- Motivation: Counter-example explanations pair a predicted answer with a similar image that the model believes has a different answer to the same question.The goal is to help users assess whether the model understands the queried concept.
- Method: At test time, the model first predicts Apred for (I, Q), then retrieves a similar image with a different answer from K nearest neighbors.Similarity is enforced by restricting retrieval to the candidate set INN.
- Limitations: Hard negative mining can select images where the question is irrelevant, such as an image without a woman for “What is the woman doing tennis?”.The baseline minimizes P(Apred|Q, Ii), but low predicted-answer probability does not ensure that Q makes sense for Ii.
- Training: Human-selected complementary images supervise counter-example learning because they are relevant to Q, similar to I, and associated with a different answer.The training candidates use I′ ∈ INN with K = 24.
- Results: Most qualitative explanations are sensible, showing images similar to I but with answers different from those predicted for I.The model outputs the predicted answer and top three negative explanations.
- Results: The explanation model significantly outperforms Random and VQA baselines, while Distance remains strongest and identifying counter-examples among nearest neighbors remains challenging.Human-picked counter-examples are valid but not necessarily unique or optimal, making evaluation inherently ambiguous.
6. Conclusion
The paper counters language priors in VQA by balancing the dataset with complementary image pairs that yield different answers. The resulting dataset exposes weaknesses in existing models and supports counter-example explanations intended to improve user trust.
- Dataset construction: The authors balance VQA by collecting complementary image pairs that look similar but produce different answers to each question.This data-collection interface elevates the role of image understanding in VQA.
- Dataset construction: The balanced dataset is more balanced by construction, about twice the original size, and has heavier tails in its answer distribution.These properties reduce the strong language priors that models may exploit.
- Benchmarking: A significant performance drop occurs when near state-of-art VQA models are tested on the balanced dataset.The result confirms that these models had exploited language biases.
- Explainability: The explainable model returns an answer and similar counter-example images where the answer differs from its prediction.Such explanations may help users understand the system’s responses and ultimately build trust.