Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

Aishwarya Agrawal; Dhruv Batra; Devi Parikh; Aniruddha Kembhavi

Source-linked AI summary

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, Aniruddha Kembhavi

arXiv:1712.00377v2Dec 1, 2017cs.CVcs.AIcs.CLcs.LG

TL;DR

VQA models can rely on superficial language priors instead of image grounding, motivating evaluation under changed answer distributions. The paper introduces VQA-CP and GVQA, which separates visual recognition from answer-space prediction; GVQA significantly outperforms SAN on both VQA-CP datasets and is more interpretable.

Problem
Existing VQA models are heavily driven by superficial training correlations and lack sufficient visual grounding, making it important to evaluate whether they use images or memorized priors.
Method
The paper creates VQA-CP splits with changed answer distributions by question type and proposes GVQA, which separates visual-concept recognition from plausible answer-space prediction.
Results
GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2, while also outperforming MCB in several cases and providing interpretable intermediate outputs.
Takeaways & Limitations
Changing-prior evaluation exposes reliance on answer priors, while GVQA offers a more transparent approach designed for visual grounding.
Takeaways & Limitations
GVQA is presented as a first step, with future work needed to combine visual grounding with world priors when grounding confidence is low.

Abstract

from arXiv · show

A number of studies have found that today's Visual Question Answering (VQA) models are heavily driven by superficial correlations in the training data and lack sufficient image grounding. To encourage development of models geared towards the latter, we propose a new setting for VQA where for every question type, train and test sets have different prior distributions of answers. Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively). First, we evaluate several existing VQA models under this new setting and show that their performance degrades significantly compared to the original VQA setting. Second, we propose a novel Grounded Visual Question Answering model (GVQA) that contains inductive biases and restrictions in the architecture specifically designed to prevent the model from 'cheating' by primarily relying on priors in the training data. Specifically, GVQA explicitly disentangles the recognition of visual concepts present in the image from the identification of plausible answer space for a given question, enabling the model to more robustly generalize across different distributions of answers. GVQA is built off an existing VQA model -- Stacked Attention Networks (SAN). Our experiments demonstrate that GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2 datasets. Interestingly, it also outperforms more powerful VQA models such as Multimodal Compact Bilinear Pooling (MCB) in several cases. GVQA offers strengths complementary to SAN when trained and evaluated on the original VQA v1 and VQA v2 datasets. Finally, GVQA is more transparent and interpretable than existing VQA models.

1. Introduction

The paper argues that VQA models often exploit language priors rather than grounding answers in images, motivating changing-prior splits and a grounded model that separates visual recognition from answer-space prediction.

Motivation: Existing VQA models often rely on superficial training correlations and insufficient visual grounding, answering from question patterns while largely ignoring images.Examples include predicting “2” for “how many,” “white” for color questions, and “yes” for yes/no questions.
Motivation: IID train-test splits can make prior-memorizing models appear successful, obscuring whether improvements come from image grounding or bias memorization.The paper identifies this ambiguity as problematic for benchmarking VQA progress.
Contributions: VQA-CP v1 and VQA-CP v2 reorganize VQA v1 and v2 so answer distributions differ by question type between training and test while image distributions remain unchanged.The splits are intended to test generalization across answer priors rather than across image domains.
Contributions: All tested existing models drop significantly on the changing-prior splits compared with the original VQA setting.This result supports the claim that conventional VQA performance can depend heavily on training priors.
Contributions: GVQA imposes architectural restrictions that disentangle visual-concept recognition from plausible answer-space prediction to reduce reliance on training priors.GVQA is built on SAN and explicitly separates what should be recognized from what kinds of answers are plausible.
Contributions: GVQA significantly outperforms SAN on both VQA-CP datasets, exceeds MCB in several cases, complements SAN on original VQA datasets, and provides interpretable intermediate outputs.The paper presents transparency and interpretability as additional strengths of GVQA.

2. Related Work

Related work addresses VQA priors through balanced data, evaluation metrics, decoy construction, adaptation, compositionality, and module-based architectures, whereas this paper evaluates visual grounding under changed answer distributions.

Countering Priors in VQA: VQA v2 weakens language priors by pairing questions with complementary images having different answers, but train and test distributions remain similar.As a result, exploiting training priors can still benefit models at test time.
Countering Priors in VQA: Other work proposes metrics correcting skewed question or answer distributions, constructs better decoys for multiple-choice VQA, and adapts models across datasets.These efforts target evaluation, shortcut resistance, or cross-dataset statistical shifts.
Compositionality: Compositionality research studies novel combinations of known concepts across zero-shot recognition, image captioning, and VQA.This connects answer-distribution generalization with broader efforts to compose learned concepts.
VQA Architectures: Neural Module Networks use question-conditioned specialized modules, yet their performance also degrades significantly on VQA-CP datasets.The paper notes that an additional LSTM question encoding may allow such models to encode dataset priors.
Related Generalization Settings: Zero-shot VQA studies unseen words or objects, while this work instead tests generalization to different answer distributions with test concepts present during training when possible.The paper characterizes these settings as orthogonal to its focus on visual grounding.

3. VQA-CP : Dataset Creation and Analysis

VQA-CP creates train-test answer-prior shifts by grouping questions by type and answer, then greedily redistributing groups while preserving concept coverage and avoiding repeated type-answer groups.

Split Construction: VQA-CP v1 and v2 differ from their source datasets by assigning different answer distributions to each question type in training and test data.The splits are formed by reorganizing the original training and validation data.
Question Grouping: Questions are grouped when they share both a question type and a ground-truth answer.For example, “What color is” questions answered “white” form a different group from those answered “black.”
Greedy Re-splitting: The greedy re-splitting procedure assigns groups to test and train to maximize test-concept coverage in training while preventing repeated question-type and answer groups across splits.The procedure first assigns groups to test unless already assigned to train, then selects training groups covering remaining concepts.
Split Statistics: The resulting VQA-CP v1 split covers 98.04% of test question concepts and 95.07% of test answers through the training split’s top 1000 answers.The corresponding VQA-CP v2 coverage values are 99.01% and 95.72%.
Distribution Analysis: VQA-CP v1 answer distributions differ substantially between train and test, such as “tennis” being most frequent for “what sport” in train and “skiing” in test.The paper contrasts this with VQA v1, whose train and validation distributions are similar for a given question type.

4. Benchmarking VQA Models on VQA-CP

The benchmark compares standard VQA baselines and models on original and changing-prior splits, finding substantial degradation across existing systems and altered model rankings under VQA-CP.

Evaluation Setup: The evaluation trains models on VQA-CP train splits and tests them on corresponding VQA-CP test splits, comparing results with standard VQA train-validation evaluations.Results are reported in Table 1.
Compared Models: The benchmark includes a question-type prior, blind question-only, normalized-image, Neural Module Network, SAN, and MCB baselines or models.These systems represent prior-only, question-only, baseline multimodal, compositional, attention-based, and bilinear approaches.
Baseline Caveat: The question-type prior baseline is ideally zero on VQA-CP because train and test answers differ by question type, but annotation disagreement makes performance slightly higher.This caveat concerns the benchmark’s human-annotated answers.
Results: All tested existing VQA models suffer significant performance drops in the VQA-CP setting compared with the original VQA setting.The result holds even for the compositional NMN architecture.
Results: Model rankings change between settings: SAN outperforms NMN on VQA, whereas NMN outperforms SAN on VQA-CP.The paper suggests that question-prior encoding may contribute to these differences.

5. GVQA model

GVQA decomposes VQA into visual concept recognition and answer-space prediction, using separate pathways for non-yes/no and yes/no questions. Its modular components locate relevant image regions, predict concepts or answer clusters, and produce answers through aligned prediction or visual verification.

Architecture: GVQA breaks VQA into visual concept recognition and answer-space prediction rather than directly mapping image-question tuples to answers.This separation is designed to reduce reliance on question-answer priors.
Architecture: The Visual Concept Classifier attends to the image region needed for a question and predicts relevant objects and attributes.For example, it can attend to a bus and output concepts related to its identity, color, and count.
Question routing: A Question Classifier routes questions to the yes/no or non-yes/no pathway, with training and testing using the corresponding module outputs.The classifier uses GloVe embeddings, an LSTM, and fully connected layers; during testing, its routing determines which components are activated.
Non-yes/no questions: For non-yes/no questions, the Answer Cluster Predictor identifies the expected answer type and the Answer Predictor combines it with visual concepts to produce the answer.Answer clusters are formed from answer classes embedded in GloVe space, while visual concepts are assigned to corresponding clusters.
Yes/no questions: For yes/no questions, the Concept Extractor identifies the concept to verify and the Visual Verifier predicts its presence or absence from visual concepts.The Concept Extractor uses part-of-speech-based extraction, while the Visual Verifier combines the extracted concept with VCC predictions and outputs yes or no.

6. Experimental Results

GVQA substantially improves over SAN on changing-prior splits, while component analyses clarify the accuracy–interpretability trade-offs and results on original VQA datasets show complementary strengths.

VQA-CP performance: 12.35% and 6.34% overall improvements over SAN are reported for GVQA on VQA-CP v1 and VQA-CP v2, respectively.On VQA-CP v1, GVQA also outperforms MCB and NMN overall; on VQA-CP v2, it outperforms NMN overall and MCB for yes/no and number questions.
VQA-CP performance: 7.14% average improvement over SAN across four VQA-CP v2 random-seed splits supports consistent GVQA performance across split constructions.The four test splits together cover 84% of the dataset, with a standard error of 1.36.
Role of GVQA components: Replacing CE with an LSTM reduces overall performance by 11.95% and yes/no performance by 28.76%, highlighting CE’s role in the yes/no pipeline.The replacement trains the LSTM end-to-end with the Visual Verifier using VQA loss.
Role of GVQA components: Removing VCC loss improves overall performance by 1.72% and performance on other questions by 3.19%, but sacrifices transparency.The authors frame VCC loss as an accuracy–interpretability design trade-off.
Original VQA performance: On original VQA datasets, GVQA trails SAN overall by 4.74% on VQA v1 and 3.78% on VQA v2, but outperforms SAN by 3.14% on VQA v2 yes/no questions.The GVQA–SAN gain on VQA-CP v1 is 12.35% absolute, larger than the 4.74% loss on VQA v1.
Complementarity and transparency: 7.91% and 9.94% higher oracle performance than SAN alone on VQA v1 and VQA v2 indicates complementary strengths, while their ensembles improve over SAN-only ensembles.Ensemble (GVQA, SAN) exceeds Ensemble (SAN, SAN) by 0.35% on VQA v1 and 0.51% on VQA v2.
Complementarity and transparency: GVQA’s intermediate VCC, ACP, and CE outputs expose why predictions succeed or fail, enabling targeted error analysis and improvements.The model’s interpretable intermediate outputs distinguish it from existing VQA models, although the supplied passage truncates the final sentence.

7. Conclusion

GVQA is presented as a first step toward models designed for visual grounding, while future work aims to combine grounding with useful world priors. The appendix documents additional analyses, implementation details, splits, component results, and qualitative examples.

GVQA is a first step toward building models that are visually grounded by design.
Future work aims to combine visual grounding with priors such as knowledge that sky is usually blue and grass is usually green.
The appendix includes additional split analyses, benchmarking details, GVQA implementation details, extra VQA-CP v2 splits, component evaluations, and qualitative examples.

Appendix I: Additional analysis of VQA-CP splits

The VQA-CP v2 train and test splits have significantly different answer distributions for a given question type, illustrated across several question categories.

Answer distributions differ significantly between VQA-CP v2 train and test splits for the same question types.For example, tennis is reported as the most frequent answer for “what sport” in the train split.

Appendix II: Details of benchmarking VQA models on VQA-CP

The benchmark covers blind, image-question, compositional, attention-based, and bilinear VQA models, and shows substantial degradation under changing priors across several question types.

Benchmarked models: The benchmark includes d-LSTM question-only and image-question baselines, Neural Module Networks, SAN, and MCB.These models span question-only prediction, multimodal classification, compositional modules, attention, and multimodal compact bilinear pooling.
Performance degradation: Performance drops from VQA to VQA-CP are larger for some question types than for others.
Performance degradation: Approximately 70% performance drops occur for “is there a” questions in VQA-CP v1 across all evaluated models.The VQA-CP v1 test prior favors “yes,” while the training prior favors “no,” encouraging prior-driven “no” answers.
Performance degradation: More than 10% performance drops affect all models on several VQA-CP v2 question types, including “is it,” “is he,” “are there,” and “what color is the.”

Appendix III: Implementation details of GVQA

GVQA’s implementation uses separate recurrent classifiers and optimizers for question, answer-cluster, visual-concept, answer, and visual-verification components. Across four VQA-CP v2 splits, GVQA consistently outperforms SAN, with a 7.14% average improvement.

Implementation: GVQA uses separate LSTM-based components for question classification, answer-cluster prediction, and visual-concept classification.The Answer Cluster Predictor uses 50 answer clusters, while the Visual Concept Classifier uses VGG-Net image features.
Implementation: GVQA uses RMSProp for the question classifier, answer-cluster predictor, and visual-concept classifier, and Adam for answer prediction and visual verification.
Implementation: Using 25 rather than 50 answer clusters reduces overall VQA-CP v2 accuracy by 1.05%, while using 100 reduces it by 0.76%.
Results: Figure 7 compares SAN and GVQA across different VQA-CP v2 splits, with GVQA outperforming SAN across all splits.
Results: 7.14% average improvement: GVQA consistently outperforms SAN across four VQA-CP v2 splits.The reported standard error is 1.36.

Appendix V: Performance of model components on VQA-CP v2

On VQA-CP v2, the question classifier performs strongly, while ACP and VCC provide weaker or partial component-level predictions. GVQA outperforms SAN and remains stronger than the Qmain SAN variant, while GVQA-VCCloss shows complementary strengths but weaker ensemble performance.

99.30% accuracy is achieved by the LSTM-based question classifier on the VQA-CP v2 test set.
0.53 is the weighted mean test F1 score across VCC classifiers.Concept weights reflect the number of positive samples and therefore test-set concept coverage.
4.98% is GVQA’s overall advantage over SAN-Qfull+Qmain, whose performance is 1.36% above original SAN.
8.61% and 9.91% are the Oracle overall gains over SAN for VQA v1 and VQA v2, respectively, showing complementary GVQA-VCCloss strengths.Despite this complementarity, its ensemble with SAN is 0.12% worse on VQA v1 and 0.66% worse on VQA v2 than SAN+SAN.

Appendix VIII: Additional qualitative examples

Figures 8–10 present qualitative VQA-CP v2 examples illustrating GVQA’s attention and intermediate predictions. These outputs provide insight into predictions and error causes, unlike existing models such as SAN.

Figure 8 shows VCC’s attention map for the example presented in Figure 5.
Figures 9 and 10 compare GVQA and SAN predictions on qualitative VQA-CP v2 test examples.
GVQA’s intermediate outputs help explain its predictions and enable system designers to identify causes of error.The examples include visual-concept predictions, attention maps, and answer-space predictions.