Source-linked AI summary
Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
Sainandan Ramakrishnan, Aishwarya Agrawal, Stefan Lee
TL;DR
VQA models often exploit language priors rather than grounding answers in images, motivating methods that separate question-based bias from visual evidence. The paper introduces adversarial regularization with a question-only adversary and an image-conditioned confidence objective. It improves performance across base models on VQA-CP, reaches state-of-the-art there, and shows smaller accuracy drops on standard VQA than existing bias-reducing models.
Problem
VQA models can rely on superficial question–answer correlations and perform poorly when answer priors change, despite images containing the necessary information.
Method
The method adversarially trains a question-only model against the base VQA model and maximizes the confidence increase attributable to processing the image.
Results
The approach consistently improves multiple base models on VQA-CP, achieves state-of-the-art performance there, and has significantly smaller accuracy drops than existing bias-reducing models on standard VQA.
Takeaways & Limitations
The regularization scheme is model agnostic, simple to implement, and can be trained end-to-end as a drop-in module for existing VQA models.
Takeaways & Limitations
The regularization strength requires balancing bias reduction against VQA performance, because excessive regularization can remove basic question-type information and harm both models.
Abstract
from arXiv · showhide
Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training such as overwhelmingly reporting the type of room as kitchen or the sport being played as tennis, irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings. In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary -- discouraging the VQA model from capturing language biases in its question encoding. Further,we leverage this question-only model to estimate the increase in model confidence after considering the image, which we maximize explicitly to encourage visual grounding. Our approach is a model agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models -- achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.
1 Introduction
VQA models can exploit question–answer correlations instead of grounding answers in images, a weakness exposed by changing-prior benchmarks. The paper addresses this by adversarially reducing language bias in question representations and explicitly encouraging image-based confidence gains.
- 1 Introduction: Many VQA models rely on superficial question–answer correlations, producing stereotyped answers and failing on novel attribute–noun combinations.Examples include answering “What sport ...” with “tennis” and missing a green hydrant despite seeing hydrants and green objects during training.
- 1 Introduction: VQA-CP changes answer distributions across training and test for each question type, exposing models that over-rely on training-set language priors.The image still contains the information needed to answer the questions.
- 1 Introduction: A question-only model measures language-prior strength by predicting answers from questions without seeing the associated images.This blind-model performance is a standard baseline alongside VQA datasets.
- 1 Introduction: The proposed adversarial scheme trains a question-only adversary against the base VQA model to make its question representation less informative about language biases.The adversary predicts from the base model’s question encoding, while the base model adjusts that encoding to reduce adversary performance.
- 1 Introduction: The method consistently improves multiple base models on VQA-CP, outperforms an existing grounded-by-design approach, and produces smaller accuracy drops on standard VQA benchmarks.Both the question-only adversary and difference-of-entropies components contribute to the gains, with their combination performing best.
2 Reducing Language Bias Through Adversarial Regularization
The paper reduces VQA language bias by adversarially weakening answer-predictive information in question encodings while encouraging image-dependent confidence gains. A question-only adversary and a difference-of-entropies regularizer modify training without requiring a specific base architecture.
- Motivation: VQA models can ignore visual features when question features already predict answers from strong language correlations.This behavior can perform acceptably on similarly biased benchmarks but is severely limited in real-world settings.
- Question-Only Adversary: The question-only adversary predicts answers from the base model’s question encoding, exposing answer-discriminative language information without using the image.It is implemented as a separate differentiable classifier operating on learned question encodings.
- Adversarial Regularization: Adversarial training updates the question encoder to make the question-only adversary perform poorly, while the adversary itself is trained to predict answers accurately.The question encoder receives the negative adversary gradient, thereby maximizing its cross-entropy loss.
- Adversarial Regularization: The regularization coefficient λQ trades off language-bias reduction against VQA performance.Excessively large λQ can remove even basic question-type information and harm both the base model and adversary.
- Difference of Entropies: The difference-of-entropies regularizer encourages the image-conditioned VQA distribution to become sharper than the question-only distribution.Its resemblance to conditional mutual information is noted, but the estimator is ill-defined because the question-only output is not constrained to be the image-conditioned marginal.
- Difference of Entropies: Only the question encoder is updated from the entropy regularizer, preventing arbitrary sharp outputs by the adversary or VQA components that could cause over-fitting.The combined regularizers improve performance, with benefits compounding across a wider range of regularization coefficients.
3 Related Work
The related work situates the method within dataset bias diagnostics, bias-reducing VQA architectures, adversarial representation learning, and inference-time recalibration. The paper distinguishes its training-time, model-agnostic procedure from approaches requiring architectural restrictions, segmentation, or test-set recalibration.
- Language Bias in VQA: VQA datasets contain strong question-answer associations, motivating balancing strategies and question-only baselines to measure language priors.VQA v2 weakened but did not eliminate these priors through paired images with different answers.
- Language Bias in VQA: VQA-CP diagnoses poor visual grounding by changing answer distributions between training and test for question types.Models that rely excessively on language biases perform poorly on this split.
- Overcoming Unwanted Biases: GVQA reduces language-bias exploitation through hand-designed separation of visual concepts from plausible answer clusters.This approach performs well on VQA-CP but is described as complicated.
- Overcoming Unwanted Biases: Neural module networks explicitly separate question-derived computation layouts from image reasoning, yet still suffer dramatic VQA-CP performance drops.Their compositional structure therefore does not by itself prevent language-bias sensitivity on this diagnostic split.
- Overcoming Unwanted Biases: Inference-time recalibration can address bias amplification but requires output distributions for every test-set element before recalibration.The paper contrasts this with its training-time procedure, while noting the methods could in principle be combined.
- Adversarial Learning: Related adversarial representation methods encourage intermediate encodings to become invariant to attributes or other predictable information.The cited image-manipulation framework uses an adversary to predict attributes from image encodings alone.
- Adversarial Learning: Unlike the related attribute-invariance approach, this method does not add removed answer information back as an input and instead requires combining question and image features.The comparison frames the VQA method as removing answer-discriminative question features while preserving recovery through multimodal inputs.
4 Experiments
Experiments evaluate the proposed regularization with multiple VQA base models on VQA-CP and standard VQA datasets, comparing adversarial components and existing approaches. Results show state-of-the-art VQA-CP performance while largely preserving standard VQA accuracy.
- Implementation: The question-only adversary is a two-layer MLP that receives the base model’s question encoding and outputs a distribution over candidate answers.The entire base VQA system and adversary are trained end-to-end.
- Optimization: Question encoding parameters are updated using the VQA, difference-of-entropies, and negative question-only losses, while other VQA parameters use only the VQA loss.The question-only model is updated only through its VQA cross-entropy loss despite contributing to the difference-of-entropies term.
- Models: The study applies the regularization to SAN and UpDn, using publicly available reimplementations and labeling the components Q-Adv and DoE.SAN uses an LSTM question encoder and VGGNet image features, while UpDn uses a GRU encoder and Faster R-CNN region features.
- Datasets and Evaluation: Models are trained on VQA-CP and evaluated on its test set using the standard VQA evaluation metric, with additional evaluation on standard VQA splits.
- Results: The method significantly improves base-model accuracy and achieves state-of-the-art performance on VQA-CP v2.Table 1 reports results on VQA-CP v2 test and VQA v2 validation.
5 Results
The proposed regularizers improve VQA-CP performance across SAN and UpDn, reduce language-biased behavior, and preserve standard VQA performance better than prior bias-reduction approaches. Their effects include improved robustness under changing priors, stabilized adversarial training, and complementary errors with the base model.
- VQA-CP performance: 8.33% improvement over SAN and 1.43% over UpDn on VQA-CP v2 result from combining Q-Adv and DoE regularizers.The combination also outperforms GVQA by 1.99% for SAN; the UpDn combination improves over GVQA by 9.87%.
- VQA-CP performance: 16.55% improvement over SAN on VQA-CP v1 exceeds the 8.33% gain on VQA-CP v2, alongside larger language biases in VQA v1.The combined regularizers outperform GVQA by 4.2% on VQA-CP v1.
- Standard VQA performance: On VQA v2, the regularizers cause an insignificant 0.1% SAN drop and a minor 0.73% UpDn drop, compared with GVQA’s 4.17% SAN drop.On VQA v2 test-std, SAN+Q-Adv+DoE reaches 52.95% overall accuracy, 2.33% below base SAN.
- Behavioral analysis: The regularized model produces less prior-aligned answer distributions and complementary strengths with SAN, improving the standard VQA v2 ensemble by 1.24%.The base SAN model aligns more strongly with the training distribution and amplifies the ‘blue’ answer bias in one example.
- Training behavior: DoE stabilizes training by allowing stronger question-only regularization with less dramatic performance loss, while sharpening the VQA posterior and weakening the question-only prior.Without DoE, performance deteriorates significantly at higher question-only regularizer coefficients.
- Language-bias reduction: The question-only model achieves 13.85% versus 24.84% for Q-only(SAN) on the VQA-CP v2 training set, indicating less discriminative question encoding.This comparison is reported after training with the proposed regularizers.
6 Conclusion
The paper proposes an adversarial regularization scheme that reduces memorization of dataset biases in VQA while improving performance under changing priors. It is designed as a simple, end-to-end trainable module for existing VQA models.
- The method combines a question-only adversary with differences in model confidence after image processing to reduce memorized dataset biases.
- Experiments on VQA-CP show that the technique significantly improves existing VQA models amid changing answer priors.
- The approach achieves state-of-the-art performance on VQA-CP and can be trained end-to-end from scratch as a drop-in module.