RUBi: Reducing Unimodal Biases in Visual Question Answering

Remi Cadene; Corentin Dancette; Hedi Ben-younes; Matthieu Cord; Devi Parikh

Source-linked AI summary

RUBi: Reducing Unimodal Biases in Visual Question Answering

Remi Cadene, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, Devi Parikh

arXiv:1906.10169v2Jun 24, 2019cs.CVcs.CLcs.LG

TL;DR

VQA models can exploit question-only biases instead of using image information, undermining robustness when test distributions change. RUBi adds a question-only training branch that dynamically downweights biased examples and guides the base model toward multimodal reasoning. On VQA-CP v2, RUBi surpasses the prior state of the art and generalizes across VQA architectures, although the authors identify grounding evaluation as future work.

Problem
VQA models often exploit question–answer regularities without using images, causing significant accuracy drops when test-time statistical distributions differ from training.
Method
RUBi adds a question-only branch during training to identify biased examples, dynamically adjust the loss, and reduce their influence on the base VQA model.
Results
+5.94 percentage point accuracy over the state-of-the-art result on VQA-CP v2.
Takeaways & Limitations
RUBi is a model-agnostic learning strategy that improves robustness-oriented VQA performance across different common VQA architectures.
Takeaways & Limitations
The authors identify evaluating RUBi’s impact on visual grounding as future work.

Abstract

from arXiv · show

Visual Question Answering (VQA) is the task of answering questions about an image. Some VQA models often exploit unimodal biases to provide the correct answer without using the image information. As a result, they suffer from a huge drop in performance when evaluated on data outside their training set distribution. This critical issue makes them unsuitable for real-world settings. We propose RUBi, a new learning strategy to reduce biases in any VQA model. It reduces the importance of the most biased examples, i.e. examples that can be correctly classified without looking at the image. It implicitly forces the VQA model to use the two input modalities instead of relying on statistical regularities between the question and the answer. We leverage a question-only model that captures the language biases by identifying when these unwanted regularities are used. It prevents the base VQA model from learning them by influencing its predictions. This leads to dynamically adjusting the loss in order to compensate for biases. We validate our contributions by surpassing the current state-of-the-art results on VQA-CP v2. This dataset is specifically designed to assess the robustness of VQA models when exposed to different question biases at test time than what was seen during training. Our code is available: github.com/cdancette/rubi.bootstrap.pytorch

1 Introduction

VQA models can exploit question–answer regularities instead of image information, causing poor robustness to shifted biases. RUBi addresses this by using a question-only branch to dynamically reduce biased training examples, improving performance on VQA-CP v2 and across architectures.

Motivation: VQA models often answer from question–answer statistical regularities without considering the image modality.For example, a model may predict “yellow” for banana-color questions without detecting or assessing the banana in the image.
Motivation: A question-only model predicts the correct answer approximately 44% of the time on the VQA v2 test set, revealing substantial language bias.
Motivation: Models trained and tested on similar distributions can exploit these shortcuts, but their accuracy drops significantly when answer regularities change at test time.VQA-CP v2 is designed to evaluate robustness under such changed question biases.
RUBi: RUBi adds a question-only branch during training to identify biased examples and dynamically adjust the base VQA model’s loss.The branch is removed after training, leaving the base VQA model for final prediction.
Results: RUBi surpasses current state-of-the-art results on VQA-CP v2 and improves several VQA architectures, while remaining competitive on standard VQA v2.

2 Related work

Prior work identifies unimodal biases through single-modality models, balances datasets, or redesigns architectures and learning objectives. RUBi builds on the question-only baseline as a learning strategy for preventing VQA models from learning question biases.

Assessing unimodal biases: Unimodal models quantify biases by measuring how well predictions can be made from only one input modality.Question-only models are particularly strong because questions contain many exploitable statistical regularities.
RUBi: RUBi instead adapts a question-only model as a branch that alters the main VQA model’s predictions during training and is removed afterward.
Assessing unimodal biases: VQA-CP v1 and v2 diagnose question bias by using different answer distributions for each question type between training and test splits.Models biased toward questions tend to fail on these benchmarks.
Balancing datasets: Dataset-based approaches reduce shortcuts through rejection sampling or complementary examples, but rejection sampling is usually impractical for real VQA datasets because annotation is costly.
Architectures and learning strategies: Architecture-based methods include GVQA, which separates visual-region grounding and recognition from identifying plausible answers conditioned on the question.
Architectures and learning strategies: Other learning strategies use an adversarial question-only branch, gradient negation, or entropy-based losses to discourage language priors.

3 Reducing Unimodal Biases Approach

RUBi reduces unimodal bias by using a question-only branch to alter the base VQA model’s predictions and dynamically reweight training loss. It downweights examples answerable from the question alone and emphasizes examples requiring both image and question modalities.

Capturing biases with a question-only branch: RUBi adds a question-only branch during training to capture question biases and influence the base VQA model’s predictions.The branch is removed after training, leaving the base VQA model for inference.
Preventing biases by masking predictions: The question-only branch produces an answer mask whose element-wise product with base predictions dynamically modifies the loss.The mask is obtained by passing the question-only network output through a sigmoid function.
Preventing biases by masking predictions: RUBi lowers loss for biased examples answerable without the image, reducing their backpropagated gradients and learning importance.For the banana-color example, the correct answer score rises from 0.8 to 0.94 while loss decreases from 0.22 to 0.06.
Preventing biases by masking predictions: RUBi raises loss for examples requiring both modalities, encouraging the VQA model to learn from visual information.For a green banana, RUBi increases the loss from 0.69 to 1.20.
Joint learning procedure: The base VQA model and question-only branch are jointly optimized with main and question-only cross-entropy losses, while the question encoder avoids direct bias learning.The final RUBi loss sums the two losses, and the question-only loss updates only the classifier and neural-network parameters.
Architecture compatibility: RUBi is compatible with many state-of-the-art VQA architectures and is tested with a Faster R-CNN image encoder, GRU question encoder, and bilinear fusion baseline.The baseline encodes image regions as visual features and questions as vectors before fusion.

4 Experiments

Experiments evaluate RUBi on bias-shifted VQA-CP v2, standard VQA v2, multiple architectures, ablations, and qualitative examples. RUBi improves robustness substantially while retaining competitive standard-dataset performance.

State-of-the-art comparison: 47.11% overall accuracy on VQA-CP v2 is a +5.94 percentage-point gain over UpDn + Q-Adv + DoE.The result averages five experiments and has a standard deviation of ±0.51.
Additional baselines: Question-Type Balanced Sampling improves accuracy from 38.46 to 42.11, but remains significantly below RUBi.Its gain is +0.94 higher than the previous state-of-the-art method.
Architecture agnostic: +11.73 accuracy points for SAN and +4.5 for UpDn demonstrate gains when RUBi is applied to additional VQA architectures.RUBi also exceeds Q-Adv + DoE by +3.4 with SAN and +3.06 with UpDn.
Impact on VQA v2: On standard VQA v2, RUBi produces a 1.94-percentage-point drop relative to its baseline while maintaining competitive performance against similar bias-reduction methods.The paper contrasts this with a 3.78-point drop between GVQA and its SAN baseline and drops of 0.05, 0.73, and 2.95 for three UpDn strategies.
Validation of the masking strategy: Replacing RUBi’s sigmoid with ReLU causes a 7.09-point drop on VQA-CP v2, while element-wise summation performs worse than the proposed product-based masking.The reported masking design relies on a sigmoid and an element-wise sum in the fusion of branch outputs.
Validation of the question-only loss: Adding the question-only loss yields gains of +0.89 for the baseline, +0.22 for SAN, and +4.76 for UpDn.The ablation attributes these comparisons to removing versus using LQO across the three architectures.

5 Conclusion

RUBi is a model-agnostic learning strategy for reducing unimodal biases in VQA models. It achieves a 5.94-percentage-point gain over the state-of-the-art result on VQA-CP v2 and works across common VQA models.

+5.94 percentage points in accuracy over the state-of-the-art result on VQA-CP v2.VQA-CP v2 is designed to account for question biases.
RUBi uses a question-only branch to capture unwanted question statistics and prevent the base VQA model from learning unimodal biases.The strategy is designed to be model agnostic.
RUBi is effective with different kinds of common VQA models.
The authors propose extending RUBi to other multimodal tasks in future work.

6 Supplementary materials

The supplementary materials report additional robustness, grounding, and implementation results for RUBi. They include VQA-CP v1 gains, grounding analyses on VQA-HAT, model details, and qualitative examples showing both improvements and failures.

Results on VQA-CP v1: +9.8 overall accuracy points with the baseline, +19.2 with SAN, and +7.66 with UpDn on VQA-CP v1.
Results on VQA-CP v1: +2.65 accuracy points over AdvReg with SAN on VQA-CP v1.All three RUBi-trained architectures also outperform GVQA.
Quantitative study of the grounding ability on VQA-HAT: +0.012 in rank-correlation with human attention maps for the baseline architecture trained with RUBi on VQA-HAT.The evaluation compares model attention maps with human annotations of relevant image regions.
Qualitative study of the grounding ability on VQA-HAT: Qualitative VQA-HAT examples show attention maps closer to human maps with RUBi, alongside examples where grounding does not improve.
Implementation details: The baseline is a simplified MuRel architecture using bilinear question–region fusion, max pooling, and a three-layer classifier over 3000 answers.
Implementation details: RUBi adds a question-only branch whose sigmoid output masks VQA predictions while a linear output computes the question-only loss.