Source-linked AI summary
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, Yueting Zhuang
TL;DR
VQA models can rely on superficial language correlations and generalize poorly when question-answer distributions change, while existing ensemble methods lack visual-explainable and question-sensitive behavior. CSS synthesizes counterfactual samples by masking critical image objects or question words and assigning different answers. It improves these abilities and performance, reaching 58.95% on VQA-CP v2 with LMH.
Problem
VQA models over-rely on superficial linguistic correlations and perform poorly when train and test question-answer distributions differ, while ensemble methods lack visual-explainable and question-sensitive abilities.
Method
CSS is a model-agnostic training scheme that masks critical image objects or question words, assigns different answers, and trains on original and synthesized samples.
Results
58.95% on VQA-CP v2 is achieved by LMH-CSS, while CSS consistently improves performance across multiple VQA architectures.
Takeaways & Limitations
CSS improves visual-explainable and question-sensitive abilities while boosting performance across VQA models, including ensemble-based methods.
Abstract
from arXiv · showhide
Despite Visual Question Answering (VQA) has realized impressive progress over the last few years, today's VQA models tend to capture superficial linguistic correlations in the train set and fail to generalize to the test set with different QA distributions. To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP. However, since the complexity of design, current methods are unable to equip the ensemble-based models with two indispensable characteristics of an ideal VQA model: 1) visual-explainable: the model should rely on the right visual regions when making decisions. 2) question-sensitive: the model should be sensitive to the linguistic variations in question. To this end, we propose a model-agnostic Counterfactual Samples Synthesizing (CSS) training scheme. The CSS generates numerous counterfactual training samples by masking critical objects in images or words in questions, and assigning different ground-truth answers. After training with the complementary samples (ie, the original and generated samples), the VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. In return, the performance of these models is further boosted. Extensive ablations have shown the effectiveness of CSS. Particularly, by building on top of the model LMH, we achieve a record-breaking performance of 58.95% on VQA-CP v2, with 6.5% gains.
1. Introduction
VQA models often exploit language biases and can fail when question-answer distributions change, while ensemble-based debiasing methods still lack visual-explainable and question-sensitive behavior. CSS addresses these gaps by synthesizing counterfactual image-question samples and improves VQA performance.
- Motivation: VQA models over-rely on superficial linguistic correlations, causing significant performance drops on VQA-CP’s changed train-test question-answer distributions.VQA-CP was introduced to diagnose this generalization problem.
- Motivation: Existing ensemble-based methods dominate VQA-CP performance but do not ensure reliance on the right visual regions or sensitivity to linguistic variations.The paper illustrates these missing properties through visual-reference and word-replacement examples.
- Method: CSS is a model-agnostic, plug-and-play training scheme with V-CSS masking critical image objects and Q-CSS replacing critical question words with [MASK].Both mechanisms create counterfactual image-question pairs and use dynamically assigned answers to avoid manual annotation.
- Method: Training on original and synthesized complementary samples encourages VQA models to focus on critical objects and words.The synthesized samples receive different ground-truth answers from the originals.
- Results: 58.95% on VQA-CP v2 is achieved by building CSS on LMH, establishing a new record-breaking performance.The paper also reports extensive qualitative and quantitative ablations demonstrating CSS effectiveness.
2. Related Work
Related work addresses VQA language biases through dataset balancing and specialized debiasing models, while separate efforts target visual explanations, question sensitivity, or counterfactual generation. CSS combines complementary-sample training with simpler masking operations and avoids extra manual annotations.
- Language Biases in VQA: Dataset balancing reduces language biases to some extent, but statistical question biases can remain and models still drop on VQA-CP.CSS follows the complementary-sample idea without requiring extra manual annotations.
- Language Biases in VQA: Ensemble-based debiasing models are among the most effective approaches and CSS is designed to integrate with them.The paper positions CSS as a training scheme that further reduces language biases in these models.
- Visual-Explainable Ability in VQA Models: Attention supervision and Grad-CAM-based methods target visual-explainable behavior, but strong biases can leave later network layers disregarding visual signals.These approaches motivate an alternative training strategy focused on critical visual objects.
- Question-Sensitive Ability in VQA Models: Prior work on question sensitivity studies robustness to linguistic rephrasing through cycle-consistent learning and sampled noise.The cited approach focuses on diverse rephrased questions rather than CSS’s critical-word masking.
- Counterfactual Training Samples for VQA: Concurrent counterfactual VQA methods use GANs to generate images, whereas CSS masks critical objects or words for easier adoption.CSS therefore uses a simpler counterfactual-sample construction approach.
3. Approach
CSS augments VQA training with counterfactual image-question pairs that mask critical visual objects or question words and assign corresponding answers. It is designed as a model-agnostic scheme for training both standard and ensemble-based VQA models.
- Ensemble-Based Models: In fusion-based ensemble models, the VQA and question-only answer distributions are combined during training, while only the VQA model is used at test time.The fused distribution receives cross-entropy supervision, and gradients pass through both models during training.
- CSS trains on original samples and counterfactual samples generated through either visual or question masking.For each training sample, the scheme first trains on (I, Q, a), then synthesizes and trains on a counterfactual pair.
- V-CSS: Critical visual objects are selected from question-related objects using local contribution scores derived from the predicted ground-truth answer probability.The method selects top-scoring objects dynamically, with K defined as the smallest number satisfying the stated criterion and η set to 0.65.
- Dynamic Answer Assigning: Dynamic answer assignment removes answers predicted for the critical subset from the original ground-truth answer set to label the counterfactual pair.The remaining answers are assigned to the complementary input; if all ground-truth answers are predicted for the critical subset, the counterfactual answer set is empty.
- Q-CSS: Q-CSS extracts question-type words, selects critical words from the remaining sentence, and masks either the critical or noncritical words to construct Q− or Q+.The dynamic answer assignment step for Q-CSS takes the pair (I, Q+) as input.
- V-CSS masks critical image objects, while Q-CSS replaces critical question words with [MASK] and forms complementary image-question pairs.V-CSS uses the original question with a counterfactual image; Q-CSS uses the original image with a counterfactual question.
4. Experiments
The experiments evaluate CSS mainly on VQA-CP and additionally on VQA v2, using standard VQA accuracy and preprocessing aligned with the widely used UpDn setup.
- CSS is evaluated mainly on the VQA-CP test set and additionally on the VQA v2 validation set.The evaluation follows the standard VQA metric and uses the same preprocessing as the widely used UpDn model.
4.1. Ablative Studies
Ablations examine CSS hyperparameters and architecture coverage, showing that performance depends on masking choices and that CSS improves multiple VQA architectures.
- V-CSS Hyperparameters: Performance gradually decreases as the size of the candidate object set |I| increases in V-CSS.This trend is reported for the ablation built on the ensemble-based model LMH.
- Q-CSS Hyperparameters: Replacing one question word, denoted top-1, achieves the best performance among the tested masking choices.The comparison is reported in the Q-CSS hyperparameter ablation.
- CSS Proportion: Performance is best when the proportion δ of V-CSS and Q-CSS is 0.5.The δ ablation is conducted on top of LMH and reports the best result at equal proportions.
- Architecture Agnostic: 6.50% and 9.79% absolute performance gains are reported for LMH and PoE, respectively, when CSS is incorporated.CSS consistently improves UpDn, PoE, RUBi, and LMH, with larger improvements in ensemble-based models.
- Architecture Agnostic: Using both V-CSS and Q-CSS often produces the best performance across the evaluated architectures.Table 1 reports accuracies on the VQA-CP v2 test set for different VQA architectures.
4.2. Comparisons with State-of-the-Arts
LMH-CSS achieves state-of-the-art performance on VQA-CP v1 and v2, while CSS improves LMH substantially on the changing-prior benchmarks.
- Figure 4 evaluates V-CSS and Q-CSS hyperparameters using accuracies on the VQA-CP v2 test set.The panels vary image and critical-object sizes, critical-word size, and δ.
- 58.95% accuracy on VQA-CP v2 represents a 6.50% absolute gain over LMH’s 52.45%.
- LMH-CSS achieves new state-of-the-art performance across all question categories on VQA-CP v2.
- 60.95% accuracy on VQA-CP v1 represents a 5.68% absolute gain over LMH’s 55.27%.
4.3. Improving Visual-Explainable Ability
CSS improves visual-explainable behavior by outperforming direct integration of SCR into LMH and aligning influential objects more closely with the question-answer pair.
- Existing visual-explainable models cannot be easily incorporated into the ensemble-based LMH framework, whereas CSS improves performance.Using a well-trained LMH initialized at 52.45% accuracy, SCR-based training caused performance to decrease from the start.
- Table 4 reports quantitative evaluations of visual-explainable and question-sensitive abilities.
- CSS dramatically improves Average Importance scores, indicating that influential objects become more related to the question-answer pair.Average Importance averages the SIM scores of the top-K objects with highest |s(a, v)|.
- Figure 5 visualizations show CSS using critical objects for predictions and suppressing irrelevant objects.Green boxes indicate positive contributions, while red boxes indicate negative contributions.
4.4. Improving Question-Sensitive Ability
CSS improves robustness to question rephrasings and increases reliance on critical words when predicting answers.
- Q-CSS dramatically improves robustness to diverse question rephrasings, while adding V-CSS further improves performance.The evaluation uses Consensus Score CS(k) on VQA-CP-Rephrasings.
- The question-sensitive evaluation compares model confidence on an original question with confidence after removing a critical noun.
- CSS makes removing critical words cause larger confidence drops for ground-truth answers.Confidence Improvement measures the confidence decrease after removing a critical noun from the question.
- Figure 5 indicates that CSS bases predictions on critical words such as “stove” or “lasagna”.
5. Conclusion
The paper proposes model-agnostic Counterfactual Samples Synthesizing (CSS) to improve VQA models’ visual-explainable and question-sensitive abilities. CSS masks critical objects or words to generate counterfactual training samples and consistently boosts different VQA models.
- CSS is a model-agnostic training scheme that improves visual-explainable and question-sensitive abilities in VQA models.
- CSS generates counterfactual training samples by masking critical objects or words in images and questions.
- CSS consistently boosts the performance of different VQA models, supported by comparative and ablative experiments.