PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual VQA: A Cause-Effect Look at Language Bias

Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, Ji-Rong Wen

arXiv:2006.04315v4cs.CVcs.CL

TL;DR

VQA models may rely on language shortcuts instead of multimodal reasoning, and existing approaches do not fully separate useful language context from language bias. CF-VQA uses counterfactual causal effects to subtract the direct question effect from the total effect. Experiments report generality across architectures and fusion strategies, competitive VQA-CP performance, and robust VQA v2 performance without augmented data.

  • Problem

    VQA models may rely on spurious linguistic correlations rather than multimodal reasoning, while unbiased inference under biased training remains a challenge.

  • Method

    CF-VQA formulates language bias as the direct causal effect of questions on answers and subtracts it from the total causal effect using conventional and counterfactual inference.

  • Results

    CF-VQA achieves a new state-of-the-art result among non-augmentation approaches on VQA-CP v2 and performs robustly across in-domain and out-of-distribution settings.

  • Takeaways & Limitations

    The framework generalizes across VQA architectures and fusion strategies and can unify recent debiasing methods within a counterfactual interpretation.

  • Takeaways & Limitations

    Estimating NDE and TIE fails when c is extremely large or small, so NDE sharpness must be controlled by selecting c.

Abstract

from arXiv · show

VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect. Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies, 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset without any augmented data. The code is available at https://github.com/yuleiniu/cfvqa.

1. Introduction

VQA models can exploit spurious question–answer correlations instead of learning multimodal reasoning, motivating CF-VQA’s causal separation of language bias from useful context. The framework estimates total and direct language effects through conventional and counterfactual scenarios, then subtracts the latter for debiased inference.

  • Motivation: Spurious linguistic correlations can yield approximately 40% accuracy for “tennis” answers and 90% for “yes” answers on VQA v1.0.These shortcuts can impair generalization when models memorize training-language priors.
  • Motivation: Extra annotations, data augmentation, and counterfactual training improve debiasing, but unbiased inference under biased training remains challenging.VQA-CP is intended to test whether models disentangle visual knowledge from memorized language priors.
  • CF-VQA: CF-VQA formulates language bias as the direct causal effect of questions on answers and reduces it by subtracting that effect from the total causal effect.The framework uses conventional VQA to estimate the total effect and counterfactual VQA to estimate the direct language effect.
  • CF-VQA: Counterfactual VQA blocks multimodal knowledge while retaining the question, exposing the model’s single-modal linguistic impact.The counterfactual scenario asks what answer would result if the model heard Q but had not extracted K or seen V.
  • Contributions: The proposed cause-effect framework is intended to apply across VQA architectures and fusion strategies while interpreting recent debiasing methods causally.The paper presents generality across baseline architectures and fusion strategies as a contribution.

2. Related Work

Language bias in VQA includes question–answer language priors and visual priming, while prior mitigation methods target visual grounding, language priors, or training-data distributions. These approaches use explanations, separate language branches, or generated data to reduce bias.

  • Language Bias: Language bias includes strong question–answer correlations, such as answering “tennis” to sport questions for approximately 40% accuracy.It also includes visual priming, where “yes” answers to “Do you see a ...” questions achieve nearly 90% accuracy.
  • Debiasing Methods: Recent debiasing methods strengthen visual grounding, weaken language priors, or use implicit and explicit data augmentation.These categories organize the main approaches reviewed in the section.
  • Debiasing Methods: Visual and textual explanations are used to improve visual grounding, while separate question–answer branches capture language priors through adversarial or multi-task learning.The reviewed methods differ in whether they strengthen visual evidence or model language bias explicitly.

3. Preliminaries

The paper introduces causal graphs, counterfactual notation, and causal-effect decompositions before applying them to VQA. Total effects can be separated into direct and indirect components by intervening on a mediator.

  • Causal Inference: Causal graphs represent variables and directed cause–effect relationships, with mediators carrying indirect effects between treatment and outcome.A direct effect is represented by X → Y, while an indirect effect through M is represented by X → M → Y.
  • Counterfactuals: Counterfactual notation describes outcomes under interventions that can assign different values to variables in the same hypothetical world.For example, Yx,Mx∗ uses X = x while setting M to the value it would have obtained under X = x∗.
  • Causal Effects: The no-treatment condition uses X = x∗, contrasted with the treatment condition X = x, to define potential outcomes and causal effects.The do operator can be omitted when X has no confounder.
  • Causal Effects: Total effect decomposes into natural direct effect and total indirect effect, with NDE blocking the mediator’s response to treatment.TIE is defined as the difference between TE and NDE.
  • Causal Effects: Total effect can alternatively decompose into natural indirect effect and total direct effect, distinguishing mediation through M from the direct X → Y path.NIE captures the effect transmitted through the mediator while the direct path is blocked.

4. Cause-Effect Look at VQA

CF-VQA models VQA causally by separating direct single-modal effects from multi-modal effects, then uses a debiased causal effect for inference. Its implementation combines language-only, vision-only, and vision-language branches while retaining counterfactual no-treatment conditions.

  • VQA selects an answer from a candidate set given an image and question.
  • The causal graph separates direct paths from image or question to answer and an indirect path through multi-modal knowledge.The direct paths represent single-modal impact, while the knowledge-mediated path represents multi-modal impact.
  • CF-VQA blocks multi-modal knowledge and visual effects to estimate the question’s natural direct effect, which captures language bias.
  • Inference selects the answer with maximum total indirect effect rather than maximum posterior probability P(a|v, q).
  • The score uses language-only, vision-only, and vision-language neural branches fused by a function h, with Harmonic and SUM variants.
  • Under no-treatment conditions, missing vision or language inputs are represented as empty values, and the model assumes uniform random guessing.A learnable parameter c controls distribution sharpness for counterfactual scores, while training combines classification and KL-divergence losses.

5. Experiments

Experiments evaluate CF-VQA on VQA-CP v2 and VQA v2 across architectures, fusion strategies, causal graphs, assumptions, and qualitative examples. CF-VQA improves non-augmentation performance, especially on Yes/No questions, while retaining useful language context and showing robustness on VQA v2.

  • Quantitative Results: CF-VQA achieves new state-of-the-art performance on VQA-CP v2 among non-augmentation approaches.Methods generating extra training samples are excluded from direct comparison because they change training priors and conflict with VQA-CP’s evaluation intention.
  • Quantitative Results: Yes/No accuracy improves from ∼70% to ∼90%, showing that language bias affects question types differently.CF-VQA remains comparable to methods using extra annotations or generated samples on Other questions.
  • Ablation Studies: CF-VQA outperforms ensemble-based strategies by over 2% for HM and over 5% for SUM across the tested settings.The ablations assess different baseline architectures, fusion strategies, causal graphs, and no-treatment distribution assumptions.
  • Ablation Studies: The uniform distribution assumption performs better than random and prior assumptions, which perform worse than the baselines.The authors associate the uniform assumption with safer estimation of the biased language effect, formalized as NDE.
  • Quantitative Results: Replacing NIE with TIE and adding one learnable parameter improves RUBi from 47.11 to 54.69, exceeding 7.5% on VQA-CP v2.Accuracy slightly drops on VQA v2, while symmetric fusion strategies show greater stability and robustness than RUBi.
  • Qualitative Results: Qualitative examples show CF-VQA reducing language-prior errors while preserving useful language context, including “large or small” and more meaningful object answers.Compared with RUBi, CF-VQA avoids some yes/no responses driven by the wrong context and responds with answers such as wood and frisbee.

6. Conclusion

The paper concludes that CF-VQA reduces language bias by estimating it as a direct linguistic causal effect and subtracting it from the total causal effect. The framework is effective and generalizable, unifies recent debiasing studies, and improves RUBi with minimal changes, while balancing robustness and debiasing remains future work.

  • Conclusion: CF-VQA formulates language bias as the direct causal effect of questions on answers and subtracts it from the total causal effect.The bias is estimated through counterfactual reasoning.
  • Conclusion: The framework demonstrates effectiveness and generalizability across VQA settings and can unify recent debiasing studies.The conclusion presents this unification as an interpretation within the proposed counterfactual framework.
  • Conclusion: CF-VQA improves RUBi by changing a few lines of code and adding only one more learnable parameter.This result follows from the paper’s cause-effect interpretation of debiasing methods.
  • Conclusion: Balancing robustness and debiasing ability remains a future research direction.The authors identify this balance as an unresolved consideration.

7. Revisiting RUBi and Learned-Mixin

RUBi and Learned-Mixin can be interpreted within CF-VQA as using natural indirect effects, while CF-VQA improves RUBi by using total indirect effects that retain language context.

  • Causal interpretation: RUBi and Learned-Mixin use ensemble architectures with vision-language and question-only branches, without modeling a direct vision-to-answer relation.Their simplified causal graph omits the direct path V → A.
  • Causal interpretation: The total effect is reduced by subtracting a direct effect, while TIE and NIE differ in whether the question is retained or blocked during inference.TIE retains question q as language context; NIE blocks q by using q∗.
  • Existing methods: RUBi and Learned-Mixin use the vision-language score z_k for inference, which is proportional to their natural indirect effect.Their inference procedure preserves the vision-language branch rather than using the total indirect effect.
  • Improving RUBi: CF-VQA improves RUBi by replacing natural indirect effect inference with total indirect effect inference.The change is implemented by modifying several lines in the RUBi procedure.
  • Improving RUBi: CF-VQA outperforms RUBi by 7% on VQA-CP v2.The comparison is reported in Table 5 of the main paper.

8. Analysis of Estimating NDE

The analysis shows that estimating NDE and TIE depends on controlling the sharpness parameter c; extreme values make language bias persist in the indirect effect.

  • Parameter analysis: The learnable parameter c controls the sharpness of Zq,v∗,k∗ when estimating NDE.For Harmonic, c_HM = (σ(c))^2 lies in (0, 1).
  • Extreme settings: For very large or small c, NDE becomes poorly estimated and TIE is dominated by zq,v,k.The analysis uses an extremely negative number C as a substitute for −∞ in valid TIE estimation.
  • Implication: In both analyzed cases, the language bias cannot be excluded when c is extremely large or small.The authors therefore identify selection of an appropriate c as necessary for controlling NDE sharpness.
  • Implication: The main paper uses a KL-divergence term to make NDE sharpness similar to that of the total effect.This regularization is intended to avoid the extreme-parameter estimation problem.

9. Implementation Details

The experiments use standardized RUBi implementations with Faster R-CNN visual features, Skip-thought question embeddings, several VQA backbones, and Adam optimization on VQA-CP and VQA datasets.

  • Experimental setup: The implementation follows RUBi for feature representation, baseline architectures, and optimization to ensure fair comparison.Experiments use Adam for 22 epochs with batch size 256 and a scheduled learning rate.
  • Representations: Visual features come from Faster R-CNN top-K region proposals, with K fixed at 36 for each image.This follows a bottom-up attention mechanism.
  • Representations: Questions are lowercased, stripped of punctuation, and encoded with a finetuned pretrained Skip-thought encoder producing 4800-dimensional embeddings.
  • Model branches: The vision-language branch combines image, question, and visual-knowledge representations using SAN, UpDn, or S-MUREL baseline encoders.S-MUREL uses BLOCK bilinear fusion per region followed by a three-layer fully connected classifier.
  • Model branches: Separate language-only and vision-only classifiers use three-layer ReLU MLPs with structures matching the vision-language classifier but different parameters.
  • Datasets: Experiments are conducted on VQA-CP v1, VQA-CP v2, and VQA datasets formed from reorganized VQA train and validation splits.VQA-CP v1 and v2 correspond to reorganized VQA v1 and v2 splits, respectively.

10. Supplementary Experimental Results

Supplementary experiments show that CF-VQA remains effective across in-domain and out-of-distribution settings, baseline architectures, and fusion strategies, while qualitative examples reveal both useful language context and failure cases.

  • Quantitative Results: CF-VQA variants outperform GRLS by large margins in both in-domain and out-of-distribution settings.The in-domain evaluation holds out 8,000 training instances as VQA-CP v2 validation data.
  • Quantitative Results: CF-VQA (HM) performs better than GradSup on both VQA-CP validation and test sets.
  • Quantitative Results: CF-VQA (SUM) is competitive with RandImg on VQA-CP v2 test data and exceeds it by over 3% in-domain.
  • Quantitative Results: Ablations show CF-VQA is general across SAN, UpDn, and S-MUREL baseline architectures and across fusion strategies.The same pattern is reported for both the standard and simplified causal graphs.
  • Qualitative Results: Qualitative examples show CF-VQA benefiting from language context such as size, depth, and object-versus-statue distinctions.
  • Qualitative Results: Failure cases include broad answers such as “houses” instead of “church” and “vegetables” instead of “peas,” alongside cases where visual information may be ignored.
Loading 2006.04315v4…