Source-linked AI summary
Self-Critical Reasoning for Robust Visual Question Answering
Jialin Wu, Raymond J. Mooney
TL;DR
VQA systems can rely on language priors and fail when QA distributions shift between training and testing. The paper introduces self-critical training that strengthens regions influential for correct answers while criticizing competitive incorrect answers, achieving state-of-the-art VQA-CP performance with human or automatically derived regions.
Problem
VQA systems capture language correlations and fail to generalize when test QA distributions differ substantially from training distributions.
Method
The self-critical objective identifies influential regions for correct answers, strengthens them, and penalizes incorrect answers for sensitivity to those regions using human explanations or QA-derived objects.
Results
49.5% overall score is achieved with VQA-X textual explanations, while QA-derived regions achieve 48.5% on VQA-CP.
Takeaways & Limitations
The approach outperforms prior VQA systems on VQA-CP, including without human explanations as additional supervision.
Takeaways & Limitations
Counting problems remain difficult because they require considering all objects jointly, so criticizing only the most sensitive objects does not improve performance.
Abstract
from arXiv · showhide
Visual Question Answering (VQA) deep-learning systems tend to capture superficial statistical correlations in the training data because of strong language priors and fail to generalize to test data with a significantly different question-answer (QA) distribution. To address this issue, we introduce a self-critical training objective that ensures that visual explanations of correct answers match the most influential image regions more than other competitive answer candidates. The influential regions are either determined from human visual/textual explanations or automatically from just significant words in the question and answer. We evaluate our approach on the VQA generalization task using the VQA-CP dataset, achieving a new state-of-the-art i.e., 49.5% using textual explanations and 48.5% using automatically annotated regions.
1 Introduction
VQA systems can exploit language priors and fail when question-answer distributions shift, even when their visual explanations appear grounded. The paper introduces self-critical training to discourage incorrect answers from relying on regions influential for the correct answer and reports new VQA-CP results.
- Motivation: VQA systems often fail to generalize when test question-answer distributions differ substantially from training distributions.Language correlations can encourage answers based on question patterns rather than image content.
- Motivation: Systems may focus on human-important regions while still producing wrong answers.The baseline predicts “hot dog” while focusing on the banana for “What is the man eating?”.
- Approach: The self-critical approach penalizes incorrect answers for being sensitive to the region most influential for the correct answer.It first identifies the correct answer’s most influential region, then criticizes incorrect-answer sensitivity to that region.
- Approach: Important regions can come from human visual explanations, human textual explanations, or objects mentioned in questions and answers.The latter alternatives reduce reliance on detailed human visual annotations.
- Results: 49.5% overall score is achieved with VQA-X textual explanations, while 49.1% uses VQA-HAT visual explanations and 48.5% uses mentioned objects.These results are reported as new state-of-the-art performance on VQA-CP.
2 Related Work
Prior VQA research uses human explanations to encourage answers based on appropriate image regions, while language priors create shortcuts that undermine generalization. VQA-CP evaluates this problem by shifting question-answer distributions between training and test data.
- Human Explanations for VQA: Human VQA explanations are visual attention maps or textual justifications for answers.VQA-HAT provides visual attention annotations, while VQA-X associates textual explanations with QA pairs.
- Language Priors in VQA: Language priors are correlations between question types and their answers.These correlations can let systems answer questions without reasoning about visual content.
- Language Priors in VQA: VQA-CP reconfigures VQA data so training and test QA distributions differ substantially.Most state-of-the-art systems experience a catastrophic performance drop on this diagnostic setting.
- Human Explanations for VQA: VQA systems pursue being “right for the right reasons” by aligning explanations with regions human experts consider important.Prior approaches include inherently interpretable models and methods that align system explanations with human judgments.
3 Preliminaries
The paper builds on UpDn VQA, which represents detected objects and predicts answers from visual and question features. It constructs potentially influential object sets from explanations or QA text, then applies self-critical strengthening and criticism to answer sensitivities.
- Bottom-Up Top-Down VQA: Bottom-up attention detects common objects and attributes so top-down attention can model higher-level concept contributions.This architecture has been widely used in recent VQA systems.
- Bottom-Up Top-Down VQA: UpDn extracts feature vectors for detected image objects and combines them with a GRU-encoded question to predict answer confidences.The task uses multi-label regression with binary cross entropy over sigmoid-normalized outputs.
- Proposed Influential Object Set Construction: The proposed influential object set may contain irrelevant objects but is assumed to include the most relevant object.This relaxes the difficulty of obtaining an exact human-defined set of influential objects.
- Self-Critical Pipeline: The model overview strengthens the correct answer’s most influential object and criticizes competitive incorrect answers’ sensitivity to it.The pipeline detects objects, identifies the influential object, strengthens it, and reduces incorrect-answer sensitivity.
- Proposed Influential Object Set Construction: Proposal objects can be constructed from visual explanations, textual explanations, or nouns extracted from questions and answers.Textual explanations use nouns grounded to detected objects, while QA-only construction parses the QA pairs directly.
4 Approach
The approach first identifies and strengthens influential image objects, then criticizes incorrect answers that rely on those objects. It combines influence strengthening with a self-critical loss over competitive answers and is trained end-to-end on a differentiable VQA system.
- Approach: The system recognizes and strengthens the most influential objects before criticizing incorrect answers ranked above the correct answer.The approach contains separate components for influential-object strengthening and criticism of dominant incorrect answers.
- Influential-object recognition: Influence is measured with a modified GradCAM sensitivity to object features, removing ReLU units and feature-vector weighting.The modifications retain negative gradients and capture how changes in any direction affect the prediction.
- Influential-object recognition: The influence-strengthening loss requires the most sensitive proposal object to be at least as sensitive as every object outside the proposal set.This assumption requires the proposal set to contain at least one influential object that a human would use to infer the answer.
- Influential-object recognition: The loss relaxes exact human object rankings and adapts to explanation types, including textual explanations without detailed rankings.This distinguishes it from the ranking-based HINT loss.
- Criticizing incorrect answers: The self-critical loss minimizes competitive answers’ weighted sensitivities to the influential object selected for the correct answer.Competitive answers are predictions with higher confidence than the correct answer, and sensitivity gaps are weighted by cosine distance between 300-d Glove embeddings.
- Training: Training fine-tunes the joint objective L = L_vqa + λ_inflL_infl + λ_critL_crit on the VQA-X and VQA-CP training-set intersection.The implementation builds on the UpDn system and uses a competitive-answer bucket of size 5.
5 Experimental Results
Experiments on synthetic data and VQA benchmarks show that self-critical training improves robustness under shifted QA distributions, with strongest gains from textual explanations and competitive performance without human explanations.
- Results on Synthetic Data: Synthetic experiments vary class ratios between training and testing to evaluate whether self-critical training improves robustness under distribution shift.Figure 3 compares training and testing data, pretrained and fine-tuned decision boundaries, and test accuracies across p values from 0.05 to 0.5.
- Results on Synthetic Data: The fine-tuned decision boundary shifts toward the correct, unbiased position, increasing robustness and test-data accuracy.
- Results on VQA Data: 1.5% improvement over HINT is obtained on VQA-CP overall score using the same VQA-HAT visual explanations.
- Results on VQA Data: QA proposal objects outperform previous approaches without human explanations, while overlapping 57.1% of VQA-X and 54.3% of VQA-HAT proposal objects on average.
- Results on VQA Data: The objective particularly improves Yes/No and Other questions but performs less well for Num questions because counting requires jointly considering all objects.
- Results on VQA Data: The method remains competitive on VQA v2 and does not hurt performance when training and testing share the same distribution.
- Results on VQA Data: Combining influence-strengthening and self-critical losses reaches 49.5%, while performance improves across a broad range of loss weights.Without the self-critical loss, influence strengthening alone improves the UpDn system by 8.1% overall.
6 Conclusion and Future Work
The self-critical approach improves VQA generalization by criticizing incorrect answers’ sensitivity to influential objects. It outperforms state-of-the-art systems on VQA-CP without requiring human explanations as additional supervision.
- The approach criticizes incorrect answers’ sensitivity to the most influential object for the correct answer.
- Influential objects come from human visual or textual explanations or from objects mentioned in the questions and answers.
- The method outperforms state-of-the-art VQA systems on VQA-CP by a clear margin without human explanations as additional supervision.