Source-linked AI summary
Towards Transparent AI Systems: Interpreting Visual Question Answering Models
Yash Goyal, Akrit Mohapatra, Devi Parikh, Dhruv Batra
TL;DR
VQA models provide answers to image questions, but their predictions can be difficult to understand. This paper uses guided backpropagation and occlusion to map important question words and image regions. Its analyses find appropriate question-word importance and sometimes relevant image-region attention, while question-word evaluation relies on POS tags because human attention annotations are unavailable for questions.
Problem
VQA predictions can be difficult to understand, leaving users without an explanation of why a system answered as it did.
Source: §1 · pp. 1–2Method
The paper uses guided backpropagation and input occlusion to estimate the importance of question words and image regions.
Source: §1 · p. 2Results
The analyses suggest that VQA models often emphasize appropriate question words and may sometimes attend implicitly to relevant image regions without explicit attention mechanisms.
Source: §1 · p. 2, §4.2 · p. 5Takeaways & Limitations
The two visualization methods provide importance maps for interpreting VQA predictions, with image maps also compared against human attention maps.
Source: §1 · p. 2, §5 · p. 5Because no human attention dataset is available for questions, question-word importance is analyzed using POS tags instead.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Deep neural networks have shown striking progress and obtained state-of-the-art results in many AI research fields in the recent years. However, it is often unsatisfying to not know why they predict what they do. In this paper, we address the problem of interpreting Visual Question Answering (VQA) models. Specifically, we are interested in finding what part of the input (pixels in images or words in questions) the VQA model focuses on while answering the question. To tackle this problem, we use two visualization techniques -- guided backpropagation and occlusion -- to find important words in the question and important regions in the image. We then present qualitative and quantitative analyses of these importance maps. We found that even without explicit attention mechanisms, VQA models may sometimes be implicitly attending to relevant regions in the image, and often to appropriate words in the question.
1 Introduction
VQA systems can produce answers without explaining why, leaving users unable to understand their predictions. This work analyzes which question words and image pixels support a VQA answer using two visualization methods.
1 Introduction
- The study interprets a VQA model that combines CNN image and LSTM question embeddings to predict an answer distribution.VQA takes an image and a free-form natural-language question and returns a natural-language answer.Source: §1 · pp. 1–2
- VQA systems can be inscrutable despite their ability to recognize image content, making it unsatisfying to lack explanations for their predictions.The paper frames this lack of transparency as a problem for systems that may also make glaring mistakes.Source: §1 · pp. 1–2
- Guided backpropagation and occlusion produce importance maps for question words and image pixels, which the paper analyzes qualitatively and quantitatively.Question maps are analyzed using POS tags, while image maps are compared with human attention maps.Source: §1 · p. 2
- The analyses suggest that VQA models may implicitly attend to appropriate question words and sometimes to relevant image regions without explicit attention mechanisms.Figure 1 illustrates “whole” as the most important question word for the example prediction “no.”Source: Figure 1 · p. 1, §1 · p. 2
2 Related Work
Prior visualization work largely focused on image classification, while posthoc explanation methods may train a separate interpretable mechanism. This work applies guided gradients to VQA inputs and computes importance maps directly from the model being interpreted.
2 Related Work
- The paper applies guided backpropagation to VQA and computes gradients with respect to both the question and image, addressing a task not previously studied in its literature review.Earlier gradient-based visualization methods focused mainly on image classification, often with iconic images dominated by a main object.Source: §2 · p. 2
- Unlike image-classification visualization, VQA can require different image regions for different questions about the same image.The paper’s occlusion experiment is inspired by Zeiler and Fergus (2014), who masked image regions and observed classification outputs.Source: §2 · p. 2
- Unlike posthoc methods that fit a separate interpretable mechanism, this work computes importance maps from the model itself without another training layer that could obfuscate analysis.The cited examples of such posthoc methods are Ribeiro et al. (2016), Baehrens et al. (2010), and Liu and Wang (2012).Source: §2 · p. 2
3 Approach
The approach estimates the importance of question words and image pixels through local derivatives or input occlusion. Guided backpropagation modifies ReLU backpropagation, while discrete derivatives measure how masking an input changes the predicted answer probability.
3.2 Discrete Derivatives
- Figure 2 visualizes discrete-derivative maps for question words and image regions, with “vegetable” and the parrot region most affecting the illustrated predictions.The examples correspond to predicting “broccoli” for the first question and the top image’s prediction for the parrot region.Source: Figure 2 · p. 3
3 Approach
- Input derivatives of the predicted answer probability approximate the model locally and indicate how image pixels or question words affect its prediction.The derivatives are with respect to input pixels and word embeddings, rather than model parameters or the ground-truth answer.Source: §3 · pp. 2–3
3.1 Guided Backpropagation
- Guided backpropagation differs from classical backpropagation in the ReLU backward pass, blocking negative gradients from flowing toward the input.The method computes gradients of the predicted answer probability with respect to image and question inputs; the language pathway described does not contain ReLUs, so its values are true gradients.Source: §3.1 · p. 3
3.2 Discrete Derivatives
- Discrete derivatives mask one input component at a time and measure the resulting change in probability of the answer predicted from the unmasked input.For questions, the method drops a word while keeping the image fixed; for images, it masks one cell of a 16 x 16 grid while retaining the full question.Source: §3.2 · p. 4
4 Results
Image-importance maps show weak positive alignment with human attention, well below inter-human agreement; question-importance analysis ranks wh-words highest, followed by adjectives and nouns. Answer changes under occlusion also correlate with model accuracy.
4.1 Analyzing Image Importance
- Across 1,374 VQA validation pairs, occlusion and guided-backprop maps correlate weakly positively with human attention, well below inter-human agreement.Human annotations mark image regions helpful for answering questions; both map types are compared with these annotations.Source: Table 1 · p. 4, §4.1 · p. 4
4.2 Analyzing Question Importance
- Among the 15 most frequent grouped POS tags, wh-words most often rank as most important, followed by adjectives and nouns.The POS categories are ordered by frequency in the dataset; adjectives and nouns rank high because questions often ask about object characteristics or objects.Source: Figure 3 · p. 4, §4.2 · p. 5
- 72% of the time, features of the importance maps predict success or failure accurately, and answer changes under occlusion correlate with model accuracy.The authors suggest these features may provide signals for predicting the model’s oncoming failures.Source: §4.2 · p. 5
5 Conclusion
The paper applies guided backpropagation and occlusion to interpret VQA models, reporting that the methods generalize beyond the single model studied. Occlusion can also be used with non-end-to-end VQA models as a black box.
5 Conclusion
- Although the experiments focus on one VQA model, the authors state that both methods generalize to other end-to-end VQA models and occlusion can apply to any VQA model as a black box.They suggest the methods and results may help interpret current VQA models and design future ones.Source: §5 · p. 5