Source-linked AI summary
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, Marcus Rohrbach
TL;DR
The paper addresses the limited ability of prior explainable models to combine visual evidence with textual justifications. It introduces human-annotated datasets and the PJ-X model for joint pointing and rationale generation, reporting that multimodal training improves both explanation modalities and reveals complementary strengths.
Problem
Prior explainable models generally offered either visual attention visualization or text-based post-hoc justification, rather than explanatory text grounded in image evidence.
Method
The paper collects ACT-X and VQA-X and trains PJ-X to jointly generate textual explanations and point to visual evidence using an explanatory attention mechanism.
Results
Multimodal explanations outperform unimodal visual or textual approaches, while textual supervision improves visual pointing and visual pointing improves textual explanation quality.
Takeaways & Limitations
Visual and textual explanations provide complementary information, with each modality offering cases that are more insightful than the other.
Abstract
from arXiv · showhide
Deep models that are both effective and explainable are desirable in many settings; prior explainable models have been unimodal, offering either image-based visualization of attention weights or text-based generation of post-hoc justifications. We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths. We collect two new datasets to define and evaluate this task, and propose a novel model which can provide joint textual rationale generation and attention visualization. Our datasets define visual and textual justifications of a classification decision for activity recognition tasks (ACT-X) and for visual question answering tasks (VQA-X). We quantitatively show that training with the textual explanations not only yields better textual justification models, but also better localizes the evidence that supports the decision. We also qualitatively show cases where visual explanation is more insightful than textual explanation, and vice versa, supporting our thesis that multimodal explanation models offer significant benefits over unimodal approaches.
1. Introduction
The paper addresses the gap between text-only and visual-only explanations by introducing multimodal explanations for visual tasks. It contributes new datasets and the PJ-X model, which jointly points to evidence and generates textual justifications.
- Prior explanation methods provided either text conditioned on an image or visualizations of active network units, but not explanatory text grounded in image evidence.
- The paper proposes PJ-X, a model that jointly generates visual and textual explanations using attention to localize salient regions during rationale generation.
- In the healthy-meal examples, PJ-X aligns textual references such as “hot dog” or “vegetables” with the corresponding visual regions rather than unrelated image items.
- VQA offers complementary image-question pairs with different answers, making fine-grained evidence use easier to assess, while activity recognition depends on cues such as pose, context, and human-object interaction.
- ACT-X and VQA-X provide human-annotated visual and textual explanations for activity recognition and visual question answering, enabling training and evaluation of multimodal explanation models.
- The authors report that multimodal explanations outperform models producing only visual or textual explanations and improve both visual pointing and textual justification quality.
2. Related Work
Related work includes textual explanation generation, visual explanation methods, VQA attention mechanisms, and activity-recognition models. The paper distinguishes PJ-X through jointly generated explanatory sentences and attention maps, supported by new explanation datasets.
- Earlier textual explanation systems were often template based, while later work generated natural-language justifications without multimodal explanations or reference human-explanation datasets.
- Visual explanation methods identify discriminative patches or interpret important intermediate features, whereas PJ-X uses attention to point to evidence and generates explanatory sentences.
- Unlike human-gaze alignment studies, the annotations ask annotators to identify the most relevant evidence for a decision while viewing the entire image.
- Prior VQA systems commonly use spatial attention, and this method differs from related multimodal pooling approaches in its normalization choices.
- Activity-recognition work uses cues such as pose and global context, often assuming the ground-truth human location, whereas this paper considers a more realistic setting without that assumption.
3. Multimodal Explanations
The paper defines multimodal explanation tasks for VQA and activity recognition, pairing human textual justifications with visual evidence annotations. It introduces VQA-X and ACT-X, including complementary VQA examples and region-level annotations for evaluating explanations.
- Datasets and task definition: VQA-X and ACT-X define explanation tasks with both textual and visual justifications for decisions in VQA and activity recognition.Human annotators provide textual explanations and segment image regions that justify the answer or activity label.
- VQA-X: VQA-X focuses on questions beyond trivial visual properties and includes complementary image pairs with different answers.Complementary pairs test whether explanations identify evidence from image content rather than merely memorizing question-specific cues.
- ACT-X: ACT-X contains 18,030 images covering 397 activities, with three textual explanations collected for each image.Annotators complete a sentence explaining why the person is performing the ground-truth activity and are asked to use at least 10 words.
- Visual annotations: Visual explanations are collected by asking annotators to segment objects or regions that most prominently justify each answer or activity label.For sampled test images, the datasets collect three visual annotations per image using a segmentation interface.
- Comparison with prior annotations: VQA-X annotations can be finer-grained and contain less extraneous information than VQA-HAT annotations in qualitative comparisons.The paper cautions that a thorough quantitative comparison is not viable because the datasets have different splits and limited overlap.
4. Pointing and Justification Model (PJ-X)
PJ-X first predicts an answer from the image and question, then uses the answer, question, and image to jointly produce visual pointing and textual justification. Attention localizes evidence and also supplies features to the text decoder.
- Visual pointing: The model learns latent pointing through an attention mechanism using natural-language justifications and classification labels as supervision.The attention mechanism focuses on a spatial subset of the visual representation without requiring explicit pointing labels for training.
- Pipeline: PJ-X predicts the task answer first, then generates visual and textual explanations conditioned on the answer, question, and image.For activity recognition, the question representation is omitted because the task has no explicit question.
- Answering model: The answering model extracts spatial ResNet-152 features, encodes the question with an LSTM, and combines them using element-wise multiplication and normalized feature processing.For activity recognition, setting the question representation to a vector of ones removes the explicit question input.
- Visual pointing: PJ-X combines image, question, and answer representations to predict a normalized spatial attention map over visual features.The answer embedding is fused with question-image features before producing the attention map and applying softmax.
- Textual justification: The attended visual representation is merged with question and answer features and fed to an LSTM decoder for textual justification generation.The decoder predicts a sequence of words conditioned on the previous word and its hidden state.
5. Experiments
The experiments evaluate textual and visual explanations on VQA-X and ACT-X using automatic, human, ablation, and qualitative analyses. Results favor multimodal training: attention improves textual justifications, textual supervision improves localization, and the modalities can be complementary.
- Experimental setup: The experiments compare PJ-X with ablations and related approaches on VQA-X and ACT-X using automatic and human evaluations.Textual justifications are evaluated with BLEU-4, METEOR, ROUGE, CIDEr, SPICE, and human judgments; visual pointing uses Earth Mover’s Distance and Rank Correlation.
- Textual Justification: Training on explanations substantially outperforms training on descriptions, showing that descriptions are insufficient for generating task-specific explanations.The comparison favors “Ours” over “Ours with Descriptions” on both datasets, while differences involving discriminative loss and policy gradients remain for future investigation.
- Textual Justification: Including attention improves textual justifications on both datasets compared with the model without attention.The attention mechanism provides visual rationale while also helping the model generate better textual explanations.
- Visual Pointing: PJ-X outperforms random, uniform, answering-model, and prior-method baselines on both visual-pointing metrics across VQA-X and ACT-X.Earth Mover’s Distance is lower-is-better and Rank Correlation is higher-is-better; textual annotations provide latent supervision for the attention maps.
- Qualitative Results: Qualitative results show textual explanations capture commonsense and image-specific cues, while attention maps identify relevant regions for VQA and activity recognition.Examples include animals in an enclosure, foam on waves, a person and lawn mower, and contextual distinctions between mountain and road biking.
- Qualitative Results: Explanations remain consistent with incorrect predictions, and humans judge explanations as more helpful than having no explanation or using weaker explanation models.In qualitative examples, the model attends to evidence supporting an incorrect “vacuuming” prediction; the diagnostic evaluation hides the predicted answer.
- Usefullness of Multimodal Explanations: Visual pointing and textual justification provide complementary explanatory value: each can clarify cases where the other modality is uninformative.Pointing conveys concepts such as leaning, whereas language can better justify properties such as whether a sky is cloudy.
6. Conclusion
The paper presents multimodal explanations that combine natural-language justifications with visual evidence pointing, supported by the VQA-X and ACT-X datasets. It reports that textual supervision improves visual localization and that the model produces human-like textual and visual explanations.
- Conclusion: PJ-X provides natural-language justifications and visual pointing for decisions in visual question answering and activity recognition.The paper introduces the VQA-X and ACT-X datasets with human-annotated multimodal explanations.
- Conclusion: Learning to point improves textual explanation quality, while reference textual explanations improve visual pointing.These results support bidirectional benefits between the two explanatory modalities.
- Conclusion: Qualitative results show PJ-X can point to decision-relevant image evidence while generating natural sentence justifications similar to human explanations.The conclusion characterizes the model as providing both explanatory modalities for the same decision.