Source-linked AI summary
Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers
Hila Chefer, Shir Gur, Lior Wolf
TL;DR
Existing Transformer explainability methods inadequately handle co-attention and other non-self-attention architectures. The paper introduces a generic relevance-propagation method for three Transformer types and reports superior performance across architectures and evaluation metrics, with important evaluation boundaries in the DETR segmentation setup.
Problem
Existing Transformer explainability methods focus on pure attention maps or self-attention and lack methodology for co-attention maps.
Method
The paper tracks attention-map evolution and mixing to produce relevancy maps for pure self-attention, self-attention with co-attention, and encoder-decoder attention.
Results
The method outperforms existing methods across Transformer architectures and evaluation metrics, often by a sizable margin.
Takeaways & Limitations
The generic prescription is applicable to attention-based architectures and supports explanations for models involving multiple attention types.
Takeaways & Limitations
DETR-derived segmentation masks depend on predicted bounding-box quality, while thresholding and noncontiguous outputs can introduce inaccuracies.
Abstract
from arXiv · showhide
Transformers are increasingly dominating multi-modal reasoning tasks, such as visual question answering, achieving state-of-the-art results thanks to their ability to contextualize information using the self-attention and co-attention mechanisms. These attention modules also play a role in other computer vision tasks including object detection and image segmentation. Unlike Transformers that only use self-attention, Transformers with co-attention require to consider multiple attention maps in parallel in order to highlight the information that is relevant to the prediction in the model's input. In this work, we propose the first method to explain prediction by any Transformer-based architecture, including bi-modal Transformers and Transformers with co-attentions. We provide generic solutions and apply these to the three most commonly used of these architectures: (i) pure self-attention, (ii) self-attention combined with co-attention, and (iii) encoder-decoder attention. We show that our method is superior to all existing methods which are adapted from single modality explainability.
1. Introduction
Multi-modal and encoder-decoder Transformers create interactions beyond self-attention that existing explainability methods do not adequately support. The paper proposes a generic method covering the main Transformer attention architectures.
- Image-text Transformers combine modalities through interactions that challenge explainability methods designed for self-attention.Existing methods do not adapt to other attention forms, despite their use in multi-modal models.
- Encoder-decoder Transformers are used in object detection, image segmentation, and natural-language processing.
- The proposed method is the first explainability approach applicable to pure self-attention, co-attention, and encoder-decoder attention.
- The method is demonstrated with exemplar models from all three architectures and is reported to outperform adapted single-modality explainability methods.The authors also describe it as easier to implement and readily applicable to attention-based architectures.
2. Related work
Prior Transformer explainability methods often use attention maps or propagate relevance through self-attention, but they do not fully capture attention computation or support non-self-attention modules.
- Transformer explainability methods commonly rely directly on attention maps, which neglect intermediate scores and other Transformer components.
- Attention mixes queries, keys, and values, so query-key inner products alone cannot fully capture the computation.
- LRP can estimate the relative importance of attention heads but does not propagate relevance to input heatmaps.
- Attention rollout can accumulate relevancy across layers when positive and negative contributions should cancel.
- Chefer et al. propagate information through all Transformer components but provide no solution for attention modules other than self-attention.
3. Method
The method propagates class-dependent relevancy through attention interactions, maintaining separate maps for self-attention and cross-modal connections. It adapts these rules to concatenated, bi-modal, and encoder-decoder Transformers.
- The method constructs separate relevancy maps for text-text, image-image, and cross-modal interactions.Cross-modal maps represent image influence on text and text influence on image.
- Attention maps are updated during a forward pass, with each layer contributing to aggregated relevance matrices through propagation rules.Initial self-attention relevancies are identity matrices, while modalities initially have no cross-modal context.
- Residual connections are handled by adding each layer’s contribution to aggregated relevancies.
- Gradient-weighted averaging across heads produces the final attention map, with negative contributions removed before averaging.
- For bi-modal attention, aggregated self-attention matrices are row-normalized so token self-influence and contextualization receive equal treatment.The normalized matrices account for prior within-modality contextualization before cross-modal updates.
- For classification, the row corresponding to the [CLS] token yields relevancies for text tokens and image tokens.The text relevancies come from Rtt, while image relevancies come from Rti when [CLS] is in the text modality.
- 3.2. Adaptation to attention types: Pure self-attention uses one joint relevancy map after concatenating modalities, while encoder-decoder models use encoder, decoder, and decoder-to-encoder maps.For object detection, each row of Rde corresponds to a detected object’s image-region relevancies.
4. Baselines
The paper compares attention-map, gradient, and relevancy-map baselines for Transformer explainability, including adaptations for non-self-attention modules. These baselines differ in which attention layer information they use and how much implementation support they require.
- Baseline classes: The baselines comprise attention-map, gradient, and relevancy-map methods.Attention baselines include raw attention and rollout; gradient baselines use Grad-CAM; relevancy baselines include partial LRP and Transformer attribution.
- Attention-map baselines: Raw attention uses the last layer’s attention map directly as the relevancy map.For example, the text relevancy is defined from the last text self-attention map.
- Attention-map baselines: Rollout propagates information through all self-attention layers, while its bi-modal variant combines self-attention relevancies with the last bi-attention map.The bi-modal formulation uses self-attention relevancies for each modality and a final bi-attention map.
- Gradient baselines: The gradient baseline adapts Grad-CAM by examining the last attention layer and applying it across that attention map’s heads.
- Relevancy-map baselines: Relevancy-map baselines include partial LRP and Transformer attribution, with the latter applying the paper’s attention rules across layers.For non-self-attention layers, the adapted Transformer attribution method averages the last attention map across heads.
- Implementation: The proposed method requires only simple attention-module hooks, whereas LRP requires custom implementations for all network layers.
5. Experiments
Experiments evaluate the method across self-attention, co-attention, and encoder-decoder Transformer architectures using perturbation tests, segmentation, and visual comparisons. The method generally outperforms existing explainability baselines, while showing especially focused image relevancies for LXMERT and strong segmentation results for DETR.
- Experimental setup: The experiments cover VisualBERT, LXMERT, and DETR as representatives of self-attention, self-attention with co-attention, and encoder-decoder architectures.VisualBERT and LXMERT are evaluated on visual question answering, while DETR is evaluated on object detection-derived segmentation.
- Experimental setup: Positive and negative perturbation tests remove tokens in opposite relevance orders and evaluate accuracy-area curves on separate image and text modalities.Positive perturbation removes highest-relevance tokens first; negative perturbation removes lowest-relevance tokens first, with AUC measuring accuracy decrease.
- LXMERT: LXMERT image perturbation gives the proposed method the largest negative-perturbation AUC and a low positive-perturbation AUC, preserving accuracy after removing over 80% of image tokens.The corresponding multimodal relevancy map exposes the advantage over methods that do not evaluate co-attention relevancies.
- LXMERT: For LXMERT text perturbation, the proposed method and Transformer attribution produce very similar results because both use the same self-attention update rule, differing mainly in head averaging.The proposed method uses LRP for head averaging, whereas the baseline does not.
- VisualBERT: For VisualBERT, the proposed method outperforms all methods and sometimes exceeds Transformer attribution by a sizeable margin, indicating that LRP is unnecessary for pure self-attention.The comparison isolates LRP-based versus non-LRP head averaging because VisualBERT uses pure self-attention.
- DETR: DETR evaluation produces segmentation masks from predicted bounding boxes on 5,000 MSCOCO validation samples, and the proposed method outperforms all other methods by a very large margin.Evaluation lowers the minimum IoU from 0.5 to 0.2 and reports categories with sufficiently good bounding boxes for mask generation.
6. Conclusions
The paper addresses limited Transformer explainability by tracking attention across pure self-attention, co-attention, and encoder-decoder architectures. Its method generally outperforms existing approaches, while released notebooks provide LXMERT and DETR examples.
- Current Transformer explainability is limited, overly focused on pure attention maps, and lacks methodology for co-attention maps.
- A stricter evaluation threshold makes baseline results slightly better than chance, while the proposed method scores only 2–3 times better than chance.
- The method empirically outperforms existing methods across Transformer architectures and evaluation metrics.Chefer et al. can provide comparable results when self-attention is prominent, but the proposed method leads by a sizable margin in most experiments.
- The released code includes Jupyter notebooks with LXMERT and DETR examples, including internet-image support and free-form questions for LXMERT.
B. Extended LXMERT VQA visual results
Figure 3 extends the LXMERT VQA explanation results with typical MSCOCO samples, comparing the proposed method against baselines.
- Figure 3 presents explanations extracted by each method for typical VQA samples using LXMERT.
C. Preparing the DETR relevancy maps for the COCO segmentation evaluation code
The DETR evaluation converts query-level relevancy maps into segmentation masks, while Figure 3 compares multimodal explanation maps for LXMERT VQA.
- DETR produces a bounding box and classification for each detected object, and evaluation uses R_qi to represent feature relevance for each query.
- The evaluation applies explainability methods to 5,000 MSCOCO validation samples to produce segmentation masks.
- The mask-extraction algorithm takes the image, detection logits, and per-query relevancy matrices, then outputs one spatial mask per bounding-box query.
- Figure 3 compares the proposed method with baselines for LXMERT VQA, using red shades for text relevance and region-weighted relevance for images.The displayed answers are no, yes, yes, and no from left to right.
D. Ablation Study
The ablation study tests normalization, aggregation, and self-attention components across DETR and LXMERT evaluations. Removing components substantially harms DETR performance, while LXMERT differences are milder for selected ablations.
- The study evaluates normalization, aggregation, and self-attention components through multiple ablations of the proposed relevancy rules.
- All included components are crucial for DETR performance, with ablations causing a sizeable decrease.
- Even without normalization or self-attention in Eq.10, the DETR ablations significantly outperform all other methods.
- LXMERT shows relatively mild differences between the full method and no-normalization or no-self-attention ablations in the image perturbation test.
- The milder LXMERT effects are attributed to its use of 36 NMS-filtered image regions, which reduces the added importance of self-attention context.
E. Using LRP with our method
On LXMERT, adding LRP does not materially improve the perturbation-test results and can slightly reduce performance. Because LRP is complex to implement, the authors advocate omitting it.
- LRP produces a large performance gap over all baseline methods for image perturbation tests, whether included or omitted.
- For text perturbation tests, the method performs similarly to Transformer Attribution, regardless of whether LRP is used.These tests are self-attention based, and the choice of LRP is insignificant.
- LRP itself causes a small degradation in performance on the evaluated LXMERT tests.
- The authors recommend eliminating LRP because its implementation is complex and its performance effect is insignificant.
F. Perturbation experiments graphs
The perturbation experiments are presented through enlarged graphs and an AUC table covering LXMERT and VisualBERT. Evaluation distinguishes negative from positive perturbations and separately varies image and text tokens.
- The enlarged graphs in Figures 4 and 5 correspond to the perturbation experiments and improve their visual clarity.
- Table 5 reports area-under-the-curve values for LXMERT baselines and the method with and without LRP.
- For negative perturbation, larger AUC is better, whereas for positive perturbation, smaller AUC is better.
- Figures 4 and 5 organize results by negative or positive perturbation applied to image or text tokens.Each figure contains four corresponding panels.