PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Attention-Model Explainability through Faithfulness Violation Test

Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, Shiqi Wang

arXiv:2201.12114v3cs.LGcs.CLcs.CV

TL;DR

Attention explanations can assign high weight to features that suppress rather than support predictions, exposing a gap in faithfulness evaluation. The paper tests polarity consistency by perturbing the feature with the largest absolute explanation weight; experiments find most tested methods violate it, with raw attention worst and gradient-based attention alleviating violations.

  • Problem

    Attention explanations can assign high weights to features that suppress rather than support predictions, while their faithfulness has not been properly evaluated.

    Source: §1 · pp. 1–2

  • Method

    The faithfulness violation test perturbs the feature with the largest absolute explanation weight and checks whether its impact polarity matches the explanation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Most tested methods show faithfulness violations; raw attention performs worst, while attention ⊙gradient alleviates violations and can pass in special cases.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The violation test detects polarity violations that existing faithfulness metrics miss, providing a complementary criterion for evaluating attention explanations.

    Source: §1 · p. 2, §5 · p. 5

    Confidence perturbation can depress test results, although the authors report using replacement functions they consider promising for addressing out-of-distribution perturbations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Attentio$\odot$Gradient and LRP-based attention explanations. We first propose an actionable diagnostic methodology (henceforth faithfulness violation test) to measure the consistency between explanation weights and the impact polarity. Through the extensive experiments, we then show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue, especially the raw attention. Empirical analyses on the factors affecting violation issues further provide useful observations for adopting explanation methods in attention models.

1. Introduction

The introduction argues that existing faithfulness evaluations of attention explanations overlook whether weights correctly indicate feature impact polarity. It proposes a test for that gap and reports widespread violations among tested methods.

1. Introduction

  • The paper frames explanation faithfulness as a concern in work on interpreting black-box models and attention mechanisms.Prior efforts have examined whether attention is explainable and how it behaves.Source: §1 · p. 1
  • Faithfulness requires both importance correlation and polarity consistency, but existing evaluations focus on importance and can mistake suppressive features for predictive contributions.The authors illustrate this issue with image regions that have high importance correlation yet suppress predictions.Source: §1 · pp. 1–2
  • The violation test checks whether the highest-absolute-weight feature’s explanation polarity matches its measured impact, a dimension existing metrics cannot assess.The test is applicable to explanation methods without modification.Source: §1 · p. 2
  • Most tested methods show polarity violations; raw attention performs worst, while attention ⊙gradient alleviates violations and passes in some cases.The paper also identifies model-architecture complexity and polarity-identification capability as important factors.Source: §1 · p. 2

2. Related Work

Related work covers broad families of neural-network explanations and methods that combine attention with other attributions. It distinguishes this paper’s focus on polarity consistency from prior faithfulness evaluations centered on importance correlation.

2.1. Model Explainability

  • Explanation research includes gradient-based, perturbation-based, and Shapley-value methods, while attention explanations may combine attention weights with gradients, relevance, or input-vector norms.Other attention studies examine model behavior or patterns attended to.Source: §2.1 · pp. 2–3

2.2. Faithfulness in Attention Models

  • This paper treats importance correlation and polarity consistency as separate properties, noting that existing faithfulness metrics assess the former but not the latter.Its polarity-consistency perspective differs from saliency work focused on arbitrary manipulation.Source: §2.2 · p. 3, §2.1 · p. 3
  • Prior attention-faithfulness work reached differing conclusions about whether attention is faithful, but its measurements focused on importance correlation.Other work compares attention with LIME or Shapley values and proposes domain-specific improvements.Source: §2.2 · p. 3

3. Faithfulness Violation Test

The faithfulness violation test evaluates whether an explanation correctly identifies the impact polarity of input features. It selects a feature by explanation magnitude, perturbs it, and compares the resulting confidence change with the explanation’s polarity.

3. Faithfulness Violation Test

  • The test selects the input feature with the largest absolute explanation weight as the feature whose impact polarity will be checked.An explanation method assigns a weight to each input token or feature.Source: §3 · p. 3
  • A highly positive explanation weight should correspond to confidence degradation after replacement; a confidence increase indicates a polarity violation.Source: §3 · p. 3

4. Experimental Setup

The experimental setup spans six attention-model tasks and nine datasets, with multiple model architectures, explanation methods, faithfulness metrics, and feature-replacement functions.

4.1. Tasks and Datasets

  • Experiments cover six exemplar tasks and nine datasets spanning different training-data scales, class counts, and learning targets.Source: §4.1 · pp. 3–4

4.2. Models

  • The models include LSTM and CNN attention variants for NLP tasks and four VQA models using transformer, cross-modal, or general attention.The VQA models use either grid-based ResNet-101 features or object-level Faster R-CNN features.Source: §4.2 · p. 4

4.3. Explanation Methods

  • The study compares generic attention, transformer-specific, and gradient-based explanations, including RawAtt, AttGrad, PLRP, GenAtt, InputGrad, and Integrated Gradients.Source: §4.3 · p. 4

4.4. Faithfulness Evaluation Metrics

  • Four comparison metrics are AUC on Threshold Performance, Sufficiency, Comprehensiveness, and absolute rank correlation.The metrics assess perturbation performance, retained-feature confidence, removed-feature effects, or relationships between perturbation variance and explanation magnitude.Source: §4.4 · pp. 4–5

4.5. Replacement Functions

  • To address out-of-distribution concerns, the study averages results across slice-out, attention-mask, and MASK-token replacements.Words are replaced with a MASK token, while image regions are replaced with zero vectors.Source: §4.5 · p. 5

5. Experimental Results

The experiments test whether existing faithfulness metrics detect polarity violations, how explanation methods perform on that test, and which factors are associated with violations.

5.1. Why violation test is necessary? (RQ1)

  • Existing evaluation metrics do not reliably detect polarity violations, while the violation test distinguishes explanations whose weights agree with input impact from those with opposite polarity.The authors suggest that metrics based on many input tokens can conceal polarity violations in the highest-weight feature.Source: §5.1 · p. 6

5.2. Evaluation results for explanation methods (RQ2)

  • All tested methods show nonzero violation ratios, with RawAtt and AttIN worst; gradient-based methods generally fare better, with AttGrad strongest on general attention models and GenAtt on transformers.RawAtt and AttIN can still score well on other faithfulness metrics despite severe violations, so those metrics alone may miss polarity problems.Source: §5 · p. 5, Figure 4 · p. 6, §5.2 · pp. 6–7

5.3. Factors affecting faithfulness violation issue (RQ3)

  • Violation levels rise with the number of deep layers, and gradient-sign information improves faithfulness over raw attention, underscoring polarity identification as a relevant factor.The example contrasts positive and negative sentiment phrases, which RawAtt and AttIN assign the same weight polarity.Source: §5.3 · pp. 7–8, Figure 6 · p. 8

6. Discussions and Conclusions

The paper finds polarity consistency essential to evaluating attention explanations: most tested methods show faithfulness violations, though gradient-based methods offer clues for identifying polarity. The authors also note limits in the test and in the scope of the methods studied.

6. Discussions and Conclusions

  • Polarity consistency is presented as a concrete criterion for evaluating attention-model explainability, and the violation test as essential compared with existing metrics.Source: §6 · p. 8
  • Most tested explanation methods show faithfulness violations, and architecture complexity and polarity-identification capability emerge as dominant factors.The authors caution that this does not mean the explanations are meaningless; they describe the research as not yet mature.Source: §6 · p. 8
  • Without polarity-identification capability, explanations cannot show whether an input contributes to a prediction, risking misleading interpretations of model decisions.Source: §6 · p. 8
  • Gradient-based explanations are not flawless, but the findings indicate they provide some clues and help identify impact polarity.Source: §6 · p. 9
  • Confidence perturbation can limit the violation test and depress its evaluation results to some extent.The authors report using several replacement functions they consider promising for the out-of-distribution problem in perturbation.Source: §6 · p. 9
  • The methods discussed are confined to an ideal, narrow scope intended to support focused analysis.Source: §6 · p. 9

A. Tasks and datasets

The experiments cover five task types using the listed datasets, including text classification, paired-question equivalence, entailment, contextual question answering, and image-text question answering.

A. Tasks and datasets

  • Sentiment analysis uses Stanford Sentiment Treebank and Yelp2, with models predicting one of five sentiment levels for each sentence.Source: A. Tasks and datasets · p. 12
  • Paraphrase detection uses Quora Question Paraphrase to train models to determine whether pairs of questions are semantically equivalent.Source: A. Tasks and datasets · p. 12
  • Natural language inference uses SNLI, a dataset of about 570K human-written English sentence pairs, to identify textual entailment.Source: A. Tasks and datasets · p. 12
  • Question answering uses bAbI-1, where correctly answering each question requires a supporting fact from the context.Source: A. Tasks and datasets · p. 12
  • Visual question answering uses VQA 2.0 and GQA, requiring models to understand both images and text to answer questions.The datasets include about 410K and 950K training samples, respectively.Source: A. Tasks and datasets · p. 12

B. Details of explanation methods

The paper studies attention-model explainability across three groups of explanation methods.

B. Details of explanation methods

B.1. Generic attention-based explanation methods

The paper compares generic attention-based, transformer-specific, and gradient-based attribution methods, each of which produces explanations through a different construction.

B.1. Generic attention-based explanation methods

  • Generic attention-based methods include direct attention output, attention scaled by gradients, and attention incorporating input-vector norms.RawAtt is the direct attention output; AttGrad scales attention weights with corresponding gradients, while AttIN incorporates input-vector norms.Source: §B.1 · p. 12

B.2. Transformer-based explanation methods

  • Transformer-based methods include head-averaged LRP scores, linear combinations of attention along graph paths, and LRP methods integrated with attention gradients.PLRP averages last-layer head scores; TransAtt integrates LRP scores with attention gradients, while GenAtt combines weights and gradients across layers.Source: §B.2 · pp. 12–13

B.3. Gradient-based attribution methods

  • Gradient-based comparison methods include input-times-gradient attribution and Integrated Gradients along a path from a baseline input to the original input.Source: §B.3 · p. 13

C. Details of faithfulness evaluation metrics

The section describes four metrics for evaluating explanation faithfulness: AUC-TP, Sufficiency, Comprehensiveness, and Rank Correlation. They assess performance under masking, retained-feature confidence, lower-weight feature necessity, and correlation with measured feature importance.

C. Details of faithfulness evaluation metrics

  • AUC-TP measures the area under the model-performance curve as explanation-ranked tokens are progressively masked.Tokens are masked at levels from 0% through 90%, including 5%, 10%, and 20%.Source: C. Details of faithfulness evaluation… · p. 13
  • Sufficiency measures whether explanation-identified important features retain confidence in the original prediction.The metric uses top-weighted features at sparsity levels B = {5, 10, 20, 50}.Source: C. Details of faithfulness evaluation… · p. 13
  • Comprehensiveness evaluates whether features assigned lower explanation weights are unnecessary for predictions.The metric uses sparsity levels B = {5, 10, 20, 50}, ordered by decreasing explanation weights.Source: C. Details of faithfulness evaluation… · p. 13
  • Rank Correlation measures the monotonic association between explanation weights and measured feature importance using Spearman’s rank correlation.It compares descending absolute explanation weights with the corresponding measured feature importance.Source: C. Details of faithfulness evaluation… · p. 13

D. Sanity faithfulness evaluation for explanation methods

The section presents average faithfulness evaluation scores across the paper’s datasets and model architectures. Its diagnosis figures cover multiple datasets, including VQA 2.0, GQA, QQP, SST, Yelp, SNLI, bAbI-1, AgNews, and 20News.

D. Sanity faithfulness evaluation for explanation methods

Loading 2201.12114v3…