Source-linked AI summary
Order in the Court: Explainable AI Methods Prone to Disagreement
Michael Neely, Stefan F. Schouten, Maurits J. R. Bleeker, Ana Lucic
TL;DR
The paper asks whether rank correlation is an appropriate way to evaluate feature-additive and attention-based explanations. It compares multiple explanation methods across two architectures and task types, finding generally low agreement and concluding that rigorous diagnostic tests are preferable to agreement-based evaluation.
Problem
It is difficult to determine when additive XAI methods fail because plausibility and faithfulness require measurable diagnostics, while explanations are task-, model-, and context-specific.
Method
The study compares LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, Deep-SHAP, and attention-based explanations using Kendall-τ agreement across architectures and language tasks.
Results
Low agreement is observed among XAI methods and with attention-based explanations, with mean correlations of 0.2684 and 0.1736, respectively, across models and tasks.
Takeaways & Limitations
Without external ground-truth explanations, rank correlation measures only similarity between rankings, so practitioners should use theoretically motivated diagnostic measures instead.
Takeaways & Limitations
The paper notes that complex tasks may admit many equally faithful or plausible rankings, especially when humans cannot reliably judge token-level importance.
Abstract
from arXiv · showhide
By computing the rank correlation between attention weights and feature-additive explanation methods, previous analyses either invalidate or support the role of attention-based explanations as a faithful and plausible measure of salience. To investigate whether this approach is appropriate, we compare LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, Deep-SHAP, and attention-based explanations, applied to two neural architectures trained on single- and pair-sequence language tasks. In most cases, we find that none of our chosen methods agree. Based on our empirical observations and theoretical objections, we conclude that rank correlation does not measure the quality of feature-additive methods. Practitioners should instead use the numerous and rigorous diagnostic methods proposed by the community.
1. Introduction
The paper questions whether agreement between explanation methods can evaluate faithfulness, since explanations are task-, model-, and context-specific. Across architectures and tasks, it finds low agreement and argues that rank correlation is unsuitable without ground-truth rankings.
- Only explanations that are both plausible to stakeholders and faithful to model reasoning are desirable, especially in critical domains.
- Agreement-based evaluation assumes that one XAI method can serve as a standard, but explanations are task-, model-, and context-specific.
- The study asks whether LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, and Deep-SHAP correlate with one another and with attention-based explanations.
- Low overall agreement occurs among XAI methods and attention-based explanations, especially for Transformer models and pair-sequence tasks.
- Rank correlation is not objective evaluation unless ground-truth rankings are available; otherwise, rigorous diagnostic measures are preferable.
2. Related Work
Prior work established agreement as a way to compare attention with other explanations, while also documenting important limitations of XAI methods. The paper situates its study among critiques of these methods and diagnostic alternatives.
- Jain & Wallace (2019) introduced agreement-based comparison of attention explanations with input × gradient and leave-one-out rankings.
- Subsequent work proposed attention modifications including guided training, sparsity, word-level objectives, and transformations of multi-head attention.
- 2.3. Limitations of XAI methods: Prior studies show that LIME and SHAP can select zero-contribution tokens, saliency methods may lack input invariance, and gradient methods can resemble random rankings under remove-and-retrain evaluation.
- 2.3. Limitations of XAI methods: Ground-truth rankings support evaluation in some settings, while Atanasova et al. (2020) propose diagnostic tests for text-classification explanations.
3. Method
The method represents token explanations as importance-score vectors and compares their ranked scores using Kendall-τ. It evaluates feature-additive methods alongside attention-based attributions, with attention rollout for Transformer self-attention.
- 3. Method: Explanations are vectors of token importance scores, and agreement is measured by Kendall-τ correlation between ranked token scores.
- 3. Method: The study compares LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, and Deep-SHAP, with the latter two based on Integrated Gradients and DeepLIFT.
- 3.2. Attention-based explanations: Attention-based explanations assign attention weights to input tokens, requiring filtering or aggregation when architecture-dependent dimensionality makes this necessary.
- 3.2. Attention-based explanations: For Transformer models, attention rollout calculates token-level attribution by linearly combining token identities through self-attention layers based on attention weights.
- 3.2. Attention-based explanations: For LSTM models, the analysis uses raw attention weights because attention over hidden representations does not provide faithful token-level attribution.
4. Experiments
The experiments cover single- and pair-sequence classification using BiLSTM and DistilBERT architectures. They reuse established model settings, train multiple seeds, and apply several attribution methods to sampled test instances.
- The study evaluates sentiment classification on SST-2 and IMDb, plus pair-sequence natural-language inference and understanding tasks using SNLI, MultiNLI, and Quora Question data.
- The BiLSTM uses additive attention and, for pair-sequence tasks, predicts from both context vectors, their absolute difference, and their element-wise product.
- The Transformer experiments fine-tune DistilBERT with a linear classification layer and concatenate pair-sequences using a [SEP] token.
- Three independently seeded instances of each model are trained for up to 40 epochs with early stopping after five patience epochs.
- The attribution analysis uses existing implementations, padding-token baselines where applicable, 1000 LIME samples, and 500 random test instances per test set.
5. Results
Across models and tasks, the XAI methods show limited agreement with one another and with attention-based explanations. Agreement varies by architecture and sequence-task structure.
- Mean Kendall-τ agreement was 0.2684 among XAI methods and 0.1736 between XAI methods and attention-based explanations across models and tasks.Comparisons with SHAP approximations were excluded from average agreement because algorithmic similarity biases them.
- Non-attention XAI agreement was lower for DistilBERT than BiLSTM, with means of 0.1088 and 0.4281, respectively.
- Total agreement was higher for single-sequence than pair-sequence datasets, with combined model means of 0.273 and 0.1883.For BiLSTM, the corresponding means were 0.4219 and 0.2308.
6. Discussion & Conclusion
The discussion argues that low agreement does not establish that most XAI methods are wrong, because multiple faithful and plausible rankings may exist. It concludes that rank agreement is unsuitable as a general evaluation measure without external ground truth.
- Multiple faithful rankings can arise because input rankings capture only part of model behavior and humans may struggle to order token importance, especially for pair-sequence tasks.Under these conditions, agreement can make XAI methods appear deceptively problematic.
- Low agreement among methods for complex models and tasks is interpreted as evidence against agreement-as-evaluation assumptions, not as proof that most methods fail.
- Without external ground-truth explanations, rank correlation only indicates whether two rankings are similar and should not evaluate XAI quality.The paper recommends robust, theoretically motivated diagnostics such as those proposed by Atanasova et al. (2020).
- Agreement remains informative for examining theoretical properties, but low agreement between Grad-SHAP, Deep-SHAP, and attention-flow explanations does not show that the methods are wrong or interchangeable.The combined Grad-SHAP/Deep-SHAP mean was 0.2839, while attention flow versus selected SHAP approximations averaged 0.1726.
B. Reproducibility Checklist
The reproducibility checklist records the paper’s code, experimental environment, evaluation metrics, and reporting practices. It also points readers to the methods, experiments, and supplementary runtime information.
- The paper states that its methods, models, datasets, and experiments are described in Sections 3 and 4.
- Source code is publicly available at github.com/sfschouten/court-of-xai, with dependencies supplied through the project resources.
- Experiments used Amazon Web Services g4dn.xlarge instances with an NVIDIA T4 GPU and PyTorch 1.6.0+cu101.
- Table 3 reports validation performance for the best model weights on each dataset.
- Models were evaluated by accuracy, while agreement between XAI methods was measured with Kendall’s-τ.The checklist refers readers to Section 3 for metric explanations and to Table 2 for average training times.
B.3. For all datasets used
This checklist subsection identifies where dataset split information is documented and where dataset links can be found.
- Table 4 lists the number of instances in each split of every dataset.
- Dataset split details are described in Section 4.1, with links to each dataset provided below.
B.3.3. EXPLANATION OF ANY DATA THAT WERE EXCLUDED, AND ALL PRE-PROCESSING STEPS
The section points to preprocessing details and dataset download resources, while accompanying tables report training time and validation accuracy.
- Preprocessing and data-exclusion details are provided in Section 4.1.
- Dataset download links are provided for SST-2, IMDb, SNLI, MNLI, and XNLI; the Quora Question Pair dataset is slated for release upon publication.
- Table 2 reports training time across models and datasets, while Table 3 reports validation accuracy for selected model epochs.
B.3.5. FOR NEW DATA COLLECTED, A COMPLETE
No new data were collected for this paper; the dataset split counts are documented before length-based exclusions.
- The paper did not collect new data.
- Table 4 reports the number of instances in each dataset split before exclusions based on sequence length.
- Because MultiNLI has no publicly available test set, the study uses the English subset of XNLI.