Source-linked AI summary
On the Faithfulness of Vision Transformer Explanations
Junyi Wu, Weitai Kang, Hao Tang, Yuan Hong, Yan Yan
TL;DR
Post-hoc Vision Transformer explanations produce salience maps, but whether those scores reflect pixels’ actual influence on predictions remains underexplored. The paper introduces SaCo, which compares pixel groups pairwise and weighs their influence ordering by salience differences. Experiments report that SaCo distinguishes meaningful explanations from Random Attribution, unlike existing metrics, and that gradient information and multi-layer aggregation can improve attention-based explanations’ faithfulness.
Problem
Whether Vision Transformer salience scores reflect the corresponding pixels’ influence on predictions remains underexplored, despite the importance of evaluating explanation faithfulness.
Source: §1 · p. 1Method
SaCo compares the predictive influence of pixel groups with different salience scores and aggregates their salience differences according to whether the influence ordering matches expectations.
Source: §1 · p. 2Results
Across datasets and Vision Transformer models, SaCo consistently scores Random Attribution near zero while most explanation methods score positively, whereas existing metrics evaluate Random Attribution inconsistently.
Source: §1 · p. 2, §5.2 · p. 7Takeaways & Limitations
The paper concludes that gradient information and multi-layer aggregation can enhance the faithfulness of attention-based explanations.
Source: §6 · p. 8
Abstract
from arXiv · showhide
To interpret Vision Transformers, post-hoc explanations assign salience scores to input pixels, providing human-understandable heatmaps. However, whether these interpretations reflect true rationales behind the model's output is still underexplored. To address this gap, we study the faithfulness criterion of explanations: the assigned salience scores should represent the influence of the corresponding input pixels on the model's predictions. To evaluate faithfulness, we introduce Salience-guided Faithfulness Coefficient (SaCo), a novel evaluation metric leveraging essential information of salience distribution. Specifically, we conduct pair-wise comparisons among distinct pixel groups and then aggregate the differences in their salience scores, resulting in a coefficient that indicates the explanation's degree of faithfulness. Our explorations reveal that current metrics struggle to differentiate between advanced explanation methods and Random Attribution, thereby failing to capture the faithfulness property. In contrast, our proposed SaCo offers a reliable faithfulness measurement, establishing a robust metric for interpretations. Furthermore, our SaCo demonstrates that the use of gradient and multi-layer aggregation can markedly enhance the faithfulness of attention-based explanation, shedding light on potential paths for advancing Vision Transformer explainability.
1. Introduction
The paper argues that post-hoc Vision Transformer explanations need to be tested for faithfulness: whether pixel salience reflects the pixels’ influence on predictions. It introduces SaCo, which compares pixel groups individually and uses their salience differences to evaluate that relationship.
1. Introduction
- Existing cumulative-perturbation metrics conflate the effects of pixel groups and ignore salience magnitudes, limiting their ability to test whether salience reflects predictive influence.The paper notes that some such metrics fail to distinguish state-of-the-art explanations from Random Attribution.Source: §1 · pp. 1–2
- SaCo compares the effects of pixel groups with different salience scores, accumulating their salience differences positively when effects align with expectations and negatively when they do not.The salience difference represents the expected disparity in pixel-group influence.Source: §1 · p. 2
- Experiments across datasets and Vision Transformer models show that current metrics do not properly evaluate faithfulness, while SaCo distinguishes meaningful explanations from Random Attribution.The contribution summary describes an examination of ten explanation methods across three datasets and three models.Source: §1 · p. 2
- The paper reports that gradient information and aggregation rules, including multi-layer aggregation, can improve the faithfulness of attention-based explanations.These observations are presented as guidance for future Vision Transformer interpretability methods.Source: §1 · p. 2
2. Related Work
Related work covers post-hoc explanations, including approaches adapted from CNNs and MLPs and methods developed around Transformer attention. Prior faithfulness evaluations use perturbations but, the paper argues, often omit salience magnitudes and individual comparisons among pixel groups.
2.1. Post-hoc Explanations
- Post-hoc explanations include gradient-based, attribution-based, saliency, Shapley, and perturbation methods, alongside Transformer-oriented approaches such as Raw Attention, Rollout, Transformer-MM, and ATTCAT.Transformer-oriented methods use attention maps to estimate token importance, then map salience scores onto image space.Source: §2.1 · pp. 2–3
2.2. Evaluation of Explanation Faithfulness
- Prior faithfulness evaluations include human-centered assessments, sanity checks, and perturbation metrics, but the paper argues perturbation methods use cumulative removal and omit salience magnitudes and individual pixel-group contrasts.The related-work discussion also notes that Sensitivity-n relies on a linearity assumption for simple DNNs and neglects salience order.Source: §2.2 · p. 3
3. Methodology
SaCo tests whether differences in salience correspond to differences in pixel groups’ effects on model confidence. It partitions pixels by salience, measures the effect of perturbing each group, then rewards or penalizes pairwise comparisons according to the expected ordering and salience gap.
3. Methodology
- SaCo reorders an image’s pixels by estimated salience and divides them into K equally sized groups ranked from highest to lowest salience.Each group contains pixels from a successive interval of the ranked image pixels.Source: §3 · p. 3
- A group’s salience is the sum of its pixels’ salience scores, and its influence is approximated by replacing that group’s pixels with the per-sample mean and observing the model-confidence change.The replacement produces a perturbed image used to measure the effect on the prediction.Source: §3 · pp. 3–4
- When one group has higher salience than another, SaCo expects it to have greater predictive influence, and a larger salience gap increases the strength of that expectation.Violations of the expected relationship are penalized more when the salience difference is larger.Source: §3 · p. 4
- SaCo compares all group pairs, adding their salience difference when influence ordering matches expectations and subtracting it when ordering is violated.The resulting coefficient ranges from -1 to 1; its sign indicates correlation direction and its absolute value measures correlation degree.Source: §3 · p. 4
4. Experimental Setup
The experiments use three image datasets, three Vision Transformer models, ten explanation methods, and SaCo alongside established perturbation metrics. The setup spans explanation categories and evaluates methods under different pixel-removal procedures.
4.1. Datasets and Models
- Experiments use CIFAR-10, CIFAR-100, and ImageNet with ViT-B, ViT-L, and DeiT-B models.The models process non-overlapping 16 × 16 image patches and add a [CLS] token for classification.Source: §4.1 · p. 4
4.2. Explanation Methods
- Ten explanation methods span gradient-based, attribution-based, and attention-based approaches, including Integrated Gradients, Grad-CAM, LRP variants, Raw Attention, Rollout, Transformer-MM, and ATTCAT.The selected methods are described as recognized in explainability research and compatible with the Vision Transformer models used.Source: §4.2 · pp. 4–5
4.3. Evaluation Metrics
- SaCo is compared with AUC, AOPC, LOdds, and Comprehensiveness, which use cumulative pixel removal; their preferred directions differ across metrics.AUC and LOdds favor lower values, AOPC favors higher values, and Comprehensiveness favors lower values.Source: §4.3 · p. 5
5. Experimental Results
Across five experiments, SaCo differs from existing metrics, distinguishes Random Attribution from meaningful explanations, and shows how gradients and multi-layer aggregation improve attention-based faithfulness scores.
5.1. Interrelationships among Evaluation Metrics
- SaCo’s correlations with existing metrics are 0.18–0.22, versus 0.4764 average intercorrelation among existing metrics, suggesting SaCo measures a complementary aspect.The correlation analysis compares rankings across samples, averaged over datasets, explanation methods, and Vision Transformer models.Source: Figure 2 · p. 5, §5.1 · p. 5
5.2. Evaluating Random Attribution
- In the linnet case study, SaCo tracks subset-influence alignment: Transformer Attribution scores high, Raw Attention lower, and Random Attribution near zero, while AOPC remains high across all three.The authors attribute AOPC’s similar scores to cumulative perturbation, which entangles individual pixel effects.Source: Figure 3 · p. 6, §5.2 · p. 6
- Across CIFAR-10, CIFAR-100, and ImageNet, SaCo scores Random Attribution near zero while most methods score positively; existing metrics sometimes rank noise comparably to or above advanced methods.Figure 4 averages results over three Vision Transformers per dataset; its axes are rescaled so greater distance from the center indicates better performance.Source: Figure 4 · p. 7, §5.2 · p. 7
5.3. Assessment of Current Explanation Methods
- Current explanation methods score moderately overall, while attention-based methods generally do better; Raw Attention underperforms, motivating gradient and cross-layer aggregation.The proposed need for auxiliary information and aggregation is presented as a hypothesis for further testing.Source: §5.3 · p. 7
5.4. Effects of Designs in Explanation Methods
- 106% (last-layer attention) and 86% (all-layer aggregation) are the respective faithfulness-score increases from adding gradients; aggregation also helps with or without gradients.The reported ablation finds gradient information more influential than multi-layer aggregation.Source: Table 1 · p. 7, §5.4 · p. 8
5.5. Exploring Influential Factors in SaCo
- As K increases, most SaCo scores decline slightly; I.G. and ATTCAT are comparatively stable, while SaCo’s score-difference weighting preserves scale invariance unlike ratios.The authors note that the optimal K balances computational demands against evaluation granularity.Source: Table 2 · p. 8, §5.5 · p. 8
6. Conclusion
The conclusion presents SaCo as a faithfulness evaluation for Vision Transformer explanations. It reports that SaCo distinguishes Random Attribution, avoids removal-order dependence, and finds attention-based methods generally more faithful, with further gains from gradient information and multi-layer aggregation.
6. Conclusion
- SaCo identifies Random Attribution as lacking significant information and produces consistent results without removal-order dependency.The authors describe existing metrics as capturing overlapping aspects while lacking consideration of faithfulness.Source: §6 · p. 8
- Attention-based methods are generally more faithful, and gradient information and multi-layer aggregation can further enhance their performance.Source: §6 · p. 8
- The authors present SaCo as a comprehensive evaluation of faithfulness in Vision Transformer explanations.Source: §6 · p. 8