Source-linked AI summary
Unraveling the Real Working Mechanism and Inherent Flaws of GAE: A Method for Interpreting Transformer Processes from an Economic Perspective
Yongjin Cui, Xiaohui Fan
TL;DR
The paper addresses the gap between metric-focused XAI research and the need to understand whether interpretation methods themselves are rational and interpretable. It analyzes GAE’s mechanism and flaws, then proposes CAH, which combines process-based and feature-based ideas for Transformer interpretation. The analysis identifies GAE’s attention-focused operation and motivates re-examination of related methods and applications.
Problem
XAI evaluation often relies on flawed proxy metrics, while the mechanisms and interpretability of XAI methods themselves remain insufficiently studied.
Method
The paper uses model simplification and analysis of additive operations to investigate GAE, then proposes CAH from an economic zero-sum-games perspective.
Results
GAE is essentially a first-order accumulation of layer-wise attention contributions, while higher-order terms are usually negligible but may interfere in special models; its errors also arise from linear approximation of nonlinear processes.
Takeaways & Limitations
GAE-related applications and subsequent methods using similar pipelines may need re-examination or verification, while CAH addresses attribution of patch tokens embedded within special tokens.
Abstract
from arXiv · showhide
We observe a phenomenon that current algorithmic research in the field of explainable artificial intelligence primarily pursues better performance on several proxy metrics. On the one hand, these proxy metrics themselves are more or less flawed and cannot properly measure the quality of methods. On the other hand, metric-oriented research approaches often lead to the neglect of the rationality and interpretability of the methods themselves. Explainable artificial intelligence is abbreviated as XAI. The metric-driven research paradigm has resulted in a lack of interpretability of the relevant XAI methods themselves. Accordingly, there is a need for interpretability research on XAI methods, which can be playfully referred to as XXAI. This paper is one of our works on XXAI. This paper takes Generic Attention-model Explainability (GAE), a widely influential model interpretation method , or rather, XAI method that represents an important technical route, as the research object, and explores the real working mechanism and flaws of this method as well as the technical route it represents. Based on the conclusions of this study, it may be necessary to re-examine or verify GAE-related methods and their domain applications. We argue that GAE is an interpretation method that focuses on the attention process. After pointing out the working mechanism and flaws of GAE, we propose Cumulative Asset Holdings (CAH), a more reasonable Transformer interpretation method integrating both process-based and feature-based ideas from an economic zero-sum games perspective. In addition, it is worth noting that our method is applicable to models with special tokens, where existing methods may suffer from limitations. The model simplification research method and the analysis of additive operations adopted in this study may provide inspiration for other research works in XAI.
I. INTRODUCTION
The paper argues that XAI research often optimizes flawed proxy metrics while neglecting the interpretability of interpretation methods themselves. It investigates GAE’s mechanism and limitations, then proposes CAH as an integrated alternative for Transformer interpretation.
- Transformer’s broad adoption has increased the need to interpret its internal mechanisms, especially amid safety and fairness concerns in high-stakes applications.
- Existing interpretation evaluation uses qualitative consistency or quantitative comparisons and perturbations, but these approaches rely on flawed assumptions about human cognition, model quality, or feature independence.
- Metric-oriented XAI research can prioritize proxy performance over the rationality and interpretability of the methods themselves, motivating interpretability research on XAI methods, termed XXAI.
- GAE weights attention maps with positive gradients and propagates them across layers through attention rollout, although its actual calculation is equivalent to the introduced correlation-propagation pipeline.
- The paper investigates GAE’s real mechanism and flaws because the method has been widely accepted, applied across fields, and influential despite incomplete mechanistic study.
- GAE uses linear approximations of nonlinear operations, introducing approximation errors and failing Sensitivity; Integrated Gradients is more accurate but cannot directly handle attention maps or fully attribute outputs in models with special tokens.
II. RELATED WORK
GAE established a widely reused Transformer-interpretation paradigm that weights attention maps and then propagates them across layers. Subsequent methods apply, extend, or reimplement this two-stage design across domains and attribution frameworks.
- 1) GAE and Its Applications:: GAE computes gradient-weighted attention by averaging attention heads, discarding negative contributions, and propagating relevance through layers with attention rollout.
- 1) GAE and Its Applications:: GAE has been applied beyond vision-language tasks, including medical prediction, speech analysis, audio deepfake detection, and agricultural multimodal modeling.
- 2) GAE as a Foundation for Subsequent Method Development:: GAE has influenced later methods that directly incorporate its gradient-weighted formula or adopt its broader weight-then-rollout design with alternative weighting mechanisms.
- 2) GAE as a Foundation for Subsequent Method Development:: GAE established a two-stage paradigm: weight attention maps using gradients or other relevance signals, then propagate the weighted maps across layers via attention rollout.
B. Integrated Gradients (IG)
Integrated Gradients (IG) attributes input changes along the path from a baseline to the current input, rather than using only the final gradient. The paper notes its axiomatic advantages, efficient approximation, and reliance on an improved baseline scheme.
- IG addresses gradient methods’ failure to satisfy Sensitivity while satisfying Implementation Invariance.
- IG tracks each step of input-contribution change from a baseline to the current input through a straight-line path.
- The IG integral can be efficiently approximated using a Riemann-sum calculation.
- This paper uses Cui et al.’s improved IG, which determines the entire model’s baseline from the input end.
A. Research Rationale
The paper analyzes GAE through model simplification, treating attention rollout as a reduced model and GAE as attention rollout augmented by positive output gradients.
- Attention rollout retains only attention and represents residual connections with identity matrices, while ignoring other model processes.
- GAE weights attention maps by their positive gradients with respect to the explained output before performing attention rollout.
B. Model Simplification and Initialization
The simplified ViT contains inputs, three single-head attention layers, residual connections, a head, and one output. Its residual connections use identity matrices, and the output is obtained from the third-layer cls token.
- The simplification models ViT with inputs, three single-head attention layers, residual connections, a head, and one output.
- The input matrix X is in R3×2, with its first row designated as the cls token.
- The three single-head attention maps A1, A2, and A3 are each in R3×3.
- Residual connections are represented by I3, a 3 × 3 identity matrix.
- The model output sums the elements in the cls token of the third-layer attention output.
C. Attention-Map-Based Interpretation Process
The analysis reconstructs GAE as gradient-weighting residual-fused attention maps before applying attention rollout, then explains why first-order accumulation usually dominates while higher-order terms can become dimensionally meaningless or interfere in special cases.
- GAE procedure: GAE computes gradient-weighted residual-fused attention maps and then applies attention rollout across layers.The procedure uses gradients as weights before recursive matrix operations over attention maps.
- GAE procedure: Gradient weighting gives attention values contribution-based meanings, so repeated multiplication combines quantities that are no longer ordinary attention weights.The paper argues that this undermines the interpretation of subsequent matrix products.
- Dimensional analysis: For output quantities measured in meters, first-, second-, and third-order terms correspond respectively to length, area, and volume, making their addition physically meaningless.The example illustrates why mixed-order matrix products are problematic after gradient weighting.
- Empirical verification: Because weighted attention values are generally below 1, higher-order products shrink, leaving accumulated first-order terms as the dominant contribution to GAE’s result.The authors call this accumulation the cumulative contribution of first-order terms.
- Empirical verification: Although A3+A2+A1 dominates in the initialized example, specially chosen values can make higher-order terms dominate instead.The alternative case is described as A3A2A1 + A3A2 + A3A1 + A2A1 dominating rather than A3+A2+A1.
- Empirical verification: In ViT base experiments, the weighted-average attention mean was 0.00002120 with standard deviation 0.00027230, making higher-order terms almost negligible.The experiment analyzed 2,328,540,000 gradient-weighted average attention values.
- Conclusion: Across the reported analysis, GAE is characterized as first-order contribution accumulation, but its higher-order terms and linear approximations warrant re-examination in special models.The perturbation results were used to verify closeness to first-order accumulation, not to compare evaluation performance, and the authors do not fully endorse that evaluation method.
IV. A METHOD FOR INTERPRETING TRANSFORMER PROCESSES FROM AN ECONOMIC PERSPECTIVE
The paper motivates integrating process-based and feature-based interpretation because IG cannot directly explain attention maps or fully attribute outputs in Transformers with special tokens. It therefore proposes interpreting Transformer processes as economic zero-sum games.
- Motivation: GAE is gradient-based and process-based, but directly improving it with IG is hindered because attention maps depend on Transformer Value features.This dependence prevents applying IG to attention maps as though they were independent inputs.
- Motivation: In Transformers with special tokens, IG assigns contribution to special tokens, making complete attribution to individual patch tokens difficult.Non-initial cls tokens contain contributions from other patch tokens, limiting single-layer feature attribution.
- Contribution: The proposed approach integrates process-based and feature-based interpretation through a zero-sum-game perspective.The paper presents this integration as the response to the two limitations above.
A. Method
CAH models attention propagation as a zero-sum game in which patches redistribute fixed total assets across successive meetings. It uses IG to measure holdings at each state and accumulates them across the process.
- Economic model: The method uses ViT as an example for an economic description of Transformer computation.The economic analogy is developed around the ViT attention process.
- Economic model: A patch clan contains n individuals with fixed total assets, while an outsider cls token enters with assets set to 0.The individuals represent patch tokens and the fixed total assets represent output.
- Economic model: Across m meetings, all n + 1 members redistribute assets while total assets remain output, after which cls absorbs the savings and debts.The original patch members’ capital eventually becomes 0, apart from the stated simplifying treatment of initial cls attribution.
- CAH computation: CAH represents each patch’s contribution by accumulating its IG-measured asset holdings across the initial state and every redistribution meeting.The final contribution is obtained by summing holdings throughout the zero-sum-game process.
- CAH computation: CAH uses input features because the final attention block attributes output contributions to cls while patch-token contributions are zero.The formulation defines IG(l)_i as the IG score of patch i in the input features of attention block l.
- Interpretive rationale: Unlike GAE’s single-step gradient approximation, CAH captures every state along the path from initial input to final attention propagation.CAH inherits IG’s process-wide characterization rather than focusing on one state or only first-order terms.
- Evaluation: CAH’s feature-region consistency is described as an intuition about model performance, while its sole interpretation metric is mean absolute IG attribution error.The paper distinguishes this performance-related intuition from CAH’s formal evaluation metric.
B. Experiments
The experiments use IG and CAH to expose limitations of IG with special tokens and to illustrate CAH across the attention process. They emphasize attribution error rather than traditional feature-region consistency as the relevant basis for trust.
- Experimental scope: CAH aggregates IG outputs, while the experiments use models with cls tokens to illustrate IG’s attribution limitations.The experiments are not designed to prove CAH’s generality or effectiveness.
- IG limitation: In models with special tokens, IG assigns only part of the contribution to patch tokens and cannot explain patch contributions embedded within special tokens.This limitation is shown in Figure 5.
- CAH interpretation: CAH measures patch-token contributions throughout the entire attention process, with error expressed as mean absolute IG attribution error per layer.Figure 6 presents CAH interpretations using this process-wide attribution view.
- Evaluation principle: Interpretation quality should be judged by attribution-error magnitude rather than whether feature regions are perfectly highlighted.The paper states that traditional consistency metrics assume both a perfect model and optimal human cognition.
V. CONCLUSION
The paper frames responsible XAI as requiring interpretable interpretation methods and accurate error measurement. It concludes that GAE’s weight-then-rollout paradigm should be reexamined and presents CAH as a process-based alternative that combines IG with feature attribution for models containing special tokens.
- Conclusion: Responsible interpretation methods should explain their own error sources and evaluate errors with accurate, reasonable methods.The paper presents this requirement as part of responsible XAI and XXAI.
- Conclusion: The paper argues that XAI should move beyond metric-oriented research because existing interpretation metrics are unreliable proxies.It contrasts this with model training, where metrics are presented as more reliable.
- Conclusion: Model simplification reveals GAE’s working mechanism and flaws, motivating reexamination of GAE-related applications and methods using similar pipelines.The paper characterizes GAE as focusing on the attention process.
- Conclusion: CAH combines process-based interpretation with IG to improve precision over gradient-based methods and address incomplete patch attribution with special tokens.The method is described as a process-based Transformer interpretation method from an economic zero-sum-games perspective.