PaperlayerView exact revision on arXiv

Source-linked AI summary

Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time

Lisa Bouger, Yannick Teglia, Philippe Loubet Moundi

arXiv:2609.05074v1cs.CL

TL;DR

Transformer classifiers used for prompt injection detection need interpretable links between internal mechanisms and classification decisions. The paper proposes an influence score combining attention-head logit direction with residual-stream contribution, then applies it to DeBERTa to characterize decision behavior. The framework distinguishes decision profiles across prediction outcomes while balancing analytical granularity and computational cost.

  • Problem

    Transformer interpretability needs methods that connect internal decision mechanisms with classifier outputs while avoiding the extremes of highly instrumented circuit analysis and purely global analysis.

  • Method

    The influence score combines each attention head’s directional influence on logits with its relative contribution to the residual representation.

  • Results

    Correct and erroneous DeBERTa predictions show distinct decision dynamics, with correct decisions more distributed and errors associated with late imbalances or dominant mechanisms.

  • Takeaways & Limitations

    The framework provides a controllable compromise between fine-grained component analysis and global model analysis for studying Transformer decision mechanisms.

  • Takeaways & Limitations

    The directional influence measure directly projects intermediate activations onto the final logit direction despite distributional shifts between intermediate and final representations.

Abstract

from arXiv · show

We propose an influence score to quantify the contribution of attention heads to classification decisions in Transformer-based models designed for prompt injection detection. The score combines directional influence on the logits with structural contribution within the residual stream, enabling a multi-scale analysis at the head, layer, and network levels. Applied to a DeBERTa model specialized for prompt injection detection, our framework reveals distinct decision behaviours between correct and erroneous predictions. Our method provides an effective compromise between fine-grained circuit analysis and global output-based methods, and offers a systematic way to study decision mechanisms in Transformer classifiers.

1 Introduction

The paper addresses the need to understand Transformer classifiers for prompt injection detection by proposing an influence score that links attention-head mechanisms to classification decisions. The framework aims to balance fine-grained interpretability with global analysis and supports automated examination of decision mechanisms.

  • Motivation: Prompt injection and jailbreak attacks motivate interpretability methods for Transformer classifiers used to detect malicious queries.Interpretability is presented as important for reliability, transparency, bias analysis, and understanding internal decision-making.
  • Interpretability gap: Existing methods trade off circuit-level granularity against scalable output-level analysis.Fine-grained circuit analyses require complex network instrumentation, while global approaches lack direct access to decision-making mechanisms.
  • Proposed approach: The proposed influence score estimates each attention head’s contribution to the final prediction by combining logit impact with internal representational importance.The score is designed to connect internal Transformer mechanisms explicitly to classification decisions.
  • Proposed approach: The approach balances analytical granularity and computational cost while retaining a global view of model behavior.It supports identification of mechanisms associated with different prediction categories.

2 Related Work

Related work spans feature-level explanations, internal-circuit analysis, layer-wise prediction tracking, and component-contribution measures. The paper positions its method as combining component-level analysis with quantitative links to final predictions.

  • Existing approaches: LIME and SHAP explain observable input features but do not identify which internal components drive a decision.These methods approximate model behavior locally around an example without accounting for internal structure.
  • Existing approaches: Circuit studies reveal specialized internal substructures but often rely on qualitative analyses or targeted case studies.This makes systematic comparisons across decision categories more difficult.
  • Existing approaches: The logit lens tracks prediction formation across layers but does not measure the relative contribution of individual components.It applies the unembedding matrix to intermediate residual representations.
  • Existing approaches: ALTI compares component contributions within and across layers but does not account for decision polarity in the logits.Its formulation uses geometric proximity to the overall residual representation.
  • Positioning: The proposed framework combines normalized component contribution with directional logit influence to analyze where decisions emerge and which heads drive them.It also supports comparisons between correct and incorrect predictions.

3 Methodology

The method is illustrated with a DeBERTa-v3-base classifier trained to detect prompt injection attacks using aggregated open-source data. Evaluation uses a filtered and re-annotated binary dataset with benign and attack prompts.

  • Model and data: The study uses a DeBERTa-v3-base classifier trained to detect prompt injection attacks.The model is based on DeBERTa’s disentangled attention mechanism and is described as a 12-layer Transformer.
  • Model and data: Training data include VMware/open-instruct, HuggingFaceH4/grok-conversation-harmless, and OpenSafetyLab/Salad-Data.The paper states that a complete description of the training data appears in Appendix A.
  • Evaluation setup: Evaluation uses a binary dataset of 30,000 examples aggregated from open-source datasets and filtered and re-annotated with an LLM-as-a-judge mechanism.The dataset contains two-thirds benign prompts and one-third attack attempts.
  • Evaluation setup: The INJECTION class is treated as positive and BENIGN as negative, with TP and TN denoting correct predictions and FP and FN denoting errors.These conventions define the prediction categories used in subsequent analyses.

4 Attention Head Influence Analysis Framework

The framework decomposes Transformer representations into layer and head contributions, measures directional logit influence and relative residual-stream importance, and combines them into an influence analysis. Its relative contribution measure is designed to remain usable with LayerNorm.

  • Residual and logit decomposition: The final residual representation is modeled across L Transformer layers, and class logits are obtained through a linear projection of that representation.The framework abstracts away normalization layers when approximating residual-stream construction.
  • Residual and logit decomposition: Each attention block is decomposed into N attention heads whose contributions can be analyzed individually.This decomposition provides the head-level basis for the influence analysis.
  • Directional influence: Raw head influence on the logit is approximated through a projection onto the relevant logit direction.The method uses this directional quantity to characterize how a head affects the class decision.
  • Relative importance: Relative importance measures how closely a component contributes to a global residual state using the ALTI formulation.The paper notes that heads do not contribute equally to residual-representation formation.
  • Relative importance: Proximity estimates component importance without requiring gradients or ablation procedures.It is defined as proximity(zi, r) = max(−∥zi −r∥1 + ∥r∥1, 0).
  • Assumption: The relative score remains suitable with LayerNorm because components are compared within the same residual state.Normalization may change activation scale or orientation, but affects within-example contributions comparably.

5 Influence score : combining contribution and direction

The influence score combines structural contribution within the residual stream with directional influence on logits to characterize attention-head effects at multiple network scales. This reveals concentrated, class-dependent contributions and competing signals whose balance shapes predictions.

  • Empirical patterns: A few attention heads dominate contributions, with concentration becoming stronger in deeper layers and varying by predicted class and correctness.Figure 2 shows shared early-layer distributions but specialized deeper-layer heads, including H6 for TN and H2 for TP; errors resemble opposite-class profiles.
  • Empirical patterns: Head projections favor opposing classes: some push toward INJECTION and others toward BENIGN, so predictions arise from competing signals.
  • Weighted influence score: The method combines each head’s directional logit influence with its structural contribution to the residual stream.A head is influential when it contributes to the residual stream, belongs to an important layer, and points toward the considered logit direction.
  • Hierarchical organization of contributions: Layer contribution measures how much a layer’s attention output modifies the residual state, while head contribution measures its relative share within that layer’s attention output.
  • Layer-wise aggregation and global score: The score can be aggregated over selected layers, enabling head-level, layer-level, and network-level analysis of how predictions form progressively.Early, intermediate, and deeper layers can be examined separately through the same aggregation.
  • Interpretation: The score’s sign indicates influence toward INJECTION or BENIGN, while its magnitude reflects influence strength and supports comparison within an example.Scores near extreme values indicate dominance by same-sign contributions, whereas values near zero indicate compensation between opposing influences.

6 Results

The influence-score analysis reveals distinct decision profiles across prediction outcomes, depth, attention heads, and TN clusters. Correct predictions show more stable or distributed evidence, whereas errors and some categories exhibit late imbalances or heterogeneous mechanisms.

  • Global score distributions: TP scores are predominantly unimodal, TN scores are multimodal, and errors show more dispersed and extreme score distributions.FP tend toward excessive INJECTION influence, while TN exhibits three modes suggesting multiple BENIGN decision regimes.
  • Evolution with depth: Across most layers, opposing contributions accumulate and compensate, but final layers produce marked changes, especially for errors.FP show a late increase in the INJECTION signal, whereas FN show a late increase in the BENIGN signal.
  • Dominant heads: Dominant influence concentrates in a small number of deeper-layer heads, while some early heads contribute moderately and frequently.The same head can favor INJECTION or BENIGN depending on the category, and early heads such as L0H0 or L0H7 may play preparatory roles.
  • Ablation validation: Ablating six dominant heads decreases performance, with the combined score producing degradation comparable to or greater than individual selection criteria.The moderate impact of ablation suggests compensatory mechanisms and distributed representation, while some TP-head ablations slightly improve TN accuracy.
  • TN cluster analysis: TN examples form three clusters whose score trajectories overlap early and progressively separate with depth.The clusters correspond to long structured prompts, general-purpose instructions, and short tasks such as translation, indicating distinct balances of pro-MALICIOUS and pro-BENIGN contributions.

7 Discussion and Conclusion

The paper frames the influence score as a practical link between attention-head mechanisms and classification decisions. Applied to DeBERTa prompt-injection detection, it finds distributed correct-decision contributions, late or concentrated error mechanisms, and decision-critical heads identified by ablation.

  • Limitation: The directional influence measure is limited by distributional shifts between intermediate-layer and final-layer representations.A layer-specific projection, such as a tuned-lens transformation, is proposed as a methodological improvement.
  • Contribution: The score combines attention heads’ residual-stream contribution with their directional influence on the logits.This explicitly links internal Transformer mechanisms to the classification decision.
  • Findings: Correct decisions rely on more distributed contributions, whereas errors are associated with late imbalances or dominance of specific mechanisms.Early layers contribute consistently but moderately, while deeper layers concentrate the decisive signal.
  • Findings: Ablation confirms that combining contribution and directional influence identifies components with the largest performance impact.The experiments support the score’s ability to identify decision-critical components.

Références

The references list prior work on tuned-lens analysis, internal circuits, Transformer models, prompting and safety, interpretability, pruning, and related language-model methods.

  • Interpretability: The cited literature includes tuned-lens and circuit-interpretability work relevant to analyzing internal Transformer mechanisms.Examples include Belrose et al. (2023), Bricken et al. (2023), and Conneau et al. (2018).
  • Related methods: Additional references address prediction explanation, logit-lens analysis, attention-head importance, pruning, and contextual-information mixing.These include LIME/SHAP-related work, the logit lens, head ablation, movement pruning, and Transformer information mixing.
  • Models and safety: The references also cover Transformer architectures, DeBERTa, large language models, prompting, and model-safety research.Cited works include Vaswani et al. (2017), He et al. (2021), Gemma Team (2024), Mistral 7B, and prompting and safety benchmarks.

A Model training data

The training dataset combines multiple public datasets and newly constructed prompt injections to cover varied prompt formulations. The source datasets use several licensing arrangements.

  • Dataset construction: The training data were assembled from several public datasets to cover a wide variety of prompt formulations.Prompt injections were additionally constructed from academic papers, specialized publications, security competitions, and LLM Guard community feedback.
  • Licensing: The source datasets span CC-BY-3.0, MIT, CC0 1.0 Universal, public-domain, and Apache License 2.0 terms.The passage lists one CC-BY-3.0 dataset, eight MIT datasets, one CC0 dataset, six without-license public-domain datasets, and five Apache-licensed datasets.

B Dataset construction

The evaluation dataset aggregates benign prompts and prompt-injection or jailbreak examples from open-source instruction, public-prompt, and attack-focused datasets. After filtering and LLM-based re-annotation, sampling controls source dominance and yields a 33,965-prompt collection spanning multiple malicious and benign use cases.

  • The dataset combines instruction datasets, public prompts, and academic or open-source collections of LLM attacks.
  • Examples are filtered and re-annotated by multiple LLM judges using a consensus labeling mechanism.A prompt is labeled MALICIOUS when at least two judges classify it as malicious, and BENIGN when at least one judge classifies it as benign.
  • 33,965 prompts are sampled with a per-dataset maximum, comprising 23,775 benign examples (70%) and 10,190 injections (30%).
  • Malicious prompts: Malicious examples include jailbreaks, DAN variants, explicit harmful instructions, roleplay attacks, and indirect circumvention through fictional or contextual framing.
  • Benign prompts: Benign prompts cover translation, factual questions, writing or rewriting, and general instructions without sensitive content.

C Descriptive statistics of the TN clusters

Table 4 presents descriptive characteristics of the three TN clusters.

  • Table 4 presents descriptive characteristics of the three TN clusters.
Loading 2609.05074v1…