PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks

Rongyuan Tan, Jue Zhang, Zhuozhao Li, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

arXiv:2604.17761v1cs.AIcs.CL

TL;DR

Realistic LLM failures are difficult to analyze because prior interpretability work often uses short or toy inputs, while behavioral analysis does not reveal internal causes. This paper applies contrastive LRP-based attribution to benchmark failures using scalable cross-layer graphs, finding informative signals in some cases but not universally.

  • Problem

    Prior interpretability analyses largely focus on toy settings or short inputs, leaving realistic benchmark failures and their internal causes underexplored.

  • Method

    The paper attributes the logit difference between an incorrect token and a correct alternative to input tokens and internal states using an efficient LRP-based cross-layer graph method.

  • Results

    Across benchmarks, model sizes, and training checkpoints, contrastive attribution yields informative failure signals in some cases but is not universally effective.

  • Takeaways & Limitations

    Contrastive attribution can reveal meaningful failure patterns and interpretable shifts accompanying scaling and continued training, while some failures remain unexplained.

  • Takeaways & Limitations

    The study focuses on coarse-grained state-level analysis and leaves systematic neuron-level attribution graph analysis to future work.

Abstract

from arXiv · show

Interpretability tools are increasingly used to analyze failures of Large Language Models (LLMs), yet prior work largely focuses on short prompts or toy settings, leaving their behavior on commonly used benchmarks underexplored. To address this gap, we study contrastive, LRP-based attribution as a practical tool for analyzing LLM failures in realistic settings. We formulate failure analysis as \textit{contrastive attribution}, attributing the logit difference between an incorrect output token and a correct alternative to input tokens and internal model states, and introduce an efficient extension that enables construction of cross-layer attribution graphs for long-context inputs. Using this framework, we conduct a systematic empirical study across benchmarks, comparing attribution patterns across datasets, model sizes, and training checkpoints. Our results show that this token-level contrastive attribution can yield informative signals in some failure cases, but is not universally applicable, highlighting both its utility and its limitations for realistic LLM failure analysis. Our code is available at: https://aka.ms/Debug-XAI.

1 Introduction

Realistic LLM failures remain difficult to debug because behavioral analysis often identifies what fails without revealing why, while interpretability studies rarely address long-context benchmark settings. This paper frames failure analysis as contrastive attribution and evaluates an efficient LRP-based approach across benchmarks, model sizes, and training checkpoints.

  • Motivation: Behavioral analyses can identify failures and sensitivity to prompts, but similar behaviors may arise from different internal causes.Possible causes include underweighting relevant context, overweighting irrelevant tokens, or biases in internal representations.
  • Motivation: Interpretability analyses of realistic benchmark and deployment failures remain scarce because failures are open-ended and inputs may span thousands of tokens.These long contexts can exceed the computational limits of existing interpretability tools.
  • Research questions: The paper asks whether interpretability can provide practical value for failure cases involving incorrect tokens, model scaling, and training checkpoints.The questions concern revealing decision-process clues, comparing stronger and weaker models, and tracking how models learn fewer mistakes.
  • Contributions: Failure analysis is formulated as token-level contrastive attribution, explaining why a model prefers an incorrect token over a correct alternative.The method attributes their logit difference to input tokens and internal model states.
  • Findings: Token-level contrastive attribution yields informative signals for some failures but is not universally effective.The result clarifies both the promise and limitations of interpretability for realistic LLM failure analysis.
  • Contributions: An efficient LRP-based extension enables scalable cross-layer hidden-state attribution graphs for long-context inputs.The study then compares attribution patterns across datasets, model sizes, and training checkpoints.

2 Related Work

Prior LLM-failure research often studies outputs, specialized phenomena, reasoning tokens, or agent-level behavior, while interpretability methods have commonly used simplified inputs. This work selects AttnLRP-based attribution graphs to better accommodate realistic long-context failure analysis.

  • LLM failure analysis: Existing LLM-failure studies cover weaknesses in language understanding, reasoning, robustness, bias, hallucination, reasoning, and agent systems.These lines of work differ in whether they analyze benchmark behavior, reasoning processes, or complete agent systems.
  • Interpretability gap: Behavioral and black-box analyses characterize observed failures but generally do not examine the internal mechanisms producing them.This leaves ambiguity about whether failures reflect context weighting, irrelevant-token emphasis, or internal representational biases.
  • Interpretability settings: Many interpretability analyses rely on short prompts, synthetic data, or highly structured formats that abstract away realistic open-ended benchmark complexity.This limitation appears in hallucination-focused and broader transformer interpretability work.
  • Method selection: AttnLRP is adopted because prior work found it achieved state-of-the-art faithfulness for transformer input-token attribution.The paper also builds its efficient extension on a backpropagation-based attribution method.
  • Alternative methods: Causal-patching approaches identify components using paired clean and corrupted prompts, but constructing such prompt pairs is difficult for real-world failures.Sparse-autoencoder methods provide fine-grained structure but are computationally intensive, prompt-length limited, expertise-heavy, and model-specific.
  • Method selection: LRP- and attention-based attribution graphs are comparatively computationally efficient while maintaining reasonable attribution faithfulness.The paper therefore develops attribution graphs based on AttnLRP for long-context failure analysis.

3 Methodology

The methodology attributes the contrastive logit difference between an incorrect target and a correct alternative through transformer representations using AttnLRP. To scale analysis to long contexts, it combines batched relevance propagation with coarse hidden-state graphs and optional fine-grained refinement.

  • Model: The model maps token embeddings through L transformer blocks, then uses an unembedding matrix to produce next-token vocabulary logits.The setup concerns transformer-based autoregressive LLMs with vocabulary size |V| and hidden dimension d.
  • Contrastive attribution: Contrastive attribution explains the logit difference between an incorrect target token and a contrast alternative by attributing it across input embeddings and hidden states.This removes shared, failure-irrelevant computation and is intended to produce more salient explanations for practical debugging.
  • LRP attribution: LRP decomposes a scalar target into additive relevance contributions and propagates the contrastive target backward through transformer layers to the embedding layer.The resulting relevances define an attribution graph whose nodes are hidden-state relevances and whose edges represent propagated relevance.
  • Efficient graph construction: Naive attribution-graph construction would require separate backward passes for the many hidden-state relevance targets.The method instead packs multiple targets into one batch and uses GPU vectorization to recover cross-layer relevance propagation efficiently.
  • Relevance granularity: The analysis aggregates relevance over hidden-state dimensions to study input-token attribution patterns, including heatmap visualizations.The formulation also supports neuron-level relevance, although the primary analysis uses aggregated state-level relevance.
  • Coarse-to-fine analysis: A coarse-to-fine strategy first constructs hidden-state-level graphs, then optionally refines selected important subgraphs with neuron-level relevance.The paper focuses on coarse-grained state-level analysis and leaves systematic neuron-level graph analysis to future work.

4 Experiments

Across benchmarks, contrastive attribution identifies recurring failure signatures, while attribution graphs explain some cases that input attribution alone cannot. Larger models and later training checkpoints show systematic attribution shifts associated with correcting incorrect token preferences, though some failures remain unexplained.

  • Attribution analysis: High-quality contrast token pairs are identified for 20.8% of IFEval, 17.0% of GAIA2, 40.7% of MATH, and 19.6% of EvalPlus failure cases.These pairs support attribution of the logit difference between incorrect and correct candidate tokens.
  • Attribution analysis: MATH retains a substantial fraction of failures unexplained after attribution graph analysis because aggregated hidden-state graphs may be too coarse for numerical reasoning errors.The authors suggest neuron-level tracing may be needed for cases such as an incorrect preference for “0” over “1”.
  • Attribution analysis: Underweighting relevant tokens is the dominant failure mode across all four benchmarks, while overweighting irrelevant tokens is especially substantial for EvalPlus and GAIA2.The latter pattern is often accompanied by underweighting relevant tokens, except in MATH.
  • Attribution analysis: Attribution graphs reveal layer-wise relevance interactions in cases where input attribution provides limited explanation.In one case, “«” relevance rises from 6.72 to 14.0 at Layer 16, but later increments from “First” make its cumulative relevance larger and produce the erroneous output.
  • Attribution shifts with model scaling: Larger Qwen3 models generally correct incorrect token preferences made by Qwen3-0.6B, although a subset of cases remains uncorrected.Most comparisons lie below the y = x line, and remaining failures appear in the yellow region.
  • Attribution shifts with model scaling: For corrected samples, all three prompt segments exhibit negative relevance, while larger models show increasingly negative relevance overall and uncorrected samples retain nearly identical distributions.The segments are Instruction, Query, and Answer; the authors associate these shifts with greater attention to task requirements and less reliance on superficial continuation.
  • Evolution across training: Training produces progressive attribution shifts: the largest changes occur during SFT, inflection points align with logit-difference reductions, DPO pushes all segments more negative, and RLVR changes are more modest.The authors relate the stage difference to IFEval’s stronger alignment with SFT and the ordering of the training pipeline.
  • Practical implications: Masking approximately 2 top-attributed input tokens on average can flip the model away from an error token, suggesting targeted prompt adjustments for debugging.The experiments are described as perturbation results in Appendix G.

5 Conclusion

The paper develops contrastive, LRP-based attribution for analyzing LLM failures and an efficient procedure for constructing cross-layer attribution graphs. The approach uses batched relevance propagation to make graph construction more practical for long-context inputs.

  • 5 Conclusion: Contrastive attribution explains why a model prefers an incorrect token over a correct alternative by attributing their logit difference to tokens and hidden states.The framework builds on AttnLRP and extends it to construct cross-layer hidden-state attribution graphs.
  • 5 Conclusion: The procedure uses modified gradient×input computations to propagate relevance through standard automatic-differentiation frameworks.This avoids implementing explicit relevance-propagation rules beyond the modified backward pass.
  • 5 Conclusion: The attribution graph represents relevance flowing between hidden states across transformer layers.Nodes receive gradient×input relevance, while directed edges describe relevance propagation between source- and target-layer hidden states.
  • 5 Conclusion: Batch-packed multi-target backpropagation reduces graph construction from O(n) backward passes to O(⌈n/B⌉) passes.A single automatic-differentiation call processes B attribution targets in parallel while remaining compatible with LRP-style propagation.

B Computational Efficiency of Batch-Packed Backpropagation

The appendix evaluates batch-packed backpropagation for attribution-graph construction and describes how resulting graphs are sparsified and connected to the final prediction. Batching provides speedups, while latency and memory grow with sequence length and model size.

  • Computational efficiency: Batching yields significant speedups across input lengths, although the gain diminishes for longer sequences.Peak memory scales linearly with input length.
  • Computational efficiency: Latency and memory scale roughly proportionally to model size at fixed batch size, indicating scalability to larger models.The model-size measurements use batch size B=8.
  • Graph pruning: Dense interaction matrices are pruned independently for each layer pair using magnitude-based criteria.Supported modes include global threshold pruning and per-layer cumulative-mass pruning.
  • Connected subgraph extraction: The connected attribution subgraph retains only nodes and edges that can reach the final-layer, last-token target node.This ensures every retained graph element lies on a relevance path to the final prediction.

D More Details on Failure Case Collection

Failure cases are collected with benchmark-specific reasoning-mode settings and a semi-automated procedure for identifying the first erroneous token. The localization workflow prioritizes the final answer, then examines reasoning for the earliest error while excluding formatting tokens.

  • Failure-case collection: Thinking mode is disabled for IFEval, MATH, and EvalPlus, but retained for GAIA2 because its tasks involve complex multi-step interactions.The choice is intended to simplify attribution for some benchmarks while preserving GAIA2’s default configuration.
  • Robustness check: 16 of 21 clean IFEval cases receive identical annotations with and without thinking mode.Of the five differing cases, one has unbounded generation and four shift target–contrast pairs.
  • Target-token identification: The first output token that causes reasoning to deviate from the correct trajectory is designated as the target token.Multiple powerful LLMs propose target tokens, majority voting selects the automated result, and human annotators verify it.
  • Localization procedure: The localization procedure examines the final answer first, then identifies the first erroneous token in the completion or reasoning when needed.If the final answer is correct, the procedure outputs <CORRECT>; otherwise, reasoning is also examined for earlier errors.
  • Localization constraints: Special formatting tokens are excluded as erroneous tokens, while subwords and punctuation may be valid localized tokens.The indexed completion provides token positions for reporting the selected error.

E.2 Target Token Agreement Statistics

The appendix validates a semi-automated target-token identification pipeline and describes how contrast tokens are selected. Strict contrast-token requirements improve pair confidence but are the main source of case filtering.

  • Target-token identification: Four independent LLMs propose target tokens, and human annotators validate cases reaching majority agreement.The pipeline uses majority agreement of at least two validators.
  • Target-token identification: Agreement among independent LLM proposers is far above chance across all benchmarks.Table 5 reports consensus within token windows alongside benchmark-specific chance baselines.
  • Target-token identification: Human approval rates confirm that LLM majority consensus aligns well with human judgment.Approval is evaluated under ±3-token matching.
  • Contrast-token selection: Contrast tokens are selected from the original model’s top-10 predictions, then a stronger sibling model, or manual inference if necessary.When multiple candidates exist, the highest-ranked candidate in the original model’s logits is chosen.
  • Filtering and alternatives: Strict contrast-token recovery is the dominant filtering factor, rather than target-token ambiguity or failure to meet the logit threshold.The requirement is retained to ensure clean, high-confidence contrastive pairs.
  • Filtering and alternatives: 7.51 out of 10 top positively contributing tokens overlap on average between baseline and additional plausible contrast-token pairs.This result comes from three additional pairs per case on a random subset of 12 failure cases.
  • Filtering and alternatives: The authors retain the stricter criterion while leaving more flexible contrast-token alternatives for future work.The appendix suggests that the requirement could in principle be relaxed.

F Inter-Annotator Agreement Statistics on Attribution Analysis

The appendix measures agreement for attribution-outcome and failure-pattern annotations using a standardized calibration protocol. Post-calibration agreement improves substantially, and the main-text results use those annotations.

  • Annotation protocol: Two annotators independently classify attribution outcomes and failure patterns under a standardized three-phase protocol.The phases are pre-calibration, calibration discussion, and post-calibration measurement.
  • Agreement results: 88.2% post-calibration agreement is achieved for attribution-outcome classification.Table 8 reports agreement rates with Wilson 95% confidence intervals.
  • Agreement results: 83.3% post-calibration agreement is achieved for failure-pattern classification.Table 9 reports agreement among cases categorized as M-IA or NC-IA+M-AG.
  • Agreement results: All main-text results use post-calibration annotations.Calibration therefore precedes the labels used in the reported analysis.

G Comparison with Alternative Attribution Methods

The appendix compares AttnLRP with CP-LRP and Gradient using perturbation-based faithfulness, attribution sharpness, and a qualitative example. LRP methods perform similarly and outperform the gradient baseline on the reported diagnostics.

  • Evaluation design: The comparison evaluates AttnLRP, CP-LRP, and Gradient on 60 randomly selected failures across IFEval, EvalPlus, MATH, and GAIA2.Tokens are ranked by attribution and masked until the model’s top-1 prediction changes.
  • Perturbation-based evaluation: 100% fix rates are achieved by both LRP-based methods, while Gradient still fails in some cases.The average numbers of tokens required are 1.7 for AttnLRP and 2.0 for CP-LRP.
  • Attribution sharpness: AttnLRP achieves the highest mean and median concentration and Gini metrics, with lower Gini variability than CP-LRP.Its Gini standard deviation is 0.066 versus 0.105 for CP-LRP.
  • Qualitative comparison: In the IFEval example, Gradient emphasizes irrelevant content and style tokens, whereas AttnLRP and CP-LRP emphasize constraint-related tokens.The contrast is examined for the target comma versus the alternative token “ and”.
  • Interpretation: The authors report that AttnLRP and CP-LRP perform comparably in this diagnostic setting.The framework’s contribution is the diagnostic approach rather than a uniquely required attribution method.

H Sample Expanded Attribution Graphs

Expanded attribution graphs expose internal relevance pathways in representative failures and support exploratory analyses of relevance profiles and source composition. These analyses show that layer-wise relevance shape and attribution origin capture distinct, largely independent structure.

  • Expanded graphs: The expanded graph preserves more relevance propagation than the reduced main-text graph while still pruning extremely low-contribution edges for readability.Figure 8 provides the expanded version for the case shown in Figure 3.
  • Representative cases: A MATH example shows the target-token preference driven overwhelmingly by internal biases, especially the BOS token, with minimal contribution from the prompt’s reasoning chain.Figure 9 presents this expanded attribution graph.
  • Relevance profiles: Relevance profiles track the prediction token across transformer layers and can show either rapid early growth followed by saturation or delayed late-layer growth.For Qwen3-0.6B, relevance averages 0.66 at the embedding layer and reaches 18.85 at the final layer.
  • Relevance-profile clustering: The exploratory clustering analysis applies k-means with k=3 to 29-dimensional normalized relevance profiles from 23 NC-IA failure traces.Cases already explained by input attribution are excluded from these graph-based analyses.
  • Attribution decomposition: Attribution composition separates self bias, BOS contribution, and other-token contribution to the prediction token.SB measures residual internal bias, BOS measures the attention-sink contribution, and OC measures context-token influence.
  • Profile shape and composition: Profile-based clusters and composition structure are largely independent, with an Adjusted Rand Index of −0.004 for direct composition-space clustering.Intra-cluster variance is 2.61× and 8.80× larger than inter-cluster variance for SB and BOS fractions, respectively.

I.3 Layer-Wise Functional Specialization

Attribution graphs reveal a robust three-phase specialization across traces: early layers establish bias, mid layers integrate context, and late layers amplify bias. Critical transitions consistently cluster in the final 6–8 layers, suggesting where failure-relevant decisions occur.

  • Functional specialization: Early-layer self bias is highest at approximately 0.73, declining to approximately 0.49–0.51 in mid and late layers.The analysis characterizes early computations as primarily driven by position-specific biases rather than contextual information.
  • Functional specialization: Other-token contribution peaks at approximately 0.51 in mid layers, where contextual information integration is most active.The finding is consistent with prior observations linking mid-layer attention heads to semantic composition.
  • Functional specialization: BOS fraction rises from approximately 0.15 in early layers to approximately 0.27–0.36 in late layers, consistent with stronger deep-layer attention-sink influence.This increase represents growing BOS influence deeper in the transformer stack.
  • Functional specialization: Across all 23 traces, attribution follows three phases: early bias establishment, mid-layer context integration, and late-layer bias amplification.This pattern holds regardless of profile cluster, dataset, or decomposition pattern.
  • Critical transitions: Median peak transition layers concentrate at layers 20–26, with BOS and OC transitions clustering around layers 23–26.The final 6–8 layers are therefore identified as the practical focus for targeted interventions such as activation editing or attention-head pruning.

I.5 Cross-Model Comparison of Attribution Decomposition

Cross-model attribution comparisons show that scaling changes how relevance is composed and propagated, not merely its token-level weighting. In corrected cases, larger models shift total relevance toward the correct alternative, reduce BOS reliance, and typically reverse contextual relevance toward the correct token.

  • Cross-model structure: Corrected cases shift total relevance from favoring the incorrect token in smaller models toward favoring the correct alternative in larger models.For Number Theory_8, total relevance changes from 0.693 in Qwen3-0.6B to −1.288 in Qwen3-1.7B.
  • Component changes: In corrected cases, the OC component typically flips from positive to negative as larger models assign contextual relevance supporting the correct token.The sign change indicates that context integration changes direction relative to the incorrect output.
  • Cross-model structure: Across Qwen3 sizes from 0.6B to 8B, larger-model corrections involve deeper structural changes in relevance composition and propagation.The comparison uses the same 23 NC-IA failure cases and decomposes relevance into SB, BOS, and OC trajectories.
  • Summary: Attribution graphs exhibit heterogeneous case-level structure but two aggregate regularities: three-phase specialization and critical transitions concentrated in the final 6–8 layers.These regularities provide actionable guidance for interventions and connect to behavioral failure patterns in the main text.
Loading 2604.17761v1…