From Understanding to Utilization: A Survey on Explainability for Large Language Models

1 Introduction

• Opacity in LLMs can accompany harmful or misleading content and hallucinations, underscoring the need for explainability in responsible and ethical application.The cited examples include harmful or misleading generation (Gehman et al., 2020) and hallucinations (Weidinger et al., 2021).Source: §1 · p. 1
• Explainability can clarify model reasoning for end users and help developers and researchers identify biases and improve downstream-task performance.The survey also notes that large models are harder to interpret and traditional methods such as SHAP become less practical at that scale.Source: §1 · p. 1
• The survey examines explainability methods for pre-trained Transformer-based LLMs, including the LLaMA family, and seeks to bridge model understanding with practical application.It reviews methods, uses of explainability insights, and evaluation methods and metrics.Source: §1 · pp. 1–2

2 Overview

• Explainability is presented as a tool for understanding complex LLMs and as essential to their improvement.The overview also highlights ethical and controllable generation challenges and proposes future research questions.Source: §2 · p. 2
• Local analysis covers feature attribution and Transformer-block analysis, while global analysis covers probing-based methods and mechanistic interpretability.The two domains distinguish detailed model operations from broader model behaviors and capacities.Source: §2 · p. 2
• The survey also examines applications of explainability insights in model editing, capability enhancement, and controlled generation.These applications focus on enhancing LLM capabilities beyond understanding model behavior.Source: §2 · p. 2

3.1.1 Feature Attribution Explanation

• Feature attribution assigns each input token a relevance score for a prediction, using perturbation-based, gradient-based, or vector-based methods.Vector-based analyses decompose token representations across input tokens, but attention rollouts often omit nonlinear FFN contributions; newer methods approximate and decompose activations.Source: §3.1.1 · pp. 2–3
• Perturbation methods such as LIME and SHAP change input features, but assume independence and ignore correlations; efficiency and reliability concerns have reduced their recent emphasis.Models may remain overconfident for nonsensical or incorrect predictions.Source: §3.1.1 · p. 2
• Integrated gradients accumulate gradients along an interpolation from a reference input, but high-quality integrals are computationally costly and attribution faithfulness remains unreliable.Attribution scores also offer limited insight into forward dynamics within hidden states.Source: §3.1.1 · p. 2

3.1.2 Dissecting Transformer Blocks

• Attention analyses use weights and gradient attribution; approaches combining attention metrics with gradient information tend to outperform either metric alone.Attention commonly emphasizes selected tokens over frequent or special tokens.Source: §3.1.2 · p. 4

3.2 Global Analysis

• Global analysis studies encoded knowledge and linguistic properties through representation probes and mechanistic reverse engineering, complementing local explanations of individual predictions.Probing classifiers distinguish input or output types using network activations.Source: §3.2 · p. 4, §3.2.1 · p. 5

3.2.1 Probing-Based Method

• Probes train classifiers on model activations; truthfulness is represented mainly in early-to-middle layers, with only a few attention heads showing standout linear probing accuracy.Other work identified nonlinear representations of board-game states in models initially lacking explicit game knowledge.Source: §3.2.1 · p. 5

3.2.2 Mechanistic Interpretability

• Circuit discovery reverse-engineered GPT-2 small’s indirect-object-identification task into a 26-head circuit, 1.1% of all head-position pairs, with later work exploring scaling to in-context learning.The circuit is a human-understandable subgraph of the computational graph.Source: §3.2.2 · p. 5
• Causal tracing identifies middle-layer states, especially last-subject-position MLPs, as important for factual recall; last-token attention modules also act as crucial mediators [Meng et al., 2023a].Figure 3 maps average causal indirect effects, with darker cells indicating stronger mediation.Source: §3.2.2 · p. 5, Figure 3 · p. 6

4 Leveraging Explainability

• The survey focuses on explainability-grounded methods for debugging and improving model capabilities rather than general fine-tuning or retraining approaches.It emphasizes methods designed with a strong foundation in explainability.Source: §4 · p. 6

4.1 Model Editing

• Explainability-based editing targets specific knowledge or behavior changes while seeking to preserve performance on other inputs.Methods include memory-based or parameter-augmented editors, locate-then-edit approaches, and updates to attention values or FFN parameters.Source: §4.1 · pp. 6–7

4.2 Enhancing Model Capability

• Explainability insights are used to improve long-text utilization and in-context learning.The survey highlights both as capability-enhancement tasks in recent work.Source: §4.2 · p. 7

4.2.1 Improving Utilization of Long Text

• StreamingLLM retains initial attention-sink tokens alongside recent KV states to support unlimited text without fine-tuning.A dedicated sink token during pre-training can further improve streaming performance.Source: §4.2.1 · p. 7

4.2.2 Improving In-Context Learning

• Mechanistic interpretability informs ICL improvements through demonstration-label reweighting, ablation of false-induction heads, and task or function vectors for few-shot prompting.The cited work reports that summing function vectors can trigger new complex tasks and improve few-shot prompting performance.Source: §4.2.2 · p. 7

4.3 Controllable Generation

• Explainability offers inference-time techniques to improve generation factuality, calibration, and controllability and better align outputs with human preference.The survey frames these techniques as fast opportunities for improving generation models.Source: §4.3 · p. 7

4.3.1 Reducing Hallucination

• Intervening on attention heads in a “truthful” direction achieves performance comparable to or better than an instruction-finetuned model.The approach locates truthful heads and directions using as few as 40 samples, then shifts activations at inference time.Source: §4.3.1 · pp. 7–8

4.3.2 Ethical Alignment

• An interpretable technique identifies social-bias neurons, whose activations are then suppressed to mitigate bias.The survey reports experiments verifying the method’s effectiveness and potential applicability to ethical-alignment research.Source: §4.3.2 · p. 8

5 Evaluation

• Evaluation needs purpose-designed methods for explainability and calibrated datasets and metrics for downstream tasks such as truthfulness evaluation.The survey notes that natural-language explanations may not actually help humans understand model reasoning.Source: §5 · p. 8

5.1 Evaluating Explanation Plausibility

• Attribution plausibility can be assessed by removing the most or least important tokens and measuring output changes, or indirectly through model-editing performance.Factual editing evaluation commonly uses ZsRE, with backtranslation-generated question rephrasings, and the more challenging CounterFact, which includes counterfacts initially scored below correct facts.Source: §5.1 · pp. 8–9

5.2 Evaluating Truthfulness

• TruthfulQA uses GPT-judge models to assess truth and informativeness, with true*informative combining both scores to discourage uninformative refusals.The benchmark is adversarially constructed to measure truthful answers.Source: §5.2 · p. 9

• The survey organizes LLM explainability methods into local and global analyses based on the objectives of explanations.Source: §6 · p. 9
• It also reviews how explanations can enhance models and how explainability methods are evaluated.Source: §6 · p. 9
• Future directions include tailoring explanation methods to different language models and using explainability knowledge to make LLMs more trustworthy and aligned with human values.The review aims to highlight open problems and directions for future research.Source: §6 · p. 9