Explainability for Large Language Models: A Survey

Motivation

• LLMs’ strong performance and growing commercial use contrast with opaque internal mechanisms that can pose risks such as harmful content or hallucinations.This motivates developing explainability to clarify how these models work.Source: §1 · p. 1
• Explainability can help end users understand model reasoning, capabilities, limitations, and flaws, while helping researchers and developers identify risks and improve downstream performance.It also supports tracking capabilities over time, comparing models, and developing reliable, ethical, and safe systems for deployment.Source: §1 · pp. 1–2

Challenges and opportunities

• LLMs’ scale makes explanations harder and more computationally costly, but also creates opportunities for models to explain their own decisions and provide post-hoc explanations for other models.The paper highlights the need to understand in-context learning, chain-of-thought prompting, and hallucination to explain and improve LLMs.Source: §1 · pp. 2–3

Paper scope

• The paper surveys explainability methods for Transformer-based language models, organizing them around traditional downstream fine-tuning and prompting paradigms.It also covers evaluation methods, research challenges, and potential future directions.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Traditional Fine-Tuning Paradigm

• Traditional fine-tuning pre-trains on unlabeled text, then adapts models to labeled downstream data, with explanations examining both acquired language knowledge and task-solving abilities.Fully connected layers can be added above the final encoder layer; this paradigm has succeeded for medium-sized models, typically up to one billion parameters.Source: §2.1 · p. 3

2.2 Prompting Paradigm

• Base models use prompts for zero- or few-shot learning without additional training data, and explanations examine how they leverage pre-trained knowledge to respond.This paradigm typically involves models exceeding 1B parameters, including GPT-3 (Brown et al., 2020) and LLaMA-2, which has up to 70B parameters.Source: §2.2 · p. 3
• Assistant models are further fine-tuned to address base models’ instruction-following and biased or toxic output limitations, aligning responses with human feedback and preferences.Instruction tuning uses prompt-response demonstrations and RLHF; explanations examine how assistant models learn open-ended interactive behavior through conversations, as in GPT-4 (Bubeck et al., 2023) and Claude.Source: §2.2 · pp. 3–4

Local explanations

• Local explanations aim to clarify how a language model predicts an output for a specific input instance.Source: §3 · p. 4

Global explanations

• Global explanations aim to provide a broad understanding of how an LLM works overall.Source: §3 · p. 4

Debugging and model improvement

• The section also discusses using explanations to debug and improve models.Source: §3 · p. 4

3.1.1 Feature Attribution-Based Explanation

• Feature attribution assigns relevance scores to input features; perturbation methods remove, mask, or alter inputs, but can ignore feature correlations, elicit overconfident outputs, and create out-of-distribution examples.Leave-one-out methods remove features to test prediction changes. Confidence-based methods can still yield high-confidence predictions on nonsensical inputs; regularization and keeping perturbed data near the original distribution are proposed mitigations.Source: §3.1.1 · pp. 4–5
• Gradient methods score output sensitivity, while integrated gradients accumulate gradients from a reference; surrogate and decomposition methods offer local approximations or relevance breakdowns.Integrated gradients address some vanilla-gradient limitations, but baseline choice, saturation, computational cost, and the mismatch between straight-line paths and discrete word embeddings remain challenges. LIME approximates behavior locally; SHAP assigns feature contributions but faces baseline-selection and computational-cost challenges.Source: §3.1.1 · pp. 5–6

3.1.2 Attention-Based Explanation

• Attention explanations visualize patterns at input, head, and model levels, and gradient-attention scores generally outperform either signal alone, though raw attention’s explanatory validity remains contested.Studies disagree on whether raw attention identifies important predictive features or captures relevant structure; the debate persists partly because evaluation criteria are not well established.Source: §3.1.2 · pp. 7–8

3.1.3 Example-Based Explanations

• Counterfactuals change significant input parts to show how predictions would differ; Polyjuice generates task-specific variants, while MiCE and CREST produce valid, fluent counterfactuals with different edit constraints.MiCE selects tokens using gradient-based attribution and optimizes minimal edits; CREST uses rationales and a sparsity budget to control closeness without the same hard minimality constraint.Source: §3.1.3 · pp. 8–9
• Replacing 5% of words reduces BERT’s accuracy from 70.6% to 2.4% in SemAttack’s white-box setting, demonstrating how adversarial examples expose model weaknesses.Adversarial examples make small input changes to alter decisions; contextualized mask-then-infill methods aim to produce more coherent perturbations than word-embedding-based approaches.Source: §3.1.3 · p. 8
• Data-influence methods trace how training examples affect test predictions or loss, but can be computationally expensive; larger-scale analyses find influential sequences better capture query reasoning.Influence functions approximate loss changes without retraining, while TracIn uses training checkpoints. EK-FAC has been used to scale influence analysis to LLMs.Source: §3.1.3 · p. 9

3.1.4 Natural Language Explanation

• Natural language explanations are generated from input sequences, often by models trained on human-annotated explanations; they can support prediction and data augmentation, but their reliability needs further study.This section’s explanations are typically generated by a separate model, unlike chain-of-thought explanations produced by the LLM itself.Source: §3.1.4 · p. 9

3.2.1 Probing-Based Explanations

• Studies find lower layers more predictive of word-level syntax and higher layers more capable of sentence-level syntax and semantic knowledge (Belinkov et al., 2017; Peters et al., 2018; Blevins et al., 2018; Jawahar et al., 2019), while probing validity remains under investigation.Selectivity constraints may help mitigate potential probe biases.Source: §3.2.1 · pp. 10–11
• Probing also uses span-based classifiers, attention analysis, and prompt-based completion, though dataset regularities can obscure whether prompted answers reflect model knowledge.Edge probing represents spans for syntactic and semantic tasks (Tenney et al., 2019b); attention patterns have been used to prune heads and trace token relevance (Kovaleva et al., 2019; Clark et al., 2019; Wu et al., 2020a). In cloze probing, BERT gave the correct hypernym answer in the top five for all samples.Source: §3.2.1 · pp. 10–11

3.2.2 Neuron Activation Explanation

• GPT-4-generated explanations captured the behavior of more than 1,000 GPT-2 XL neurons, while missing ground-truth annotations make component-level evaluation difficult.Neuron methods rank candidate units and test their importance through ablations; the Summarize and Score pipeline generates and tests module explanations, which sometimes resemble human labels (Singh et al., 2023).Source: §3.2.2 · pp. 11–12

3.2.3 Concept-Based Explanation

• Concept-based methods score human-readable concepts using examples and concept activation vectors, but depend on descriptive data and reliable concept classifiers.For negative-sentiment sentences, TCAV assigns Positive Adjectives a lower score than Neutral; neuron-derived concepts can also combine into logical forms, and greater neuron interpretability is associated with greater model accuracy (Mu & Andreas, 2021).Source: §3.2.3 · p. 12

3.2.4 Mechanistic Interpretability

• Mechanistic interpretability studies features, neuron connections, and circuits; transformer circuit work includes induction heads and feedforward-layer knowledge, but its explanatory hypotheses remain uncertain at LLM scale.Two-layer attention-only models exhibit induction heads that may support in-context learning, while feedforward-layer analyses connect keys to textual patterns and identify middle layers relevant to facts. Circuit methods have been adapted to a 70B Chinchilla model, but many hypotheses remain untested on LLMs (Lieberum et al., 2023).Source: §3.2.4 · p. 13

3.3.1 Debugging Models

• Feature-importance patterns can reveal biases or limitations, such as context-insensitive attention that suggests heuristic reliance rather than understanding (Du et al., 2023).Source: §3.3.1 · p. 14
• Integrated Gradients reveal that language models can favor head features in long-tailed distributions as early-learned shortcuts, harming robustness and OOD generalization (Du et al., 2021).Source: §3.3.1 · p. 14
• Integrated Gradient explanations show that adversarially robust models rely on similar features, unlike non-robust models, motivating more robust adversarial training (Chen & Ji, 2022).Source: §3.3.1 · p. 14

3.3.2 Improving Models

• AMPLIFY improves LLM accuracy by 10-25% across various tasks, including when human rationales are lacking.It generates automated rationales with post-hoc explanation methods and feeds them into prompts for prediction (Ma et al., 2023).Source: §3.3.2 · p. 14
• ER-TEST finds that explanation regularization has little impact on in-distribution performance but yields large OOD gains across sentiment analysis and natural language inference.It evaluates generalization with unseen-dataset, contrast-set, and functional tests, using Input*Gradient, attention-based, and learned rationales (Joshi et al., 2022; Stacey et al., 2022; Chan et al., 2022a).Source: §3.3.2 · p. 14
• XMD improves text-classification OOD performance by up to 18% by using user feedback on explanations to regularize models in real time.Users can provide flexible feedback on task- or instance-level explanations through an intuitive interface.Source: §3.3.2 · p. 14

4 Explanation for Prompting Paradigm

• Prompting-based LLMs’ reasoning makes example-specific explanations less meaningful, while their scale makes costly methods infeasible and their complexity exceeds simplified surrogate models.These challenges motivate explanation methods tailored to prompting-based models.Source: §4 · p. 14
• Emerging approaches use chain-of-thought explanations, identify influential examples to illuminate dataset composition, and apply high-level concept- and module-based global explanations.The passage notes that global explanation techniques developed for traditional fine-tuning paradigms are also used for prompting-based LLMs.Source: §4 · p. 14
• The section covers explanations for base models and assistant models, then examines using LLM reasoning and explanation to improve prediction and enable beneficial applications.It identifies LLaMA-2 and LLaMA-2-Chat as examples of the two model types.Source: §4 · p. 15

4.1 Base Model Explanation

• The section identifies three goals: explain few-shot learning, explain chain-of-thought prompting, and engineer representations.Understanding how models grasp tasks from limited examples can help end-users interpret model reasoning.Source: §4.1 · p. 15

4.1.1 Explaining In-context Learning

• ICL studies find that large models can learn contradictory input-label mappings while small models rely more on semantic priors (Wei et al., 2023b).Flipping labels can reduce salience for smaller models such as GPT-2 but have the opposite impact on large models such as InstructGPT; effects vary by scale and task (Li et al., 2023b).Source: §4.1.1 · p. 15

4.1.2 Explaining CoT Prompting

• Saliency analysis suggests CoT stabilizes attention to question tokens, while perturbed demonstrations indicate intermediate reasoning steps cue symbol replication into factual answers (Wu et al., 2023a; Madaan & Yazdanbakhsh, 2022; Wang et al., 2022a).The saliency analysis uses gradient-based feature attribution; counterfactual prompts perturb symbols, patterns, and text.Source: §4.1.2 · p. 15

4.1.3 Representation Engineering

• Representation engineering reads and controls high-level concepts: adding or subtracting reading vectors can induce honest or lying outputs (Zou et al., 2023).Probes reveal linear truth directions and spatial or temporal representations that become more accurate with model scale; representation control is promising, but further ablations are needed (Marks & Tegmark, 2023; Gurnee & Tegmark, 2023).Source: §4.1.3 · p. 16

4.2.1 Explaining the Role of Fine-tuning

• Assistant models’ knowledge is mainly acquired during pre-training; 1,000 carefully selected instructions brought LLaMA-65B to performance comparable to GPT-4, underscoring pre-training’s importance.The study suggested alignment may chiefly teach interaction styles and formats, and that instruction-data quality can matter more than quantity; reinforcement learning can further align models with human values.Source: §4.2.1 · pp. 16–17
• Instruction fine-tuning improves instruction-following and instruction/context distinction, shifts focus toward middle and tail prompt content, and adjusts concepts for downstream users, while imitation improves style and following but not factuality, coding, or problem solving.Fine-tuning changes how semantic knowledge is distributed across layers and makes more lower-layer neurons encode word-word patterns for recognizing instruction verbs; LLaMA-2’s report describes fine-tuning as increasing helpfulness and safety.Source: §4.2.1 · pp. 16–17

4.2.2 Explaining Hallucination

• Hallucinations are associated with missing or repeated training data and model limitations, including weak long-tail knowledge, reliance on memorization and statistical patterns, reasoning failures, overconfidence, and sycophancy.Repeating 10% of training data reportedly degraded an 800M-parameter model’s performance to that of a 400M-parameter model; scaling or instruction tuning may increase sycophancy.Source: §4.2.2 · p. 17
• Hallucinations can be addressed through scaling, extractive fine-tuning data, adapter fine-tuning, improved datasets, synthetic-data interventions, and model-side optimization, though extractive data and adapters do not change the hallucination distribution.PaLM’s 540-billion-parameter model improved on varied tasks and showed better memorization than smaller models, while still struggling with long-tail knowledge.Source: §4.2.2 · pp. 17–18

4.2.3 Uncertainty Quantification

• Because logits are less suitable for many LLMs, especially closed-source models, confidence elicitation estimates uncertainty through response consistency, verbal confidence statements, or aggregation of token-level uncertainty.Consistency methods vary generation with randomness or misleading prompt hints; greater consistency across responses indicates higher estimated confidence.Source: §4.2.3 · p. 18

4.3.1 Improving LLMs

• Few-shot explanations, including chain-of-thought reasoning steps, can improve performance, with benefits depending on model scale and explanation quality and increasing with task-specific validation-set customization (Wei et al., 2022).The reported gains span arithmetic, symbolic, and commonsense reasoning tasks.Source: §4.3.1 · p. 18
• Explanation tuning uses detailed step-by-step responses from GPT-4 to train smaller models, which outperform conventional instruction-tuned models on complex zero-shot BigBench Hard reasoning benchmarks (Mukherjee et al., 2023).This richer supervision aims to teach reasoning processes rather than merely imitate larger models’ style.Source: §4.3.1 · p. 18
• Instruction analysis finds label-relevant information most important for classification, and automatically removing 60% of task-definition tokens maintained or improved T5-XL performance on a holdout dataset (Yin et al., 2023).Performance dropped substantially only when output-label information was removed.Source: §4.3.1 · p. 19
• Intermediate computation steps boosted multi-step symbolic reasoning, but textual-reasoning gains were limited; explanations could be unrealistic yet prediction-consistent, motivating selection of explanations associated with high performance (Nye et al., 2021; Ye & Durrett, 2022).For textual reasoning, only text-davinci-002 was observed to improve; the other four listed models showed no clear improvement or performed worse.Source: §4.3.1 · p. 19

4.3.2 Downstream Applications

• A training-free pipeline combines ChatGPT-generated descriptions of disease symptoms and visual features with CLIP for explainable zero-shot medical-image classification (Radford et al., 2021).Prompts target high-quality descriptions of visually identifiable symptoms to address potential inaccuracies in ChatGPT’s medical descriptions; experiments across multiple datasets showed effectiveness and explainability.Source: §4.3.2 · p. 19

5 Explanation Evaluation

• Evaluation covers local explanations for fine-tuned models and natural-language CoT explanations for prompting, with plausibility and faithfulness as key dimensions.Human evaluations compare rationales with human explanations or rely on subjective judgments, but can neglect faithfulness; automatic tests perturb rationales to assess importance and avoid human bias.Source: §5 · p. 19

5.1 Explanation Evaluations in Traditional Fine-tuning Paradigms

• Local-explanation plausibility can be assessed across grammar, semantics, knowledge, reasoning, and computation against human-annotated rationales.Benchmark rationales should be sufficient for correct predictions and compact enough that removing any part changes the prediction; metrics include token-level IOU, precision and recall, plus overall F1 or AUPRC.Source: §5.1 · p. 20
• Faithfulness principles include implementation and input invariance, input sensitivity, completeness, polarity consistency, prediction consistency, and sufficiency.Completeness applies only to differentiable methods, and each principle addresses only one facet, so no single framework provides an all-in-one solution.Source: §5.1 · p. 20
• Perturbation metrics measure how rationale tokens affect predictions: higher comprehensiveness and lower sufficiency indicate influential, prediction-supporting rationales.TaSc favors higher DFMIT and lower DFFOT; shortcut faithfulness is assessed with precision@k and mean rank, while violation tests check feature importance and polarity.Source: §5.1 · p. 21
• Faithfulness metrics can rank the same explanation model inconsistently: DFFOT may favor an explanation that SUFF ranks worst.TaSc finds attention-based importance metrics more robust than non-attention metrics, although treating attention as an explanation remains debated.Source: §5.1 · p. 21
• Natural-language explanation faithfulness tests use counterfactual insertion and input reconstruction, but new linguistic variants can make both tests unfair.Rev instead assesses whether explanations support predictions and whether information added by explanations justifies them.Source: §5.1 · p. 21

5.2 Evaluation of Explanations in Prompting Paradigms

• GPT-3.5 and GPT-4 explanations have low simulation precision, potentially misleading people about model behavior on counterfactual inputs.Counterfactual simulatability is measured by simulation generality and simulation precision; plausibility optimization alone may not improve it.Source: §5.2 · p. 22
• CoT explanations can be systematically unfaithful; question decomposition can approach CoT performance while improving faithfulness on several metrics.Models may omit prompt biases, and one study found smaller models more faithful on some tasks than larger, more capable models; further work is needed.Source: §5.2 · p. 22

6.1 Explanation without Ground Truths

• Without ground-truth explanations or global-component benchmarks, designing accurate explanations, evaluating faithfulness and fidelity, and selecting among methods are difficult; human evaluations and synthetic datasets are proposed.Source: §6.1 · p. 22

6.2 Sources of Emergent Abilities

• The origins of scale- and data-linked emergent abilities remain unclear, motivating study of model architecture, training-data subsets and quality, and whether contamination explains them.Open questions include the minimum model complexity for strong performance across diverse tasks; proprietary models’ unpublished architectures and training data and limited interpretability of open models impede investigation. Researchers also ask whether emergent abilities arise from training or contamination (Blevins et al., 2023); better understanding could support more controllable and reliable models.Source: §6.2 · pp. 22–23

6.3 Comparing Two Paradigms

• Fine-tuned and prompted models can differ in in-distribution and out-of-distribution performance, but their explanations have not been comprehensively compared.Research questions include differences in prediction rationales and whether divergent OOD robustness reflects reasoning differences; answers could guide paradigm choice and improve robustness.Source: §6.3 · p. 23

6.4 Shortcut Learning of LLMs

• Fine-tuned models exploit NLI dataset biases, while prompted models perform best when relevant information is at context edges; both paradigms can use shortcuts that harm generalization.Reported fine-tuning shortcuts include lexical, overlap, position, and style bias (Du et al., 2023); prompted models’ performance worsened when relevant information appeared in the middle of long contexts (Liu et al., 2023b).Source: §6.4 · p. 23

6.5 Attention Redundancy

• Studies find redundant attention heads in BERT-base that can be pruned with little downstream impact, and redundant attention heads and feedforward networks in OPT-66B (Bansal et al., 2022).These findings suggest opportunities to compress models by pruning redundant components while preserving downstream performance.Source: §6.5 · p. 23

6.6 Shifting from Snapshot Explainability to Temporal Analysis

• Snapshot explanations can miss training dynamics, phase transitions, and feature interactions; checkpoint analysis found syntactic attention structure emerging alongside a steep training-loss drop (Chen et al., 2023a).That structure was critical for grammatical learning, and temporal analysis may reveal useful training patterns for understanding and improving model performance.Source: §6.6 · pp. 23–24

6.7 Safety and Ethics

• Explainability can support LLM audits for harms including misinformation, bias, and social manipulation, while tools can expose gender stereotypes (Li et al., 2023a) and problematic learned associations.The section calls for audits, external oversight, and transparency regulations, and proposes explainability audits to supplement human feedback as alignment systems scale.Source: §6.7 · p. 24

• The survey presents a comprehensive overview of LLM explainability methods, organizing local and global explanations around model training paradigms.Source: §7 · p. 24
• Future work includes tailoring explanation methods to different LLMs, evaluating faithfulness, and improving human interpretability.Source: §7 · p. 24
• The survey identifies explainability as increasingly important for making advancing LLMs transparent, fair, and beneficial.Source: §7 · p. 24