PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainability for Large Language Models: A Survey

Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Mengnan Du

arXiv:2309.01029v3cs.CLcs.AIcs.LG

TL;DR

Opaque internal mechanisms and uncertain explanation quality make LLM behavior difficult to understand and evaluate. The paper surveys Transformer-based explainability methods by fine-tuning and prompting paradigms, covering local and global explanations, evaluation, model improvement, and open challenges. It concludes that tailored methods, faithfulness evaluation, and human interpretability remain important directions.

  • Problem

    LLMs’ internal mechanisms remain unclear, and it is uncertain whether their generated explanations help humans understand reasoning and generalize to new inputs.

  • Method

    The paper organizes Transformer-based LLM explanation techniques by fine-tuning and prompting paradigms, covering local and global explanations, evaluation, and explanation-based improvement.

  • Results

    The survey presents a comprehensive overview of local and global explanation methods, evaluation approaches, model-improvement uses, and key challenges for LLM explainability.

  • Takeaways & Limitations

    Future explainability work should develop methods tailored to different LLMs, evaluate explanation faithfulness, and improve human interpretability.

  • Takeaways & Limitations

    Traditional explanation methods face scaling and evaluation constraints, while prompting-based explanations may be less meaningful for reasoning-centered models.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their internal mechanisms are still unclear and this lack of transparency poses unwanted risks for downstream applications. Therefore, understanding and explaining these models is crucial for elucidating their behaviors, limitations, and social impacts. In this paper, we introduce a taxonomy of explainability techniques and provide a structured overview of methods for explaining Transformer-based language models. We categorize techniques based on the training paradigms of LLMs: traditional fine-tuning-based paradigm and prompting-based paradigm. For each paradigm, we summarize the goals and dominant approaches for generating local explanations of individual predictions and global explanations of overall model knowledge. We also discuss metrics for evaluating generated explanations, and discuss how explanations can be leveraged to debug models and improve performance. Lastly, we examine key challenges and emerging opportunities for explanation techniques in the era of LLMs in comparison to conventional machine learning models.

1 Introduction

LLMs’ growing capabilities and deployment make their opaque internal mechanisms a consequential explainability problem. This survey organizes explanation methods across training paradigms, while highlighting their uses, evaluation needs, and scaling challenges.

  • LLMs’ complex, opaque mechanisms make interpreting their behavior difficult despite strong NLP performance and widespread commercial deployment.
  • Explainability helps users understand model capabilities, limitations, and flaws, while helping researchers identify biases, risks, and performance-improvement opportunities.
  • The survey categorizes LLM explainability into two major paradigms and reviews techniques and evaluations associated with each.
  • LLM scale makes traditional explanation methods more computationally demanding and complicates analysis of in-context learning, chain-of-thought prompting, and hallucination.
  • The paper organizes research on interpreting Transformer-based language models and identifies challenges and future directions for improving understanding of LLMs.

2 Training Paradigms of LLMs

LLM use is organized around traditional fine-tuning and prompting paradigms. Prompting supports zero- or few-shot adaptation, while assistant models add supervised alignment to improve instruction following and dialogue.

  • LLM training and downstream adaptation are broadly divided into traditional fine-tuning and prompting paradigms.
  • Traditional Fine-Tuning Paradigm: Traditional fine-tuning pre-trains on unlabeled text and then adapts a model to labeled downstream data through task-specific layers.
  • Prompting Paradigm: Prompting enables zero-shot or few-shot learning by using natural-language prompts without requiring additional training data.
  • Prompting Paradigm: Base models are very large models that develop capabilities such as few-shot learning through prompting as scale and training data increase.
  • Prompting Paradigm: Assistant models are further fine-tuned with supervised instruction tuning and human-feedback alignment to support instruction following and open-domain dialogue.

3 Explanation for Traditional Fine-Tuning Paradigm

For traditionally fine-tuned LLMs, the survey distinguishes local explanations of individual predictions from global explanations of overall model behavior, then considers how explanations can improve models.

  • Local Explanations: Local explanations describe how an LLM produces a prediction for a specific input instance.
  • Global Explanations: Global explanations provide a broad understanding of how the LLM works overall.
  • Organization: The survey reviews explanation techniques for fine-tuned LLMs before discussing explanation-based model improvement.

3.1 Local Explanation

Local explanations clarify how an LLM produces an individual prediction through feature attribution, attention, example-based, and natural-language approaches. These methods expose influential inputs, model sensitivities, counterfactual behavior, and training-data influence, while raising reliability and evaluation challenges.

  • Local explanations target how a language model generates a prediction for one input instance.
  • Feature Attribution-Based Explanation: Feature attribution assigns relevance scores to words, phrases, or spans and includes perturbation-, gradient-, surrogate-, and decomposition-based methods.
  • Feature Attribution-Based Explanation: Perturbation methods alter or remove input features to measure changes in model outputs, but perturbed examples can fall outside the original data distribution.Constraining perturbed data near the original distribution is one proposed mitigation.
  • Feature Attribution-Based Explanation: Gradient methods measure output sensitivity through derivatives, while integrated gradients accumulate gradients between a reference point and the actual input.Vanilla gradients can violate input invariance, mishandle zero-valued inputs, and suffer from saturation; the reference point remains critical for reliable evaluation.
  • Attention-Based Explanation: Attention explanations visualize attention heads with graphs or heatmaps, but raw attention may identify unimportant features, omit syntactic structure, or contain redundant information.The survey notes that enhanced gradient-based attention variants and other technical interventions seek to improve attention explanations, while evaluation criteria remain unsettled.
  • Example-Based Explanations: Counterfactual explanations change selected input features to show how predictions would differ, while data-influence methods trace how individual training samples affect test-point loss.Polyjuice generates realistic counterfactuals, and other methods select and edit important tokens under different minimality or sparsity constraints.
  • Natural Language Explanation: Natural-language explanations generate textual accounts of model decisions and may improve prediction accuracy or augment data, although their reliability requires further investigation.They can be produced through separate explanation models, unlike chain-of-thought explanations produced by the LLM itself.

3.2 Global Explanation

Global explanations examine what language models encode beyond individual predictions, using probing, neuron analysis, concept-based methods, and related representation-focused approaches.

  • Global explanations study knowledge and linguistic properties encoded by neurons, hidden layers, and larger model modules.
  • Probing-Based Explanations: Probing methods analyze model representations and parameters to measure encoded syntactic and semantic knowledge.
  • Probing-Based Explanations: High probing performance does not establish that representations are interpretable, motivating selectivity constraints to mitigate probe bias.
  • Probing-Based Explanations: Prompt-based probing can reveal knowledge through generation or completion, but dataset regularities may obscure the factual knowledge captured by models.
  • Neuron Activation Explanation: Neuron analysis identifies individual dimensions associated with model performance or linguistic properties, often ranking a small subset because examining all neurons is expensive.
  • Neuron Activation Explanation: GPT-4-generated explanations for more than 1,000 GPT-2 XL neurons achieved high correlation with true activation and accounted for most of their behavior, though ground-truth annotations remain scarce.
  • Concept-Based Explanation: Concept-based methods map inputs to predefined concepts and quantify each concept’s importance for predictions, translating latent information into human-understandable explanations.

3.3 Making Use of Explanations

Explanations can diagnose model biases and limitations, and explanation regularization can connect model rationales with human rationales to improve reliability and task performance.

  • Debugging Models: Post-hoc explanations reveal feature-importance patterns that may indicate heuristic or biased behavior, such as context-insensitive attention to tokens.
  • Improving Models: AMPLIFY uses automated post-hoc rationales in prompts and improves LLM accuracy by 10-25% across various tasks, even without human rationales.
  • Improving Models: ER-TEST evaluates explanation-regularized models’ out-of-distribution generalization through unseen-dataset, contrast-set, and functional tests.

4 Explanation for Prompting Paradigm

Prompting-based models require explanation techniques suited to emergent abilities, scale, and complex reasoning; chain-of-thought and influential-example methods are emerging responses.

  • Prompting-based models’ reasoning abilities make localized or example-specific explanations less meaningful than in traditional fine-tuning settings.
  • At hundreds of billions of parameters or more, computationally demanding explanation techniques become infeasible and simplified surrogates cannot effectively capture intricate reasoning processes.
  • Chain-of-thought explanations and methods identifying influential examples are emerging to explain prompting-based model behavior.

4.1 Base Model Explanation

Base-model explanation research studies how prompting changes behavior, how models use representations, and how chain-of-thought affects input-token influence.

  • Explaining In-context Learning: Explainability studies examine how in-context learning influences the behavior of prompting-based language models.
  • Explaining In-context Learning: In SST-2 analyses, flipping labels reduces salience more often for GPT-2 but has the opposite impact for InstructGPT, with effects varying by model scale and task type.
  • Explaining In-context Learning: Large models can override semantic priors and learn contradictory input-label mappings from in-context examples, whereas small models rely more heavily on priors.
  • Explaining CoT Prompting: Chain-of-thought prompting makes models consider question tokens more stably according to gradient-based saliency analysis, potentially supporting more consistently accurate answers.
  • Representation Engineering: Representation engineering analyzes representation-space structure through representation reading and representation control to capture emergent representations and high-level cognitive phenomena.

4.2 Assistant Model Explanation

Assistant-model explanations examine how pre-training, alignment fine-tuning, and data-related limitations shape knowledge, instruction following, hallucinations, and uncertainty.

  • Assistant Model Explanation: Assistant models combine unsupervised pre-training for general language knowledge with supervised and reinforcement-learning alignment for tasks and user preferences.
  • Explaining the Role of Fine-tuning: 1,000 carefully selected instructions produced LLaMA-65B performance comparable to GPT-4 without reinforcement learning, highlighting pre-training and data quality over fine-tuning scale.
  • Explaining the Role of Fine-tuning: Imitation improves style, persona, and instruction following, but does not improve factuality, coding, or problem solving on more complex dimensions.
  • Explaining the Role of Fine-tuning: Pre-training appears to capture most model knowledge, while instruction fine-tuning activates that knowledge toward useful outputs and reinforcement learning further aligns models with human values.
  • Hallucination and Uncertainty: Hallucinations are linked to missing or repeated training data and to model reliance on sentence-level memorization and corpus-level statistical patterns rather than robust reasoning.
  • Hallucination and Uncertainty: Because logits are less suitable for especially closed-source LLMs, confidence elicitation estimates uncertainty through response consistency or verbal confidence statements.

4.3 Making Use of Explanations

The paper reviews ways to use explanations to improve prompting-based models, supervise smaller models, compress instructions, and support applied systems.

  • Improving LLMs: Chain-of-thought and other few-shot explanations can improve task performance, but benefits depend on model scale, explanation quality, and task-specific customization.
  • Improving LLMs: Explanation tuning trains smaller models with detailed step-by-step explanations from advanced models to provide richer supervision than input-output imitation.
  • Compressing Instructions: Ablation analysis identifies label-relevant instruction content as important for classification, enabling removal of unnecessary task-definition content.
  • Evaluating and Selecting Explanations: Intermediate computation steps significantly boost multi-step symbolic reasoning, whereas textual-reasoning gains are inconsistent across models.
  • Evaluating and Selecting Explanations: LLMs can generate unrealistic explanations that remain consistent with predictions, motivating explanation optimization to select explanations associated with higher performance.

5 Explanation Evaluation

Explanation evaluation distinguishes human plausibility from faithfulness to model reasoning, using quantitative tests while exposing inconsistencies and limits across explanation types.

  • Evaluation Dimensions: The survey evaluates explanations along two dimensions: plausibility to humans and faithfulness in capturing LLMs’ internal logic.
  • Evaluation Dimensions: Human judgments can assess plausibility but may neglect or misrepresent faithfulness, motivating automatic evaluations based on rationale perturbations.
  • Traditional Fine-tuning Paradigms: Local-explanation faithfulness is assessed through principles including implementation invariance, input invariance, sensitivity, completeness, polarity consistency, prediction consistency, and sufficiency.
  • Traditional Fine-tuning Paradigms: Comprehensiveness measures prediction-probability change after removing important tokens, while sufficiency measures whether extracted rationales alone support the prediction.
  • Traditional Fine-tuning Paradigms: Existing metrics can rank the same explanation model inconsistently, with the best DFFOT explanation potentially worst under SUFF.
  • Prompting Paradigms: Natural-language explanations require specialized tests because they lack a straightforward input relationship; counterfactual methods evaluate whether explanations reflect prediction-changing features.
  • Prompting Paradigms: Chain-of-thought explanations can be systematically unfaithful, and smaller models may produce more faithful explanations than larger, more capable models on some tasks.

6 Research Challenges

The survey identifies unresolved challenges in establishing explanation ground truth, understanding emergent abilities and shortcuts, comparing paradigms, and analyzing training dynamics and societal risks.

  • Explanation without Ground Truths: Ground-truth explanations are usually inaccessible, making explanation design, faithfulness evaluation, and method selection difficult.
  • Emergent Abilities: Proprietary models obscure the origins of emergent abilities because their architectures and training data are unpublished, while open models remain only partly interpretable.
  • Model Perspective: Key model-level questions concern which architectures produce emergent abilities and the minimum scale needed for strong performance across diverse language tasks.
  • Paradigm Differences: Fine-tuning and prompting can show markedly different in-distribution and out-of-distribution performance, yet comprehensive explanation comparisons between them remain lacking.
  • Shortcut Learning of LLMs: Both paradigms exploit shortcuts: fine-tuned models use dataset artifacts and biases, while prompted models perform worse when relevant information appears in the middle of long contexts.
  • Training Dynamics: Interpretability is shifting from static post-hoc analysis toward temporal analysis of training checkpoints to capture phase transitions and feature interactions.
  • Ethical Risks: Explainability can help audit misinformation, bias, and social manipulation risks by tracing training-data attribution or visualizing attention patterns.

7 Conclusions

The survey organizes explainability techniques for LLMs and identifies evaluation, interpretability, and model-specific adaptation as priorities for future work.

  • The paper provides a comprehensive overview of local and global explainability methods organized by LLM training paradigms.
  • It discusses how explanations can support model improvement and evaluates challenges in assessing their quality.
  • Future work should develop methods tailored to different LLMs, evaluate explanation faithfulness, and improve human interpretability.
  • The survey frames explainability as important for making advanced LLMs transparent, fair, and beneficial.
Loading 2309.01029v3…