PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-hoc Interpretability for Neural NLP: A Survey

Andreas Madsen, Siva Reddy, Sarath Chandar

arXiv:2108.04840v5cs.CLcs.LGcs.NE

TL;DR

Complex and widespread neural NLP models raise concerns about responsible use, while interpretability evaluation remains difficult and produces conflicting findings. This survey organizes post-hoc methods by their communication approaches, reviews how they are evaluated, and offers a broad overview with discussion of validation and future research.

  • Problem

    Neural NLP models are increasingly complex and difficult to assess, while interpretability methods lack clear quantitative evaluation and can yield conflicting findings.

  • Method

    The survey categorizes post-hoc interpretability methods by communication approach, reviews diverse methods, and discusses their evaluation.

  • Results

    The survey provides a diverse overview of post-hoc communication approaches and consistently examines how their explanations are validated.

  • Takeaways & Limitations

    Interpretability research should give dedicated attention to principled measurement for the integrity of the field.

  • Takeaways & Limitations

    The survey cannot provide unified quantitative comparisons because no unified, principled interpretability benchmark exists, and its visual examples are fictive and optimistic.

Abstract

from arXiv · show

Neural networks for NLP are becoming increasingly complex and widespread, and there is a growing concern if these models are responsible to use. Explaining models helps to address the safety and ethical concerns and is essential for accountability. Interpretability serves to provide these explanations in terms that are understandable to humans. Additionally, post-hoc methods provide explanations after a model is learned and are generally model-agnostic. This survey provides a categorization of how recent post-hoc interpretability methods communicate explanations to humans, it discusses each method in-depth, and how they are validated, as the latter is often a common concern.

1 INTRODUCTION

This survey organizes post-hoc interpretability methods by how they communicate explanations and by the information and abstraction involved. It covers diverse approaches, discusses their validation, and notes limitations of both the categorization and post-hoc explanations.

  • Complex, widespread neural NLP models are often black boxes, raising concerns about responsible deployment.
  • Interpretability presents model explanations in understandable terms, but effective explanations require selectively communicating causes to humans.
  • The survey categorizes communication approaches in Table 1 by increasing abstraction and the information used to produce explanations.
  • It provides a more diverse overview than surveys focused on a few popular approaches and consistently discusses how methods are validated.
  • Post-hoc methods explain trained models and are often model-agnostic, but they can be criticized for producing false explanations.
  • The categorization is approximate, omits some categories such as intermediate representations, and does not support unified quantitative comparisons.

2 MOTIVATIONS FOR INTERPRETABILITY

Interpretability is motivated by safety, ethics, accountability, and scientific understanding. The survey frames explanations around different questions about individual observations, classes, or the whole model.

  • Safety and ethics motivate interpretability because perfect optimization against all possible ethical issues and deployment failure modes is unlikely.
  • Ethical assessment may require qualitative human judgment because fairness metrics depend on a potentially incomplete list of protected attributes.
  • Adversarial examples and counterfactuals evaluate models outside the test distribution, supporting safety assessment under distributional shifts.
  • Accountability concerns explaining production failures, including contexts involving rights to explanation and model auditing.
  • Scientific understanding uses explanations to generate hypotheses, knowledge, and model-debugging insights.
  • The survey distinguishes local explanations for observations, class explanations for selected classes, and global explanations for the entire model.

4 MEASURES OF INTERPRETABILITY

Interpretability evaluation asks whether explanations reflect model behavior and help humans, but quantitative comparison remains unsettled. The survey distinguishes functional, human, and application-grounded evaluation and emphasizes combining these perspectives.

  • There is no consensus on measuring interpretability, while input-feature metrics can be especially challenging and even intrinsically interpretable methods may fail to reflect models.
  • Application-grounded evaluation tests explanations in deployment settings, such as medical survival, homework performance, or label correction, against human-provided explanations.
  • Application-grounded studies are rare in NLP because they are application-specific and time-consuming, so functional and human evaluation are often used together.
  • Human-grounded evaluation checks whether explanations are useful to people through tasks such as model selection, behavior prediction, or intruder identification.
  • Human evaluation is more efficient than application evaluation, but simulated users can produce overoptimistic results because human informativeness is non-trivial.
  • Functionally-grounded evaluation checks whether an explanation reflects the model, also known as faithfulness or fidelity.

5 METHODS OF INTERPRETABILITY

The survey organizes post-hoc interpretability methods by how they communicate explanations and by the information they use. It covers input features, gradients, LIME, and related approaches while consistently examining their validation and limitations.

  • The survey’s main objective is to categorize post-hoc interpretability methods by their communication approach and review how each method is evaluated.
  • Input features: Input-feature explanations identify how important tokens, sub-words, or characters are for a given prediction and may return an importance measure.
  • Input features: Input-feature explanations explain one scalar at a time, typically one class at one timestep, which can be repeated across sequence-to-sequence outputs.
  • Gradient: Gradient importance measures estimate output change from small input-feature changes, but vocabulary dimensions must be reduced for word-level visualization.
  • Gradient: Gradient explanations may miss important areas with zero gradients, and their functional grounding is task dependent for nonlinear NLP models.
  • Gradient: Integrated Gradients integrates gradients between an uninformative baseline and the observation to address sensitivity and completeness.
  • LIME: LIME fits a logistic regression to model outputs on nearby perturbed observations, using its parameters as importance scores and an L1 regularizer for sparse explanations.
  • LIME and SHAP: LIME and SHAP depend on input perturbations, allowing adversarial models to appear ethical under perturbation while behaving unethically without it.

7 ADVERSARIAL EXAMPLES

Adversarial examples explain predictions contrastively by finding similar inputs that change the model’s output while preserving the gold label. The survey discusses HotFlip and paraphrase-based search, alongside search and scope limitations.

  • Adversarial examples provide local, contrastive explanations by comparing an original example with another input that causes a wrong prediction.
  • An adversarial method maps an original input x to an adversarial example ˜x, ideally preserving similarity while changing the predicted label.
  • Functional grounding requires the predicted label to change while the gold label remains the same; similarity can be framed as paraphrasing.
  • Adversarial explanations do not generalize easily to sequence-to-sequence problems, although offensive-text classification can reduce such settings to sequence classification.
  • HotFlip: HotFlip uses input gradients to select token changes, and word-level applications constrain replacements to paraphrases with cosine similarity of at least 0.8.
  • SEA: SEA samples from a paraphrasing model and can change or delete multiple tokens simultaneously while requiring a semantic-equivalency score of at least 0.8.
  • HotFlip’s greedy sequential search misses combinatorial effects such as double-negation, while SEA depends on an expensive paraphrase-generation model.
  • Adversarial methods usually provide little control over search direction, potentially overlooking changes that are more interesting than the largest robustness error.

8 SIMILAR EXAMPLES

Influential-example methods explain predictions by identifying training observations that resemble an input and affect its loss or representation. The section covers influence functions, Representer Point Selection, and TracIn, while emphasizing limited NLP validation and practical constraints.

  • Influential examples identify training observations that resemble a specific input according to the model, making them local explanations.
  • Influence Functions: Influence functions estimate an observation’s importance from the loss change caused by removing it from training.Their computation relies on Hessian-based approximations and can be numerically unstable or expensive.
  • Representer Point Selection: Representer Point Selection decomposes a test logit into training-sample contributions and is more numerically stable than influence functions.It depends on the final-layer representation and a regularized last layer, but the stationary solution can be achieved post-hoc.
  • TracIn: TracIn accumulates training-time loss changes using gradients, minibatches, and checkpoints rather than requiring every optimization step.
  • Groundedness and Future Work: Functionally grounded validation is scarce in NLP: label-correction experiments are used, but they mainly test an observation’s influence on itself.The survey recommends information-removal experiments as a complementary evaluation.
  • Groundedness and Future Work: Future work should identify which parts of training examples or sequential outputs influenced a particular generation, especially for large language models.

9 COUNTERFACTUALS

Counterfactual explanations ask how an input must change to produce a different prediction, ideally through minimal and fluent edits. Polyjuice and MiCE illustrate different balances between model dependence, interpretability, and robustness-oriented generation.

  • Definition: Counterfactual explanations compare an input with a minimally edited, fluent example whose prediction differs, whereas adversarial examples retain the original gold label.
  • Definition: Counterfactual datasets are generated without using a model and therefore cannot by themselves explain model behavior.
  • Polyjuice: Polyjuice generates counterfactuals with GPT-2, then filters them for examples that change the model’s prediction most, adjusted by a SHAP importance measure.Its human-grounded validation found that people could not predict the model’s behavior on the generated examples.
  • MiCE: MiCE masks the most gradient-important consecutive tokens, inverts the gold label for T5 infilling, and uses beam search until the prediction changes.
  • MiCE: Using gradients significantly improves MiCE’s fluency and reduces the number of edits required to change a prediction compared with masking random tokens.
  • Groundedness: Counterfactual methods face a groundedness tension: opposite-label generation is model-independent, while search should respond to model behavior.Polyjuice is only weakly model-dependent, whereas MiCE provides a stronger example of combining the objectives.

10 NATURAL LANGUAGE

Natural-language explanations are designed to be directly understandable, but post-hoc rationalization methods may reflect human reasoning rather than the explained model’s behavior. The section therefore highlights a central groundedness concern.

  • Motivation: Natural-language explanations communicate a single example in readable prose, making them local explanations with relatively high accessibility.
  • Definition: Post-hoc natural-language explanations are called rationalization methods because they explain after a prediction, although the term can include false justifications.
  • CAGE: CAGE uses human-written explanations for Commonsense Question Answering and fine-tunes a separate GPT explainer-model on questions, answers, and explanations.
  • CAGE: CAGE can use generative explanations with sequence-valued answers, so it is not limited to sequence-to-class problems.
  • Groundedness: Because Mechanical Turk annotators lacked specialist insight into the explained model, CAGE’s explainer has little basis for reflecting that model’s behavior.
  • Groundedness and Future Work: Natural-language explanation research faces a recurring groundedness problem: explanation models may learn human thought processes rather than the model’s logical process.The survey also notes criticism of the claim that intrinsic self-reasoning explanations necessarily improve accuracy.

11 CONCEPTS

Concept explanations describe model behavior through abstractions such as occupations and gender associations rather than individual input tokens. Natural Indirect Effect analysis applies this idea to identify attention heads associated with gender bias, while groundedness and annotation costs remain open issues.

  • Definition: Concept explanations relate predictions to abstractions of the input, such as how occupation words relate to gendered classifications.
  • Scope: Concept explanations generally analyze one class or a small subset of classes, placing them within the survey’s class-explanation category.
  • Natural Indirect Effect: Natural Indirect Effect measures how a latent mediator, such as an attention head, affects the model output through an intervention on a concept.For gender bias, the intervention replaces an occupation word such as “nurse” with “man” or “woman.”
  • Natural Indirect Effect: Vig et al. apply Natural Indirect Effect to a small GPT-2 model to identify attention heads most responsible for gender bias involving occupation concepts.
  • Groundedness: Natural Indirect Effect lacks established human-groundedness and functionally-groundedness evaluations, and it is unclear how functional groundedness should be measured.
  • Future Work: Concept explanations may require new datasets or annotations, making exploratory use expensive and impractical when no specific concept is predefined.

12 VOCABULARY

Vocabulary explanations describe models through words or embedding spaces, using projections or rotations to expose semantic structure while retaining different amounts of information. Their validation commonly relies on word intrusion, but replacing human judgments with algorithms weakens that test.

  • Vocabulary explanations are global methods that relate the model to each word in its vocabulary.
  • Embedding-based vocabulary explanations can apply to sequence-to-class and sequence-to-sequence models, especially when using the embedding matrix.
  • PCA and t-SNE project fixed-dimensional embeddings into two or three dimensions, with t-SNE preserving local proximity through nonlinear transformations.
  • Supervised projections define directions around a subject of interest, such as gender bias, rather than relying only on unsupervised contrast.
  • Rotating embeddings can align new basis dimensions with significant concepts without losing information through dimensional projection.
  • Groundedness: The word intrusion test asks humans to identify one semantically unrelated word among six, but some work replaces this judgment with a distance ratio.The survey calls algorithmic replacement problematic because relatedness is defined by the same distance-based selection process.
  • Future work: Neural vocabulary-explanation research remains limited and largely embedding-based, although aggregating input-feature contributions could incorporate additional information.

13 ENSEMBLE

Ensemble explanations seek a selective global account by combining representative local explanations that capture different model modes. SP-LIME demonstrates human-grounded evaluation, but ensemble methods face computational, representational, and scalability constraints.

  • Ensemble explanations collect multiple local explanations so selected examples represent different modes of a model.
  • Because explaining every possible input overwhelms humans, ensemble methods strategically select representative examples and their local explanations.
  • The approach assumes that one or a few examples from each model mode can represent the model’s entire behavior.
  • Limitations: Few ensemble methods target non-tabular data, SP-LIME uses a Bag-of-Words proxy for NLP, and scaling becomes harder as datasets and model modes grow.
  • SP-LIME: SP-LIME selects a budget of observations whose LIME explanations collectively represent important features, using a greedy optimization objective.
  • Limitations: SP-LIME requires a LIME explanation for every observation, making it expensive; a proposed selection shortcut has not yet been applied to NLP.
  • SP-LIME: 89% of humans selected the better classifier with SP-LIME, compared with 75% using a random-observation baseline.
  • Groundedness: Ensemble functional groundedness depends on both the local explanation and the selection algorithm being validated.

14 LINGUISTIC INFORMATION

Linguistic-information explanations compare model behavior or representations with linguistic theory through behavioral and structural probes. The survey highlights evidence of heuristic reliance and probes’ communication complexity, while emphasizing insufficient groundedness evaluation.

  • Linguistic-information methods provide global explanations by aligning model behavior or latent representations with linguistic theory.
  • Behavioral probes modify inputs to observe model reactions, whereas structural probes align latent representations with linguistic representations.
  • BERTology: BERTology focuses on probing BERT-like models, including post-hoc analyses of intermediate representations alongside intrinsic attention-based explanations.
  • Behavioral probes: HANS tests lexical-overlap, subsequence, and constituent heuristics in NLI, and results support the hypothesis that models rely on these heuristics rather than content understanding.
  • Structural probes: Probing commonly trains a simple neural network to map intermediate representations to linguistic properties such as part of speech.
  • Structural probes: Probe accuracy can match a control task while the control requires higher Minimum Description Length, indicating greater communication complexity for the probe.
  • Groundedness: The survey finds insufficient evaluation of whether linguistic explanations are useful to humans and accurately reflect model functions.

15 RULES

Rule explanations summarize model behavior with simple rules, often by targeting a particular aspect such as adversarial failures or neuron activation. Their construction and validation remain difficult, and the survey emphasizes limited coverage and missing groundedness evidence.

  • Rule explanations use simple rules as global explanations, but simplifying a neural network generally requires restricting attention to one model aspect.
  • SEAR: SEAR extends adversarial-example sampling to generate rules describing changes that break sequence-to-class models.
  • SEAR: SEAR derives rules from word changes and adjacent-part-of-speech statistics, treating high flip-rate rules as semantically equivalent adversarial patterns.
  • SEAR: Expert evaluation found that SEAR-generated rules had higher success-rates than human-generated rules.
  • Compositional Explanations of Neurons: Compositional neuron explanations construct rules with beam search, scoring feature indicators and logical combinations by their IoU with neuron activation.
  • Groundedness: Mu and Andreas provide no groundedness validation, and the neuron–input relation does not establish how strongly the neuron affects the output.
  • Groundedness: Rule groundedness can be assessed by testing rules against datasets and model responses, while human groundedness can involve predicting outputs or choosing the better model.
  • Survey limitations: The survey is selective rather than exhaustive, and it lacks unified quantitative comparisons because no principled interpretability benchmark exists.

17 FINDINGS

The survey identifies challenges in terminology, measurement, explanation scope, and model structure, while highlighting connections among methods and directions for future research. It organizes post-hoc interpretability methods by communication approach and relates them to evaluation practices.

  • Terminology: Terminology varies across interpretability papers, with human-groundedness and functionally-groundedness sometimes confused and communication types inconsistently named.The survey does not seek to unify terminology, but aims to clarify which terms are equivalent or distinct.
  • Synergy: Methods from different communication approaches can share components, as HotFlip and MiCE both use gradients with respect to the input.This creates flexibility in method design, while criticisms of input-feature faithfulness may affect dependent methods.
  • Helpful complex models: Complex models such as GPT and T5 contribute to interpretability challenges but also generate fluent explanations in methods including Polyjuice, MiCE, and CAGE.The survey therefore characterizes complex models as neither exclusively harmful nor exclusively counterproductive to interpretability.
  • Measuring Interpretability: Interpretability measurement lacks standardization, because papers often introduce or modify their own human-groundedness and functionally-groundedness measures.The survey warns that this reduces comparability and can invalidate measures, while encouraging principled measurement research.
  • Measuring Interpretability: Functionally-groundedness is difficult to measure for input-feature explanations because humans cannot provide gold labels for explanations they would otherwise not need.Evaluation therefore relies on proxy measures and axioms, although principled proxies may still be developed.
  • Explanation scope: Class explanations remain underrepresented between local and global explanations, while sequence-to-sequence models receive less methodological attention than sequence-to-class models.The survey advocates more sequence-to-sequence explanations or evaluations because such models are widely used in applications such as machine translation.
Loading 2108.04840v5…