PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Faithful Model Explanation in NLP: A Survey

Qing Lyu, Marianna Apidianaki, Chris Callison-Burch

arXiv:2209.11326v4cs.CL

TL;DR

End-to-end neural NLP models are difficult to understand, while explanations that are plausible but unfaithful can mislead users. This survey reviews over 110 NLP explanation methods through faithfulness, synthesizes evaluation approaches and five method categories, and identifies progress alongside unresolved challenges.

  • Problem

    End-to-end neural NLP models are opaque, and unfaithful explanations can mislead users into over-trusting models despite unwanted biases.

  • Method

    The survey reviews over 110 NLP explanation methods through faithfulness, evaluating their approaches, strengths, shortcomings, and future directions.

  • Results

    The survey finds varied faithfulness across explanation families, with promising improvements for attention and later backpropagation variants but persistent evaluation and methodological challenges.

  • Takeaways & Limitations

    Faithfulness remains fundamental for explainability, but the field lacks a universally accepted technical definition and evaluation framework for comparing methods.

  • Takeaways & Limitations

    Many existing methods capture individual feature contributions but not higher-order feature interactions, motivating richer explanations such as feature subsets and causal graphs.

Abstract

from arXiv · show

End-to-end neural Natural Language Processing (NLP) models are notoriously difficult to understand. This has given rise to numerous efforts towards model explainability in recent years. One desideratum of model explanation is faithfulness, i.e. an explanation should accurately represent the reasoning process behind the model's prediction. In this survey, we review over 110 model explanation methods in NLP through the lens of faithfulness. We first discuss the definition and evaluation of faithfulness, as well as its significance for explainability. We then introduce recent advances in faithful explanation, grouping existing approaches into five categories: similarity-based methods, analysis of model-internal structures, backpropagation-based methods, counterfactual intervention, and self-explanatory models. For each category, we synthesize its representative studies, strengths, and weaknesses. Finally, we summarize their common virtues and remaining challenges, and reflect on future work directions towards faithful explainability in NLP.

1. Introduction

This survey reviews over 110 NLP model explanation methods through the lens of faithfulness, addressing how faithfulness should be defined, evaluated, and pursued. It organizes methods into five families and synthesizes their strengths, shortcomings, challenges, and future directions.

  • Scope and motivation: The survey examines over 110 model explanation methods in NLP, focusing on whether explanations accurately reflect models’ reasoning processes.Faithfulness is distinguished from plausibility and is important because plausible but unfaithful explanations can conceal unwanted biases.
  • Survey contributions: The authors introduce five method families: similarity-based, model-internal analysis, backpropagation-based, counterfactual intervention, and self-explanatory models.The taxonomy is intended to clarify previously ambiguous terminology.
  • Faithfulness evaluation: The survey clarifies faithfulness and critically reviews its varied evaluation methodologies rather than adopting a single evaluation standard.The authors note that no universally accepted standard currently exists.
  • Scope boundaries: The survey does not declare one method family categorically most faithful, instead comparing how families and individual methods confront faithfulness challenges.The authors emphasize diversity within each family as a reason to avoid sweeping judgments.
  • Implications and future work: The survey summarizes common virtues and remaining challenges, identifies future work directions, and aims to support method selection for practical use cases.It is presented as an introductory and comprehensive resource for students, researchers, and practitioners.

2. Explainability in NLP

The survey frames NLP explainability as presenting why a model makes predictions in understandable terms to a target audience. It distinguishes explaining encoded knowledge from explaining prediction causes, then reviews why explanations matter and how methods can be compared.

  • What versus why: NLP explainability separates “what” questions about encoded knowledge from “why” questions about factors that led to predictions.The survey focuses primarily on the causal “why” question rather than the already reviewed “what” question.
  • Methods and scope: Methods addressing encoded knowledge include probing classifiers, information-theoretic measurements, behavioral tests, and visualization.These approaches characterize what models encode, not necessarily what they use to produce task predictions.
  • Definition: Explainability concerns presenting why a model makes particular predictions in understandable terms to a specified human audience.The survey notes that explainability lacks an established technical definition and that interpretations of internal mechanics and human audiences vary.
  • Why explainability matters: Explainability can reveal dataset artifacts, diagnose model strengths and weaknesses, support debugging, and help calibrate trust in high-stakes applications.Examples include detecting shortcuts in NLI data, unwanted biases, perturbation vulnerabilities, and supporting evidence in computer-aided diagnosis.
  • Performance caveat: Interpretability and task performance can exhibit an empirical trade-off, although some studies report that explanations can also improve performance.The relationship is therefore not uniformly negative across tasks and models.
  • Properties and principles: The survey compares explanation methods across explicit properties to reduce confusion caused by taxonomies that mix distinct dimensions.It distinguishes, for example, saliency as a form and unit of explanation from a particular methodological family.
  • Properties and principles: The survey focuses on faithfulness while acknowledging that other explanatory principles lack universally established definitions, evaluation standards, or consensus about necessity.Faithfulness is treated as a central requirement and is revisited in later analysis.

3. Faithfulness

The survey treats faithfulness as a fundamental explanatory principle and examines its meaning, importance, relationships with other principles, and measurement.

  • Faithfulness: The survey analyzes faithfulness as a fundamental principle of model explanations, alongside its definition, importance, relationships, and evaluation.This focus prepares the analysis of explanation methods in later sections.

3.1 Definition

Faithfulness means that an explanation accurately reflects the reasoning process behind a model’s prediction. The survey emphasizes that this intuitive notion still lacks a consistent formal definition and is often specified through ad hoc evaluation metrics.

  • Definition: A faithful explanation accurately reflects the reasoning process behind the model’s prediction.The survey presents this as a loose description rather than a settled formal definition.
  • Definition: Faithfulness is often defined ad hoc through different evaluation metrics because the community lacks a consistent formal definition.These metrics are discussed as part of the survey’s evaluation review.

3.2 Importance

Faithfulness requires explanations to reveal the model’s actual reasoning process, not merely what the model knows or what humans find convincing. It matters because it establishes causal insight into predictions and can prevent plausible explanations from concealing harmful model behavior.

  • Faithfulness requires an explanation to accurately reflect the model’s underlying reasoning process.
  • Faithfulness helps distinguish features encoded by a model from features it actually uses to make predictions.Language models may encode linguistic features such as tense and number even when those features are irrelevant to an end-task label.
  • Unfaithful but plausible explanations can cause users to trust incorrect predictions because the explanation appears convincing.
  • Attention weights can deceive users by minimizing attention to gender-related tokens while hiding reliance on them.

3.3 Relationship between Faithfulness and Other Principles

Faithfulness and plausibility are distinct: an explanation can match human expectations without reflecting the model’s reasoning, or faithfully expose reasoning without being human-plausible. The survey therefore treats human alignment as insufficient evidence of faithfulness.

  • Faithfulness and plausibility can conflict because faithful explanations need not resemble human reasoning, while plausible explanations may ignore the model’s mechanism.
  • An implausible explanation may reflect either unusual model reasoning or an unfaithful explanation.
  • Human-aligned explanations are often mistaken for faithful explanations, although they may only be more plausible.
  • Input sensitivity, model sensitivity, and completeness are treated as necessary-condition checks that can disprove faithfulness through counterexamples.

3.4 Evaluation of Faithfulness

The survey finds no universally accepted or directly comparable faithfulness metric and reviews evaluation through multiple methodological lenses. It recommends combining complementary tests while accounting for their assumptions and confounds.

  • Faithfulness metrics are often ad hoc, non-comparable, and inconsistent, making objective progress difficult to assess.
  • The survey categorizes evaluation into axiomatic, predictive-power, robustness, perturbation-based, white-box, and human-perception methods.
  • Axiomatic evaluation: Axiomatic tests assess necessary conditions such as model sensitivity, input sensitivity, polarity consistency, robustness equivalence, and feature-importance agreement.Passing all axiomatic tests does not guarantee faithfulness, although passing more tests can increase confidence.
  • Predictive-power evaluation: Predictive-power evaluation measures whether an explanation enables prediction of the model’s decisions on unseen examples, but external simulators introduce label leakage and expressiveness concerns.Directly deriving predictions from explanations avoids an external simulator but makes large-scale, high-quality counterfactual generation difficult.
  • Predictive-power evaluation: Human simulation can conflate faithfulness with plausibility, because failures may reflect either an unfaithful explanation or an explanation humans cannot understand.
  • Robustness evaluation: Robustness evaluation is problematic when similar inputs and outputs may still arise from different reasoning mechanisms.

3.4.4 Perturbation-based Evaluation.

Perturbation-based evaluation tests explanations by removing or altering input features and observing prediction changes, but its interpretation depends on strong assumptions. The survey emphasizes distribution shift, feature dependence, and circularity as unresolved concerns in NLP.

  • Perturbation-based evaluation: Perturbation-based evaluation removes or alters input features and checks whether prediction changes match their explained importance.Removing highly important features should cause larger prediction changes than removing less important features; sufficiency and comprehensiveness are widely used metrics.
  • Assumptions and limitations: The method assumes that input features contribute independently, although correlated features can change one another’s effects when perturbed.
  • Assumptions and limitations: NLP perturbations can create ungrammatical, meaningless, or out-of-distribution inputs, making observed performance changes difficult to interpret.ROAR addresses distribution shift by removing features from both training and testing data before retraining, but no perfect NLP fix exists.
  • Assumptions and limitations: Perturbation-based metrics implicitly treat another perturbation-based explanation method as ground truth, creating a circular evaluation assumption.AOPC can reduce to leave-one-out under constraints on the number of features.
  • White-box evaluation: White-box evaluation compares explanations against known feature importance from transparent tasks or inherently interpretable models.Because these setups are simplified, passing the test does not guarantee faithful generalization to real-world scenarios.
  • Human-perception evaluation: Human-perception evaluation compares explanations with human judgments, but often measures plausibility unless the model is assumed to reason like humans.Human evaluation remains valuable for assessing whether target audiences understand models.
  • Meta-evaluation: Meta-evaluation can assess whether metrics detect known unfaithful explanations, are algorithmically solvable, and require few model forward passes.
  • Recommendations: The survey calls for faithfulness to be defined before evaluation, assumptions to be stated, and model capacity to be separated from explanation quality.

4. Families of Model Explanation Methods

The survey groups faithful NLP explanation methods into five families, each offering a different way to relate model predictions to examples, internal structures, feature relevance, interventions, or generated rationales.

  • The five families are similarity-based methods, model-internal structure analysis, backpropagation-based methods, counterfactual intervention, and self-explanatory models.
  • Similarity-based methods explain predictions through similar training examples or concepts in the model’s representation space.
  • Model-internal analysis examines activations and mechanisms such as attention, using visualization, clustering, or correlation analysis.
  • Backpropagation-based methods assign feature importance from gradients or related backward computations, often highlighting tokens with large relevance values.
  • Counterfactual intervention perturbs one feature while controlling others and observes the resulting change in the prediction.
  • Self-explanatory models produce explanations during inference rather than applying a separate post-hoc explanation method.

4.1 Similarity-based Methods

Similarity-based methods explain predictions by retrieving similar training examples in the model’s learned representation space. They are intuitive and model-agnostic, but their faithfulness and coverage remain uncertain.

  • Similarity-based explanations retrieve the most similar training examples to support a test prediction in the learned representation space.
  • Case-based explanation uses hidden activations to find a test example’s k-nearest neighbors, with similarity defined in representation rather than input space.
  • Wallace, Feng, and Boyd-Graber replace the final softmax classifier with a kNN classifier at test time while keeping trained model parameters fixed.
  • Strengths: These methods are intuitive, easy to implement, and model-agnostic because similarity scores can be computed from model representations.
  • Strengths: Human subjects rated some similarity-based explanations as more understandable and trustworthy than several backpropagation and counterfactual baselines.
  • Weaknesses: Most methods reveal which examples are similar but not how the representation space was learned, making causal faithfulness difficult to establish.
  • Weaknesses: Because explanations are confined to training data, their diversity and reliability depend on how well that data represents the broader distribution.

4.2 Analysis of Model-Internal Structures

Model-internal analysis studies neurons and attention to expose internal patterns, but raw attention weights do not reliably represent causal contribution. Regularization and alternative analyses seek to improve attention faithfulness.

  • Analysis approach: Internal-structure analysis uses visualization, clustering, and correlation analysis to study neurons, layers, attention, and other model mechanisms.
  • Analysis on Neurons: Neuron studies identify units that activate for patterns such as quotation marks, conditional statements, line endings, negation, intensification, and adjective-noun composition.
  • Analysis on Attention Mechanism: Attention weights encode weighted interactions among transformed input representations, which can make them appear to indicate token importance.
  • Analysis on Attention Mechanism: Adversarial attention distributions can differ substantially from observed weights while preserving the model prediction, indicating that attention does not always causally influence outputs.
  • Faithfulness improvements: Weight tying and an auxiliary masked-language-modeling task make hidden states more representative of inputs and increase the causal influence of attention modification on predictions.
  • Faithfulness improvements: Attention Flow can equal Shapley values under certain conditions, whereas raw attention cannot, but Attention Rollout performs almost as poorly as raw attention in polarity consistency.
  • Strengths: Attention can be visually intuitive, support interactive hypothesis formation, and capture interactions between features.
  • Weaknesses: Raw attention may be unfaithful because intermediate hidden states mix contextual information before attention weights are interpreted as input-feature importance.

4.3 Backpropagation-based Methods

Backpropagation-based methods propagate prediction relevance backward to input features through gradients or customized redistribution rules. They are relatively efficient and can reflect the full computation path, but remain limited by faithfulness tests and explanation scope.

  • Overview: Backpropagation-based methods attribute output relevance to input features through gradient methods or propagation methods.
  • Gradient methods: Gradient methods treat the gradient of the output with respect to each feature as relative importance, with sign and magnitude indicating direction and influence.
  • Gradient methods: Gradient×Input incorporates feature values into relevance scores, but improves plausibility without necessarily improving faithfulness and fails Input Sensitivity.
  • Gradient methods: Integrated Gradients compares an input with a baseline, satisfying Input Sensitivity while still showing empirically observed visual noise.
  • Propagation methods: Propagation methods redistribute relevance through network layers, using customized backward rules to control how importance reaches input features.
  • Propagation methods: DeepLift satisfies Input Sensitivity and outperformed Integrated Gradients, Gradient×Input, Simple Gradients, and Guided BackPropagation in a pixel-flipping test, but fails Implementation Invariance.
  • Strengths: Backpropagation methods are generally efficient, and some variants show greater faithfulness than earlier baselines under perturbation-based evaluations.
  • Weaknesses: Most methods target low-level features, leaving their application to higher-level linguistic features and non-classification tasks unclear.

4.4 Counterfactual Intervention

Counterfactual intervention methods explain predictions by manipulating input features, examples, or model representations and observing prediction changes. They can target causal effects, but their faithfulness is limited by computational cost, out-of-distribution interventions, and imperfect feature erasure or counterfactual construction.

  • 4.4.1 Input Intervention: Counterfactual methods intervene on inputs or model representations to estimate how features, examples, or internal factors affect predictions.Input interventions manipulate the original input, whereas representation interventions directly modify neurons or layers.
  • 4.4.1 Input Intervention: Input interventions target features or examples and use erasure, such as masking or deletion, or perturbation, such as changing the target value.Leave-one-out methods erase one feature at a time and assess the resulting prediction change.
  • 4.4.1 Input Intervention: Surrogate methods such as LIME locally approximate a black-box model with an interpretable model learned from nearby masked instances and weighted predictions.LIME uses an interpretable model, such as sparse linear regression, to approximate the local decision boundary.
  • 4.4.1 Input Intervention: LIME recovers 90% of important features in white-box tests, but its faithfulness depends on neighborhood choice and surrogate expressivity.A linear surrogate cannot faithfully approximate some nonlinear local decision boundaries, such as a circular boundary around an interior example.
  • 4.4.2 Model Representation Intervention: Representation erasure may be post-hoc or adversarially trained, but linear guardedness does not guarantee feature removal and erasure can destroy task-relevant information.Post-hoc methods remove features after representation training; adversarial methods jointly train the model with an adversarial predictor.
  • 4.4.3 Strengths and Weaknesses: Counterfactual interventions are designed to capture causal rather than merely correlational effects and are often evaluated explicitly for faithfulness, but they require multiple forward passes and can yield nonsensical inputs.High-quality counterfactuals must match the target label, remain semantically coherent, and differ only in the intended feature; manual creation remains the most reliable yet expensive approach.

4.5 Self-Explanatory Models

Self-explanatory models provide explanations through transparent architectures or explicit explanation generation alongside prediction. They offer flexible, potentially supervised explanations, but architectural transparency and explanation faithfulness can still be undermined by learned intermediate behavior or predictor–explainer dependencies.

  • 4.5 Self-Explanatory Models: Self-explanatory models provide built-in explanations through transparent architectures or explicit explanations generated during inference.Explanations may take the form of feature scores, natural language, causal graphs, or the network architecture itself.
  • 4.5.1 Explainable Architecture: Neural Module Networks parse questions into programs of learnable modules that execute on context to derive answers, making intermediate reasoning steps explicit.Their potentially learnable components include the question parser, network layout predictor, and module parameters.
  • 4.5.1 Explainable Architecture: On DROP, Gupta et al.’s Neural Module Network supports symbolic operations such as counting, finding maximum numbers, and comparing numbers.Earlier language-only NMNs for HotpotQA could answer directly retrievable questions but lacked symbolic reasoning capabilities.
  • 4.5.1 Explainable Architecture: NMN modules may not faithfully implement their intended functions because their parameters are typically learned only from end-to-end supervision on final answers.Pretraining modules independently on synthetic tasks and fixing their parameters could help ensure faithfulness and transfer knowledge.
  • 4.5.2 Generating Explanations: Models that generate explanations include predict-then-explain, explain-then-predict, and jointly-predict-and-explain frameworks.Predict-then-explain uses an independent post-hoc explainer, whereas explain-then-predict restricts the predictor to the generated explanation.
  • 4.5.2 Generating Explanations: Explain-then-predict is intended to be faithful by construction, but natural-language explanations can trade off task accuracy and plausibility, become self-inconsistent, or contain label cues.NILE addresses label-specific cues by structuring natural-language inference around explanations associated with each class.
  • 4.5 Self-Explanatory Models: Self-explanatory models avoid separate post-hoc explanations, support flexible explanation forms, and can use human explanations to encourage plausible or desired reasoning mechanisms.Their interpretability can still involve a trade-off with performance, especially in models with architectural constraints.

5. Summary and Discussion

The survey compares five explanation families through theoretical and empirical faithfulness, identifying distinct assumptions, failure modes, and progress. It also highlights unresolved evaluation, feature-level, task-format, actionability, and performance–interpretability challenges.

  • Similarity-based methods: Similarity-based methods assume similar representations imply similar reasoning, but this assumption can fail under subtle representation changes and is rarely tested for faithfulness.
  • Analysis of model-internal structures: Raw attention explanations face faithfulness concerns from information mixing, locality, and lack of causality, although regularization and hybrid approaches show promise.
  • Backpropagation-based methods: Backpropagation methods are faithfulness-motivated but face saturation, input sensitivity, implementation variance, and evidence of partial input recovery.
  • Counterfactual intervention: Counterfactual methods draw on causality but face feature-independence and hindsight-bias concerns; perturbations may be more faithful than erasures because erasures can create OOD inputs.
  • Self-explanatory models: Self-explanatory models may lack intermediate supervision and suffer label leakage or social misalignment, weakening claims of faithfulness by construction.
  • Challenges and future work: Faithfulness lacks consensus measurement, while explanations often remain surface-level, omit higher-order interactions, target limited tasks, and offer unclear paths for user action.The survey also notes that interpretability and performance can be synergistic or tense, and that explanations may increase over-trust without improving decisions.

6. Conclusion

The survey examines NLP explanation methods through faithfulness, which lacks a universally accepted definition and evaluation framework. It organizes methods into five families, compares their strengths and weaknesses, and proposes priorities for more standardized and capable interpretability research.

  • Faithfulness has no universally accepted technical definition or evaluation framework, making method comparison difficult and leaving many methods without quantitative faithfulness results.
  • The survey critically reviews five explanation families, presenting representative work, strengths, and weaknesses with special attention to faithfulness.
  • Similarity-based methods are rarely assessed for faithfulness, while attention, gradient, counterfactual, and self-explanatory approaches each retain distinct theoretical or practical concerns.
  • Future work should establish universal faithfulness standards, study interpretability–performance relationships, and support high-level features, flexible explanations, and alternative task formats.

Appendix A: Additional Details

The appendix supplements the survey with formal summaries, visualizations, and implementation resources for explanation-method families. It includes tables for gradient and propagation methods, figures, and tools spanning neuron, attention, gradient, and counterfactual analysis.

  • Technical Details: Table 1 summarizes gradient methods by how they compute r_i(x), the relevance of feature x_i.
  • Tools: The appendix lists tools for neuron and attention visualization, backpropagation-based explanations, and counterfactual intervention.
  • Visualizations: Figure 1 visualizes different gradient methods on image-classification examples, where brighter shades indicate higher feature relevance for the prediction.
  • Technical Details: Table 2 summarizes propagation methods by how they define the recursive function D(), with Simple Gradients included for comparison.
Loading 2209.11326v4…