Towards Faithful Model Explanation in NLP: A Survey

Motivation

• End-to-end neural language models achieve broad NLP success but remain black boxes, hindering trustworthy use and leaving explanation quality, selection, and performance trade-offs unresolved.These concerns are especially important in high-stakes scenarios where user trust matters.Source: §1 · p. 1

Why Faithfulness

• Faithfulness requires explanations to reflect models’ reasoning rather than merely appear plausible, because unfaithful explanations can conceal reliance on sensitive features and induce over-trust.Such risks have been demonstrated for recidivism, college acceptance, and credit prediction, including reliance on race and gender (Pruthi et al. 2020; Slack et al. 2020).Source: §1 · pp. 1–2

Faithfulness Evaluation

• Faithfulness is often conflated with plausibility, lacks a quantitative consensus, and produces incompatible findings when studies rely on isolated methods with problematic assumptions.The survey therefore treats faithfulness as a central principle requiring critical evaluation rather than a single accepted score.Source: §1 · p. 2

Organization and Outlook

• The survey concludes by synthesizing common virtues and challenges and identifying future directions for improving faithfulness in explainable NLP.Its taxonomy and faithfulness-focused review aim to support a more transparent and standardized interpretability field.Source: §1 · pp. 2–3

Contributions

• The survey reviews over 110 explanation methods, introduces five families, and critically examines faithfulness concepts and evaluation methodologies.The families are similarity-based methods, analysis of model-internal structures, backpropagation-based methods, counterfactual intervention, and self-explanatory models.Source: §1 · p. 3

Scope and Limitations

• Rather than empirically evaluating every method or declaring one family universally most faithful, the survey compares motivations, assumptions, constraints, and within-family challenges.It recommends methods that are intrinsically motivated and empirically validated for faithfulness while recognizing potential pitfalls.Source: §1 · p. 3

2.1 What Is Explainability

• NLP explainability concerns presenting why a model makes particular predictions in understandable terms to a specified target audience.The survey distinguishes this focus from explaining what knowledge a model encodes.Source: §2.1 · pp. 5–6

2.2 Why Is Explainability Important

• Explainability is important because neural NLP models are opaque, may exploit dataset artifacts or unwanted biases, and require diagnosis before deployment and calibrated user trust.The section also notes that explainability can trade off with task performance in some settings but improve performance in others, particularly under low-resource conditions.Source: §2.2 · pp. 6–7

2.3 Properties of Explanations

• Explanation methods are categorized by when they operate, model access, scope, explanatory unit and form, while target audience captures differing explanation goals.This multidimensional taxonomy avoids conflating properties such as example-based, local, global, and saliency-based explanations.Source: §2.3 · pp. 7–8

2.4 Principles of Explanations

• Faithfulness requires an explanation to accurately reflect the reasoning behind a prediction, whereas plausibility concerns whether the explanation is understandable and convincing to its audience.The survey treats faithfulness as more central because plausible explanations can appear convincing while misrepresenting the model’s causal reasoning, such as interpreting attention weights as feature importance without evidence of causality.Source: §2.4 · p. 9
• Other proposed principles include input and model sensitivity, completeness, and minimality, but their definitions and evaluation standards lack universal consensus.The survey therefore focuses primarily on faithfulness while referring to the other principles when relevant.Source: §2.4 · pp. 9–10

• Faithfulness is analyzed in terms of its definition, significance, connections to other explanatory principles, and evaluation methods.Source: §3 · p. 10

Definition

• Faithfulness broadly requires explanations to reflect the reasoning process behind model predictions (Harrington et al. 1985; Ribeiro, Singh, and Guestrin 2016; Jacovi and Goldberg 2020), yet no consistent formal definition exists.Researchers instead commonly define faithfulness ad hoc through different evaluation metrics.Source: §3.1 · p. 11

• Faithfulness is fundamental to explainability because an explanation must accurately represent the model’s underlying reasoning process; otherwise, it does not satisfy the definition of an explanation [Jain and Wallace 2019; Bastings and Filippova 2020; Jacovi and Goldberg 2020].Source: §3.2 · p. 11
• Faithful explanations establish causality by distinguishing features encoded by a model from those actually used for prediction, which correlation-based analyses cannot determine (Ravichander, Belinkov, and Hovy 2021).Language models can encode tense and number even when these features are irrelevant to an artificial task’s label.Source: §3.2 · p. 11
• Unfaithful but plausible explanations can dangerously increase user trust in incorrect or biased predictions, as deceptive attention weights can conceal reliance on gender-related tokens (Pruthi et al. 2020; Bansal et al. 2021).Source: §3.2 · p. 11

Faithfulness vs. Plausibility

• Plausibility does not guarantee faithfulness, and faithfulness does not guarantee plausibility because the two principles assess different relationships.A faithful explanation may be incomprehensible to humans, while a plausible explanation may reflect human reasoning rather than the model’s reasoning.Source: §3.3 · pp. 11–12
• An implausible explanation may indicate either that the model uses non-human reasoning or that the explanation fails to faithfully capture the model’s reasoning.For example, reliance on function words rather than sentiment words could reflect spurious model behavior or an unfaithful explanation.Source: §3.3 · p. 12

Faithfulness vs. Input Sensitivity, Model Sensitivity, and Completeness

• Input sensitivity, model sensitivity, and completeness are often treated as necessary faithfulness conditions and used as sanity checks, where violating one can establish unfaithfulness.These principles are frequently applied through counterexamples, although their relationship to faithfulness is not always stated explicitly (Sundararajan, Taly, and Yan 2017; Kindermans et al. 2019; Yeh et al. 2019; Adebayo et al. 2018).Source: §3.3 · p. 12

3.4 Evaluation of Faithfulness

• Faithfulness evaluation lacks an established formal definition, while existing metrics are often incomparable and produce inconsistent results.The survey therefore reviews axiomatic, predictive-power, robustness, perturbation-based, white-box, and human-perception evaluation methods.Source: §3.4 · p. 12
• Evaluation should explicitly target faithfulness rather than plausibility, avoid human judgments of explanation quality, and require no human-provided gold labels.A faithful method should explain any model prediction, whether correct or incorrect.Source: §3.4 · p. 12

3.4.1 Axiomatic Evaluation.

• Axiomatic evaluation tests necessary conditions such as model sensitivity, input sensitivity, polarity consistency, robustness equivalence, and feature-importance agreement, but passing them cannot establish faithfulness.These tests can disprove faithfulness through counterexamples, while more passed tests increase confidence only practically.Source: §3.4.1 · pp. 13–14
• Implementation Invariance is not a necessary faithfulness condition because functionally equivalent models can use different reasoning mechanisms despite identical outputs.The survey considers sensitivity to meaningful model changes, such as randomized weights, a more sensible model-sensitivity check.Source: §3.4.1 · p. 13

3.4.2 Predictive Power Evaluation.

• Predictive-power evaluation estimates faithfulness by how accurately an explanation supports prediction of the model’s decisions on unseen examples, or its simulatability.Predictions may be derived directly from executable explanations or rules, or through external models and human simulators.Source: §3.4.2 · p. 14
• External-simulator evaluations risk label leakage and simulator-expressivity confounds, while human simulators can conflate explanation plausibility with faithfulness.The survey regards direct prediction from explanations as most rigorous, but notes that generating large-scale, high-quality counterfactual examples is difficult; leakage-adjusted and conditional-information alternatives are proposed.Source: §3.4.2 · pp. 15–16

3.4.3 Robustness Evaluation.

• Robustness evaluation requires similar explanations for similar inputs with similar model outputs, but NLP’s discrete inputs and potentially non-robust reasoning undermine this criterion.Semantically similar inputs may induce different model features or reasoning mechanisms, so unstable explanations cannot distinguish unfaithfulness from genuinely unstable model reasoning (Ju et al. 2022).Source: §3.4.3 · p. 16
• The survey does not recommend robustness as a faithfulness metric, although stable explanations may still improve user understandability (Zhou, Ribeiro, and Shah 2022).Robustness is therefore potentially useful for comprehensibility even when it is not diagnostically reliable for faithfulness.Source: §3.4.3 · p. 16

3.4.4 Perturbation-based Evaluation.

• Perturbation-based evaluation removes or alters input features, expecting important-feature removal to change predictions more than unimportant-feature removal, often measured by sufficiency and comprehensiveness.Its validity depends on feature independence and meaningful perturbed inputs; NLP perturbations can create out-of-distribution or nonsensical sentences, while embedding perturbations introduce non-uniform neighborhood-density concerns.Source: §3.4.4 · pp. 16–17
• Perturbation-based evaluation also implicitly treats another perturbation-based explanation as ground truth, and no perfect NLP fix currently resolves these assumptions.ROAR addresses distribution shift by retraining after feature removal, but changes the model being explained; AOPC can reduce to leave-one-out under feature-count constraints.Source: §3.4.4 · p. 17

3.4.5 White-box Evaluation.

• White-box evaluation compares candidate explanations with known ground-truth explanations from transparent tasks or inherently interpretable models.Synthetic or modified datasets control informative features, while models such as Logistic Regression and Decision Tree expose feature importance through weights or prediction structure.Source: §3.4.5 · p. 18
• White-box tests are necessary sanity checks but do not guarantee faithful explanations in real-world settings because their simplified setups may not generalize.Source: §3.4.5 · p. 18

3.4.6 Human Perception Evaluation.

• Human perception evaluation measures alignment between explanations and human judgments, but generally assesses plausibility rather than faithfulness unless model and human reasoning are assumed identical.Despite this limitation, human evaluation remains valuable because explanations ultimately aim to improve the target audience’s understanding of models.Source: §3.4.6 · p. 18

3.4.7 Meta-evaluation of Faithfulness Evaluation Metrics.

• Faithfulness metrics can be meta-evaluated by detecting known unfaithful explanations, testing solvability, and measuring computational cost through model forward passes.Known unfaithful explanations are often random or artificial, but selecting relatively faithful comparisons is itself difficult; sufficiency and comprehensiveness are solvable with beam search.Source: §3.4.7 · p. 19

3.4.8 Summary.

• The authors recommend axiomatic, predictive-power, and white-box evaluations, preferably combined because many tests establish only necessary conditions for faithfulness.They also call for explicit definitions, transparent assumptions and implications, and caution against conflating plausibility with faithfulness.Source: §3.4.8 · p. 19
• Evaluation should disentangle model capacity from explanation quality, while metric quality itself remains under-studied and requires further systematic comparison.For example, non-robust explanations may reflect inconsistent model features or unfaithful explanations rather than a single cause.Source: §3.4.8 · pp. 19–20

Overview

• The five families are similarity-based methods, model-internal analysis, backpropagation-based methods, counterfactual intervention, and self-explanatory models.The survey introduces each family through its core intuition before discussing representative studies, strengths, and weaknesses.Source: §4 · p. 20

Similarity-based methods

• Similarity-based methods explain predictions by identifying similar training examples or concepts in the model’s induced representation space.They assume that similar examples share intrinsically similar reasoning mechanisms.Source: §4 · p. 20

Analysis of model-internal structures

• Model-internal analysis explains predictions by examining activations in nodes, layers, or mechanisms such as attention (Bahdanau, Cho, and Bengio 2015).Visualization, clustering, and correlation analysis can identify highly weighted tokens interpreted as contributing most to the prediction.Source: §4 · p. 20

Backpropagation-based methods

• Backpropagation-based methods rank features by the absolute gradient of the prediction, treating highly sensitive features as most important.The intuition is that small feature changes could substantially alter the model output; for discrete tokens, change extent uses embedding-space similarity metrics.Source: §4 · p. 20

Counterfactual intervention

• Counterfactual intervention perturbs one feature while controlling others and measures the resulting change in the model prediction.Masking or replacing a token and observing a considerable probability decrease can indicate that the token was important.Source: §4 · p. 20

Self-explanatory models

• Self-explanatory models generate explanations during inference rather than applying post-hoc methods.They can jointly predict a label and produce a natural-language justification for that prediction.Source: §4 · p. 21

4.1 Similarity-based Methods

• Similarity-based methods explain a test prediction by identifying the most similar training examples in the model’s learned representation space, typically using kNN.Case-based explanation formalizes this approach using hidden activations, while other methods replace the final classifier with kNN or retrieve similar concepts.Source: §4.1 · p. 21

4.1.1 Strengths and Weaknesses.

• These methods are intuitive, easy to implement without retraining or data manipulation, and model-agnostic because neural networks provide representation spaces to which common similarity metrics apply.Some similarity-based explanations were judged more understandable and trustworthy than several backpropagation-based and counterfactual baselines (Rajagopal et al. 2021).Source: §4.1.1 · pp. 21–22
• Their explanations usually reveal which examples are similar rather than how the model learned its representation space, and existing evaluations largely measure plausibility rather than faithfulness or causal support.For example, Wallace, Feng, and Boyd-Graber (2018) assessed alignment with human feature-importance judgments, while Rajagopal et al. (2021) mainly evaluated human-rated plausibility and reported only relative human-simulatability differences.Source: §4.1 · p. 21, §4.1.1 · p. 22
• Because they search only the training set and provide instance-level explanations, similarity-based methods can depend on biased or unrepresentative data, miss edge cases, and omit actionable feature-level contributions.Their explanation outcomes may vary with the training distribution, while limited granularity constrains insight into individual feature contributions.Source: §4.1.1 · p. 22

4.2 Analysis of Model-Internal Structures.

• Model-internal analysis commonly uses visualization, clustering, and correlation to study individual neurons or mechanisms such as convolution and attention (Vig 2019; Brunner et al. 2018; Qian, Qiu, and Huang 2016).The survey emphasizes attention because it underlies widely adopted NLP architectures.Source: §4.2 · p. 22

4.2.1 Analysis on Neurons.

• Neuron studies identify units responding to quotes, control structures, negation, intensification, adjective-noun composition, and orthographic or grammatical patterns (Karpathy, Johnson, and Fei-Fei 2015; Li et al. 2016; Strobelt et al. 2018).Reverse analyses also group inputs with similar neuron activations, revealing patterns captured by GRU and LSTM models.Source: §4.2.1 · pp. 22–23, Figure 2 · p. 23

4.2.2 Analysis on Attention Mechanism.

• Attention weights can be quite dissimilar yet yield effectively the same prediction, conceal reliance on gender biases, and, in machine translation, encoder-decoder attention modifications substantially influence generations.Source: Figure 4 · p. 24, §4.2.2 · p. 25
• Attention-faithfulness remedies address information mixing, locality, and causality through regularization, global-flow methods, and links to backpropagation-based explanations.Weight tying and auxiliary masked-language modeling increase causal influence, while Attention Flow has theoretical Shapley-value connections but Attention Rollout performs nearly as poorly as raw attention in polarity consistency.Source: §4.2.2 · pp. 25–26

4.2.3 Strengths and Weaknesses.

• Model-internal analyses are intuitive, interactive, computationally efficient, and able to represent feature interactions, but raw attention may not reflect causal contribution or global computation.Intermediate hidden states mix contextual information, while single-layer or single-position analyses omit broader information flow.Source: §4.2.2 · p. 25, §4.2.3 · p. 26

4.3 Backpropagation-based Methods

• Gradient methods use output derivatives as feature importance, whereas propagation methods recursively redistribute relevance with custom rules for layers such as ReLU.Gradients measure local responsiveness, while propagation methods aim to capture relevance redistribution through special layers.Source: §4.3 · p. 27, §4.3.1 · p. 27, §4.3.2 · p. 29

4.3.2 Propagation methods.

• LRP produced faster performance drops than Simple Gradients and a random baseline when deleting relevant tokens from correctly classified inputs, indicating more faithful reasoning insights.For incorrectly classified inputs, deleting irrelevant tokens with LRP most effectively improved performance.Source: §4.3.2 · p. 31
• DeepLift most effectively manipulated predictions toward a target class in pixel-flipping tests, outperforming Integrated Gradients, Gradient×Input, Simple Gradients, and Guided BackPropagation.DeepLift avoids saturation and satisfies Input Sensitivity but still fails Implementation Invariance; DTD’s faithfulness remains unverified.Source: §4.3.2 · p. 31

4.3.3 Strengths and Weaknesses.

• These methods are relatively efficient and computation-path-aware, but often target low-level features, struggle with non-classification outputs, violate faithfulness axioms, and can be unstable or adversarially manipulated.Guided BackPropagation and DeconvNet can perform partial input recovery unrelated to model decisions, while some gradient methods admit manipulated attribution distributions.Source: §4.3.3 · pp. 31–32

4.4 Counterfactual Intervention

• Counterfactual intervention estimates feature or representation importance from prediction changes after erasure or perturbation, with interventions occurring either on inputs or inside model representations.Input interventions manipulate tokens or examples and pass them through the original model, whereas representation interventions directly modify neurons or higher-level features.Source: §4.4 · pp. 32–33

4.4.1 Input Intervention

• Input interventions target tokens, examples, or feature interactions through erasure or perturbation, but their faithfulness depends on approximation quality and the validity of generated counterfactual inputs.LIME locally fits an interpretable surrogate from masked neighbors, recovering 90% of important features in white-box tests; however, surrogate neighborhoods and linear expressivity are critical limitations, while perturbations are difficult to make coherent and minimally changed.Source: §4.4.1 · pp. 33–35

4.4.2 Model Representation Intervention

• Model-representation interventions test whether neurons or high-level features affect predictions, finding evidence for syntax use and concentrated gender-bias neurons but limited empirical faithfulness validation.AlterRep counterfactually increases the probability ratio of are versus is in subject-verb agreement, while representation-erasure methods may fail to remove the target feature or destroy task-relevant features.Source: §4.4.2 · pp. 36–37

4.4.3 Strengths and Weaknesses

• Counterfactual methods capture causal rather than merely correlational effects and are often explicitly evaluated for faithfulness, but they are costly, vulnerable to OOD manipulations, miss interactions, and may suffer hindsight bias.Single-feature erasure can miss relationships such as the OR interaction between mediocre and bad, while dropping a feature without changing the prediction does not establish that the model originally ignored it (De Cao et al. 2020).Source: §4.4.3 · p. 38

4.5 Self-Explanatory Models

• Self-explanatory models replace post-hoc explanations with built-in feature scores, natural language, causal graphs, or transparent architectures.They divide into explainable architectures and models that generate explicit explanations as part of inference.Source: §4.5 · p. 39

4.5.1 Explainable Architecture.

• Neural Module Networks compose learnable modules into question-specific programs, extending from fixed components toward jointly learned parsers, layouts, and parameters.They support visual and language-only question answering, including symbolic reasoning on DROP through dedicated operation modules.Source: §4.5.1 · pp. 39–40
• NMNs can be unfaithful when learned modules ignore their intended functions, while symbolic modules struggle with flexible semantics; intermediate supervision improves faithfulness but may reduce accuracy.Independent pretraining with fixed module parameters offers a route to more faithful and transferable compositional reasoning.Source: §4.5.1 · p. 40

4.5.2 Generating Explanations.

• Generating-explanation models include predict-then-explain, explain-then-predict, and jointly-predict-and-explain frameworks, whose faithfulness depends on whether predictions causally rely on explanations.Explain-then-predict is faithful by construction because the predictor receives only the generated explanation, unlike frameworks allowing access to the original input.Source: §4.5.2 · p. 41, Figure 9 · p. 41
• Rationales and free-text explanations remain vulnerable to superficial cues, instability, inconsistency, label leakage, and task-specific insufficiency, despite restricting prediction to generated explanations.NILE generates label-specific explanations and achieves comparable SNLI accuracy with better out-of-distribution transferability, while comparing faithfulness across variants.Source: §4.5.2 · p. 42

4.5.3 Strengths and Weaknesses.

• Symbolic reasoning chains executed by external solvers guarantee faithful derivations, but the language model’s generation process remains opaque; explanation quality and performance effects vary widely.Iterative ProofWriter verifies each step but is less efficient, whereas METGEN freezes pretrained reasoning modules and improves performance over earlier EntailmentWriter versions; CoT-style prompting can improve accuracy but natural-language chains need not cause answers.Source: §4.5.2 · pp. 43–44, §4.5.3 · p. 45

5. Summary and Discussion

• The five method families each face distinct faithfulness limitations, from invalid similarity assumptions and noncausal attention to gradient artifacts, counterfactual pitfalls, and weaknesses in intrinsic explanations.Backpropagation methods are more often tested for faithfulness, while counterfactual methods show improved faithfulness over earlier baselines but may generate OOD inputs; self-explanatory models can suffer from label leakage and social misalignment.Source: §5 · pp. 45–46

5.1 Virtues

• Faithfulness research increasingly addresses the gap between model competence and task performance, offers quantitative evaluation starting points, and often produces explanations understandable to nonexperts.Many explanation tactics are also model-agnostic, particularly for classification, and toolkits support their application to users’ models.Source: §5.1 · p. 46

5.2 Challenges and Future Work

• A central challenge is the absence of consensus on faithfulness measurement, because existing evaluations rely on differing assumptions, are often incomparable, and can produce inconsistent results.The survey calls for a universal evaluation framework, potentially complemented by a meta-evaluation framework for existing evaluation frameworks.Source: §5.2 · p. 47
• Future explanations should capture higher-level features and feature interactions, support tasks beyond classification and span identification, and enable interactive model improvement.Suggested forms include feature subsets, causal graphs, and explanations generalizable across tasks; knowledge editing offers initial evidence for interactive approaches.Source: §5.2 · p. 47
• The relationship between model performance and interpretability can be synergistic or conflicting, requiring theory about when explanations help and cautious balancing across application scenarios.This tension is especially evident in self-explanatory models, while recent studies investigate when explanations improve model performance.Source: §5.2 · p. 47
• Faithfulness is fundamental but insufficient: explanations should also support user goals, yet current methods can worsen over-trust and rarely improve human decision accuracy in practice.The survey therefore calls for studying faithfulness in real-world applications together with user-oriented utility.Source: §5.2 · pp. 47–48

• Faithfulness lacks a universally accepted technical definition or evaluation framework, limiting comparison and leaving many methods without quantitative faithfulness results.These gaps make it difficult to assess competing explanation methods consistently.Source: §6 · p. 48
• The survey critically synthesizes five families—similarity-based, model-internal, backpropagation-based, counterfactual, and self-explanatory methods—by representative work, strengths, weaknesses, and faithfulness.Source: §6 · p. 48
• Faithfulness varies substantially across families: attention and self-explanatory claims face important pitfalls, gradients suffer saturation and input sensitivity, and perturbation can be preferable to erasure under out-of-distribution concerns.The survey also highlights feature independence and hindsight bias as practical concerns, and notes gradual improvements in later attention and backpropagation variants.Source: §6 · p. 48
• Future work should establish universal faithfulness standards, clarify interpretability–performance relationships, and develop explanations using high-level features, flexible forms, and alternative task formats.The survey is intended both as an overview for interpretability researchers and as a practical guide for users seeking to understand their models.Source: §6 · p. 48

Appendix A: Additional Details

• The appendix provides additional mathematical formalization, example visualizations, and implementation resources for selected model explanation method families.Source: Appendix A · pp. 48–49

1.2 Backpropagation-based Methods

• Table 1 summarizes how gradient methods compute feature relevance ri(x), while Table 2 formalizes propagation methods through the recursive function D().Figure 1 visualizes gradient-method relevance on image-classification examples, where brighter shades indicate higher relevance; implementation packages include AllenNLP Interpret36, Captum37, RNNbow38, and DeepExplain39.Source: Table 1 · p. 49, §1.2 · p. 49, Figure 1 · p. 50

1.1 Analysis of Model-Internal Structures

• Neuron- and attention-visualization tools include RNNvis30, LSTMVis31, Seq2Seq-Vis32, BertViz33, and LIT34.Source: §1.1 · p. 49

1.3 Counterfactual Intervention

• Tools implementing counterfactual intervention include Captum40, LIT41, LIME42, SHAP43, Anchors44, Seq2Seq-Vis45, and the What-if Tool46.Source: §1.3 · p. 52