Source-linked AI summary
Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?
Alon Jacovi, Yoav Goldberg
TL;DR
Deep-learning NLP systems increasingly support consequential decisions, but interpretation quality and faithfulness lack consistent definitions and evaluations. The paper surveys faithfulness methods through three underlying assumptions, proposes evaluation guidelines, and argues that binary faithfulness should be replaced with a graded criterion. It concludes that graded evaluation is more practically useful because strict faithfulness is often difficult to establish.
Problem
Interpretability research lacks consistent definitions of faithfulness and often conflates faithfulness with other interpretation criteria, making evaluation unclear.
Method
The paper surveys faithfulness-evaluation literature, organizes approaches around three assumptions, and develops guidelines for evaluating interpretation methods.
Results
The paper finds that faithfulness is commonly evaluated as a binary property and argues that this standard is difficult to satisfy and easy to disprove through counterexamples.
Takeaways & Limitations
Faithfulness should be evaluated on a graded scale that measures the extent and likelihood of faithful interpretation in practice.
Takeaways & Limitations
The paper notes that utility-based HCI evaluations can show correlations between explanation plausibility and model performance rather than faithfulness.
Abstract
from arXiv · showhide
With the growing popularity of deep-learning based NLP models, comes a need for interpretable systems. But what is interpretability, and what constitutes a high-quality interpretation? In this opinion piece we reflect on the current state of interpretability evaluation research. We call for more clearly differentiating between different desired criteria an interpretation should satisfy, and focus on the faithfulness criteria. We survey the literature with respect to faithfulness evaluation, and arrange the current approaches around three assumptions, providing an explicit form to how faithfulness is "defined" by the community. We provide concrete guidelines on how evaluation of interpretation methods should and should not be conducted. Finally, we claim that the current binary definition for faithfulness sets a potentially unrealistic bar for being considered faithful. We call for discarding the binary notion of faithfulness in favor of a more graded one, which we believe will be of greater practical utility.
1 Introduction
Deep-learning NLP systems increasingly influence decisions, creating a need for explanations whose quality is defined and evaluated clearly. The paper focuses on faithfulness, distinguishes it from other criteria, organizes existing evaluation approaches, and argues for graded rather than binary faithfulness.
- Deep-learning NLP models are increasingly used for prediction and decision-making, including in sensitive fields such as health, commerce, and law.
- Interpretability research often conflates readability, plausibility, and faithfulness, despite these being distinct aspects of interpretation quality.
- Faithfulness concerns whether an explanation accurately represents the reasoning process behind a model’s prediction.
- The paper surveys faithfulness evaluation and organizes existing approaches around three underlying assumptions.
- The paper argues that binary faithfulness is impractical because its assumptions are difficult to satisfy fully and easy to disprove with counterexamples.
- It proposes evaluating faithfulness with graded criteria measuring the extent and likelihood that an interpretation is faithful in practice.
2 Faithfulness vs. Plausibility
The paper distinguishes plausibility, which concerns how convincing an interpretation is to humans, from faithfulness, which concerns whether it reflects the model’s actual reasoning. It argues that these criteria can diverge and should not be conflated.
- Plausibility measures how convincing an interpretation is to humans, whereas faithfulness measures how accurately it reflects the model’s true reasoning process.
- An interpretation can be plausible without being faithful, as illustrated by post-hoc text generation trained on human-written explanations.
- Many studies fail to distinguish plausibility from faithfulness or leave the evaluation criterion unnamed.
- Conflating the criteria can be dangerous when users believe a persuasive explanation reflects the model’s reasoning and act on it in consequential settings.
3 Inherently Interpretable?
Inherently interpretable models are presented as potentially more faithful than post-hoc explanations, but the paper argues that this remains an unverified claim. Such models should therefore undergo the same faithfulness evaluations as post-hoc methods.
- Inherently interpretable models are designed to provide interpretations directly, unlike post-hoc techniques applied to existing black-box models.
- The paper warns that inherent interpretability does not establish faithfulness and must be verified before the model is trusted.
- Recent studies have raised doubts about the faithfulness of attention mechanisms despite their treatment as inherently interpretable.
4 Evaluation via Utility
Utility-based HCI evaluations can show that explanations help users perform tasks, but they do not establish that explanations faithfully reflect model reasoning. A non-faithful explanation can improve user performance when plausibility correlates with model correctness.
- Evaluation via Utility: HCI evaluations assess explanations through user task performance, often aiming to increase trust and help users judge whether model decisions are likely correct.The final metric in this setting is generally the user's performance at their task.
- Evaluation via Utility: Increased user performance is not indicative of faithfulness; it reflects correlation between explanation plausibility and model performance.The authors distinguish utility from faithfulness in IUI evaluations.
- Evaluation via Utility: A fictional heat-map system can improve user performance while assigning relevance to random tokens unrelated to the model's decisions.The system presents plausible-looking content-word explanations for correct outputs and punctuation explanations for incorrect outputs.
- Evaluation via Utility: Any correlation between explanation plausibility and model performance can increase user performance without establishing faithfulness.The paper presents this as a broader concern beyond the deliberately extreme fictional example.
5 Guidelines for Evaluating Faithfulness
The paper recommends evaluating faithfulness explicitly and independently from plausibility, without human judgments, human-provided gold labels, user performance, or unverified claims of inherent interpretability.
- Guidelines for Evaluating Faithfulness: Evaluators should state whether they measure plausibility or faithfulness and use methodologies appropriate to the selected property.The same distinction should guide the design of interpretation techniques.
- Guidelines for Evaluating Faithfulness: Faithfulness evaluation should not use human judgments because humans cannot determine whether an interpretation matches the model's reasoning.Human supervision is likewise unavailable for this problem, so such judgments measure plausibility instead.
- Guidelines for Evaluating Faithfulness: Faithfulness evaluation should not rely on human-provided gold labels, which import human priors about what the model should do.The guideline applies to interpreting incorrect as well as correct model predictions.
- Guidelines for Evaluating Faithfulness: Explanations from inherently interpretable models should be evaluated for faithfulness using the same standards as post-hoc methods.Inherent interpretability is treated as an unverified claim rather than a guarantee.
- Guidelines for Evaluating Faithfulness: IUI faithfulness evaluations should not use end-task user performance, which measures utility-related correlation rather than faithfulness.User performance remains relevant for evaluating interpretation utility in some use cases.
6 Defining Faithfulness
The paper organizes faithfulness evaluation around three assumptions that provide the research community's implicit working definitions. These assumptions motivate tests based on model behavior, prediction consistency, and feature importance.
- Defining Faithfulness: Existing faithfulness tests are ad hoc and inconsistent because the field lacks a standard definition.The paper argues that this makes commonalities between evaluation methods difficult to identify.
- Defining Faithfulness: The paper identifies three assumptions underlying current faithfulness evaluation methods and uses them to organize the literature.Making the assumptions explicit also supports discussion of their validity and merit.
- Model Assumption: The Model Assumption treats identical predictions as evidence of identical reasoning and uses differing interpretations from decision-equivalent models to disprove faithfulness.Adversarially trained models can mimic an original model's decisions while producing different explanations.
- Prediction Assumption: The Prediction Assumption links similar inputs and decisions with similar reasoning, implying that faithful interpretations should vary consistently with model decisions.This motivates tests of interpretation consistency across similar or dissimilar inputs and outputs.
- Prediction Assumption: The Prediction Assumption is easier to use for disproving faithfulness than proving it because proof would require checking many examples or the entire input space.Counterexamples can establish a violation, whereas confirming the assumption globally is much more demanding.
- Linearity Assumption: The Linearity Assumption treats input parts as independently contributing to model reasoning, motivating heat-map stress tests and erasure-based measures.Erasure removes highly ranked features expecting decisions to change, or low-ranked features expecting decisions to remain stable; comprehensiveness and sufficiency generalize this idea.
- Linearity Assumption: The paper notes that the Linearity Assumption has been scrutinized and is not necessarily endorsed, despite its use in parts of the literature.Feature-attribution explanations are another name for heat-map interpretations.
7 Is Faithful Interpretation Impossible?
The paper argues that binary faithfulness tests set an unrealistic bar because interpretation is an approximation that can diverge from complex model reasoning. It therefore frames current counterexamples as expected and the binary evaluation trend as unproductive.
- Is Faithful Interpretation Impossible?: Current assumptions are commonly used to classify interpretations as strictly faithful or unfaithful, especially through counterexamples showing global failure.This pattern appears across attention, saliency, and other interpretation methods.
- Is Faithful Interpretation Impossible?: The binary approach is considered unproductive because interpretations approximate the model's reasoning and therefore lose information.The paper invokes the pigeonhole principle to argue that some inputs will exhibit deviation between interpretation and reasoning.
- Is Faithful Interpretation Impossible?: Adversarial and pathological behaviors arise from the nonlinear, high-dimensional decision boundaries of current models, while supervision about human-readable mappings is unavailable.These conditions prevent simply ignoring approximation errors.
- Is Faithful Interpretation Impossible?: The resulting standard is a high bar that the paper estimates will not be overcome soon, if at all.The section poses the practical question of how to build systems that provide faithful explanations under this constraint.
8 Towards Better Faithfulness Criteria
The paper calls for a more practical and nuanced way to define and evaluate faithfulness, allowing interpretations to be judged as sufficiently faithful for practical use. It proposes graded evaluation across models, tasks, and regions of the input space rather than a universal binary judgment.
- The authors propose formal faithfulness definitions and evaluations that can identify when an interpretation is sufficiently faithful for practical use.
- Faithfulness can be evaluated across specific models and tasks, since some settings may permit sufficiently faithful interpretations while others may not.For example, the paper contrasts question answering with movie-review sentiment tasks.
- Faithfulness can also be evaluated across subspaces of the input space, including neighborhoods of similar inputs or individual instances.This permits using an interpretation method only for decisions whose explanations are sufficiently faithful.
9 Conclusion
The paper argues that interpretability evaluation should separate faithfulness from plausibility and replace binary faithfulness judgments with a graded view. This approach treats strictly faithful interpretations as unlikely while preserving practical utility for interpretations that are not globally definitive.
- Interpretability evaluation often conflates faithfulness with plausibility, so faithfulness should be evaluated independently of an interpretation’s convincing power.
- Strictly faithful interpretations are described as unlikely to be found, making binary “faithful or not faithful” evaluation impractical.
- A graded faithfulness scale can preserve the usefulness of interpretations that are not globally and definitively faithful.