Source-linked AI summary
A Survey of the State of Explainable AI for Natural Language Processing
Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, Prithviraj Sen
TL;DR
Modern NLP models often achieve stronger quality at the expense of interpretability, creating a need to understand how their predictions are produced. This survey synthesizes NLP XAI categories, techniques, visualizations, and evaluations, finding gaps in terminology and evaluation that motivate future work.
Problem
NLP models have become less interpretable as model quality has advanced, while explainability lacks formal definitions and standardized evaluation.
Method
The survey reviews NLP XAI literature, categorizes explanation approaches, and details techniques, operations, visualizations, and evaluation methods.
Results
The survey identifies commonly used feature-importance and surrogate-model approaches, with saliency the dominant visualization technique among reviewed papers.
Takeaways & Limitations
The survey serves as a resource for developers and highlights the need for clearer terminology and expanded evaluation, especially human evaluation.
Takeaways & Limitations
The survey’s coverage depends on selected NLP conferences and title-based XAI terms, so related papers may be omitted.
Abstract
from arXiv · showhide
Recent years have seen important advances in the quality of state-of-the-art models, but this has come at the expense of models becoming less interpretable. This survey presents an overview of the current state of Explainable AI (XAI), considered within the domain of Natural Language Processing (NLP). We discuss the main categorization of explanations, as well as the various ways explanations can be arrived at and visualized. We detail the operations and explainability techniques currently available for generating explanations for NLP model predictions, to serve as a resource for model developers in the community. Finally, we point out the current gaps and encourage directions for future work in this important research area.
1 Introduction
NLP has shifted from inherently explainable white-box methods toward higher-quality but less interpretable black-box models, motivating XAI research tailored to NLP. This survey organizes the field’s explanation categories, techniques, visualizations, evaluations, and open gaps.
- Deep learning and language embeddings can improve model quality while making NLP systems less interpretable.
- The survey focuses on XAI work from major NLP conferences over the last seven years and presents itself as the first survey focused specifically on NLP.
- Explainability can help end users understand model outcomes, build trust in predictions, and provide feedback that may improve model quality.
- The survey analyzes NLP XAI literature across explanation dimensions, generation operations, visualization techniques, and evaluation methods.
- Its goals include clarifying the state of XAI in NLP, directing developers to available techniques, and highlighting gaps such as limited formal definitions and evaluation.
2 Methodology
The survey identifies relevant NLP XAI papers, classifies them using predefined aspects, and applies multiple-reviewer analysis to improve classification consistency. It also provides a website containing the covered papers’ aspects and techniques.
- Relevant papers were identified and classified according to the aspects defined in Sections 3 and 4.
- Each paper was individually analyzed by at least two reviewers, with additional reviewers consulted when disagreements occurred.
- The survey’s website collects relevant aspects for every covered paper and helps readers discover explainability techniques and ideas.
3 Categorization of Explanations
Explanations are categorized by whether they target an individual prediction or the overall predictive process, and by whether they arise during prediction or afterward. These dimensions produce four high-level categories.
- Local explanations justify a model’s prediction for a specific input, whereas global explanations describe the predictive process independently of a particular input.
- The two classification dimensions induce four high-level explanation categories summarized in Table 1.
- Self-explaining approaches generate explanations during prediction using information emitted by the model.
- Post-hoc approaches perform an additional operation after prediction, such as applying a surrogate model to produce a local explanation.
- A paper may span both self-explaining and post-hoc categories when it presents techniques from both groups.
4 Aspects of Explanations
The survey separates explanation derivation from presentation and catalogs five major techniques, their enabling operations, and visualization choices. It also identifies common combinations across high-level explanation categories and reports feature importance and surrogate models as the most frequent techniques.
- Explanation derivation and presentation: Explainability techniques derive raw mathematical justifications, whereas visualization techniques determine how those justifications are presented to end users.Attention scores exemplify raw explanations, while saliency heatmaps exemplify their presentation.
- Explainability techniques: The survey identifies five major techniques: feature importance, surrogate models, example-driven explanations, provenance-based explanations, and declarative induction.These techniques differ in the mechanisms used to generate raw explanations for end users.
- Explainability techniques: Feature importance examines feature scores, while surrogate models learn a more explainable proxy for the original model’s predictions.Surrogate models can provide model-agnostic local or global explanations; feature importance can use manual, lexical, or latent features.
- Prevalence: 29 of the 50 reviewed papers use feature importance and 8 use surrogate models, making them the most frequent techniques in the survey.The survey attributes this pattern to features being machine-learning building blocks and recent NLP models commonly being neural-network black boxes.
- Operations: The survey covers operations including first-derivative saliency, layer-wise relevance propagation, input perturbations, attention, LSTM gating signals, and explainability-aware architecture design.These operations enable techniques such as feature importance, surrogate models, example-driven explanations, and human-readable program induction.
- Visualization techniques: Saliency is the most dominant visualization technique, and all surveyed feature-importance approaches use saliency-based visualizations.Other presentation forms include raw declarative representations and natural-language explanations; complex systems may combine multiple explanation techniques.
5 Explanation Quality
Explanation quality is evaluated through multiple approaches and dimensions, but the field has little agreement on standardized evaluation. Coverage of the predictive process is also important because many explanations leave users to fill in gaps.
- Evaluation landscape: 32 of 50 reviewed works lacked standardized evaluation or used only informal evaluation.More formal approaches included ground-truth comparisons and human evaluation.
- Evaluation techniques: Informal examination assesses whether generated explanations align with human intuition, either alone or against reference approaches such as LIME.
- Evaluation techniques: Ground-truth comparison quantifies explanation performance with task-dependent metrics including P/R/F1, perplexity, and BLEU.Multiple annotators, inter-annotator agreement, mean human performance, and different granularities can address alternative valid explanations and annotation disagreement.
- Evaluation techniques: Human evaluation directly assesses explanation effectiveness without assuming a single valid ground-truth explanation.Reliable human evaluation still requires multiple annotators, agreement reporting, and attention to subjectivity and response variance.
- Operation-specific evaluation: Attention explanations can be tested with counterfactual or erasure-based procedures that remove dominant attention weights and examine whether predictions change.These tests evaluate whether attention weights reflect the model’s output prediction.
- Evaluation targets and coverage: Evaluation should distinguish explanation aspects such as fidelity and comprehensibility and consider how much of the input-to-output process the explanation covers.Many approaches explain only part of that process, leaving end users to fill in the gaps; coverage should be interpreted relative to the target audience and use.
6 Insights and Future Directions
The survey organizes recent NLP XAI advances, identifies gaps in terminology and evaluation, and highlights an imbalance toward local explanations of black-box models. It also suggests that white-box models could support explanation-evaluation research.
- Survey contributions: The survey categorizes explanations, reviews ways to derive and visualize them, and details available techniques for developers building explainable NLP models.
- Gaps and future directions: The field needs clearer terminology, better links between explainability and target audiences, and expanded evaluation processes and human-evaluation metrics.The survey also notes trade-offs among explainability, predictive quality, runtime performance, complexity, and memory usage.
- Insights: Only four surveyed papers addressed global explanations, while explicit XAI research mainly focuses on local explanations for black-box models.
- Future directions: White-box models may be overlooked as test beds for studying techniques that evaluate explanations.
A Appendix A - Methodology
The survey identifies recent NLP XAI work through keyword-filtered searches of major conferences from 2013 to 2019, then reviews the papers for scope and classification. The included corpus contains 50 papers, with topics and conferences summarized in Table 4.
- Search strategy: The methodology targets papers from ACL, NAACL, EMNLP, and COLING published between 2013 and 2019.Search titles used lemmatized terms related to explainability, interpretability, and transparency.
- Search strategy: Keyword-based search criteria yielded 107 papers, although the authors acknowledge that related papers may have been missed.The approach assumes authors treating explainability as a major component are more likely to use related title keywords.
- Corpus construction: After scope review excluded 57 papers, the survey retained 50 papers focused on explainability as a vehicle for understanding model predictions.Table 4 reports the top NLP topics and conferences represented in the included set.