PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Explainability of Natural Language Processing Deep Models

Julia El Zini, Mariette Awad

arXiv:2210.06929v1cs.CLcs.HC

TL;DR

Textual input structure, opaque embeddings, and hard-to-visualize model internals complicate explainability for NLP, while existing methods lack a comprehensive framework. This survey organizes NLP methods across model types, explanation approaches, and levels, and synthesizes what is known about their evaluation and interpretability.

  • Problem

    Textual data, opaque embeddings, and difficult-to-visualize model internals create NLP-specific explainability challenges, and methods have not been studied in a comprehensive framework.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    The survey classifies NLP explainability methods by model-specific versus model-agnostic scope, by design versus post-hoc approach, and by input, processing, or output level.

    Source: §1 · p. 3, §10 · p. 29

  • Results

    The survey finds that referenced NLP explainability work frequently addresses embeddings, deep networks, attention, and visualizations, with greater focus on model internals than individual classification outputs.

    Source: §9 · p. 27

  • Takeaways & Limitations

    Attention weights are relatively more inherently interpretable than traditional model parameters, but further analysis is needed to assess their explanations.

    Source: §10 · p. 29

    A common limitation of attention approaches is the lack of a unified definition of attention explainability.

    Source: §7.2 · p. 19

Abstract

from arXiv · show

While there has been a recent explosion of work on ExplainableAI ExAI on deep models that operate on imagery and tabular data, textual datasets present new challenges to the ExAI community. Such challenges can be attributed to the lack of input structure in textual data, the use of word embeddings that add to the opacity of the models and the difficulty of the visualization of the inner workings of deep models when they are trained on textual data. Lately, methods have been developed to address the aforementioned challenges and present satisfactory explanations on Natural Language Processing (NLP) models. However, such methods are yet to be studied in a comprehensive framework where common challenges are properly stated and rigorous evaluation practices and metrics are proposed. Motivated to democratize ExAI methods in the NLP field, we present in this work a survey that studies model-agnostic as well as model-specific explainability methods on NLP models. Such methods can either develop inherently interpretable NLP models or operate on pre-trained models in a post-hoc manner. We make this distinction and we further decompose the methods into three categories according to what they explain: (1) word embeddings (input-level), (2) inner workings of NLP models (processing-level) and (3) models' decisions (output-level). We also detail the different evaluation approaches interpretability methods in the NLP field. Finally, we present a case-study on the well-known neural machine translation in an appendix and we propose promising future research directions for ExAI in the NLP field.

1 INTRODUCTION

Deep NLP models can be opaque despite strong performance, while textual data and model representations pose challenges for explaining them. This survey organizes NLP interpretability work across models, explanation levels, and evaluation approaches.

1 INTRODUCTION

  • Interpretability supports trust in opaque deep-learning predictions and can help investigate failure cases or teach people from models’ decisions.The need is especially pronounced for systems performing comparably to or better than humans.Source: §1 · p. 2
  • Text-trained models may require explanations during learning as well as prediction, including investigation of possible bias from curated datasets.Interpreting hidden representations and prediction patterns may indicate whether protected attributes affect predictions.Source: §1 · p. 2
  • The survey distinguishes inherently interpretable models from post-hoc methods, and model-specific from model-agnostic approaches across input, processing, and prediction levels.The levels cover embeddings, model internals, and model decisions.Source: §1 · p. 3
  • Beyond decision explanations, the survey covers word embeddings, NLP networks’ inner representations, transformers and attention, and methods for explaining individual decisions.It also discusses evaluation setups, metrics, datasets, challenges, and opportunities.Source: §1 · p. 3
  • The paper proceeds from NLP-specific challenges and terminology to surveys of input-, processing-, and output-level interpretability, then concludes with future directions and an NMT case study.The case study is in an appendix.Source: §1 · p. 3

2 CHALLENGES IN EXAI ON NLP MODELS

Text presents challenges for explainability methods designed around more explicit input features: words combine multiple kinds of information, while embeddings and model internals are difficult to interpret directly. Long-range dependencies and multilingual language semantics add further challenges.

2 CHALLENGES IN EXAI ON NLP MODELS

  • Text-specific challenges motivate treating explainability for NLP separately from methods developed for other data types.The section names polysemy, sarcasm, slang, cultural effects, and ambiguity among the challenges.Source: §2 · pp. 3–4
  • Pixel- or feature-based explanations do not transfer straightforwardly to text, where embeddings are opaque and model internals are difficult to visualize.Explaining embedding dimensions may not have practical implications; further processing of inner encodings is needed to dissect learned knowledge.Source: §2 · p. 4
  • A word-level explanation can conflate syntax, semantics, and accumulated context, leaving unclear which aspect of a word supports a decision.The section raises whether an explanation reflects part of speech, entity, meaning, or context.Source: §2 · p. 4
  • Textual explanations may depend on nonadjacent words, while multilingual models add language-specific semantics and context that need further deciphering.Source: §2 · p. 4

3 RELATED SURVEYS

Earlier explainability surveys often focus on particular methods, domains, or data types, while NLP surveys have addressed narrower scopes. This work positions its coverage as broader, especially for word embeddings and attention mechanisms.

3 RELATED SURVEYS

  • Earlier surveys cover distinct subsets: [28] presents evaluation settings, focuses on post-hoc methods, and surveys visualization methods applied to imagery.[39] primarily studies methods on image and tabular datasets.Source: §3 · p. 4
  • Other surveys address inherently explainable approaches, activation maximization, medical explainability, general opportunities and challenges [23], or reinforcement learning [84].Source: §3 · pp. 4–5
  • Figure 1 compares surveys by year, citation count, evaluation, and data type; bar thickness distinguishes broad from narrow scope.The figure is described as visualizing surveys since 2017 and their scope, data types, impact, and inclusion of evaluation methods.Source: §3 · p. 5, Figure 1
  • The NLP-focused surveys [22] and [56] address individual predictions and human-in-the-loop debugging, respectively, whereas this work surveys knowledge encoded and processed by language models.The authors report surveying over 50 papers across three sections that they say [22] missed.Source: §3 · p. 5

4 TERMINOLOGY

The survey classifies interpretability along three dimensions: how an explanation is produced, what it explains, and which models it targets. Its framework contrasts these dimensions with the terminology used in prior work.

4 TERMINOLOGY

  • The “how” dimension separates inherently interpretable models developed by design from post-hoc explanations of pretrained black-box models.The paper notes that inherently interpretable models require architectural modification and retraining large models.Source: §4 · p. 5
  • The “what” dimension covers input-level embeddings, processing-level RNN and transformer representations, and output-level decisions tied to input features or neuron activations.Source: §4 · p. 6
  • The “which” dimension distinguishes model-agnostic methods that can operate on any machine-learning model from model-specific methods tailored to particular architectures.Figure 2 compares the paper’s dimensions with [22]; missing arrows indicate that integrating consecutive dimensions is infeasible.Source: §4 · p. 6, Figure 2 · p. 6

5 INTERPRETING WORD EMBEDDINGS

Word embeddings encode useful semantic and syntactic information but are difficult to interpret; the survey reviews approaches to make them more transparent and examines how their interpretability is evaluated. It covers sparsification, rotation, external knowledge, and contextualized representations, while noting that evaluation methods often measure embedding quality rather than interpretability.

5.1 Sparsification of Embedding Spaces

  • Sparse embeddings yield more semantically coherent dimensions than dense embeddings, supporting the use of sparsification to make dimensions more interpretable.NNSE derives sparse, non-negative representations through matrix factorization.Source: §5.1 · p. 7, Figure 3 · p. 8
  • Other embedding approaches use online projected gradients, sparse coding, denoising autoencoders, or post-hoc sparsity and sense-based representations,,,,.OIWE addresses NNSE’s heavy memory use and difficulty with streaming text; SPINE learns sparse, non-negative representations.Source: §5.1 · pp. 7–8

5.2 Rotation of Embedding Spaces

  • Rotation methods use linear transformations to produce simpler embedding spaces, including low-complexity rotations, orthogonal transformations, and lexicon-guided interpretable subspaces,,.A separate account derives linear embedding relationships from paraphrasing and word transformations.Source: §5.2 · pp. 8–9

5.3 Integrating External Knowledge

  • External lexical resources can refine embeddings to reflect semantic relations, while domain knowledge graphs and ego-networks can encode biomedical categories or induce word senses,,.A gender-neutral embedding approach aims to remove gender influence while preserving essential gender information in specific dimensions [128].Source: §5.3 · pp. 9–10

5.4 Contextualized Embeddings

  • Morphology appears in contextual embedding layers, local syntax in early contextual layers, and semantics in deeper layers; these representations outperform non-contextualized embeddings on syntactic but not semantic tasks [109].They also encode long-range linguistic information that helps disambiguate longer-range syntactic dependencies; some training data’s male-female imbalance is reflected in unequal gender information in embeddings.Source: §5.4 · p. 11

5.5 Evaluating Embeddings Interpretability

  • Embedding evaluations include word intrusion, synonym and polarity judgments, analogy and linguistic tasks, and probing contextual representations across NLP tasks,,,.Word-intrusion evaluations ask human judges to identify an outlier among highly ranked words; an alternative distance-ratio metric does not require human assessment.Source: §5.5 · pp. 11–12
  • Many evaluations measure task or embedding quality rather than interpretability because they do not identify which dimensions encode a property or context.The survey notes that a rigorous, shared benchmark is still lacking.Source: §5.5 · p. 11, §5.6 · p. 12

5.6 Discussion

  • Qualitative dimension studies are advancing evaluation, but methods still lack commonality and objectivity; surveyed interest in embedding interpretability generally increases, with contextualized embeddings drawing attention after 2017.Sparsification was among the earliest methods, while interest in other methods appears consistent.Source: §5.6 · pp. 12–13, Figure 4 · p. 13

6 WHAT DO RNNS LEARN?

The section surveys how RNNs encode semantic and syntactic structure, contrasting post-hoc analysis with inherently interpretable architectures. It also highlights that current evaluation probes encoded properties without establishing full transparency.

6.1 Post-hoc Interpretation

  • RNNs capture local compositionality, including negation asymmetry, and LSTMs focus on essential keywords; sentiment tasks are sensitive to embedding dimensions.These post-hoc analyses use heatmaps, t-SNE visualizations, and unit salience to examine how input contributes to sentence meaning.Source: §6.1 · pp. 13–14
  • RNN representations encode syntax beyond training supervision, with deeper layers capturing higher-level syntax; studies also find long-distance agreement and context use alongside limits on distant-history discrimination.LSTMs used about 200 tokens of context on average, while more expressive architectures may be needed for particularly complex agreement sentences.Source: §6.1 · p. 14
  • An FSA-based framework learns interpretable state clusters and transitions from trained RNNs, enabling global explanations for classification tasks.Hidden states gathered from input sequences are clustered into FSA states, and transitions correspond to processing an input item.Source: §6.1 · p. 15

6.2 Inherently Interpretable Models

  • SISTA-RNN makes parameters interpretable by mapping the RNN to a principled probabilistic optimization model, and achieves higher performance on sequential compressive sensing.The approach replaces the conventional black-box RNN mapping with an optimization problem corresponding to a probabilistic model.Source: §6.2 · p. 15
  • Tree- and graph-structured LSTMs encode semantic or linguistic structure, but evaluations are limited, including structure-evolving LSTMs tested mainly on image segmentation.Graph LSTMs are reported to improve relation-extraction performance.Source: §6.2 · p. 15

6.3 Discussion

  • Syntax- and semantics-probe performance can indicate that a property is encoded, but explainability also requires locating it in the model and analyzing relevant inputs.Evaluations vary: some are subjective, while others use human-annotated labels that reflect linguistic properties rather than explainability itself.Source: §6.3 · p. 16

7 WHAT DO TRANSFORMERS LEARN?

The section surveys visualization and post-hoc methods for understanding transformer representations and attention. It presents evidence that attention can reflect linguistic structure, alongside limitations on its interpretation and evaluation.

7.1 Visualization of Transformers

  • Transformer visualization tools expose attention, latent states, and contextual representations through interactive views at multiple model and component levels.SEQ2SEQ-VIS supports debugging across encoder, decoder, attention, prediction, and beam search; ExBERT and Vig provide additional views of transformer internals.Source: §7.1 · pp. 17–18

7.2 Is the Attention Mechanism Inherently Interpretable?

  • Studies find attention can align with linguistic features, but results vary: some distributions lack frequent correlation with feature-importance methods, and equivalent predictions can arise from different distributions.Other analyses report attention capturing synonymy and multiword expressions, while unrelated sentence pairs may leave it dominated by the last output vector.Source: §7.2 · pp. 18–19
  • GPT-2 attention heads show layer-dependent patterns: early layers target determiners and positions, while deeper layers target proper nouns, linguistic constructs, and longer-distance relations.Attention-dependency alignment is strongest in middle layers, and attention distance is negatively correlated with dependency alignment.Source: §7.2 · p. 19
  • Discussed approaches lack a unified definition of attention explainability; attention weights are more inherently interpretable than general deep-network parameters, but some interpretability aspects require further processing and investigation.Source: §7.2 · p. 19

7.3 Interpretability of BERT

  • The BERT subsection frames interpretability around understanding why and how bidirectional training works well.It presents dissection as a way to understand BERT and its weights.Source: §7.3 · p. 19

7.4 Discussion

The discussion weighs attention’s relative interpretability against limits on transparency, especially for complex tasks and deeper layers. It also notes that transformer explainability evaluations remain aspect-specific.

7.4 Discussion

  • Attention weights are relatively more interpretable than conventional deep-network weights, but alone may not provide full transparency, particularly for translation, question answering, and natural language inference.Attention may also be less meaningful when computed over highly similar input representations.Source: §7.4 · p. 22
  • Deeper-layer attention weights become less interpretable, and output vectors may dominate them, so deeper-layer explanations can require further processing [91].Source: §7.4 · p. 23
  • Transformer evaluations test selected semantic or syntactic aspects, while the explainability definition and metric proposed in need extension to cover other aspects.The paper attributes the focus on post-hoc methods partly to the computational cost of retraining transformers.Source: §7.4 · p. 23

8 EXPLAINING MODEL’S DECISIONS

This section surveys methods for explaining individual NLP decisions, including post-hoc interpretations of pretrained models and models designed to produce explanations inherently.

8.1 Post-hoc Interpretation

  • Post-hoc methods analyze a pretrained model’s processing of text; they are model-agnostic when the model is treated as a black box and model-specific when they assume its architecture.Most model-agnostic methods perturb inputs to explain individual predictions.Source: §8.1 · p. 23, §8.1.1 · p. 23

8.1.1 Model-agnostic Explanations.

  • LIME locally approximates a classifier with an interpretable model, while anchors use if-then rules; SHAP unifies several feature-importance approaches, and L-Shapley and C-Shapley reduce model evaluations using graph structure.A limitation of LIME explanations is that they may not clarify whether a word such as “not” has positive or negative influence.Source: §8.1.1 · pp. 23–24

8.1.2 Model-specific Explanations.

  • RNN-specific methods quantify input contributions or propagate relevance, while rationalization methods use selected text spans as explanations; rationale-based methods include RA-CNN and approaches that encourage more comprehensive extraction.The ERASER framework provides a common benchmark for rationalization, alongside contrastive explanations, representation erasure, and information-theoretic measures.Source: §8.1.2 · pp. 24–25

8.2 Inherently Interpretable Models While the previous interpretability methods work on pre-trained models, the methods discussed

  • Inherently interpretable models jointly produce predictions and explanations rather than explaining a pretrained model after the fact.Lei et al. train a generator and encoder to select short, coherent rationales that remain sufficient for prediction.Source: §8.2 · p. 25, §8.2.1 · p. 25

8.2.1 Model-agnostic Explanations.

  • A model-agnostic generative framework produces fine-grained, attribute-specific explanations from a text representation alongside category predictions.The cited work reports applying its framework to two datasets collected by its authors.Source: §8.2.2 · p. 26

8.2.2 Model-specific Explanations.

  • A model-specific explanation factor trains a classifier to predict categories from explanations, strengthening their connection to predictions and improving base-predictor performance in experiments.Knowledge-graph reasoning is also surveyed as a structured approach to explaining NLP systems.Source: §8.2.2 · p. 26

8.3 Discussion

  • Inherently interpretable NLP models for individual decisions are limited in the surveyed work, while modifying and retraining costly language models may be expensive.Evaluation is uneven: some general-purpose methods lack NLP testing, and the inherently interpretable approaches use differing evaluation procedures.Source: §8.3 · pp. 26–27

9 DISCUSSION AND FUTURE DIRECTIONS

The survey describes trends in referenced NLP explainability work and identifies gaps in evaluation and coverage. It calls for common evaluation practices and further study of embedding spaces, inherently interpretable models, and explanation levels.

9 DISCUSSION AND FUTURE DIRECTIONS

  • Referenced titles frequently concern deep networks, representations or embeddings, and attention; the cited work largely emphasizes model internals over individual classification outputs.The survey also notes that its referenced venues include few journals devoted to ExAI, with work appearing in general AI or language-focused conferences.Source: §9 · p. 27, Figure 5 · p. 28, Figure 6 · p. 28
  • ExAI research interest has grown across embeddings, transformers, and neural machine translation, but evaluation lacks a shared framework of well-designed datasets and metrics.Existing NLP tasks are paper-specific and are not used to compare methods; quantitative assessment of embedding dimensions’ semantic and syntactic encoding also remains unexplored.Source: §9 · pp. 27–29, Figure 7 · p. 28
  • Inherently interpretable models are under-examined partly because modifying and retraining costly language models may be expensive and may not retain the same performance.The survey identifies explainable model designs as a future research direction.Source: §9 · p. 29
  • Current explanations focus on individual words or layers and neurons, while text understanding involves a hierarchy from extracted information to knowledge, wisdom, and logic.The survey cautions that focusing on individual input words can affect explanation efficacy by ignoring this hierarchy.Source: §9 · p. 29

10 CONCLUSION

The conclusion presents a survey organized across input, processing, and output explanations, and distinguishes model-agnostic from model-specific and post-hoc from inherently interpretable approaches. It synthesizes evaluation practices and research directions.

10 CONCLUSION

  • The survey combines NLP explainability methods for inputs, model processing, and outputs, while distinguishing model assumptions and post-hoc from inherently interpretable approaches.The authors describe it as the first comprehensive survey combining these levels.Source: §10 · p. 29
  • The survey covers embedding interpretation, hidden representations in RNNs and transformers, decisions, and visualization methods; it concludes that attention weights are relatively more inherently interpretable than traditional parameters but need further analysis.Embedding methods include sparsity constraints, rotations, external knowledge, and contextualized embeddings.Source: §10 · p. 29
  • Figure 8 maps research on embeddings, RNNs and transformers, model decisions, and visualization methods, showing connections among these areas.The survey also groups evaluation approaches by whether they address embeddings, model internals, or decisions, and distinguishes interpretability assessments from qualitative evaluations.Source: §10 · pp. 29–30, Figure 8 · p. 30

Related papers

Loading 2210.06929v1…