Source-linked AI summary
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, Sameer Singh
TL;DR
Neural NLP models are imperfect and opaque, while existing interpretation codebases make it difficult to apply explanations across new models and tasks. AllenNLP Interpret provides model-agnostic interpretation primitives, built-in methods, and reusable visualizations. Its demos span multiple interpretation methods, tasks, and model architectures, illustrating flexible analysis of biases, decision rules, and errors.
Problem
Existing interpretation codebases are often model- or task-specific and support few methods, making models difficult for practitioners to interpret and methods difficult for researchers to evaluate across models.
Method
AllenNLP Interpret combines model-agnostic input-gradient APIs, built-in saliency and adversarial methods, and reusable front-end visualization components.
Results
The toolkit demonstrates flexible interpretation across six NLP tasks and varied model architectures, including examples that uncover biases, decision rules, and errors.
Takeaways & Limitations
AllenNLP Interpret facilitates developing and evaluating interpretation methods across a wide range of NLP models and tasks.
Abstract
from arXiv · showhide
Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior. Model interpretation methods ameliorate this opacity by providing explanations for specific model predictions. Unfortunately, existing interpretation codebases make it difficult to apply these methods to new models and tasks, which hinders adoption for practitioners and burdens interpretability researchers. We introduce AllenNLP Interpret, a flexible framework for interpreting NLP models. The toolkit provides interpretation primitives (e.g., input gradients) for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. We demonstrate the toolkit's flexibility and utility by implementing live demos for five interpretation methods (e.g., saliency maps and adversarial attacks) on a variety of models and tasks (e.g., masked language modeling using BERT and reading comprehension using BiDAF). These demos, alongside our code and tutorials, are available at https://allennlp.org/interpret .
1 Introduction
Neural NLP models can rely on superficial patterns and remain opaque, creating a need for instance-level explanations. AllenNLP Interpret addresses limited, inflexible interpretation tooling with an extensible toolkit, reusable visualizations, and demonstrations across model behaviors.
- State-of-the-art NLP models remain imperfect and opaque, prompting questions about why they make particular predictions.
- Existing interpretation codebases are often model- or task-specific and support few methods, limiting practitioners and interpretability researchers.
- AllenNLP Interpret provides interpretation techniques, model- and task-agnostic APIs, and reusable front-end components for visualizing NLP model explanations.
- Its demonstrations expose model biases, decision rules, and errors across tasks including question answering, entailment, named entity recognition, and sentiment analysis.
2 Interpreting Model Predictions
AllenNLP Interpret offers gradient-based saliency maps and adversarial attacks for examining why models make predictions and how their outputs respond to input changes. The toolkit applies these methods across six tasks and varied model architectures.
- 2.2 Saliency Map Visualizations: Saliency maps identify important input tokens using gradients, while adversarial attacks modify tokens to change or preserve model predictions.
- 2.2 Saliency Map Visualizations: The three saliency methods are Vanilla Gradient, Integrated Gradients, and SmoothGrad, with token-gradient vectors reduced to scalar importance values using the L2 norm.
- 2.3 Adversarial Attacks: HotFlip uses gradients for word substitutions that probe sensitivity or target a specified prediction, whereas Input Reduction removes low-gradient words without changing the prediction.
- 2.4 Currently Available Models: The toolkit supports six tasks spanning diverse input-output formats and model architectures.
- 2.4 Currently Available Models: The available models include reading comprehension, masked language modeling, text classification, textual entailment, named entity recognition, and coreference resolution.
3 AllenNLP Interpret Under the Hood
AllenNLP Interpret computes input gradients across heterogeneous NLP models by converting predictions into labeled instances and hooking gradients at the token-embedding interface. This model-agnostic backbone supports interpretation methods and handles contextual embeddings for attacks such as HotFlip.
- 3.1 Model-Agnostic Input Gradients: The toolkit extends AllenNLP’s Predictor so interpretation methods can compute input gradients across output formats and embedding types.
- 3.1 Model-Agnostic Input Gradients: The predictions to labeled instances() function converts a model’s predictions into labels, creating separate instances for structured outputs such as individual NER tags.
- 3.1 Model-Agnostic Input Gradients: A backward hook on AllenNLP’s Token-Embedder interface enables gradients for varied embedding methods, including word vectors, ELMo, and BERT embeddings.
- 3.1 Model-Agnostic Input Gradients: The resulting API computes input gradients by calling predictions to labeled instances() followed by get gradients().
- 3.2 Contextual Embeddings: For contextual embeddings without a searchable embedding matrix, the toolkit builds a context-independent feature matrix so HotFlip can search discrete substitutes while retaining contextual gradients.
4 Adding a Model or Interpretation
AllenNLP Interpret is designed to make adding interpretation methods and models lightweight through tutorials, simple backend integrations, and reusable visualization components. Existing demo tasks can use a new model with minimal code changes.
- New Interpretation: Adding SmoothGrad requires implementing the method, registering a demo backend interpreter, and connecting the reusable front-end visualization.
- New Interpretation: Reusable visualization components let developers add saliency-map or adversarial-attack interfaces with a one-line HTML call.
- New Model: Replacing a demo model for an already supported task requires changing a single line of code.
- New Model: A new task requires a predictions to labeled instances() function, a backend model path, and a front-end page using reusable templates.
5 Related Work
The paper situates AllenNLP Interpret among interpretation toolkits, emphasizing that NLP coverage remains scarcer and often model- or task-specific. It focuses on gradient-based methods because they apply broadly across models.
- Alternative Interpretation Methods: Gradient-based methods are emphasized because they apply to many models.
- Existing Interpretation Toolkits: Existing NLP toolkits are relatively scarce and typically target specific models or tasks.
- Existing Interpretation Toolkits: Prior NLP tools include attention visualizations for particular models, whereas this work addresses broader toolkit flexibility.
6 Conclusion
AllenNLP Interpret is an open-source, flexible toolkit for developing and evaluating NLP model interpretation methods across diverse models and tasks. The project will continue expanding through new methods, models, and open-source contributions.
- AllenNLP Interpret facilitates interpretation of NLP models across a wide range of models and tasks.
- The toolkit supports developing and evaluating interpretation methods.
- The project plans to add new interpretation methods and models as they become available.
- Open-source contributions are welcomed from model developers and interpretability researchers.