PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions

Xiaochuang Han, Byron C. Wallace, Yulia Tsvetkov

arXiv:2005.06676v1cs.CLcs.LG

TL;DR

Black-box NLP models remain difficult to interpret, and token-level saliency may not suit reasoning-heavy tasks. This paper evaluates influence functions, which identify influential training examples, against saliency methods and finds them especially useful for NLI while also quantifying data artifacts.

  • Problem

    Existing NLP interpretation methods emphasize input tokens, but their suitability and comparative evidence for complex tasks and training-data artifacts remain limited.

  • Method

    The paper compares gradient-based saliency maps with influence functions that trace predictions to influential training examples in BERT-based sentiment-analysis and NLI experiments.

  • Results

    Saliency maps and influence functions are largely consistent for sentiment analysis but not for NLI, where influence functions may be more suitable for interpretation.

  • Takeaways & Limitations

    Influence functions offer an alternative for interpreting complex NLP predictions and a way to reveal and quantify effects of artifacts in training data.

  • Takeaways & Limitations

    Influence-based artifact evidence is necessary but not sufficient to conclude that a model exploited the artifact, and influence computation is expensive.

Abstract

from arXiv · show

Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting important words in the corresponding input text. While this might be useful for tasks where decisions are explicitly influenced by individual tokens in the input, we suspect that such highlighting is not suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers. Influence functions explain the decisions of a model by identifying influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which 'saliency maps' may not have clear interpretation. Furthermore, we develop a new quantitative measure based on influence functions that can reveal artifacts in training data.

1 Introduction

The paper compares input-token saliency maps with training-example influence functions for interpreting black-box NLP predictions. It evaluates their reliability, consistency, and use for uncovering training-data artifacts.

  • Saliency maps may fail to explain complex reasoning tasks such as natural language inference because they highlight individual tokens rather than interactions.They can remain useful for lexicon-driven tasks, but may indicate importance without explaining why the model made a prediction.
  • Influence functions instead rank training examples by their influence on a test prediction, tracing decisions through model parameters without modifying model structure.The approach identifies examples most responsible for particular predictions.
  • The experiments assess whether influence-function approximations interpret transformer-based models such as BERT and compare them with gradient-based saliency scores.The comparison covers sentiment analysis and natural language inference.
  • The paper proposes a quantitative influence-function measure for revealing how hypothesized artifacts or confounds affect model predictions.This extends influence functions beyond individual prediction explanations to analysis of training data.
  • The work presents an NLP comparison between input saliency methods and explanations based on influential training examples.The authors identify this as the first such comparison in NLP to their knowledge.

2 Explaining Black-box Model Predictions

The paper contrasts gradient-based token saliency with influence functions that trace a prediction back to training examples. Each method assigns signed, prediction-specific importance through a different part of the model.

  • Gradient-based saliency maps: Gradient-based saliency scores use the gradient of the predicted-loss with respect to each input-token embedding.The score is −∇e(t)L̂y · e(t), combining loss sensitivity with the embedding’s sign and magnitude.
  • Gradient-based saliency maps: Signed saliency distinguishes tokens supporting the model’s prediction from tokens providing counter-evidence.The method retains the sign rather than taking the absolute value, then L1-normalizes scores across tokens.
  • Influence functions: Influence functions estimate how upweighting a training example changes learned parameters and then how that parameter change affects the test input’s predicted loss.This provides a route from a prediction to influential training examples.
  • Influence functions: The influence score is defined as the negative derivative of predicted loss with respect to an example’s upweighting and is z-normalized across training examples.Scores are specific to individual test instances because the loss is defined for a particular test input.
  • Influence functions: Positive influence means removing an example is expected to reduce confidence in the test prediction, whereas negative influence means removal is expected to increase it.The signs therefore distinguish examples supporting versus opposing a particular prediction.

3 Experimental Setup

The study compares both interpretation approaches on sentiment analysis and natural language inference using BERT-based models. It also addresses approximation assumptions, computational cost, and artifact-focused evaluation with HANS.

  • Tasks and models: The experiments compare gradient-based attributions and influence functions on lexicon-driven sentiment analysis and reasoning-driven natural language inference.Both models use BERT encoders, with the first eight of twelve layers frozen and only the final four layers and projection layer fine-tuned.
  • Influence-function assumptions: Influence functions are theoretically guaranteed accurate only for strictly convex models trained to convergence, unlike the non-convex, often early-stopped BERT setting.The study uses Hessian damping and a practical sanity check to address this setting.
  • Sentiment analysis: The sentiment model trains on 10k SST-2 examples, reaches 89.6% dev accuracy, and explanations are extracted for 50 sampled dev examples.The SST-2 development set contains 872 examples.
  • Computational constraints: Influence computation remains expensive: scoring 10k training examples for one test input takes approximately 10 minutes on one NVIDIA GeForce RTX 2080 Ti GPU.This motivates using smaller BERT models and reduced training sets.
  • Natural language inference: The MNLI model uses 10k sampled examples with neutral and contradiction collapsed into non-entailment, reaching 84.6% accuracy on a 9815-example dev set.MNLI supplies premise–hypothesis pairs across three original relations and ten genres.
  • Artifact evaluation: HANS provides balanced diagnostic examples across 30 heuristic sub-categories, and the study tests 30 examples covering all sub-categories.The dataset probes artifacts such as lexical overlap and subsequence heuristics.

4 Evaluating Influence Functions for NLP

The evaluation tests whether influence-function approximations are reliable for BERT and whether influential training examples align with token-level saliency. Influence functions behave reasonably in both tasks, but agreement with saliency differs sharply between sentiment analysis and NLI.

  • The sanity check removes the most positive, most negative, least influential, or random 10% of training examples and measures prediction-confidence changes across five seeds.
  • Removing the most positively influential examples substantially lowers prediction confidence, while removing negatively influential examples slightly increases it in both tasks.
  • Removing the least influential examples has an effect closest to removing the same number of random examples, although the NLI difference is larger than random removal.
  • Influence functions behave reasonably and reliably for BERT in both sentiment analysis and NLI.
  • In sentiment analysis, training examples containing the most positively salient test token generally have higher influence, unlike in NLI.
  • The token-removal experiment compares overlap among the top 0.1%, 0.2%, 0.5%, and 1% influential training examples before and after removing salient tokens.
  • For NLI, removing the most negatively salient token produces the most different influence distribution, contrary to the sentiment-analysis pattern.
  • Overall, saliency maps and influential examples are consistent in sentiment analysis but can tell very different stories for NLI.

5 Interpreting NLI Predictions with Influence Functions

Influence functions provide example-based explanations for NLI, where token-level saliency may miss interactions among words or spans. Their influential training examples expose lexical-overlap artifacts and support quantitative artifact analysis, while requiring cautious interpretation.

  • Comparing explanations: Influential training examples are presented as supporting or opposing instances, contrasting with saliency maps that highlight positive or negative token scores.Table 5 uses both explanation types for a correctly predicted HANS example.
  • Motivation: NLI decisions often depend on interactions among multiple words or spans, limiting the insight provided by individual-token importance scores.Influence functions likewise do not expose latent word interactions directly, but influential training examples can reveal artifact patterns post hoc.
  • Artifact patterns: HANS influential examples exhibit lexical overlap and sometimes reverse span ordering, suggesting reliance on artifacts rather than the intended relation.The test input shares high lexical overlap and similar reverse ordering with these examples.
  • Quantifying artifacts: The influence-artifact method analyzes individual examples by testing whether their most influential training examples contain a hypothesized artifact.A positive quadratic coefficient indicates that the artifact appears more in the most influential examples, while an irrelevant feature should yield zero.
  • Quantifying artifacts: +3.28×10^-3 and +3.30×10^-3 are the average quadratic coefficients for HANS examples predicted as entailment and non-entailment, respectively.Both groups show influential training examples with high lexical-overlap rates, indicating that the artifact remains recognized in predicted non-entailment cases.
  • Quantifying artifacts: +0.65 × 10^-3 is the average quadratic coefficient for 50 natural MNLI examples, considerably smaller than the HANS cases.The authors report that natural examples therefore show less reliance on lexical overlap than the diagnostic dataset.
  • Limitations: Artifacts in influential examples are necessary but not sufficient evidence that the model exploited them, and group-level imbalance may not determine each example’s behavior.The method can identify candidate artifacts but should be complemented by broader probing.

6 Related Work

Related work situates influence functions among token-level, perturbation, probing, generative, and training-example-based interpretation methods, alongside research on artifacts in NLI and other text-classification tasks.

  • Input-based interpretation: Token-level interpretation includes attention inspection, gradient-based attribution, input perturbation, and LIME, though attention’s faithfulness remains debated.Gradient variants seek robustness to input noise, while input perturbation measures importance through token removal and LIME fits a local sparse linear model.
  • Training-example explanations: Influence functions trace predictions to influential training examples without requiring special model designs that may sacrifice performance or generalizability.Related approaches include classifiers based on weighted training-example averages and architectures activated by sampled training instances.
  • Other approaches: Other interpretability approaches use adversarial edits, probing of internal representations, or models that extract or generate natural-language explanations.These approaches target changes in decisions, internal properties, or explicit explanatory outputs.
  • Artifacts in NLP: NLI artifact research identifies annotation artifacts and uses diagnostic datasets to examine whether models exploit them, while other work applies adversarial learning to demote artifacts.The related literature extends artifact analysis beyond NLI to other text-classification tasks.

7 Conclusion

The paper concludes that influence functions complement saliency maps for interpreting neural NLP models and can reveal and quantify data artifacts. It also identifies future work on standalone influence analysis, model improvement, and user trust.

  • Conclusion: The study compares gradient-based saliency maps and influence functions on sentiment analysis and NLI, finding broad consistency in sentiment analysis but not NLI.The authors posit that influence functions may better suit complex understanding tasks, while gradients may suffice for simpler sentiment analysis.
  • Conclusion: The paper introduces influence functions as a way to reveal and quantify data artifacts affecting model predictions.The conclusion notes that such artifacts are common in NLI.
  • Future work: Future work includes standalone analysis of influence rankings, using them to improve model performance, and studying whether these explanations establish user trust.The authors also seek to understand how users interpret the explanations.

A Implementation Details

Experiments use a partially fine-tuned BERT-Base model, vanilla gradient saliency maps, and influence functions implemented with LiSSA and damping for recursive convergence.

  • Model: The BERT-Base model freezes the embedding layer and first 8 transformer layers, fine-tuning only the last 4 layers and final projection layer.Training uses a learning rate of 5e−5, 3 epochs, sequence length 128, and batch size 32.
  • Interpretation methods: Gradient-based saliency maps use a vanilla implementation from Wallace et al. (2019).
  • Interpretation methods: Influence functions adapt Koh and Liang’s code to PyTorch and use LiSSA with damping 3e−3 and recursion depth 2500.The damping value supports convergence of the recursive inverse Hessian-vector approximation within reasonable time.
Loading 2005.06676v1…