Source-linked AI summary
An Empirical Comparison of Instance Attribution Methods for NLP
Pouya Pezeshkpour, Sarthak Jain, Byron C. Wallace, Sameer Singh
TL;DR
Instance attribution could help explain model predictions by identifying relevant training examples, but approximating influence functions can be prohibitively expensive. This paper compares simpler similarity-based methods with gradient-based attribution through ranking, explanation-quality, and artifact-detection evaluations. Similarity methods select different examples from gradient methods yet prove fairly competitive in the reported evaluations.
Problem
Approximating influence functions to identify training examples relevant to a prediction can be prohibitively expensive, raising the question of whether simpler similarity retrieval performs comparably.
Source: §1 · pp. 1–2Method
The paper compares similarity-based retrieval with gradient-based attribution using ranking correlations, leave-some-out and randomized-test analyses, and HANS artifact detection.
Source: §1 · p. 2Results
Takeaways & Limitations
The comparisons suggest future work on fast and useful instance-attribution methods.
Source: §5 · p. 5For modern medium-to-large models such as BERT, computing IF over all parameters is effectively infeasible; last-layer IF has an O(p2) inverse-Hessian bottleneck, while similarity measures can be calculated in O(p).
Source: §4 · p. 5
Abstract
from arXiv · showhide
Widespread adoption of deep models has motivated a pressing need for approaches to interpret network outputs and to facilitate model debugging. Instance attribution methods constitute one means of accomplishing these goals by retrieving training instances that (may have) led to a particular prediction. Influence functions (IF; Koh and Liang 2017) provide machinery for doing this by quantifying the effect that perturbing individual train instances would have on a specific test prediction. However, even approximating the IF is computationally expensive, to the degree that may be prohibitive in many cases. Might simpler approaches (e.g., retrieving train examples most similar to a given test point) perform comparably? In this work, we evaluate the degree to which different potential instance attribution agree with respect to the importance of training samples. We find that simple retrieval methods yield training instances that differ from those identified via gradient-based methods (such as IFs), but that nonetheless exhibit desirable characteristics similar to more complex attribution methods. Code for all methods and experiments in this paper is available at: https://github.com/successar/instance_attributions_NLP.
1 Introduction
Instance attribution seeks to reveal which training examples matter to a prediction, complementing feature-based explanations that focus on input parts. The section asks whether simpler similarity methods can provide useful explanations without the computational cost of influence functions.
1 Introduction
- The motivating gap is that feature-based explanations highlight input parts but do not communicate training data as a basis for model outputs.Prior interpretability methods are described as focusing on feature-based explanations.Source: §1 · p. 1
- The study compares influence functions and variants, Representer Points, and similarity measures as methods for scoring training examples for a prediction.Figure 1 depicts the methods as assigning importance scores to training examples.Source: Figure 1 · p. 1, §1 · p. 1
- Influence functions can be prohibitively expensive to approximate, motivating the question of whether simpler similarity-based retrieval can identify comparably important training points.The evaluation asks both how rankings correlate and how explanation quality compares.Source: §1 · pp. 1–2
- The experiments evaluate whether similarity methods yield useful explanations and rankings that agree with more complex gradient-based methods.The two research questions concern ranking correlation and relative explanation quality.Source: §1 · pp. 1–2
- The paper evaluates methods on SST-2 and MNLI, compares explanation quality, and tests whether similarity methods can surface HANS training-data artifacts.The explanation-quality analyses include leave-one-out and randomized-test evaluations.Source: §1 · p. 2
2 Attribution Methods
The section defines similarity-based scores in embedding space and describes influence functions and related gradient-based alternatives. It also introduces Representer Points and untuned sentence-BERT representations as comparison methods.
2 Attribution Methods
- Similarity-based attribution scores a training example by its representation’s Euclidean distance, cosine similarity, or dot product with the target representation.The representations are taken from the model’s penultimate layer.Source: §2 · p. 2
- The study also ranks examples using similarities between untuned sentence-BERT representations to examine fine-tuning’s effect on similarity measures.The cited representations are from sentence-BERT.Source: §2 · p. 2
- Influence functions estimate how upweighting a training instance changes model parameters, then use that change to estimate its effect on a specific test prediction.The parameter-change estimate uses the inverse Hessian of the loss and the training-instance gradient.Source: §2 · p. 2
- The paper does not assume that similarity measures actual influence; it tests whether their rankings agree with more complex methods and identify useful training points.This distinction motivates evaluating similarity as a simple retrieval strategy.Source: §2 · p. 2
- The gradient-based comparisons include RIF, gradient dot product, and gradient cosine; the latter two are variants without Hessian information.The section identifies RIF as proposed by Barshan et al. (2020), and GD and GC by Charpiat et al. (2019).Source: §2 · p. 2
- Representer Points express a target instance’s output as a linear decomposition of training-instance data-importance contributions.This formulation is described for a feature extractor combined with an L2-regularized linear layer.Source: §2 · p. 2
3 Experimental Setup
The experiments use SST-2 and MNLI classification with BERT, alongside HANS examples for artifact detection. The setup also limits the parameters used for influence-function approximations because of their computational cost and instability.
3 Experimental Setup
- The experiments use binarized SST-2 and MNLI, randomly sampling 10,000 MNLI training instances and 1,000 HANS instances for artifact detection.SST-2 is the sentiment task, MNLI the NLI task, and HANS supplies examples for testing artifact discovery.Source: §3 · p. 2
- BERT models tuned by validation-set grid search achieve 90.6% accuracy on SST and 71.2% on MNLI.These are the reported accuracies of the experimental models.Source: §3 · p. 2
- Computing the full BERT inverse Hessian is impractical, so the study approximates inverse-Hessian vector products with LiSSa and restricts IF to the top five or last linear layer.The full matrix would require approximately 12 PB of storage; LiSSa remains expensive and becomes increasingly unstable as more layers are included.Source: §3 · pp. 2–3
- Figure 2 reports that Hessian use does not change training-example ordering, adding BERT layers to IF changes it little, and nearest-neighbor rankings correlate poorly with gradient-based ones.The reported pairwise similarities use Spearman correlation on SST and MNLI.Source: Figure 2 · p. 3
4 Experiments
The experiments compare attribution rankings, test explanation quality through sample removal and randomized-model correlations, and examine whether methods recover lexical-overlap artifacts. Similarity methods are fairly competitive, while gradient normalization affects consistency and the examples surfaced.
4 Experiments
- The study evaluates attribution methods via average Spearman correlations over 100 test and 500 training samples, leave-some-out retraining after removing most influential samples, trained-versus-random-model ranking correlations, and HANS lexical-overlap recovery.Source: §4 · p. 3
Attribution Methods’ Correlation
- Similarity-based rankings have only modest correlation with gradient-based methods, while gradient methods with and without normalization appear similar to one another.IF and GD are similar, as are RIF and GC; IF over BERT’s top five layers also correlates highly with IF over only the last linear layer.Source: §4 · p. 3
Removing ‘Important’ Samples
- On both tasks, at least one similarity-based approach performs comparably or better than gradient-based methods when top-ranked samples are removed.The comparison measures reductions in the predicted probability of the original prediction; normalized gradient methods appear more consistent.Source: §4 · p. 4
Randomized-Test
- On SST, IF, REP, and GD have negative trained-versus-random ranking correlations; overall, unnormalized gradient methods may be inferior by this measure.NN-DOT has the best performance according to this metric.Source: §4 · p. 4
Artifacts and Attribution Methods
- Similarity-based methods surface higher lexical-overlap examples than gradient-based methods for MNLI test instances mispredicted as entailment.Unnormalized IF, GD, and REP perform similarly to random selection and sentence-BERT similarity baselines on this artifact-focused analysis.Source: §4 · pp. 3–5
Computational Complexity
- Similarity-based approaches require < 0.0001 seconds per test sample, compared with ∼5 minutes for top-five-layer IF on SST.The reported estimates extrapolate to about six days for top-five-layer IF across 1,821 SST test samples versus around 0.2 seconds for similarity methods.Source: §4 · p. 5
5 Conclusions
The paper compares simpler instance-attribution approaches with more complex gradient-based methods. It reports strong agreement among some simpler gradient-based variants and finds similarity methods competitive in evaluation.
5 Conclusions
- Approximating influence functions to quantify training-sample importance is prohibitively expensive.The conclusion frames simpler, more efficient alternatives as the subject of the investigation.Source: §5 · p. 5
- Gradient-based methods using more parameters correlate highly with simpler counterparts, as do methods with and without Hessian information.The comparisons are IF and GD (top-5) versus IF and GD (linear), and IF versus GD and RIF versus GC.Source: §5 · p. 5
- Similarity-based methods are surprisingly competitive with gradient-based methods in leave-some-out, randomized-test, and artifact-detection experiments.The authors suggest this points toward future work on fast and useful instance-attribution methods.Source: §5 · p. 5
6 Ethical Considerations
The paper links model opacity to concerns about predictions based on unwanted attributes, including demographic categories or data artifacts. It presents instance attribution as a possible mechanism for addressing these concerns and compares complex and simpler approaches to characterize their tradeoffs.
6 Ethical Considerations
- Opaque models may make predictions based on unwanted attributes such as demographic categories or data artifacts.The paper notes that it can be difficult to know why deep models make specific predictions.Source: §6 · p. 5
- Instance attribution identifies training samples that influenced a given prediction and might help address these concerns.The paper contrasts complex approaches with simpler alternatives to characterize tradeoffs in attribution techniques.Source: §6 · p. 5
- The authors suggest these comparisons may, in turn, improve model robustness and potentially reduce implicit biases in predictions.These are presented as possible downstream benefits, not demonstrated outcomes.Source: §6 · p. 5
A Experimental Details
Experiments cover sentiment analysis and natural language inference, with a separate HANS sample used to examine annotation artifacts. The evaluated models use BERT and achieve different reported accuracies on the two main tasks.
A Experimental Details
- The experiments use SST-2 with 6,920 training and 1,821 test samples, and MNLI with 393k premise-hypothesis pairs across 10 genres.MNLI fine-tuning uses a random sample of 10k training instances.Source: A Experimental Details · p. 7
- A random sample of 1,000 HANS examples serves as test data for analyzing attribution methods’ ability to reveal NLI annotation artifacts.HANS examples exhibit previously identified artifacts such as lexical overlap between hypotheses and premises.Source: A Experimental Details · p. 7
- 90.6% accuracy on SST and 71.2% accuracy on MNLI are reported for the final BERT-based models.Hyperparameters are tuned on validation data by grid search.Source: A Experimental Details · p. 7
B Attribution Methods’ Correlation
The reported correlation results distinguish between the gradient-based variants and sentence-BERT similarity approaches. IF (linear) has high top-sample agreement with IF (top-5), while sentence-BERT similarity approaches show very small correlation with other methods.
B Attribution Methods’ Correlation
- Sentence-BERT similarity-based approaches show very small correlation with other attribution methods.The complete Spearman-correlation results, including sentence-BERT, are provided in Figure 3.Source: B Attribution Methods’ Correlation · p. 7
- IF (top-5) and IF (linear) have very high correlation among their top retrieved samples.The authors describe this as validating the high quality of the simpler IF (linear) version relative to IF (top-5).Source: B Attribution Methods’ Correlation · p. 7
- Figure 4 reports the proportion of shared examples among the top samples retrieved by IF (top-5) and IF (linear).The accompanying text describes their top-sample correlation as very high.Source: B Attribution Methods’ Correlation · p. 7
C Removing ‘Important’ Samples
The experiment tests whether removing highly ranked training instances changes predictions, and relates shared top-ranked examples to similar leave-some-out performance across some method pairs.
C Removing ‘Important’ Samples
- For each of 50 random test samples from MNLI and SST, the experiment removes the top 50 or 500 training instances and retrains the model.The authors then calculate the change in the prediction for that test sample.Source: C Removing ‘Important’ Samples · p. 7
- Very high top-example overlap for IF vs GD, RIF vs GC, and NN-EUC vs NN-COS helps explain these pairs’ similar leave-some-out performance.The overlap is depicted in Figures 5 and 6.Source: C Removing ‘Important’ Samples · p. 7
- The reported average is the prediction change across the selected 50 test samples.The proportion of shared examples between method pairs is also shown for top-10 and top-50 samples.Source: C Removing ‘Important’ Samples · p. 7
D Near Training Samples Explanations
The analysis tests attribution methods on targets that are training samples or lightly modified versions of them. Similarity-based methods recover the corresponding original samples better, while some gradient-based methods repeatedly rank high-gradient training points first.
D Near Training Samples Explanations
- Targets include original training samples, samples with a random token added, removed, or replaced, with MNLI modifications applied to both premise and hypothesis.The four target scenarios are original samples, random-token addition, random-token removal, and random-token replacement.Source: D Near Training Samples Explanations · p. 7
- Similarity-based methods recover the original training samples better across the training-sample targets and their modifications.The analysis results are reported in Tables 3 and 4 for SST and MNLI.Source: D Near Training Samples Explanations · p. 7, Table 3 · p. 9, Table 4 · p. 9
- IF, GC, and REP perform very poorly because they select high-magnitude-gradient training points as top instances for any target sample.This is the authors’ explanation for these methods’ low recovery performance.Source: D Near Training Samples Explanations · p. 7