Combining Feature and Instance Attribution to Detect Artifacts

1 Introduction

• The paper’s motivation is that deep networks may exploit training-data correlations that do not represent the intended task.Such artifacts can make model behavior appear successful without reflecting the relationship the task is intended to capture.Source: §1 · p. 1
• The paper evaluates feature attribution, which highlights input features, and instance attribution, which retrieves training examples influential to a prediction.These model-centric approaches complement data-centric artifact-discovery work (Gardner et al., 2021).Source: §1 · p. 1
• TFA highlights influential features inside training examples, addressing feature attribution’s focus on the test input and instance attribution’s need to inspect whole examples.For “yo! that’s sick,” TFA can highlight “!” in an influential training example, “shut up!”; the passage describes an example involving AAVE and toxicity.Source: §1 · pp. 1–2
• The authors report that training-feature attribution can outperform other methods for artifact detection and identify previously unreported artifacts in a few datasets.A small user study also found that combining feature and instance attribution was more effective than using either alone.Source: §1 · p. 2

2 Background and Notation

• The setup represents a text classifier as φ, mapping inputs x_i to labels y_i, with a linear layer on BERT fine-tuned on training set D using cross-entropy loss.Each training instance is z_i = (x_i, y_i), and each input is a token sequence.Source: §2 · p. 2
• Feature attribution highlights important tokens in a test sample; this work considers gradient-based feature attribution.The passage also names input gradients and LIME as examples of feature-attribution methods.Source: §2 · p. 2
• Instance attribution scores training samples for their influence on a test prediction, using formal approximations or heuristic methods.These methods can require practitioners to infer which patterns in retrieved instances mattered.Source: §2 · p. 2

3.1 What is an Artifact?

• A dataset artifact is a feature-label correlation in training distribution D that is absent from the idealized target distribution D*.The paper frames D* as the task distribution one would ideally learn from.Source: §3.1 · p. 2
• The paper also considers artifacts model-centered: a model prediction may differ from the ideal conditional distribution because the model exploits an artifact.This can be examined for a specific prediction.Source: §3.1 · p. 2
• The paper distinguishes granular artifacts, such as individual tokens, from abstract patterns, such as lexical overlap in NLI.The distinction is between discrete input units and higher-level patterns.Source: §3.1 · p. 3

3.2 Training-Feature Attribution

• Training-feature attribution highlights features in influential training instances, so practitioners need not infer manually what made a retrieved example important.Unlike feature attribution, it can surface a training pattern absent from the test input; unlike instance attribution, it can summarize influence without inspecting entire examples.Source: §3.2 · p. 3
• One TFA variant uses the gradient of instance influence with respect to training-input features, then uses heatmaps for abstract artifacts or aggregation for granular ones.Heatmaps show token importance in influential examples; the paper describes presenting the top and bottom k examples.Source: §3.2 · p. 3
• Aggregated token analysis ranks tokens from the most and least influential training examples by frequency, while a discriminator uses logistic regression over bag-of-words features.The discriminator’s token weights indicate association with influence for a given test point.Source: §3.2 · p. 3

4 A Procedure for Artifact Discovery

• Start by constructing a validation set, using a standard split or a deliberately difficult sample set; making it useful for dataset debugging is the biggest challenge.The procedure applies feature, instance, and training-feature attribution to its examples.Source: §4 · p. 3
• For granular artifacts, aggregate important features across validation examples; for abstract artifacts, inspect heatmaps of influential training examples.The figure describes explaining each validation prediction, then using frequency aggregation or heatmap analysis.Source: §4 · pp. 3–4, Figure 2 · p. 4
• Verify candidate artifacts by manipulating validation data and observing the effects on model outputs.Token rankings for feature attribution and TFA can be aggregated across instances without thresholding scores per instance.Source: §4 · p. 4
• The procedure rediscovered known artifacts and identified new ones in widely used NLP benchmarks.The passage points to Table 1 for these findings.Source: §4 · p. 4

• The study uses MNLI, IMDB sentiment, BoolQ, and DWMW17 as text-classification case studies.Source: §5 · p. 4
• BERT-based models reach accuracy within ∼1% of prior reported performance; gradient-based instance attribution considers only BERT’s last layer.Source: §5 · p. 4
• The attribution baselines are RIF and Euclidean Similarity for instances, and Gradients and Integrated Gradients for features.For RIF, the setup uses the TFA aggregated-token approach but assigns uniform importance to document tokens.Source: §5 · p. 4
• PMI and competency score provide dataset-centered comparisons, but are limited to granular artifact detection because they cannot show model exploitation or reveal abstract artifacts.Source: §5 · p. 4
• Direct influence-function instance attribution can be prohibitively expensive, so TFA uses efficient heuristic strategies and inherits known issues from its component techniques.Source: §5 · p. 4

6 Case Studies

• Across the case studies, the authors report recovering known artifacts and identifying previously unknown artifacts using attribution methods.Table 1 summarizes which methods apply to the investigated artifacts.Source: §6 · p. 4

6.1 Known Granular Artifact: Sentiment Analysis with IMDB Ratings

• On IMDB, training-feature attribution surfaced numerical ratings in top-token lists more often than feature attribution, but the comparison with PMI and competency is not direct.TFA uses a validation set containing the artifact, whereas PMI and competency rely solely on training data.Source: §6.1 · p. 5

6.2 Known Abstract Artifact: Natural Language Inference with HANS

• For HANS lexical overlap examples, TFA variants achieved fair-to-good performance, with similarity-based instance attribution appearing better at detecting the artifact.The analysis treats a training token appearing in both premise and hypothesis as evidence of overlap.Source: §6.2 · p. 6

6.3 Unknown Granular Artifact: Bias in Hate Speech Detection

• For hate-speech misclassifications, instance attribution did not make anomalies readily identifiable, while feature attribution surfaced punctuation and other tokens as candidate artifacts.The reported feature list, excluding hate-speech lexicon terms, includes ., you, @, the, :, and &.Source: §6.3 · p. 6
• Aggregated training-feature attribution surfaced @, white, trash, !, you, and is among tokens associated with toxicity predictions.PMI and competency also ranked is, ., trash, and the highly.Source: §6.3 · p. 6
• Replacing punctuation tokens with [MASK] produced substantially more prediction flips than masking a random token; the random-token baseline yielded a 1.8% label-flip rate on average over 10 runs.Source: §6.3 · p. 7, Table 3 · p. 7

6.4 Unknown Abstract Artifact: Structural Bias in BoolQ

• For BoolQ mispredictions, influential training examples often shared query structure with the test instance, and “same” correlated with influence for half of the examined examples.The latter analysis considered test queries containing “same.”Source: §6.4 · p. 7
• BoolQ paraphrase tests found 10% of predictions flipped for “Is X and Y the same?” and 23% for “Is X different from Y?”, suggesting query structure may not track meaning.The authors conclude the model might use query structure in a manner that does not track its meaning.Source: §6.4 · p. 7

7 User Study

• The study’s query structure was not correlated with a specific dataset label and therefore did not exactly meet the paper’s operational definition of an artifact.The authors excluded purely data-centered methods because the study targeted debugging specific test-set mispredictions.Source: §7 · pp. 7–8
• The study simulated adjective, first-name, and pronoun artifacts by making them label-correlated in training and validation data but not in the test set.For example, positive training reviews received adjective artifacts, while test artifacts were added to both positive and negative samples.Source: §7 · p. 8
• Nine graduate students experienced with similar models examined 10 reviews per method-artifact pairing and could query the model to verify suspected artifacts.The study recorded model calls and engagement time alongside users’ artifact and label judgments.Source: §7 · p. 8
• Users identified artifacts better with TFA, but spent the most time and made the most model calls; instance attribution had the fewest calls and least time.The authors suggest TFA's interaction cost may reflect the effort of inferring artifacts from influential training features, while users often gave up early with instance attribution.Source: §7 · p. 8

Artifact Discovery

• Prior studies examined artifacts through datasets, analyses of their effects, instance attribution to surface artifacts and reduce bias, or interpretability evaluation; only Han et al. (2020) pursued methodical artifact detection.Source: §8 · p. 8
• Han et al. (2020) used influence functions to extract lexical overlap from HANS, assuming influential training instances exhibit a known artifact.The authors note this approach retains limitations of instance attribution alone and assumes the artifact is known in advance.Source: §8 · p. 9

Features of Training Instances

• Where Koh and Liang (2017) approximated training-feature influence for adversarial attacks and defenses, this paper applies TFA to artifact identification and introduces a broader set of methods.Source: §8 · p. 9

9 Conclusions

• The paper defines artifacts as spurious feature-label correlations arising from incidental data-collection properties, which can mislead benchmark performance and harm generalization.Source: §9 · p. 9
• TFA showed advantages over feature and instance attribution in artifact discovery, recovered previously reported artifacts, surfaced previously unreported artifacts on some datasets, and best facilitated identification in a small user study.Source: §9 · p. 9
• The approach’s biggest caveat is reliance on a good validation set to compute training-instance and feature influence; constructing useful challenge sets remains a future-work direction.Source: §9 · p. 9

Broader Impact Statement

• Because deployed pretrained models can exhibit failure modes and biases, the paper frames better artifact-identification tools as one way to help practitioners mitigate risks.Source: Broader Impact Statement · p. 9
• The evaluated hybrid methods might eventually help practitioners deploy more robust and fairer models, but the authors caution that no method is foolproof.Source: Broader Impact Statement · p. 9
• The discussion leaves open whether a model’s deployment benefits justify potential harms when its behavior is not fully understood.Source: Broader Impact Statement · p. 9

A Experimental Setup

• HANS supplies 1,000 test instances with known NLI artifacts such as lexical overlap, while 10,000 MNLI instances serve as training data.Source: A Experimental Setup · p. 12
• The experiments also examine DWMW17, with 25K labeled tweets, and BoolQ, with 16K yes/no question-answer pairs, as benchmarks with previously unreported artifacts.Source: A Experimental Setup · p. 12
• IMDB: 93.2%, DWMW17: 91.1%, and BoolQ: 77.5% were the final model accuracies.Source: A Experimental Setup · p. 12
• The classifier uses BERT with a linear model on top, with hyperparameters tuned on validation data via grid search.Source: A Experimental Setup · p. 12
• Token gradients are computed with respect to embeddings and averaged, indicating positively and negatively influential tokens in predictions.The gradient is taken for the pre-softmax output or instance attribution score.Source: A Experimental Setup · p. 12

B User Study

• User-study artifacts included sampled adjectives, names, and pronouns; Figure 3 shows the study interface.Source: B User Study · p. 12, Figure 3 · p. 13