Debugging Tests for Model Explanations

Bug categorization

• The paper categorizes supervised-learning bugs by source into data, model, and test-time contamination and asks which explanation methods work for each class.These classes cover defects in training data, model specification or parameters, and test-time inputs.Source: §1 · p. 2, Figure 1 · p. 2

Empirical assessment

• Feature attribution methods identify spurious background bugs but do not conclusively distinguish normal from mislabeled training examples.The study also assesses re-initialized weights and out-of-distribution shift as representative bugs.Source: §1 · p. 2

Insights

• Positive-aggregation back-propagation methods are invariant to higher-layer parameters of deep neural networks, limiting their ability to diagnose model contamination.This finding concerns attribution methods that modify the back-propagation computation.Source: §1 · p. 2
• Attributions for out-of-distribution examples can visually resemble attributions produced by an in-domain model, making debugging based solely on visual inspection misleading in specific settings.The result highlights a limitation in detecting test-time contamination through attribution patterns alone.Source: §1 · p. 2

Human subject study

• In a 54-person IRB-approved study, users primarily relied on model predictions rather than attributions to identify defective models.The study evaluates whether end-users can identify defective models when attributions are available.Source: §1 · p. 2

2.1 Characterizing Model Bugs.

• Model bugs are contamination in supervised learning or prediction that causes incorrect predictions or error-causing associations, categorized as data, model, or test-time contamination.Data contamination affects features or labels, model contamination affects parameters, and test-time contamination affects inputs through domain shift or preprocessing mismatch.Source: §2.1 · pp. 3–4
• The study evaluates explanations against specific bug classes, including mislabeled or spuriously correlated training data, accidentally re-initialized model weights, and defective test inputs.These examples instantiate the section’s data, model, and test-time contamination categories and are assessed across explanation methods in Table 1.Source: §2.1 · p. 4, Table 1 · p. 4

2.2 Explanation Methods

• The explanation survey covers feature attributions, gradient variants, Integrated Gradients, Expected Gradients, LIME, SHAP, modified back-propagation methods, and similarity metrics.Attributions produce relevance scores or heat maps; comparisons use SSIM for visual similarity and Spearman correlation for feature-ranking similarity.Source: §2.2 · pp. 4–5

2.3 Overview of Human Subject Study

• The human study asked QA participants to judge animal-classification models using original images, predictions, and attribution maps, then recommend models on a five-point Likert scale.Participants evaluated four dog breeds at a time, provided rationales, and reported machine-learning expertise verified by three questions.Source: §2.3 · p. 5
• The user study focused on Gradient, Integrated Gradients, and SmoothGrad, testing models containing the Table 1 bugs alongside an unbiased model.These methods formed a representative subset selected for the human-subject evaluation.Source: Table 1 · p. 4, §2.3 · p. 5

3 Debugging Data Contamination

• On a birds-versus-dogs CNN task, attribution methods diagnose spurious backgrounds but not conclusively mislabeled training examples.The BVD-CNN uses five convolutional and three fully connected layers and achieves 94-percent test accuracy.Source: §3 · pp. 5–6

3.1 Spurious Correlation Training Artifacts

• SSIM scores for the relaxed spurious-background ground truth range from 0.78 to 0.98 across methods, with similar evidence for the ideal mask.The spurious model achieves 97 percent accuracy on sky-versus-bamboo backgrounds without birds or dogs, confirming that it learned the spurious association.Source: §3.1 · pp. 6–7, Table 2 · p. 7
• Participants were unsure about the spurious model’s reliability and did not outright reject it, suggesting that an attribution method’s ability to diagnose spurious correlation may not ensure reliable decision making.Source: §3 · p. 6, §3.1 · p. 8

3.2 Mislabelled Training Examples

• Attributions for correctly and incorrectly labeled versions of the same training examples are visually similar, with SSIM values from 0.73 −0.99 across all methods.This similarity indicates that the tested attribution methods might be ineffective for identifying mislabeled examples.Source: §3 · p. 6, §3.2 · p. 8, Figure 6 · p. 8
• Participants rejected mislabeled examples, but overwhelmingly relied on the model’s prediction rather than attribution methods.Median Likert scores were 1, 2, and 1 for Gradient, SmoothGrad, and Integrated Gradients, respectively.Source: §3.2 · p. 8

4 Debugging Model Contamination

• The evaluated bug concerns accidental model-weight re-initialization before prediction, particularly testing whether attribution methods respond to altered higher-layer parameters.The evaluated methods included Guided Back-Propagation, DConvNet, and variants of layer relevance propagation such as PatternNet and Pattern Attribution.Source: §4 · p. 8

Bug Implementation

• The bug was implemented by successively re-initializing VGG-16 weights from the top layer through the first layer and comparing each model’s attributions with the original.The experiment used a pre-trained VGG-16 model on ImageNet.Source: §4 · pp. 8–9

Results

• Modified back-propagation methods were visually and quantitatively invariant to higher-layer parameters in VGG-16, limiting their ability to diagnose model contamination.The methods included Guided BackProp, Deconvnet, DeepTaylor, PatternNet, Pattern Attribution, and LRP-SPAF; the result corroborates prior findings and rank-1 convergence analyses.Source: §4 · p. 9, Figure 7 · p. 9

Insights from Human Subject Study

• Participants conclusively rejected models with re-initialized top layers using classification labels, but rarely used incorrect attributions.This human-subject result indicates that users relied primarily on predictions rather than attribution methods.Source: §4 · p. 9

Bug Implementation

• The test compares Fashion MNIST attributions from a Fashion MNIST model with those from models trained on MNIST, birds-versus-dogs, and ImageNet.The study uses four dataset-model pairs and reports results on Fashion MNIST.Source: §5 · p. 10

Results

• Visual similarity coexists with essentially no feature-ranking correlation across in-domain and out-of-domain models, limiting attribution-based diagnosis of domain shift.This pattern was especially visible for VGG-16 trained on ImageNet when evaluated on Fashion MNIST inputs.Source: §5 · p. 10, Table 3 · p. 10, Figure 8 · p. 10

Insights from Human Subject Study

• In the human study, users rejected models for wrong prediction labels rather than relying on the displayed attributions.Participants evaluated dogs from breeds different from those used during training.Source: §5 · p. 11

Findings

• Feature attributions diagnose the tested spatial spurious-correlation bug but do not conclusively distinguish mislabeled examples, and some are ineffective for model contamination.Modified-backpropagation attributions using positive aggregation can be invariant to higher-layer parameters, limiting their diagnostic value.Source: §6 · p. 11
• Attributions for out-of-domain inputs can resemble those from an in-domain model, warranting caution when visually inspecting explanations, especially for image tasks.Source: §6 · p. 11

Human evaluation

• Human participants primarily relied on model predictions rather than feature attributions when diagnosing model bugs.This reliance may reflect the dog-breed classification task and may not generalize to tasks requiring less prior knowledge.Source: §6 · p. 11

Limitations

• The bug taxonomy covers standard supervised learning but may not capture bugs arising from combined factors, and the study evaluates only feature attributions.Concept activation and training-point ranking showed promising initial results, while users’ reliance on predictions may depend on task expertise.Source: §6 · p. 11

Conclusion

• The work provides guidance for using feature attributions in model debugging and a first step toward more rigorous evaluation of explanation methods.Source: §6 · p. 11

• The work received feedback from Hal Abelson, Danny Weitzner, Taylor Reynolds, and anonymous reviewers, cloud-computing credits from MIT Quest for Intelligence, and Open Philanthropy Fellowship support for JA.Source: Acknowlegdements · p. 12

A Additional Discussion of Bug Categorization Setup

• The appendix adds frozen layers and pre-processing mismatch as formalized bug categories in Table 4.Frozen layers keep a deep-network layer fixed during training, whereas pre-processing mismatch applies test-time input processing different from that used for training data.Source: A Additional Discussion of Bug… · p. 19
• Frozen layers accidentally leave one deep-network layer fixed during training, while pre-processing mismatch can cause wrong predictions when test inputs are processed differently from training data.Source: A Additional Discussion of Bug… · p. 19
• The Table 4 examples are specific instantiations, and other examples can be proposed under the same bug categorization.Source: A Additional Discussion of Bug… · p. 19

B Detailed Overview of Attribution Methods

• The implementations were re-created and benchmarked against public author-released packages, including innvestigate, LIME-IMAGE, and SHAP.Attribution comparisons use SSIM for visual similarity and Spearman correlation for feature-ranking similarity.Source: B Detailed Overview of Attribution… · pp. 20, 22
• The cited passages document Gradient, SmoothGrad Squared, VarGrad, Integrated Gradients, and a multiple-baseline variant, while specifying Gaussian-noise settings for VarGrad and baselines for integrated-gradient methods.VarGrad uses 50 noisy inputs with zero-mean Gaussian noise at scale 0.15 × input range; the multiple-baseline variant uses 200 training examples as baselines.Source: B Detailed Overview of Attribution… · pp. 20–21
• Surrogate explanations use LIME and SHAP to fit local models around individual inputs, with 1000 samples used for model fitting.LIME uses 50 image segments because this was more stable for the considered input sizes.Source: B Detailed Overview of Attribution… · p. 22
• Modified-backpropagation methods include Deconvnet, Guided Backpropagation, PatternNet, Pattern Attribution, DeepTaylor, and four LRP variants.LRP-Z is equivalent to Input-Grad for ReLU networks, while Pattern methods propagate estimated signal components.Source: B Detailed Overview of Attribution… · pp. 22–23

C.1 Datasets.

• Data-contamination experiments use a 10,000-image birds-versus-dogs dataset with segmentation masks, comprising 8,600 training examples and evenly split validation and test sets.The dataset combines dog breeds from Cats-v-Dogs with bird species from Caltech-UCSD.Source: §C.1 · p. 23
• Test-time contamination tests use three-channel modified MNIST and Fashion-MNIST with attributions derived from a VGG-16 model, while model-contamination tests use 200 ImageNet validation images.The user study instead restricts evaluation to 10 dog classes and constructs spurious correlations by replacing training-image backgrounds.Source: §C.1 · pp. 23–24

C.2 Models.

• The standard data-contamination model is a five-convolutional-layer, three-fully-connected-layer BVD-CNN trained with Adam for 40 epochs to 94-percent test accuracy.The user-study model is a fine-tuned ResNet-50 trained on the dog-only dataset.Source: §C.2 · p. 24

D.1 Data Contamination: Spurious Correlation Artifacts

• 97 percent spurious accuracy on backgrounds alone confirms that the BVD-CNN relies on sky and bamboo-forest signals.The same 97 percent accuracy persists when birds and dogs are inserted into their appropriate backgrounds.Source: §D.1 · p. 24
• The spurious-correlation test removes dogs and birds, isolating whether background signals drive the model’s predictions.The evaluation uses a test set containing only Sky and Bamboo-forest images.Source: §D.1 · p. 24

D.2 Data Contamination: Mislabelled Examples

• 10 percent of birds-versus-dogs training samples have flipped labels to implement the mislabeled-example bug.The contaminated BVD-CNN reaches 94.2, 91.7, and 88 percent accuracy on training, validation, and test sets, respectively.Source: §D.2 · p. 24
• The mislabeled-example experiment evaluates whether explanations can diagnose a model trained on corrupted supervision.Its reported accuracies are 94.2 percent training, 91.7 percent validation, and 88 percent test accuracy.Source: §D.2 · p. 24

E Model Contamination

• Model contamination tests target defects in model parameters, using accidental reinitialization during or after model use as the test setting.Such reinitialization is expected to produce observable accuracy differences.Source: E Model Contamination · p. 25
• The model-attribution objective is to determine whether explanations identify models operating in different parameter regimes, beyond ordinary accuracy changes.This frames parameter-regime identification as the central diagnostic question.Source: E Model Contamination · p. 25

F Test-Time Contamination

• Test-time contamination covers preprocessing mismatches between training and testing and out-of-domain inputs presented to a model.Additional figures address this contamination setting.Source: F Test-Time Contamination · p. 25

Concept and Influence Functions

• Influence functions (Koh and Liang (2017)) and TCAV (Kim et al. (2018)) were tested across spurious correlation, mislabeled examples, domain shift, and model contamination settings.TCAV was evaluated for spurious correlation and model contamination, while influence functions were assessed for spurious correlation, mislabeled examples, and domain shift.Source: G Other Methods · p. 25
• 91 percent (2.4 standard error) of the top 1 percent of training points were spurious across 50 background-correlated examples, indicating influence functions may detect spurious correlation.The top 1 percent comprised 86 training points.Source: G Other Methods · p. 25
• Self-influence required checking an average of 11 percent of the training set (5.9 standard error) across 50 examples to assess mislabeled data, but further validation is needed.TCAV background-concept rankings also indicated potential for identifying spurious correlation, while the authors caution that both methods require additional empirical assessment.Source: G Other Methods · p. 25

H User-Study Overview

• Participants judged models from original images, predictions, and attribution maps, rating sale-readiness on a 5-point Likert scale and explaining their decisions.Each participant answered 21 unique questions plus one attention-check repeat; 54 participants completed the study.Source: H User-Study Overview · pp. 25–27
• The study used Gradient, SmoothGrad, and Integrated Gradient to represent broad attribution-method classes in debugging tests.Selection was based on generality, prior use in model debugging, and resistance to manipulation, with randomization tests narrowing the candidate methods.Source: H User-Study Overview · p. 26
• The experiment covered a normal control, top-layer randomization, randomized labels, spurious backgrounds, and out-of-distribution dog breeds in a 10-class dataset.The dataset contained 4,000 images across ten breeds, split IID into 350 training, 25 validation, and 25 test images per class.Source: Datasets · pp. 27, 29, 30
• Attribution choices significantly affected recommendations for normal, random-label, and spurious models, but not out-of-distribution or top-layer models.For normal models, Gradient (µ = 3.88) and SmoothGrad (µ = 3.77) exceeded Integrated-Gradients (µ = 2.805); for random labels, Integrated-Gradients (µ = 1.94) exceeded SmoothGrad (µ = 1.31); for spurious models, Integrated-Gradients (µ = 3.05) exceeded SmoothGrad (µ = 1.98) and Gradient (µ = 2.35).Source: Datasets · pp. 30–31

I Collection of Additional Figures

• Additional figures show spurious and mislabeled examples, VGG-16 model-contamination randomization, test-time randomization across datasets, and images used in the user study.The VGG-16 quantitative figure reports SSIM, rank correlation, and normalized attribution differences, while noting that Expected Gradients was omitted because it was computationally intensive.Source: Figure 18 · p. 31, Figure 21 · p. 33, Figure 26 · p. 35
• The collection includes images used as part of the user study.Source: Figure 63 · p. 63, Figure 64 · p. 64