PaperlayerView exact revision on arXiv

Source-linked AI summary

Debugging Tests for Model Explanations

Julius Adebayo, Michael Muelly, Ilaria Liccardi, Been Kim

arXiv:2011.05429v1cs.CVcs.LG

TL;DR

The paper investigates whether post-hoc explanations can diagnose model errors, addressing uncertainty about which methods work for which bug types. It categorizes contamination sources, evaluates feature attributions and human debugging, and finds success for spurious backgrounds but failures for some mislabeled, model, and human-identification cases.

  • Problem

    It remains unclear which explanation methods are effective for which classes of model bugs, despite increasing use of explanations in high-stakes applications.

  • Method

    The authors categorize bugs as data, model, or test-time contamination and evaluate feature attribution methods across four bugs plus a human subject study.

  • Results

    Attributions identify spurious background artifacts but not conclusively mislabeled examples; positive-aggregation methods are invariant to higher-layer parameters, and users primarily rely on predictions rather than attributions.

  • Takeaways & Limitations

    The findings provide guidance for researchers and practitioners using feature attributions for model debugging and support more rigorous evaluation of explanation utility.

  • Takeaways & Limitations

    The bug characterization covers only the standard supervised-learning pipeline, may not capture bugs caused by combined factors, and the study focuses only on feature attributions.

Abstract

from arXiv · show

We investigate whether post-hoc model explanations are effective for diagnosing model errors--model debugging. In response to the challenge of explaining a model's prediction, a vast array of explanation methods have been proposed. Despite increasing use, it is unclear if they are effective. To start, we categorize \textit{bugs}, based on their source, into:~\textit{data, model, and test-time} contamination bugs. For several explanation methods, we assess their ability to: detect spurious correlation artifacts (data contamination), diagnose mislabeled training examples (data contamination), differentiate between a (partially) re-initialized model and a trained one (model contamination), and detect out-of-distribution inputs (test-time contamination). We find that the methods tested are able to diagnose a spurious background bug, but not conclusively identify mislabeled training examples. In addition, a class of methods, that modify the back-propagation algorithm are invariant to the higher layer parameters of a deep network; hence, ineffective for diagnosing model contamination. We complement our analysis with a human subject study, and find that subjects fail to identify defective models using attributions, but instead rely, primarily, on model predictions. Taken together, our results provide guidance for practitioners and researchers turning to explanations as tools for model debugging.

1 Introduction

This work asks which explanation methods can effectively diagnose different classes of model bugs. It categorizes contamination sources, evaluates attribution methods and human debugging, and finds substantial task- and method-specific limitations.

  • Model debugging is increasingly important because learned systems are deployed in high-stakes settings where errors can have devastating consequences.
  • The paper categorizes supervised-learning bugs into data, model, and test-time contamination based on where defects arise.The categories cover training data, model specification and parameters, and test-time inputs.
  • The authors assess feature attribution methods against spurious correlation artifacts, mislabeled training examples, re-initialized weights, and out-of-distribution shift.
  • Attributions identify spurious background artifacts but do not conclusively distinguish normal from mislabeled training examples; positive-aggregation methods are invariant to higher-layer parameters.
  • In a 54-person study, users primarily relied on model predictions rather than attributions when identifying defective models.
  • The work builds on prior evidence that explanation quality rankings can be inconsistent and that human studies are a gold standard for evaluating explanation effectiveness.

2 Bug Characterization, Explanation Methods, & User Study

The paper defines contamination bugs across the supervised-learning pipeline, surveys attribution methods, and designs evaluations for automated and human debugging. Its framework covers data, model, and test-time defects and represents explanations as input-feature relevance scores.

  • 2.1 Characterizing Model Bugs.: Model bugs are contamination in the learning or prediction pipeline that causes incorrect predictions or error-causing associations.
  • 2.1 Characterizing Model Bugs.: Data contamination affects training inputs or labels, model contamination affects parameters, and test-time contamination affects inputs through domain shift or preprocessing mismatch.
  • 2.1 Characterizing Model Bugs.: The categorization enables assessment of explanation methods against specific bug classes and instances formalized in Table 1.
  • 2.2 Explanation Methods: Feature attribution methods assign relevance scores to input dimensions, which can be visualized as heat maps for image-based deep neural networks.
  • 2.2 Explanation Methods: The study includes gradient-based, surrogate, and modified-back-propagation methods, comparing attribution maps with SSIM and Spearman rank correlation.
  • 2.3 Human Subject Study: The human study shows participants original images, predictions, and attribution maps, then asks them to rate model-sale recommendations on a 5-point Likert scale.

3 Debugging Data Contamination

Feature attributions expose spurious background correlations but do not conclusively reveal mislabeled training examples. In the human study, participants often failed to reject the spurious model despite attribution evidence, while they rejected mislabeled examples mainly using predictions.

  • Spurious Correlation Training Artifacts: Attributions placed emphasis on the spurious background in dog and bird inputs.The experiments used a BVD-CNN trained with birds on sky backgrounds and dogs on bamboo backgrounds.
  • Spurious Correlation Training Artifacts: SSIM-GT2 scores ranged from 0.78 to 0.98, supporting attribution sensitivity to the spurious background signal.Similar evidence was reported for GT-1; the table’s random-Gaussian calibration mean was 3e−06.
  • Spurious Correlation Training Artifacts: Spurious-model Likert medians were 2, 2, and 3 for Gradient, SmoothGrad, and Integrated Gradients, with some users still recommending it.A majority said the attributions did not highlight the expected image region, yet users did not convincingly reject the model.
  • Mislabelled Training Examples: Attributions for correctly and incorrectly labeled training examples had SSIM values between 0.73 and 0.99 across all tested methods.Their visual similarity indicates that the methods might be ineffective for identifying mislabeled examples.
  • Mislabelled Training Examples: Participants rejected mislabeled examples with median Likert scores of 1, 2, and 1, but overwhelmingly relied on model predictions.The methods’ attribution maps were not the primary basis for these decisions.

4 Debugging Model Contamination

The paper tests whether attributions can detect models whose parameters were re-initialized. Modified back-propagation methods were invariant to higher-layer parameters, while human participants rejected defective models primarily from classification labels rather than attributions.

  • Model Parameter Bugs: The study re-initialized VGG-16 weights successively from the top layer toward the first layer and compared resulting attributions with those of the original model.This setup directly tests sensitivity to partial and complete parameter re-initialization.
  • Results: Modified back-propagation methods were visually and quantitatively invariant to higher-layer VGG-16 parameters.The methods included Guided BackProp, Deconvnet, DeepTaylor, PatternNet, Pattern Attribution, and LRP-SPAF.
  • Human Subject Study: Participants conclusively rejected a model with a re-initialized top layer based on classification labels, rarely because of wrong attributions.This human-study result reinforces the limited diagnostic value of the attributions in this setting.

5 Debugging Test-Time Contamination

The study tests whether feature attributions can diagnose test-time contamination by comparing attributions for the same inputs across in-domain and out-of-domain models. Attributions can look visually similar across models, while feature-ranking agreement remains absent, and users primarily rely on prediction labels.

  • Test-time contamination: The experiment compares attributions for the same inputs from an in-domain model and models trained on different datasets.Fashion MNIST examples are compared with attributions from MNIST, birds-vs-dogs, and ImageNet models.
  • Results: Table 3 reports visual similarity but essentially no correlation in feature ranking for the compared attributions.The table presents SSIM and rank-correlation metrics for 190 examples.
  • Results: Visual similarity appears between in-domain Fashion MNIST attributions and attributions from other models, especially VGG-16 on ImageNet.The comparison covers four dataset-model pairs and reports results on Fashion MNIST.
  • Human subject study: In the human study, users primarily used prediction labels rather than attributions when assessing models under domain shift.Participants did not recommend the model because of wrong prediction labels.

6 Discussion & Conclusion

The paper evaluates feature attributions as tools for debugging model errors across data, model, and test-time contamination. It finds important diagnostic gaps and frames the results as guidance for more rigorous evaluation of explanation utility.

  • Discussion: The paper categorizes model bugs into data, model, and test-time contamination and evaluates feature attribution methods against these categories.The study focuses on spatial spurious correlation, mislabeled examples, model contamination, and out-of-domain inputs.
  • Findings: Feature attributions diagnose the tested spatial spurious-correlation bug but do not conclusively distinguish mislabeled examples from normal ones.This conclusion concerns the specific bugs and attribution methods studied.
  • Findings: Positive-aggregation attributions computed with modified back-propagation are invariant to higher-layer parameters, limiting their usefulness for diagnosing model contamination.The paper presents this as a suggestion about the effectiveness of these approaches for model-contamination bugs.
  • Limitations: The study is limited to feature attributions and the standard supervised-learning pipeline, which may not capture combined-factor bugs.The authors note that concept activation and training-point ranking may be better suited to some studied tasks.
  • Conclusion: The paper aims to guide researchers and practitioners using feature attributions for model debugging and to encourage more rigorous assessment of explanation utility.The authors describe these findings as a first step toward rigorous evaluation.

Acknowlegdements

The authors acknowledge contributors and funding sources supporting the work.

  • Acknowledgements: The authors thank several individuals and anonymous reviewers for feedback on the work.They also acknowledge MIT Quest for Intelligence cloud-computing credits and an Open Philanthropy Fellowship.

Part

The appendix formalizes additional bugs within the paper’s categorization and notes that other instantiations are possible.

  • Additional bug formalizations: Frozen layers and pre-processing mismatch are formalized as additional bugs beyond those discussed in the main document.Frozen layers keep a network layer fixed during training; pre-processing mismatch applies different test-time preprocessing than during training.
  • Scope of categorization: The Table 4 examples are specific instantiations, and other bugs can be proposed under the same categorization.The categorization is therefore not limited to the listed examples.

B Detailed Overview of Attribution Methods

The paper surveys gradient-based, surrogate, and modified back-propagation attribution methods, together with the metrics and implementations used to compare them.

  • Gradient & Variants: Gradient maps quantify output sensitivity to each input dimension, while SmoothGrad, SmoothGrad Squared, and VarGrad derive variants from noisy gradients.SmoothGrad uses 50 noisy inputs with noise standard deviation set to 0.15 * input range; VarGrad uses the analogous variance construction.
  • Gradient & Variants: Input-Grad multiplies gradients element-wise by the input, and Integrated Gradients sums gradients along a path from a baseline input.The Integrated Gradients baseline is the minimum value in each input dimension, often corresponding to an all-black image.
  • Gradient & Variants: Expected Gradients extends Integrated Gradients by averaging over multiple training-set baselines, using 200 training examples in this study.The paper notes that this construction is equivalent to Integrated Gradients with multiple baselines.
  • Surrogate Approaches: LIME and SHAP locally approximate the model around an input with a simple interpretable function.For image inputs, LIME uses 50 segments and 1000 model-fitting samples; SHAP also uses 1000 samples.
  • Modified Back-Propagation: Modified back-propagation methods redistribute output relevance through altered ReLU gradients or layer-specific relevance rules.The methods include Deconvnet, Guided Backpropagation, LRP variants, PatternNet, Pattern Attribution, and DeepTaylor.
  • Evaluation: Attribution comparisons use SSIM for visual similarity and Spearman rank correlation for feature-ranking similarity.The attribution functional maps an input, model, and output to an attribution map.

D.1 Data Contamination: Spurious Correlation Artifacts

The experiments examine contamination in data, model parameters, and test-time inputs, with spurious-background and mislabeled-example settings used for data contamination.

  • Spurious Correlation: The spurious model achieves 97 percent accuracy on backgrounds alone, showing that it uses sky and bamboo-forest signals.The same 97 percent accuracy is maintained when birds and dogs are inserted on the appropriate backgrounds.
  • Mislabeled Examples: The mislabeled-example bug flips labels for 10 percent of birds-vs-dogs training samples.The resulting model achieves 94.2, 91.7, and 88 percent accuracy on training, validation, and test sets, respectively.
  • Model Contamination: Model contamination tests study accidental model reinitialization and whether attributions distinguish different parameter regimes.The setup expects reinitialization to produce observable accuracy differences, while testing attribution-based identification separately.
  • Test-Time Contamination: Test-time contamination includes preprocessing mismatches and out-of-domain inputs received by a model trained in another setting.The paper treats out-of-distribution inputs as a representative test-time defect.

G Other Methods

Beyond feature attributions, the paper evaluates influence functions for training-point ranking and TCAV for concept-based analysis across selected bug settings.

  • Concept and Influence Functions: Influence functions are assessed under spurious correlation, mislabeled examples, and domain shift, while TCAV is tested under spurious correlation and model contamination.The paper focuses on these methods to broaden the inquiry while keeping it thematically centered on attribution-style explanations.

H User-Study Overview

The user study asks participants to assess animal-classification models using predictions and feature attributions across normal and contaminated conditions. Statistical analyses compare attribution methods and model manipulations.

  • Study Materials: Additional figures present the attribution combinations and model-condition examples used in the study.The supplementary visual materials include study interfaces, demographics, spurious-background examples, and mislabeled-example examples.
  • Study Task: Participants acted as quality-assurance testers for a hypothetical animal-classification company and rated whether models should be recommended for sale.They viewed original images, model predictions, and attribution maps, then supplied rationales and expertise information.
  • Methods: The study focused on Gradient, Integrated Gradients, and SmoothGrad as a representative subset selected after randomization tests.The methods fell into gradient-related methods and Integrated-Gradients-related methods in the broader comparison.
  • Participants: 54 participants completed the study, and more than 80% reported prior machine-learning experience.Participants were recruited through a medium-sized North-American university mailing list.
  • Task, Procedure, & Structure of Experiment: The task used 10 dog breeds and asked participants to evaluate model labels and attributions for each model condition.Participants received a guide to the breeds and study interface before evaluating the models.
  • Model Conditions: The tested conditions included a normal model, a randomly reinitialized top layer, randomized labels, spurious backgrounds, and out-of-distribution dog breeds.The normal model served as the control, while the other conditions represented model, data, and test-time contamination.
  • Data Analysis: The analysis used one-way ANOVA and post-hoc Tukey-Kramer tests to assess whether attributions affected participants’ ability to reject defective models.Within the normal condition, attribution maps differed significantly: F(2, 54) = 14.35, p < 0.0001.
  • Data Analysis: Within the spurious condition, attribution maps also differed significantly: F(2, 54) = 15.9, p < 0.0001.Participants were more likely to recommend Integrated Gradients (µ = 3.05) than SmoothGrad (µ = 1.98), with p < 0.0001.
Loading 2011.05429v1…