PaperlayerView exact revision on arXiv

Source-linked AI summary

Sanity Checks for Saliency Maps

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim

arXiv:1810.03292v3cs.CVcs.LGstat.ML

TL;DR

Saliency methods are popular, but visual inspection alone does not establish whether explanations reflect a model or its training data. The paper introduces randomization-based sanity checks and finds that some methods are insensitive to both, limiting their use for model- or data-sensitive explanation tasks.

  • Problem

    The paper addresses the lack of principled ways to assess whether saliency explanations reflect the learned model and the relationship between inputs and outputs in training data.

  • Method

    The authors evaluate explanation methods with model-parameter and data-randomization tests, supported by experiments and analyses of simple models and edge detectors.

  • Results

    Some saliency methods are independent of both model parameters and the data-generating process, while visual inspection can make them appear relevant.

  • Takeaways & Limitations

    Methods failing the tests are inadequate for tasks requiring explanations faithful to the model or data-generating process.

  • Takeaways & Limitations

    The paper primarily investigates model-randomization and label-randomization invariances and presents its findings as a step toward broader evaluation rather than a verdict on existing methods.

Abstract

from arXiv · show

Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.

1 Introduction

The paper proposes actionable randomization tests for evaluating whether explanation methods depend on a model and its training data, finding that visual appeal alone can be misleading.

  • Motivation: The authors identify a methodological challenge: practitioners lack principled guidance for assessing the scope and quality of model explanations.Saliency methods are increasingly used to highlight relevant input features and may support debugging, regulatory requirements, and detection of unintended effects.
  • Method: The paper proposes easy-to-implement model-parameter and data-randomization tests for evaluating explanation methods.The methodology compares explanations under randomized model parameters or permuted training labels with explanations from the original setting.
  • Results: Some widely deployed saliency methods are independent of both model parameters and the data used to train the model.The finding comes from experiments spanning numerous saliency methods, model architectures, and data sets.
  • Implications: Visual similarity to an edge detector can make saliency maps appear relevant even when the method is insensitive to the underlying model and data.Because edge detectors use neither model parameters nor training data, visual inspection alone is a poor guide to explanation sensitivity.
  • Results: Guided Backpropagation and Guided GradCAM fail the sanity checks, whereas Gradients and GradCAM pass them.The failing methods are invariant to higher-layer parameters in the reported tests.
  • Implications: Methods that fail the tests cannot support tasks requiring explanations faithful to the model or the data-generating process.Such tasks include model debugging and explaining relationships between inputs and outputs encoded in the data.

2 Methods and Related Work

The paper situates saliency explanations within a broader family of attribution methods, reviews common techniques and assessment approaches, and emphasizes their differing computational constructions and vulnerabilities.

  • Formal setup: An explanation method maps an input vector to an explanation object with the same shape, while the model maps inputs to class scores.The formal setup represents inputs as x ∈ R^d and model outputs as S: R^d → R^C.
  • Gradient methods: Gradient explanations measure how changes in input dimensions affect model predictions locally.Gradient-based explanations use the derivative of the score with respect to the input.
  • Gradient methods: Gradient ⊙Input multiplies each input dimension by its gradient to address gradient saturation and reduce visual diffusion.This method is presented as a variant of the basic gradient explanation.
  • Gradient methods: Integrated Gradients sums gradients along scaled versions of the input relative to a baseline representing feature absence.The baseline input is denoted by x̄ in the supplied description.
  • Backpropagation methods: Guided Backpropagation sets negative gradient entries to zero while back-propagating through ReLU units.Guided GradCAM combines GradCAM with Guided Backpropagation through an element-wise product for pixel-level explanations.
  • Smoothing: SmoothGrad averages explanations of noisy input copies to reduce noise and visual diffusion.The noise vectors are independently sampled from a normal distribution.
  • Related work and assessment: The paper reviews learned per-sample explanations, equivalences among attribution methods, SHAP, adversarial manipulation, and existing saliency-quality metrics.It does not aim to exhaustively evaluate all prior methods, but instead highlights how its methodology applies to selected cases.
  • Assessment: The study compares explanations using rank correlation, SSIM, and gradient-histogram correlation under absolute-value and diverging visualizations.These metrics quantify similarity between explanation maps.

3 Model Parameter Randomization Test

The model parameter randomization test evaluates whether saliency explanations depend on learned model parameters by comparing explanations after complete, cascading, or independent weight randomization. Results show that gradients are sensitive to parameter randomization, whereas Guided Backpropagation and Guided GradCAM remain invariant to higher-layer weights, making visual similarity an unreliable diagnostic.

  • Test design: The test compares explanations from the original model with explanations after complete, cascading, or single-layer weight re-initialization.Cascading randomization proceeds from top layers toward the input, while independent randomization isolates each layer’s contribution.
  • Results: Gradients are sensitive to model parameter randomization, while GradCAM is sensitive when randomization occurs downstream of its last convolutional layer.
  • Results: Guided Backpropagation masks remain visually and quantitatively similar to trained-model masks until lower-layer weights closest to the input are randomized.
  • Interpretation: Visual similarity can persist after parameter randomization even though signed rank correlation approaches zero when top layers are re-initialized.The discrepancy between visual appearance, SSIM, absolute-value rank correlation, and signed rank correlation makes naive visual inspection unreliable.
  • Results: Guided Backpropagation and Guided GradCAM are invariant to higher-layer weights but change when lower-layer convolutional weights are randomized.The resulting Guided Backpropagation masks remain dominated by input structure after lower-layer randomization.

4 Data Randomization Test

The data randomization test evaluates whether explanations depend on the relationship between inputs and labels by comparing models trained on true versus permuted labels. The experiments show that some saliency maps can remain visually plausible despite models learning randomized labels, while gradients and SmoothGrad undergo substantial changes.

  • Test rationale: Randomizing labels breaks the instance-label relationship, so an explanation insensitive to this change cannot explain mechanisms depending on that relationship.
  • Test design: The test permutes training labels, trains each model to greater than 95% training accuracy, and compares explanations on the same test inputs with true-label models.Test accuracy for permuted-label models is never better than random guessing, up to sampling error.
  • Results: Gradients and SmoothGrad undergo substantial changes after label randomization, while GradCAM develops masks with disconnected patches.
  • Results: Guided Backpropagation still assigns positive relevance across most of a digit for a model trained on random labels, despite visually plausible input-aligned masks.
  • Interpretation: Gradient⊙input and integrated-gradients masks visibly change, particularly in attribution signs, while input structure remains prevalent and can appear legitimate under visual inspection.This effect is especially prominent for sparse MNIST inputs but also appears on Fashion MNIST.

5 Discussion

The paper interprets saliency behavior through simple models and edge detectors, showing that visual similarity can arise without sensitivity to learned parameters or data. These analyses explain why some methods produce compelling but potentially misleading explanations.

  • Architecture and explanation behavior: Randomly initialized networks can provide non-trivial representations, so architecture itself may influence explanations independently of learned parameters.The discussion connects this possibility to prior work using random networks for denoising, super-resolution, and in-painting.
  • Input-gradient methods: Input-times-gradient methods can be dominated by the input even when gradient vectors change drastically.Experiments on Fashion MNIST and ImageNet found that these methods mostly return the input when gradients are visually noisy.
  • Linear models: In a linear model, gradients and SmoothGrad return model weights, while Integrated Gradients reduces to the input multiplied by the weights.Accordingly, basic gradients pass the random-model sanity check, whereas Integrated Gradients produces a noisy version of the input for a random model.
  • One-layer convolutional models: A one-layer sum-pooling convolutional network applies a 3 × 3 filter, ReLU, and global sum pooling, yet its saliency maps can act like edge detectors.The analysis attributes this behavior to the convolutional structure and local activation patterns.
  • One-layer convolutional models: Uniform image regions share activation patterns and gradients, whereas edge regions have distinct patterns that appear in the saliency mask.The ReLU derivative makes the local 3 × 3 activation pattern determine the gradient for each interior pixel.
  • Edge detectors: Edge detectors depend only on the input image, so their visual similarity to saliency maps can create confirmation bias about model relevance.The paper therefore argues that visual inspection alone cannot distinguish model-sensitive explanations from implicit image-processing techniques.

6 Conclusion and future work

The paper presents its experimental method as guidance for assessing the scope of model explanations, while emphasizing that the evaluation is a step toward broader analysis rather than a final verdict.

  • Conclusion: Visual inspection can favor compelling explanation images that lack sensitivity to the model and data-generating process.The paper positions this as a reason to assess explanation scope beyond appearance.
  • Future work: Invariances under model or label randomization can rule out an explanation method for tasks requiring sensitivity to those transformations.The authors focused primarily on model and label randomization but identify other transformations for future investigation.

A Explanation Methods

An explanation method maps an input to an object of the same shape, assigning each input dimension a relevance or importance value for a model output.

  • Explanation methods: An explanation method is represented as E : R^d → R^d, mapping inputs to explanations with the same shape.Each dimension of the explanation corresponds to the relevance or importance of the matching input dimension for a class-specific output.

A.1 Gradient with respect to input

The gradient explanation is defined as the gradient of a scalar class logit with respect to the input.

  • Gradient with respect to input: The gradient explanation measures the scalar logit for a particular class with respect to the input.

A.2 Gradient ⊙Input

This section defines several saliency methods, including gradient-based, GradCAM-based, Integrated Gradients, SmoothGrad, and VarGrad explanations. It also states key assumptions and relationships among these methods.

  • Gradient ⊙Input: Gradient⊙input multiplies gradients element-wise by the input and is equivalent to DeepLift and ϵ-LRP for bias-free ReLU networks.The equivalence holds for networks with only ReLUs and no additive biases.
  • Guided Backpropagation: Guided Backpropagation modifies ReLU back-propagation by retaining only positive gradients and positive forward activations.Its backward mask keeps entries satisfying both positivity conditions.
  • GradCAM and Guided GradCAM: GradCAM uses gradients of the class score with respect to the last convolutional feature maps, then applies global averaging, weighted combination, and ReLU.Guided GradCAM combines GradCAM with Guided Backpropagation through an element-wise product for pixel-level explanations.
  • Integrated Gradients: Integrated Gradients uses a baseline input representing feature absence, typically set to zero.The baseline is denoted by x̄ and is compared with the original sample x_t.
  • SmoothGrad and VarGrad: SmoothGrad averages explanations over noisy copies of an input, while VarGrad uses the corresponding variance and captures higher-order partial derivatives independently of the gradient.Both methods draw independent Gaussian noise vectors with variance σ^2.

B DNN Architecture, Training, Randomization & Metrics

The experiments evaluate randomization tests across image datasets and neural-network architectures, using model and label randomization alongside several similarity metrics. Calibration compares explanation masks with random masks and with one another.

  • Data sets & Models: Experiments use Inception v3 on ImageNet, a CNN on MNIST and Fashion MNIST, and an MLP on MNIST and Fashion MNIST.These models provide varied architectures and datasets for the randomization tests.
  • Randomization Tests: Model parameter tests reinitialize network parameters, while random-label tests train on fully randomized labels until exceeding 95 percent training accuracy.Parameter reinitialization uses a truncated normal distribution and was replicated with a uniform distribution yielding identical results.
  • Inception v3: The Inception v3 tests randomize parameters on a per-block basis, including all filter weights within each inception block.The network is a pretrained ImageNet model with 93.9 top-5 test accuracy.
  • CNN on MNIST and Fashion MNIST: The CNN uses convolutional, pooling, and fully connected layers with ReLU nonlinearities and achieves 99.2 percent final test accuracy.It is trained with ADAM for 20 thousand iterations and weight decay of 0.001.
  • MLP trained on MNIST: The MLP uses four fully connected layers with ReLU nonlinearities and achieves 98.7 percent final test accuracy on MNIST.Randomization reinitializes layers successively or independently depending on the experiment.
  • Similarity Metrics: Similarity is measured with absolute and diverging Spearman correlation, SSIM, and Pearson correlation of gradient histograms.SSIM and HOG metrics are applied to ImageNet explanation masks because they suit natural images.
  • Calibration: Calibration compares explanation masks with uniformly or Gaussian-sampled random masks and compares pairs of randomly sampled masks.ImageNet experiments average results across 50 validation images for gradient saliency, Integrated Gradients, and gradient⊙input.
  • Calibration: For uniform random masks, the reported similarity tuple is (0.00016, −0.0015, 0.078, 0.00076).The tuple corresponds to the reported similarity metrics in their stated order.

C Additional Figures

The supplementary material presents additional randomization figures across ImageNet, MNIST, Fashion MNIST, and multiple architectures. The figures cover cascading and independent randomization, visualization schemes, rank correlations, random-label comparisons, and an edge-detector analogy.

  • Additional Figures: The supplementary material explicitly introduces these figures as additional figures referenced in the main text.The section also labels a grayscale visualization for cascading randomization.
  • Guided Backpropagation: Figures 8 and 9 provide cascading-randomization visualizations for several examples using Guided Backpropagation.Figure 9 uses a different visualization scheme from Figure 8.
  • Inception V3 Examples: Additional figures show cascading and independent randomization results for bird, dog, and corn examples on Inception V3.The examples are presented with grayscale and diverging visualizations.
  • Additional Metrics: Further figures report input⊙random-gradient experiments and additional cascading and independent rank-correlation metrics across MNIST and Fashion MNIST CNNs.The rank-correlation figures separately cover cascading and independent randomization.
  • Random Labels: Random-label comparisons show explanations for true and random-label models using CNNs and MLPs on MNIST and Fashion MNIST.The comparisons are presented in Figures 22–25.
  • Layer Randomization: Successive and independent layer-randomization figures cover Fashion MNIST, MNIST, and MLP variants with diverging and grayscale visualizations.The material includes separate CNN and MLP figure sets.
  • Additional Architectures and Analogy: Additional visualizations include MLP MNIST and Fashion MNIST results, AlexNet perturbation-method randomization, and an input-image times edge-detector comparison.The edge-detector figure contrasts saliency maps with an image transformed by an edge detector.
Loading 1810.03292v3…