Source-linked AI summary
The (Un)reliability of saliency methods
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, Been Kim
TL;DR
Saliency methods can attribute importance to input changes that do not affect a model’s prediction, undermining explanation reliability. The paper introduces input invariance and tests it with constant input shifts, showing that numerous methods can produce misleading attributions, including deceptive explanations of a hand-drawn cat image.
Problem
Saliency explanations lack reliability when they respond to factors that do not contribute to model predictions.
Method
The paper introduces input invariance and evaluates saliency methods using a constant input shift across neural-network experiments.
Results
Numerous saliency methods fail input invariance under constant shifts, and deceptive attribution can be purposefully created using a hand-drawn cat image.
Takeaways & Limitations
Understanding where saliency methods fail helps researchers appropriately weigh the explanations they provide.
Takeaways & Limitations
The analysis demonstrates failure for at least one transformation and does not establish methods or reference points that guarantee reliability for all possible transformations.
Abstract
from arXiv · showhide
Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a constant shift to the input data--- to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. In order to guarantee reliability, we posit that methods should fulfill input invariance, the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy input invariance result in misleading attribution.
1 INTRODUCTION
The paper argues that reliable saliency explanations require input invariance, because transformations that leave model predictions unchanged can still alter attributions. It introduces this axiom and demonstrates failures using constant input shifts and reference-point choices.
- Reliable explanations matter because they can build user trust, identify model failures, and support deployment in high-stakes domains.
- Input invariance requires saliency methods to mirror model sensitivity to input transformations.
- A constant input shift can change attributions without affecting model predictions or weights, enabling misleading explanations.
- Reference-point methods such as Integrated Gradients and Deep Taylor Decomposition can produce divergent attributions depending on the reference and transformation.
- The paper proposes data normalization for some transformations but states that normalization does not guarantee reliable attribution for all transformations.
2 THE MODEL IS INVARIANT TO A CONSTANT SHIFT IN INPUT
The authors construct a second network that cancels a constant input shift through first-layer bias adjustment, preserving the original network’s weights, activations, gradients, and predictions. They evaluate this invariance using paired MNIST networks with identical accuracy.
- The bias compensation produces two networks with identical weights and predictions for corresponding samples.Network 2 cancels the mean-shift transformation, yielding f1(x1) = f2(x2).
- The experiment compares attribution across a network trained on x1 and a network classifying the shifted inputs x2.The two inputs differ by the constant vector m2, and the networks are constructed to implement corresponding classifications.
- A constant input shift is canceled by changing the first-layer bias to b2 = b1 − wT m2.The transformed input is x2 = x1 + m2, and the adjusted bias preserves the first-layer pre-activation.
- The first-layer activations remain the same for f1(x) and f2(x), while the input gradient also remains unchanged.
- For MNIST, Network 1 is a three-layer, 1024-neuron-per-layer ReLU MLP using [0,1] inputs, while Network 2 uses [-1,0] inputs after m2 = −1; both reach 98.3% accuracy.Network 1 is trained for 10 epochs with mini-batch SGD.
3 THE (IN)SENSITIVITY OF SALIENCY METHODS TO MEAN SHIFTS
The evaluation tests whether saliency methods mirror a model’s invariance to constant input shifts. Gradient and signal methods remain invariant, while several attribution methods depend on input values or reference-point choices and can produce different explanations for identical predictions.
- Experiment setup: The experiment compares saliency heatmaps for two networks whose inputs differ by a constant shift, while their predictions remain equivalent.A method satisfies input invariance when the two networks receive identical heatmaps despite the shifted encoding.
- Gradient and signal methods: Gradient, PatternNet, and GuidedBackprop produce identical heatmaps across the shifted networks and therefore satisfy input invariance in this setting.These methods determine attribution from network or pattern weights, which are identical for the compared networks.
- Reference-point methods: Integrated Gradients and Deep Taylor Decomposition satisfy input invariance only conditionally, depending on the chosen reference point and the input transformation.Zero-vector references fail, whereas a black-image IG reference and a PatternAttribution DTD reference remain invariant for the tested shift.
- Gradient x Input: Gradient x Input fails input invariance because multiplying gradients by the input carries the constant shift into the final attribution.The method also restricts attribution without justification to inputs that are not zero.
- Reference-point methods: PatternAttribution compensates for a constant shift through a covariance-based reference, producing identical attributions across the two networks.By contrast, a zero-vector reference changes the input-reference difference after shifting, causing attribution to change.
- SmoothGrad: SmoothGrad inherits the input-sensitivity properties of its underlying attribution method rather than changing them.It remains invariant with gradient and signal methods, but not with Gradient x Input, zero-reference IG, or zero-reference DTD.
4 THE IMPORTANCE OF CHOOSING AN APPROPRIATE REFERENCE POINT
Reference-point choice materially affects attribution reliability: IG and DTD can satisfy input invariance only under particular references or transformations, while the MNIST experiment shows how shifts can induce deceptive explanations.
- Reference-point dependence: IG and DTD satisfy input invariance only for certain reference points or input transformations.The same method can produce visibly divergent attributions under different reference points.
- Constructing the transformation: The experiment constructs a constant vector shift using a hand-drawn cat image and adds it to the input.The shift is designed to assign a desired attribution to a specific sample under gradient-times-input attribution.
- MNIST evaluation: Gradient x Input, IG with black or zero references, and DTD with an LRP reference produce different attributions after the transformation.DTD with a PA reference is not sensitive to the transformation.
- MNIST evaluation: Clipping the shift to [-.3,.3] keeps the MNIST digit visible; without clipping, the attribution would show only the cat.The clipping preserves visibility of the original digit while retaining the deceptive attribution effect.
- MNIST evaluation: In Fig. 6, all methods except PA fail input invariance and visibly show a cat as the explanation for an MNIST prediction.The result demonstrates purposeful misrepresentation under the tested transformation.
5 CONCLUSION
The paper argues that saliency methods can be unreliable under transformations that leave model predictions unchanged. It introduces input invariance and shows that constant shifts can produce deceptive explanations, motivating systematic reliability evaluation.
- Conclusion: The paper shows that numerous saliency methods fail to attribute correctly when a constant vector shift is applied to the input.The shift does not affect the model prediction or weights but changes attribution.
- Conclusion: A hand-drawn cat image can be used to purposefully create a deceptive explanation of the network.The paper identifies this manipulation as a concrete failure mode of saliency methods.
- Conclusion: Input invariance is introduced as a prerequisite for reliable attribution.It requires explanations to track the model’s sensitivity to input transformations.
- Implications: Reliability is especially crucial when attribution is costly to inspect or incorrect attribution has high costs, including medicine.Image inspection can reveal the cat attack, whereas language and audio settings may be opaque.
- Future research: The paper calls for evaluating which methods or reference points guarantee reliability across all possible transformations.Determining failure modes is presented as a step toward understanding where and how to use saliency methods.