Source-linked AI summary
Explanations can be manipulated and geometry is to blame
Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller, Pan Kessel
TL;DR
Explanation maps can be arbitrarily manipulated by visually imperceptible input perturbations that leave network outputs approximately constant, undermining their reliability for interpretation. The paper analyzes this vulnerability geometrically, derives a manipulation bound, and proposes smoothing-based defenses that increase explanation robustness while leaving the model unchanged.
Problem
Explanation methods intended to improve neural-network trustworthiness and interpretability can be manipulated while preserving the network output, challenging robust use in safety-critical settings.
Method
The paper combines a theoretical analysis of output-manifold geometry with targeted perturbation algorithms and smoothing-based robustness mechanisms.
Results
The authors demonstrate arbitrary explanation manipulation with hardly perceptible perturbations and approximately constant outputs, then report increased robustness from smoothing.
Takeaways & Limitations
Explanation robustness can be improved by smoothing the explanation process without changing the underlying model.
Takeaways & Limitations
The theoretical theorem applies only to one-layer neural networks, while extension from gradient to propagation-based methods remains future work.
Abstract
from arXiv · showhide
Explanation methods aim to make neural networks more trustworthy and interpretable. In this paper, we demonstrate a property of explanation methods which is disconcerting for both of these purposes. Namely, we show that explanations can be manipulated arbitrarily by applying visually hardly perceptible perturbations to the input that keep the network's output approximately constant. We establish theoretically that this phenomenon can be related to certain geometrical properties of neural networks. This allows us to derive an upper bound on the susceptibility of explanations to manipulations. Based on this result, we propose effective mechanisms to enhance the robustness of explanations.
1 Introduction
The paper shows that explanation maps can be arbitrarily targeted through visually imperceptible input perturbations while keeping network outputs approximately constant. It links this vulnerability to output-manifold geometry and proposes smoothing-based defenses.
- Core finding: Explanation maps can be changed to an arbitrary target map using a visually hardly perceptible perturbation that approximately preserves the network output.The class probabilities also remain approximately unchanged.
- Core finding: This vulnerability is problematic when users rely on explanation maps for clinical decision making.
- Theory: Theoretical analysis relates the phenomenon to output-manifold geometry and bounds manipulation using principal curvatures and geodesic distance.
- Evidence: The manipulation algorithm is demonstrated across six explanation methods, four network architectures, and two datasets.
- Robustness: Smoothing-based methods are proposed to undo manipulations and increase robustness for both gradient- and propagation-based explanations.
2 Manipulating explanations
The authors optimize small input perturbations so explanations match a target while network outputs remain stable. Experiments show this targeted manipulation across methods, architectures, and datasets, with high image and explanation similarity.
- Manipulation setup: A manipulated image xadv = x + δx is constructed so the network output stays approximately constant, the explanation approaches a target map, and ∥δx∥ remains small.
- Explanation methods: The study covers gradient-based and propagation-based explanation methods, including Gradient, Gradient × Input, Guided Backpropagation, LRP, and Pattern Attribution.
- Optimization: The loss balances explanation-target similarity against output preservation, with relative weighting controlled by γ.
- Optimization: Replacing ReLU with softplus during optimization supplies a well-defined second derivative, after which the manipulated image is tested with the original ReLU network.
- Results: Across the similarity metrics, manipulated explanations closely resemble targets while perturbed images remain very similar to originals.SSIM and PCC increase with similarity; MSE decreases with similarity.
- Results: Comparable results are obtained on ResNet-18, AlexNet, and DenseNet-121, and the algorithm is also successfully tested on CIFAR-10.
3 Theoretical considerations
The paper links explanation sensitivity to the geometry of constant-output manifolds, showing that curvature and geodesic distance bound how much gradient explanations can change under small, output-preserving perturbations.
- Geometric formulation: The analysis studies gradient explanations by viewing the network output as a hypersurface and the gradient as its normal.The theoretical discussion uses differential geometry to formalize this sensitivity.
- Output-preserving manipulation: Small input perturbations can drastically change explanations while leaving the network output constant.The manipulated input remains on a hypersurface of equal network output.
- Geometric formulation: The sensitivity of the gradient map is described by the principal curvatures of the constant-output hypersurface.Principal curvatures are the real eigenvalues of the second fundamental form.
- Bound on sensitivity: An upper bound on the maximal gradient-map change depends on the largest absolute principal curvature and the geodesic distance between inputs.The theorem assumes a connected local hypersurface neighborhood and a lower bound on the gradient norm.
- Implications: Small Euclidean input distance does not ensure similar explanations when geodesic distance is large on a curved manifold.The paper also states that the bound scales with the softplus β parameter, so smaller β yields greater robustness.
4 Robust explanations
The paper proposes smoothing explanation methods to reduce manipulation susceptibility, using β-smoothing and comparing it with SmoothGrad in robustness and computational cost.
- β-smoothing: β-smoothing replaces ReLU nonlinearities with softplus nonlinearities using a small β to smooth principal curvatures.The paper reports that a value around one works well in practice.
- Relation to SmoothGrad: For a one-layer ReLU network, Theorem 2 relates β-smoothing to a noise-averaged gradient representation associated with SmoothGrad.The stated relationship uses a perturbation distribution pβ and an infinite-sample SmoothGrad limit.
- Relation to SmoothGrad: Theorem 2 is limited to one-layer networks, while deeper networks show empirically similar, less noisy explanation maps under the two smoothing methods.The paper presents this theorem as evidence that SmoothGrad can also smooth curvatures and improve robustness.
- β-smoothing: Figure 4 shows that lowering β with β-smoothing recovers the original explanation map across the considered methods.The figure compares correlations of manipulated explanations with target and original explanations for Gradient and LRP.
- Robustness and cost: β-smoothing and SmoothGrad are substantially more robust than unsmoothed methods, while β-smoothing requires one forward-backward pass versus typically 10 to 50 noise samples for SmoothGrad.Figure 5 compares robustness between the methods and reports lower computational cost for β-smoothing.
5 Conclusion
The paper shows that imperceptible input manipulations can drastically alter explanation maps while preserving model outputs, and proposes smoothing to improve resilience. It identifies network curvature as an important vulnerability factor while leaving broader training-based defenses and propagation-method theory for future work.
- Imperceptible input manipulations can produce arbitrary, drastic changes in explanation maps while keeping the model output constant.
- Large curvature of the network’s decision function is identified as an important source of explanation vulnerability.
- Smoothing only the explanation process can substantially increase resilience without changing the model itself.
- Future work will examine modifying neural-network training and extending the theoretical analysis from gradient to propagation-based methods.
A Details on experiments
The experiments use beta growth during optimization to approximate relu behavior with softplus networks, while analyzing output changes and explanation-map similarity. The strategy is not essential, and LRP does not receive its optimization speed-up because its rules do not explicitly depend on relu second derivatives.
- A.1 Beta growth: Figure 6 compares image and explanation-map MSE during training with and without beta growth.The left panel measures x versus xadv, while the right measures ht versus h(xadv).
- A.1 Beta growth: Beta growth is used for all methods except LRP, whose rules do not explicitly depend on relu activation second derivatives.
- A.1 Beta growth: For large β, softplus networks approximate the relu network well, whereas small β makes gradient maps more similar to LRP, GPB, and PA maps.
- B Difference in network output: Manipulated images retain the original classification result, with small confidence changes and a very small change in the vector of class probabilities.
C.1 Additional architectures
The manipulation algorithm generalizes beyond VGG-16 to AlexNet, DenseNet, and ResNet architectures. Experiments compare output changes across architectures and use architecture-specific beta-growth settings where softplus maps otherwise fail to resemble relu maps.
- C.1 Additional architectures: The explanation-manipulation algorithm is evaluated on AlexNet, DenseNet, and ResNet in addition to VGG-16.
- C.1 Additional architectures: DenseNet uses β0 = 30 and βe = 300 because smaller beta values produce softplus maps unlike the relu maps.
- C.1 Additional architectures: Figures 11 and 12 report change in output and gradient-method similarity measures across architectures.
- C.1 Additional architectures: Figures 13–16 provide one example image for each evaluated architecture.
C.2 Additional datasets
An additional experiment trains VGG-16 on CIFAR-10 and applies the manipulation algorithm to LRP explanations. The section includes architecture-specific explanation-map figures and reports approximately 92% test accuracy for the trained model.
- C.2 Additional datasets: VGG-16 is trained on CIFAR-10, achieving approximately 92% test accuracy.
- C.2 Additional datasets: The algorithm is then used to manipulate explanations for the LRP method on CIFAR-10.
- C.2 Additional datasets: Figures 13–16 show gradient explanation maps for VGG-16, ResNet-18, AlexNet, and DenseNet-121.
- C.2 Additional datasets: Figure 17 shows two example LRP-manipulated images for the CIFAR-10 experiment.
D Smoothing explanation methods
Replacing ReLU with softplus or averaging explanations over noisy inputs smooths explanation methods and makes targeted manipulations less effective. The theoretical analysis links this robustness to reduced curvature and bounds involving principal curvature and geodesic distance.
- Softplusβ activations create a smoothing effect when substituted for ReLU activations in explanation methods.
- SmoothGrad adds random image perturbations and averages the resulting explanation maps across 10 perturbed images.The noise level is parameterized by σ = n · (xmax − xmin).
- β-smoothing and SmoothGrad produce more robust explanations under 100 targeted attacks evaluated with MSE, SSIM, and PCC.The comparison covers Gradient and LRP explanation methods.
- β-smoothed explanations show higher MSE and lower SSIM and PCC than original methods, while manipulated-image similarities remain comparable.This pattern indicates increased robustness without a large change in image similarity measures.
- Reducing β lowers the maximal curvature of equal-output surfaces, and the explanation-change bound scales with maximal curvature and geodesic distance.The proof proceeds through bounds on the Hessian, second fundamental form, principal curvature, and explanation change.
E.2 Theorem 2
Theorem 4 relates ReLU gradients to gradients of β-smoothed softplus networks through a perturbation distribution. This connection motivates smoothing-based explanations as a way to reduce noise and improve robustness.
- Theorem 4 compares one-layer ReLU networks g(x) = relu(wT x) with β-smoothed softplus networks gβ(x) = softplusβ(wT x).
- The theorem is proved first for scalar inputs by defining the perturbation distribution implicitly and differentiating the resulting identity twice.The vector case then defines the perturbation distribution using the components of the perturbation vector.
- The theorem expresses the β-smoothed gradient through an expectation of gradients evaluated at perturbed inputs.For vector inputs, the perturbation distribution is defined componentwise and aligned with the network weight direction.