Source-linked AI summary
On the Connection Between Adversarial Robustness and Saliency Map Interpretability
Christian Etmann, Sebastian Lunz, Peter Maass, Carola-Bibiane Schönlieb
TL;DR
The paper asks whether the apparent interpretability of robust networks’ saliency maps can be quantified through alignment with the input image. It develops a linear theory and evaluates the relationship empirically in nonlinear networks, finding that robustness and alignment are connected on average but less consistently when nonlinear effects are stronger.
Problem
The paper addresses how adversarial robustness relates to the interpretability of neural-network saliency maps and whether that interpretability can be quantified through image–saliency alignment.
Method
The paper defines alignment, analyzes a linear binary classifier, derives robustness-related bounds, and tests the relationship using adversarial examples and models with local Lipschitz regularization.
Results
The robustness–alignment relationship holds well on average, but is less pronounced for individual points, especially on ImageNet, and becomes stronger for more linear models.
Takeaways & Limitations
Saliency-map interpretability can be understood as stronger image–saliency alignment associated with robustness, particularly when the network behaves more linearly.
Takeaways & Limitations
The linear approximation may fail near data points in practice, and apparent robustifying effects of alignment can be confounded by the Lipschitz penalty introduced by the analysis.
Abstract
from arXiv · showhide
Recent studies on the adversarial vulnerability of neural networks have shown that models trained to be more robust to adversarial attacks exhibit more interpretable saliency maps than their non-robust counterparts. We aim to quantify this behavior by considering the alignment between input image and saliency map. We hypothesize that as the distance to the decision boundary grows,so does the alignment. This connection is strictly true in the case of linear models. We confirm these theoretical findings with experiments based on models trained with a local Lipschitz regularization and identify where the non-linear nature of neural networks weakens the relation.
1. Introduction
Neural networks can be highly accurate yet vulnerable to small adversarial perturbations, while robustified models tend to produce more interpretable saliency maps. This paper connects that interpretability to alignment between an image and its saliency map, with the connection strongest for linear models.
- Small, often imperceptible adversarial perturbations can change neural networks’ classification decisions, creating risks for security-critical deployments.
- Adversarial training increases robustness by requiring larger perturbations to change classification, although it decreases classifier accuracy.
- Standard neural networks often produce unstructured saliency maps, whereas robustified classifiers tend to reveal structures present in the input image.
- The paper explains saliency-map interpretability through alignment between the input image and its gradient, first establishing the principle for linear binary classifiers.
- For nonlinear networks, the robustness–alignment relationship holds on average but weakens as the model becomes less linear.
2. Adversarial Robustness and Saliency Maps
The paper defines adversarial robustness through distance to the nearest decision boundary and alignment through similarity between an input image and its saliency map. It establishes their connection exactly for linear classifiers, then develops approximations and reductions for affine, locally affine, and multi-class settings while identifying limits of the relationship.
- Robustness: Adversarial robustness is the distance from an input to its closest decision boundary, and distributional robustness averages this pointwise quantity.The definition does not depend on the ground-truth class label.
- Linear case: For a linear binary classifier, robustness grows with input–gradient alignment and is maximized when the input and classifier vector are collinear.The relationship follows from ρ(x) = ∥x∥·|cos(δ)|, where δ is the angle between the input and gradient.
- Alignment: Alignment measures similarity between an input image and its saliency map, coinciding with absolute Pearson correlation when both vectors are unit-norm and zero-centered.For linear binary classifiers, alignment increases with robustness.
- Affine case: In affine classifiers, robustness and alignment no longer coincide, so the paper relates them through bounds or alignment evaluated at a shifted point.The affine shift is uniform across data points, while the two viewpoints yield different bounds in the nonlinear case.
- General case: For locally affine multi-class networks, the paper introduces linearized robustness and conditions under which it approximates distance to the decision boundary.The relevant local-affinity condition is typically violated in practice, but the identity can remain approximately valid when the local linear approximation is sufficiently accurate.
- General case: The multi-class reduction compares the predicted class with its closest competing class, using score differences and their gradient differences to define binarized saliency and alignment.This reduction reflects that score differences, rather than individual score values, determine the classification decision.
3. Decompositions and Bounds for Neural Networks
For neural networks, positive one-homogeneous structure preserves the robustness–alignment connection, while decomposition and pointwise bounds identify the error terms that weaken it. When these errors are small, greater robustness should correspond roughly to greater saliency-map alignment and interpretability.
- Homogeneous decomposition: Positive one-homogeneous score functions are sufficient for robustness and binarized alignment to coincide, beyond the stronger assumption of linearity.Many bias-free feedforward networks with (leaky) ReLU activations satisfy this property.
- Homogeneous decomposition: Neural networks can be decomposed into a homogeneous term and a remainder involving bias terms, with the remainder locally constant within each linear region.This decomposition separates the component preserving the robustness–alignment relation from the bias-dependent remainder.
- Pointwise bounds: For nonlinear networks, bounds relate linearized robustness to alignment, but their tightness depends on the gradient difference, bias remainder, and shifted-image approximation being small.The relevant conditions include a small angle between the gradients and a small bias contribution relative to alignment.
- Pointwise bounds: When the error terms in the bounds are small, the inequalities explain why more robust networks can yield more interpretable saliency maps.In this regime, alignment should roughly increase with linearized robustness.
- Pointwise bounds: The nonlinear bounds may become substantially loose when the image and saliency map are nearly orthogonal, so they explain deviations rather than provide a precise equivalence.This looseness follows from the large Cauchy–Schwarz upper bound in the near-orthogonal case.
- Alignment and interpretability: The alignment–interpretability correspondence is not exact: zero-valued image regions and image representation choices can affect alignment independently of visual interpretability.For quasi-binary images, relevance assigned to absent features may not change the inner product, although it still affects alignment through normalization.
4. Experiments
Experiments on MNIST and ImageNet support an increasing median alignment with robustness, but pointwise relationships differ across datasets. The analyses attribute these differences to how nonlinear terms and alignment contributions change with regularization.
- Experimental setup: Models trained with varying local-Lipschitz regularization were evaluated on MNIST and ImageNet using multiple adversarial attacks and linearized robustness estimates.The regularization strength λ controls the models’ robustness, while 1000 adversarial examples per model were generated for validation analysis.
- Median relationship: Median alignment increases with median robustness on both ImageNet and MNIST, although MNIST shows saturation toward the highest robustness levels.The ImageNet trend is supported across the attacks, and the linearized robustness estimator closely approximates algorithmically calculated robustness.
- Pointwise relationship: Pointwise robustness and alignment are highly correlated on MNIST but show no visible correlation on ImageNet.For ImageNet, the increasing median alignment can instead reflect increasing dispersion of the image–saliency inner product when its median is approximately zero.
- Robustness estimation: The linearized robustness estimate is tightly correlated with attack-based robustness on both datasets, indicating that it remains an adequate approximation for these nonlinear models.The comparison uses CW-attack estimates of actual robustness and 1000 validation points per model.
- Explaining the observations: For ImageNet, alignment-related terms increasingly dominate the theoretical bounds, whereas MNIST shows weaker or largely constant alignment contributions across the corresponding analyses.The ImageNet α† fraction rises from below 0.55 to almost 0.85, and the α contribution in one bound increases roughly sevenfold; MNIST follows different trends.
- Explaining the observations: The authors conjecture that regularization makes ImageNet models increasingly similar to linear models, strengthening the robustness–alignment connection, while nonlinearity weakens the pointwise relation.On MNIST, the alignment eventually saturates as deviations from linearity weaken the connection despite increasing robustness.
5. Conclusion and Outlook
The paper links robustness to saliency-map alignment and suggests alignment-aware defenses, while cautioning that the proposed alignment penalty may confound robustness with Lipschitz regularization.
- 5. Conclusion and Outlook: The paper concludes that robustness and saliency-map alignment are generally linked, but the connection is weaker for individual ImageNet examples and depends on local linearity.The conclusion attributes the weaker ImageNet relationship to the greater nonlinearity required for accurate models on that complex task.
- 5. Conclusion and Outlook: Alignment may be incorporated directly into new adversarial defenses alongside robustness.The paper identifies alignment as an additional design objective rather than only an interpretability measure.
- 5. Conclusion and Outlook: The proposed alignment penalty is bounded below by zero, but its robustifying effects may be confounded by the Lipschitz penalty introduced by its first term.The authors therefore call for careful experimental evaluation of any robustness gains attributed to increased alignment.
- 5. Conclusion and Outlook: Figure 10 compares gradient images and their alignments for the most and least robust ImageNet classifiers.Odd rows use the most robust classifier, even rows the least robust, and each gradient image is independently scaled to [0, 255].
Appendix
The appendix proves a local robustness characterization by restricting perturbations to the locally affine component and selecting the closest boundary-crossing direction from competing class gradients.
- Appendix: Lemma 1 assumes the local affine radius is no larger than the robustness and reformulates the robustness calculation within that affine component.The infimum is taken over perturbations e satisfying ∥e∥≤l(x).
- Appendix: The appendix identifies i∗ as the predicted class at x and uses the nearest competing class in the robustness minimization.This notation anchors the subsequent class-wise boundary calculation.
- Appendix: The class-change condition is expressed by the existence of a competing class whose gradient difference has sufficiently large inner product with the perturbation.The relevant comparison is between the gradient difference ∇Ψj(x) − ∇Ψi∗(x) and the score gap Ψi∗(x) − Ψj(x).
- Appendix: The infimum is attained by choosing the perturbation as a multiple of the gradient difference between the competing and predicted classes.A direct computation then completes the proof.
Proofs of Homogenization results
The homogenization results establish that the considered ReLU-style neural-network components produce positive one-homogeneous functions, enabling the paper’s decomposition and gradient calculations.
- Proofs of Homogenization results: Euler’s homogeneous-function theorem is introduced for continuously differentiable positive one-homogeneous functions.This theorem supplies the mathematical basis for evaluating such functions through their gradients.
- Proofs of Homogenization results: The proof section also states that the linearized robustness lemma follows directly from the preceding result.This connects the homogenization derivation to the robustness analysis.
- Proofs of Homogenization results: The neural-network class includes affine transforms with ReLU or leaky ReLU activations, plus skip connections, normalization, and pooling layers.The definition covers many state-of-the-art classification architectures.
- Proofs of Homogenization results: A single affine-ReLU layer is positive one-homogeneous, and compositions preserve this property.The proof extends the result to networks built from the specified affine and activation operations.
- Proofs of Homogenization results: The appendix applies the homogenization theorem to network logits and explicitly computes their values and gradients at x.These calculations support the later decomposition into gradient and locally constant terms.
- Proofs of Homogenization results: The subsequent theorem defines g as the predicted-class gradient, g† as the competing-score difference gradient, and β† as the corresponding bias-term difference.These quantities are used in the neural-network robustness bounds.
MNIST Model Architecture
The MNIST models use a convolutional architecture with increasing feature-map widths, repeated max pooling, and a final dense hidden layer with ReLU activation.
- MNIST Model Architecture: The architecture begins with a 3 × 3 same-padded convolution using 64 feature maps and ReLU, followed by factor-2 max pooling.
- MNIST Model Architecture: A second 3 × 3 same-padded convolution expands the model to 128 feature maps with ReLU, followed by factor-2 max pooling.
- MNIST Model Architecture: The pooled representation passes through a 128-neuron dense ReLU layer and another factor-2 max-pooling stage.