PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks

Yannick Lunk, Atell Yehor Krasnopolsky, Damien Garreau, Leon Bungert

arXiv:2610.10666v1cs.LGmath.OCstat.ML

TL;DR

The paper addresses the observed sparsity of adversarially trained models' gradient saliency maps, which are used to explain neural-network predictions. It analyzes regularized empirical risk for two-layer ReLU networks and shows how the limiting gradient penalty relates to sparse maps. Experiments also report reduced gradient ℓ1-norm and thresholded sparsity in adversarially trained models.

  • Problem

    Gradient saliency maps are widely used to explain predictions, and prior work observes that adversarially trained models produce sparse maps, but the theoretical explanation is unclear.

    Source: §1 · p. 1

  • Method

    The paper analyzes a total-variation- and weight-decay-regularized empirical-risk problem for two-layer ReLU networks using Gamma-convergence and approximation results.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    For ℓ∞ attacks, the limiting regularizer includes an average gradient ℓ1 penalty, and experiments find reduced average input-gradient ℓ1-norm and thresholded sparsity under adversarial training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis selects Bayes classifiers balancing average gradient ℓ1-norm and Barron norm, favoring axis-aligned gradients that explain sparse gradient maps.

    Source: §3 · p. 5, §7 · p. 9

    The analysis requires weight decay as strong as adversarial regularization and does not cover Madry et al.'s original adversarial training formulation.

    Source: §3 · p. 7, §7 · p. 9

Abstract

from arXiv · show

Understanding why deep neural networks make a given prediction is of great importance for their safe deployment. In computer vision, saliency maps, which highlight the image region most influential for a prediction, remain a widely-used form of explanation. An empirical observation is the apparent sparsity of gradient saliency maps of adversarially-trained neural networks. In this paper, we propose a theoretical explanation of this phenomenon for two-layer ReLU networks. We build on the established equivalence of adversarial training to the minimization of the empirical risk with weight-decay penalization and an added adversarial total variation term -- valid for certain loss functions. As the number of data points and neurons grows and the regularization parameters are sent to zero at appropriate rates, we prove that minimizers converge to a Bayes classifier with minimal gradient and Barron norm. Sparsity appears since for adversarial training with $\ell_\infty$-attacks the gradient norm is anisotropic and favors axis-aligned / sparse gradients. We illustrate our theoretical findings experimentally by evaluating the gradient $\ell_1$-norm and thresholded sparsity of naturally versus adversarially trained models.

1 INTRODUCTION

The paper explains why adversarially trained models can have sparse gradient saliency maps, connecting adversarial training to a gradient-regularized limit problem. Its theory and experiments link ℓ∞ attacks to an ℓ1 gradient penalty and sparse maps.

1 INTRODUCTION

  • Saliency maps visualize input gradients to show which image features influence a model's prediction, addressing the black-box nature of neural-network decisions.Gradient times input, integrated gradients, SmoothGrad, and GradCAM are cited as related explanation methods.Source: §1 · p. 1
  • Adversarial training has been observed to improve robustness while producing sparse, perceptually aligned gradient saliency maps, despite having no explicit interpretability or gradient-structure objective.The cited empirical observations are Zhang & Zhu (2019), Etmann et al. (2019), and Chalasani et al. (2020).Source: §1 · pp. 1–2
  • For ℓ∞ attacks, the dual-norm penalty becomes an ℓ1 penalty on input gradients, providing a mathematical explanation for sparse saliency maps.The paper describes the selected Bayes classifier as balancing accuracy, complexity, and average gradient ℓ1-norm.Source: §1 · p. 2
  • Gamma-convergence, generalization bounds, and approximation results connect the limit analysis to empirical risk minimizers on finite datasets and neural networks.The paper embeds the problem in homogeneous Barron space to make this transition.Source: §1 · p. 2
  • As data and network width grow while adversarial budget and weight decay vanish at coupled rates, empirical minimizers converge subsequentially to Bayes classifiers balancing gradient regularization and Barron complexity.The limit regularizer uses the average dual norm of the classifier's input gradients.Source: §1 · p. 2
  • Experiments show that adversarial training reduces average input-gradient ℓ1-norm and that thresholded gradient sparsity validates earlier observations of sparse saliency maps.The cited earlier observations are Zhang & Zhu (2019) and Etmann et al. (2019).Source: §1 · p. 2

2 SETTING

The setting is binary classification with two-layer ReLU networks and a regularized empirical-risk objective. The analysis relates its adversarial total-variation term to adversarial training and studies a growing-data, growing-width limit.

2 SETTING

  • The classification data are modeled by a probability measure on a bounded domain with Lipschitz boundary and binary labels.The label-specific measures describe the data-point distributions for each class.Source: §2 · p. 2
  • In the toy setting with non-unique Bayes classifiers, natural training has an oscillatory boundary, while nonlocal TV regularization and adversarial training show approximate gradient sparsity.The regularized boundaries have small anisotropic perimeter, with axis-parallel segments indicating approximate sparsity.Source: Figure 2 · p. 3
  • The adversarial population risk uses a perturbation budget and an attack norm; for ℓ∞ attacks, its dual norm is ℓ1.The attack norm need not be Euclidean.Source: §2 · p. 4
  • The motivating total-variation objective is loosely equivalent to population-level adversarial training, with strict equivalence for specific losses and hypothesis classes.Its nonlocal TV term is connected to an ε-dependent surface content of the decision boundary and quantifies classifier robustness.Source: §2 · p. 4
  • The model class consists of two-layer neural networks with ReLU activation, with weight decay added to ensure existence and compactness of minimizers.The Barron semi-norm measures function complexity by minimizing weight decay over representations; weight decay does not control the bias vector.Source: §2 · pp. 3–4
  • The finite-sample objective uses two-layer ReLU networks, empirical risk, weight decay, and adversarial budget and weight-decay parameters that depend on sample size.It is loosely equivalent to finite-sample adversarial training; experiments report similar results and a slight robust-accuracy and sparsity advantage on CIFAR-10.Source: §2 · pp. 4–5

3 MAIN RESULT

Under suitable parameter scalings, empirical minimizers converge along a subsequence to a minimizer of a functional that selects Bayes classifiers with minimal combined total-variation and Barron complexity. For ℓ∞ attacks, the gradient penalty is ℓ1-based and promotes sparse saliency maps.

3 MAIN RESULT

  • The result assumes a Barron function minimizing population risk, a bounded Lipschitz loss with trivial constant classifiers strictly suboptimal, and continuous class densities positive on Ω.The class-conditionals are assumed absolutely continuous with continuous densities whose sum is positive throughout the domain.Source: §3 · p. 5
  • Minimizers of F are Bayes classifiers minimizing TV(f;µ) plus α[f]B; with ℓ∞ attacks, the gradient penalty is ℓ1-based and promotes sparse saliency.The regularizer promotes group sparsity: gradients vanish at many data points and are sparse at the remaining points, especially near the decision boundary.Source: §3 · pp. 5–6
  • Under the stated scaling, empirical minimizers have an almost-surely convergent subsequence in L1(Ω) whose limit minimizes F.The scaling sends ε_n and λ_n to zero while network width grows.Source: §3 · pp. 5–6
  • The analysis covers only positive finite α; the α=0 case would require Barron approximation bounds expected to scale with dimension.At α=0, the proof would need to approximate general BV functions rather than only Barron functions.Source: §3 · p. 6
  • The analysis treats one-hidden-layer networks; a comparable framework for deeper feedforward architectures is not currently available, though the authors expect extension to deep ResNets.The stated exception is deep ResNets embedded in a flow-induced Barron space.Source: §3 · p. 7
  • The objective matches Madry et al.'s original adversarial training only for zero-one loss; with other losses it is a related variant.The authors report similar experimental results for this objective and adversarial training, including experiments using cross-entropy loss.Source: §3 · p. 7

4 RELATED WORK

Prior work connects adversarial robustness with saliency-map interpretability, while theoretical accounts and regularization approaches examine how training objectives relate to gradients.

Adversarial Robustness and Interpretability

  • Adversarially trained CNNs have been reported to produce sparser saliency maps focused on object shapes and contours, alongside perceptual alignment.The related work also notes theoretical justifications for linear classifiers.Source: §4 · p. 7

Gradient Regularization

  • Direct input-gradient regularization has been reported to improve both adversarial robustness and interpretability.Other approaches include penalty-based gradient regularization and TRADES, which modifies adversarial training with a nonlocal loss-gradient [Zhang et al., 2019].Source: §4 · p. 7

Theoretical Foundations

  • Prior theory formalizes adversarial training as nonlocal total-variation regularization for ℓ1-loss, while related work studies its convergence and gradient-based risk bounds [Bungert et al., 2023; Bungert & Stinson, 2024; Yeh et al., 2019; García Trillos & García Trillos, 2022].Park et al. (2023) use Gamma-convergence to study weight-decay training of two-layer ReLU networks in a similar setting.Source: §4 · p. 7

5 SKETCH OF PROOF

The proof establishes Gamma-convergence of the empirical objectives to a limiting functional and combines it with compactness to show that subsequential limits of minimizers minimize that functional.

Proof strategy

  • Gamma-convergence plus compactness yields convergence of empirical minimizers to minimizers of the limiting functional F.The objective is divided by ε_n to recover F's selection principle, beyond convergence to Bayes classifiers alone.Source: §5 · p. 8

Liminf inequality

  • The lim inf inequality bounds F at any convergent sequence's limit by the limiting lower bound of F_n, almost surely.Its proof combines empirical-risk generalization bounds with inequalities for empirical nonlocal total variation and weight decay.Source: §5 · p. 8

Limsup inequality

  • The lim sup inequality constructs a recovery sequence converging to f whose objective has limit superior at most F(f), almost surely.For finite F, the construction uses Barron-function approximation with dimension-independent rates.Source: §5 · p. 8

Proof of Theorem 1

  • The proof compares minimizers of F_n with a recovery sequence for an approximate minimizer of F, then uses compactness and the lim inf result to conclude that the limit minimizes F.The conclusion holds along a subsequence converging in L1(Ω).Source: §5 · p. 9

6 NUMERICAL ILLUSTRATIONS

The experiments compare natural and adversarial training on CIFAR-10 and ImageNet, finding that adversarially trained models have sparser input gradients and lower gradient ℓ1-norms, including at larger scale.

Numerical illustrations

  • The CIFAR-10 experiments compare clean and robust accuracy, test-set average gradient ℓ1-norm, and thresholded gradient sparsity across adversarial budgets.Sparsity is the ratio of gradient components below each threshold.Source: §6 · p. 9
  • Small-budget adversarial training yields comparable accuracy to standard training but higher input-gradient sparsity and significantly lower average gradient ℓ1-norm on CIFAR-10; the pattern also holds for ImageNet.The CIFAR-10 comparison reports mean results over five seeds, with robust accuracy evaluated against 50-step PGD at 8/255; ImageNet uses pretrained ResNet50 models.Source: §6 · p. 9
  • On ImageNet, the same metrics are evaluated for pretrained ResNet50 models, and the earlier observations persist at the larger dataset scale.The models are from Salman et al. (2020).Source: §6 · p. 9

7 CONCLUSION

The paper explains sparse saliency gradients through a regularized empirical-risk limit that favors Bayes classifiers balancing gradient and Barron norms. Its analysis has scope limits, and the authors describe their use of generative AI tools.

7 CONCLUSION

  • The analysis explains sparse gradient maps by selecting Bayes classifiers that minimize a balance of average gradient ℓ1-norm and Barron norm, favoring axis-aligned gradients.The conclusion reports numerical experiments as confirming the theory and indicating generalizability.Source: §7 · p. 9
  • The analysis requires weight decay as strong as adversarial regularization and does not cover the original Madry et al. (2018) adversarial-training form.The authors identify other saliency maps, including integrated and smoothed gradients, as future study targets.Source: §7 · p. 9

AI USE STATEMENT

  • The authors report using generative AI tools to polish the manuscript, discuss parts of proofs, and assist with experiment code, while stating they reviewed the assisted work.They say the tools were not used to generate the core hypothesis or conduct literature reviews.Source: AI USE STATEMENT · p. 10

SUPPLEMENTARY MATERIAL

The supplementary material includes an alternative proof route for the main theorem, while the main paper also provides a direct proof.

A THE HOMOGENEOUS BARRON SPACE

This appendix develops properties of the homogeneous Barron space and derives generalization bounds for empirical risk and nonlocal total variation.

A THE HOMOGENEOUS BARRON SPACE

B GENERALIZATION BOUNDS

  • The empirical-risk and empirical nonlocal-total-variation bounds support generalization analysis for both terms of the objective.Source: B GENERALIZATION BOUNDS · p. 19
  • Under suitable scalings, empirical nonlocal total variation converges almost surely to its population version for sequences whose scaled Barron seminorm remains bounded.Source: B GENERALIZATION BOUNDS · p. 21

C RESULTS FOR THE NONLOCAL PERIMETER AND TOTAL VARIATION

The appendix extends nonlocal-perimeter Gamma-convergence results from Euclidean balls to general norm balls and establishes related compactness and total-variation limits.

D RESULTS FOR THE FULL REGULARIZER

The appendix establishes liminf and recovery-sequence results for the full regularizer, connecting its limiting problem to neural networks.

E PROOFS OF PROPOSITIONS 5.1 TO 5.3

The proof establishes Gamma-convergence of the induced functionals to the limit objective in the L1 topology. Its liminf, recovery-sequence, and compactness arguments support convergence of empirical minimizers.

  • The induced functionals Gamma-converge to F in the L1(Ω) topology.Theorem 2 defines the functionals by taking the infimum of the parameterized network objective over representations of each function.Source: E PROOFS OF PROPOSITIONS 5.1 TO 5.3 · p. 29
  • The liminf argument rules out limits outside the Barron class or with nonminimal risk by making the limiting objective diverge.For a Barron limit with risk above the minimum, the proof uses bounded Barron semi-norms and uniform Lipschitz control before applying the generalization bound.Source: E PROOFS OF PROPOSITIONS 5.1 TO 5.3 · p. 29
  • The limsup argument builds recovery sequences from neural-network approximations and combines them with a generalization bound.For a Bayes-risk minimizer in the Barron class, the proof uses the recovery sequence from Proposition D.2 and bounds the approximation and generalization errors.Source: E PROOFS OF PROPOSITIONS 5.1 TO 5.3 · pp. 29–30
  • Compactness follows from bounded Barron semi-norms, which give uniform Lipschitz bounds and an L1-convergent subsequence.The proof also bounds the functions at the origin; otherwise the risk would diverge under Assumption 1.Source: E PROOFS OF PROPOSITIONS 5.1 TO 5.3 · p. 30

F ADDITIONAL NUMERICAL ILLUSTRATIONS

Additional experiments compare decision boundaries, accuracy, and gradient sparsity across toy, CIFAR-2, and ImageNet settings. The patterns vary by attack setting, and ImageNet gradients concentrate near zero without being strictly sparse.

Toy Dataset

  • On the toy dataset, natural training has the highest clean accuracy but is most susceptible to the considered attacks.TV regularization has better clean accuracy and robustness to ℓ2 and small ℓ∞ attacks, while adversarial training slightly dominates at larger budgets; the Bayes accuracy is 80%.Source: F ADDITIONAL NUMERICAL ILLUSTRATIONS · p. 31, Figure 4 · p. 31
  • With ℓ2 attacks, gradients in the toy dataset’s undetermined region are not axis-parallel, while TV and adversarial training yield shorter, simpler boundaries than natural training.The theory does not predict gradient sparsity for ℓ2 attacks; the paper attributes the boundary behavior to an implicit gradient-norm and Barron-norm regularizer.Source: F ADDITIONAL NUMERICAL ILLUSTRATIONS · p. 31, Figure 5 · p. 32

CIFAR-2 and Two-Layer Neural Networks

ImageNet

  • On ImageNet, adversarially trained ResNet50 gradients concentrate much more around zero than naturally trained gradients, but are not strictly sparse.Very few components are exactly zero; most have magnitudes extremely close to zero.Source: F ADDITIONAL NUMERICAL ILLUSTRATIONS · pp. 32–33, Figure 8 · p. 33
  • For incorrect target classes, adversarially trained models also produce sparser, more structured gradients than the naturally trained model.The theory concerns true-class gradients; the authors say its alignment with off-target behavior depends on sufficient overlap between class-conditionals.Source: F ADDITIONAL NUMERICAL ILLUSTRATIONS · p. 33, Figure 9 · p. 33

Related papers

Loading 2610.10666v1…