PaperlayerView exact revision on arXiv

Source-linked AI summary

There and Back Again: Revisiting Backpropagation Saliency Methods

Sylvestre-Alvise Rebuffi, Ruth Fong, Xu Ji, Andrea Vedaldi

arXiv:2004.02866v1cs.CV

TL;DR

Opaque, highly parameterized neural networks motivate tools for understanding model behavior, while backpropagation-based saliency methods still need their relationships and desiderata clarified. The paper unifies several methods in a framework and introduces NormGrad, cross-layer analysis, and class-sensitivity approaches. It reports that combining maps across layers consistently improves weak-localization performance for several methods and explains a Grad-CAM failure at earlier layers.

  • Problem

    Opaque, highly parameterized neural networks create a need for tools to help practitioners understand model behavior, and saliency methods have varied approaches worth clarifying.

    Source: §1 · p. 1

  • Method

    A two-phase framework unifies several backpropagation-based methods and yields NormGrad; the paper also introduces a class-sensitivity metric and meta-learning-inspired paradigm.

    Source: §1 · pp. 1–2

  • Results

    Weighted maps from all layers consistently improve weak-localization performance over the single best layer for several methods; earlier layers lose class-sensitive signals that Grad-CAM needs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports systematic exploration of saliency methods, while cross-layer maps can improve weak localization and produce high-resolution maps.

    Source: §5 · p. 8

    The class-sensitivity metric expects maps for the maximum class to be salient and maps for the minimum class to be uninformative because that class is absent from the image.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Saliency methods seek to explain the predictions of a model by producing an importance map across each input sample. A popular class of such methods is based on backpropagating a signal and analyzing the resulting gradient. Despite much research on such methods, relatively little work has been done to clarify the differences between such methods as well as the desiderata of these techniques. Thus, there is a need for rigorously understanding the relationships between different methods as well as their failure modes. In this work, we conduct a thorough analysis of backpropagation-based saliency methods and propose a single framework under which several such methods can be unified. As a result of our study, we make three additional contributions. First, we use our framework to propose NormGrad, a novel saliency method based on the spatial contribution of gradients of convolutional weights. Second, we combine saliency maps at different layers to test the ability of saliency methods to extract complementary information at different network levels (e.g.~trading off spatial resolution and distinctiveness) and we explain why some methods fail at specific layers (e.g., Grad-CAM anywhere besides the last convolutional layer). Third, we introduce a class-sensitivity metric and a meta-learning inspired paradigm applicable to any saliency method for improving sensitivity to the output class being explained.

1. Introduction

The introduction motivates tools for understanding opaque neural-network behavior and focuses on backpropagation-based saliency. It presents a unifying framework, NormGrad, cross-layer analysis, and methods for improving class sensitivity.

1. Introduction

  • Opaque, highly parameterized neural networks used in high-risk applications create a need for tools that help practitioners understand model behavior.The introduction identifies healthcare and automated driving as examples.Source: §1 · p. 1
  • Backpropagation-based saliency methods produce image heatmaps that rank pixels by importance to a model’s decision.The introduction frames its analysis around their similarities, differences, and potential improvements.Source: §1 · p. 1
  • The framework separates saliency generation into extracting spatial contributions of parameter gradients and aggregating them into a heatmap.NormGrad uses the Frobenius norm for aggregation; identity layers allow maps at spatially grounded locations, including after ReLU.Source: §1 · p. 1
  • Weighted maps from all layers consistently improve weak-localization performance over the single best layer for several methods.The paper also explains why Grad-CAM produces unmeaningful maps at some earlier layers: spatial gradient averaging eliminates class sensitivity across locations.Source: §1 · pp. 1–2
  • A class-sensitivity metric and meta-learning-inspired paradigm are introduced to improve a saliency method’s sensitivity to the output class.The paradigm adds an inner SGD step to the heatmap computation.Source: §1 · p. 2

2. Related work

Related work spans multiple backpropagation-based saliency families, tests of saliency properties and utility, and prior attempts to unify attribution methods.

Saliency methods

  • Backpropagation methods include input gradients, modified-ReLU methods such as Deconvnet and guided backprop, and intermediate-layer methods such as CAM and GradCAM.Other families include noise averaging, conservation-preserving methods, and reference-based methods.Source: §2 · p. 2

Assessing and unifying saliency methods

  • Prior work examined sensitivity to model weights and output classes, as well as invariance to input-intensity shifts.Gradient, deconvnet, and guided backprop were reported to produce class-insensitive heatmaps.Source: §2 · p. 2
  • Saliency utility has been evaluated for weak localization, attribution quality, and effects of perturbing image regions.One cited approach instead retrains classifiers on images with 20% of pixels perturbed.Source: §2 · p. 2
  • Earlier unification work emphasized methods that change backpropagation rules, while additive feature attribution unified examples including LIME, LRP, and DeepLIFT.The cited account also relates α-LRP, Excitation Backprop, and DeepLIFT.Source: §2 · p. 2

3. Method

The method considers labeled color-image training data and a neural network trained with cross-entropy. Its Extract & Aggregate framework interprets non-invasive saliency through spatial contributions to weight gradients.

3. Method

  • The setup models labeled color images as inputs to a neural network whose parameters are optimized with cross-entropy loss.The training examples are pairs of images and labels, and the network predicts their labels.Source: §3 · p. 2

3.1. Extract & Aggregate framework

  • The framework applies to methods such as gradient, linear approximation, and NormGrad variants that do not modify backpropagation.These methods use forward-pass activations together with information propagated backward from the network output.Source: §3.1 · pp. 2–3
  • Extract & Aggregate treats non-invasive saliency as spatial contributions to weight gradients followed by scalar aggregation into a heatmap.The gradient of spatially shared weights is expressed as a sum over locations, using activation gradients and input features.Source: §3.1 · p. 3

1. Extract.

The extraction step targets spatial contributions to a weight-gradient summation by selecting a spatially shared layer or inserting an identity layer.

1. Extract.

  • Choose a layer with spatially shared parameters, or insert an identity layer at the desired location, to compute spatial gradient contributions.Source: §1 · p. 3

2. Aggregate.

The section frames saliency construction as extracting location-wise gradient contributions and aggregating them into heatmaps, extending the approach with a virtual identity for arbitrary CNN locations.

3.1.1 Phase 1: Extract

  • For convolutional layers, each location contributes an outer product to the filter-weight gradient; spatially shared bias and scaling layers contribute channel-sized vectors.The convolutional contribution has size K′ × N^2K, while the bias and scaling contributions have size K.Source: §3.1.1 · p. 3

3.1.2 Virtual identity layer

  • A virtual identity layer extends contribution extraction to any CNN location without changing forward or backward information propagation.It is conceptual rather than physically added to the model, and allows activations and gradients to be examined at the same location.Source: §3.1.2 · p. 4

2. Aggregate.

  • Aggregation maps each location’s vector or matrix contribution to one scalar per spatial position.The framework identifies voting or summing, filtering, and maximum as aggregation choices.Source: §2 · p. 3, §3.1.3 · p. 4

3.1.3 Phase 2: Aggregate

  • Linear approximation uses sum aggregation, gradient uses the maximum absolute component, and CAM or Grad-CAM combine a virtual scaling identity with summing and positive filtering.The CAM and Grad-CAM description applies to architectures with global average pooling after convolutional layers.Source: §3.1.3 · p. 4
  • NormGrad uses a norm to aggregate spatial contributions, including the Frobenius norm for convolutional contribution matrices.For an existing 1 × 1 convolutional layer, the map is ∥gout∥∥xin∥.Source: §3.1.3 · pp. 4–5
  • Selective NormGrad applies positive filtering before the norm to make NormGrad maps more class selective.For a scaling identity layer, its map is ∥(gout⊙xout)+∥.Source: §3.1.3 · p. 5

4. Experiments

The experiments evaluate backpropagation-based saliency methods, test the virtual identity trick for NormGrad, and examine weighting saliency maps across layers.

4. Experiments

  • The experiments evaluate saliency maps on PASCAL VOC and ImageNet using VGG16 and ResNet50, with Pointing Game as the weak-localisation evaluation.The setup also specifies Spearman correlation for comparing saliency maps upsampled to 224 × 224.Source: §4 · p. 5

4.1. Justifying the virtual identity trick.

  • Across tested layers, the virtual identity trick yielded over 95% mean map correlation and a 0.53% ± 0.62% mean absolute difference in Pointing Game accuracy.These comparisons used 50k ImageNet validation images and tested VGG16 and ResNet50 at 1 × 1 and 3 × 3 convolutional layers.Source: §4.1 · p. 5

4.2. Combining saliency maps across layers.

  • Layer-combination experiments compare weighting methods on the PASCAL VOC Pointing Game, motivated by the possibility that saliency maps vary in meaningfulness across layers.The tested weights include feature spread and linear-classifier accuracy; performance is measured on both the “difficult” and “all” image sets.Source: §4.2 · p. 5

4. Uniform.

Layer-combined saliency maps improve weak-localization performance, while class sensitivity varies by depth and method. The section introduces a class-sensitivity metric and meta-saliency, which reduces max–min class-map correlations for nearly all methods.

4. Uniform.

  • Weighted saliency maps perform best across all four key test cases, and linear approximation with layer combination consistently leads overall at weak localization.The combined linear-approximation method beats more complex methods using a single forward-backward pass.Source: §4 · pp. 5–6, Table 3 · p. 6
  • Feature-spread and classification-accuracy weights generally increase with depth, while Grad-CAM performs poorly before the last convolutional layer.Grad-CAM spatially averages gradients, giving every heatmap location the same gradient vector; non-class-selective activations can therefore produce non-class-selective maps.Source: §4 · pp. 6–7, Figure 3 · p. 6

4.3. An explicit metric for class sensitivity

  • The class-sensitivity metric correlates saliency maps for the maximally and minimally predicted classes, with scores closer to zero preferred; methods show markedly different correlations.Excitation backprop and guided backprop score near 1, contrastive excitation backprop is closest to 0, and sum-aggregation methods have negative scores.Source: §4.3 · p. 7

4.4. Meta-saliency analysis

  • Meta-saliency adds an inner SGD step before saliency gradients, reducing max–min class-map correlation strength for nearly all methods and suggesting broader gains in class sensitivity.It requires one additional forward-backward pass compared with usual saliency backpropagation methods.Source: §4.4 · p. 8

4.5. Model weights sensitivity

  • With cascading weight randomization, linear approximation and selective NormGrad maps become scrambled by late convolutional layers, while meta-saliency maps take greater depth to randomize.The authors attribute the latter pattern to meta-saliency’s increased class selectivity and its target-class SGD step.Source: §4.5 · p. 8, Figure 8 · p. 8

5. Conclusions

The conclusion presents a unified framework for backpropagation-based saliency methods and summarizes its use in developing NormGrad, layer combination, and meta-saliency.

5. Conclusions

  • A framework based on spatial contributions to weight gradients unifies existing backpropagation saliency methods and supports NormGrad, layer combination for improved weak localization, and meta-saliency for greater target-class sensitivity.The authors report that layer combination can also produce high-resolution maps, and that meta-saliency applies to existing methods.Source: §5 · p. 8

Supplementary Material

The supplementary material specifies the Pointing Game evaluation, tests the virtual identity trick against direct convolutional contributions, and examines feature sensitivity and layer selection. It reports high identity-trick correlations and shows that the earliest layer is not the best individual choice.

A. Pointing Game details

  • The Pointing Game records a hit when the most salient point for an object class falls within 15 pixels of its ground-truth annotation, then averages per-class accuracy.Evaluation uses VOC 2007 test images and COCO 2014 validation images, including difficult subsets with small annotated object areas and another object class present.Source: A. Pointing Game details · p. 11

B. Virtual identity trick correlations

  • 94.33 is the minimum correlation between saliency maps with and without the virtual identity trick across VGG16 and ResNet50; pointing-game performance differs by 0.53%±0.62%.The authors take these results as evidence that the trick closely approximates spatial contributions from the original convolutional layers.Source: Table 4 · p. 11, B. Virtual identity trick correlations · p. 11

C. Performance of combining saliency maps

  • Both feature spread and classification accuracy generally rise with network depth, although VGG16 feature spread falls in its last two layers while classification accuracy rises; the earliest layer is never the best individual choice.Classification accuracy is described as more reliable because it directly reflects feature separability by class, whereas feature spread can vary with activation scale.Source: C. Performance of combining saliency… · p. 11

D. Meta-saliency analysis

The analysis compares meta- and non-meta saliency maps, tests sensitivity to model weights, and illustrates selective NormGrad for image captioning. Validation maps show lower meta/non-meta correlation, meta-saliency delays loss of focus under cascading randomization, and combining layers sharpens captioning maps.

D. Meta-saliency analysis

  • For NormGrad and selective NormGrad, meta/non-meta map correlation is lower on validation than training images, and selective NormGrad maps stabilize near the end of training.The lower validation correlation matches the authors’ hypothesis that the inner gradient step has greater impact on unseen images.Source: D. Meta-saliency analysis · pp. 11–12, Figure 10 · p. 13, Figure 11 · p. 13

E. Model weights sensitivity

  • For linear approximation and selective NormGrad, saliency maps progressively lose target focus as layer depth decreases, while meta-saliency delays that degradation to earlier layers.The comparison uses cascading randomization to illustrate model-weight sensitivity.Source: E. Model weights sensitivity · p. 12, Figure 13 · p. 14

F. Image captioning

  • In image captioning with a ResNet101 and LSTM, combining selective NormGrad maps from selected layers yields sharper, clearer image focus than individual-layer maps.The maps are computed before final global average pooling and at selected ReLU layers after downsampling shortcuts.Source: F. Image captioning · p. 12, Figure 14 · p. 15
  • Across VGG16 layers, gradient backpropagation works well except at the network end, while selective NormGrad and linear approximation appear more class selective than NormGrad.The comparison also shows NormGrad and selective NormGrad performing well across the network.Source: Figure 15 · p. 16
Loading 2004.02866v1…