PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation

Brian Kenji Iwana, Ryohei Kuroki, Seiichi Uchida

arXiv:1908.04351v3cs.CVcs.LGcs.NE

TL;DR

CNNs are difficult to interpret, and LRP may misattribute input regions because it considers only the target class. The paper introduces SGLRP, which propagates softmax gradients to produce class-discriminative relevance maps. Its evaluations report localization of contributing regions and better performance than other LRP-based visualizations.

  • Problem

    CNNs are often treated as black boxes, and LRP's focus on only the target class can lead to input-region misattribution.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    SGLRP extends LRP by using the softmax gradient as output relevance and subtracting relevance from non-target classes to produce class-discriminative maps.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Qualitative and quantitative evaluations report localization of classification-contributing regions and better performance than other LRP-based visualizations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that SGLRP attributes input regions associated with object classification and distinguishes target-class relevance from non-target classes.

    Source: §6 · p. 8

    SGLRP tends to perform poorly when the target fills the image or is an insignificant part of it, where non-target occluding classes can overwhelm the target.

    Source: §5.2.3 · p. 6

Abstract

from arXiv · show

Convolutional Neural Networks (CNN) have become state-of-the-art in the field of image classification. However, not everything is understood about their inner representations. This paper tackles the interpretability and explainability of the predictions of CNNs for multi-class classification problems. Specifically, we propose a novel visualization method of pixel-wise input attribution called Softmax-Gradient Layer-wise Relevance Propagation (SGLRP). The proposed model is a class discriminate extension to Deep Taylor Decomposition (DTD) using the gradient of softmax to back propagate the relevance of the output probability to the input image. Through qualitative and quantitative analysis, we demonstrate that SGLRP can successfully localize and attribute the regions on input images which contribute to a target object's classification. We show that the proposed method excels at discriminating the target objects class from the other possible objects in the images. We confirm that SGLRP performs better than existing Layer-wise Relevance Propagation (LRP) based methods and can help in the understanding of the decision process of CNNs.

1. Introduction

CNNs can be difficult to interpret, motivating a method for visualizing pixel-wise contributions to object classification. SGLRP uses softmax gradients to produce class-discriminative relevance maps and is reported to outperform other LRP-based visualizations.

1. Introduction

  • Deep CNNs such as VGG can have millions of parameters, making their internal representations and decision processes difficult to understand.The introduction frames this as a challenge for interpreting CNN predictions.Source: §1 · p. 1
  • SGLRP visualizes pixel-wise input contributions to an object's classification by propagating the gradient of softmax through a trained network.It uses the gradient of softmax as the output layer's starting relevance signal, subtracting relevance from non-target classes based on their classification probabilities.Source: §1 · p. 2
  • Qualitative evaluations show that SGLRP produces class-discriminative maps that localize contributing regions and objects, including when multiple classes are present.The paper describes the method as robust in images containing multiple different classes.Source: §1 · p. 2
  • Quantitative evaluations report that SGLRP performs better than other LRP-based visualizations.The evaluations use an object pointing game and a patch removal evaluation.Source: §1 · p. 2

2. Related Work

CNN visualization methods include input modification, back-propagation, and CAM approaches, alongside other ways of inspecting activations or hidden representations. These approaches differ in how they generate explanations and in their stated constraints.

2.1. Input Modification Methods

  • Input modification methods probe output changes using perturbations, masks, or noise, but can be computationally costly because they require exhaustive input modifications.Zeiler and Fergus [41] generate heatmaps from prediction-probability drops after masking image patches.Source: §2.1 · p. 2

2.2. Back Propagation-based Methods

  • Back-propagation methods trace output contributions toward the input; DTD-based LRP and CLRP redistribute relevance, while DeepLIFT, SHAP, and Integrated Gradients address gradient-related attribution issues.DeconvNet [41] [28] and Guided Back Propagation [30] modify backward signals; gradients can mislead with discontinuous or vanishing activations [27] [17] [32] [22].Source: §2.2 · p. 2

2.3. Class Activation Mapping Methods

  • CAM combines output weights and final convolutional feature maps into a heatmap, but requires a final GAP layer and cannot be applied to every network.Grad-CAM [26] can target any layer using gradient information, while Guided Grad-CAM [26] combines CAM with pixel-wise methods.Source: §2.3 · p. 2

2.4. Miscellaneous Methods

  • Other approaches inspect maximal layer activations or attention, or analyze hidden layers with dimensionality reduction, graphs, matrix factorization, and community detection.The cited examples include feature-map visualization [41] [40], attention visualization [5] [39], and Relative Neighbor Graphs.Source: §2.4 · p. 3

3. Relevance Propagation

Layer-wise Relevance Propagation (LRP) traces output relevance backward through a network to produce pixel-wise input maps, but considering only the target class can misattribute shared evidence. Contrastive LRP (CLRP) subtracts non-target relevance, balancing target and non-target contributions while risking cancellation when non-target classes are penalized equally.

3.1. Layer-wise Relevance Propagation

  • LRP propagates target-output relevance backward through the network to produce a pixel-wise map of input contributions.The method recursively propagates relevance across layers, applying different rules for positive activations and cases that can include negative values.Source: §3.1 · p. 3
  • Because LRP considers only the target class, shared evidence can be misattributed even when another input distinguishes the classes.In the example, z(2)2 contributes to both outputs, while x(1)1 is the discriminating input leading to ŷ2.Source: §3.1 · p. 3

3.2. Contrastive Layer-wise Relevance Propagation

  • CLRP subtracts relevance for non-target classes from target relevance to separate target-class contributions from competing classes.This produces maps in which non-target classes are penalized.Source: §3.2 · p. 4
  • CLRP balances target relevance against the summed penalties from other classes, which helps prevent non-target relevance from overpowering it, especially with many classes.The stated balance equates propagated target relevance with the sum of penalties from non-target classes.Source: §3.2 · p. 4
  • Equal penalties can reduce a strongly target-correlated node’s relevance to zero and ultimately misattribute relevance to another input.The example attributes this behavior to equal weighting of non-target nodes.Source: §3.2 · p. 4

4. Softmax-Gradient Layer-wise Relevance Propagation

SGLRP extends LRP by initializing relevance with the gradient of the softmax output for the target class, then propagating it through the network. This makes non-target subtraction responsive to class probabilities and specifically targets conflicting objects.

  • SGLRP uses the gradient of the target softmax output with respect to each output node’s intermediate value as the initial relevance for LRP propagation.The resulting input relevance map is calculated using the LRP propagation equations with this changed output-layer relevance.Source: §4 · p. 4
  • The softmax gradient weights non-target classes by their predicted probabilities, unlike LRP’s omission of other classes or CLRP’s fixed penalty.The authors describe this as a more natural way to remove relevance from non-target classes.Source: §4 · p. 4
  • SGLRP preserves a balance between target and other classes while assigning greater subtractive relevance to interfering objects present in the image.Unlike CLRP’s uniform penalty, the subtraction specifically targets non-target objects found in the image.Source: §4 · p. 4

5. Experimental Results

The section compares the proposed method with existing methods using qualitative and quantitative evaluations.

  • The proposed method is compared with existing methods through qualitative and quantitative evaluations.Source: §5 · p. 5

5.1. Dataset and Architecture

The experiments use VGG16 trained on ImageNet ILSVRC2012, with relevance maps prepared for the dataset’s 50,000 test images.

  • The experiments use VGG16 trained on ILSVRC2012, a 1,000-class dataset with about 1.2 million images.The model has 69.63% accuracy and the authors say the method can be applied to arbitrary neural networks, including other deep CNNs.Source: §5.1 · p. 5
  • For quantitative evaluation, relevance maps are prepared for 50,000 test images resized and cropped to 224 × 224 × 3.Source: §5.1 · p. 5

5.2. Qualitative Evaluation

The qualitative evaluation compares relevance maps from SGLRP, LRP, CLRP, and Guided Grad-CAM, including target classes that may differ from the image’s predicted class. Across examples, SGLRP separates target objects from conflicting objects, though it performs poorly when the target fills the image or is a minor part of it.

5.2.1 Visualizations of Contrastive Classes

  • SGLRP isolates target-object relevance with less background noise and class overlap than CLRP, while LRP often highlights non-target regions and background.Figure 6 visualizes relevance for “Zebra”; the authors attribute the difference to SGLRP specifically removing non-target objects found in the image.Source: §5.2.1 · p. 5, Figure 6 · p. 5

5.2.2 Visualizations Using Other Models

  • Examples on ResNet152 and DenseNet121 resemble VGG16 results, while LRP-based methods produce very sparse maps on InceptionV3.The authors suggest InceptionV3’s complex structure may account for the sparse maps.Source: §5.2.2 · p. 5, Figure 8 · p. 6

5.2.3 Performance Targeting the Ground Truth Class

  • When targeting ground-truth classes, SGLRP removes conflicting objects in examples including “Spider Web,” “Submarine,” and “Banjo,” and tends to outperform CLRP among similar classes.The authors note CLRP has more difficulty for “Water Ouzel” among a dataset with about 60 bird classes.Source: §5.2.3 · pp. 5–6, Figure 7 · p. 6
  • SGLRP performs poorly when the target object fills the image or forms an insignificant part of it, as non-target occluding classes can overwhelm it.The authors report the same problem for CLRP.Source: §5.2.3 · p. 6

5.3. Quantitative Evaluation

The quantitative evaluations compare relevance methods through patch masking and the Pointing Game. SGLRP generally outperforms other LRP-based methods, with its advantage depending on the target class and evaluation setting.

5.3. Quantitative Evaluation

  • Three evaluations compare LRP, CLRP, Guided Grad-CAM, and Random using patch masking for the ground-truth and second-most-probable classes, plus the Pointing Game.Patch masking removes the region around a method’s highest relevance value and measures the target output drop after reclassification.Source: §5.3 · p. 6, §5.3.1 · p. 7

5.3.2 Maximal Patch Masking of the Second Probable Class

  • For the second-highest-probability class, SGLRP performed best across all patch sizes; Guided Grad-CAM fell below CLRP, while masking increased target probability for LRP and Random.The authors interpret this as evidence that SGLRP and CLRP can discriminate target classes, while Guided Grad-CAM is more limited to the dominant, most salient class.Source: Figure 11 · p. 7, §5.3.2 · p. 8

5.3.1 Maximal Patch Masking of the Ground Truth Class

  • Ground-truth patch masking reduced the target output more for SGLRP than other LRP-based methods at every patch size, but Guided Grad-CAM was slightly stronger for p ≥5.The tested patch sizes were p = 1, 3, 5, 7, 9.Source: Figure 11 · p. 7, §5.3.1 · p. 7

5.3.3 Pointing Game

  • The Pointing Game counts thresholded relevance points inside the target bounding box as hits and points outside it as misses.The threshold is set by the percent energy in the relevance map.Source: Figure 12 · p. 8, §5.3.3 · p. 8
  • For energy levels E ≥67%, SGLRP had the highest Pointing Accuracy; Guided Grad-CAM was more accurate for E < 67%.Higher accuracy at high energy means fewer identified relevant pixels fall outside the target bounding box.Source: Figure 13 · p. 8, §5.3.3 · p. 8

6. Conclusion

SGLRP extends LRP by using the softmax gradient to subtract non-target-class relevance, adding class discrimination. The paper reports qualitative and quantitative evidence that it attributes input regions contributing to object classification.

6. Conclusion

  • SGLRP extends LRP by subtracting relevance from non-target classes using the softmax gradient to enable class discrimination.Source: §6 · p. 8
  • Qualitative and quantitative evaluations demonstrate SGLRP’s strength at attributing input regions that lead to object classification.Source: §6 · p. 8
  • The authors propose that future work could extend SGLRP to any neural-network structure and applications such as localization and visualization.Source: §6 · p. 8

Related papers

Loading 1908.04351v3…