PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks

Jörg Wagner, Jan Mathias Köhler, Tobias Gindele, Leon Hetzel, Jakob Thaddäus Wiedemer, Sven Behnke

arXiv:1908.02686v1cs.CVcs.LG

TL;DR

Black-box CNN decisions are difficult to inspect, and existing defenses against faulty explanation evidence can constrain fine-grained results. FGVis filters gradients while optimizing perturbed images to produce fine-grained, valid model inputs. The authors report evaluations across models and datasets, including medical-image results and visual indication of color bias.

  • Problem

    Black-box CNNs are difficult to understand, and optimization-based explanations face adversarial evidence while existing defenses can add hyperparameters or constrain fine-grained output.

    Source: §1 · pp. 1–2

  • Method

    FGVis optimizes perturbed images and filters gradients that would otherwise lead to adversarial evidence, without requiring hyperparameters.

    Source: §1 · p. 2

  • Results

    The authors report evaluations across models and datasets, comparisons with other methods, faithfulness evaluation, class-discriminative visualizations, medical-dataset results, and visual indication of training-data color bias.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Because FGVis explanations are valid model inputs, their faithfulness can be directly verified with the model.

    Source: §1 · p. 2

    The method is formulated assuming a given CNN that maps an input image to softmax scores.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

To verify and validate networks, it is essential to gain insight into their decisions, limitations as well as possible shortcomings of training data. In this work, we propose a post-hoc, optimization based visual explanation method, which highlights the evidence in the input image for a specific prediction. Our approach is based on a novel technique to defend against adversarial evidence (i.e. faulty evidence due to artefacts) by filtering gradients during optimization. The defense does not depend on human-tuned parameters. It enables explanations which are both fine-grained and preserve the characteristics of images, such as edges and colors. The explanations are interpretable, suited for visualizing detailed evidence and can be tested as they are valid model inputs. We qualitatively and quantitatively evaluate our approach on a multitude of models and datasets.

1. Introduction

CNNs are used in real-world systems, but their black-box character makes it difficult to understand decisions, failure cases, and possible training-data shortcomings. FGVis addresses limitations of fine-grained optimization-based explanations with gradient filtering, producing detailed explanations that are valid model inputs.

1. Introduction

  • Black-box CNNs pose challenges in safety-critical use, while understanding their decisions can help examine failures and training-data shortcomings.Users may also trust a model more when they can understand its decision process and verify or anticipate outputs.Source: §1 · p. 1
  • Existing defenses against adversarial evidence add hyperparameters and can require reduced-resolution or smoothed explanations, limiting fine-grained evidence.Adversarial evidence is faulty evidence caused by artifacts introduced during explanation computation.Source: §1 · p. 2
  • FGVis selectively filters gradients that would otherwise lead to adversarial evidence, without hyperparameters, smoothing, or regularization.The authors describe this defense as the key to producing fine-grained explanations directly in image space.Source: §1 · p. 2
  • FGVis produces perturbed images that remove irrelevant or most relevant pixels, yielding explanations that are valid model inputs and can be directly checked for faithfulness.The paper presents this as a way to verify explanations using the model itself.Source: §1 · p. 2

2. Related Work

Visual explanation methods include gradient-based importance maps, activation-based heat maps, and perturbation-based approaches. FGVis builds on optimization of perturbed images, adding gradient filtering and per-pixel optimization to target fine-grained explanations without resolution limits.

2. Related Work

  • Backpropagation-based methods produce fine-grained importance maps but are generally less interpretable and may need proxy measures or preprocessing to assess faithfulness.Activation-based methods produce interpretable heat maps but are generally poorly suited to fine-grained evidence or color dependencies, and faithfulness is not guaranteed.Source: §2 · p. 2
  • Perturbation-based methods monitor predictions after input changes, but many produce coarse explanations or are time-consuming.Optimization-based perturbation methods generate valid model inputs, but existing approaches use additional defenses or constraints against adversarial evidence.Source: §2 · pp. 2–3
  • FGVis optimizes a perturbed input while filtering gradients during optimization and optimizing each pixel individually, avoiding hyperparameter tuning and resolution limits.The related-work discussion contrasts this with approaches that use regularization, coarse masks, stochasticity, or surrogate models to address adversarial evidence.Source: §2 · pp. 2–3

3. Explaining Model Predictions

The section frames explanations as local accounts of individual model predictions and optimizes perturbed images to produce interpretable, testable inputs optimized for faithfulness. It introduces fine-grained visual explanations and a gradient-filtering defense against adversarial evidence.

3.1. Perturbation based Visual Explanations

  • Preservation retains the smallest region needed to preserve the target output, while deletion removes the smallest region needed to change it.Both are defined using perturbed versions of the input image.Source: §3.1 · p. 3
  • A zero image serves as the reference, and a similarity metric compares the target-class outputs for the original and perturbed images to maintain faithfulness.The metric should be small when the target output is preserved and large when its probability drops substantially.Source: §3.1 · p. 3
  • Preservation masks are sparse and highlight important pixels, while deletion explanations are generally sparser because fewer pixels must be removed than retained.The deletion mask uses complementary colors; its complementary mask provides a true-color representation.Source: §3.1 · pp. 3–4

3.2. Defending against Adversarial Evidence

  • The defense filters gradients to restrict explanation activations to features present at the same locations in the original image, preventing new evidence without constraining resolution or smoothness.It applies activation bounds after nonlinearities and clips gradients during the backward pass, without adding hyperparameters or changing the model's forward structure.Source: §3.2 · p. 5
  • Without defense, adversarial explanations were generated for 100% of 1,000 ImageNet images; with defense, optimization nearly never succeeded.The two VGG16 examples generated with defense had low confidence, which the authors attribute to possible evidence for the target class in the images.Source: Table 1 · p. 6, §3.2 · p. 6

4. Qualitative Results

FGVis produces fine-grained, target-specific masks that support interpretability checks and expose architectural patterns and possible training-data biases. The qualitative evaluation examines these properties across models and image examples.

4.1. Interpretability

  • FGVis gives the most fine-grained GoogleNet masks; BBMP, Grad-CAM, and Occlusion are coarser and tend to include unnecessary background, while highlighted FGVis pixels form object edges.Gradient and Guided Backprop masks are most similar to FGVis, but FGVis explanations are valid network inputs whose highlighted-pixel deletion prevents correct object prediction.Source: §4.1 · p. 6, Figure 5 · p. 7
  • ResNet50 explanations show a grid-like pattern, also visible outside the object but to a lesser extent; the authors leave its detailed investigation for future work.The pattern is also observed in prior work, which proposes that ResNet skip connections may influence explanation behavior [31].Source: §4.1 · p. 6, Figure 6 · p. 7

4.2. Class Discriminative / Fine-Grained

  • FGVis isolates the chosen target class even when objects overlap, and its fence example shows precise pixel-level detail.The examples use the deletion game with GoogleNet.Source: §4.2 · pp. 6–7, Figure 4 · p. 6

4.3. Investigating Biases of Training Data

  • FGVis can depict object-co-occurrence bias, including sports equipment appearing alongside players in ImageNet.Source: §4.3 · p. 7
  • In VGG16 ImageNet examples, minivan masks emphasize edges, while school-bus masks consistently preserve yellow, indicating color importance for school-bus predictions.This is a first visual indication of color importance, which the authors then estimate quantitatively.Source: §4.3 · p. 7
  • 83.3% of 21 correctly classified minivans retained labels versus 8.3% of 42 school buses after channel swaps; for 80 classes, at least 75% of images lost true classification.The authors describe this as a quantitative estimate of color bias.Source: §4.3 · p. 7

5. Quantitative Results

The deletion metric evaluates explanation faithfulness by measuring prediction changes as important pixels are removed; FGVis outperforms other methods on both tested models. On DiaretDB1, FGVis performs comparably to or better than fully supervised approaches for lesion detection.

5.1. Faithfulness of Explanations

  • The deletion metric removes pixels ranked by an importance map and measures the resulting predicted-class probability across a deletion curve.The area under that curve is used as the faithfulness measure.Source: §5.1 · pp. 7–8
  • FGVis outperforms other explanation methods by a large margin on both models in the ImageNet validation-set deletion metric.The authors attribute the increase to fine-grained evidence; Table 2 reports deletion metric results, for which lower is better.Source: §5.1 · p. 8, Table 2 · p. 8

5.2. Visual explanation for medical images

  • FGVis performs comparably to or better than fully supervised approaches for detecting lesion presence, with particular strength on small RSD lesions.The evaluation uses 89 DiaretDB1 fundus images and compares image-level sensitivity for four lesion types.Source: §5.2 · p. 8, Table 3 · p. 8

6. Conclusion

The conclusion presents FGVis as a fine-grained image-space explanation method using a defense against adversarial evidence. Its evaluations cover faithfulness, class discrimination, medical images, and color bias in training data.

6. Conclusion

  • FGVis generates fine-grained visual explanations in image space using a technique to defend against adversarial evidence.The conclusion reports that the defense does not introduce hyperparameters.Source: §6 · p. 8
  • The paper evaluates FGVis across models and compares its explanations with other methods, including quantitative evaluation of faithfulness.It also reports class-discriminative visualizations and observations about ResNet50 explanations.Source: §6 · p. 8
  • The conclusion highlights FGVis results on a medical dataset and its use to visually indicate color bias in training data.Source: §6 · p. 8

A1. Defending against Adversarial Evidence

The appendix reports that FGVis explanations are based on image evidence and that the defense suppresses adversarial evidence hallucination. Without the defense, optimization can generate an explanation for a visually absent class, including from a black input.

A1. Defending against Adversarial Evidence

A2. Implementation Details

The appendix gives default optimization settings and a line-search procedure for selecting λ. It also reports that larger λ values yield sparser masks, especially masks relatively independent of learning rate.

A2. Implementation Details

  • Unless otherwise specified, explanations use SGD at learning rate 0.1 for 500 iterations, with pixelwise uniform noise initialization and gradient normalization by its maximum value.Initialization uses U(0, 0.01) for generation and repression, and U(0.99, 1) for preservation and deletion.Source: A2. Implementation Details · p. 12
  • The appendix illustrates an iguana-class explanation from a black image and describes a test generating explanations for 998 ImageNet classes from the same black input.The test averages how often an adversarial class could be generated with the generation game at λ = 0.Source: Figure A1 · p. 12, Table A1 · p. 12
  • For most-likely-class explanations, a line search tests 13 equally spaced λ values from 10^-4 to 10^-10 and stops according to the game-specific class-probability condition.The condition is a class shift for deletion and repression, or the highest probability among classes for preservation and generation.Source: A2. Implementation Details · p. 12
  • Higher λ produces sparser masks, and at especially high λ values the masks are rather independent of the SGD learning rate.The passage attributes sparsity to greater weighting of the loss function's sparsity term.Source: A2. Implementation Details · p. 13

A3. Qualitative Results

The appendix examines reference-image entropy, FGVis’s class discrimination and fine-grained detail, learned training-data biases, architecture-specific explanations, and differences among explanation games.

A3.1. Entropy of Reference Images

  • For all models except GoogleNet, the zero-image reference has the highest entropy; increasing noise or reducing blur generally lowers entropy, with GoogleNet as an exception.The authors choose the zero reference for its entropy and low computational effort.Source: A3.1. Entropy of Reference Images · p. 13

A3.2. Class Discriminative / Fine-Grained

  • FGVis distinguishes overlapping objects and produces fine-grained explanations in the deletion-game examples.The examples use GoogleNet and objects chosen from different categories to limit overlapping evidence.Source: A3.2. Class Discriminative /… · p. 13

A3.3. Investigating Biases of Training Data

  • Sports equipment co-occurrence appears as learned bias: deletion explanations include player pixels, while channel-swapping analysis examines ImageNet color dependence.The table reports classes most and least affected by swapping color channels, including redbone and potpie among the most dependent and zebra and electric fan among the least dependent.Source: A3.3. Investigating Biases of Training… · p. 13, Table A3 · p. 18
  • School-bus explanations retain yellow, whereas white or grey minivans can appear greenish-blue rather than preserving their original color.The examples show explanations for correctly classified school-bus and minivan images.Source: A3.3. Investigating Biases of Training… · pp. 13–14

A3.4. Comparison of Networks

  • ResNet50 and VGG16 explanations are denser than those of the other compared networks; VGG16 also highlights image edges, while ResNet50 shows a grid-like pattern.The authors suggest density could indicate greater robustness, but say further research is needed.Source: A3.4. Comparison of Networks · p. 20

A3.5. Comparison of Games

  • Repression and deletion explanations are qualitatively similar, as are generation and preservation explanations; repression and deletion are sparser than generation and preservation.Repression and deletion use the same optimization with different starting masks; the authors attribute sparsity differences to the distinct evidence needed to suppress versus evoke a class output.Source: A3.5. Comparison of Games · pp. 20–21

A4. Quantitative Results

This appendix describes the deletion metric used to evaluate explanation faithfulness and presents FGVis localization results on retinal images, including alignment with lesion markings and additional highlighted structures.

A4.1. Faithfulness of Explanations

  • The deletion metric ranks pixels by evidence for the predicted class, removes them successively, and summarizes the resulting class-probability curve with area under the curve.The reported procedure uses the mean mask as the importance map and sets removed pixels to zero.Source: A4.1. Faithfulness of Explanations · p. 21, Figure A20 · p. 26

A4.2. Visual Explanation for Medical Images

  • FGVis highlights red small dots, or microaneurysms, which cover only a few pixels and require zooming in to see in the examples.The passage identifies these as the earliest sign of diabetic retinopathy.Source: A4.2. Visual Explanation for Medical… · p. 21
  • On DiaretDB1, weakly supervised FGVis predictions mostly fall within lesion ground-truth markings, while optic-disk and blood-vessel changes can also be highlighted despite lacking labels.The dataset contains 89 color fundus images, and pixels marked by at least three of four experts are treated as lesions.Source: A4.2. Visual Explanation for Medical… · p. 21, Figure A21 · p. 27
Loading 1908.02686v1…