PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the visualization of what a Deep Neural Network has learned

Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Bach, Klaus-Robert Müller

arXiv:1509.06321v1cs.CV

TL;DR

DNNs are powerful but opaque when explaining individual predictions, and heatmap quality lacks an objective quantitative measure. The paper evaluates heatmaps through region perturbation and compares three methods across multiple datasets. It reports that LRP provides a better explanation of classification decisions than sensitivity and deconvolution methods, while also examining heatmaps for unsupervised performance assessment.

  • Problem

    DNNs are difficult to interpret for individual unseen inputs, and heatmap quality lacks an objective quantitative evaluation method.

  • Method

    The paper introduces a generic region-perturbation framework for evaluating ordered heatmaps and extends prior evaluation from binary inputs to color images.

  • Results

    LRP provides a better explanation of DNN classification decisions than sensitivity-based and deconvolution methods across comparisons on SUN397, ILSVRC2012, and MIT Places.

  • Takeaways & Limitations

    Heatmaps can support quantitative comparison of explanation methods and may prioritize image regions for more detailed analysis or efficient processing.

Abstract

from arXiv · show

Deep Neural Networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multi-layer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the ''importance'' of individual pixels wrt the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012 and MIT Places data sets. Our main result is that the recently proposed Layer-wise Relevance Propagation (LRP) algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of neural network performance.

I. INTRODUCTION

DNNs achieve strong performance but remain difficult to interpret for individual predictions. The paper addresses the missing objective evaluation of heatmaps and introduces a generic framework for comparing them.

  • Motivation: DNNs are difficult to interpret because their nonlinear structure obscures why a particular unseen input receives its prediction.This individual-prediction question differs from identifying features that are salient across the training set.
  • Motivation: Heatmaps visualize the impact of image regions on a fixed image’s prediction, but their quality is difficult to evaluate quantitatively.Human assessment can use prior knowledge, whereas practical applications require an automated objective measure.
  • Contributions: The paper identifies objective heatmap-quality evaluation as an open issue and introduces a generic framework extending prior evaluation from binary inputs to color images.The framework is designed to support quantitative comparison of heatmap methods.
  • Contributions: The study compares three heatmap methods on SUN397, ILSVRC2012, and MIT Places, finding LRP more suitable for explaining DNN classification decisions than sensitivity and deconvolution methods.The paper also investigates heatmaps for assessing neural-network performance.
  • Heatmaps: A heatmap assigns each pixel a value derived from a class-discriminant function and can therefore be visualized in the same dimensionality as the input image.The paper reviews sensitivity analysis, deconvolution, and layer-wise relevance propagation as heatmap methods.

A. Sensitivity Heatmaps

Sensitivity heatmaps use partial derivatives to identify pixels whose small changes locally affect the network output. Their gradients are propagated through network layers and tend to emphasize local features.

  • Sensitivity analysis: Sensitivity heatmaps compute pixel importance from norms of neural-network partial derivatives.Large heatmap values indicate pixels whose small changes strongly affect the classification function.
  • Sensitivity analysis: The sensitivity measure captures how small pixel changes locally affect the network output, while taking the derivative norm discards the direction of change.Thus, the heatmap reflects local influence rather than a signed contribution.
  • Backpropagation: Backpropagation efficiently computes the partial derivatives across the network’s layers.The layer-to-layer rule propagates gradient signals through the model.
  • Backpropagation: During backpropagation, max-pooling redirects the gradient to the input neuron with the maximum activation.The backward procedure also filters signals through transposed convolutional filters and rectification operations.
  • Interpretation: Because rectification makes the backward mapping discontinuous and local, gradient-based heatmaps are expected to contain mostly local rather than global features.The paper contrasts evidence for a specific change with the full set of features composing an object.

B. Deconvolution Heatmaps

Deconvolution maps output activations back to pixel space through layer-specific backward rules. Its filtering projection ignores neuron activations, limiting image-specific explanations, while ReLU rectification discards negative evidence.

  • Deconvolution maps activations from the network output back to pixel space using a backpropagation rule.
  • Unpooling: The method uses recorded pooling maxima to redirect the backward signal to corresponding locations.
  • Nonlinearity: At ReLU layers, deconvolution passes the relevance signal through a ReLU function.
  • Filtering: Convolutional filtering uses transposed trained filters, and this projection does not depend on neuron activations.
  • Continuous rectification in the backward mapping can capture more global features, but deconvolution uses no activations in filtering.

C. Relevance Heatmaps

LRP decomposes a classification decision into pixel-wise relevances using layer-wise conservation. Its propagation rules preserve evidence, accommodate non-differentiable activations, and incorporate activations for image-specific explanations.

  • LRP decomposes a classification decision into pixel-wise relevances representing pixel contributions to the overall classification score.
  • Layer-wise conservation preserves the propagated quantity between adjacent neuron layers and can impose local relevance redistribution.
  • LRP repeatedly propagates relevance through layers so the resulting heatmap remains consistent with evidence for the predicted class.
  • Unlike gradient-based techniques, LRP applies to non-differentiable neuron activation functions.
  • LRP filtering uses both filter weights and lower-layer activations, enabling individual explanations without pooling layers.
  • In the MNIST example, LRP heatmaps adapt to rotated digits, whereas deconvolution heatmaps depend on weights rather than each image.

A. What makes a good heatmap?

A good heatmap should be evaluated against the classifier’s decision rather than human saliency intuitions. The paper contrasts LRP’s image-specific positive and negative evidence with deconvolution’s limitations.

  • Objective heatmap quality is difficult to define because human assessment relies on prior knowledge and experience of relevance.
  • The paper evaluates heatmap relevance using the classifier’s output and a perturbation method instead of modeling human perception.
  • LRP heatmaps show positive and negative evidence and adapt to individual images, whereas deconvolution lacks image-specific heatmaps in the example.
  • Deconvolution cannot explain evidence against the classification ‘9’ because negative evidence is discarded through ReLU backpropagation.
  • For distinguishing digit ‘3’ from ‘8’ and ‘9’, the relevant explanation depends on the image information provided to the classifier.

B. Salient Features vs. Individual Explanations

The framework distinguishes average class-level salient features from image-specific explanations and evaluates ordered heatmaps by progressively perturbing their most relevant regions. A steepest early score decline yields a larger AOPC.

  • B. Salient Features vs. Individual Explanations: Salient features describe average explanations distinguishing image categories, but may be meaningless or wrong for individual images.
  • C. Heatmap Evaluation Framework: The framework generalizes state-flip perturbation from binary pixels to arbitrary locations and local randomization or blurring.
  • C. Heatmap Evaluation Framework: The framework defines a heatmap as an ordered set of image locations, with ordering induced by scores from a heatmapping function.
  • C. Heatmap Evaluation Framework: Locations with higher heatmap scores appear earlier in the ordering, while mostly irrelevant regions appear later.
  • C. Heatmap Evaluation Framework: Most relevant first perturbation progressively removes information at the ordered locations using a local perturbation function.
  • C. Heatmap Evaluation Framework: AOPC is larger when the ordering ranks the most sensitive regions first, producing a steeper MoRF score decrease.

IV. EXPERIMENTAL RESULTS

The experiments compare LRP, deconvolution, and sensitivity heatmaps against a random baseline across three data sets, using the proposed heatmap evaluation procedure.

  • The evaluation compares LRP, deconvolution, and sensitivity heatmaps with a random-order baseline.The procedure also examines heatmap quality in relation to network performance.
  • Experiments use unchanged MIT Places and ImageNet classifiers, with MIT Places evaluated on MIT Places and SUN397 images and ImageNet on ILSVRC2012 images.The evaluation covers 5040 images per reported data-set condition.
  • Heatmaps are computed for each classifier’s predicted label, making the perturbation analysis fully unsupervised during testing.Perturbation replaces pixels in 9×9 non-overlapping regions, each covering 0.157% of the image, with uniformly sampled values.

B. Quantitative Comparison of Heatmapping Methods

LRP achieves the strongest quantitative heatmap evaluation across all three data sets, while also producing less complex explanations than the competing methods.

  • LRP heatmaps have the largest AOPC values relative to random ordering across all three data sets.The ϵ-LRP variant performs slightly better than α, β-LRP, while both variants are expected to perform similarly after parameter optimization.
  • The deconvolution method is the closest competitor and significantly outperforms the random baseline.LRP’s positive-negative evidence distinction and score normalization are reported to produce less noisy heatmaps than deconvolution.
  • Sensitivity heatmaps provide suboptimal explanations because local gradients may fail to capture global features of a class.On SUN397, sensitivity’s AOPC curve falls below random ranking, whereas on MIT Places it is initially clearly better than random.
  • LRP and deconvolution are less affected by off-manifold testing because they provide global explanations.The reported dataset differences are linked to image clutter and the distributional effects of region perturbation.
  • LRP heatmaps have the smallest file sizes and lowest complexity, followed by deconvolution, while sensitivity performs worst on both measures.The same ordering is reported using image entropy, and file-size differences are highly significant.
  • Subjective visualizations align with the quantitative results: sensitivity and deconvolution are noisier and less sparse than LRP heatmaps.For SUN397 and MIT Places, sensitivity is close to random, while LRP and deconvolution highlight structural scene elements.

D. Heatmap Quality and Neural Network Performance

The paper extends heatmap evaluation beyond subjective inspection by relating heatmap quality to neural-network performance and practical image-region prioritization.

  • AOPC-based heatmap quality can provide information about overall DNN performance.The authors suggest that better-trained networks capture relevant image structures more effectively and therefore produce more meaningful heatmaps.
  • The study evaluates heatmap methods across SUN397, ILSVRC2012, and MIT Places.The compared methods are image sensitivity, deconvolution, and LRP.
  • The work targets understanding and transparency of trained DNN decision-making by attributing individual-pixel contributions to inference results.
  • Heatmaps can prioritize image regions for more detailed analysis and potentially enable time-efficient processing focused on relevant areas.

APPENDIX CHOOSING A PERTURBATION METHOD

The appendix compares region perturbation strategies and formalizes evaluation through the contrast between most- and least-relevant-first perturbations. It emphasizes removing information while preserving image statistics and proximity to the data manifold.

  • Choosing a perturbation method: An ideal perturbation removes information without introducing spurious structures, substantially disrupting image statistics, or moving images far from the data manifold.
  • Choosing a perturbation method: Uniform, Dirichlet, and Constant replace local pixels using sampled or fixed RGB values, while Dirichlet is designed to retain image statistics.
  • Choosing a perturbation method: Blur applies a Gaussian filter with σ = 3 and is the only listed perturbation method that retains local information.
  • LeRF and MoRF: Least relevant first, or LeRF, perturbs locations in reverse relevance order.
  • LeRF and MoRF: The perturbation gap compares LeRF and MoRF to assess whether highly relevant regions lose class information while least relevant regions preserve it.The area between perturbation curves, or ABPC, indicates heatmap quality, class-scoring quality, and perturbation effectiveness.

COMPARISON OF PERTURBATION METHODS ON SUN397

On SUN397, the authors compare four perturbation functions using LRP heatmaps and classification outputs during most- and least-relevant-first removal. The reported comparison shows that blurring fails to remove information effectively.

  • The SUN397 comparison applies four perturbation functions to LRP heatmaps while measuring the highest linear-layer classification output.
  • Blurring fails to remove information on SUN397, with a relatively flat MoRF curve and continued DNN classification ability.
  • MoRF and LeRF curves are used to represent removal of the most and least relevant information, respectively.
Loading 1509.06321v1…