PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values

Julius Adebayo, Justin Gilmer, Ian Goodfellow, Been Kim

arXiv:1810.03307v1cs.CVcs.LGstat.ML

TL;DR

Assessing whether local explanations faithfully reflect a DNN’s parameters matters as reliance on explanations for decisions grows. The paper tests this by reinitializing network weights and comparing explanations, finding similarity between randomly initialized and learned-weight DNNs that the authors conjecture reflects lower-level features and architectural priors.

  • Problem

    The paper asks how faithful local explanations are to a DNN’s parameter settings, a concern as explanations are increasingly relied on in decision making.

    Source: §1 · p. 1

  • Method

    The authors reinitialize DNN weights in different ways and compare explanations from randomized and learned-weight models, using visual inspection and Spearman rank correlation.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Explanations from randomly initialized and learned-weight DNNs are visually and quantitatively similar.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors posit that explanations are dominated by lower-level features, whose representations are significantly affected by DNN architecture even with random weights.

    Source: §4 · p. 3

    The proposed account is a hypothesis: explanations are modeled as depending on lower-layer features through a method-specific function.

    Source: §1 · p. 1

Abstract

from arXiv · show

Explaining the output of a complicated machine learning model like a deep neural network (DNN) is a central challenge in machine learning. Several proposed local explanation methods address this issue by identifying what dimensions of a single input are most responsible for a DNN's output. The goal of this work is to assess the sensitivity of local explanations to DNN parameter values. Somewhat surprisingly, we find that DNNs with randomly-initialized weights produce explanations that are both visually and quantitatively similar to those produced by DNNs with learned weights. Our conjecture is that this phenomenon occurs because these explanations are dominated by the lower level features of a DNN, and that a DNN's architecture provides a strong prior which significantly affects the representations learned at these lower layers. NOTE: This work is now subsumed by our recent manuscript, Sanity Checks for Saliency Maps (to appear NIPS 2018), where we expand on findings and address concerns raised in Sundararajan et. al. (2018).

1 INTRODUCTION

This work tests whether local explanations reflect a DNN’s learned parameter values by comparing explanations from learned and randomly initialized networks. The explanations remain visually and quantitatively similar, which the authors hypothesize reflects the influence of lower-level features.

1 INTRODUCTION

  • Because decisions increasingly rely on local explanations, the authors assess their fidelity to the parameter settings of the DNN being explained.A local explanation identifies input dimensions most responsible for a single DNN output.Source: §1 · p. 1
  • Explanations from randomly initialized and learned-weight DNNs are visually and quantitatively similar.The authors assess this similarity after reinitializing model weights in different ways.Source: §1 · p. 1
  • The study reinitializes DNN weights in different ways to empirically assess local explanations for faithfulness.It compares explanations produced with random weights against those from learned weights.Source: §1 · p. 1
  • The authors hypothesize that explanations depend on lower-level features, with DNN architecture shaping representations even under random initialization.They express this account as E(x_t) ∝ h(g(x; γ)), where g denotes the lower-layer function and h captures the explanation method.Source: §1 · p. 1

2 LOCAL EXPLANATION METHODS

The section introduces local explanation maps as input-shaped outputs and reviews the explanation methods examined in the work. These methods include gradient-based approaches, guided variants, and methods that average or measure variation across noisy inputs.

2 LOCAL EXPLANATION METHODS

  • An explanation map E maps an input in R^d to an object of the same shape.The formal setup treats the model as a function from inputs to class scores.Source: §2 · p. 2

2.1 OVERVIEW OF METHODS

  • The gradient explanation measures how input-dimension changes affect predictions locally, while Integrated Gradients sums gradients over scaled inputs to address saturation.Integrated Gradients uses a baseline input representing the absence of a feature.Source: §2.1 · p. 2
  • Guided Backpropagation sets negative gradient entries to zero during backpropagation through ReLU, while Guided GradCAM combines GradCAM with Guided Backpropagation for pixel-level detail.GradCAM uses the class-score gradient with respect to the last convolutional unit’s feature map.Source: §2.1 · pp. 2–3
  • SmoothGrad averages explanations of noisy input copies, and VarGrad measures their variance.The noise vectors are independently drawn from a normal distribution.Source: §2.1 · p. 3
  • The methods were selected for ease of implementation, running time, and memory requirements, and prior work reports equivalences among several approaches.The authors note equivalences shown by Ancona et al. (2017) and Lundberg & Lee (2017).Source: §2.1 · p. 3

3 LOCAL EXPLANATION SPECIFICITY

The experiments compare explanations from learned and reinitialized networks, finding method-dependent stability as reinitialization reaches lower layers. The authors relate this pattern to architectural priors that preserve low-level input structure.

3 LOCAL EXPLANATION SPECIFICITY

  • The study compares learned-weight explanations with explanations after cascading and independent layer reinitialization, measuring similarity with Spearman’s rank correlation.The experiments assess changes both visually and quantitatively.Source: §3 · p. 3
  • Guided back-propagation and Guided Grad-CAM show no change, while input-output gradients change most as reinitialization approaches lower layers.Integrated gradients and gradients initially decline but remain consistent through major degradation, particularly in middle blocks; similar results appear for MNIST CNN and MLP models.Source: §3 · p. 3
  • Prior work reports that random CNN features preserve translation invariance and frequency selectivity, and randomly initialized MNIST CNN features achieve 2 percent test error accuracy (Alain & Bengio, 2016).Alain & Bengio (2016) found the best-performing random features just after the first-layer ReLU.Source: §3 · p. 3
  • The authors posit that DNN architecture acts as a strong prior on image inputs, capturing low-level structure even with random initialization.They connect this proposed architectural prior to local explanations’ reliance on lower-level features.Source: §3 · p. 3

4 CONCLUSION

The paper assesses local explanations across several methods and finds that randomly initialized networks can produce explanations similar to those from learned networks. It proposes lower-level features and architectural priors as a possible explanation.

4 CONCLUSION

  • The authors empirically assess local explanations from several methods by comparing networks with randomly initialized and learned weights.They report visual and quantitative similarity between the resulting explanations.Source: §4 · p. 3
  • Explanations from randomly initialized DNNs are visually and quantitatively similar to those from DNNs with learned weights.Source: §4 · p. 3
  • The authors posit that explanations are dominated by lower-level features, whose representations are significantly affected by the DNN architecture even under random initialization.Source: §4 · p. 3

SUCCESSIVE RANDOMIZATION VISUALIZATION

The figures encode explanation similarity as rank correlation across layer-reinitialization conditions and model types. Figure 2 shows successive reinitialization, while Figure 3 shows independent reinitialization of each layer.

SUCCESSIVE RANDOMIZATION VISUALIZATION

  • Figure 2 plots rank correlation against layers or blocks, proceeding from the output layer, as successive reinitialization begins at the first layer.The black dashed line marks where successive randomization begins.Source: Figure 2 · p. 6
  • Figure 2 compares Inception v3 on ImageNet, a CNN on MNIST, and an MLP on MNIST.Source: Figure 2 · p. 6
  • Figure 3 plots rank correlation by layer or block for independent reinitialization, with a red dashed line marking zero correlation.Its panels cover Inception v3 on ImageNet, a CNN on MNIST, and an MLP on MNIST.Source: Figure 3 · p. 6

BACKGROUND & EXPERIMENTAL DETAILS

Experiments span ImageNet and MNIST across three model setups. Explanation similarity is assessed with Spearman rank correlation on 200 examples per dataset.

Loading 1810.03307v1…