PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating the influence of noise and distractors on the interpretation of neural networks

Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, Sven Dähne

arXiv:1611.07270v1stat.MLcs.LG

TL;DR

The paper addresses how noise and distractor dimensions affect neural-network explanations, a concern for interpreting methods whose rules encode assumptions about data variation. It analyzes those assumptions within deep Taylor decomposition and reports qualitative differences among methods as noise increases, while noting that broader evaluation is needed.

  • Problem

    Explanation methods’ behavior under noise and distractor dimensions is not explicitly considered, despite gradient-based approaches being described as uninformative for noisy linear classifiers.

    Source: §2 · p. 2

  • Method

    The paper relates deep Taylor explanation rules to generative assumptions about task-related and unrelated variation, and proposes rules that estimate task-related directions from data.

    Source: §3 · pp. 2–3

  • Results

    As noise increases, the z-rule’s heatmap quality degrades, while w+ and a+ explanations are less affected; a+ focuses more on the MNIST center region where class-relevant variation is concentrated.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Different rules correspond to different generative models and influence explanations, so selecting an appropriate model depends on the task and network architecture.

    Source: §5 · p. 4

    Further research on large networks and additional datasets, plus benchmarks with known relevant components, is needed for more reliable evaluation and model selection.

    Source: §5 · p. 5

Abstract

from arXiv · show

Understanding neural networks is becoming increasingly important. Over the last few years different types of visualisation and explanation methods have been proposed. However, none of them explicitly considered the behaviour in the presence of noise and distracting elements. In this work, we will show how noise and distracting dimensions can influence the result of an explanation model. This gives a new theoretical insights to aid selection of the most appropriate explanation model within the deep-Taylor decomposition framework.

1 Introduction

The introduction situates explanation methods within efforts to make neural networks interpretable and motivates examining them under noise and distractor signals. It traces methods from feature visualization and saliency maps to LRP and deep Taylor decomposition.

1 Introduction

  • Zeiler et al. [7] proposed Deconvnet to visualize learned features, while Simonyan et al. [6] proposed class-specific saliency maps based on output gradients with respect to inputs.The approaches mainly differ in the positioning of ReLU units [6].Source: §1 · p. 1
  • Bach et al. introduced layer-wise relevance propagation by defining relevance as each input variable’s contribution to a prediction.Source: §1 · p. 1
  • Montavon et al. [4] provided a theoretical basis for LRP and generalized it to deep Taylor decomposition; noise and distractors motivate the paper’s analysis of explanation methods.The introduction notes that gradient-based procedures are considered unsuitable for neuroimaging interpretation because of noise and distractor signals.Source: §1 · p. 1

2 Linear projections in the presence of noise and distractors

The section models observed data as task-related signal, task-unrelated variation, and noise, then examines what a linear projection must satisfy to recover the task signal. Its weight vector need not align with the task-related direction, which matters for interpreting gradient-based explanations.

2 Linear projections in the presence of noise and distractors

  • Observed data comprise a task-related component, task-unrelated latent signals, and zero-mean Gaussian noise, with shared patterns across the dataset.The task signal is example-specific, while its pattern and the distractor patterns are shared across examples.Source: §2 · p. 1
  • The task-related pattern describes changes in observed data that alter the desired output, whereas distractor directions allow data to change without altering it.The task and distractor patterns are not required to be orthogonal.Source: §2 · pp. 1–2
  • To recover the target signal, a linear projection must satisfy w^T a_t = 1 and be orthogonal to task-unrelated directions, w^T A_n = 0.Given a_t, infinitely many weight vectors can meet the recovery condition.Source: §2 · p. 2
  • Because the orthogonality constraint can make the projection’s steepest-ascent direction differ from task-related variation, gradient-based explanations can be uninformative for noisy linear classifiers.The analysis also makes explicit that explanation rules and visualizations implicitly impose assumptions about the data and networks.Source: §2 · p. 2

3 What assumptions do explanation rules make implicitly?

The section interprets deep Taylor rules as encoding assumptions about data variation through their root-point search directions. It introduces a data-adaptive rule that estimates task-related directions, alongside a modified gradient-based rule.

3 What assumptions do explanation rules make implicitly?

  • Deep Taylor decomposition recursively redistributes output relevance to input features using layer-wise first-degree Taylor expansions around root points while conserving layer relevance.For ReLU nonlinearities, the residual vanishes; neuron relevance is redistributed to the preceding layer.Source: §3 · p. 2
  • The paper proposes choosing root-point search directions according to the generative model, ideally along task-related data variation, rather than only input-domain constraints.The authors state that space constraints prevent consideration of every known explanation rule.Source: §3 · p. 2
  • The z-rule uses a constant-zero root point, corresponding to an assumption that noise is absent and all input directions are informative.For ReLU and max-pooling, the input-weighted saliency map corresponds to the z-rule under the stated bias assumption.Source: §3 · pp. 2–3
  • The w2-rule selects the closest root along the gradient; the proposed w+-rule restricts its direction to active neurons and assumes detected features follow the gradient there.The gradient-direction assumption is also used by saliency maps and Deconvnet.Source: §3 · p. 3
  • The proposed a-rule estimates task-related directions by regressing layer outputs on inputs under an independence assumption, while a+-rule adapts those directions to active neurons and rescales after ReLU.The a-rule’s explanation is not data-point dependent; the a+-rule adds adaptivity.Source: §3 · p. 3

4 Experiments and Discussion

On MNIST, the z-rule’s heatmaps degrade as noise increases, while w+ and a+ are less affected. The rules also differ in where they assign relevance, including whether it falls outside the digit.

4 Experiments and Discussion

  • A preliminary MNIST digit-classification evaluation uses an MLP with images scaled to [0, 1] and normal noise standard deviation 0.0 to 0.8, comparing saliency maps and the z-rule with proposed w+ and a+ rules.The MLP has one hidden layer with 200 fully connected ReLU neurons and a softmax output layer.Source: §4 · p. 3
  • As noise increases, the z-rule heatmap degrades, while w+ and a+ explanations are much less affected.The z-rule’s implied generative model assumes no noise is present.Source: §4 · p. 4
  • The a+ rule emphasizes the image center, where MNIST digits vary and class information is present, whereas w+ assigns positive relevance across the whole image.The authors attribute this difference to w+’s gradient having non-zero weights over the entire image, while learned a vectors show class-relevant variation directions.Source: §4 · p. 4
  • For the digit four, Simonyan’s saliency map and a+ identify the gaps above and below its vertical strokes as important, since changing them could produce a nine or eight.This is a qualitative observation in the reported setting.Source: §4 · p. 4
  • Across the additional MNIST heatmaps, the z-rule focuses on the digit, while the other methods assign substantial relevance outside it, especially a+ on nearby black regions.Figure 2 compares heatmaps for different MNIST digits at noise level 0.2.Source: §4 · p. 4, Figure 2 · p. 4

5 Conclusion

The conclusion presents a generative-model perspective on deep Taylor explanations and introduces a rule that learns task-related variation directions. It calls for broader evaluation before choosing an appropriate model for each task and architecture.

5 Conclusion

  • The paper relates different explanation rules to different generative models of observed data, which influence the resulting explanations.The conclusion frames this as a theoretical view using deep Taylor decomposition.Source: §5 · p. 4
  • The introduced explanation rule learns directions of variation related to the task.Source: §5 · p. 4
  • Choosing an appropriate model for each task and architecture requires further research on larger networks and additional datasets, plus benchmarks with known relevant components for more reliable evaluation.Source: §5 · pp. 4–5

A The connection between the z-rule and the Simonyan saliancy map.

The section states an equivalence between the z-rule and the input-weighted gradient under ReLU or max-pooling activations. It accounts for inactive neurons and blocked paths by summing only over active neurons.

A The connection between the z-rule and the Simonyan saliancy map.

A.1 Proof

The proof defines relevance at the top layer, redistributes it to the layer below, and expresses that redistribution using the previous layer’s activation and the gradient.

A.1 Proof

  • The proof begins by defining relevance for a neuron in the top layer.Source: §A.1 · p. 6
  • It then defines the relevance of the layer below.Source: §A.1 · p. 6
  • The lower-layer relevance is expressed as the gradient multiplied by the previous layer’s neuron activation.Source: §A.1 · p. 6
  • The z-rule is equivalent to the saliency map multiplied elementwise with the input.Source: §A.1 · p. 6

A.2 Implications

For ReLU/max-pooling networks, the weight-vector gradient yields a datapoint-specific linear projection equivalent to the network’s operation, supporting the z-rule as a valid decomposition of the computed function. This implication assumes no structured noise or distracting pattern in the data.

  • For networks with only ReLU and max-pooling units, the weight-vector gradient yields a linear projection that performs the same operation as the network for that datapoint.Source: §A.2 · p. 7
  • Because the z-rule decomposes into the gradient multiplied by the input, it validly decomposes the function computed by these networks.Source: §A.2 · p. 7
  • This decomposition assumes the data contain no structured noise or distracting pattern and treats the network as a datapoint-specific function generator.Source: §A.2 · p. 7
Loading 1611.07270v1…