Source-linked AI summary
Learning how to explain neural networks: PatternNet and PatternAttribution
Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, Sven Dähne
TL;DR
The paper addresses whether neural-network explanation methods produce theoretically correct explanations even in the simple linear-model limit. It analyzes signal and distractor separation, proposes PatternNet and PatternAttribution, and reports improved signal visualizations and attributions for deep networks.
Problem
Existing explanation methods are used for deep networks but do not provide the correct explanation even for a simple linear model, making the linear-model limit a relevant test.
Method
The paper analytically studies linear models, derives an objective for neuron-wise signal estimation, and uses learned signal estimates for PatternNet and PatternAttribution.
Results
PatternNet and PatternAttribution produce qualitatively improved signal visualizations and attributions, with experiments showing optimized estimators outperforming random and filter-based alternatives.
Takeaways & Limitations
Explanation quality depends on estimating the detected signal rather than treating model weights or implicit backward rules as the signal itself.
Takeaways & Limitations
The analytical treatment restricts networks to linear neurons with selected nonlinearities and treats nonlinearities independently.
Abstract
from arXiv · showhide
DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.
1 INTRODUCTION
The paper argues that existing neural-network explanation methods should first be tested on linear models, where signal and distractors can be controlled and explanations tracked analytically. This analysis motivates PatternNet and PatternAttribution, which are then evaluated on practical networks and datasets.
- Deep classifiers detect relevant signals by filtering out nonrelevant and distracting components through many layers with millions of parameters.
- Existing explanation methods aim to propagate a classifier’s output signal back to the input to show which aspects support its decision.
- The paper first analyzes explanation methods on purely linear models and linearly generated data, enabling control of signal and distractor encoding and analytical tracking of explanations.
- PatternNet and PatternAttribution are proposed to alleviate these flaws, then evaluated qualitatively and empirically on practical networks and datasets.
2 UNDERSTANDING LINEAR MODELS
In a linear model with signal and distractors, the optimal weight vector primarily cancels distractors rather than indicating the signal direction. The paper uses this distinction to motivate tests of whether the same regime holds in deep networks.
- The toy data decomposes as x = s + d, with s = a_sy carrying information about y and d representing a distractor.
- The trained regression filter w = [1, −1]^T extracts y by cancelling the distractor rather than directly representing the signal.
- The optimal weight vector generally does not align with the signal direction; it is optimized so w^T d = 0, and therefore follows changes in distractor direction.
- With signal and distractors, the weight direction is largely determined by the distractor, so the signal direction cannot be inferred from weights alone.
- Isotropic Gaussian noise cannot be effectively cancelled by a weight direction; it shrinks weights like l2 regularization, while structured distractors alter their direction.
- In VGG16, learned directions capture more information than weight-vector directions, indicating that the network operates largely in a distractor regime.
3 OVERVIEW OF EXPLANATION APPROACHES AND THEIR BEHAVIOR
The paper distinguishes function, signal, and attribution explanations. It shows that gradient-based signal visualizations reduce to filters in linear models, while attribution methods require estimating and removing distractors.
- Function, signal, and attribution visualizations present different, complementary information about a network.
- Functions – gradients, saliency map: A saliency map approximates function sensitivity by estimating how movement along the model-gradient direction changes the output.
- Signal – DeConvNet, Guided BackProp, PatternNet: Signal methods seek the input component that caused network activations; DeConvNet and Guided BackProp modify rectifier treatment during backward visualization.
- Signal – DeConvNet, Guided BackProp, PatternNet: For linear models, DeConvNet and Guided BackProp reduce to the gradient and show the filter w rather than the pattern a_s or signal s.
- Signal – DeConvNet, Guided BackProp, PatternNet: PatternNet estimates the correct signal direction because filter direction does not necessarily coincide with signal direction, despite producing compelling visualizations.
- Attribution – LRP, Deep Taylor Decomposition, PatternAttribution: For linear models, optimal attribution is r_input = w ⊙ a_y, while PatternAttribution learns the distractor-related root point from data for deep Taylor decomposition.
4 LEARNING TO ESTIMATE THE SIGNAL
The paper defines a criterion for learning signal estimators from data, showing why common filter- and input-based explanations retain distractor information. It then derives estimators for linear and ReLU settings and uses them to construct PatternNet and PatternAttribution.
- Quality criterion for signal estimators: Signal estimation is evaluated by how much information about the output remains reconstructable from residuals after subtracting the estimated signal.The criterion favors estimators whose residuals contain little linearly recoverable information about y, yielding larger ρ(S).
- Existing signal estimators: The identity estimator treats the entire input as signal, so LRP’s z-rule includes distractor contributions in its attribution.Forward filters cancel distractors, but backward element-wise multiplication cannot, producing noisy visualizations.
- Existing signal estimators: The filter-based estimator assumes the signal direction follows the weight vector, but it fails on the toy linear example and is empirically sub-optimal in Fig. 3.Its linear-model attribution is proportional to w⊙w multiplied by the model output.
- Linear estimator: The learned linear estimator recovers the toy-example signal and works well for convolutional layers, although ReLU-equipped dense layers retain considerable correlation.It is equivalent to the filter-based approach only when distractors are orthogonal to the signal.
- Two-component estimator: Because ReLU creates distinct positive and negative regimes, the two-component estimator learns separate patterns a+ and a− to approximate the signal.The distinction is needed because otherwise information about whether a neuron fired remains in the distractor.
- PatternNet and PatternAttribution: PatternNet back-projects estimated signal using informative directions, while PatternAttribution exposes w⊙a+ and improves LRP by ignoring distractors.Both methods are implemented through backward passes that replace ordinary weights with learned signal directions or their attribution-weighted versions.
5 EXPERIMENTS AND DISCUSSION
Experiments evaluate signal estimators and attribution methods on ImageNet using quantitative correlation, image degradation, and qualitative visualization. The proposed two-component estimator and PatternNet produce stronger signal recovery, sharper attributions, and improved visualizations than comparison methods.
- Experimental setup: The validation set contains 50000 previously unseen samples, with signal estimators and evaluation vectors trained separately from validation data.The vector for measuring signal-estimator quality was optimized on the second half of the training set; the official validation set was not used for either training process.
- Computational cost: Individual explanations are computationally cheap after learning because they use a back-propagation pass with a modified weight vector.The method is reported to produce explanations at least as fast as real-time saliency work and is not restricted to image models.
- Signal-estimator quality: Higher correlation-measure values indicate better signal estimators; optimized estimators remove more correlations than random, filter-based, and gradient baselines across layers.The gradient performs little better than the random estimator in higher layers, while the two-component estimator is best in dense layers.
- Image degradation: The two-component estimator Sa+− produces the fastest decrease in confidence under patch corruption by a large margin.Patches are ordered by attribution and replaced with their mean color-channel values; a steeper confidence decay indicates a better heat map.
- Qualitative evaluation: The two-component model captures the original input during signal estimation and produces a crisp attribution heat map, unlike noisier or denser alternatives.The trivial estimator includes distractors, while Sa struggles with color information and produces dense heat maps.
- ImageNet visualizations: PatternNet recovers a signal close to the original on randomly selected ImageNet images without adding rectifiers, while comparison methods fail to capture the true signal in correlation tests.The authors report that the proposed approach produces the crispest visualization and is measurably better than DeConvNet, Guided BackProp, LRP, and DTD.
6 CONCLUSION
The paper concludes that gradient directions need not represent data signal, so several popular explanation methods can fail even for linear models. It proposes data-distribution-aware objectives that improve signal visualizations and attribution decompositions for deep networks.
- 6 CONCLUSION: Model gradients can reflect relations between signal and distracting noise rather than estimate the signal itself.This conclusion is drawn from the linear-model analysis and motivates correcting explanation methods using data distribution information.
- 6 CONCLUSION: DeConvNet, Guided BackProp, and LRP do not provide theoretically correct explanations even for a simple linear model.The paper extends this reasoning to nonlinear models and frames linear reliability as a test for neural-network explanations.
- 6 CONCLUSION: PatternNet and PatternAttribution optimize a neuron-wise objective to correct signal visualizations and decomposition methods using the data distribution.PatternNet targets signal visualizations, while PatternAttribution targets decompositions.
- 6 CONCLUSION: The proposed methods constitute theoretical, qualitative, and quantitative improvements toward understanding deep neural networks.The conclusion presents this as the paper’s overall demonstrated outcome.