Source-linked AI summary
Methods for Interpreting and Understanding Deep Neural Networks
Grégoire Montavon, Wojciech Samek, Klaus-Robert Müller
TL;DR
High-accuracy neural networks still require interpretation to verify that they use proper features rather than data artifacts. The tutorial presents prototype-based concept interpretation and feature-based explanations, connecting these methods to practical transparency and validation. It also identifies model- and setting-specific limitations that affect reliable use.
Problem
High predictive accuracy does not by itself verify that a neural network relies on proper problem representations rather than data artifacts, especially in high-stakes applications.
Method
The tutorial surveys conceptual techniques for interpreting learned concepts with prototypes and explaining decisions by identifying relevant input variables.
Results
The tutorial connects these interpretation directions to practical successes and describes benefits of data-linked prototypes, sensitivity-versus-decomposition distinctions, and graph-propagation explanations.
Takeaways & Limitations
Interpretability is presented as a complementary technological direction that can benefit new neural-network models and learning algorithms.
Takeaways & Limitations
Direct heatmaps from convolutional networks are not strictly equivalent to sliding-window explanations and can be unreliable with border-padded convolutions.
Abstract
from arXiv · showhide
This paper provides an entry point to the problem of interpreting a deep neural network model and explaining its predictions. It is based on a tutorial given at ICASSP 2017. It introduces some recently proposed techniques of interpretation, along with theory, tricks and recommendations, to make most efficient use of these techniques on real data. It also discusses a number of practical applications.
1. Introduction
Deep neural networks achieve very high predictive accuracy, but interpretation is needed to verify that they rely on proper representations rather than data artifacts. This tutorial surveys conceptual techniques for interpreting and explaining DNNs across applications.
- High predictive accuracy makes deep neural networks useful across image, speech, and language applications.
- Interpretation helps validate whether accuracy reflects proper problem representations rather than exploited artifacts in the data.
- Interpretability is especially important when applications require assurance that models rely on correct features, including medicine and self-driving cars.
- Interpretation techniques are also used with deep nonlinear models to extract insights from complex physical, chemical, and biological systems.
- The tutorial covers interpreting modeled concepts and explaining individual model decisions, while abstracting from specific network structures and application domains.
2. Preliminaries
The tutorial distinguishes post-hoc functional understanding from mechanistic analysis and defines interpretation and explanation as different mappings between model concepts, interpretable domains, and decision-relevant features.
- The tutorial focuses on post-hoc interpretability: understanding predictions of a trained model in terms of readily interpretable input variables.
- Understanding means characterizing black-box behavior functionally, without elucidating internal workings or representations.
- An interpretation maps an abstract concept, such as a predicted class, into a domain that humans can make sense of.
- Interpretable domains include images and texts, whereas abstract vector spaces and undocumented input features are not directly interpretable.
- An explanation is the collection of interpretable-domain features that contributed to a particular classification or regression decision.
- Explanations may highlight image pixels or regions, color components, or text spans at different levels of granularity.
3. Interpreting a DNN Model
The tutorial interprets learned DNN concepts by constructing input prototypes, then refines activation maximization with data or generative models and extends analysis toward local, feature-based questions.
- Global concept interpretation: A learned concept represented by a top-layer neuron can be interpreted by constructing an interpretable and representative prototype in the input domain.
- Global concept interpretation: Activation maximization searches for an input pattern that maximizes a model response, such as a class probability.
- Global concept interpretation: Standard activation maximization often produces unnatural gray images with sparse edge and color patterns despite strong class responses.
- Global concept interpretation: An expert data-density model encourages prototypes to both produce strong class responses and resemble the data, approximating class-conditioned density.
- Global concept interpretation: On MNIST, all prototype variants receive full-certainty classification, but only expert- or decoder-based methods produce sharp, realistic-looking prototypes.
- Global concept interpretation: When concepts are multimodal, multiple local prototypes may represent a class better than one global prototype.
- From global to local analysis: Increasing localization forces the prototype toward the reference point and reduces its information content, motivating feature-based local analysis.
4. Explaining DNN Decisions
The section explains DNN decisions by assigning feature-level relevance scores to inputs, then covers sensitivity, Taylor decomposition, and graph-based relevance propagation. These methods support quantitative and visual analyses of which input features contribute to predictions.
- Core setup: Feature-level explanations assign each input feature a relevance score for explaining an output neuron’s prediction.For images, relevance can be mapped back to pixels and displayed as a heatmap.
- Sensitivity analysis: Sensitivity analysis uses locally evaluated gradients to identify features to which the model output is most sensitive.The gradient can be computed by backpropagation.
- Sensitivity analysis: Sensitivity analysis explains local function variation rather than the function value itself, distinguishing what changes a prediction from what makes an input represent a concept.For car detection, it addresses what makes an image more or less like a car rather than what makes it a car.
- Taylor decomposition: Taylor decomposition explains a decision by decomposing the function value into relevance scores from first-order terms around a root point.Higher-order terms are typically non-zero, so the resulting explanation is generally only partial.
- Graph-based propagation: Graph-based propagation redistributes the output prediction score backward through the network while enforcing local and global relevance conservation.Relevance is passed between successive layers according to neuron contributions, then accumulated at lower-layer neurons.
- Practical representations: Decomposition methods quantify feature contributions and can pool or filter relevance to produce histograms, heatmaps, regional scores, or isolated feature-map components.For RGB images, channel relevance can be summed per pixel; relevance may also be pooled over an object bounding box and the remaining image.
5. The LRP Explanation Framework
LRP explains deep neural network predictions by propagating relevance scores through successive layers, with rules that conserve relevance locally and can be connected to deep Taylor decomposition. The framework provides parameter choices, layer-specific rules, and practical guidance for producing stable, interpretable explanations.
- Propagation framework: The αβ-rule separates excitatory and inhibitory contributions, with α − β = 1 and β ≥ 0, while stabilizing terms can prevent division by zero.The αβ parameters control how positive and negative contributions are redistributed.
- Propagation framework: LRP-α2β1 worked well on BVLC CaffeNet, whereas LRP-α1β0 was more stable for the deeper GoogleNet; nonzero β also sparsified image explanations.Different α, β settings modulate the qualitative behavior of explanations.
- LRP and deep Taylor decomposition: For deep ReLU networks, LRP-α1β0 corresponds to Taylor decomposition at each layer, and iterating this process to the input layer yields deep Taylor decomposition.The connection relies on representing relevance as a positive, approximately constant coefficient times neuron activation and repeating the decomposition across layers.
- LRP and deep Taylor decomposition: The relevance-neuron construction searches for a root point on a specified segment, whose first-order Taylor terms produce lower-layer relevance contributions and recover the LRP-α1β0 rule.The root search also guarantees membership in the ReLU domain and positivity of relevance scores.
- Practical layer rules: Deep Taylor LRP supplies rules for hidden, input, pooling, and normalization layers, with activation-proportional pooling redistribution ensuring explanation continuity.The tutorial recommends trying the Table 2 rules as a default because they are described as stable and theoretically understood.
6. Recommendations and Tricks for LRP
The tutorial recommends model architectures and propagation rules that make LRP explanations selective, stable, and theoretically grounded, with implementation strategies for images of varying size. It also describes practical refinements for noisy heatmaps and cluttered scenes.
- Model choice: LRP is most thoroughly studied on convolutional ReLU networks, so the tutorial recommends applying it to model classes with prior successful use.When no suitable trained model exists, the authors suggest training one.
- Model choice: Keep fully connected layers few because LRP can redistribute relevance across too many lower-layer neurons and lose selectivity.The tutorial suggests convolutional ReLU networks with dropout in the fully connected layers.
- Propagation rules: Prefer sum- or average-pooling and zero or negative biases to better match deep Taylor LRP and disentangle relevance.Negative biases further sparsify activations, which helps separate relevance at each layer.
- Propagation rules: Use deep Taylor LRP rules as the default, switching from LRP-α1β0 to LRP-α2β1 when negative relevance is needed or heatmaps are too diffuse.If these rules remain unsatisfactory, pixel-flipping can select among a larger set of propagation rules.
- Heatmap refinement: Translated-image recombination can mitigate unattractive LRP heatmaps caused by noisy first-layer filters or large first-layer strides.Heatmaps from multiple slightly translated versions are recombined into an improved explanation.
- Large-image explanations: For large images, sliding windows explain patches across all locations and combine their relevance so pixels receive contributions from every relevant patch.The method is illustrated for CIFAR-10 horse classification in Figure 5.
7. Quantifying Explanation Quality
The tutorial evaluates explanation quality through continuity and selectivity: similar inputs should yield similar explanations, while removing highly relevant features should rapidly reduce the model output. Deep Taylor LRP is reported to perform better on both illustrative comparisons.
- Explanation continuity: Explanation continuity requires nearly equivalent inputs to produce nearly equivalent explanations, assuming a continuous prediction function.The strongest explanation variation can be quantified over the input domain.
- Explanation continuity: For deep ReLU networks, sensitivity analysis and simple Taylor decomposition have sharp explanation discontinuities, whereas deep Taylor LRP produces continuous explanations.For f(x) = max(x1, x2), deep Taylor LRP produces a smooth transition where the other methods are discontinuous.
- Explanation continuity: On MNIST, only deep Taylor LRP produces reasonably continuous explanations despite the relatively low variation of the underlying function.Figure 7 tracks relevance pooled into four quadrants as a handwritten digit moves across the receptive field.
- Explanation selectivity: Selectivity is measured by removing features in descending relevance order and recording the model output, with a low area under the curve indicating that relevant features were identified.The procedure was introduced for images as pixel-flipping and adapted to text by zeroing selected word embeddings.
- Explanation selectivity: In the illustrated MNIST case, deep Taylor LRP is more selective than sensitivity analysis and simple Taylor decomposition.At each iteration, a 4 × 4 patch with the highest relevance is set to black before recording the function score.
- Explanation selectivity: Selectivity results depend on the feature-removal process, which should preserve as much of the image’s data-manifold structure as possible.The removal strategy is therefore a practical source of variation in the analysis.
8. Applications
Explanation techniques support model validation by revealing what features models use and support scientific analysis by exposing relevant structure in complex data. The tutorial illustrates applications across text, images, EEG, genomes, faces, and molecular simulations.
- Model validation: Explanation-based validation complements held-out error by allowing human inspection of the features that an apparently accurate model uses.Validation error is only a proxy for true error because data distributions and labeling processes may differ.
- Model validation: On 20-Newsgroup classification, a CNN and SVM had similar test error but relied on different evidence: keywords for the CNN and word-count regularities for the SVM.The explanations compare word2vec-based CNN inputs with bag-of-words SVM representations.
- Model validation: For PASCAL VOC horse images, the DNN used the horse contour while the Fisher vector classifier relied mostly on a copyright tag despite similar classification accuracy.Removing the tag would consequently reduce the Fisher vector model’s measured accuracy while leaving the DNN predictions unaffected.
- Scientific data: Explanation techniques are also applied to molecular electronic properties and facial images, including pixel-wise explanations for age and gender attributes.Figure 10 summarizes molecular response maps, EEG heatmaps, gene-sequence relevance, and facial-appearance analysis.
- Scientific data: In EEG analysis, LRP decomposes movement predictions in the spatiotemporal electrode domain and pools relevance over time to produce EEG heatmaps.The application addresses variation in EEG patterns across users, environments, and acquisition calibration.
- Scientific data: For genomic sequences, convolutional networks map DNA to protein-binding sites and explanation methods identify nucleotides relevant to predicted binding-site presence.Heatmaps show relevant genes for different sequences and prediction outcomes.
9. Conclusion
The tutorial presents interpretability as a way to improve machine-learning transparency for both model validation and extracting domain knowledge. It focuses on post-hoc, concept-level and decision-level techniques that apply broadly across nonlinear models.
- Transparent machine-learning systems support both extracting novel domain knowledge and validating models.
- The tutorial covers prototypes for interpreting learned concepts and relevant input variables for explaining model decisions.
- Linking prototypes to data through density functions or generative models connects abstract concepts to the data domain.
- Graph propagation techniques are presented as more robust for explanations than standard analysis techniques.
- The techniques target general nonlinear models, including fully trained downloadable models, without requiring control over their structure or training.
- Interpretability is framed as an orthogonal research strand that can complement advances in neural-network models and learning algorithms.