Source-linked AI summary
Interpreting the Predictions of Complex ML Models by Layer-wise Relevance Propagation
Wojciech Samek, Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Klaus-Robert Müller
TL;DR
Deep neural networks can perform strongly yet offer little insight into which inputs support a prediction. This paper presents Layer-wise Relevance Propagation (LRP), which decomposes predictions into input relevance and propagates it through network layers. Its image-classification examples distinguish pixel contribution from sensitivity and reveal context use and architecture differences.
Problem
Deep neural networks perform well on practical tasks yet predict as black boxes, making it difficult to identify which input aspects support a decision.
Source: §1 · p. 1Method
LRP decomposes a classifier’s decision into input relevance and propagates relevance through layers; its decomposition accounts for both classifier response and feature presence.
Source: §1 · p. 2, §2 · pp. 2–3Results
LRP explanations distinguish pixel contribution from sensitivity and reveal context use and architecture-specific focus in image-classification examples.
Source: §3 · pp. 3–4Takeaways & Limitations
LRP can support model comparison, validation, and visualization, but its redistribution rules should fit the model and input domain.
Source: §4 · p. 4No single LRP rule works equally well for all models and datasets, so redistribution choices must account for layer mappings and input domains.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Complex nonlinear models such as deep neural network (DNNs) have become an important tool for image classification, speech recognition, natural language processing, and many other fields of application. These models however lack transparency due to their complex nonlinear structure and to the complex data distributions to which they typically apply. As a result, it is difficult to fully characterize what makes these models reach a particular decision for a given input. This lack of transparency can be a drawback, especially in the context of sensitive applications such as medical analysis or security. In this short paper, we summarize a recent technique introduced by Bach et al. [1] that explains predictions by decomposing the classification decision of DNN models in terms of input variables.
1 Explaining Predictions by Decomposition
Complex models can predict effectively while revealing little about which input features support a decision. Decomposition assigns relevance to input variables so their scores sum to the prediction, explaining the whole output rather than local sensitivity to perturbations.
1 Explaining Predictions by Decomposition
- Decomposition assigns pixel relevance scores whose sum equals the model’s prediction for the input.The scores indicate each pixel’s relevance to that prediction.Source: §1 · pp. 1–2
- Unlike sensitivity analysis, which measures local changes under pixel perturbations, decomposition explains what makes the model reach its output rather than zero.For a scooter image, sensitivity highlights pixels whose modification changes the class score; decomposition attributes support to scooter-like structures.Source: §1 · p. 2
- For a linear classifier, decomposition uses both pixel activation and its weight, whereas sensitivity scores need not account for the activation.The text gives Rp = xpwp for decomposition and Rp = wp^2 (or |wp|) for sensitivity analysis.Source: §1 · p. 2
2 Layer-wise Relevance Propagation
Layer-wise relevance propagation (LRP) redistributes a prediction backward through a network to produce input-level relevance. The section describes redistribution rules, their conservation condition, and a connection between some rules and deep Taylor decomposition.
2 Layer-wise Relevance Propagation
- LRP builds local redistribution rules for neurons and applies them in a backward pass to produce pixel-wise relevance.The method has also been applied to models and tasks beyond image classification, including text, facial features, and EEG.Source: §2 · p. 2
- The alpha-beta rule redistributes relevance using positive and negative contributions; choosing α − β = 1 enforces layer-wise conservation.The α = 2, β = 1 setting is reported to produce sharp heatmaps and allow negative relevance scores; α = 1, β = 0 connects to the z+-rule and Excitation Backprop redistribution.Source: §2 · p. 2
- For ReLU networks, some redistribution rules result from deep Taylor decomposition of the network function [14].The account treats a neuron’s relevance as its ReLU activation times a positive constant, then uses a first-order Taylor expansion around a reference point.Source: §2 · p. 3
- Pooling relevance messages from upper-layer neurons recovers the propagation rule, and an approximately constant factor supports decomposition at the preceding layer.The factor’s dependence on the lower-layer activations is described as weak, so the product structure holds approximately there.Source: §2 · p. 3
3 LRP in Practice
The paper demonstrates LRP in practice by comparing explanations with sensitivity analysis, measuring context relevance, and examining differences between two DNN architectures.
3 LRP in Practice
- On ILSVRC2012 images, LRP explanations outperform sensitivity analysis in the perturbation-based comparison and are qualitatively less noisy.Sensitivity highlights empty street regions in the scooter example, while LRP points to scooter-like structures; a faster score drop under perturbation indicates a more meaningful relevance ranking.Source: §3 · p. 3
- For PASCAL VOC 2007, relevance lies mostly inside airplane and sheep bounding boxes, while context matters more for indoor classes such as chair and sofa.Aggregating pixel scores inside and outside bounding boxes provides an outside-inside relevance ratio for the 20 categories.Source: §3 · p. 4
- Across many animal images, GoogleNet focuses more on the face and produces sparser heatmaps than BVLC CaffeNet.The analysis suggests GoogleNet focuses better on relevant image information, often the animal face rather than body shape or fur.Source: §3 · p. 4
4 Conclusion
LRP is a general framework for interpreting predictions from complex machine-learning systems, including deep neural networks. Its uses and rules depend on the models and datasets involved.
4 Conclusion
- LRP can support model comparison, validation, and visualization of complex-ML predictions.The framework has been applied to various tasks and machine-learning models.Source: §4 · p. 4
- Some redistribution rules for DNNs can be viewed as deep Taylor decompositions of the network function.Source: §4 · p. 4
- The alpha-beta LRP rule was shown to work well for DNNs with ReLU activations.Source: §4 · p. 4
- No single LRP rule serves all models and datasets equally well, because layer mappings and input domains differ.Source: §4 · p. 4