Source-linked AI summary
Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, Anshul Kundaje
TL;DR
Neural networks’ black-box reputation motivates feature-importance methods that remain informative when gradients vanish. The paper presents DeepLIFT, which compares activations with reference activations and reports contribution scores; it argues that DeepLIFT improves on gradient-based approaches in practice, while relying on assumptions such as including bias terms for certain conservation properties.
Problem
Gradient-based feature-importance methods can fail for ReLUs, sigmoid gates, and tanh gates when gradients are zero or near zero despite meaningful activations.
Method
DeepLIFT compares each neuron’s activation with its activation under a task-appropriate reference input and propagates contribution scores from these differences.
Results
In practice, DeepLIFT produces superior results to LRP when biases are often non-zero, while matching LRP when all reference activations are zero under stated conditions.
Takeaways & Limitations
DeepLIFT can handle recurrent memory units gated by sigmoid or tanh activations that have vanishing gradients.
Takeaways & Limitations
The stated conservation property requires including the bias term in the calculation.
Abstract
from arXiv · showhide
Note: This paper describes an older version of DeepLIFT. See https://arxiv.org/abs/1704.02685 for the newer version. Original abstract follows: The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Learning Important FeaTures), an efficient and effective method for computing importance scores in a neural network. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. We apply DeepLIFT to models trained on natural images and genomic data, and show significant advantages over gradient-based methods.
1. Introduction
Gradient-based feature importance can fail when nonlinear units have zero or near-zero gradients despite carrying meaningful information. DeepLIFT addresses this by comparing neuron activations with task-appropriate reference activations.
- Motivation: Improved interpretability can build user trust and support scientific discovery when neural networks are used in applications where interpretability is paramount.
- Motivation: Gradient-based methods can assign zero importance to inactive ReLUs even when those units carry information.Sigmoid and tanh gates can likewise have near-zero gradients at extreme inputs despite significant inputs.
- Motivation: Figure 1 illustrates the failure: with x1 = x2 = −1, the output is 0.1 while both input gradients are zero.The ReLU is inactive for the observed input but has activation 2 under the reference input.
- DeepLIFT: DeepLIFT assigns feature importance by comparing each neuron’s activation with its activation under a task-appropriate reference input.The activation difference can remain non-zero when the local gradient is zero.
2. DeepLIFT Method
DeepLIFT defines neuron-to-neuron contributions and requires them to satisfy conservation over minimally sufficient, non-redundant input sets. The resulting contributions sum to the target neuron’s difference from reference.
- Definitions: DeepLIFT denotes the contribution from neuron x to neuron y as Cxy and each neuron’s activation and reference activation as An and A0n.
- Summation to δ: For a minimally sufficient, non-redundant input set S, contributions from S to y sum to y’s difference-from-reference.No proper subset of S is sufficient to compute y.
2.2. Linear composition
DeepLIFT propagates contributions through a neuron’s outputs so that each neuron inherits contribution in proportion to its contribution to the outputs’ difference from reference.
- Linear composition: For output neurons Ox of x, DeepLIFT defines a linear composition property for propagating contributions through the network.
- Linear composition: Each neuron inherits contribution through its outputs in proportion to how much it contributes to the outputs’ difference-from-reference.
2.3. Backpropagation Rules
DeepLIFT computes backpropagation multipliers that propagate contribution scores backward while avoiding numerical instability. Rules cover general composition, max operations, maxout units, and nonlinearities.
- General rules: DeepLIFT uses GPU-implementable rules and multipliers to backpropagate contribution scores from later to preceding layers.Multipliers avoid explicitly computing contributions when a neuron’s difference-from-reference is small.
- General rules: For a target neuron t and outputs Ox, the backpropagation rule derives contributions from the linear composition property and multiplier definition.
- General rules: The backpropagation equations define Iy as the set of inputs of neuron y and use reference activations to compute difference-from-reference quantities.
- Max operation: For max operations, contributions select the input attaining the maximum activation, using an indicator function or an equivalent gradient.
- Maxout units: Maxout units are handled by partitioning the activation difference into regions where one affine function dominates, then weighting each region’s coefficient by its share.
- Element-wise nonlinearities: A shared multiplier for all inputs to y satisfies summation-to-delta and can be used for ReLU, PReLU, sigmoid, and tanh nonlinearities.Near-zero denominators can be handled with L’Hopital’s rule.
2.4. A note on final activation layers
Final activation functions can attenuate individual feature contributions when redundant inputs jointly produce a saturated output. Computing contributions to the pre-activation rather than the final activation avoids this attenuation.
- Using contributions to the pre-activation y rather than the sigmoid output t avoids redundancy-driven attenuation.For both one and two redundant inputs, the contribution C_x1y is 100.
2.5. A note on Softmax activation
For softmax outputs, a feature that contributes equally to every class has zero effective contribution. Mean-normalizing weights preserves the softmax output while enforcing this behavior in DeepLIFT scores.
- A feature contributes zero to a softmax output when it contributes equally to all softmax classes.Equal coefficients cancel after substituting the affine inputs into the softmax expression.
- To avoid attenuation for highly confident predictions, DeepLIFT should compute contributions to each pre-softmax activation rather than to the softmax output.
- Mean-normalizing weights leaves the softmax output unchanged and makes DeepLIFT scores zero for equal contributions across classes.
2.6. Weight normalization for constrained inputs
For constrained one-hot inputs, weight normalization preserves the neural-network output while making contribution estimates less sensitive to arbitrary weight offsets. Mean-normalizing filter weights also prevents systematic over- or underestimation across columns.
- For constrained inputs, subtracting the mean weight and compensating the bias preserves the neural-network output.The inputs satisfy a sum-to-one constraint, so the weight shift can be absorbed into the bias.
- Mean-normalizing convolutional filter weights prevents contributions from different columns from being systematically overestimated or underestimated.For affine functions, DeepLIFT multipliers equal the weights and are therefore sensitive to their mean offset.
3. Results
DeepLIFT was evaluated on image and genomic models using reference inputs, with comparisons against gradient-based importance methods. In the genomic experiment, DeepLIFT identified both required DNA patterns while gradient-based methods missed GATA.
- 3.1. Image results: VGG16 was trained on 64 × 64 Tiny Imagenet images spanning 200 output classes, using an all-zero preprocessed reference input.
- 3.2. Genomics: The genomic positive class required both GATA and CAGATG patterns in length-200 DNA sequences, whereas negative examples contained only one pattern.
- 3.1. Image results: DeepLIFT importance scores were compared with gradient-based methods for image models, including absolute gradients and positive gradient*input.
- 3.2. Genomics: DeepLIFT discovered both genomic patterns and assigned them large importance scores, while gradient-based methods missed the GATA pattern.
4. Discussion
The discussion shows that LRP can reduce to gradient*input under piecewise-linear conditions, while DeepLIFT differs when nonzero biases and reference activations matter. It also relates LRP filtering and pooling rules to gradient*input and addresses numerical stability.
- With zero reference activations, DeepLIFT and LRP give similar results, but nonzero biases explain why DeepLIFT performs better in Figures 2 and 3.DeepLIFT also avoids LRP’s numerical stability problems by computing contributions with multipliers.
- When all activations are piecewise linear and biases are included, LRP reduces to gradient*input.
- For unpooling, LRP’s backward signal and gradient*input agree because only the forward-recorded pooling location receives nonzero signal.
- As epsilon approaches zero, gradient*input gives the exact LRP result, eliminating the numerical-instability issue addressed by epsilon regularization.
- For sequential filtering, relevance is propagated from a neuron two layers later through the intermediate layer, with the denominator linked to the next-layer activation.
- When final-layer relevance differs from activation, gradient*input can instead be computed with respect to the linear term before the final nonlinearity.
- For ReLU nonlinearities, LRP and gradient*input converge: inactive units receive zero, while active units receive the output.
5. Author contributions
The authors divided responsibility for conceiving, implementing, applying, and preparing the DeepLIFT work across the listed contributors.
- AS and PG conceived DeepLIFT, while AS implemented the software.
- PG led the genomics application, and AYS led the Tiny Imagenet application.
- AK provided guidance and feedback, and AS, PG, AYS, and AK prepared the manuscript.