PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Important Features Through Propagating Activation Differences

Avanti Shrikumar, Peyton Greenside, Anshul Kundaje

arXiv:1704.02685v2cs.CVcs.LGcs.NE

TL;DR

Neural networks’ black-box reputation limits their use when interpretability is essential. DeepLIFT assigns input-importance scores by propagating differences from a reference state, outperforming other backpropagation-based methods on MNIST and reducing noise in genomic analyses.

  • Problem

    Neural networks’ black-box reputation creates a need for interpretable importance scores, while gradient-based methods can fail when gradients are zero or discontinuous.

  • Method

    DeepLIFT attributes output differences from a reference output to input differences from a reference input, optionally separating positive and negative contributions.

  • Results

    DeepLIFT-RevealCancel outperformed other backpropagation-based methods on MNIST, while a hybrid DeepLIFT variant reduced noise in genomic analyses.

  • Takeaways & Limitations

    Difference-from-reference propagation can transmit information when gradients are zero and reveal dependencies missed by other methods.

  • Takeaways & Limitations

    The Rescale rule can misleadingly assign all importance to one input in min operations, obscuring that both inputs are relevant.

Abstract

from arXiv · show

The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Deep Learning Important FeaTures), a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. By optionally giving separate consideration to positive and negative contributions, DeepLIFT can also reveal dependencies which are missed by other approaches. Scores can be computed efficiently in a single backward pass. We apply DeepLIFT to models trained on MNIST and simulated genomic data, and show significant advantages over gradient-based methods. Video tutorial: http://goo.gl/qKb7pL, ICML slides: bit.ly/deeplifticmlslides, ICML talk: https://vimeo.com/238275076, code: http://goo.gl/RM8jvH.

1. Introduction

DeepLIFT is introduced as an algorithm for assigning input-importance scores by comparing neuron activations with a reference state. It can separately track positive and negative contributions to reveal missed dependencies, while computing scores efficiently in one backward pass.

  • 1. Introduction: DeepLIFT assigns importance scores to a model’s inputs for a given output by comparing activations against a reference state.The method frames importance in terms of differences from a reference state.
  • 1. Introduction: Separate treatment of positive and negative contributions enables DeepLIFT to reveal dependencies missed by other approaches.This separation is applied at nonlinearities.
  • 1. Introduction: DeepLIFT scores are computed efficiently with a backpropagation-like algorithm in a single backward pass after prediction.The backward pass follows the model’s prediction.

2. Previous Work

Previous work assigns importance through input or neuron perturbations and through backpropagated signals, including gradients, LRP, integrated gradients, and Grad-CAM. These methods offer useful efficiency or saturation-handling properties but retain limitations that motivate DeepLIFT.

  • Perturbation-based approaches: Perturbation methods alter individual inputs or neurons and measure effects on later activations or outputs.Examples include image occlusion, in-silico mutagenesis of genomic sequences, and related perturbation approaches.
  • Backpropagation approaches: Backpropagation methods propagate importance from output to input in one pass, making them efficient.DeepLIFT is presented as one such approach.
  • Gradient-based approaches: Gradient × input is often preferable to gradients alone because it uses the input’s sign and strength, but it does not solve saturation or thresholding artifacts.LRP rules for ReLU networks were shown to be equivalent within a scaling factor to gradient × input absent numerical-stability modifications.
  • Gradient-based approaches: Integrated gradients address saturation and thresholding by integrating gradients from a starting value to the current input, but high-quality numerical integrals add computational overhead and can still mislead.The passage gives all-zeros inputs as one example of a starting value.
  • Gradient-based approaches: Grad-CAM produces coarse-grained class-specific importance maps from final-layer feature maps and gradients, with elementwise products proposed for finer-grained importance.It uses weighted feature-map activations to indicate important inputs.

3. The DeepLIFT Method … 3.3. Defining the Reference

DeepLIFT attributes output differences to input differences from a chosen reference, using finite-difference multipliers and a backpropagation-compatible chain rule. Its explanations depend critically on selecting an appropriate reference input, which is propagated through the network to define all reference activations.

  • 3.1. The DeepLIFT Philosophy: DeepLIFT explains the difference in a target output from a reference output by attributing it to differences between the input and a neutral reference input.The reference input is chosen according to what is appropriate for the problem.
  • 3.1. The DeepLIFT Philosophy: The summation-to-delta property assigns each input contribution as the portion of the target’s difference-from-reference attributed to that input’s difference-from-reference.When transfer functions are well-behaved, local linearity provides additional motivation for this property.
  • 3.1. The DeepLIFT Philosophy: DeepLIFT can assign non-zero contributions when the corresponding gradient is zero, addressing meaningful signals and discontinuous importance changes missed by gradients.The method uses difference-from-reference values rather than infinitesimal gradient behavior.
  • 3.2.1. DEFINITION OF MULTIPLIERS: A multiplier measures the contribution of an input difference to a target difference divided by that input difference.Unlike a partial derivative, it is defined over finite rather than infinitesimal differences.
  • 3.2.2. THE CHAIN RULE FOR MULTIPLIERS: The chain rule for multipliers combines successive-neuron multipliers, allowing target-specific multipliers to be computed efficiently by backpropagation.This parallels backpropagation for gradients obtained through the chain rule for partial derivatives.
  • 3.3. Defining the Reference: DeepLIFT defines each neuron’s reference activation by propagating activations through the network from a selected reference input.The reference of a neuron is assumed to be its activation on that reference input.
  • 3.3. Defining the Reference: Reference selection is critical and domain-dependent; multiple references may sometimes be needed to obtain insightful DeepLIFT results.The guiding question is what differences one wants to measure against.
  • 3.3. Defining the Reference: Gradient×input implicitly uses an all-zeros reference, while integrated gradients requires a conceptually similar starting point; pure gradients and Guided Backprop do not use one.The paper argues that lacking a reference is a limitation of the latter methods.

3.4. Separating Positive and Negative Contributions · 3.5. Rules for Assigning Contribution Scores

DeepLIFT separates positive and negative activation differences when cancellation can distort contribution assignments, while defining neuron-to-input rules that support backpropagation to arbitrary target outputs. The Linear, Rescale, and RevealCancel rules provide distinct treatments for linear, single-input nonlinear, and cancellation-sensitive operations, with tradeoffs between faithful relevance and noise suppression.

  • 3.4. Separating Positive and Negative Contributions: DeepLIFT represents each neuron’s activation difference as positive and negative components, Δy+ and Δy−, because some contribution rules require treating them separately.Under only the Linear or Rescale rules, the positive, negative, and total multipliers coincide; RevealCancel can make them differ.
  • 3.5.1. THE LINEAR RULE: The Linear rule applies to Dense and Convolutional layers without nonlinearities by decomposing activation differences into positive and negative input terms.When an input difference is zero but its positive and negative components cancel, the rule sets the corresponding multiplier to 0.5wi to propagate importance to those components.
  • 3.5.2. THE RESCALE RULE: The Rescale rule handles single-input nonlinear transformations such as ReLU, tanh, and sigmoid by assigning the input the full output difference, CΔxΔy = Δy.As the input approaches its reference, its multiplier approaches the derivative, allowing gradients to replace multipliers near the reference for numerical stability.
  • 3.5.2. THE RESCALE RULE: The Rescale rule addresses saturation and thresholding cases where gradients are zero, allowing difference-from-reference information to flow without assigning importance to bias terms.In the thresholding example, it assigns the input contribution ϵ, whereas gradient×input assigns 10+ϵ to x and −10 to the bias term.
  • 3.5.3. AN IMPROVED APPROXIMATION OF THE SHAPELY VALUES: THE REVEALCANCEL RULE: For min(i1, i2), the Rescale rule can assign all importance to whichever input is smaller, obscuring that both inputs are relevant.This failure motivates an alternative rule for interactions involving cancellation.
  • 3.5.3. AN IMPROVED APPROXIMATION OF THE SHAPELY VALUES: THE REVEALCANCEL RULE: RevealCancel separately evaluates positive terms without negative terms and negative terms without positive terms, reducing distortions caused by cancellation.For min(i1, i2), it assigns 0.5 min(i1, i2) to both inputs.
  • 3.5.3. AN IMPROVED APPROXIMATION OF THE SHAPELY VALUES: THE REVEALCANCEL RULE: Rescale may be preferable to RevealCancel for thresholded ReLUs when sub-threshold activation differences represent noise, because it assigns zero contributions to both positive and negative components.RevealCancel can assign nonzero contributions by evaluating each component in the absence of the other.

3.6. Choice of Target Layer

For sigmoid or softmax outputs, DeepLIFT may target the preceding linear layer rather than the final nonlinearity to avoid attenuation from the summation-to-delta property. In a sigmoid example, saturated outputs can produce misleadingly lower scores for stronger logit contributions, so contributions are computed to the logit instead.

  • 3.6. Choice of Target Layer: DeepLIFT may compute contributions to the linear layer preceding a softmax or sigmoid rather than to the final nonlinearity.This avoids attenuation caused by the summation-to-delta property.
  • 3.6. Choice of Target Layer: For a sigmoid o = σ(y), with y = x1 + x2, saturation can make contribution scores misleading across different inputs.The example compares inputs with x1 = 50, x2 = 0 and x1 = 100, x2 = 100, while the output remains very close to 1.
  • 3.6. Choice of Target Layer: When x1 = 50 and x2 = 0, the saturated sigmoid assigns contributions of 0.5 and 0, respectively.The output o is very close to 1.
  • 3.6. Choice of Target Layer: When x1 = 100 and x2 = 100, the same near-1 output assigns contributions of 0.25 to each input.A stronger contribution to the logit therefore does not always translate into a higher DeepLIFT score.

Adjustments for Softmax Layers

For softmax models, DeepLIFT normalizes contributions computed at the preceding linear layer by subtracting their mean across classes. This preserves the softmax output because subtracting the same value from every softmax input leaves it unchanged.

  • Adjustments for Softmax Layers: Contributions to the linear layer preceding softmax are normalized by subtracting the mean contribution across all classes.This addresses the mismatch between class-wide softmax normalization and the unnormalized preceding linear layer.
  • Adjustments for Softmax Layers: The normalized contribution is defined from the unnormalized contribution to each class in the preceding linear layer.The passage introduces the normalized contribution using the unnormalized class-specific contribution as its basis.
  • Adjustments for Softmax Layers: Subtracting a fixed value from every softmax input leaves the softmax output unchanged.This invariance justifies subtracting the same mean contribution from all classes.

4. Results

DeepLIFT achieved strong MNIST classification performance and, with RevealCancel, outperformed other backpropagation-based methods in identifying pixels that convert one digit to another. On simulated DNA tasks, RevealCancel-based DeepLIFT better captured motif relevance and cooperativity than competing importance-scoring methods.

  • MNIST: 99.2% test-set accuracy was achieved by the convolutional neural network trained for MNIST digit classification.The architecture used two convolutional layers, a fully connected layer, and a softmax output layer.
  • MNIST: The MNIST evaluation erased up to 157 pixels ranked by the difference between original-class and target-class scores, then measured the change in class log-odds.Pixels were erased only when their score difference was positive.
  • MNIST: DeepLIFT with RevealCancel outperformed other backpropagation-based methods on the MNIST pixel-erasure task.Integrated gradients with 5 or 10 intervals performed comparably to gradient×input on MNIST.
  • Simulated DNA: Guided Backprop×input assigned positive importance to TAL1 on task 1, failed to identify task-0 cooperativity, and showed poor differentiation at high motif log-odds scores.These failures corresponded to properties evaluated in the simulated DNA experiment.
  • Simulated DNA: DeepLIFT-fc-RC-conv-RS reduced noise relative to pure RevealCancel and corrected missed TAL1 relevance on task 0.Gradient×input, integrated gradients, and DeepLIFT-Rescale occasionally missed TAL1 relevance, while RevealCancel at the fully connected layer corrected it.

5. Conclusion

DeepLIFT computes importance scores by explaining output differences relative to a reference through input differences from corresponding reference inputs. This difference-from-reference formulation can propagate information even when gradients are zero, potentially benefiting recurrent networks with saturating activations.

  • 5. Conclusion: DeepLIFT computes importance scores by explaining the difference between an output and a reference output through differences between inputs and reference inputs.The method is presented as a novel approach based on reference comparisons.
  • 5. Conclusion: Difference-from-reference propagation allows information to pass through the network even when the gradient is zero.The passage identifies this behavior as illustrated in Fig. 1.
  • 5. Conclusion: This capability could be especially useful for recurrent neural networks using saturating sigmoid or tanh activations.The passage specifically highlights sigmoid and tanh as popular saturating activations in recurrent networks.

6. Appendix

The appendix is available as a downloadable supplementary PDF.

  • 6. Appendix: The appendix can be downloaded as supplementary material.The paper provides a direct URL to the appendix PDF.

8. Funding

The authors report funding from fellowships and NIH grants supporting AS, PG, and AK.

  • 8. Funding: AS received support from a Howard Hughes Medical Institute International Student Research Fellowship and a Bio-X Bowes Fellowship.These fellowships supported AS.
  • 8. Funding: PG was supported by a Bio-X Stanford Interdisciplinary Graduate Fellowship.
  • 8. Funding: AK was supported by NIH grants DP2-GM-123485 and 1R01ES025009-02.

9. Author Contributions

AS and PG conceptualized DeepLIFT, while AS implemented it and led the MNIST experiments. AS and PG conducted genomic-data experiments, AK provided guidance and feedback, and all three wrote the manuscript.

  • AS and PG conceptualized DeepLIFT.
  • AS implemented DeepLIFT and ran experiments on MNIST, while AS and PG ran experiments on genomic data.
  • AK provided guidance and feedback, and AS, PG, and AK wrote the manuscript.
Loading 1704.02685v2…