PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards better understanding of gradient-based attribution methods for Deep Neural Networks

Marco Ancona, Enea Ceolini, Cengiz Öztireli, Markus Gross

arXiv:1711.06104v4cs.LGstat.ML

TL;DR

DNN predictions remain challenging to explain because black-box models lack tools for inspecting their behavior, reducing trustworthiness in interpretability-critical domains. The paper analyzes four gradient-based methods, reformulates ϵ-LRP and DeepLIFT (Rescale) as modified-gradient backpropagation, and introduces Sensitivity-n for evaluation. The four analyzed methods are strongly related, with proven equivalence or approximation conditions, and empirical results support the theoretical findings.

  • Problem

    DNN predictions remain challenging to explain because black-box models lack tools for inspecting their behavior, reducing trustworthiness in interpretability-critical domains.

  • Method

    The paper analyzes four gradient-based methods, reformulates ϵ-LRP and DeepLIFT (Rescale) as modified-gradient backpropagation, and introduces Sensitivity-n for evaluation.

  • Results

    The four analyzed methods are strongly related, with proven equivalence or approximation conditions, and empirical results support the theoretical findings.

  • Takeaways & Limitations

    Sensitivity-n helps reveal attribution-method properties and suggests directions for developing more general attribution methods.

  • Takeaways & Limitations

    ϵ-LRP can produce unmeaningful, concentrated attributions for nonlinearities such as Sigmoid or Softplus, while DeepLIFT can fail on RNNs with multiplicative interactions.

Abstract

from arXiv · show

Understanding the flow of information in Deep Neural Networks (DNNs) is a challenging problem that has gain increasing attention over the last few years. While several methods have been proposed to explain network predictions, there have been only a few attempts to compare them from a theoretical perspective. What is more, no exhaustive empirical comparison has been performed in the past. In this work, we analyze four gradient-based attribution methods and formally prove conditions of equivalence and approximation between them. By reformulating two of these methods, we construct a unified framework which enables a direct comparison, as well as an easier implementation. Finally, we propose a novel evaluation metric, called Sensitivity-n and test the gradient-based attribution methods alongside with a simple perturbation-based attribution method on several datasets in the domains of image and text classification, using various network architectures.

1 INTRODUCTION AND MOTIVATION

The paper addresses the challenge of explaining DNN predictions by studying attribution methods and the lack of comprehensive theoretical and empirical comparison. It introduces a unified gradient-based framework and Sensitivity-n for comparing methods across datasets and architectures.

  • DNN interpretability remains challenging, limiting trust in applications where reliability is crucial.
  • Existing attribution methods lack a comprehensive comparison because of differing formulations, architecture compatibility, and the absence of a common benchmark.
  • ϵ-LRP and DeepLIFT (Rescale) can be reformulated as backpropagation with modified gradients, yielding a unified framework for gradient-based methods.
  • The reformulation also enables easier implementation with modern graph computational libraries.
  • Sensitivity-n generalizes Completeness and Summation to Delta and is used to compare methods on widely adopted datasets and architectures.

2 OVERVIEW OVER EXISTING ATTRIBUTION METHODS

The paper contrasts perturbation-based and backpropagation-based attribution methods, emphasizing their different computational costs and interpretive behavior. It reviews Gradient * Input, Integrated Gradients, LRP, DeepLIFT, and related methods alongside occlusion-based attribution.

  • 2.1 PERTURBATION-BASED METHODS: Perturbation methods alter or remove features, rerun the network, and measure output differences, directly estimating marginal feature effects.
  • 2.1 PERTURBATION-BASED METHODS: Occlusion-1 replaces each feature with a zero baseline and measures Sc(x) − Sc(x[xi=0]) as a perturbation benchmark.
  • 2.2 BACKPROPAGATION-BASED METHODS: Backpropagation methods compute all feature attributions through network passes and are generally faster, but their outputs are less directly related to output variation.
  • 2.2 BACKPROPAGATION-BASED METHODS: Gradient * Input multiplies signed input gradients by the input, whereas Integrated Gradients averages gradients along a baseline-to-input path.
  • 2.2 BACKPROPAGATION-BASED METHODS: LRP redistributes the target prediction score backward through the network, while DeepLIFT compares activations at the input and a reference baseline using relevance propagation.
  • 2.2 BACKPROPAGATION-BASED METHODS: Gradient-based attributions on Inception V3 show higher local variance than perturbation-based attributions, while absolute-gradient saliency cannot represent positive and negative evidence.

3 A UNIFIED FRAMEWORK

The unified framework expresses gradient-based attribution methods through modified chain-rule derivatives, enabling theoretical comparison, equivalence results, and implementation with standard graph backpropagation.

  • 3 A UNIFIED FRAMEWORK: The modified chain rule sums products of weights and nonlinear response functions along paths, and applies to fully connected, convolutional, recurrent, and pooling architectures without multiplicative units.Setting g=f′ recovers the ordinary partial derivative of the target output with respect to an input unit.
  • 3 A UNIFIED FRAMEWORK: ϵ-LRP and DeepLIFT (Rescale) become feature-wise products of input differences and modified partial derivatives, placing them in a common gradient-based formulation.For ϵ-LRP, the multiplier is x and g_LRP(z)=f(z)/z; for DeepLIFT, it is x−x̄ and g_DL uses output differences divided by input differences.
  • 3 A UNIFIED FRAMEWORK: The reformulation permits implementation of all considered methods by overriding graph nonlinearities and using standard backpropagation, avoiding custom layer implementations.The paper illustrates this approach with TensorFlow gradient registration for ϵ-LRP.
  • 3.1 INVESTIGATING FURTHER CONNECTIONS: ϵ-LRP equals Gradient * Input for ReLU networks and equals zero-baseline DeepLIFT when biases are absent and nonlinearities satisfy f(0)=0.These conditions include ReLU and Tanh as examples of nonlinearities crossing the origin.
  • 3.1 INVESTIGATING FURTHER CONNECTIONS: ϵ-LRP’s average-gradient interpretation holds for ReLU and Tanh but can fail for Sigmoid or Softplus, where small inputs may produce concentrated, less meaningful attributions.The paper attributes this behavior to g_LRP becoming extremely large near zero, with its upper bound limited by the stabilizer.
  • 3.1 INVESTIGATING FURTHER CONNECTIONS: DeepLIFT approximates Integrated Gradients in a single step and is often close on varied tasks, but diverges for recurrent networks with multiplicative interactions and can violate Completeness.The paper distinguishes this approximation from general equivalence because average gradients do not generally obey the chain rule.

4 EVALUATING ATTRIBUTIONS

The evaluation frames attribution as a trade-off between explaining individual features and capturing multi-feature effects, using Sensitivity-n across tasks and architectures. Results show method behavior depends on model linearity, nonlinearities, interactions, and the desired scope of explanation.

  • 4.1 MEASURING SENSITIVITY: No listed attribution method satisfies Sensitivity-n for every n in nonlinear models, because scalar feature scores cannot fully represent nonlinear interactions.All methods satisfy Sensitivity-n across all n only for linear models or models behaving linearly for the selected task.
  • 4.1 MEASURING SENSITIVITY: The evaluation uses random subsets to estimate Pearson correlation between summed attributions and target-output variation across n, averaging results over one thousand samples per dataset.Tests cover MNIST with multiple architectures and activations, CIFAR10, ImageNet samples, and a simple sentiment-analysis model.
  • 4.1 MEASURING SENSITIVITY: On complex Inception V3 models, gradient-based methods have low accuracy in predicting attribution signs, producing heatmaps affected by high-frequency noise.The experiments also indicate that inputs can contain negative evidence, which signed attributions can report.
  • 4.1 MEASURING SENSITIVITY: Integrated Gradients, DeepLIFT, and ϵ-LRP are preferable for capturing global nonlinear effects and feature interactions, whereas Occlusion-1 better identifies a few individually important features.Occlusion-1 satisfies Sensitivity-1, but its correlation decreases as n increases; it is also much slower than gradient-based methods.
  • 4.1 MEASURING SENSITIVITY: In MNIST-MLP with Tanh, Gradient * Input approximates Occlusion-1 better than the other gradient-based methods.This suggests that its instantaneous gradient is feature-wise close to the average gradient for these models.
  • 4.1 MEASURING SENSITIVITY: Integrated Gradients and DeepLIFT show very high correlation in practice, making DeepLIFT a faster approximation of Integrated Gradients except with multiplicative feature interactions.The analyzed DeepLIFT formulation should be avoided in cases such as IMDB-LSTM, where multiplicative interactions are present.

5 CONCLUSIONS

The paper finds that four attribution methods are strongly related despite different formulations, and it combines theoretical equivalence results with a unified implementation perspective and the Sensitivity-n metric.

  • 5 CONCLUSIONS: Gradient * Input, ϵ-LRP, Integrated Gradients, and DeepLIFT are theoretically and practically strongly related, with proven conditions of equivalence or approximation.The reformulation of ϵ-LRP and DeepLIFT makes their implementation as easy as other gradient-based methods.
  • 5 CONCLUSIONS: Sensitivity-n helps expose properties of existing attribution methods and identifies directions for developing more general methods.

A.1 PROOF OF PROPOSITION 1

The proof derives the ϵ-LRP explanation recursively through the network, using its propagation rule and a modified chain rule in which nonlinear gradients are replaced by LRP ratios.

  • A.1 PROOF OF PROPOSITION 1: The ϵ propagation rule assigns bias terms part of the relevance and uses a stabilizer in the denominator, whose contribution is assumed negligible for relevance conservation.
  • A.1 PROOF OF PROPOSITION 1: The ϵ-LRP proof begins by setting the target neuron's relevance equal to its output and then propagates relevance backward layer by layer.The inductive step assumes the LRP explanation at a generic layer and derives the relevance at the preceding layer.
  • A.1 PROOF OF PROPOSITION 1: The modified chain rule differs from the ordinary gradient chain rule by replacing each intervening nonlinearity derivative with gLRP, the output-to-input ratio at that nonlinearity.The proof applies the LRP propagation rule recursively using this modified derivative.

A.2 PROOF OF PROPOSITION 2

The proof shows how DeepLIFT multipliers correspond to modified gradients, including linear, nonlinear, and multi-input operations.

  • DeepLIFT’s multiplier chaining follows the same structure as the gradient chain rule, enabling comparison through modified gradients.The proof examines operation-specific multipliers and their composition across network paths.
  • For linear and convolutional layers, DeepLIFT multipliers equal the unchanged gradients w_ji.
  • For nonlinear single-input operations, the proof analyzes DeepLIFT’s multiplier through the operation’s modified gradient function.
  • Nonlinear operations with multiple inputs, such as 2D pooling, are not covered by DeepLIFT’s formulation and retain their original gradients.

A.3 PROOF OF PROPOSITION 4

The proof establishes that attribution methods coincide on linear models, while nonlinear interactions prevent scalar feature attributions from satisfying Sensitivity-n universally.

  • All listed attribution methods are equivalent when the model is linear or behaves linearly on the selected task.The proof represents the target as a sum of feature-wise linear components and derives identical attributions.
  • In the linear case, ϵ-LRP and zero-baseline DeepLIFT reduce to Gradient * Input, while Integrated Gradients and Occlusion-1 yield the same feature contribution.
  • All methods satisfy Sensitivity-n for every n on linear models, but nonlinear interactions force at least one of Sensitivity-1 or Sensitivity-2 to fail.
  • Scalar feature attributions cannot fully capture nonlinear interactions, so methods may provide partial explanations that emphasize different aspects.
  • Zero is a common baseline because it can be neutral for networks with zero biases and origin-crossing nonlinearities, although the choice remains arbitrary when biases differ.

C EXPERIMENTS SETUP

The MNIST experiments normalize images and compare DNN and CNN architectures across four activation functions to test attribution-method generalization.

  • MNIST images are normalized between -1 for background and 1 for digit strokes before training.
  • Both a DNN and a CNN are trained with four activation functions to evaluate attribution methods across architectures and nonlinearities.
  • The CNN configuration includes two 3x3 convolutional layers with 32 and 64 kernels, followed by two dense layers of 512 units.

C.2 CIFAR-10

The CIFAR-10 setup uses normalized images, a ReLU CNN with dropout, and channel-aggregated pixel attributions for gradient and occlusion methods.

  • CIFAR-10 inputs are normalized to [-1; 1], and the ReLU CNN reaches a final test accuracy of 80.5%.
  • Pixel attributions sum the three color-channel attributions, while Occlusion-1 sets all channels of a pixel to zero together.
  • The CNN uses paired 3x3 convolutional layers with 32 and 64 kernels, dropout, and a 256-unit dense layer.

C.3 INCEPTION V3

The experiments used pre-trained Inception V3 on normalized ImageNet-compatible images and shallow MLP and LSTM models for IMDB sentiment analysis.

  • C.3 INCEPTION V3: The Inception V3 test set contained 1000 ImageNet-compatible images and achieved 95.9% classification accuracy.Images were normalized to [-1; 1], with color channels handled as for CIFAR-10.
  • C.3 INCEPTION V3: The IMDB experiments trained shallow MLP and LSTM sentiment classifiers using embeddings for the 5000 most frequent words.Reviews were capped at 500 words, shorter reviews were padded, and hidden layers used ReLU nonlinearities.
  • C.3 INCEPTION V3: Both IMDB architectures reached a final test accuracy of 87.3%.Training used Adam with early stopping.
Loading 1711.06104v4…