Axiomatic Attribution for Deep Networks

Mukund Sundararajan; Ankur Taly; Qiqi Yan

Source-linked AI summary

Axiomatic Attribution for Deep Networks

Mukund Sundararajan, Ankur Taly, Qiqi Yan

arXiv:1703.01365v2Mar 4, 2017cs.LG

TL;DR

Attributing deep-network predictions to input features is difficult to evaluate, so the paper uses axioms to design Integrated Gradients. The method satisfies the identified axioms and supports applications across image, text, and chemistry networks, including debugging and rule extraction.

Problem
The paper addresses how to attribute a deep network’s prediction to its input features while evaluating whether attribution methods satisfy desirable properties.
Method
The paper uses an axiomatic framework centered on Sensitivity and Implementation Invariance to design Integrated Gradients, a gradient-based attribution method.
Results
Integrated Gradients satisfies Sensitivity and Implementation Invariance and applies across image, text, and chemistry networks for understanding, debugging, rule extraction, and prediction explanation.
Takeaways & Limitations
Integrated Gradients provides a theoretically justified way to attribute deep-network predictions using a few gradient-operator calls across varied network applications.
Takeaways & Limitations
Simple gradient-product attributions are limited because they break the Sensitivity axiom, motivating the paper’s axiomatic design of Integrated Gradients.

Abstract

from arXiv · show

We study the problem of attributing the prediction of a deep network to its input features, a problem previously studied by several other works. We identify two fundamental axioms---Sensitivity and Implementation Invariance that attribution methods ought to satisfy. We show that they are not satisfied by most known attribution methods, which we consider to be a fundamental weakness of those methods. We use the axioms to guide the design of a new attribution method called Integrated Gradients. Our method requires no modification to the original network and is extremely simple to implement; it just needs a few calls to the standard gradient operator. We apply this method to a couple of image models, a couple of text models and a chemistry model, demonstrating its ability to debug networks, to extract rules from a network, and to enable users to engage with models better.

1. Motivation and Summary of Results

The paper frames attribution as assigning a deep network’s prediction to input features relative to a baseline, motivated by interpretability and practical model improvement. It proposes an axiomatic approach because attribution methods are difficult to evaluate empirically, and demonstrates integrated gradients across image, text, and chemistry networks.

Problem formulation: Attribution assigns each input feature a contribution to a deep network’s prediction relative to a baseline input.Formally, the attribution is a vector whose components represent the contributions of the corresponding input features.
Motivation: Attribution supports understanding input-output behavior, improving networks, providing recommendation rationales, and helping users interpret predictions.The passage specifically gives medical imaging as an example in which attribution could inform a doctor about a predicted condition.
Motivation: Because attribution methods are hard to evaluate empirically, the paper uses axioms to separate model behavior from attribution-method behavior.The authors identify two axioms and report that most previous methods fail to satisfy one of them.
Method: Integrated gradients require no network instrumentation and can be computed with a few gradient-operation calls.This makes the technique accessible to novice practitioners.
Applications: Applications span two image networks, two text-processing networks, and one chemistry network, supporting network understanding, debugging, rule extraction, and user assistance.The experiments demonstrate ease of applicability across these model types.

2. Two Fundamental Axioms

The section defines Sensitivity and Implementation Invariance as desirable axioms for attribution methods, then shows how gradients and backpropagation-based methods violate them in distinct ways.

Sensitivity: Sensitivity(a) requires a feature that differs between an input and baseline with different predictions to receive non-zero attribution.This axiom is introduced for every input and baseline differing in one feature.
Sensitivity: Gradients violate Sensitivity(a) when the prediction function is flat at the input, assigning zero attribution despite a different prediction from the baseline.For f(x) = 1 − ReLU(1−x), the baseline x = 0 and input x = 2 change the function from 0 to 1, but the input gradient is zero.
Sensitivity: DeConvNets and Guided back-propagation also violate Sensitivity(a) because they back-propagate through ReLU only when it is active at the input.Their attributions are therefore zero for features with zero input gradient even when the baseline gradient is non-zero.
Implementation Invariance: Implementation Invariance requires functionally equivalent networks to produce identical attributions, since attribution should not depend on implementation details.Functionally equivalent networks have equal outputs for all inputs despite potentially different implementations.
Implementation Invariance: DeepLift and LRP fail Implementation Invariance because discrete gradients do not generally obey the chain rule used to compose attributions.Consequently, their attributions can depend on unimportant model details even when the underlying network function is unchanged.

3. Our Method: Integrated Gradients

Integrated Gradients attributes a network’s prediction by integrating gradients along the straight-line path from a baseline to the input. The method combines gradient-based Implementation Invariance with Sensitivity and satisfies the stronger Completeness axiom.

3. Our Method: Integrated Gradients: Integrated Gradients computes gradients along the straight-line path from baseline x′ to input x and accumulates them as a path integral.For image networks, the baseline may be a black image; for text models, it may be the zero embedding vector.
3. Our Method: Integrated Gradients: Completeness strengthens Sensitivity(a), so Integrated Gradients satisfies Sensitivity(a) when baseline and input differ in only one variable.In that case, Completeness equates the output difference with the attribution to the differing variable.
3. Our Method: Integrated Gradients: Completeness requires attributions to sum to F(x) − F(x′), providing a comprehensive accounting of the prediction difference.The paper presents this as a desirable sanity check, especially when the output has numeric meaning rather than merely identifying the top label.
3. Our Method: Integrated Gradients: Integrated Gradients satisfies Implementation Invariance because its attributions are based only on the network’s gradients.The supplied passage introduces the method as combining Implementation Invariance of Gradients with the Sensitivity of methods such as LRP or DeepLift.

4. Uniqueness of Integrated Gradients

The section motivates integrated gradients through axioms because empirical evaluations cannot reliably distinguish data-perturbation artifacts from model or attribution failures. It shows that path methods satisfy key axioms, while integrated gradients is uniquely symmetry-preserving among them, with practical advantages over Shapley-Shubik averaging.

Axiomatic motivation: Empirical evaluations could not distinguish artifacts caused by data perturbations, a misbehaving model, or a misbehaving attribution method.This motivates the paper’s axiomatic approach to designing attribution methods.
Path methods: Path methods integrate gradients along paths from the baseline to the input, with integrated gradients corresponding to the straightline path.The straightline path is γ(α) = x′ + α × (x − x′).
Path methods: Path methods satisfy Implementation Invariance, Completeness, and Sensitivity(a), and are the only methods satisfying the section’s specified desirable axioms.Sensitivity(b) requires zero attribution for variables on which the implemented function does not mathematically depend; Linearity requires weighted-sum attributions under linear network composition.
Canonical path: Integrated gradients is the unique path method that is symmetry-preserving.Symmetry preservation requires identical attributions when symmetric variables have identical input and baseline values.
Alternative methods: Shapley-Shubik also satisfies the relevant axioms by averaging attributions over n! extremal paths, but this is computationally expensive.For a 100X100 image, n = 10000; integrated gradients instead operates with 20 to 300 calls.

5. Applying Integrated Gradients

Applying Integrated Gradients requires selecting a baseline with a near-zero score and approximating the path integral by summing gradients along the straight-line path to the input. In practice, 20–300 approximation steps typically achieve 5% accuracy, with an attribution-sum check recommended.

Selecting a Benchmark: A suitable baseline should have a near-zero score, allowing attributions to be interpreted as a function of the input.The paper cautions that baseline selection is a key step in applying Integrated Gradients.
Computing Integrated Gradients: Integrated Gradients can be approximated by summing gradients at sufficiently small intervals along the straight-line path from baseline x′ to input x.This converts the integral into a discrete summation over points on the path.
Computing Integrated Gradients: The Riemann approximation uses m steps and can be implemented with a gradient call inside a loop, including TensorFlow’s tf.gradients.The inputs are x′ + k × (x − x′) for k = 1, . . . , m, and the computations can also be batched.
Computing Integrated Gradients: 20–300 steps usually approximate the integral within 5%.Developers should check whether attributions approximately sum to the score difference between the input and baseline, increasing m if necessary.

6. Applications

Integrated Gradients was applied across image, language, and chemistry networks to visualize feature importance, uncover rules and undesirable correlations, align translations, and diagnose architectural anomalies. These applications show how attribution can support model interpretation, debugging, and trust.

Image models: In ImageNet object recognition, integrated-gradient visualizations better reflected distinctive image features than gradients multiplied by the actual image.Pixel gradients were computed for the highest-scoring class using a black-image baseline.
Image models: For diabetic-retinopathy prediction, attributions localized to retinal lesions, with negative attribution inside lesions and positive attribution around their periphery.This indicated that the network focused on lesion boundaries and matched lesions visible to human observers.
Natural language models: Question-classification attributions recovered familiar rules, revealed a novel “total number” numeric-answer rule, and exposed the undesirable “charles” yes/no correlation.They largely agreed with commonly used triggers such as “how many” for numeric questions.
Natural language models: In neural machine translation, token-level attributions aligned output wordpieces with intuitive input counterparts, such as “und” with “and” and “morgen” with “morning”.The approximation used 100 −1000 steps because the network was highly nonlinear.
Chemistry model: In ligand-based virtual screening, integrated gradients visualized atom and atom-pair contributions and supported accounting for each feature’s contribution to the prediction.The network used molecular graph convolution, with zeroed atom and atom-pair feature vectors as the baseline.
Chemistry model: Attributions exposed identical atom contributions in W1N2 despite different bonding neighborhoods, tracing the anomaly to incomplete convolution of atom and atom-pair features.Atoms with the same type and the same number of bonds of each type therefore contributed identically.

7. Other Related work

This section situates attribution among broader efforts to interpret deep networks, discussing local surrogate models and attention as related but limited approaches. It contrasts their limitations with the proposed technique’s use of only a few gradient-operation calls.

Other interpretability work: Broader deep-network interpretability work has largely focused on computer-vision models, examining what individual neurons compute and how their representations contribute during prediction.The cited examples include neuron visualization and representation interpretation.
Local surrogate models: Local surrogate approaches are implementation-agnostic and satisfy implementation invariance, but may fail to escape flat regions of the prediction function.Their local exploration does not guarantee reaching an informative region around the input.
Local surrogate models: For dense inputs such as images, local surrogate methods are expensive because they explore a region proportional to the number of pixels and train a model there.The computational burden arises from the size of the explored local region and the additional model training.
Proposed technique: In contrast, the proposed technique requires only a few calls to the standard gradient operation.This is presented as an implementation advantage over locally approximating dense-input networks.
Attention mechanisms: Attention is an incomplete proxy for attribution because input tokens can influence outputs through memory cells, recurrent states, and attention itself.Focusing only on attention ignores other modes of influence in attention-equipped LSTMs.

8. Conclusion

The paper’s primary contribution is Integrated Gradients, a theoretically justified method for attributing deep-network predictions to inputs. It also develops an axiomatic framework while leaving feature interactions, network logic, and many debugging questions unresolved.

Conclusion: Integrated Gradients attributes a deep network’s prediction to its inputs using a few gradient-operator calls across varied deep networks.The method has a strong theoretical justification.
Conclusion: The axiomatic framework clarifies desirable attribution-method properties and rules out artifacts caused by the attribution method itself.It is inspired by cost-sharing literature from economics and helps distinguish data, network, and method artifacts.
Conclusion: The paper leaves unanswered how input features interact, what logic the network employs, and how to debug its input-output behavior.The authors report progress on understanding the relative importance of input features but do not address these broader questions.

A. Proof of Theorem 1

The proof constructs a symmetric function whose Integrated Gradients attributions differ for two input dimensions, contradicting the requirement that symmetric variables receive identical attributions.

Construction: The proof assumes γ_i(t_0) > γ_j(t_0) and selects the maximal interval (t_1, t_2) where γ_i(t) > γ_j(t).At the interval endpoints, γ_i and γ_j coincide at values a and b.
Construction: It defines a piecewise function f based symmetrically on x_i and x_j, using thresholds a and b.The function is 0 when min(x_i, x_j) ≤ a, constant when max(x_i, x_j) ≥ b, and (x_i − a)(x_j − a) otherwise.
Contradiction: Although x_i and x_j are symmetric and should receive identical attributions, the integrand assigns γ_j(t) − a to x_i and γ_i(t) − a to x_j on (t_1, t_2).The latter attribution is strictly larger throughout the interval, while outside it the function is constant and all attributions are zero.

B. Attribution Counter-Examples

The section constructs equivalent-network examples showing that DeepLift and LRP violate implementation invariance, while Deconvolution and Guided backpropagation violate sensitivity. In the example, functionally equivalent networks receive different attributions from methods other than Integrated Gradients, and some methods assign zero attribution to an input that affects the output.

Axiomatic violations: DeepLift and LRP break implementation invariance, whereas Deconvolution and Guided backpropagation break sensitivity.These violations are demonstrated through explicit counterexamples.
Implementation invariance: For two functionally equivalent networks, Integrated Gradients gives identical attributions, while DeepLift and LRP give different attributions.The networks are evaluated at x1 = 3, x2 = 1 with reference input x1 = 0, x2 = 0 for Integrated Gradients and DeepLift.
Implementation invariance: The networks are equivalent because any disagreement between their internal forms occurs only when x1 < 1, where both ReLU outputs equal 0.Their internal functions h and k differ for x1 < 1, but the outer ReLU makes f and g identical there and elsewhere.
Sensitivity: For fixed x1 > 1, the output decreases as x2 rises from 0 to x1 −1, yet Deconvolution and Guided backpropagation assign x2 zero attribution.Their back-propagated signal at ReLU(x2) is negative, producing zero attribution despite the output’s sensitivity to x2.