Source-linked AI summary
Gradients of Counterfactuals
Mukund Sundararajan, Ankur Taly, Qiqi Yan
TL;DR
In deep nonlinear networks, saturation can leave important input features with tiny gradients, limiting their usefulness for assessing feature importance. Interior Gradients examine gradients at scaled counterfactual inputs; the paper reports class-representative visualizations, supporting ablations, and additive attributions. The approach is applicable across varied deep networks with little code, though image perturbations can be unnatural.
Problem
In deep nonlinear networks, saturation can give important input features tiny gradients, limiting gradients as measures of feature importance.
Source: §1 · p. 1Method
Interior Gradients assess feature importance by inspecting gradients at counterfactual inputs formed by scaling down the original input.
Source: §1 · p. 1, §2.3 · p. 4Results
Visualizations highlight class-representative features that ablations confirm the network treats as representative, and the attributions satisfy additivity with the prediction score.
Source: §2.4 · p. 6, §2.7 · p. 8Takeaways & Limitations
Interior Gradients apply to varied deep networks with little code, lowering the effort needed to debug them without detailed implementation knowledge.
Source: §4 · pp. 13–14Image perturbations can be unnatural, so score drops may reflect inputs unlike those encountered in training rather than feature relevance.
Source: §2.5 · p. 7
Abstract
from arXiv · showhide
Gradients have been used to quantify feature importance in machine learning models. Unfortunately, in nonlinear deep networks, not only individual neurons but also the whole network can saturate, and as a result an important input feature can have a tiny gradient. We study various networks, and observe that this phenomena is indeed widespread, across many inputs. We propose to examine interior gradients, which are gradients of counterfactual inputs constructed by scaling down the original input. We apply our method to the GoogleNet architecture for object recognition in images, as well as a ligand-based virtual screening network with categorical features and an LSTM based language model for the Penn Treebank dataset. We visualize how interior gradients better capture feature importance. Furthermore, interior gradients are applicable to a wide variety of deep networks, and have the attribution property that the feature importance scores sum to the the prediction score. Best of all, interior gradients can be computed just as easily as gradients. In contrast, previous methods are complex to implement, which hinders practical adoption.
1 INTRODUCTION
Input gradients offer a familiar way to assess feature importance, but saturation can make important features appear to have tiny gradients. Interior Gradients inspect gradients at scaled counterfactual inputs, avoiding the instrumentation and implementation effort associated with earlier techniques.
1 INTRODUCTION
- Saturation can give important input features tiny gradients, complicating the use of gradients to understand and debug deep nonlinear models.For linear models, gradients equal coefficients; for deep nonlinear models, they act as a local linear approximation.Source: §1 · p. 1
- Earlier techniques require network instrumentation and can take substantial developer effort because they are not primitive operations in standard machine-learning libraries.The authors also note that these techniques can be difficult to understand and may produce different importances for functionally equivalent networks.Source: §1 · p. 1
- Interior Gradients inspect gradients of carefully chosen counterfactual inputs scaled from the original, using the familiar gradient operation.The authors say this can let practitioners debug networks without detailed knowledge of their implementations.Source: §1 · p. 1
2 OUR TECHNIQUE
In the Inception image-classification example, gradients often highlight pixels that do not appear relevant to the predicted object. Ablating those pixels leaves the prediction intact, illustrating that image gradients can fail to capture feature importance.
2.1 GRADIENTS DO NOT REFLECT FEATURE IMPORTANCE
- For the highest-scoring ImageNet class, the authors compute input gradients and aggregate them across color channels into pixel-importance scores.They visualize those scores by scaling original pixel intensities, so higher-scoring pixels appear brighter.Source: §2.1 · p. 2
- In the reflex-camera example, many highlighted pixels fall left of or above the camera, rather than on the object or useful context such as its lens cap.The authors consider whether these pixels matter to the network's internal computation or whether image gradients fail to capture importance.Source: §2.1 · p. 3
- After off-camera pixels were ablated, Inception still predicted “reflex camera” with a softmax score of 0.9966, supporting that those pixels were irrelevant to this classification.Gradients on the ablated image still highlighted pixels outside the camera; the authors report similar failures across more images.Source: §2.1 · p. 3
2.2 SATURATION
Saturation, where a prediction function flattens near its input, can make gradients poor indicators of feature importance. In Inception, scaled-image tests show saturation across images, including in pre-softmax scores and intermediate layers.
2.2 SATURATION
- Saturation means the prediction function flattens near the input, leaving a tiny gradient that may not reflect feature importance.The authors describe this as saturation of the prediction function with respect to the input.Source: §2.2 · p. 3
- Across 30 randomly chosen ImageNet images, output trends flattened as pixel intensities were scaled from zero to their original values, often by α = 0.2.Saturation also appeared for images whose final score was below 1.0.Source: §2.2 · p. 3
- Inception's pre-softmax activation scores also saturate, so saturation is not limited to the softmax output.The authors additionally report saturation in intermediate-layer activations.Source: §2.2 · p. 3
- The authors report widespread saturation across images and greater network activity for counterfactual inputs at relatively low scaling values.They propose that this observation underlies their feature-importance technique; they also hypothesize that gradients saturate after features are learned adequately.Source: §2.2 · p. 3
2.3 INTERIOR GRADIENTS
Interior gradients inspect gradients at counterfactual inputs formed by scaling the original input, revealing how pixel importance changes along the scaling path. For the camera image, lower-scale gradients emphasize the camera, while higher-scale gradients shift importance above it.
2.3 INTERIOR GRADIENTS
- Interior gradients are gradients of counterfactual inputs formed by uniformly scaling the original image from zero to its actual values.They probe the network along the scaling curve rather than at a single input.Source: §2.3 · p. 4
- For the reflex-camera image, average pixel-importance magnitude is much larger at low α and nearly zero at high α, where saturation occurs.Each visualization shows relative pixel-score distribution, not absolute magnitude.Source: §2.3 · p. 4
- At lower α, the camera pixels are most important; as α increases, the region above the camera gains importance.The authors consider the high-magnitude gradients at lower α the primary drivers of the final prediction and more indicative of feature importance than gradients at α = 1.Source: §2.3 · p. 4
- The visualizations can be chained into an animation summarizing how pixel importance moves as α increases from zero to one.The paper motivates probing related counterfactuals because distributed network computations are difficult to inspect directly.Source: §2.3 · p. 4
- The authors speculate that the network first uses distinctive patterns for an initial prediction, then fine-tunes it using weaker patterns.This interpretation is based on the observed shift in attention across scaled inputs.Source: §2.3 · p. 4
2.4 CUMULATING INTERIOR GRADIENTS
Integrated gradients cumulate gradients along a path of counterfactual inputs, yielding feature attributions that sum to the output difference between the input and baseline. The paper applies this approach to images and reports that the resulting visualizations better capture important features.
2.4 CUMULATING INTERIOR GRADIENTS
- Integrated gradients cumulate interior gradients along a path of counterfactual inputs, with the baseline at γ(0) and the actual input at γ(1).For Inception, the path interpolates from a black image to the actual image.Source: §2.4 · pp. 5–6
- Integrated gradients sum to the difference between the function output at the final counterfactual and at the baseline.When the baseline prediction is near zero, they almost exactly distribute the output across input features.Source: §2.4 · p. 6
- The additivity property provides a sanity check against under- or over-attributing features, particularly when prediction scores are numerically meaningful.The authors contrast this with feature-ablation schemes, whose score changes can depend on conjunctive or disjunctive feature interactions.Source: §2.4 · p. 6
- A Riemann sum approximates integrated gradients by summing gradients at sufficiently close points along the counterfactual path.The paper describes this as straightforward to implement using repeated gradient calls, which may be batched.Source: §2.4 · p. 6
- For Inception, integrated-gradient visualizations appear better at capturing important image features than gradients at the actual image.The method scales the image along a path and aggregates gradients across the color channel for pixel-importance visualizations.Source: §2.4 · p. 7
- Integrated-gradient attributions depend on the network function rather than its implementation, unlike several other attribution methods discussed by the authors.The paper names DeepLift, LRP, DeConvNets, and Guided backpropagation as methods that do not share this property.Source: §2.4 · p. 7
2.5 EVALUATING OUR APPROACH
The section evaluates integrated gradients against gradients at the image using pixel ablations and localization, while noting limits of the evaluation and its scope.
2.5 EVALUATING OUR APPROACH
- Ablating top-ranked pixels caused a larger score drop for integrated gradients than for gradients at the image, averaged over 150 images.The score drop was measured across 50 steps, each ablating 100 pixels.Source: §2.5 · p. 7
- Pixel perturbations often produce unnatural images, so score drops could reflect inputs unlike those seen during training.The authors also note that they did not compare against white-box attribution methods because their focus was on easy-to-implement black-box techniques.Source: §2.5 · pp. 7–8
- Integrated gradients placed 8.4% more pixel attribution inside bounding boxes on average and did so for 82 of 100 images.The evaluation used images with bounding boxes smaller than two thirds of the image and whose ablation significantly lowered the object-class score.Source: §2.5 · p. 7
2.6 DEBUGGING NETWORKS
The authors use interior-gradient visualizations for both the predicted and ground-truth labels to inspect why Inception misclassifies images.
2.6 DEBUGGING NETWORKS
- The approach uses feature importance to investigate misclassified ImageNet images whose ground-truth label is outside Inception’s top five predictions.The authors visualize pixel importance for both the Inception label and the ground-truth label.Source: §2.6 · p. 8
- For one image with two objects, each label’s interior gradients emphasized its corresponding object, suggesting a label-ranking rather than recognition error.The image contained one object matching the ground-truth label and another matching Inception’s label.Source: §2.6 · p. 8
- For another image, similar visualizations for both labels emphasized the cricket, suggesting a more serious recognition error for the mantis.The cricket was emphasized by interior gradients computed for the Inception label, mantis.Source: §2.6 · p. 8
2.7 DISCUSSION
The discussion argues that scaled counterfactuals can yield reasonable attributions, while noting that attributions cannot represent feature interactions and may reflect unintelligible choices among correlated features.
2.7 DISCUSSION
- The authors speculate that scaling from a suitable baseline explores neuron activations in ReLU networks, helping explain why counterfactuals can seem reasonable even outside the training set.They extend the analogous argument to max pooling, average pooling, and softmax.Source: §2.7 · p. 8
- Attributions cannot represent interactions such as a molecule-binding prediction that depends on a conjunction of atoms and bonds.The limitation applies to attributions or importance scores generally, not just this technique.Source: §2.7 · p. 9
- With correlated features, the model may assign weight to either or both, and its chosen assignment may not be human-intelligible.The authors say it is unclear how feature-selection approaches to reduce correlations apply to deep models on dense input.Source: §2.7 · p. 9
2.8 RELATED WORK
The section places the approach among gradient-based, score-backpropagation, and model-approximation methods, contrasting their mechanisms and implementation demands.
2.8 RELATED WORK
- Input gradients are easy to implement, but saturation can make gradients at the actual input inaccurate measures of feature importance.Prior gradient-based methods include Baehrens et al. (2010) and Simonyan et al. (2013).Source: §2.8 · p. 9
- Score-backpropagation methods differ in how they propagate relevance through nonlinearities, while DeepLift uses activation deviations from a baseline.DeepLift (Shrikumar et al. (2016)), LRP (Binder et al. (2016)), DeConvNets (Zeiler & Fergus (2014)), and Guided backpropagation (Springenberg et al. (2014)) are examples.Source: §2.8 · p. 9
- DeepLift and LRP distribute prediction scores exactly, but their attributions can vary across functionally equivalent networks; the proposed method needs only gradients and is architecture-agnostic.The authors characterize its ease of implementation as its primary advantage over these methods.Source: §2.8 · p. 9
- Model-approximation methods learn a simpler local model from sampled outputs, but require training a new model around each input; integrated-gradient visualizations better reflected distinctive image features than raw gradients.Ribeiro et al. (2016a;b) proposed the local-approximation approach.Source: §2.8 · pp. 9–10, Figure 4 · p. 10
3 APPLICATIONS TO OTHER NETWORKS
Interior gradients extend feature attribution to molecular and language networks by probing gradients along scaled counterfactual inputs. Across these applications, they reveal saturation and provide interpretable attributions, including a network-architecture anomaly and word-level examples.
3 APPLICATIONS TO OTHER NETWORKS
- The applications include molecular graph convolutions for ligand screening and an LSTM language model for Penn Treebank, alongside vision networks.For non-vision networks, counterfactuals can scale an input embedding rather than pixel intensities.Source: §3 · pp. 10–11
3.1 LIGAND-BASED VIRTUAL SCREENING
- For molecular inputs, the method scales atom and atom-pair features toward zero and accumulates gradients to attribute the prediction to those features.These counterfactuals are not valid molecule featurizations, but the authors argue they remain valid network inputs because the operators accept continuous values bounded between zero and one.Source: §3.1 · p. 12
- For 25 active molecules on PCBA-588342, the W2N2-simple network’s softmax scores saturate near inputs at 0.9 ≤α ≤1; total feature gradients vary significantly along the scaling path, and molecule gradients are fully indicative.Source: §3.1 · p. 12
- Bonded atom pairs cumulatively contribute 46% of the prediction score, while other atom pairs contribute −3%.Attributions also helped chemists inspecting 100 active molecules spot dominant functional groups such as aromatic rings.Source: §3.1 · p. 12
- Molecular attribution highlights atoms and atom pairs with strongly positive or negative contributions, supporting inspection of features associated with activity.The ligand-screening task is to predict whether a molecule is active against a target.Source: §3.1 · pp. 11–12
- Integrated gradients exposed identical 0.0043 attributions for several differently connected carbon atoms in the W1N2 network.The authors traced this to its single convolution layer, which leaves atom and atom-pair features not fully convolved; this does not occur in networks with two or more convolutional layers.Source: §3.1 · p. 12
3.2 LANGUAGE MODELING
- The LSTM language model also saturates as word embeddings are scaled, with prediction curves generally flattening near α = 1.The reported exception is a prediction for an out-of-vocabulary word, whose softmax score is nonzero at α = 0.Source: §3.2 · p. 13
- In language modeling, integrated gradients yield attributions close to prediction scores, while ordinary gradients are tiny or misleading across the two examples.They highlight “more” for the prediction “than” and the first “ual” for the second “ual.”Source: §3.2 · p. 13
4 CONCLUSION
The conclusion presents Interior Gradients as a lightweight way to quantify feature importance across deep networks without instrumenting them. It frames easier network debugging as a practical consequence and suggests further exploration of the method.
4 CONCLUSION
- Interior Gradients quantifies feature importance across varied deep networks without network instrumentation and with little code.The method uses gradients at carefully chosen counterfactual inputs scaled from the original input.Source: §4 · p. 13
- The authors argue that the method can lower the effort needed to debug deep networks without requiring detailed knowledge of their implementation.They describe this as opening the possibility of easy and wide application.Source: §4 · p. 14
- The authors raise as an open question whether Interior Gradients could help address saturation during training or be useful wherever gradients are used.This is posed as a possibility for future investigation.Source: §4 · p. 14
A APPENDIX
The appendix supplies additional comparisons and saturation visualizations. It also illustrates that several attribution methods can differ across networks that compute the same function, unlike integrated gradients in the displayed example.
A APPENDIX
- The appendix includes further image comparisons showing integrated-gradient and image-gradient visualizations alongside each original image and its top-class label and score.Source: Figure 11 · p. 17
- For Inception’s deepest layer, mixed5b, the cosine distance between scaled-input and original-image activations is below 0.01 when α > 0.6.The figure plots L2 and cosine distances for 30 randomly selected ImageNet images and reports flattening trends across the shown layers.Source: Figure 12 · p. 18
- The W2N2 visualization plots softmax scores for task PCBA-58834 and sums of feature gradients for twenty active molecules.Source: Figure 13 · p. 18
- For two functionally equivalent networks, every displayed method except integrated gradients assigns different attributions.The comparison includes DeepLift, layer-wise relevance propagation, DeConvNets, and guided backpropagation.Source: Figure 14 · p. 19