Source-linked AI summary
Full-Gradient Representation for Neural Network Visualization
Suraj Srinivas, Francois Fleuret
TL;DR
Saliency maps cannot generally capture structured feature interactions while satisfying both local sensitivity and global completeness. The paper introduces full-gradients, attributing responses to inputs and neurons, and aggregates them into FullGrad for convolutional networks. FullGrad outperforms competitive methods in pixel perturbation and remove-and-retrain evaluations, while the authors note scope limitations for signed-saliency tasks and approximate aggregation.
Problem
Saliency maps assign scores to individual features, but structured groups can matter jointly, and completeness and weak dependence cannot both be achieved by saliency maps.
Method
Full-gradients decompose network responses into input and neuron attributions, while FullGrad aggregates these components into an approximate convolutional saliency map.
Results
FullGrad outperforms existing competitive methods on pixel perturbation and remove-and-retrain tests for explaining model behavior.
Takeaways & Limitations
Input attributions capture individual-feature sensitivity, while neuron attributions account for interactions among input features and together satisfy completeness.
Takeaways & Limitations
FullGrad’s post-processing does not work well for signed-saliency digit-flipping tasks, and its convolutional saliency map remains approximate.
Abstract
from arXiv · showhide
We introduce a new tool for interpreting neural net responses, namely full-gradients, which decomposes the neural net response into input sensitivity and per-neuron sensitivity components. This is the first proposed representation which satisfies two key properties: completeness and weak dependence, which provably cannot be satisfied by any saliency map-based interpretability method. For convolutional nets, we also propose an approximate saliency map representation, called FullGrad, obtained by aggregating the full-gradient components. We experimentally evaluate the usefulness of FullGrad in explaining model behaviour with two quantitative tests: pixel perturbation and remove-and-retrain. Our experiments reveal that our method explains model behaviour correctly, and more comprehensively than other methods in the literature. Visual inspection also reveals that our saliency maps are sharper and more tightly confined to object regions than other methods.
1 Introduction
Saliency maps struggle to represent structured feature interactions while satisfying both local sensitivity and global completeness. The paper introduces full-gradients, which attribute importance to inputs and neurons, and evaluates their convolutional approximation, FullGrad.
- Motivation: Individual pixels may be unimportant alone even when groups of pixels form critical object regions.The paper argues that saliency methods should capture structured groups, not only individual feature scores.
- Motivation: Saliency methods are expected to reflect output sensitivity while their feature scores completely explain the network output.These are presented as local and global attribution requirements.
- Contribution: Full-gradients assign importance to both input features and individual neurons, capturing pixel sensitivity and structured groups of pixels.The representation is more expressive than a saliency map and satisfies both importance notions simultaneously.
- Core result: Weak dependence and completeness cannot be satisfied simultaneously by any saliency method.The paper identifies this incompatibility as a source of counter-intuitive attribution behavior.
- Evaluation: FullGrad is an approximate convolutional saliency map derived by aggregating full-gradient components.The paper evaluates it using pixel perturbation and remove-and-retrain tests, reporting that it outperforms competitive existing methods.
3 Local vs. Global Attribution
The paper formalizes local attribution as weak dependence and global attribution as completeness, then proves that ordinary saliency maps cannot satisfy both for piecewise-linear functions. Full-gradients address this restriction by representing input and neuron contributions.
- Definitions: Weak dependence requires a saliency map to depend only on the local linear neighborhood and its parameters.For piecewise-linear models, the map is independent of the exact input within a neighborhood.
- Definitions: Completeness requires recovering the network output, or its baseline difference, from saliency information and the input.The baseline formulation uses both the input and baseline saliency maps.
- Impossibility result: For any piecewise-linear function, no saliency map can generally satisfy completeness and weak dependence simultaneously.The proof establishes an incompatibility between these two attribution notions.
- Implications: Integrated gradients, Deep Taylor decomposition, and DeepLIFT satisfy completeness but therefore do not satisfy weak dependence.The paper illustrates the resulting issue with integrated gradients on a piecewise-linear function.
- Implications: Integrated gradients can assign different relative importances at points lying on the same linear function even when x1 is more sensitive than x2.The reported attributions are (10, 6), (2.5, 3.5), and (4, 4) at three points.
- Resolution: The restriction arises because saliency maps cannot summarize both weights and biases, especially across neurons in neural networks.Full-gradients account for bias terms while satisfying weak dependence and completeness.
4 Full-Gradient Representation
Full-gradients extend attribution beyond input features by assigning importance to neurons, enabling a complete representation that also preserves local sensitivity. For convolutional networks, FullGrad aggregates these components into approximate spatial saliency maps.
- Properties of Full-Gradients: For piecewise-linear functions, both the input-gradient and bias-gradient components are locally constant within a linear region.This establishes weak dependence on inputs for the full-gradient representation under the stated assumptions.
- Properties of Full-Gradients: Full-gradients exactly recover the neural network output, satisfying completeness.The representation combines input-gradients with bias-gradients, including implicit biases from nonlinearities or batch normalization.
- Properties of Full-Gradients: Full-gradients address saturation by attributing importance to function parameters when input gradients are zero.In the example f(x) = a − ReLU(b − x) at x = 2, the method attributes (1, 0) to the two biases.
- FullGrad: Full-Gradient Saliency Maps for Convolutional Nets: For convolutional networks, spatially shaped bias-gradients support per-neuron and per-layer visualizations, which are aggregated into FullGrad maps.The maps combine input-gradients and intermediate bias-gradients; the network-wide saliency map remains approximate and does not preserve both completeness and weak dependence.
5 Experiments
The experiments evaluate FullGrad with pixel perturbation and remove-and-retrain benchmarks, alongside qualitative comparisons with established saliency methods. FullGrad most strongly identifies important pixels and produces sharper, more object-confined visualizations.
- Quantitative Evaluation: FullGrad is evaluated with pixel perturbation on Imagenet 2012 and remove-and-retrain on CIFAR100.The two tests assess saliency using complementary perturbation procedures and datasets.
- 5.1 Pixel perturbation: Pixel perturbation removes k least-salient pixels and measures the absolute fractional change in the most confident class output; lower curves are better.The evaluation uses a VGG-16 model with batch normalization on the Imagenet 2012 validation dataset.
- 5.2 Remove and Retrain: Remove-and-retrain removes the top-k pixels across a dataset, retrains a classifier, and evaluates accuracy; lower accuracy indicates better identification of crucial pixels.The benchmark is designed to test whether removed pixels were important to classification.
- 5.2 Remove and Retrain: FullGrad decreases CIFAR100 accuracy the most among the compared methods in the remove-and-retrain benchmark.Comparisons include gradCAM, input-gradients, integrated gradients, and smooth grad squared.
- 5.3 Visual Inspection: Qualitatively, FullGrad combines highlighted regions with boundaries tightly confined to objects, unlike noisy input-gradient-based maps and broad grad-CAM regions.The authors attribute this combination to including both input-gradients and intermediate-layer gradients.
6 How to Choose ψ(·)
FullGrad’s post-processing function involves unavoidable trade-offs: different choices favor different properties and evaluation settings. The paper recommends selecting ψ(·) according to the metrics and datasets relevant to the application.
- Any post-processing function cannot preserve all desired properties because the resulting representation remains saliency-based.Thus, each choice prioritizes some properties over others.
- The paper’s default post-processing performs well on pixel perturbation and ROAR for image data, where saliency magnitude matters more than sign.
- Signed-saliency tasks such as digit-flipping expose a weakness of the proposed FullGrad post-processing.Minimal bilinear upsampling performs better on digit-flipping but lowers performance on pixel perturbation and ROAR.
- Practitioners should choose ψ(·) based on the evaluation metrics and datasets most relevant to their application.The authors consider FullGrad sufficient for most computer vision applications but recommend domain-specific metrics and expert consultation elsewhere.
7 Conclusions and Future Work
The paper concludes that FullGrad visualizes neural-network functions by attributing importance to both inputs and intermediate neurons. It also argues that interpretability methods require explicit awareness of trade-offs and clearer evaluation metrics.
- FullGrad attributes importance to inputs and intermediate neurons, combining individual-feature sensitivity with interactions among input features.
- Input and neuron attributions individually satisfy weak dependence, while together they satisfy completeness.
- Saliency methods’ inability to satisfy multiple intuitive properties suggests a need for more expressive interpretability schemes or explicit trade-offs.
- The paper identifies unambiguous evaluation metrics for saliency maps as an open issue and suggests interpretability-by-design models as future work.
8 Proof of Incompatibility
The proof shows that, in general, no saliency map for a piecewise-linear function can satisfy both completeness and weak dependence. The argument relies on saliency maps encoding fewer parameters than the underlying local functions.
- Weak dependence makes a saliency map within each linear region depend only on that region’s local parameters.
- Completeness requires recovering either f(x) or f(x)−f(x0) from saliency maps and inputs.
- The resulting contradiction shows that completeness and weak dependence cannot both hold for piecewise-linear functions in general.
- A D-dimensional saliency map cannot generally summarize a D+1-parameter linear model, producing distinct functions with identical saliency representations.
- The incompatibility disappears in corner cases such as bias-free piecewise-linear networks or linear regions satisfying the required bias condition.
9 Full-gradient Proofs
The full-gradient proofs derive the network output from input and bias sensitivities. They use ReLU homogeneity for bias-free networks and represent biases as additional constant inputs for biased networks.
- For bias-free ReLU networks, positive homogeneity yields f(x)=∇_x f(x)^T x.The proof applies first-order Taylor expansion around a scaled input.
- Biases are handled by introducing an all-ones bias input multiplied by the bias parameters.
- For biased ReLU networks, chain-rule relations connect derivatives with respect to bias inputs, intermediate pre-activations, and bias parameters.
- Bias gradients are constant within each linear region of a piecewise-linear neural network.The argument is shown for a one-hidden-layer ReLU network and extended recursively to deeper networks.
10 Experiments to Illustrate Post-Processing Trade-offs
The experiments compare FullGrad variants and other saliency methods on MNIST digit flipping and pixel perturbation, revealing metric-dependent trade-offs rather than one consistently superior method.
- Digit Flipping: FullGrad (minimal) outperforms FullGrad and other methods on the digit flipping task, where larger numbers are better.The task converts digit 8 images into digit 3 images by removing pixels supporting 8 and opposing 3.
- Pixel Perturbation: Pixel perturbation removes the least salient pixels and measures the fractional change in neural-network output; smaller changes indicate better saliency.The experiment evaluates saliency maps on MNIST and compares methods including Integrated gradients and FullGrad variants.
- Pixel Perturbation: Integrated gradients performs best overall on MNIST pixel perturbation, while regular FullGrad outperforms FullGrad without absolute values.The authors hypothesize that MNIST’s binary pixels may favor Integrated gradients in this test.
- Cross-Metric Comparison: Digit flipping favors FullGrad (no abs), whereas pixel perturbation favors Integrated Gradients followed by FullGrad and FullGrad (no abs).The differing rankings suggest that no single saliency or post-processing method is consistently better across metrics.
11 Saliency Results
Figures 4 and 5 compare different neural network saliency methods.
- Figure 4 presents a comparison of different neural network saliency methods.
- Together, the two figures provide comparisons among neural network saliency methods.
- Figure 5 presents a comparison of different neural network saliency methods.