Source-linked AI summary
How Important Is a Neuron?
Kedar Dhamdhere, Mukund Sundararajan, Qiqi Yan
TL;DR
The paper addresses how to measure the importance of hidden units when activation-based and gradient-based proxies can miss influence direction. It introduces conductance by decomposing Integrated Gradients through hidden units and evaluates it using theoretical and empirical comparisons.
Problem
Existing hidden-unit importance proxies based on activation or activation multiplied by gradient can fail to identify whether a unit positively or negatively influences a prediction.
Method
Conductance decomposes Integrated Gradients through a hidden unit using the chain rule, interpreting the result as attribution flow through that unit.
Results
Conductance correlated with ablation scores at 0.79 in one study and 0.88 on average in another, while identifying filters whose ablation changed predictions after 3.7 filters on average.
Takeaways & Limitations
Conductance provides a way to assess hidden-unit importance for individual predictions and to identify filters that materially affect network predictions.
Takeaways & Limitations
The paper does not establish that conductance is the unique best definition, instead supporting it through theoretical and empirical arguments.
Abstract
from arXiv · showhide
The problem of attributing a deep network's prediction to its \emph{input/base} features is well-studied. We introduce the notion of \emph{conductance} to extend the notion of attribution to the understanding the importance of \emph{hidden} units. Informally, the conductance of a hidden unit of a deep network is the \emph{flow} of attribution via this hidden unit. We use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We evaluate the effectiveness of conductance in multiple ways, including theoretical properties, ablation studies, and a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a sentiment analysis network over reviews. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.
1 Background and motivation
Existing attribution methods explain how input features influence predictions, but hidden-unit importance remains difficult to assess. Prior approaches use activation proxies, optimization-based visualizations, or auxiliary explanatory models with important limitations.
- Input attribution identifies how base features such as pixels contribute to a network prediction.
- Hidden-unit importance is often approximated with activation or gradient-times-activation, but these measures can miss whether a unit helps or opposes the prediction.ReLU activations are always positive even when a neuron's influence may be positive or negative.
- Optimization-based analyses can indicate what hidden units represent, but they do not establish their importance to predictions and may not work for every unit.
- Auxiliary linear models may change the relationships among features, making influence in the explanatory model differ from influence in the original network.
- The paper instead analyzes the network's own hidden-layer filters, accounting for the whole prediction while potentially producing less intuitive explanations.
2 Our Contribution
The paper introduces conductance to trace attribution from input features through hidden units and evaluates it with theoretical analysis, ablations, feature selection, and network-inspection studies. Across image and sentiment models, conductance identifies prediction-relevant filters and reveals distinct internal behaviors.
- Conductance decomposes Integrated Gradients through the chain rule to measure attribution flow through a hidden unit.
- The evaluation combines theoretical properties, ablations, class-level feature selection, and qualitative inspection across two large-scale networks.
- In Inception on ImageNet, 5 to 10 of 5484 filters account for each input's classification, and ablating high-conductance filters alters the output.
- Filters with high average conductance across images from one class are highly predictive of that class, while some filters influence multiple images and labels.
- In the sentiment network, many filters influence each input, filters predominantly specialize in positive or negative sentiment, and some capture negation.
3 Conductance
Conductance lifts Integrated Gradients from base features to hidden neurons by decomposing attribution along a baseline-to-input path. It can be measured for individual input variables, summed across variables, and aggregated across related neurons such as filters.
- Integrated Gradients computes attribution by integrating gradients along the straight-line path from baseline x′ to input x.The baseline may be a black image for image models or a zero embedding vector for text models.
- The baseline defines the reference for the sensitivity analysis used to produce attribution values.
- Conductance lifts base-feature attribution to a hidden neuron by decomposing Integrated Gradients through that neuron.
- Neuron-specific conductance can be computed for an individual input variable or summed across input variables to obtain total conductance.
- The paper uses variable-level conductance to explain a neuron's effects on input features and total conductance to discuss the neuron's importance.
- Conductances of logically related neurons, such as those in one filter, can be summed to measure the conductance of the set.
4 Evaluation of Conductance
The evaluation compares conductance with activation, Gradient*Activation, and internal influence using theoretical properties and examples of saturation, polarity, and sensitivity. Conductance satisfies conservation-related properties and avoids several incorrect signs or magnitudes exhibited by competing methods.
- Theoretical properties: Conductance satisfies completeness: conductances within a hidden layer sum to F(x) − F(x′), preserving the network’s redistributed evidence.This also yields layerwise conservation, whereas the three comparison methods do not satisfy completeness or layerwise conservation.
- Theoretical properties: Conductance satisfies linearity and assigns each linearly composed hidden neuron its output-weighted change from baseline.This provides an intuitive solution when a network behaves mostly linearly from a hidden layer.
- Theoretical properties: Conductance is insensitive when changing a hidden unit cannot change the prediction or when inputs cannot change that unit’s value.In either case, the hidden unit receives zero conductance.
- Saturation: In a saturated network z = max(2x, 1), Gradient*Activation assigns zero importance at x = 1, whereas conductance assigns the hidden unit a score of 1.Conductance integrates gradients along extrapolated inputs between the baseline and input, including the non-saturated region.
- Polarity and sensitivity: Activation cannot identify influence polarity for ReLU units, while conductance distinguishes positive from negative influence.ReLU activations are always positive even though upstream weights can make their influence positive or negative.
- Polarity and sensitivity: Gradient*Activation can overshoot: it assigns 1 − ϵ to a hidden unit whose network output is zero, while conductance assigns zero to all hidden units.The example uses f(x) = x followed by g(y) = max(y −1, 0), with baseline 0 and input 1 − ϵ.
- Polarity and sensitivity: Internal influence can reverse polarity: for g(f(x)) with f(x) = −x and g(y) = y, the output is −1 but internal influence is +1, whereas conductance is −1.The polarity mismatch is especially problematic because the represented unit is the only unit in its layer.
- Polarity and sensitivity: The authors report that incorrect-sign issues found in these examples also occur in real networks, including their sentiment analysis network.This motivates evaluating attribution methods beyond idealized examples.
5 Applying Conductance to an Object Recognition Model
The study applies conductance to Inception filters on ImageNet, testing whether high-conductance filters affect predictions and support class-level feature selection. Conductance identifies a small set of influential filters, including filters shared across images and filters predictive of classes.
- Experimental setup: The Inception network was analyzed at several hidden layers using filters as the unit of conductance analysis on ImageNet.The dataset has 1000 labels with about 1000 training images and 50 validation images per label.
- Experimental setup: Conductance was computed for each filter relative to the top predicted label, with pixel visualizations formed by aggregating conductance across color channels.The resulting visualizations highlight image regions associated with high-conductance filters.
- Cross-image insights: Filters with high conductance could be shared across images with different labels and corresponded to features such as glare, color, and red regions.Figure 1 shows filter 52 highlighting glare and filter 76 highlighting red-colored regions across example images.
- Ablation study: 0.79 was the Pearson correlation between conductance and ablation scores, compared with 0.487 for internal influence and 0.07 for activations.Gradient*activation had a correlation of 0.92 in the same comparison.
- Ablation study: 3.7 filter ablations were sufficient on average to change the network prediction, while only 3 of 100 images required more than 10 ablations.The maximum number of ablations required was 16; ablating the top 10 internal-influence filters changed predictions for only 5 of 100 images.
- Feature selection study: Aggregated high-conductance filters were used for class-level feature selection, and exemplar images outside the identification set showed related visual concepts for meerkat and cheeseburger labels.The examples include a titi for the meerkat-selected filter and a bagel for the cheeseburger-selected filter.
6 Applying Conductance to a Sentiment Model
The sentiment model’s filters show strong agreement with ablation-based importance, while conductance reveals signed and functionally specialized contributions. Most filters focus on either positive or negative sentiment, and selected filters capture negation-related patterns.
- Model and evaluation: The model embeds words into 50 dimensions, applies four convolutional window sizes with 64 feature maps each, and 1-max pools 256 filters.These pooled filters form the representation analyzed through conductance.
- Ablation study: Conductance correlates with filter ablation scores at 0.88 Pearson correlation, compared with 0.19 for activations.Gradient ∗activation correlates at 0.99, although the paper notes that this high value is somewhat artifactual.
- Ablation study: Internal influence correlates negatively with ablation scores at −0.43 Pearson correlation because it omits scaling terms included by conductance.The paper attributes conductance’s correct scale to its completeness property.
- Ablation study: Most filter ablation effects have opposing signs within an input, indicating that prediction scores combine several positive and negative hidden-unit effects.The ratio of the absolute summed ablation scores to the sum of absolute ablation scores is far below 1.0 for most inputs.
- Division of labour: Most filters contribute to either positive or negative sentiment, but not both, across around 4000 examples.Figure 3 encodes negative conductance in red, positive conductance in blue, and near-zero conductance in green using a 0.01 threshold.
- Negation: Three filters with high conductance for “not good” show negation-related patterns, while filters selected for “not bad” are largely focused on negation.The “not good” examples also include diminishing patterns and some stray errors.