Source-linked AI summary
Neuron Shapley: Discovering the Responsible Neurons
Amirata Ghorbani, James Zou
TL;DR
The paper addresses the challenge of identifying which neurons drive a neural network’s performance, especially when neurons interact and networks contain many elements. It introduces Neuron Shapley with an efficient multi-armed-bandit estimator and applies it to accuracy, fairness, and robustness. The resulting analyses identify sparse critical or responsible neurons and support post-training model repair without retraining.
Problem
Existing neuron-interpretation approaches can be ad hoc, while neuron contributions involve interactions and are difficult to compute efficiently in large networks.
Method
Neuron Shapley assigns neurons interaction-aware performance contributions, estimated efficiently with a multi-armed-bandit algorithm and evaluated by removing neurons without retraining.
Results
Neuron Shapley discovers sparse critical neurons across accuracy, fairness, and robustness, including neurons whose removal reduces fairness disparities or adversarial vulnerability.
Takeaways & Limitations
Removing a few responsible neurons can support post-training model interpretation and repair without access to training data.
Takeaways & Limitations
The paper focuses on convolutional networks because most interpretation methods considered use image data, although the approach is applicable to other architectures and machine-learning models.
Abstract
from arXiv · showhide
We develop Neuron Shapley as a new framework to quantify the contribution of individual neurons to the prediction and performance of a deep network. By accounting for interactions across neurons, Neuron Shapley is more effective in identifying important filters compared to common approaches based on activation patterns. Interestingly, removing just 30 filters with the highest Shapley scores effectively destroys the prediction accuracy of Inception-v3 on ImageNet. Visualization of these few critical filters provides insights into how the network functions. Neuron Shapley is a flexible framework and can be applied to identify responsible neurons in many tasks. We illustrate additional applications of identifying filters that are responsible for biased prediction in facial recognition and filters that are vulnerable to adversarial attacks. Removing these filters is a quick way to repair models. Enabling all these applications is a new multi-arm bandit algorithm that we developed to efficiently estimate Neuron Shapley values.
1 Introduction
The paper introduces Neuron Shapley to systematically identify neurons that contribute most to network performance while accounting for neuron interactions. It combines this framework with efficient estimation and applies it to interpretation, fairness, and robustness.
- Framework: Neuron Shapley quantifies each neuron’s contribution to network function while accounting for complex interactions between neurons.Its multi-armed bandit algorithm is designed to estimate these values efficiently in large networks.
- Empirical findings: Fewer than 30 neurons or filters can be crucial for achieving good prediction accuracy on standard image-recognition tasks.Interpreting these critical neurons provides systematic insights into network function.
- Applications: Neuron Shapley identifies neurons responsible for biased facial-recognition predictions and vulnerability to adversarial attacks.These applications extend beyond conventional model interpretation.
- Applications: Zeroing out culprit neurons can reduce fairness disparities without much degradation to overall accuracy, enabling post-hoc model repair without retraining.The paper contrasts this with related fairness-repair approaches that typically involve retraining.
- Contributions: The paper presents conceptual, algorithmic, and empirical contributions linking neuron valuation to model interpretation and repair.Its experiments report sparse critical neurons across accuracy, fairness, and robustness.
- Related work: Compared with neuron-importance methods based on extending feature-importance algorithms, Neuron Shapley accounts for broader neuron interactions and fair credit allocation.The related methods are also mainly tailored to a specific sample’s prediction.
2 Shapley Value for Neurons
The paper formulates neuron contribution as a performance-allocation problem over subnetworks and uses Shapley value to assign interaction-aware, axiomatically justified credit. Contributions are evaluated by removing neurons without retraining.
- Model formulation: A trained model is treated as a set of elements, such as convolutional filters, whose contributions are partitioned from a performance metric.The metric may represent accuracy, loss, or disparity across minority groups.
- Model formulation: Neuron Shapley assigns each neuron a contribution that sums to the model’s overall performance metric.The framework applies to convolutional networks and can extend to other architectures and machine-learning models.
- Evaluation procedure: Neuron contributions are evaluated by fixing removed filters to their mean validation-set output while keeping all network weights fixed.This preserves propagated mean statistics and avoids the prohibitive cost of fine-tuning every subnetwork.
- Axiomatic properties: The Shapley formulation is the unique allocation satisfying properties such as additivity, zero contribution, and symmetry.These properties require null neurons to receive zero value and exchangeable neurons to receive equal value.
- Shapley valuation: A neuron’s contribution is its marginal performance effect across subnetworks, normalized across subnetworks of the same cardinality.This construction accounts for interactions among neurons, including cases where neurons help only jointly.
3 Estimating Neuron Shapley
Exact Neuron Shapley computation is exponential, so the paper reframes estimation statistically and combines Monte Carlo sampling, early truncation, and adaptive multi-armed-bandit sampling. The resulting method targets efficient identification of top-contributing neurons.
- Computational challenge: Computing exact Shapley values requires exponentially many operations because it involves exponentially many neuron subsets.The paper therefore develops approximation methods intended to provide orders-of-magnitude speedups.
- Monte Carlo estimation: Monte Carlo estimation approximates each Shapley value as the mean marginal contribution over uniformly sampled permutations of model elements.The elements preceding a neuron in a permutation define the subnetwork used for that marginal contribution.
- Approximation methods: Early truncation skips marginal-contribution computations when a sampled subnetwork’s performance falls below a threshold at which the model is considered dead.This can produce computational savings close to one order of magnitude.
- Adaptive sampling: Adaptive sampling tracks confidence bounds and samples only neurons that could still belong to the top-k set.The method stops when the top-k neurons are confidently separated within an error tolerance.
- Combined algorithm: Combining the approximation techniques yields Truncated Multi Armed Bandit Shapley, or TMAB-Shapley.The algorithm prioritizes identifying important neurons rather than estimating every Shapley value exactly.
4 Experiments & Applications
Experiments show that Neuron Shapley identifies sparse, critical filters for global and class-specific accuracy, while supporting efficient estimation, interpretation, fairness repair, and adversarial-robustness interventions.
- Experimental setup: 78.1% test accuracy Inception-v3 and 98.0% test accuracy SqueezeNet provide the two main experimental settings.The models contain 17,216 and 2,976 filters, respectively.
- Efficient estimation: TMAB-Shapley requires around one order of magnitude fewer samples than MC-Shapley, with most filters converging using less than 10k samples.MC-Shapley requires on the order of 100k samples.
- Global accuracy: Removing the top 10 Inception-v3 filters drops accuracy from 74% to 38%, while removing the top 20 drops it to 8%; 20 random filters do not change accuracy.The result demonstrates that the highest-Shapley filters concentrate global predictive importance.
- Interpretation: High-Shapley filters provide interpretable visualizations: earlier layers capture color and texture, whereas later layers capture concepts such as colorfulness and crowdedness.The top 100 filters tend to be concentrated in earlier layers, while deeper filters are more class-specific.
- Class-specific accuracy: For four representative classes, removing the top 40 class-specific filters causes a dramatic class-accuracy decline while leaving overall model performance intact.Neuron Shapley outperforms alternative identification approaches, and critical filters include interpretable class-related features such as stripes and dumbbell shapes.
- Fairness repair: Zeroing negatively contributing fairness filters raises black-female accuracy from 54.7% to 81.9% and average PPB accuracy from 84.9% to 91.7%.Accuracy on the original CelebA data drops only slightly, supporting rapid post-training repair.
- Adversarial robustness: Zeroing the top 16 vulnerable filters reduces adversarial success from nearly 100% to 0.1%, while clean-image performance falls from 74% to 67%.The modified model remains vulnerable to a new white-box attack tailored to the altered network, but black-box attack success drops by 37% on average.
- Comparison with prior methods: Compared with neuron conductance, Neuron Shapley finds critical filters with fewer removals across global accuracy, fairness, adversarial robustness, and class-specific accuracy.For ImageNet, 30 Shapley filters achieve a random-accuracy reduction that requires twice as many conductance filters; both methods have similar computational costs.
5 Discussion
The discussion presents Neuron Shapley as a post-training framework for finding sparse critical neurons and applying targeted model repairs without training data. It also emphasizes the computational cost of Shapley estimation and the multi-arm bandit algorithm introduced to reduce it.
- Framework and findings: Neuron Shapley quantifies individual neuron contributions while accounting for neuron interactions and reveals sparse critical structure at global and class levels.The framework is positioned as theoretically principled through its connection to game theory.
- Model repair: Removing a few responsible neurons can improve fairness or reduce adversarial fragility without access to the training data.The discussion frames these edits as post-training model repairs that warrant further investigation.
- Computational cost: Neuron Shapley has large computational cost, while the proposed multi-arm bandit algorithm reduces that cost by orders of magnitude.This reduction enables efficient Shapley computation on widely used deep networks.
Broader Impact
Neuron Shapley is introduced as a framework for assigning responsibility to individual neurons, with an adaptive algorithm for estimating contributions. The paper also examines how training choices may alter these values.
- Broader Impact: Neuron Shapley quantifies each neuron's contribution to prediction successes and failures while accounting for interactions among neurons.The framework assigns responsibility to model elements through a contribution value.
- Broader Impact: The paper proposes an efficient adaptive algorithm for estimating Neuron Shapley scores.
- Broader Impact: Dropout training may change Shapley values because it encourages filters to become more independent.The authors test this hypothesis using a SqueezeNet model trained with filter dropout p_drop = 0.5.
- Broader Impact: The uniqueness proof maps directly to the original Shapley value proof in cooperative game theory.
- Broader Impact: The contribution scheme includes zero contribution, symmetry, and additivity properties.
B.2 Proof of uniqueness
The proof establishes that any contribution scheme satisfying zero contribution, symmetry, and additivity must equal Neuron Shapley. It does so by decomposing arbitrary performance metrics into simple subset-based metrics and applying the required properties.
- B.2 Proof of uniqueness: Any contribution scheme satisfying the three desired properties is identical to Neuron Shapley.
- B.2 Proof of uniqueness: For a simple metric w_R, neurons outside R receive zero contribution, while neurons inside R are exchangeable and receive equal contributions.
- B.2 Proof of uniqueness: Any performance metric V(.) is represented as a linear combination of the simple metrics w_R.
- B.2 Proof of uniqueness: The proof uses binomial cancellation to show that only the term T = S remains in the subset expansion.
- B.2 Proof of uniqueness: Additivity requires the contribution scheme to preserve sums of performance metrics.
- B.2 Proof of uniqueness: The final summation argument completes the uniqueness proof after pairing subsets that differ in the i-th filter.
C Implementation Details
The implementation evaluates Neuron Shapley and comparison methods across image datasets, adversarial settings, and two convolutional architectures. Experiments use specified sampling, attack, approximation, and computational parameters.
- Datasets: The experiments use ImageNet validation images for Inception-v3 and class-specific neuron Shapley evaluation.Overall importance uses 25,000 validation images, while class-specific values use 25 images per class and evaluate on the remaining 25.
- Adversarial attack parameters: Vulnerability values are computed using projected gradient descent attacks with maximum perturbation 16/255 for 30 iterations.The attack step size is 0.8/255.
- Alg 1 Parameters: Algorithm 1 uses tolerance ϵ = 0.0001, k = 100 important neurons, and empirical Bernstein bounds with δ = 0.1.Early truncation parameters are v_T = 20% for Inception-v3 and v_T = 60% for Squeezenet.
- Neuronconductance implementation: Neuron conductance scores are computed using 100 randomly selected ImageNet images for Inception-v3.For Squeezenet and the fairness experiment, the implementation uses half of the PPB dataset images.
- Computational details: The experiments use a cluster of 100 machines with 12 CPU cores each and parallelize Neuron Shapley iterations across machines.
D Early truncation
Early truncation estimates marginal filter effects by assigning zero contribution once performance falls below a threshold. Random removal trajectories show that performance breaks after removing roughly 10% of Inception-v3 filters and 20% of Squeezenet filters.
- D Early truncation: Early truncation assigns zero marginal contribution to filters appearing early in sampled removal permutations.The approximation removes filters in random order and monitors the resulting performance.
- D Early truncation: 10% of Inception-v3 filters is enough to degrade the model under random removal.The model has nearly 17,000 filters, and the figure uses 100 removal trajectories.
- D Early truncation: 20% of Squeezenet filters breaks performance under random removal.The figure reports 100 random removal trajectories for the two models.
E Model interpretation through Neuron Shapley
Figure 6 presents important filters from the Inception-V3 model, extending the paper’s visualization of filters identified by Neuron Shapley.
- Figure 6 visualizes a more complete set of important filters from the Inception-V3 model.
F Squeezenet Interpretation
Figure 7 shows important filters in a SqueezeNet model trained for gender detection on CelebA, including filters associated with interpretable facial and background properties.
- Figure 7 shows the most important filter in each layer of the SqueezeNet gender-detection model.
- The filters can be interpreted as detecting background color, skin color, face angle, and amount of hair.
G Alg. 1 Sample Efficiency
The sample-efficiency evaluation compares Monte Carlo Shapley with TMAB-Shapley using empirical Bernstein bounds and shows that the bandit-based method requires substantially fewer samples.
- The evaluation compares the original Monte Carlo Shapley algorithm with TMAB-Shapley for estimating filter importance.
- On average, TMAB-Shapley requires 8.7% of the samples required by MC-Shapley.This corresponds to around 11 times fewer forward passes on the model.
- TMAB-Shapley uses fewer samples for most filters while allocating more samples to a small group near the kth filter value.