Source-linked AI summary
Ablation Studies in Artificial Neural Networks
Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, Tobias Meisen
TL;DR
The paper asks whether ablation studies can reveal how knowledge is organized in increasingly complex artificial neural networks. It applies controlled ablations to two different ANNs and finds selective, partly redundant representations, structural indicators of unit importance, and substantial recovery after retraining.
Problem
As ANNs and their tasks grow more complex, the paper asks how trained networks represent knowledge and whether neuroscience-inspired ablations can investigate that organization.
Method
The study performs single, pairwise, and filter ablations in two fundamentally different ANNs, analyzes performance and weight structure, and applies recovery training to damaged networks.
Results
Features related to local and global data structure are selectively represented; some representations are redundant, and most original performance can be recovered after one training epoch even after ablating about 80% of filters.
Takeaways & Limitations
Ablation studies are a feasible way to investigate knowledge representations and network robustness to structural damage.
Abstract
from arXiv · showhide
Ablation studies have been widely used in the field of neuroscience to tackle complex biological systems such as the extensively studied Drosophila central nervous system, the vertebrate brain and more interestingly and most delicately, the human brain. In the past, these kinds of studies were utilized to uncover structure and organization in the brain, i.e. a mapping of features inherent to external stimuli onto different areas of the neocortex. considering the growth in size and complexity of state-of-the-art artificial neural networks (ANNs) and the corresponding growth in complexity of the tasks that are tackled by these networks, the question arises whether ablation studies may be used to investigate these networks for a similar organization of their inner representations. In this paper, we address this question and performed two ablation studies in two fundamentally different ANNs to investigate their inner representations of two well-known benchmark datasets from the computer vision domain. We found that features distinct to the local and global structure of the data are selectively represented in specific parts of the network. Furthermore, some of these representations are redundant, awarding the network a certain robustness to structural damages. We further determined the importance of specific parts of the network for the classification task solely based on the weight structure of single units. Finally, we examined the ability of damaged networks to recover from the consequences of ablations by means of recovery training. We argue that ablations studies are a feasible method to investigate knowledge representations in ANNs and are especially helpful to examine a networks robustness to structural damages, a feature of ANNs that will become increasingly important for future safety-critical applications.
1 Introduction
As deep neural networks grow more complex, the paper asks whether neuroscience-inspired ablation studies can reveal how knowledge is represented and organized within them. It applies this approach to two neural networks, finding class- and structure-specific effects, variable damage sensitivity, and recoverability through training.
- Motivation: The paper motivates neuroscience-inspired ablation studies as a way to investigate knowledge representation in increasingly large and complex DNNs.Earlier work emphasized benchmark performance and speed rather than how trained networks represent knowledge.
- Approach: Two ablation studies examine inner representations in fundamentally different ANNs trained on benchmark computer-vision datasets.The studies include single and pairwise unit ablations in an MLP and larger-scale network ablations.
- Findings: Single-unit ablations identify units that are universally important, selectively important for specific classes, or not important for classification.Unit importance also correlates with how much incoming weight distributions change during training.
- Findings: Ablation effects generally strengthen with larger damaged network portions, but their magnitude varies across network depths and classes.Some layers are universally more important than others, while class-specific importance differs across classes.
- Findings: Ablations can negatively affect overall classification while improving performance for specific classes, suggesting that controlled structural manipulation may alter trained performance.The paper presents this as a possibility for exceeding the local optimum reached during training.
2 Related Work
Pruning applies the ablation idea to reduce neural-network size and computational cost while retaining as much original performance as possible.
- Pruning: Pruning removes trained parameters that contribute little or nothing to network output, reducing model size and computational cost.The passage describes pruning as a related use of removing trainable weights from trained networks.
3 Methods
The study compares ablation effects in a shallow MNIST MLP and a pretrained VGG-19, using unit- and filter-level interventions across network locations. It also evaluates recovery training after damaging important convolutional layers.
- Architectures: The study investigates two fundamentally different architectures trained on datasets with varying complexity.The architectures are a small shallow MLP and a state-of-the-art convolutional network.
- MLP setup: The MNIST MLP maps 28x28-pixel inputs through hidden layers of 20 and 10 ReLU units to 10 softmax output classes.It was trained on 60,000 images for 100 epochs and reached 94.64% test accuracy.
- VGG-19 setup: The VGG-19 experiment uses a batch-normalized network with 16 convolutional and 3 fully connected learnable layers, enabling depth-resolved ablation analysis.The network was pretrained on ImageNet for object-recognition tasks.
- Ablation procedure: Filter ablations zero all incoming weights and biases, eliminating the affected filter’s activation; filter similarity is computed from normalized-weight Euclidean distance.The same intervention principle is applied to remove functional contributions from selected filters.
- Recovery training: Recovery training retrains damaged networks while freezing layers above the ablated layer, forcing adaptation to the structural change.The recovery experiments target the two layers identified as most important for classification.
- Evaluation: t-SNE visualization provides a basis for evaluating whether global and local structure in the MNIST data is represented in an organized manner.The visualization embeds the complete 10,000-digit test set from 784 dimensions into two dimensions.
4 Results
Single-unit and filter ablations revealed selective, redundant, and depth-dependent representations in both the MLP and VGG-19, while recovery training restored most lost classification ability.
- Single-unit ablations: 44.5%p was the largest overall-accuracy drop from ablating one MLP unit, with classes 1, 4, 7, and 9 each losing more than 80%p class-specific accuracy.Ablating unit 12 had smaller effects on classes 6 and 8 and less than 10%p effects on the remaining classes.
- Single-unit ablations: 11.6%p overall-accuracy loss from ablating MLP unit 19 disproportionately affected class 1, indicating selective single-class feature representation without exceptional overall importance.The unit was interpretable by its class selectivity, but its ablation was not more damaging overall than other units.
- Single-unit ablations: 1.4%p overall-accuracy loss from ablating unit 6 marked it as a minor contributor; four of 20 first-layer units were identified as pruning candidates.The candidate units were 6, 11, 13, and 18.
- Single-unit ablations: 14.6%p overall-accuracy loss after ablating unit 20 reflected features for subtle, smoothly changing characteristics spanning classes 1, 6, and 9.Misclassified digits within a class clustered near one another in the t-SNE visualization.
- Single-unit ablations: 25.4%p overall-accuracy loss from ablating unit 3 coincided with a 5.7% increase in class-5 accuracy, demonstrating class-specific trade-offs in ablation effects.The increase was the strongest positive class-specific effect among first-layer units.
- Weight structure and representation: Incoming-weight distribution changes during training indicated unit importance, and correlation analyses across 20 networks supported this relationship as a possible pruning signal.Network-specific differences also showed that class selectivity was not universal across independently initialized networks.
- VGG-19 ablations: In VGG-19, larger ablated portions generally caused stronger performance effects, but importance varied by depth and class; some filters even increased top-1 accuracy.Layers 33 and 46 showed significantly higher top-1 and top-5 drops, while class-specific importance varied across layers.
- Recovery training: After 25% filter ablations in VGG-19, most lost classification ability returned after one recovery epoch, and recovery remained possible despite initial accuracy drops of up to 30%p.Iterative ablations ultimately removed approximately 80% of filters in either layer, yet the network recovered almost completely.
5 Conclusion and Future Work
The ablation studies show that units and layers encode class-related features with varying importance, redundancy, and recoverability. They support ablations as a method for studying ANN representations and robustness to structural damage.
- Single-unit ablations revealed universally important units, class-selective units, and units that were not important for classification.
- Incoming-weight distributions changed more during training for units that were more important to classification, enabling importance estimation without functional testing.
- t-SNE visualizations indicated that individual units represented features associated with both global and local dataset structure in localized network areas.
- Redundant feature representations made networks robust to structural damage, with pairwise ablations producing effects exceeding the combined effects of corresponding single ablations.
- VGG-19 ablations showed that deeper layers 33 and 46 were significantly more important than other layers, while feature importance varied across classes.
- Most original classification performance was recovered after one training epoch, although recovery became harder as more filters were ablated; even ∼80% ablation in layers 33 and 46 remained within 4%p.
A Appendix
The appendix illustrates how selected first-hidden-layer unit ablations affect overall and class-specific MLP accuracy. The examples contrast broadly represented, low-importance, redundant, and interaction-dependent units.
- Ablating unit 16 affected features corresponding to many different classes, as shown through overall accuracy, class-specific accuracy, and t-SNE visualization.
- Ablating units 6, 11, 13, and 18 produced little classification impact, making them top pruning candidates.
- Pairwise ablation of units 12 and 19 impaired class-six performance substantially more than either single-unit ablation, indicating redundant representation.
- For class five, the pairwise ablation of units 5 and 10 produced a larger positive effect than the sum of their single-ablation effects.