On the importance of single directions for generalization

Ari S. Morcos; David G. T. Barrett; Neil C. Rabinowitz; Matthew Botvinick

Source-linked AI summary

On the importance of single directions for generalization

Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, Matthew Botvinick

arXiv:1803.06959v4Mar 19, 2018stat.MLcs.AIcs.LGcs.NE

TL;DR

Deep networks can memorize large datasets yet generalize well, and the learned-solution differences behind this remain unclear. The paper measures reliance on single directions through activation-space ablations and related selectivity analyses, finding that lower reliance predicts better generalization across multiple settings. It also identifies batch normalization as implicitly discouraging such reliance, while class selectivity is a poor predictor of task importance.

Problem
It remains unclear why some deep-network solutions generalize while others do not, and the importance of highlighted single-direction tuning properties has not been evaluated.
Method
The paper uses activation-space ablation analyses and class-selectivity measurements to relate single-direction reliance and selectivity to network generalization and task importance.
Results
Reliance on single directions predicts generalization across corrupted and uncorrupted datasets, network ensembles, hyperparameters, and training, while batch normalization implicitly discourages such reliance.
Takeaways & Limitations
Networks that generalize well appear less dependent on individual directions, and individually class-selective units may not be necessary for strong performance.
Takeaways & Limitations
The viability of single-direction reliance as a predictor remains to be evaluated on more complicated datasets, and feature selectivity was not measured.

Abstract

from arXiv · show

Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance. However, the differences between the learned solutions of networks which generalize and those which do not remain unclear. Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated. Here, we connect these lines of inquiry to demonstrate that a network's reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyperparameters, and over the course of training. While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units. Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance.

1 INTRODUCTION

Deep networks can memorize very large datasets while often generalizing well, leaving unclear why some learned solutions generalize and others do not. This work links that question to reliance on single directions and shows that reliance differs systematically across networks and training conditions.

Deep neural networks can memorize extremely large datasets while achieving low generalization error on tasks including image classification and language translation.
Prior explanations have connected generalization to minimum flatness, PAC-Bayes bounds, weight information, and stochastic gradient descent, but sharp minima can also generalize.
Single directions are individual units, feature maps, or linear combinations whose reliance differs between memorizing networks and networks that do not memorize.
Networks that memorize training data are substantially more dependent on single directions, including among identical-topology networks trained on identical data.
As networks begin to overfit, their reliance on single directions increases, suggesting this metric could signal when to stop training.
Batch normalization discourages single-direction reliance partly by decreasing individual feature-map class selectivity, while class selectivity poorly predicts output importance.

2 APPROACH

The paper uses activation-space perturbations across several neural-network architectures and datasets to measure reliance on single directions and compare it with class selectivity. Cumulative ablations reveal how quickly performance falls as directions are removed, while noise tests probe random directions.

Models and datasets: The study examines an MNIST MLP, a CIFAR-10 convolutional network, and an ImageNet ResNet, using ReLU nonlinearities and batch normalization for convolutional networks unless noted otherwise.
Models and datasets: Datasets with different fractions of randomized labels create varying degrees of memorization while preserving the label distribution.
Perturbation analyses: Ablation measures a direction’s importance by clamping its activation to a fixed value and observing the resulting performance degradation in activation space.
Perturbation analyses: Cumulative ablation curves plot accuracy against the number of removed directions; faster accuracy decline indicates greater reliance on low-dimensional activation subspaces.
Perturbation analyses: All ablation experiments clamp activity to zero because clamping to empirical mean activation was more damaging to performance.
Perturbation analyses: Gaussian noise with progressively increasing, activation-variance-normalized variance tests reliance on random rather than coordinate-aligned single directions.
Class selectivity: Class selectivity is computed from class-conditional mean activity using the highest class mean and the mean across all other classes.
Class selectivity: The selectivity index ranges from 0 for identical average activity across classes to 1 for activity restricted to one class.

3 EXPERIMENTS

Experiments show that reliance on single directions tracks generalization across label corruption, independently trained networks, training time, and hyperparameter settings. Regularizers affect this reliance differently, while class selectivity is not a reliable proxy for direction importance.

3.1 GENERALIZATION: Networks trained on corrupted labels were consistently more sensitive to random noise than networks trained on true labels.Noise was added along random bases after scaling each unit’s variance by its empirical activation variance.
3.1 GENERALIZATION: Among 200 identical-topology networks trained on unmodified CIFAR-10, the best-generalizing models were more robust to single-direction ablations than the worst models.Normalized ablation-curve area also varied with generalization error, including a negative within-cluster correlation.
3.2 RELIANCE ON SINGLE DIRECTIONS AS A SIGNAL FOR MODEL SELECTION: AUC began dropping when train and test loss diverged, while AUC and test loss were negatively correlated (Spearman’s correlation: -0.728).This result came from an MNIST MLP tracked over training.
3.2 RELIANCE ON SINGLE DIRECTIONS AS A SIGNAL FOR MODEL SELECTION: AUC and test accuracy were highly correlated across 192 CIFAR-10 models (Spearman’s correlation: 0.914), selecting a top-10 setting 98% of the time.The best AUC-selected model differed from the optimal model by 1 ± 1.1% test accuracy on average.
3.3 RELATIONSHIP TO DROPOUT AND BATCH NORMALIZATION: Dropout improved robustness to ablations only up to its training fraction and did not prevent over-reliance on single directions beyond that point.With corrupted labels, cumulative ablation curves began dropping soon past the dropout fraction used in training.
3.3 RELATIONSHIP TO DROPOUT AND BATCH NORMALIZATION: Batch normalization substantially increased robustness to cumulative ablations and decreased class selectivity while increasing mutual information in feature maps.These comparisons used CIFAR-10 convolutional networks trained with versus without batch normalization.
3.4 RELATIONSHIP BETWEEN CLASS SELECTIVITY AND IMPORTANCE: Class selectivity was a poor predictor of task importance: highly selective units or feature maps were often no more impactful, and sometimes less impactful, when ablated.The MNIST correlation was 0.095, while CIFAR-10 showed a negative correlation of -0.428, driven largely by early layers.

4 RELATED WORK

Prior work links generalization to minima geometry, weight information, optimization, and capacity, while this paper relates it to reliance on single directions and class selectivity.

Memorizing networks are more reliant on single directions than networks that generalize, answering whether their learned solutions differ empirically.
Flat-minimum theories are consistent with the finding that perturbations along single directions should have little effect on network outputs.
Dropout can discourage memorization but does not discourage reliance on single directions beyond the dropout probability.
Class selectivity is a poor predictor of unit importance, consistent with population coding by neurons having low individual selectivity.
Increasing numbers of concept-selective units may correlate with performance because networks differ in total units and depth, not necessarily selectivity itself.

5 DISCUSSION AND FUTURE WORK

The paper finds that generalization relates to reliance on single directions and that batch normalization discourages this reliance, while identifying extensions and scope limits.

Generalization capability relates to reliance on single directions in networks trained on corrupted or uncorrupted data and throughout training.
Batch normalization seems to implicitly discourage reliance on single directions.
Single-direction reliance could support regularizer design and computationally cheap signals for early stopping or hyperparameter selection.
The metric has been shown viable on simple datasets, but its viability on more complicated datasets remains to be evaluated.
The relationship should be tested under distribution-shifted generalization, where test data come from a unique but overlapping distribution.
Class selectivity is largely uncorrelated with output importance, and its decrease under batch normalization may make highly selective units harmful or misleading analysis targets.

A.1 COMPARISON OF ABLATION METHODS

The ablation analysis compares fixing directions at zero with fixing them at their empirical training-set mean, because the chosen fixed value affects measured performance.

Ablation removes a direction’s input dependence by fixing its activation, but an unlikely fixed value can drastically damage performance.
The study compares ablation to zero with ablation to the empirical feature-map mean.

A.2 TRAINING DETAILS

The experiments span MNIST MLPs, CIFAR-10 convolutional networks, and ImageNet ResNets, with selectivity increasing with depth and filter-weight L1-norm largely unrelated to selectivity.

MNIST MLPs: MNIST MLPs used layers of 128, 512, 2048, and 2048 units; training lasted 640 epochs, or 5000 epochs for dropout networks.
CIFAR-10 ConvNets: CIFAR-10 convolutional networks were trained for 100 epochs with 3x3 kernels and layer widths from 64 to 512.
ImageNet ResNet: ImageNet experiments used 50-layer ResNets trained for 200,000 steps with 32 workers and batch size 32.
Selectivity analyses: Class selectivity increases with depth for CIFAR-10 and ImageNet networks.
Selectivity analyses: Class selectivity is largely unrelated to filter-weight L1-norm for CIFAR-10 and ImageNet.
ImageNet ResNet: Data augmentation was omitted for partially corrupted-label ImageNet training to preserve the ability to memorize.

A.3 DEPTH-DEPENDENCE OF CLASS SELECTIVITY

Class selectivity increased with network depth in both CIFAR-10 and ImageNet models, consistent with prior findings that deeper representations become more class- or concept-selective.

Class selectivity increased as a function of depth in networks trained on both CIFAR-10 and ImageNet.The result is consistent with prior findings that concept selectivity and linear decodability of class information increase with depth.

A.4 RELATIONSHIP BETWEEN CLASS SELECTIVITY AND THE FILTER WEIGHT L1-NORM

The analysis clarifies that class selectivity does not predict unit importance: it is not correlated with filter-weight L1-norm, despite that norm predicting feature-map removal impact.

Class selectivity is not a good predictor of importance, although some directions can still be more important than others.The result rules out selectivity as a predictor, not the existence or predictability of importance differences.
Class selectivity and filter-weight L1-norm were not correlated in the ImageNet network.This comparison tests whether selectivity predicts importance through a metric associated with filter-removal impact.

A.5 RELATIONSHIP BETWEEN MUTUAL INFORMATION AND IMPORTANCE

Mutual information was generally a poor predictor of unit importance across the tested MNIST, CIFAR-10, and ImageNet models, with less consistent relationships in some layers.

Mutual information was generally a poor predictor of unit importance across the tested networks.The analysis covered an MNIST MLP, a CIFAR-10 convolutional network, and an ImageNet ResNet.
Relationships between mutual information and importance were less consistent, with apparent associations in some early and late CIFAR-10 layers.The figure includes layer-specific regression lines for CIFAR-10 and ImageNet analyses.