Source-linked AI summary
Reducing Overfitting in Deep Networks by Decorrelating Representations
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, Dhruv Batra
TL;DR
Deep networks need methods to reduce overfitting without large training sets. This paper proposes DeCov, which minimizes hidden-activation cross-covariance to encourage non-redundant representations, and reports reduced overfitting across varied settings while generally preserving or improving generalization. Fine-tuning results and some combinations with Dropout indicate that its benefits are not uniformly additive.
Problem
The paper addresses overfitting in deep networks and the prior lack of supervised-learning applications of representation redundancy minimization.
Method
DeCov regularizes one or more hidden layers by explicitly penalizing cross-covariance among their activations without additional supervision.
Results
Across varied datasets and architectures, DeCov reduces overfitting and performs better than unregularized models, while competing well with Dropout.
Takeaways & Limitations
DeCov is a useful alternative regularizer, although combining it with Dropout almost always works best and it does not completely replace Dropout.
Takeaways & Limitations
Fine-tuning pretrained networks with DeCov left performance unchanged or slightly decreased, and some DeCov–Dropout combinations also slightly reduced performance.
Abstract
from arXiv · showhide
One major challenge in training Deep Neural Networks is preventing overfitting. Many techniques such as data augmentation and novel regularizers such as Dropout have been proposed to prevent overfitting without requiring a massive amount of training data. In this work, we propose a new regularizer called DeCov which leads to significantly reduced overfitting (as indicated by the difference between train and val performance), and better generalization. Our regularizer encourages diverse or non-redundant representations in Deep Neural Networks by minimizing the cross-covariance of hidden activations. This simple intuition has been explored in a number of past works but surprisingly has never been applied as a regularizer in supervised learning. Experiments across a range of datasets and network architectures show that this loss always reduces overfitting while almost always maintaining or increasing generalization performance and often improving performance over Dropout.
1 INTRODUCTION
The paper links hidden-activation correlation with overfitting and proposes DeCov, a loss that directly encourages decorrelated representations in supervised deep networks.
- Motivation: Dropout and increased training data reduce both overfitting and hidden activation cross-covariance.Figure 1 examines validation error and cross-covariance as functions of training-data amount and Dropout.
- Research questions: The paper asks whether directly reducing correlation between hidden units can produce representations that generalize better.DeCov is introduced to study the relationship between decorrelation and generalization.
- Approach: DeCov applies an unsupervised decorrelation loss to activations in one or more chosen hidden layers without requiring additional supervision.The approach defines covariances among hidden activations within a minibatch.
- Approach: The loss penalizes off-diagonal covariance while subtracting diagonal variance, avoiding a requirement that activation dynamic ranges become small.Subtracting the diagonal made little difference in small networks but increased stability in larger networks.
- Approach: DeCov is data-dependent, affects parameters up to its application layer, and is typically applied to deep fully connected layers.Unlike L1 or L2, its value depends on input data rather than only on the weight vector.
- Interpretation: Its gradient suppresses activations that are correlated with important features, while specialized features contribute little to gradients of other specialized features.Covariance measures linear redundancy, and importance weights the effect of correlated features.
3 RELATED WORK
Prior redundancy-minimization research mainly addressed unsupervised feature learning, whereas this paper applies the idea as supervised-learning regularization with modern networks and datasets.
- Redundancy-based representations: Earlier work studied non-redundant representations through information-theoretic, sparse-coding, and related unsupervised feature-learning objectives.These approaches formalized low redundancy in several ways.
- Research gap: Prior studies did not experiment with supervised models and generally evaluated small networks under older training settings.The paper contrasts those settings with modern architectures, ReLUs, Dropout, SGD, and batch normalization.
- This paper: This paper introduces redundancy minimization as regularization for supervised objectives and evaluates it with end-to-end SGD on modern datasets.The experiments span contemporary challenges rather than only small-network feature learning.
- Correlation losses: Deep CCA and CorrNets maximize correlation between aligned views or modalities, unlike DeCov, which minimizes hidden cross-covariance.These methods learn common-space features from multiple views or modalities.
4 EXPERIMENTS
Experiments on synthetic MNIST, autoencoding, CIFAR10/100, and ImageNet evaluate whether DeCov reduces redundant representations, overfitting, and train-test gaps. Across these settings, DeCov generally improves generalization, with effects varying when combined with Dropout or model capacity.
- MNIST dual-modality experiment: DeCov applies to shared fully connected representations in the MNIST dual-digit task, where training-time bias makes the right digit weakly informed by the left.The model uses a fully connected layer with 500 hidden units shared between two softmax outputs.
- MNIST results: ∼0.6% higher right-digit test accuracy is obtained when switching from Dropout-alone to DeCov-alone, compared with ∼0.3% for left digits.The right classifier also has a larger train-test gap under the injected bias, and DeCov improves generalization for both classifiers.
- Autoencoder comparison: DeCov produces lower autoencoder mean-square reconstruction error than the other compared models and qualitatively different representations from Dropout.The visualizations compare models with no regularizer, DeCov, and Dropout.
- CIFAR10 results: ∼4.5% higher CIFAR10 test accuracy than no regularization is reported, while the train-validation gap falls by ∼15% without Dropout and ∼16% with Dropout.Using both regularizers further improves the generalization gap, but its absolute test-performance improvement is not statistically significant.
- CIFAR100 results: On CIFAR100, combining DeCov and Dropout achieves the highest test accuracy and the smallest train-test gap, approximately 34% smaller than using neither regularizer.Dropout alone has higher test performance than DeCov alone, while DeCov alone yields the smaller gap.
- ImageNet results: In AlexNet, explicitly minimizing DeCov produces much lower DeCov losses and coincides with significantly reduced overfitting; higher image resolution also increases capacity and overfitting.Dropout usually has the best validation accuracy across resolutions, while DeCov reduces redundancy and overfitting-related gaps.
5 DISCUSSION AND CONCLUSION
The experiments indicate that DeCov reduces overfitting and acts as a strong regularizer, while remaining an alternative to rather than a complete replacement for Dropout.
- Combining DeCov and Dropout almost always works best, although the combination can sometimes decrease performance slightly.
- DeCov reduces overfitting, measured by the gap between train and test performance.
- Performance with DeCov is always better than performance without DeCov or Dropout.
- DeCov is not supported as a complete replacement for Dropout, but is a useful alternative in some scenarios.
- DeCov explicitly penalizes covariance between same-layer activations in an unsupervised fashion and competes well with Dropout across datasets and architectures.
A DETAILS OF THE BIAS IN THE MNIST EXPERIMENT
The MNIST bias experiment constructs paired-digit examples with specified left-right sampling probabilities, then compares conditional entropies to characterize the induced asymmetry.
- The experiment samples horizontally concatenated MNIST digit pairs, with the left and right digits assigned biased joint probabilities.
- The induced distribution creates more uncertainty about left digits given right digits than about right digits given left digits.
- The analysis computes conditional entropies H(l|r) and H(r|l) to quantify the directional bias.
- The derived conditional probabilities assign 0.05 when r is 0–4 and 0.15 when r is 5–9.
- The joint distribution includes probabilities 2/15 and 3/15 for the two mixed digit-group cases, and 1/15 when both groups are 5–9.
- The entropy calculation uses the convention 0 log 0 = 0.