Source-linked AI summary
Regularizing CNNs with Locally Constrained Decorrelations
Pau Rodríguez, Jordi Gonzàlez, Guillem Cucurull, Josep M. Gonfaus, Xavier Roca
TL;DR
Common regularizers can limit model capacity, while feature-decorrelation methods have offered small gains relative to their computational overhead. The paper introduces OrthoReg, which locally constrains weight decorrelation by regularizing positively correlated feature weights. The authors report reduced overfitting and improved performance across CIFAR-10, CIFAR-100, and SVHN, including with batch normalization and dropout.
Problem
Common capacity-reducing regularizers do not use all model capacity, while feature-decorrelation methods have offered limited improvement margins relative to their computational overhead.
Source: §1 · p. 1Method
Results
The authors report lower overfitting and improved performance for state-of-the-art ReLU networks on CIFAR-10, CIFAR-100, and SVHN, including with batch normalization and dropout.
Source: §4 · p. 9Takeaways & Limitations
Because it regularizes weights directly, OrthoReg is especially suitable for fully convolutional networks, where weight space is constant compared with feature-map space.
Source: §4 · p. 9The authors note that the choice of activation function could affect regularizers such as OrthoReg.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Regularization is key for deep learning since it allows training more complex models while keeping lower levels of overfitting. However, the most prevalent regularizations do not leverage all the capacity of the models since they rely on reducing the effective number of parameters. Feature decorrelation is an alternative for using the full capacity of the models but the overfitting reduction margins are too narrow given the overhead it introduces. In this paper, we show that regularizing negatively correlated features is an obstacle for effective decorrelation and present OrthoReg, a novel regularization technique that locally enforces feature orthogonality. As a result, imposing locality constraints in feature decorrelation removes interferences between negatively correlated feature weights, allowing the regularizer to reach higher decorrelation bounds, and reducing the overfitting more effectively. In particular, we show that the models regularized with OrthoReg have higher accuracy bounds even when batch normalization and dropout are present. Moreover, since our regularization is directly performed on the weights, it is especially suitable for fully convolutional neural networks, where the weight space is constant compared to the feature map space. As a result, we are able to reduce the overfitting of state-of-the-art CNNs on CIFAR-10, CIFAR-100, and SVHN.
1 INTRODUCTION
Deep networks can overfit as their architectures grow, while common regularizers either reduce model capacity or offer limited gains at computational cost. OrthoReg addresses the decorrelation bottleneck by regularizing positively correlated feature weights, and the paper reports strong results on image benchmarks.
1 INTRODUCTION
- Capacity-reducing methods such as weight decay, Dropout, and DropConnect can be effective, but do not leverage all of the models’ capacity.Weight decay reduces the effective number of parameters, while Dropout and DropConnect randomly drop activations or weights to prevent feature co-adaptation.Source: §1 · p. 1
- Feature-decorrelation regularizers preserve model capacity, but have seen limited use in large-scale image recognition because of small improvement margins and computational overhead.The passage cites slow decorrelated features and incoherent training as examples of this strategy.Source: §1 · p. 1
- Weight-space regularization is presented as computationally efficient for CNNs; Table 1 reports it is orders of magnitude faster than activation regularization.The comparison covers a 3-hidden-layer MLP and a 110-layer ResNet, with batch size 128 for the ResNet.Source: Table 1 · p. 2, §1 · p. 2
- OrthoReg regularizes positively correlated feature weights to address the performance-margin issue, and the authors report the lowest error rates on MNIST, CIFAR-10, CIFAR-100, and SVHN to their knowledge.The work first uses MNIST as a proof of concept, then evaluates wide residual networks on CIFAR-10, CIFAR-100, and SVHN.Source: §1 · p. 2
2 DEALING WITH WEIGHT REDUNDANCIES
The section links redundant feature weights to overfitting and proposes locally constrained weight decorrelation: OrthoReg leaves negatively correlated pairs unregularized. The authors report higher decorrelation bounds than global regularization, especially when layer dimensions prevent all weight vectors from being orthogonal.
2 DEALING WITH WEIGHT REDUNDANCIES
- The section identifies similar or redundant feature weights as a source of overfitting and focuses on weight correlation rather than activation independence.The authors describe weight correlation as an open problem in many network models that can be addressed without too much overhead.Source: §2 · p. 2
2.1 ORTHOGONAL WEIGHT REGULARIZATION
- Orthogonal weight regularization measures feature-detector relationships with cosine similarity and adds a squared-cosine cost whose local minima occur at orthogonal vectors.The added cost is weighted by γ in the model's global objective.Source: §2.1 · p. 3
2.2 NEGATIVE CORRELATIONS
- On a 2D toy dataset, local loss produces more uniform angles and a more stable mean nearest-neighbor angle than global loss.The global loss lets vectors in opposite quadrants influence each other; local loss reaches a stable configuration for feature weights with angle π.Source: §2.2 · p. 5
- OrthoReg reaches higher decorrelation bounds than global regularization, particularly when input dimensionality is smaller than output dimensionality.In that setting, all vectors cannot be orthogonal simultaneously, and local regularization removes interference from negatively correlated feature weights.Source: §2.2 · p. 5
3 EXPERIMENTS
Experiments evaluate OrthoReg on MNIST and residual CNNs on CIFAR-10, CIFAR-100, and SVHN, including comparisons with global regularization and existing approaches. The reported results include improved benchmark performance and lower error rates across these settings.
3.1 VERIFICATION EXPERIMENTS
- 1.45% error at γ = 1 versus 1.74% at γ = 0 on MNIST, while higher regularization rates decrease overfitting.The best error rate was obtained at the highest tested γ; λ near 10 corresponded to orthogonality and gave the best value tested.Source: Figure 4 · p. 7, §3.1 · p. 7
- Local regularization achieved better MNIST test error than global regularization, and both improved on the unregularized model; reducing negative correlations was associated with worse performance.The comparison distinguishes the local loss (eq. 7) from the global loss (eq. 2).Source: Figure 5 · p. 7, §3.1 · p. 7
- OrthoReg improved the small CNN results both without Dropout and LSUV and, by smaller increments, when those techniques were present.The authors report that the margin of improvement for the other methods is reduced when one is already present.Source: §3.1 · p. 7, Table 2 · p. 8
3.2 REGULARIZATION ON CIFAR-10 AND CIFAR-100
- 5.1% relative error improvement on CIFAR-10 and 1.5% on CIFAR-100 are reported against state-of-the-art results, with OrthoReg also beating the Wide ResNet baselines.The comparison reports single-crop results for fairness.Source: §3.2 · p. 8, Figure 6 · p. 8, Table 3 · p. 9
3.3 REGULARIZATION ON SVHN
- 1.54% vs 1.64% error rate on SVHN is reported for the OrthoReg model and comparison, respectively; the paper describes 1.54% as the lowest reported value to its knowledge.The experiment uses a depth-28, width-4 Wide ResNet with dropout.Source: §3.3 · pp. 8–9, Table 4 · p. 9
4 DISCUSSION
Locality constraints let feature-decorrelation regularization reduce overfitting more effectively, including alongside batch normalization and dropout. The weight-based approach is suited to fully convolutional networks, and improves performance on the reported CIFAR and SVHN models.
4 DISCUSSION
- Locality constraints remove interference from negatively correlated feature weights, enabling higher decorrelation bounds and more effective overfitting reduction.Feature-decorrelation regularization can still reduce overfitting alongside other regularization methods, but its capacity is limited, especially when feature detectors outnumber input dimensions.Source: §4 · p. 9
- Constrained regularization lowers overfitting even when batch normalization and dropout are present.It reduces overfitting in 110-layer ResNets and wide ResNets on CIFAR-10, CIFAR-100, and SVHN, improving their performance.Source: §4 · p. 9
- Weight-based regularization is especially suitable for fully convolutional networks because weight space remains constant relative to feature-map space.The discussion notes that the activation function could affect regularizers of this kind, and calls for further study of asymmetrical activations.Source: §4 · p. 9