Source-linked AI summary
Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
Nitin Bansal, Xiaohan Chen, Zhangyang Wang
TL;DR
Deep CNN training is difficult, and the paper asks how orthogonality can be enforced more effectively for non-square weights. It introduces plug-and-play regularizers based on Frobenius norms, Mutual Coherence, and RIP, finding consistent accuracy gains alongside faster and more stable convergence across CNN benchmarks.
Problem
The paper addresses how to enforce near-orthogonality in deep CNN weights more effectively, especially because non-square matrices lack a sole optimal enforcement strategy.
Method
The paper develops three plug-and-play orthogonality regularizers using Frobenius norms, Mutual Coherence, and Restricted Isometry Property tools.
Results
Across evaluated CNN architectures and datasets, the regularizers consistently improve accuracy and yield faster, more stable convergence, including a 2.31% CIFAR-100 top-1 accuracy gain for WideResNet.
Takeaways & Limitations
Orthogonality regularizations are presented as general, easy-to-use tools for training deeper CNNs, with stronger regularizer designs producing larger gains.
Takeaways & Limitations
Soft Orthogonality cannot make W^T W close to identity for overcomplete matrices, while the Mutual Coherence approximation can experience column-norm deviations during training.
Abstract
from arXiv · showhide
This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available: https://github.com/nbansal90/Can-we-Gain-More-from-Orthogonality.
1 Introduction
The paper studies how to enforce orthogonality in deep CNN training more effectively, motivated by training difficulties and the potential of orthogonality to stabilize activations and improve optimization.
- Deep CNN training is difficult because of issues including vanishing or exploding gradients, feature statistic shifts, and proliferating saddle points.
- Orthogonality can preserve activation energy, stabilize layer-wise activation distributions, and make optimization more efficient.
- Non-square weight matrices lack a single optimal way to evaluate and enforce orthogonality, motivating more sophisticated regularizers.
- The paper introduces three plug-and-play orthogonality regularizers based on Frobenius norms, Mutual Coherence, and Restricted Isometry Property tools.
- The proposed regularizations improve accuracy and produce faster, more stable convergence across ResNet, ResNeXt, and WideResNet experiments.
2 Related Work
Prior work addresses orthogonality through initialization, hard constraints, soft penalties, and spectral approaches, but these methods differ in computational cost and treatment of non-square CNN weights.
- Orthogonal initialization and normalization-related methods aim to stabilize layer outputs and improve optimization in CNNs and other deep models.
- Hard orthogonality constraints require repeated singular value decompositions during training, making them expensive for high-dimensional matrices.
- Non-square CNN weight matrices may be too thin or fat to satisfy exact Stiefel-manifold constraints, requiring special treatments such as group-wise orthogonalization.
- Soft Frobenius norm regularization is differentiable and avoids SVD, but it is only a rough approximation for fat matrices.
- Spectral regularization connects orthogonality with singular-value behavior, robustness, Lipschitz constants, and Hessian-related analyses.
3 Deriving New Orthogonality Regularizations
The paper derives soft, mutual-coherence, and SRIP regularizers to handle both undercomplete and overcomplete weights while retaining practical differentiability and efficiency.
- 3.1 Baseline: Soft Orthogonality Regularization: Soft Orthogonality penalizes the Frobenius distance between W^T W and I, and its gradient can be appended directly to the current weight gradient.
- 3.1 Baseline: Soft Orthogonality Regularization: Soft Orthogonality is limited for overcomplete W because W^T W has rank at most m and cannot be close to the n-dimensional identity.
- 3.1 Baseline: Soft Orthogonality Regularization: Combining ordinary ℓ2 weight decay with Soft Orthogonality throughout training hurts final accuracy, motivating a scheme-change approach.
- 3.2 Double Soft Orthogonality Regularization: Double Soft Orthogonality adds both W^T W and WW^T penalties so either columns or rows can remain orthogonal depending on matrix shape.
- 3.3 Mutual Coherence Regularization: Mutual Coherence measures the highest correlation between any two columns of W, so lower values indicate more orthogonal columns.
- 3.3 Mutual Coherence Regularization: The proposed Mutual Coherence approximation often encourages near-unit column norms without explicitly normalizing them, though deviations can occur during training.
- 3.4 Spectral Restricted Isometry Property Regularization: RIP requires every sufficiently small set of columns to behave like an orthogonal system and is defined for both undercomplete and overcomplete matrices.
- 3.4 Spectral Restricted Isometry Property Regularization: SRIP minimizes the spectral norm of W^T W − I, requiring all singular values to be close to one and thereby encouraging well-conditioned weights.
4 Experiments on Benchmarks
Experiments across state-of-the-art CNNs and benchmark datasets show that orthogonality regularization improves accuracy and early training behavior, with SRIP generally the strongest option. A coefficient-switching scheme helps preserve final accuracy while retaining early-stage training benefits.
- Experimental setup: Experiments retain the original models’ preprocessing, augmentation, training splits, hyperparameters, and architectural details for fair comparisons.The benchmark models include ResNet, Wide ResNet, and ResNeXt, evaluated primarily on CIFAR-10 and CIFAR-100.
- Regularization scheme: Orthogonality regularization accelerates and stabilizes early training, but fully replacing ℓ2 weight decay can reduce final accuracy.The authors therefore investigate switching between regularization schemes across training stages.
- CIFAR results: SRIP is the strongest regularizer in almost all CIFAR experiments, including a 2.31% top-1 error reduction for Wide ResNet-28-10.SO is often the next-best baseline, MC is generally weaker, and DSO can underperform the original model.
- CIFAR results: Removing scheme change increases Wide ResNet 28-10 top-1 error by 0.33% on CIFAR-10 and 0.90% on CIFAR-100, while remaining better than unregularized models.Other regularizers perform even worse without the scheme change.
- Comparisons with other methods: SRIP outperforms OMDSM on Wide ResNet 28-10, reaching 3.60% versus 3.73% error on CIFAR-10 and 18.19% versus 18.76% on CIFAR-100.This compares soft regularization with a hard orthogonality-constraint approach.
- Comparisons with other methods: SRIP achieves 4.28% top-1 error on CIFAR-10 for the compared Wide ResNet variant, versus 5.68% with Jacobian norm regularization.Under the reported comparison, SRIP also outperforms the original model’s 6.66% error rate.
5 Conclusion
The paper presents efficient orthogonality regularizers for deep CNNs and reports better accuracy, more stable training, and smoother convergence across state-of-the-art models.
- The proposed regularizers improve accuracy, stabilize training curves, and smooth convergence across several state-of-the-art convolutional CNNs.The paper reports that SRIP almost consistently outperforms the other regularizers.