Source-linked AI summary
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, Bo Li
TL;DR
The paper addresses the limited study of rectangular orthogonal filters and their dependent layerwise optimization in feed-forward networks. It formulates OMDSM, proposes orthogonal weight normalization and an orthogonal linear module, and reports improved benchmark performance, including reduced wide-residual-network CIFAR-100 test error from 20.04% to 18.61%.
Problem
Rectangular orthogonal filters and their dependent optimization across layers are insufficiently studied in deep feed-forward networks, despite orthogonality's established use in RNNs.
Method
The paper solves OMDSM with proxy-parameter-based orthogonal weight normalization and uses the resulting orthogonal linear module as a standard linear-module alternative.
Results
18.61% test error was obtained on CIFAR-100 with a wide residual network, compared with 20.04% in prior reported results, while improvements were also shown across CNN architectures and datasets.
Takeaways & Limitations
Orthogonal linear modules can improve optimization efficiency and generalization ability when substituted into deep networks without revising experimental protocols.
Takeaways & Limitations
The paper identifies distortion minimization as essential for stability because an alternative transformation without it suffers instability and fails to converge; it also describes robustness to adversarial examples as future potential.
Abstract
from arXiv · showhide
Orthogonal matrix has shown advantages in training Recurrent Neural Networks (RNNs), but such matrix is limited to be square for the hidden-to-hidden transformation in RNNs. In this paper, we generalize such square orthogonal matrix to orthogonal rectangular matrix and formulating this problem in feed-forward Neural Networks (FNNs) as Optimization over Multiple Dependent Stiefel Manifolds (OMDSM). We show that the rectangular orthogonal matrix can stabilize the distribution of network activations and regularize FNNs. We also propose a novel orthogonal weight normalization method to solve OMDSM. Particularly, it constructs orthogonal transformation over proxy parameters to ensure the weight matrix is orthogonal and back-propagates gradient information through the transformation during training. To guarantee stability, we minimize the distortions between proxy parameters and canonical weights over all tractable orthogonal transformations. In addition, we design an orthogonal linear module (OLM) to learn orthogonal filter banks in practice, which can be used as an alternative to standard linear module. Extensive experiments demonstrate that by simply substituting OLM for standard linear module without revising any experimental protocols, our method largely improves the performance of the state-of-the-art networks, including Inception and residual networks on CIFAR and ImageNet datasets. In particular, we have reduced the test error of wide residual network on CIFAR-100 from 20.04% to 18.61% with such simple substitution. Our code is available online for result reproduction.
1 Introduction
The paper extends orthogonal transformations from square RNN matrices to rectangular filters in feed-forward networks, formulates their learning as OMDSM, and introduces a stable normalization method with an orthogonal linear module. Experiments show improved performance across architectures and datasets without changing experimental protocols.
- Properties of orthogonal filters: Orthogonality can stabilize activation distributions and regularize feed-forward networks through energy preservation and constrained weights.The paper links energy preservation to preventing activation energy from being amplified across layers.
- Problem formulation: Orthogonal rectangular filters are formulated as optimization over multiple dependent Stiefel manifolds (OMDSM) for deep feed-forward networks.The formulation reflects layerwise orthogonality constraints and dependencies between weights in successive layers.
- Optimization challenge: Riemannian optimization methods for single or independent manifolds show convergence instability or inferior performance on OMDSM.The paper motivates a method specifically designed for dependent manifolds in deep feed-forward networks.
- Proposed method: Orthogonal weight normalization maps proxy parameters to orthogonal canonical weights, back-propagates gradients through that transformation, and minimizes parameter distortions.This design is intended to provide stable and efficient convergence when solving OMDSM.
- Practical module: The orthogonal linear module provides an orthogonal alternative to standard linear modules and can cooperate with batch normalization, Adam, and Dropout.It is designed for practical substitution within existing deep-learning architectures.
- Empirical results: 3.73% and 18.61% test error were achieved on CIFAR-10 and CIFAR-100 with wide residual networks, versus 4.17% and 20.04% in prior reported results.The substitution was made without revising experimental protocols, and improvements were also reported for BN-Inception and residual networks on ImageNet.
2 Optimization over Multiple Dependent Stiefel Manifolds
The paper formulates learning orthogonal rectangular weights in deep feed-forward networks as OMDSM, where layer-wise Stiefel constraints are dependent through preceding layers. It analyzes activation stabilization and network regularization as properties of orthogonal weights.
- Formulation: A feed-forward network is modeled as layered linear transformations with learnable weights and biases followed by element-wise nonlinearities.Training minimizes expected loss between the desired output and the network prediction.
- Formulation: The target is an orthogonal rectangular weight matrix W_l in each layer, with orthogonal filters represented by its rows.The constraint is W_l(W_l)^T = I.
- Formulation: OMDSM contains multiple embedded Stiefel submanifolds whose optimization problems are dependent because preceding-layer weights affect later layers.These dependencies amplify as network depth increases.
- Stabilize the Distribution of Activations: For s = Wx with W W^T = I, zero-mean inputs with covariance σ^2I produce zero-mean outputs with covariance σ^2I.When n = d, the transformation also preserves activation norms.
- Stabilize the Distribution of Activations: Orthogonal weights preserve activation and back-propagated gradient norms, while normalized and decorrelated activations can improve conditioning and accelerate training.These properties follow from the stated theorem and its interpretation for deep networks.
- Regularize Neural Networks: Orthogonal filters impose pairwise orthogonality and unit-norm constraints, regularizing networks through the Stiefel manifold's reduced degrees of freedom.The paper cautions that this regularization may reduce representation capacity in insufficiently deep networks; relaxing unit norms or adding a learnable scalar can trade off capacity and regularization.
3 Orthogonal Weight Normalization
The paper addresses unstable Riemannian optimization over multiple dependent Stiefel manifolds by reparameterizing orthogonal weights through proxy parameters and differentiable transformations. It extends the approach to practical orthogonal linear and convolutional modules using group-based orthogonalization.
- Riemannian methods for OMDSM can be unstable or converge to inferior solutions in deep feed-forward networks.
- Proxy parameters V are transformed into orthogonal weights W, while gradients back-propagate through the transformation during optimization.
- The transformation is selected by minimizing distortion between proxy parameters and canonical weights to promote stable solutions.
- Orthogonal linear modules replace standard linear modules and support orthogonality-constrained training without revising training protocols.
- Group-based orthogonalization handles n > d by partitioning weights into groups of size N_G <= d whose weights are orthogonal within each group.
- O(nN_GdF_hF_w + nN_G^2 + nmdhwF_hF_w) per iteration approaches standard convolutional complexity O(nmdhwF_hF_w) when N_G ≪ mhw.
4 Experiments
The experiments compare optimization methods for OMDSM and evaluate OLM in MLPs. OLM is reported as stable, fast, and consistently effective across tested configurations.
- Comparing OMDSM solvers: Riemannian methods became unstable or divergent at learning rates above 0.05 and were slower with small learning rates.These methods could perform well initially but later converged more slowly than the plain baseline and achieved worse performance.
- Comparing OMDSM solvers: OLM converged fastest and most stably, while QR was stable but reached a worse final training loss than the plain baseline.OLMvar was unstable, supporting distortion minimization as an essential part of the proposed transformation.
- Comparing OMDSM solvers: Deeper MLPs and smaller minibatches showed similar behavior, with OLM gaining more advantage as depth increased.The experiments used 6-layer and 8-layer MLPs in addition to the 4-layer setup.
- Using different group sizes: OLM outperformed plain and weight-normalized MLPs across group sizes on the PIE dataset.Increasing group size improved orthogonalization, but NG = 128 imposed excessive regularization and reduced performance.
- Combining practical techniques: OLM consistently achieved the best performance when combined with batch normalization or Adam optimization.The comparisons evaluated training error and test error on the PIE dataset.
4.3 CNN Architectures
CNN experiments replace standard linear or convolutional modules with OLM in VGG-style, BN-Inception, residual, and ImageNet architectures. The reported results show improved accuracy or optimization behavior, with added per-iteration cost in the current implementation.
- VGG-style architectures: OLM achieved the best reported VGG-style test performance consistently on CIFAR-10 and CIFAR-100.Table 1 reports test error averaged over five independent runs as mean ± standard deviation.
- VGG-style architectures: Increasing the number of OLM layers improved VGG-style network optimization efficiency and generalization.The study compared replacing the first two or four convolutional layers with OLM modules.
- BN-Inception: OLM converged faster by training epochs in BN-Inception experiments on CIFAR datasets.The experiments report test errors in Table 2 and training/test error curves for CIFAR-10 and CIFAR-100.
- Wide Residual Networks: 3.73% test error was obtained by WRN-OLM-L1 on CIFAR-10, compared with 4.17% for WRN.The result came from constraining only the first convolutional layer, with negligible extra computation cost reported for that setup.
- Computation cost: OLM required 0.46s per iteration versus 0.26s for plain and 0.38s for WN in small VGG, and 3.12s versus 1.1s for plain in large WRN.The authors note that the current implementation can be further optimized.
- ImageNet: The ImageNet evaluation covered AlexNet, BN-Inception, ResNet-34, and Pre-ResNet using OLM replacements under a single-GPU protocol.The evaluation used top-5 test error and kept baseline settings largely unchanged, except for minibatch size and training epochs.
5 Related Work and Discussion
Prior work studied orthogonal transformations mainly in RNNs or limited Stiefel-layer settings. The paper positions OMDSM and orthogonal weight normalization as a stable approach for deeper feed-forward networks.
- Manifold optimization: Matrix-manifold optimization methods were generally limited to one manifold or one linear mapping and often relied on full-batch gradients.The paper reports instability or inferior performance when such methods were applied to DNNs.
- Orthogonal transformations: Earlier orthogonal-matrix methods primarily addressed square hidden-to-hidden transformations in RNNs.These methods were designed to mitigate gradient vanishing and explosion through energy preservation.
- Stiefel layers: Stiefel layers extended orthogonality to fully connected layers but were evaluated only in limited network placements and were reported as unstable in deeper constraints.The cited prior method used Riemannian gradients with QR retraction.
6 Conclusions and Further Work
The paper formulates orthogonal linear transformations in DNNs as OMDSM and proposes OLM as a stable replacement for standard linear modules. It also identifies potential extensions to other deep-learning applications.
- Conclusions: OMDSM captures dependencies among orthogonally constrained layer weights that amplify as feed-forward networks become deeper.The paper presents this as the first formulation of learning orthogonal filters for deep feed-forward neural networks.
- Conclusions: Orthogonal weight normalization yields OLM, an alternative linear module reported to improve optimization efficiency and generalization ability.The paper states that OLM can be applied to large and deep networks by replacing standard linear modules.
- Further work: The authors identify potential applications in unsupervised and semi-supervised DNN methods and robustness to adversarial examples.These applications are described as potential benefits rather than demonstrated results in the supplied conclusion passage.
A.1 Proof of Theorem 1
Theorem 1 establishes that a row-orthogonal transformation preserves zero mean and isotropic covariance, while square orthogonal transformations also preserve vector and gradient norms.
- For WWT = I, zero-mean inputs with covariance σ2I produce outputs with zero mean and covariance σ2I.
- When n = d, the orthogonal transformation preserves the input norm, ∥s∥ = ∥x∥.
- The theorem also derives preservation of the back-propagated gradient norm under the corresponding orthogonality conditions.
A.2 Derivation of Minimizing Orthogonal Vectors Transformation Problem
The derivation seeks the closest row-orthogonal matrix W = PV to a full-row-rank matrix V, reducing the problem to covariance decomposition and an orthogonal transformation.
- The problem transforms full-row-rank V into W = PV while enforcing WWT = I and minimizing least-squares distortion from V.
- The covariance matrix Σ = VVT is positive definite and is decomposed as Σ = DΛDT using eigenvalues and eigenvectors.
- The constraint becomes PΣPT = I, motivating the use of Σ−1/2 in constructing feasible transformations.
- The resulting transformation uses the covariance-based solution P = Σ−1/2.
- Minimizing distortion is converted into maximizing a trace over an orthogonal matrix Q, whose optimum is Q = I.
A.3 Riemannian Optimization over Stiefel Manifold
Riemannian optimization updates weights on the Stiefel manifold by computing tangent-space gradients, selecting descent directions, and retracting updates back onto the manifold.
- The Stiefel-manifold objective constrains W to have orthonormal columns, WT W = I.
- Each iteration computes a Riemannian gradient in the current tangent space and seeks a descent direction that remains feasible.
- Tangent-space gradients depend on the chosen Euclidean or canonical inner product.
- QR-type retraction maps a tangent update onto the Stiefel manifold through the Q factor of a QR decomposition.
- Cayley transformation provides another way to move along a descent direction while maintaining feasibility.
A.4 More Experimental Results
Additional MNIST MLP experiments vary batch size and depth to evaluate Riemannian methods and OLM; the results support poor reliability of Riemannian optimization and easier optimization with OLM.
- Batch size 1024: Batch size 1024 experiments evaluate 6-layer and 8-layer MLPs using training-loss curves across EI+QR, CI+QR, CayT, and plain baselines.
- Batch size 512: Batch size 512 experiments cover 4-layer, 6-layer, and 8-layer MLPs, reporting training-loss curves and best performance across learning rates.
- Empirical conclusions: The experiments support that Riemannian methods probably do not work reliably for OMDSM without carefully designed algorithms or tuned hyperparameters.
- Empirical conclusions: The experiments also support that deep feed-forward networks with orthogonal weights are easier to optimize using OLM.
- Batch size 256: Batch size 256 experiments similarly evaluate 4-layer, 6-layer, and 8-layer MLPs.