Source-linked AI summary
On orthogonality and learning recurrent networks with long term dependencies
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, Chris Pal
TL;DR
Long-term dependencies make deep and recurrent networks difficult to train because gradients can vanish or explode. The paper factorizes recurrent weight matrices to control their singular spectra and tests configurable deviations from orthogonality. It finds that hard orthogonality constraints can reduce convergence speed and model performance, while excessive relaxation can destabilize optimization.
Problem
Deep and recurrent networks are difficult to train on long-term dependencies because vanishing or exploding gradients impair backpropagation.
Method
The paper factorizes weight matrices into orthogonal bases and controllable singular values, using margins and regularization to govern deviation from orthogonality.
Results
Hard orthogonality constraints can reduce convergence speed and model performance, whereas excessive spectral relaxation can produce unstable optimization.
Takeaways & Limitations
Orthogonal initialization may help stabilize training, but orthogonality constraints may need to be loosened to preserve model representational ability.
Abstract
from arXiv · showhide
It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and may therefore be a desirable property. This paper explores issues with optimization convergence, speed and gradient stability when encouraging or enforcing orthogonality. To perform this analysis, we propose a weight matrix factorization and parameterization strategy through which we can bound matrix norms and therein control the degree of expansivity induced during backpropagation. We find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance.
1. Introduction
The paper motivates orthogonality as a way to stabilize gradients in recurrent networks, while examining whether strict constraints impair optimization and representational capacity.
- Deep and recurrent networks are difficult to optimize because gradients can vanish or explode during backpropagation.
- Prior approaches include gradient or signal-norm regularization, orthogonal initialization, unitary recurrent models, and optimization directly on the Stiefel manifold.
- Orthogonal transformations preserve norm because their singular values have unit gain, making them attractive for long-term gradient propagation.
- The paper proposes studying real-valued matrices within a configurable margin around the Stiefel manifold to test whether hard orthogonality limits performance.The stated hypothesis is that strong orthogonality constraints may reduce representational power and make optimization more difficult.
- Gradient growth or shrinkage depends on the gains of both the recurrent matrix W and the activation Jacobian D.In RNNs, the shared transition matrix can amplify a non-unitary gain exponentially across time steps.
2. Our Approach
The approach factorizes recurrent transition matrices into orthogonal bases and controllable singular values, allowing bounded deviation from exact orthogonality during optimization.
- The method explicitly controls expansion and contraction by bounding the largest and smallest singular values of W.
- The singular value decomposition separates W into orthogonal basis matrices and a diagonal matrix of positive singular values.This makes the matrix’s maximum and minimum gains controllable through its largest and smallest singular values.
- Geodesic gradient descent with a Cayley transform keeps the basis matrices U and V orthogonal during updates.The factorized transition matrix can therefore move away from the Stiefel manifold through its singular spectrum while retaining orthogonal bases.
- A spectral margin constrains each singular value to the range [1 −m, 1 + m] while allowing the underlying parameters to update freely.The margin also slows singular-value movement toward its bounds.
- Soft regularization provides an alternative that encourages W toward norm preservation without imposing hard singular-value bounds.
3. Experiments
Experiments show that loosening orthogonality constraints often improves convergence and performance, while orthogonal initialization can preserve useful gradient behavior. The best margin depends on sequence length, task structure, activation, and whether constraints are hard or soft.
- 3.1.1. CONVERGENCE ON SYNTHETIC MEMORY TASKS: Larger spectral margins generally accelerate copy-task convergence, but margins above 0.001 slow convergence at T=10000.The unconstrained model also failed to converge at the longest sequence length.
- 3.1.1. CONVERGENCE ON SYNTHETIC MEMORY TASKS: For the adding task at T=1000, purely orthogonal and unconstrained transition matrices failed to improve beyond baseline MSE within 2000 epochs.The tested margins were m = 0, 1, 10, and 100, alongside no margin.
- 3.1.1. CONVERGENCE ON SYNTHETIC MEMORY TASKS: Nonlinear transition functions made the copy task substantially harder; only an identity-equivalent PReLU slope solved the T=1000 task with margin 0.3.A trainable slope initialized at 0.7 converged to 0.96, further favoring no transition nonlinearity for this memory task.
- 3.1.2. PERFORMANCE ON REAL DATA: On sequential MNIST, non-zero margins outperformed purely orthogonal transitions, with m = 0.1 reaching 94.10% on MNIST and 91.44% on permuted MNIST.LSTMs performed better overall, but orthogonally initialized RNNs nearly matched the LSTM on permuted MNIST.
- 3.1.2. PERFORMANCE ON REAL DATA: Orthogonal initialization helped MNIST, whereas strict orthogonality constraints were mainly detrimental; on PTB, constraints hurt short sentences but helped when long sentences were included.The authors suggest that PTB’s per-character outputs permit local learning before longer dependencies become useful.
- 3.1.3. SPECTRAL AND GRADIENT EVOLUTION: With margin zero, gradient norms were preserved across time; increasing the margin shortened the duration of norm preservation, yet unconstrained spectra often remained near orthogonal during training.Singular values generally stayed within prescribed bounds and reached the margin only under very large learning rates that prevented convergence.
4. Conclusions
The experiments indicate that hard orthogonality constraints can hurt optimization and performance, while moderate relaxation can improve both. Excessive relaxation, however, can reverse these gains and destabilize optimization.
- Hard orthogonality constraints can be detrimental despite potentially beneficial orthogonal initialization.
- Moving away from hard matrix orthogonality can improve optimization convergence rate and model performance.
- Overly relaxed spectral regularization or excessively wide spectral-norm bounds can reverse the gains and lead to unstable optimization.