Source-linked AI summary
Full-Capacity Unitary Recurrent Neural Networks
Scott Wisdom, Thomas Powers, John R. Hershey, Jonathan Le Roux, Les Atlas
TL;DR
Recurrent networks suffer from gradient instability, and prior uRNNs used recurrence parameterizations that may not represent every unitary matrix. The paper tests this capacity gap theoretically and proposes direct optimization over the unitary manifold. Full-capacity uRNNs generally outperform restricted-capacity uRNNs and LSTMs across synthetic and natural tasks, while the proposed optimization avoids gradient clipping and can work without learning-rate adaptation.
Problem
Prior uRNN recurrence matrices used products of parameterized unitary matrices, leaving open when those parameterizations fail to represent all unitary matrices and how that restriction affects learning.
Method
The paper uses parameter counting with Sard’s theorem to assess capacity and optimizes complete unitary recurrence matrices through manifold-constrained stochastic-gradient updates.
Results
Full-capacity uRNNs generally achieve equivalent or superior performance to LSTMs and restricted-capacity uRNNs on synthetic and natural-data tasks.
Takeaways & Limitations
The previously proposed unitary parameterization is restricted for hidden-state dimensions greater than 7, whereas direct full-capacity optimization provides a practical alternative without gradient clipping or learning-rate adaptation.
Takeaways & Limitations
The system-identification evaluation uses synthetic sequences with T = 150, identity input and output transformations, zero initial state, and a fixed bias initialization range.
Abstract
from arXiv · showhide
Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second, we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.
1 Introduction
The paper addresses gradient instability in recurrent networks by analyzing the capacity of unitary parameterizations and introducing full-capacity optimization. Experiments compare full- and restricted-capacity uRNNs with LSTMs on synthetic and natural tasks.
- Repeated use of recurrent weight matrices can amplify gradient instability, causing vanishing or exploding gradients that impede learning.
- The paper’s first contribution uses parameter counting and Sard’s theorem to identify when a unitary parameterization cannot cover the full unitary manifold.
- For hidden-state dimensions larger than 7, the previously proposed parameterization cannot represent all unitary matrices and therefore has restricted capacity.
- The second contribution directly optimizes a complete unitary recurrence matrix by constraining stochastic-gradient updates to the differentiable unitary manifold.
- Full-capacity uRNNs generally match or outperform both LSTMs and the original restricted-capacity uRNNs across synthetic and natural-data tasks.
2 Unitary recurrent neural networks
The original uRNN uses complex-valued hidden states and a unitary recurrent transition within a nonlinear dynamical system. Its recurrence matrix is parameterized as a product of structured unitary components for computational efficiency.
- The uRNN dynamical system uses complex-valued inputs and hidden states, with outputs formed from a hidden-to-output transformation.
- Its element-wise nonlinearity soft-thresholds activation magnitudes using a bias vector.
- The recurrent transition matrix W is unitary, while V, b, U, and c parameterize input transformation, nonlinearity bias, output transformation, and output bias.
- The original parameterization constructs W as a product of diagonal unitary, Householder reflection, Fourier-transform, and permutation matrices.
- This decomposition remains unitary because each component is unitary and is computationally efficient through O(N) structured operations and O(N log N) FFTs.
3 Estimating the representation capacity of structured unitary matrices
The paper tests whether structured unitary parameterizations can represent every N × N unitary matrix. A parameter-counting theorem using manifold dimension and Sard’s theorem shows the cited parameterization is incomplete for N > 7.
- Manifold dimension: The set of all N × N unitary matrices forms a differentiable manifold with dimension N^2.This dimension is established by identifying unitary matrices with the Lie group U(N) and its skew-Hermitian Lie algebra.
- Capacity criterion: Any smooth unitary parameterization with fewer than N^2 real-valued parameters cannot represent all N × N unitary matrices.Sard’s theorem implies that the parameterization image has measure zero in the unitary manifold and is therefore not onto.
- Application: The structured parameterization has 7N real-valued parameters, so it cannot represent all unitary matrices when N > 7.The threshold follows directly from solving 7N < N^2 for N.
4 Optimizing full-capacity unitary matrices on the Stiefel manifold
The paper directly optimizes a complete unitary recurrence matrix on the Stiefel manifold rather than relying on a restricted parameterization. Its multiplicative Cayley-based update follows the manifold geometry and can use a fixed learning rate, with only a matrix inverse added beyond ordinary network passes.
- Stiefel-manifold formulation: The Stiefel manifold contains complex N × N matrices whose columns are N orthonormal vectors, providing the space for full-capacity optimization.The unitary matrices of interest lie in this manifold, which is equipped with a tangent-space geometry for optimization.
- Manifold gradient: The manifold gradient is AW, where A = G^HW − W^HG is skew-Hermitian and G is the ordinary loss gradient with respect to W.This construction projects the usual gradient into a direction compatible with the Stiefel-manifold structure.
- Multiplicative update: Training updates W by multiplying the current solution by a Cayley transformation evaluated along a descent curve on the Stiefel manifold.The update uses the Cayley transformation of A^(k) and the current matrix W^(k).
- Optimization practice: A fixed learning rate often works well, while RMSprop-style gradient scaling can improve convergence without requiring an expensive line search.The main additional substantial computation beyond forward and backward passes is an N × N matrix inverse.
5 Experiments
Experiments compare full-capacity and restricted-capacity uRNNs with LSTMs across synthetic, speech, and handwritten-digit tasks. Full-capacity uRNNs generally perform best when restricted parameterizations cannot represent the relevant unitary matrices, while restricted models can remain competitive at smaller hidden dimensions.
- Synthetic data: Synthetic evaluations cover system identification and long-term copy memory, testing whether recurrence capacity affects learning on controlled tasks.The copy-memory sequences extend to T = 1000 and T = 2000.
- System identification: For system identification, full-capacity uRNNs outperform restricted-capacity uRNNs for N > 7, confirming the predicted capacity boundary.For N < 7, restricted-capacity models achieve comparable or better performance; at N = 7, performance is relatively comparable.
- Copy memory problem: At T = 1000 and T = 2000, full-capacity uRNNs reach zero average cross entropy, whereas restricted-capacity uRNNs settle at baseline losses of 0.020 and 0.010, respectively.The full-capacity model reaches zero after about 2000 iterations for T = 1000 and after about 5000 iterations for T = 2000.
- Speech data: On speech log-magnitude STFT prediction, full-capacity uRNNs generally outperform restricted-capacity uRNNs with the same parameter count, and both uRNN types significantly outperform LSTMs.The task uses objective and perceptual metrics evaluated on reconstructed time-domain signals.
6 Conclusion
The paper concludes that unitary recurrence can address vanishing and exploding gradients, and that directly optimized full-capacity matrices improve performance across several tasks. It also identifies broader restricted-capacity constructions as future work.
- Unitary recurrent matrices are presented as effective for addressing vanishing and exploding gradient problems.
- The paper quantifies constrained-unitary capacity and directly optimizes full-capacity matrices on the differentiable manifold of unitary matrices.
- Full-capacity uRNNs outperform restricted-capacity uRNNs and LSTMs across the reported synthetic and natural-data evaluations.The conclusion specifically cites system identification, memory, speech STFT prediction, and permuted handwritten-digit classification.
- Future work will explore more general restricted-capacity unitary matrices, including products of Householder or Givens operators.