PaperlayerView exact revision on arXiv

Source-linked AI summary

Orthogonal Recurrent Neural Networks with Scaled Cayley Transform

Kyle Helfrich, Devin Willmott, Qiang Ye

arXiv:1707.09520v3stat.MLcs.LG

TL;DR

RNNs can lose the ability to learn long-term dependencies because gradients vanish or explode, motivating recurrent architectures with structured weight matrices. The paper introduces scoRNN, which uses real-valued orthogonal recurrent weights generated through a scaled Cayley transform and straightforward updates. Across experiments, scoRNN performs strongly, including 96.6% test accuracy on permuted MNIST, while addressing representational limitations of the standard Cayley transform.

  • Problem

    Vanishing or exploding gradients hinder RNN learning on long sequences, while existing unitary or orthogonal RNNs are often complex-valued or restricted in capacity.

  • Method

    scoRNN parametrizes real orthogonal recurrent weights with a skew-symmetric matrix through a scaled Cayley transform and updates that matrix by gradient descent.

  • Results

    96.6% test-set accuracy was achieved by the 512-hidden-unit scoRNN on permuted MNIST, outperforming all other machines in that experiment.

  • Takeaways & Limitations

    The experiments indicate that state-of-the-art performance can be achieved with real-valued orthogonal recurrent matrices rather than complex matrices.

  • Takeaways & Limitations

    The standard Cayley parametrization cannot represent orthogonal matrices with −1 eigenvalues, while banding the skew-symmetric matrix reduces recurrent-weight representational capacity and adds a hyperparameter.

Abstract

from arXiv · show

Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices. This is done by parametrizing with a skew-symmetric matrix using the Cayley transform. Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones. The proposed training scheme involves a straightforward gradient calculation and update step. In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.

1. Introduction

RNNs struggle with vanishing or exploding gradients, especially on long sequences, while existing remedies trade simplicity or parameter efficiency against representational capacity. This paper proposes a real-valued scaled Cayley approach that maintains orthogonal recurrent weights and performs strongly across sequential tasks.

  • Motivation: Vanishing or exploding gradients can make gradient-descent steps ineffective or excessively large, diminishing RNNs’ ability to learn long-range dependencies.The issue arises because gradient magnitudes may grow or decay exponentially.
  • Existing approaches: LSTMs and GRUs control information with gating mechanisms but require additional trainable parameters.These architectures are described as the preferred RNN alternatives for addressing the gradient difficulty.
  • Existing approaches: Unitary and orthogonal RNNs preserve recurrent-matrix structure, but prior schemes vary in capacity, complexity, and parameterization.Earlier methods include restricted-capacity and full-capacity unitary models, as well as Givens-rotation and Householder-based parameterizations.
  • Contribution: The proposed method uses a skew-symmetric matrix and a scaled Cayley transform to construct orthogonal recurrent weights while avoiding the standard transform’s −1-eigenvalue singularity.A diagonal scaling matrix enables access to appropriate orthogonal matrices through a simple gradient-descent update.
  • Contribution: The method operates entirely with real matrices and achieves state-of-the-art performance without using complex matrices to optimize along the Stiefel manifold.The paper states that the experimental results support this conclusion.

2. Background

The background defines the recurrent computation and explains why orthogonal or unitary recurrent matrices are attractive: they preserve vector norms. It then contrasts restricted-capacity unitary parameterizations with full-capacity approaches that optimize along the Stiefel manifold.

  • Recurrent neural networks: An RNN maps input sequences to output sequences using input, recurrent, bias, and output parameters, with hidden states updated recursively.The hidden-state activation is commonly a pointwise nonlinearity such as tanh or ReLU.
  • Orthogonal and unitary matrices: Orthogonal matrices satisfy W^T W = I, while unitary matrices satisfy W*W = I in the complex case.Both preserve the ℓ2 norm of any vector.
  • Unitary RNNs: The restricted-capacity uRNN represents its recurrent matrix as a product of structured unitary factors, including diagonal, Householder, Fourier, and permutation matrices.This construction yields a unitary recurrent matrix but uses a constrained product parameterization.
  • Unitary RNNs: The restricted representation has 7n parameters, which is insufficient to represent all unitary matrices when n > 7.The full-capacity uRNN addresses this limitation with a multiplicative update capable of reaching all unitary matrices of order n.
  • Optimization on the Stiefel manifold: Full-capacity uRNN optimization constructs a descent direction in the tangent space of the Stiefel manifold and updates the unitary matrix along a curve.The learning rate controls the update magnitude.

3. Scaled Cayley Orthogonal RNN

scoRNN parametrizes an orthogonal recurrent matrix with a skew-symmetric matrix through a scaled Cayley transform, using diagonal ±1 scaling to represent all orthogonal matrices. Gradient updates preserve skew-symmetry and orthogonality, while experiments report machine-precision orthogonality and stable hidden-state gradients.

  • Cayley transform limitations: The Cayley transform maps a skew-symmetric matrix to orthogonal matrices without −1 eigenvalues, but cannot represent matrices containing −1 eigenvalues.Matrices arbitrarily close to −1 may require large entries in the skew-symmetric parameter matrix, making gradient-based learning very slow.
  • Scaled Cayley transform: The scaled Cayley transform adds a diagonal matrix with entries ±1, enabling representation of any orthogonal matrix while keeping the skew-symmetric matrix bounded.The number of −1 entries in the diagonal scaling matrix is treated as a task-dependent hyperparameter.
  • Parameterization and training: scoRNN uses W = (I + A)−1(I − A)D, with skew-symmetric A updated by gradient descent and fixed diagonal D supplying the scaling.The parameterization contains n(n−1)/2 trainable weights in A.
  • Parameterization and training: Backpropagation first computes ∂L/∂W, then applies the derived gradient with respect to A before reconstructing W.The update preserves skew-symmetry in A and therefore orthogonality in the recurrent matrix.
  • Reported properties: Orthogonality is maintained to machine precision, and hidden-state gradient norms remain stable over time with small computational overhead over a standard RNN.The paper compares scoRNN with full-capacity uRNN, EURNN, and oRNN, which use different optimization or matrix representations.
  • Capacity trade-off: Restricting A to a banded skew-symmetric matrix can reduce trainable parameters, but introduces a bandwidth hyperparameter and reduces recurrent-matrix representational capacity.This modification does not reduce the hidden-state dimension.

4. Other Architecture Details

scoRNN applies its recurrent transformation through a sequence of real-valued layers and uses modReLU as its activation function. The paper reports that real modReLU outperformed other activations, which the authors associate with allowing both positive and negative states without vanishing-gradient issues.

  • Recurrent transformation: The recurrent transformation can be viewed as three layers that apply D, then (I − A), and then the remaining inverse factor to the hidden state.This layered process is equivalent to applying W to the current state.
  • modReLU activation: scoRNN uses modReLU, an activation originally introduced for complex-valued functions and weights but applied here with real-valued functions and weights.The real-case form simplifies to sign(z)σReLU(|z|+b), where b is a trainable bias.
  • Activation behavior: The authors report that real modReLU performed better than other activation functions in their experiments.They associate this improvement with modReLU admitting both positive and negative activation values, which appears important for state transitions in orthogonal RNNs.
  • modReLU activation: The modReLU output is zero when |z| + b < 0.This condition is part of the activation definition used by the network.
  • Initialization: The initialization of recurrent parameter matrix A uses zero entries except for 2×2 diagonal blocks whose values distribute Cayley-transform eigenvalues along the right unit half-circle.The scaling matrix reflects a selected fraction of these eigenvalues across the imaginary axis.

5. Experiments

The copying task tests whether recurrent models can preserve information across long sequences. At T = 1000 and T = 2000, scoRNN converges successfully, while several baselines remain near chance or fail to converge.

  • 5.1. Copying Problem: The copying task requires reproducing ten symbols presented at the sequence beginning after a marker near the end.The input contains zeros, ten symbols from classes 1–8, and a marker 9; the model must output the stored symbols after seeing the marker.
  • 5.1. Copying Problem: The LSTM, restricted-capacity uRNN, and EURNN remain at the baseline or fail to converge throughout the experiment.The baseline corresponds to outputting zeros until the marker and then randomly guessing the ten symbols.
  • 5.1. Copying Problem: At T = 1000, the full-capacity uRNN and scoRNN converge immediately to zero-entropy solutions.The full-capacity uRNN converges slightly faster than scoRNN in this setting.
  • 5.1. Copying Problem: At T = 2000, full-capacity uRNN eventually finds a correct solution, whereas scoRNN converges smoothly but more slowly.The full-capacity uRNN stays at baseline for several thousand iterations; scoRNN bypasses the baseline during convergence.

5.2. Adding Problem

The adding task requires selecting two marked values from noisy sequences and summing them over increasing sequence lengths. Across T = 200, 400, and 750, models generally move from baseline error toward zero, with convergence taking longer for longer sequences.

  • 5.2. Adding Problem: The adding problem presents a digit sequence and a binary marker sequence, requiring the model to sum the two marked digits.The markers identify one value in the first half and one in the second half of the sequence.
  • 5.2. Adding Problem: 0.167 MSE is the approximate baseline from always predicting one.This baseline is used for evaluating the adding experiments.
  • 5.2. Adding Problem: Approximately 15k trainable parameters were used for the LSTM, scoRNN, and uRNN comparisons.The corresponding hidden sizes were n = 170 for scoRNN, n = 60 for LSTM, and n = 120 for the full-capacity uRNN.
  • 5.2. Adding Problem: For T = 200, 400, and 750, networks generally drop from near-baseline MSE toward zero after several epochs.The number of epochs before this drop increases with sequence length.
  • 5.2. Adding Problem: The oRNN descends irregularly and can return to baseline, whereas the LSTM error drops precipitously across sequence lengths.The oRNN begins dropping before scoRNN but shows less stable descent behavior.

5.3. Pixel-by-Pixel MNIST

Pixel-by-pixel MNIST evaluates recurrent models on unpermuted and permuted sequences of 784 image pixels. The 512-unit scoRNN achieves 96.6% accuracy on permuted MNIST and outperforms the other machines.

  • 5.3. Pixel-by-Pixel MNIST: The MNIST experiments feed each 28×28 image to the RNN as a sequence of 784 pixels.Unpermuted MNIST presents pixels row-by-row, while permuted MNIST applies a fixed permutation to both training and testing sequences.
  • 5.3. Pixel-by-Pixel MNIST: Each experiment used 55,000 training images, 10,000 test images, and 70 training epochs.Test accuracy was evaluated at the end of every epoch, with the best value reported for each machine.
  • 5.3. Pixel-by-Pixel MNIST: The 170-unit scoRNN performs similarly to both 512-unit uRNNs with a much smaller hidden dimension and, versus full-capacity uRNN, an order of magnitude fewer parameters.At matched parameter counts, the restricted-capacity uRNN is comparable on unpermuted MNIST but worse on permuted MNIST than the 360-unit scoRNN.
  • 5.3. Pixel-by-Pixel MNIST: Orthogonal and unitary RNNs do not outperform the LSTM in the unpermuted case.This pattern is also reported in the cited prior experiments.
  • 5.3. Pixel-by-Pixel MNIST: 96.6% test-set accuracy is achieved by the 512-unit scoRNN on permuted MNIST, outperforming all other machines.The authors describe this as a state-of-the-art result.

5.4. TIMIT Speech Dataset

On TIMIT speech prediction, scoRNN models achieved lower MSE than LSTM and unitary RNN models, with stronger SegSNR but slightly weaker intelligibility metrics than full-capacity uRNN.

  • TIMIT Speech Dataset: The TIMIT evaluation used 3,696 training and 192 testing audio files after excluding dialect SA sentences and retaining the core test set.Audio was downsampled to 8kHz and transformed with a Hann-windowed short-time Fourier transform.
  • TIMIT Speech Dataset: All scoRNN models achieved smaller MSE than all LSTM and unitary RNN models on TIMIT validation and test sets.The task predicted the next frame’s log-magnitudes using mean squared error over each sequence.
  • TIMIT Speech Dataset: scoRNN predictions achieved better SegSNR scores than the compared models.SegSNR evaluated the quality of reconstructed audio predictions.
  • TIMIT Speech Dataset: Full-capacity uRNN predictions slightly outperformed scoRNN on STOI and PESQ, metrics measuring human intelligibility and perception.The comparison concerns reconstructed audio rather than the primary MSE objective.

5.5. Further Analysis

Further analyses examined numerical orthogonality, gradient propagation, and runtime, finding that scoRNN preserves orthogonality and attenuates gradients less while incurring transform-related computational cost.

  • 5.5.1. LOSS OF ORTHOGONALITY: With n = 512 hidden units, scoRNN orthogonality remained unaffected by roundoff errors, while full-capacity uRNN became less unitary over training.The comparison used the unpermuted MNIST experiment and checked scores at each epoch.
  • 5.5.2. VANISHING GRADIENTS: LSTM gradient norms decreased steadily away from the sequence end, with the vanishing effect exacerbated after 300 training iterations.The gradients were measured on the adding problem with sequence length T = 500.
  • 5.5.2. VANISHING GRADIENTS: After 300 iterations, scoRNN hidden-state gradients decayed from 10^-3 at t = 500 to 10^-4 at t = 0.At the beginning of training, gradients remained near 10^-2 across timesteps and decayed by less than an order of magnitude.
  • 5.5.3. COMPLEXITY AND SPEED: The n = 170 scoRNN was approximately 1.5 times faster than restricted-capacity uRNN with n = 512 and twice as fast as full-capacity uRNN with n = 116 at matched hidden parameters.The n = 170 scoRNN had similar speed to LSTM, whereas n = 512 scoRNN was approximately twice as slow as LSTM.

6. Conclusion

The paper concludes that scoRNN uses real-valued orthogonal recurrent matrices with a simpler update scheme, while achieving strong performance and stable convergence across experiments.

  • 6. Conclusion: scoRNN uses real-valued orthogonal recurrent matrices and can represent all orthogonal matrices through Cayley-transform parametrization.Its additive update maintains recurrent-weight orthogonality in the presence of roundoff errors.
  • 6. Conclusion: Across experiments, scoRNN was among the top performers with smooth and stable convergence curves.The conclusion reports this pattern across the paper’s tests without claiming universal superiority.
  • 6. Conclusion: In some experiments, scoRNN achieved this performance with a smaller hidden-state dimension than other models.The conclusion presents reduced hidden-state dimension as an observed experimental outcome.

Supplemental Material: Proof of Theorem 3.2

The supplemental material proves the gradient formula for the loss with respect to the skew-symmetric parameter in the scaled Cayley representation of the recurrent matrix.

  • Supplemental Material: Proof of Theorem 3.2: Theorem 3.2 defines W(A) = (I + A)^-1(I − A)D, where A is skew-symmetric and D has diagonal entries −1 or 1.The theorem considers a differentiable loss L(W) and derives its gradient with respect to A.
  • Supplemental Material: Proof of Theorem 3.2: The proof introduces Z = (I + A)^-1(I − A) and differentiates the identity (I + A)Z = I − A.This identity supplies the derivative relation used to obtain the gradient expression.
  • Supplemental Material: Proof of Theorem 3.2: Because A is skew-symmetric, its derivative with respect to an off-diagonal entry satisfies ∂A/∂A_i,j = E_i,j − E_j,i.The proof uses this structure when differentiating off-diagonal parameters.
  • Supplemental Material: Proof of Theorem 3.2: The diagonal derivative is zero, while off-diagonal derivatives are obtained from the differentiated matrix identity and rearrangement.The derivation explicitly states ∂L/∂A_j,j = 0 before giving the off-diagonal result.
Loading 1707.09520v3…