PaperlayerView exact revision on arXiv

Source-linked AI summary

Quaternion Recurrent Neural Networks

Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi, Renato De Mori, Yoshua Bengio

arXiv:1806.04418v3stat.MLcs.LG

TL;DR

Multidimensional sequential inputs contain internal dependencies that standard recurrent models do not explicitly capture. The paper introduces QRNN and QLSTM models based on quaternion algebra, which outperform real-valued counterparts on TIMIT while using substantially fewer parameters.

  • Problem

    Multidimensional inputs contain internal feature dependencies, and efficient representations should encode these relations with fewer parameters.

  • Method

    QRNN and QLSTM use quaternion algebra to process multidimensional features as structured entities while modeling sequence relations.

  • Results

    QRNNs and QLSTMs outperform RNNs and LSTMs on TIMIT phoneme recognition with 2 to 3 times fewer free parameters.

  • Takeaways & Limitations

    Quaternion algebra provides a more compact representation of multidimensional information while improving the reported phoneme-recognition results.

  • Takeaways & Limitations

    The speech-recognition setup treats acoustic derivatives as correlated views of the same input.

Abstract

from arXiv · show

Recurrent neural networks (RNNs) are powerful architectures to model sequential data, due to their capability to learn short and long-term dependencies between the basic elements of a sequence. Nonetheless, popular tasks such as speech or images recognition, involve multi-dimensional input features that are characterized by strong internal dependencies between the dimensions of the input vector. We propose a novel quaternion recurrent neural network (QRNN), alongside with a quaternion long-short term memory neural network (QLSTM), that take into account both the external relations and these internal structural dependencies with the quaternion algebra. Similarly to capsules, quaternions allow the QRNN to code internal dependencies by composing and processing multidimensional features as single entities, while the recurrent operation reveals correlations between the elements composing the sequence. We show that both QRNN and QLSTM achieve better performances than RNN and LSTM in a realistic application of automatic speech recognition. Finally, we show that QRNN and QLSTM reduce by a maximum factor of 3.3x the number of free parameters needed, compared to real-valued RNNs and LSTMs to reach better results, leading to a more compact representation of the relevant information.

1 INTRODUCTION

RNNs model sequential dependencies, but multidimensional inputs also contain internal dependencies that standard representations do not explicitly capture. Quaternion networks address this gap by bundling features and processing recurrent sequences with fewer parameters.

  • Quaternions contain one real and three imaginary components, fitting three- and four-dimensional feature vectors.
  • Quaternion networks bundle related input features to code latent inter-dependencies during learning with fewer parameters than RNNs.The Hamilton product replaces the ordinary product between quaternions.
  • Previous quaternion and complex networks addressed image and language processing, but not recurrent neural networks with quaternion-defined operations.
  • QRNN and QLSTM extend recurrent and gated recurrent models with quaternion algebra for multidimensional sequential features.The approach targets both inter-dependencies across sequence elements and intra-dependencies within feature groups.
  • On TIMIT phoneme recognition, QRNN and QLSTM outperform RNN and LSTM while using up to 3.3 times fewer free parameters.Reported best PERs are 18.5% versus 19.0% for QRNN versus RNN, and 15.1% versus 15.3% for QLSTM versus LSTM.

2 MOTIVATIONS

The motivation is to represent multidimensional features as structured entities while reducing the parameters needed to learn their internal relations. Quaternion representations use shared weights to model these relations efficiently, including correlated acoustic derivatives.

  • The motivation combines improved multidimensional representation with fewer parameters to support efficient models and reduce overfitting risk.
  • Quaternion representations treat related feature components as a single entity, making internal relations easier to encode than with real-valued RNNs.The paper uses RGB components as an illustrative example of strongly related dimensions.
  • Quaternion weights encode internal relations during the Hamilton product, while the layer can learn global relations between different features.
  • A quaternion connection has 4 degrees of freedom versus 16 for a standard 4-by-4 real-valued connection, giving a four-fold parameter saving.
  • Speech-recognition inputs concatenate static log Mel features with derivatives even though derivatives provide correlated views of the same signal.The paper motivates modeling these relations explicitly in quaternion-valued recurrent networks.

3 QUATERNION RECURRENT NEURAL NETWORKS

The QRNN represents vectors as quaternions and replaces real-valued affine products with Hamilton products in recurrent computation. Its training uses quaternion backpropagation and an initialization scheme adapted to quaternion-valued weights.

  • 3.1 QUATERNION ALGEBRA: A quaternion consists of one real component and three imaginary components expressed as Q = r1 + xi + yj + zk.The unit basis is 1, i, j, and k; r, x, y, and z are real numbers.
  • 3.1 QUATERNION ALGEBRA: The Hamilton product combines two quaternions and is used to transform vectors, scale them, and interpolate between rotations.
  • 3.2 QUATERNION REPRESENTATION: A QRNN splits each vector into four components composing quaternions, then replaces real-valued matrix dot products with Hamilton products.Inputs, outputs, weights, and biases are quaternion-valued, while the implementation manipulates real-valued matrices.
  • 3.3.1 FORWARD PHASE: QRNN forward propagation updates the hidden state with recurrent and input Hamilton products, then computes outputs from the hidden state with another Hamilton product.Split activation functions apply standard nonlinearities separately to the quaternion components.
  • 3.3.2 QUATERNION BACKPROPAGATION THROUGH TIME: Quaternion backpropagation through time extends standard quaternion backpropagation by differentiating losses with respect to quaternion parameter components.The recurrent and input weights and biases depend on the hidden-state recurrence, while the output weights enter the output computation directly.
  • 3.4 PARAMETER INITIALIZATION: The proposed initialization adapts Glorot and He criteria by accounting for quaternion weight variance and sampling an angle and normalized imaginary direction.The algorithm samples θ, ϕ, and imaginary components before constructing each quaternion weight.

4 EXPERIMENTS

Experiments evaluate quaternion recurrent models on TIMIT speech recognition, comparing QRNN and QLSTM variants with real-valued baselines under matched conditions. Both quaternion models achieve lower phoneme error rates while using substantially fewer parameters.

  • 4.1 QUATERNION ACOUSTIC FEATURES: The TIMIT experiments use 40-dimensional log Mel-filterbank features with first-, second-, and third-order derivatives, organized into acoustic quaternions.Each acoustic quaternion represents energy, slope, concavity, and the rate of change of concavity at one frequency and time frame.
  • 4.2 THE TIMIT CORPUS: The study compares four-layer models while varying neuron counts from 256 to 2,048 for RNNs and from 64 to 512 for QRNNs.Training, validation, and testing use the standard TIMIT sentence and speaker splits.
  • 4.2 THE TIMIT CORPUS: 18.5% versus 19.0% test PER gives QRNN a 0.5% absolute improvement over RNN while using 3.8M versus 9.4M parameters.Both models have the same internal dimensionality, corresponding to 1,024 real-valued or 256 quaternion-valued neurons.
  • 4.3 QUATERNION LONG-SHORT TERM MEMORY NEURAL NETWORKS: QLSTM extends QRNN with quaternion-valued gates and is trained bidirectionally under the same conditions as the LSTM baseline.The extension is motivated by applying quaternion recurrent propagation equations to the LSTM architecture.
  • 4.3 QUATERNION LONG-SHORT TERM MEMORY NEURAL NETWORKS: 15.1% versus 15.3% test PER gives QLSTM a 0.2% absolute improvement over LSTM using 3.3 times fewer parameters.The comparison is reported for QLSTM and LSTM models on TIMIT.

5 CONCLUSION

The paper concludes that QRNNs and QLSTMs compactly represent multidimensional sequential information by modeling feature interdependencies with quaternion algebra. On TIMIT phoneme recognition, they outperform real-valued counterparts with substantially fewer free parameters.

  • 5 CONCLUSION: QRNNs and QLSTMs outperform RNNs and LSTMs on TIMIT phoneme recognition while using 2 to 3 times fewer free parameters.The conclusion attributes this compact representation to quaternion algebra and Hamilton-product modeling of internal feature dependencies.
  • 5 CONCLUSION: Future work will investigate multi-view features, quaternion Fourier transforms, and other high-dimensional neural networks such as manifold and Clifford networks.These directions are presented as ways to improve phoneme representation or extend research beyond quaternion models.

6.1 WALL STREET JOURNAL EXPERIMENTS AND COMPUTATIONAL COMPLEXITY

Experiments on the Wall Street Journal corpus test whether QLSTM advantages observed on TIMIT scale to larger speech-recognition datasets. QLSTMs outperform LSTMs across training conditions while retaining lower parameter counts, but their Hamilton-product implementation can slow training.

  • 6.1.1 SPEECH RECOGNITION WITH THE WALL STREET JOURNAL CORPUS: The WSJ evaluation uses 14-hour and 81-hour training subsets, with test-dev93 for validation and test-eval92 for testing.The experiment is designed to validate QLSTM scaling on a larger, more realistic corpus.
  • 6.1.1 SPEECH RECOGNITION WITH THE WALL STREET JOURNAL CORPUS: QLSTMs outperform LSTMs on both 14-hour and 81-hour WSJ training conditions across validation and testing sets, using 2.9 times fewer neural parameters.The results are reported in Table 3 for the WSJ corpus.
  • 6.1.2 NOTES ON COMPUTATIONAL COMPLEXITY: QLSTM computational complexity remains O(n^2), matching real-valued LSTM complexity despite using matrices four times larger.Here n denotes the number of hidden states.
  • 6.1.2 NOTES ON COMPUTATIONAL COMPLEXITY: A quaternion-neuron forward propagation uses 28 operations versus one for two real-valued neurons, making training up to 3 times slower.The passage suggests an engineered cuDNN Hamilton-product kernel could alleviate this worst-case speed difference.

6.2 PARAMETERS INITIALIZATION

Quaternion weight initialization models the weight magnitude through a four-dimensional Gaussian-derived distribution and converts its Cartesian coordinates into polar form. The resulting Jacobian supplies the polar volume element used to derive expectations and variance.

  • The quaternion weight is represented in polar coordinates using a magnitude and three phase variables, with the imaginary part formed from normalized components.The phases satisfy 0 ≤ θ ≤ π, 0 ≤ φ ≤ π, and 0 ≤ ψ ≤ 2π.
  • Quaternion weights are modeled with four independent, zero-centered normally distributed components, so their magnitude follows a Chi-distribution with four degrees of freedom.The squared magnitude is evaluated from the four-dimensional vector X = {A, B, C, D}.
  • The four-dimensional Cartesian volume element transforms to da db dc dd = ρ^3 sin^2 θ sin φ dρ dθ dφ dψ through the Jacobian.This transformation is used to rewrite the weight distribution in polar coordinates.
  • The derivation computes the weight-magnitude expectation by differentiating the cumulative distribution, integrating by parts, and evaluating limiting terms.The expectation is assembled from two terms, with limits handled using L’Hôpital’s rule where needed.
  • The squared-magnitude expectation is 4σ^2, which is then used to obtain the variance of the quaternion weight.The derivation notes that the distribution is symmetric around zero, making the squared mean term zero.

6.3 QUATERNION BACKPROPAGATION THROUGH TIME

Quaternion backpropagation through time extends recurrent forward equations and loss differentiation to quaternion-valued weights and their individual components. Its recursive treatment differs from real-valued BPTT and may act as a training regularizer.

  • QBPTT propagates the recurrent hidden state from the previous state and current input using quaternion matrix products, bias, and a split quaternion activation.The output is then computed from the hidden state with a quaternion output-weight matrix.
  • The objective is a real-valued component-wise loss whose gradients are computed for output, recurrent, input, and bias parameters.The forward equations define the hidden state, activation, output vector, and target needed for these derivatives.
  • Unlike real-valued BPTT, QBPTT differentiates the loss with respect to each component of every quaternion neural parameter.The paper characterizes this as a bi-level process that may regularize training.
  • Gradients for recurrent weights recursively accumulate contributions across the N timesteps composing the sequence.The recurrent derivation separates non-recursive and recursive elements before regrouping component-wise derivatives.
  • Input-weight gradients follow the same procedure as recurrent-weight gradients, while bias gradients simplify because biases are not connected to inputs or hidden states.The bias derivative uses a matrix of ones in the intermediate derivative expression.
Loading 1806.04418v3…