Source-linked AI summary
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton
TL;DR
Long-term dependencies are difficult for RNNs because training gradients can vanish or explode. The paper uses ReLU recurrent networks with identity-based initialization, finding performance comparable to LSTMs across toy, language-modeling, and speech tasks, while noting that LSTM tuning may affect some comparisons.
Problem
RNNs have difficulty learning long-term dependencies because backpropagated gradients can vanish or explode.
Method
The paper trains ReLU recurrent networks using the identity matrix or a scaled identity to initialize recurrent weights, with zero biases.
Results
IRNNs were comparable to LSTMs across long-range toy problems, large language modeling, and TIMIT acoustic modeling, including 3% versus 34% test error on sequential MNIST.
Takeaways & Limitations
Simple identity-based initialization makes ReLU recurrent networks relatively easy to train and effective at modeling long-range dependencies.
Takeaways & Limitations
The reported LSTM comparisons may improve with better tuning, including different architectures, hidden-state sizes, or initial forget-gate biases.
Abstract
from arXiv · showhide
Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients. To overcome this difficulty, researchers have developed sophisticated optimization techniques and network architectures. In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units. Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix. We find that our solution is comparable to LSTM on our four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem.
1 Introduction
RNNs naturally model sequences, but training them to learn long-term dependencies is difficult because gradients can vanish or explode. The paper motivates simpler recurrent architectures by examining which LSTM design elements matter and whether ReLUs can be trained effectively in RNNs.
- RNN training with back-propagation through time can suffer from vanishing and exploding gradients, hindering long-term dependency learning.
- Sophisticated optimization methods and careful initialization with gradient clipping have been explored as alternatives for training recurrent networks.
- LSTM improves gradient behavior by using gated memory cells that can preserve stored values and maintain gradients over time.Input, output, and forget gates control writing, reading, and decay of each analog memory value.
- LSTM’s strong results motivate identifying which parts of its complicated architecture are essential rather than merely incidental.
- The paper also asks whether ReLUs, despite potentially large outputs, can be made effective in RNNs and retain their optimization advantages.
2 The initialization trick
The proposed IRNN initializes ReLU recurrent networks with the identity matrix and zero biases, preserving hidden states and error derivatives when inputs are absent. A scaled identity provides faster forgetting when tasks require less long-range memory.
- IRNNs initialize the recurrent weight matrix as the identity and biases as zero, so hidden states copy forward before current inputs and rectification are applied.
- Without input, identity-initialized ReLU networks remain in the same state indefinitely, keeping backpropagated error derivatives constant when no extra derivatives are added.
- This constant-gradient behavior resembles LSTM memory cells with forget gates configured to avoid decay, supporting very long-range dependency learning.
- Scaling the identity matrix by a small scalar helps tasks with weaker long-range dependencies forget earlier effects, analogous to rapidly decaying LSTM memory.
- Unlike related approaches that fix part of the recurrent matrix or use scaled identity without ReLUs, this method uses identity only for initialization with rectified units.
3 Overview of the experiments
Across long-range toy problems, language modeling, and acoustic modeling, IRNNs performed comparably to LSTMs and sometimes better. The results support a simpler ReLU recurrent model, while the authors note that better-tuned LSTMs could change some comparisons.
- IRNNs learned the adding problem with sequence length 300, a challenging regime for other algorithms.
- 3% test error for IRNN versus 34% for LSTM was reported on sequential MNIST classification.
- The authors caution that better-tuned LSTMs with different architectures, hidden-state sizes, or initialization choices might outperform IRNNs on the toy tasks.
- IRNN performance was comparable to an equivalent LSTM on a large language-modeling task under parameter and architecture balancing.
- IRNNs were comparable to LSTMs on TIMIT acoustic modeling while using a simpler implementation and a recurrent matrix initialized as 0.01 times the identity.
4 Experiments
Experiments compare IRNNs with LSTMs and other recurrent baselines on long-range toy tasks, language modeling, and speech recognition. IRNNs use identity-initialized recurrent weights and show strong performance across these settings, with scaled identity preferred for speech.
- Experimental setup: IRNN experiments compare against LSTMs, tanh RNNs, and randomly initialized ReLU RNNs using shared training procedures and tuned learning rates and clipping.IRNN recurrent weights use identity initialization, while non-recurrent weights use a small Gaussian initialization; all methods share batch size 16.
- 4.1 The Adding Problem: The adding problem selects two masked values from a random sequence and tests whether recurrent models can preserve their sum as sequence length increases.Experiments vary sequence length through T = 150, 200, 300, and 400, with a 100-unit hidden state for every network.
- 4.1 The Adding Problem: IRNN convergence is as good as LSTM on the adding problem, although each LSTM step is at least 4x more expensive and both models find sequences of length 400 challenging.The reported comparisons use test-set RMSE, with best hyperparameters selected by grid search.
- 4.2 MNIST Classification from a Sequence of Pixels: On sequential MNIST, IRNN achieves 3% test error, while LSTM achieves 34%, and permuting pixels makes the task harder but leaves IRNN better than LSTM on standard MNIST.The task presents all 784 pixels sequentially before classification; the same setup is also evaluated with a fixed random pixel permutation.
- 4.3 Language Modeling: On the one-billion-word benchmark, IRNN performance is closer to LSTM than to standard RNN performance.The benchmark uses a 1,000,000-word vocabulary and controlled experiments with similar parameter counts.
- 4.4 Speech Recognition: For TIMIT speech recognition, scaled identity initialization with 0.01I gives iRNNs that outperform tanh RNNs and are comparable to LSTMs.Full identity initialization converged slowly, performed worse, and sometimes diverged; a five-layer bidirectional LSTM remained best, followed closely by a five-layer bidirectional iRNN.