Source-linked AI summary
Learning Longer Memory in Recurrent Neural Networks
Tomas Mikolov, Armand Joulin, Sumit Chopra, Michael Mathieu, Marc'Aurelio Ranzato
TL;DR
The paper addresses whether recurrent networks can learn longer-term patterns with standard stochastic gradient descent despite vanishing gradients. It modifies the SRN by constraining part of the recurrent weight matrix close to identity, and reports performance similar to LSTM in language modeling, with advantages under parameter constraints. The authors also distinguish this learned longer-term information from truly long-term memory.
Problem
Simple recurrent networks struggle to learn long-term patterns because gradients shrink during backpropagation through time, while LSTM uses a more sophisticated gated structure.
Method
The model constrains part of the SRN recurrent matrix close to identity, producing slowly updating context units that retain longer-term information.
Results
SCRN shows similar behavior and gains to LSTM on two language modeling tasks, and often outperforms LSTM by a large margin when models have similar numbers of parameters.
Takeaways & Limitations
The findings support learning longer-term patterns in real sequential data with standard stochastic gradient descent through a structural constraint on the recurrent weight matrix.
Takeaways & Limitations
The authors note that none of the evaluated models can learn truly long-term memory, which has a different nature.
Abstract
from arXiv · showhide
Recurrent neural network is a powerful model that learns temporal patterns in sequential data. For a long time, it was believed that recurrent networks are difficult to train using simple optimizers, such as stochastic gradient descent, due to the so-called vanishing gradient problem. In this paper, we show that learning longer term patterns in real data, such as in natural language, is perfectly possible using gradient descent. This is achieved by using a slight structural modification of the simple recurrent neural network architecture. We encourage some of the hidden units to change their state slowly by making part of the recurrent weight matrix close to identity, thus forming kind of a longer term memory. We evaluate our model in language modeling experiments, where we obtain similar performance to the much more complex Long Short Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997).
1 INTRODUCTION
Sequential models must capture long-range dependencies, but simple recurrent networks are difficult to train because gradients vanish over time. The paper proposes a simpler alternative to sophisticated architectures for addressing this problem.
- Fixed-length windows make long-term dependencies harder to learn and increase parameters linearly with window size.
- Simple recurrent networks recursively store hidden states, theoretically allowing information to persist for arbitrarily long periods.
- Vanishing gradients cause simple recurrent networks to focus on short-term patterns and practically ignore longer-term dependencies.
- Gradients in recurrent networks are empirically usually close to zero after 5 - 10 steps.
- LSTM addresses vanishing gradients with gates that control information flow, but uses a fairly sophisticated structure.
- The paper modifies the SRN by constraining part of its recurrent matrix close to identity, creating context units that capture long-term information and achieve competitive performance with LSTM on language modeling datasets.
2 MODEL
The model combines a standard recurrent hidden layer with slowly changing context units to retain longer-term information while remaining trainable with stochastic gradient descent. Its structural constraint uses a diagonal recurrent block that encourages context units to change slowly, complementing fast hidden-unit dynamics.
- 2.1 SIMPLE RECURRENT NETWORK: An SRN predicts the next token from the current token and recurrent hidden state, whose units store information about preceding tokens.The hidden state is updated recursively from the current token encoding and previous hidden state, then mapped to a next-token probability vector.
- 2.2 CONTEXT FEATURES: SCRN adds context units with a diagonal recurrent matrix so their state changes slowly, while a fully connected recurrent matrix preserves fast hidden-unit dynamics.The model therefore combines quickly changing representations resembling n-grams with slowly changing context representations resembling cache-based topic information.
- 2.2 CONTEXT FEATURES: The context state is an exponentially decaying bag-of-words history, updated without a nonlinearity using a recurrent retention weight α.Learning separate diagonal weights can let different context units capture information at different time delays.
- 2.2 CONTEXT FEATURES: The structural modification constrains a diagonal recurrent block to a reweighted identity and sets an off-diagonal block to zero, defining the Structurally Constrained Recurrent Network.An alternative interpretation treats context units as additional hidden units with no activation function.
- 2.2 CONTEXT FEATURES: Learning context-unit self-recurrent weights does not appear important when the model also includes the standard hidden layer.The paper studies when learning these weights helps and reports this finding as an experimental result.
3 EXPERIMENTS
The experiments evaluate SCRN against SRN and LSTM on Penn Treebank and Text8, showing that contextual features improve language-modeling performance and can match or exceed stronger recurrent baselines in several settings.
- Experimental setup: The experiments compare SCRN with standard SRNs and LSTM RNNs on the Penn Treebank and Text8 language-modeling datasets.Penn Treebank contains 930K training words, while Text8 uses 99M training characters and a vocabulary of about 44K tokens.
- Penn Treebank results: SCRN achieves performance comparable with LSTM models on small Penn Treebank datasets using relatively few parameters.For the same hidden-layer size, LSTM models have about 4x more parameters than SRNs.
- Penn Treebank results: Perplexity falls from 136 (SRN) to 129 with leaky neurons, whereas SCRN reduces it from 129 (SRN) to 115 on the same dataset.The reported comparison favors SCRN over the leaky-neuron approach.
- Penn Treebank results: SCRN outperforms SRN with fewer parameters: 40 hidden and 10 contextual units obtain test perplexity 127 versus 129 for SRN with 300 hidden units.The authors interpret this as evidence that recurrent-matrix structure helps the learning algorithm capture additional information.
- Text8 results: On Text8, adding 40 contextual units to an SRN with 100 hidden units reduces perplexity from 245 to 189, outperforming a 300-hidden-unit SRN at 202.This corresponds to a reported 23% reduction.
- Text8 results: On larger Text8 models, the best LSTM is slightly better than the best SCRN, with perplexities 156 and 161, respectively.For smaller models, SCRN with 100 hidden and 80 contextual features performs better than LSTM with 100 hidden units.
4 CONCLUSION
The paper shows that recurrent networks can learn longer-term patterns with a structural constraint on the recurrent weight matrix, while retaining standard stochastic gradient descent. SCRN matches LSTM behavior in two language-modeling tasks, performs especially well under model-size constraints, and does not provide truly unlimited memory.
- 4 CONCLUSION: A structural constraint on the recurrent weight matrix enables standard stochastic gradient descent to learn longer-term patterns.The model separates rapidly changing hidden states for short-term patterns from a slowly updating context layer retaining longer-term information.
- 4 CONCLUSION: SCRN shows very similar behavior to LSTM in two language-modeling tasks, with similar gains over simple recurrent networks when tuned for accuracy.
- 4 CONCLUSION: With similar numbers of parameters, SCRN often outperforms LSTM by a large margin when model size is constrained.This advantage is especially relevant when training data is practically unlimited and even models with thousands of hidden neurons severely underfit.
- 4 CONCLUSION: The findings are intended to improve understanding of longer-term memory in sequential data and simplify recurrent-network analysis and implementation.The authors also published code for reproducing the experiments.
- 4 CONCLUSION: None of the discussed models can learn truly long-term memory because finite-capacity models cannot store and later reproduce arbitrarily long symbol sequences.Using a recurrent network to control unlimited external memory is proposed as a possible direction, but substantial research remains.