Source-linked AI summary
Multiplicative LSTM for sequence modelling
Ben Krause, Liang Lu, Iain Murray, Steve Renals
TL;DR
The paper addresses limits of fixed recurrent transitions in generative sequence models, especially their difficulty representing input-specific changes while retaining information. It introduces mLSTM, combining mRNN factorized transitions with LSTM gating, and reports improvements over LSTM baselines across character-level tasks. The discussion also leaves word-level, continuous-input, and broader discrete-task performance as open questions.
Problem
Fixed-transition generative RNNs can struggle to recover from surprising inputs, motivating more flexible input-dependent transitions for sequence density estimation.
Method
mLSTM combines mRNN factorized hidden-to-hidden transitions with LSTM gating for discrete multinomial sequence modelling.
Results
mLSTM improved over LSTM and its deep variants across character-level language-modelling experiments and performed competitively with word-level models on WikiText-2.
Takeaways & Limitations
Flexible input-dependent transitions can achieve competitive character-level language modelling without nonlinear recurrent depth, while remaining more easily parallelizable than deeper recurrent approaches.
Takeaways & Limitations
Performance remains to be established for word-level modelling, other discrete multinomial tasks, and continuous or non-sparse inputs.
Abstract
from arXiv · showhide
We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level language modelling tasks. In this version of the paper, we regularise mLSTM to achieve 1.27 bits/char on text8 and 1.24 bits/char on Hutter Prize. We also apply a purely byte-level mLSTM on the WikiText-2 dataset to achieve a character level entropy of 1.26 bits/char, corresponding to a word level perplexity of 88.8, which is comparable to word level LSTMs regularised in similar ways on the same task.
1 INTRODUCTION
RNNs estimate sequence densities by recursively updating hidden states, but fixed transitions can limit distinct representations and recovery from surprising inputs. The paper motivates input-dependent transitions and combines multiplicative-RNN transitions with LSTM gating in mLSTM.
- Sequence density estimation: RNNs factorize sequence probabilities into next-element conditionals and use hidden states to summarize previous inputs.The hidden state is updated from the previous hidden state and current input, then mapped to a next-element distribution.
- Motivation: Generative RNNs evaluate sequence log-likelihoods exactly, but training can suffer from vanishing gradients and recurrent models can struggle to recover from mistakes.Limited hidden-state capacity forces the model to decide what prior information to store at every step.
- Input-dependent transition functions: Input-dependent transition functions can vary hidden-state transformations across possible inputs, widening the representational tree without overwriting as much past information.This is proposed to support rapid adjustment to surprising inputs while retaining longer-term information.
- Multiplicative RNN: mRNNs approximate tensor-RNN input-specific transitions with factorized hidden-to-hidden weights, but their standard units may make long-term information difficult to retain.For one-hot language-model inputs, the selected input determines the corresponding transition matrix in the tensor-RNN view.
- Long short-term memory: LSTM uses input, output, and forget gates to control information written, preserved, and exposed across time steps.Its gating framework is designed to regulate information flow through internal states.
- Multiplicative LSTM: mLSTM combines mRNN factorized transitions with LSTM gating, targeting controlled and flexible input-dependent transitions for discrete multinomial sequences.The hybrid adds the mRNN intermediate state to the LSTM gating units.
2 MULTIPLICATIVE LSTM
The multiplicative LSTM combines mRNN factorized hidden-to-hidden transitions with LSTM gating to support flexible input-dependent transitions while retaining long-term information control.
- mLSTM combines mRNN factorized hidden-to-hidden transitions with the gating framework of LSTMs.
- The architecture connects the mRNN intermediate state to each LSTM gating unit.
- The implementation shares the intermediate state across LSTM unit types and uses 1.25 times as many recurrent weights as an equal-width LSTM.
- LSTM input and forget gates allow flexible transitions while controlling or bypassing complex factorized hidden-state transitions.
3 RELATED APPROACHES
Related approaches increase transition flexibility through recurrent depth or multiplicative integration, while mLSTM emphasizes input-dependent flexibility independently of nonlinear recurrent depth.
- Recurrent depth adds nonlinear transformations between recurrent steps, enabling more flexible combinations of inputs and previous hidden states.
- mLSTM prioritizes maximizing input-dependent transition flexibility independently of nonlinear recurrent depth.
- Multiplicative integration RNNs use Hadamard products when combining input and hidden contributions, whereas mLSTM applies the product between matrix multiplications.
4 EXPERIMENTS
The experiments evaluate mLSTM against LSTM variants across character-level language-modelling datasets, using both initial comparisons and more standard training and regularisation methods. Results show broad performance gains, while also exposing dependence on regularisation and differences across architectures.
- Experimental design: Experiments compare mLSTM with regular, stacked, and other recurrent language models across text8, Hutter Prize, and WikiText-2.The study includes initial convergence and performance comparisons plus follow-up training and regularisation experiments.
- Hutter Prize dataset: mLSTM improves over 2-layer stacked LSTM across network sizes from about 3–20 million parameters on Hutter Prize.The comparison used constant hyperparameters across model sizes and measured test performance after training on the first 95 million characters.
- Hutter Prize dataset: 1.42 bits/char versus 1.53 bits/char overall, mLSTM’s advantage over stacked LSTM grows after surprising inputs.Losses one to four steps afterward were lower for mLSTM: (2.26, 2.04, 1.61, 1.51) versus (2.48, 2.25, 1.79, 1.67) bits per character.
- Hutter Prize dataset: Regularisation materially affects Hutter Prize results: weight normalization and embeddings hurt without regularisation, but the combination with variational dropout outperformed previous static single-model neural-network results.The authors did not test variational dropout on mLSTM without weight normalization.
- Hutter Prize dataset: 1.40 bits/char for mLSTM versus 1.53 bits/char for MI-LSTM under the same settings, although the comparison is not conclusive.MI-LSTM also converged more slowly, demonstrating substantially different outcomes under similar hyperparameter settings.
- Text8 dataset: On text8, well-regularised mLSTM tied recurrent highway networks with recurrent depth 10 for the dataset’s best result.The model reused the Hutter Prize architecture and hyperparameters, with around 45 million parameters after adapting the input character set.
5 DISCUSSION
The discussion attributes mLSTM’s performance to flexible input-dependent transitions combined with LSTM information control, while noting unresolved scope beyond the demonstrated settings.
- 5 DISCUSSION: mLSTM combines mRNN factorized hidden weights with LSTM units for generative modelling of discrete multinomial sequences.The design aims to support controlled, flexible input-dependent transitions without erasing stored information.
- 5 DISCUSSION: mLSTM improved over LSTM and deep variants across character-level language modelling experiments.Variationally regularised mLSTM outperformed previous neural models on Hutter Prize and tied the best previous text8 result.
- 5 DISCUSSION: mLSTM uses only two linear recurrent transition matrices rather than nonlinear recurrent depth of up to ten in comparable approaches.The authors state that this makes mLSTM more easily parallelizable and suggests large recurrent depth is unnecessary for competitive character-level modelling.
- 5 DISCUSSION: The authors hypothesize that distinct transition functions for each possible input explain mLSTM’s success and provide greater efficiency than recurrent depth.The efficiency comparison is presented as a hypothesis about the mechanism behind the results.
- 5 DISCUSSION: It remains unresolved whether mLSTM performs well for word-level modelling, other discrete multinomial tasks, or continuous and non-sparse inputs.These settings define the paper’s stated boundary for future evaluation and broader formulation.