Source-linked AI summary
Grid Long Short-Term Memory
Nal Kalchbrenner, Ivo Danihelka, Alex Graves
TL;DR
The paper addresses limitations of deep and recurrent computation by introducing Grid LSTM, which places LSTM cells across data and depth dimensions. It evaluates this unified architecture on algorithmic and empirical tasks, reporting stronger performance across the tested settings, including 1.47 bits per character on Wikipedia and better Chinese-to-English translation than a phrase-based reference.
Problem
Deep and recurrent networks can suffer vanishing gradients and limited input selection, motivating a unified way to extend LSTM to deep computation.
Method
Grid LSTM arranges LSTM cells in one or more dimensions, including network depth, and modulates their multi-way interaction.
Results
Grid LSTM outperforms standard LSTM on the reported algorithmic tasks and achieves 1.47 bits-per-character on Wikipedia while outperforming a phrase-based Chinese-to-English translation reference.
Takeaways & Limitations
The architecture provides a unified approach for applying LSTM to feed-forward, recurrent, sequential, translation, image, and character-prediction computations.
Takeaways & Limitations
The addition experiment fixes inputs at 15 digits, omits curriculum learning, and requires the network to remember partial predictions.
Abstract
from arXiv · showhide
This paper introduces Grid Long Short-Term Memory, a network of LSTM cells arranged in a multidimensional grid that can be applied to vectors, sequences or higher dimensional data such as images. The network differs from existing deep LSTM architectures in that the cells are connected between network layers as well as along the spatiotemporal dimensions of the data. The network provides a unified way of using LSTM for both deep and sequential computation. We apply the model to algorithmic tasks such as 15-digit integer addition and sequence memorization, where it is able to significantly outperform the standard LSTM. We then give results for two empirical tasks. We find that 2D Grid LSTM achieves 1.47 bits per character on the Wikipedia character prediction benchmark, which is state-of-the-art among neural approaches. In addition, we use the Grid LSTM to define a novel two-dimensional translation model, the Reencoder, and show that it outperforms a phrase-based reference system on a Chinese-to-English translation task.
1 INTRODUCTION
Grid LSTM extends LSTM to deep computation by placing LSTM cells along the data dimensions and the network’s depth dimension. Across algorithmic and empirical tasks, it reports stronger performance than standard stacked LSTM and other neural or phrase-based references.
- Deep networks face vanishing gradients and cannot dynamically select or ignore inputs, motivating LSTM-like gating for deep computation.
- Grid LSTM arranges LSTM cells in a one-or-more-dimensional grid, treating network depth like the other dimensions for direct layer-to-layer communication.
- N-dimensional Grid LSTM supports feed-forward and recurrent applications, with one-dimensional versions replacing transfer functions such as tanh and ReLU.
- Grid LSTM differs from related stacked and multidimensional LSTM models by adding depth cells and a mechanism for modulating N-way interaction.
- Grid LSTM outperforms standard stacked LSTM on 15-digit addition and sequence memorization, with depth cells and tied weights more effective than their alternatives.
- 1.47 bits-per-character is achieved on Wikipedia character prediction, while the Reencoder outperforms a phrase-based CDEC reference on Chinese-to-English translation.
2 BACKGROUND
The background reviews standard LSTM gating, stacked LSTM, and multidimensional LSTM before motivating Grid LSTM’s alternative treatment of multidimensional memory updates. These models differ in how recurrent state and memory are propagated across temporal, spatial, and depth dimensions.
- 2.1 LONG SHORT-TERM MEMORY: Standard LSTM processes input-target sequences by updating a hidden vector and memory vector, then estimating each target from the new hidden state.
- 2.1 LONG SHORT-TERM MEMORY: LSTM gates control memory deletion, writing, and reading, helping preserve signals, reduce vanishing gradients, and attend to selected inputs across steps.
- 2.1 LONG SHORT-TERM MEMORY: Stacked LSTM adds capacity by stacking layers, but lacks LSTM cells in the vertical computation from one layer to the next.
- 2.3 MULTIDIMENSIONAL LSTM: Multidimensional LSTM propagates hidden and memory vectors across an N-dimensional input grid, combining multiple incoming states at each location.
- 2.3 MULTIDIMENSIONAL LSTM: Because grid paths grow combinatorially, unconstrained memory summation can cause instability, especially when depth adds another dimension.
3 ARCHITECTURE
Grid LSTM places LSTM cells along temporal, depth, and other data dimensions, using multidimensional blocks to transform and route hidden and memory states. Its design also supports non-LSTM dimensions, multiple input sides, and weight sharing across grid directions.
- Grid LSTM architecture: Grid LSTM places LSTM cells along any or all dimensions, including network depth, to connect computation across temporal and vertical directions.For sequence prediction, the two dimensions are the sequence's temporal axis and the network's depth axis.
- Grid LSTM blocks: Each N-dimensional block receives N hidden and memory vectors, then produces N distinct output hidden and memory vectors.The block computes one LSTM transform per dimension, with shared concatenated hidden input but dimension-specific memory inputs and transforms.
- Grid LSTM blocks: Blocks concatenate hidden vectors from all dimensions before applying the dimension-specific LSTM transforms.The resulting shared vector H contains the input hidden vectors, while the respective memory vectors are not directly combined.
- Priority dimensions: A block can prioritize one dimension by first computing the other dimensions' transforms, then using their outputs with that dimension's input.The prioritized final transform produces the selected dimension's output hidden and memory vectors.
- Non-LSTM dimensions: A grid dimension may use an ordinary nonlinear or identity transformation instead of LSTM cells, recovering Stacked LSTM as a special case.For temporal sequences, a 2D Grid LSTM with temporal cells but no depth cells corresponds to Stacked LSTM; analogous reductions apply to multidimensional LSTM.
- Inputs and weight sharing: Different input types can enter through different grid sides, while weight sharing can be specified along any dimension to induce computational invariance.The translation model projects source and target words onto separate sides, and sharing across all dimensions defines a Tied N-LSTM.
4 EXPERIMENTS
Experiments evaluate Grid LSTM on algorithmic tasks, character prediction, and translation. Across these settings, depth-direction cells, weight tying, and two-dimensional source-target processing are associated with strong learning or task performance.
- 4.1 ADDITION: 15-digit addition is evaluated without curriculum learning or feeding partially predicted digits back into the network.Inputs are presented one digit at a time, and outputs contain 15 or 16 digits.
- 4.1 ADDITION: Less than 550K training samples were needed for the best tied 2-LSTM to solve addition perfectly.The best model was 18 layers deep; the best untied 2-LSTM reached 67% per-digit accuracy after 5 million examples.
- 4.2 MEMORIZATION: Less than 150K samples were needed by the 43-layer tied 2-LSTM to solve the 20-symbol memorization task.Deeper networks tended to learn faster, while Stacked LSTM networks above 16 layers did not exceed 50% accuracy.
- 4.3 CHARACTER-LEVEL LANGUAGE MODELLING: 1.47 bits-per-character was achieved on the 100M-character Wikipedia dataset by a tied 2-LSTM.The model used 1,000 hidden units and six depth layers, and significantly outperformed other models despite having fewer parameters.
- 4.4 TRANSLATION: The Reencoder repeatedly re-encodes the source sentence conditioned on generated target words, implementing translation as a two-dimensional mapping.The source and target dimensions share weights, and the resulting model has O(nm) complexity.
- 4.4 TRANSLATION: Perplexity of 4.54 was obtained by the best translation model, which significantly outperformed the CDEC baseline on validation and test data.The model uses bidirectional processing across two grids and averages output probabilities across seven models.
5 CONCLUSION
The paper introduces Grid LSTM, which uses LSTM cells across grid dimensions and modulates their multi-way interaction. It reports advantages over regular connections across algorithmic and empirical tasks.
- Grid LSTM places LSTM cells along all grid dimensions and uses a novel mechanism to modulate multi-way interaction.
- The model shows advantages over regular connections on parity, addition, and memorization tasks.
- Grid LSTM supports character prediction, machine translation, and image classification with strong reported performance.
APPENDIX
The appendix extends Grid LSTM evaluation to parity and MNIST image classification. One-dimensional models learn parity for up to 250 bits, while three-dimensional image models perform near state-of-the-art without pooling.
- 5.1 PARITY: Parity learning is difficult because changing one input bit changes the target and produces highly non-linear decision boundaries.
- 5.1 PARITY: 1-LSTM networks learn k-bit parity for up to k = 250 bits, whereas feed-forward ReLU and tanh networks fail at k = 35 bits and beyond.
- 5.1 PARITY: The minimum 1-LSTM depth increases sub-linearly with input length, and activation visualizations reveal counting neurons tracking processed layers.
- 5.2 MNIST DIGIT RECOGNITION: 3-LSTM applies shared three-way transforms across non-overlapping image patches, with LSTM cells spanning both spatial dimensions and network depth.
- 5.2 MNIST DIGIT RECOGNITION: The 3-LSTM with depth-direction cells performs near state-of-the-art on MNIST despite using no pooling.