Source-linked AI summary
Factorization tricks for LSTM networks
Oleksii Kuchaiev, Boris Ginsburg
TL;DR
Large LSTMs can have millions of parameters and require weeks of multi-GPU training. This paper proposes factorized and independent-group projected LSTM cells to reduce parameters and speed training. After one week, the G-LSTM and F-LSTM reached evaluation perplexities of 36 and 36.3, respectively, versus BIGLSTM's 35.1, while using two to three times fewer RNN parameters.
Problem
LSTMs can have millions of parameters and require weeks of training on multi-GPU systems.
Source: §1 · p. 1Method
The paper reduces projected LSTM parameters by factorizing its affine matrix or partitioning inputs and states into independent groups.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
After one week, BIGLSTM's evaluation perplexity was 35.1, compared with 36 for G-LSTM and 36.3 for F-LSTM, which used two to three times fewer RNN parameters.
Source: §1 · p. 1Takeaways & Limitations
Within the one-week training budget, G-LSTM trained faster and achieved similar evaluation perplexity while using almost 2 times fewer RNN parameters than BIGLSTM.
Source: §3 · p. 3Initial experiments found no immediate benefit from approximating transform T with an arbitrary feed-forward neural network.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.
1 INTRODUCTION
LSTMs are effective across several tasks, but their millions of parameters make training costly. This paper introduces factorized and grouped projected LSTM cells, focusing on reducing parameters in the affine transform that dominates computation.
1 INTRODUCTION
- LSTMs have millions of parameters and can require weeks of training on multi-GPU systems.Source: §1 · p. 1
- The paper introduces factorized and grouped projected LSTM cells to reduce parameters and speed training.Related work discusses grouping in convolutional networks and factorization of large convolutional layers into smaller ones.Source: §1 · p. 1, §1.2 · p. 2
1.1 LONG SHORT-TERM MEMORY OVERVIEW
- Projected LSTM computes its gates with an affine transform using a 4n × 2p matrix, motivating the focus on reducing that matrix's parameters.The next state is projected to size p, while the memory has size n.Source: §1.1 · p. 1
2 MODELS
The paper proposes two ways to reduce the projected LSTM's affine-transform parameters: approximate its matrix with a low-rank product or partition computation into independent groups.
2.1 FACTORIZED LSTM CELL
- F-LSTM approximates W as a product of smaller matrices, assuming W is well approximated by rank r; this reduces parameter count and can speed computation and synchronization.Its factors have sizes 2p × r and r × 4n, with r < p ≤ n.Source: §2.1 · p. 2
2.2 GROUP LSTM CELL
- G-LSTM partitions inputs and hidden states into independent feature groups, with each group's transform operating on its own input and state.For k groups, each transform maps R^(2p/k) to R^(4n/k).Source: §2.2 · p. 2
- The group computations are independent, making the architecture suited to model parallelism; unlike an ensemble, group outputs are concatenated rather than combined as ensemble outputs.Source: §2.2 · pp. 2–3
3 EXPERIMENTS AND RESULTS
On One Billion Word language modeling, the grouped and factorized LSTMs train faster than the baseline under a one-week budget, with the G-LSTM achieving similar perplexity using fewer RNN parameters. Exploratory results also report similar perplexities for a hierarchical-group variant, while arbitrary feed-forward approximations remain future work.
3 EXPERIMENTS AND RESULTS
- Models were trained for one week on four GPUs, using Adagrad, projection size 1024, cell size 8192, and sampled softmax with 8192 samples.Source: §3 · p. 3
- BIG G-LSTM-G2 uses almost 2 times fewer RNN parameters than BIGLSTM, trains faster, and achieves similar evaluation perplexity within the one-week budget.At the same step count, the model with more parameters has lower training loss.Source: §3 · p. 3
3.1 FUTURE RESEARCH
- Initial experiments found no immediate benefit from approximating transform T with an arbitrary feed-forward neural network, leaving that approach for future research.Source: §3.1 · p. 3
- In a smaller experiment, BIG G-LSTM-G4-G8 achieved very similar perplexities to BIG G-LSTM-G4, ran faster, and got better perplexity.The variant replaces the second layer's four groups with eight.Source: §3.1 · p. 3
APPENDIX: TRAINING LOSS FOR 4 LSTM-LIKE MODELS
Figure 2 compares training loss against step count and wall time for four LSTM-like models trained for one week. More-parameter models perform better at equal step counts, while factorized models achieve better results in the same time by completing more iterations.
APPENDIX: TRAINING LOSS FOR 4 LSTM-LIKE MODELS
- At equal training time, factorized models complete more iterations and reach better results than the other models.All four models were trained for exactly one week.Source: Figure 2 · p. 6
- At the same step count, the model with more parameters performs better.Source: Figure 2 · p. 6
- Panel A plots log-scale training loss against steps or mini-batch count, while panel B plots it against hours of wall time.Source: Figure 2 · p. 6