Learning Intrinsic Sparse Structures within Long Short-Term Memory

1 INTRODUCTION

• The work targets compact DNN structures for efficient inference, situated at the intersection of model compression and structure learning.Prior compact-structure learning had been widely explored for CNNs.Source: §1 · p. 1
• Because recurrent units are shared across sequence steps, connection pruning leaves their basic structures unchanged, while irregular sparsity can hinder efficient hardware computation.The introduction also notes that CPU speed gains from non-structured sparsity appear only beyond 80% sparsity.Source: §1 · pp. 1–2
• At 95% sparsity, non-structured CPU matrix multiplication achieved only about 3×–4× speedup; Figure 1 contrasts this with equal-sparsity structured row and column removal.The figure compares random sparsity with structured sparsity using removed rows and columns in W.Source: §1 · p. 2, Figure 1 · p. 2
• ISS groups recurrent structures so removing one component reduces their dimensions together, avoiding the mismatch that can make independently pruned recurrent units invalid.The introduction describes the resulting models as retaining the original schematic with smaller basic structures.Source: §1 · p. 2
• 10.59× inference speedup was achieved with unchanged perplexity using learned LSTM sizes of 373 and 315, rather than the baseline size of 1500.Directly training LSTMs at sizes 373 and 315 did not achieve the same perplexity.Source: §1 · p. 2
• The evaluation includes recurrent highway networks and the BiDAF question-answering model in addition to LSTM language modeling.The introduction identifies the RHN and BiDAF models as state-of-the-art models used in the evaluation.Source: §1 · pp. 2–3

2 RELATED WORK

• DNN compression studies include removing redundant structures, approximating the original function, and designing inherently compact structures; this method belongs to the first category.Source: §2 · p. 3
• FNN and CNN compression used connection pruning and group Lasso to remove coarse structures, while prior RNN pruning focused on connections and coarse-grained RNN structure reduction remained little explored.Source: §2 · p. 3
• Structure-learning methods explored FNN or CNN architectures for classification accuracy, whereas this work learns compact LSTM structures for model compression.Source: §2 · p. 3

3.1 INTRINSIC SPARSE STRUCTURES

• Removing one ISS component reduces the dimensions of all basic LSTM structures together, preserving their dimension consistency.The tied structures include input updates, gates, hidden states, cell states, and outputs.Source: §3.1 · pp. 3–4
• Each ISS component corresponds to an ISS weight group whose associated rows and columns can be zeroed to remove that component.The grouping follows connections to and from the hidden state, gates, cell state, and input updates; ISS variants can also be realized for vanilla RNNs, GRUs, and RHNs.Source: §3.1 · p. 4
• Independent ℓ1 regularization can already reveal substantial intrinsic ISS sparsity, while the proposed method explicitly targets removing more components.The reported intrinsic-sparsity observation is covered in Appendix A.Source: §3.1 · p. 4

3.2 LEARNING METHOD

• Group Lasso regularization is added to the loss to encourage ISS sparsity, with its coefficient trading off recognition accuracy and sparsity.Its gradient squeezes each component's weight-group length toward zero.Source: §3.2 · p. 5
• Because SGD updates can leave near-zero groups fluctuating, weights below threshold τ are zeroed after each mini-batch.A tiny ε is added to the norm to avoid division by zero in the regularization gradient.Source: §3.2 · p. 5

4 EXPERIMENTS

• The evaluation spans Penn TreeBank language modeling and SQuAD question answering, using LSTMs and RHNs, with both scratch training and fine-tuning.The experiments also use hidden sizes of 1,500 and 100 and select a sparsification threshold by cross-validation.Source: §4 · p. 5

4.1.1 STACKED LSTMS

• 10.59× inference speedup reduced the stacked-LSTM model from 66.0M to 21.8M weights, with ISS sizes of 373 and 315 versus 1,500.A smaller model achieved lower perplexity at 25.2M weights and 7.10× speedup; direct design at the same structure had test perplexity 85.66, 7.01 higher than the ISS method.Source: §4.1.1 · p. 6, Table 1 · p. 6

4.1.2 EXTENSION TO RECURRENT HIGHWAY NETWORKS

• RHN width fell from 830 to 517 without perplexity loss, reducing model size to 11.1M, a 52.8% reduction.A width-726 RHN also improved state-of-the-art perplexity; larger λ produced smaller widths but higher perplexity.Source: §4.1.2 · p. 7, Table 2 · p. 7

4.2 MACHINE READING COMPREHENSION

• BiDAF sparsification excludes contextual-embedding LSTMs and targets the other LSTM layers, which consume 76.47% of inference time.After training, weights with absolute values below 0.02 are zeroed without impacting EM and F1 scores, while increasing sparsity.Source: §4.2 · pp. 7–8
• With fine-tuning, stronger group Lasso increased BiDAF sparsity at an EM/F1 cost; a 2.63 F1-score loss reduced OutFwd and OutBwd ISS sizes to 15 and 12.The authors report difficulty reducing ISS sizes without losing EM or F1, describing BiDAF as compact enough for its accuracy and scale.Source: §4.2 · p. 8, Table 3 · p. 8
• Training BiDAF from scratch balanced sparsity across layers better than fine-tuning, yielding better model-size compression and inference speedup.The results are summarized in Table 4.Source: §4.2 · p. 8, Table 4 · p. 8

• ISS simultaneously reduces the sizes of LSTM input updates, gates, hidden states, cell states, and outputs.Source: §5 · p. 8
• The resulting structurally sparse LSTM is a regular LSTM with reduced hidden dimension, requiring no software- or hardware-specific customization for storage saving and computation acceleration.Source: §5 · p. 9
• The approach can also extend to vanilla RNNs, GRUs, and Recurrent Highway Networks.Source: §5 · p. 9

APPENDIX A ISS UNVEILED BY ℓ1-NORM REGULARIZATION

• Figure 5 contrasts original weight matrices with their ISS-format views, where white strips mark all-zero ISS components; the matrices are down-sampled 10 × 10 for display.Source: Figure 5 · p. 12, Appendix A · p. 12
• 82.40 validation and 78.60 test perplexity after ℓ1 fine-tuning approximately match the baseline values of 82.57 and 78.57, respectively.Source: Appendix A · p. 12
• 91.66% first-layer, 90.32% second-layer, and 90.22% output-layer weight sparsity accompanied the approximately unchanged perplexity.Source: Appendix A · p. 12
• 504 and 220 ISS components in the first and second LSTM layers, respectively, were all-zero after independently sparsifying weights.Source: Appendix A · p. 12

APPENDIX B ISS IN BIDAF

• Table 5 presents the ISS in BiDAF.Source: Table 5 · p. 13

APPENDIX C HISTOGRAM OF VECTOR LENGTHS OF ISS WEIGHT GROUPS IN BIDAF

• Figure 6 plots BiDAF ISS weight-group vector lengths; a peak at zero indicates high ISS sparsity.Source: Figure 6 · p. 14