Rank and run-time aware compression of NLP Applications

1 Introduction

• Edge devices’ lower-bandwidth memory and smaller caches motivate compression, but compressed models must also preserve inference runtime for real-time tasks.The introduction notes increasing use of sequence-based NLP applications on phones and smart watches.Source: §1 · p. 1
• Pruning can yield sparse matrices inefficient on most modern CPUs, while LMF preserves dense runtime benefits but can lose accuracy as reduced rank limits expressibility.Structured-matrix computation reductions translate to runtime gains only for large matrices or specialized hardware; HMF reports 30× speed-up over that technique on the paper’s benchmarks.Source: §1 · p. 1
• Compression choices should balance compression factor, inference runtime speedup, and accuracy rather than improve one axis at another’s expense.The passage contrasts pruning, low-rank matrix factorization, and structured matrices as having different trade-offs across these criteria.Source: §1 · p. 1
• 2.32× faster CPU runtime than pruning, with 16.77% better model accuracy than LMF, is reported alongside iso-accuracy at 2× to 4× compression and 9% better accuracy than smaller baselines.HMF is proposed for NLP edge use cases on embedded CPUs.Source: §1 · p. 2

2 Related Work

• Block-based pruning still requires a certain amount of block sparsity to run faster than baseline, whereas HMF avoids a strict compression-factor requirement for better runtime.The passage describes pruning as a successful compression technique across neural networks (Han et al., 2016; Zhu and Gupta, 2017).Source: §2 · p. 2
• HMF is presented as superior to block circular decomposition, an extension of structured-matrix compression that converts each matrix block into a structured matrix.The passage cites structured-matrix compression work (Sindhwani et al., 2015; Ding et al., 2018; Cheng et al., 2015; Thakker et al., 2020).Source: §2 · p. 2
• HMF is presented as improving accuracy over LMF-compressed RNNs, within a related-work discussion that also lists CP, Kronecker, and Tucker tensor decompositions.The passage cites tensor-decomposition and matrix-factorization methods (Tjandra et al., 2017; Thakker et al., 2019c,a; Kuchaiev and Ginsburg, 2017; Chen et al., 2018; Grachev et al., 2017; Thakker et al., 2019).Source: §2 · p. 2
• Quantization can further compress HMF networks, and HMF is described as orthogonal to dynamic RNN methods that skip selected state updates or inputs.The dynamic methods use a small predictor to learn which inputs and associated computations to skip.Source: §2 · p. 2
• HMF can further optimize the matrices in hand-crafted LSTM/GRU structures such as SRU, QRNN, and PRU to reduce parameters and runtime.These structures are described as distinct from structured matrices and designed after understanding the application domain.Source: §2 · p. 2
• Embedding-footprint reduction techniques can further optimize HMF-based RNNs, and the paper reports compressing networks with compressed word-embedding layers.The passage cites embedding-compression work (Acharya et al., 2019; Mehta et al., 2019).Source: §2 · p. 2

3.1 Why LMF can potentially lead to loss in accuracy

• LMF can lose accuracy because its lower-rank representation reduces matrix expressibility and projects outputs through a lower-dimensional embedding.For a 256 × 256 matrix, 2× compression yields a rank-64 matrix, reducing rank by a factor of four.Source: §3.1 · pp. 2–3

3.2 Hybrid Matrix Factorization

• HLF’s design assumes matrix rank matters for accuracy, mixed-dimensional output features can improve on LMF’s constraint, and output-element order is not strictly important.The latter assumption relies on subsequent fully connected or recurrent layers adjusting to the reordered output.Source: §3.2 · p. 3
• HLF splits an RNN weight matrix into a fully parameterized upper part and a low-rank lower part, with j and k controlling compression and rank.The lower part comprises k rank-1 blocks; HLF preserves a dense matrix representation for fast run-time properties.Source: §3.2 · pp. 3–4

3.3 Why HLF leads to larger rank than LMF for same number of parameters?

• HLF can double matrix rank versus iso-parameter LMF by using a sparsity-forcing mask to reduce the parameters needed for the higher-rank representation.Table 1 compares maximum possible rank across three techniques and four compression factors.Source: Table 1 · p. 4, §3.2 · p. 4, §3.3 · p. 5

3.2 Hybrid Matrix Factorization

• At batch size one, HLF uses matrix-product associativity to compute the matrix-vector product without expanding the component matrices into the original matrix.The operation count is j×n + k×n + k×(m−j), compared with m×n for the uncompressed product.Source: §3.2 · p. 4

3.2.1 Impact on output feature vector

• HLF divides the output into stacked sub-vectors: richer features from a higher-dimensional embedding and constrained features from a lower-dimensional embedding.The paper argues subsequent RNN or softmax layers can accommodate the restructuring.Source: §3.2.1 · p. 4

4 Results

• The comparison includes HLF, LMF, pruning, a smaller baseline, and block circular decomposition.Pruning is included despite sparse-matrix traversal overhead that can hurt CPU inference; the smaller baseline matches the compressed model's parameter count.Source: §4 · p. 5
• The smaller baseline tests whether compressing a larger network outperforms reducing its dimensions and whether HLF's lower-dimensional output features add useful information.Its output features come from a high-dimensional embedding, whereas HLF also concatenates features from a lower-dimensional embedding.Source: §4 · p. 5
• Block circular decomposition results are omitted because its inference slowdown was significant.Source: §4 · p. 5

4.1 Experiment Setup

• Runtime measurements use C++ Eigen implementations of RNN cells and assume batch size 1 for edge-device inference.The authors state that runtime observations should remain consistent for larger batches.Source: §4.1 · pp. 5, 7
• Figure 2 compares speed-up over baseline with accuracy across compression schemes and compression factors; the top-right choice is preferred.Speed-up above 1 means reduced inference run-time, while below 1 means increased run-time; perplexity is plotted so lower values remain preferable.Source: Figure 2 · p. 6
• The experiments compress LSTM/GRU layers, not embedding layers, and compare accuracy and inference run-time at equal compression factors.Training uses TensorFlow 1.14 on a cluster with two RTX 2080 Ti GPUs.Source: §4.1 · p. 7
• For HLF, parameters j and k control matrix rank, and a sweep starting at k=1 selects values for good accuracy.Source: §4.1 · p. 7

4.2.1 Language Translation

• HLF improves BLEU over LMF by 2.3% to 4.5% and over Small Baseline by 2.8% to 4.1%, while improving inference runtime over pruning by 1.5× to 1.74×.The English-to-Vietnamese model uses two 512-unit LSTM layers, a bidirectional encoder, 512-dimensional embeddings, and attention; HLF uses k=2.Source: §4.2.1 · p. 7

4.2.2 Language Modeling

• 17% better perplexity than LMF, 9% better perplexity than small baseline, and 2.32× better inference run-time than pruning are reported for PTB Medium.Across compression factors, HLF also has better perplexity than LMF and faster run-time than baseline and pruned networks.Source: §4.2.2 · p. 7

4.2.3 Text Classification

• HLF improves text-classification accuracy over LMF by up to 1.2% and over Small Baseline by up to 1.3%, and improves run-time over pruning by almost 1.20×.The SemEval-2010 model is a bidirectional LSTM with hidden size 256.Source: §4.2.3 · pp. 7–8

4.2.4 Intent Detection and Slot Filling

• For slot filling, HLF improves F1-accuracy over LMF and Small Baseline by up to 1.2% and run-time over pruning by up to 1.26×; intent accuracy improves by up to 1%.The ATIS benchmark jointly trains intent detection and slot filling using one 128-unit LSTM layer with attention; the shared network gives the same runtime improvement for both tasks.Source: §4.2.4 · p. 8

4.3.1 Compressing Word Embedding layers using HLF

• 8% loss in perplexity results when HLF compresses the PTB-LM input word-embedding layers by 3×, with other layers left uncompressed.Source: §4.3.1 · p. 8

4.3.2 Orthogonality of word embedding compression methods and HLF

• After pruning embeddings by 2×, another 2× HLF compression produces only a 1-point perplexity loss, indicating compatibility between the methods.The LSTM layers remain uncompressed in this experiment.Source: §4.3.2 · p. 8

• HLF is an alternative when pruning does not provide the required run-time benefit and LMF loses accuracy.Source: §5 · p. 8
• HLF has better accuracy than LMF for most evaluation points, supporting the assumption that matrix rank matters for NLP task accuracy.Source: §5 · p. 8
• HLF has better accuracy than the smaller baseline, supporting the importance of constrained features alongside richer features.Source: §5 · p. 8

• The RNN assumptions underlying HLF are not valid for the final classification layer.Source: §6 · p. 8
• In classification layers, HLF gives more expressive outputs to some classes and less expressive outputs to the rest.Source: §6 · p. 8
• HMD is training-aware compression and cannot be applied to a pre-trained network.Source: §6 · p. 8

• 2× faster inference run-time than pruning and up-to 16% better accuracy than LMF are reported for HLF.Source: §7 · p. 8
• HLF preserves the dense structures of LMF while effectively doubling matrix rank through an intelligent structure by design.Source: §7 · p. 8
• The conclusion says compression choices should consider compression factor, accuracy, and run-time together.Source: §7 · p. 8