Source-linked AI summary
Run-Time Efficient RNN Compression for Inference on Edge Devices
Urmish Thakker, Jesse Beu, Dibakar Gope, Ganesh Dasika, Matthew Mattina
TL;DR
RNNs can be large and compute-intensive, while edge devices have limited compute and storage and applications may impose run-time constraints. The paper introduces HMD, which divides weight matrices into an unconstrained upper part and a rank-1-block lower part. It reports iso-accuracy at 2× to 3× compression, faster run-time than pruning, and better accuracy than LMF.
Problem
Edge applications need RNN compression that reduces model size without negatively affecting inference run-time or task accuracy.
Source: §1 · p. 1Method
HMD splits a weight matrix into an unconstrained upper part and a lower part composed of rank-1 blocks.
Source: §3 · p. 2Results
HMD reaches iso-accuracy at 2× to 3× compression, runs 2× faster than pruning, and achieves better model accuracy than LMF.
Source: §1 · p. 1Takeaways & Limitations
HMD is effective when pruning does not speed inference and LMF degrades accuracy.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Recurrent neural networks can be large and compute-intensive, yet many applications that benefit from RNNs run on small devices with very limited compute and storage capabilities while still having run-time constraints. As a result, there is a need for compression techniques that can achieve significant compression without negatively impacting inference run-time and task accuracy. This paper explores a new compressed RNN cell implementation called Hybrid Matrix Decomposition (HMD) that achieves this dual objective. This scheme divides the weight matrix into two parts - an unconstrained upper half and a lower half composed of rank-1 blocks. This results in output features where the upper sub-vector has "richer" features while the lower-sub vector has "constrained features". HMD can compress RNNs by a factor of 2-4x while having a faster run-time than pruning (Zhu &Gupta, 2017) and retaining more model accuracy than matrix factorization (Grachev et al., 2017). We evaluate this technique on 5 benchmarks spanning 3 different applications, illustrating its generality in the domain of edge computing.
1 Introduction
RNNs face pressure to run on edge devices, where storage, bandwidth, cache, and inference-time constraints make compression trade-offs important. The paper introduces HMD and compares its accuracy and run-time with other compression approaches.
1 Introduction
- Edge devices’ limited bandwidth and smaller caches motivate compression, while real-time deadlines make preserving inference run-time important.The paper identifies fitting large RNNs on edge devices or running them efficiently with smaller caches as a need.Source: §1 · p. 1
- Pruning can be inefficient on modern CPUs, LMF can reduce accuracy, and structured matrices may improve run-time only for larger matrices or specialized hardware.The introduction frames compression as a trade-off among compression factor, speedup, and accuracy.Source: §1 · p. 1
- HMD reaches iso-accuracy at 2× to 3× compression, runs 2× faster than pruning, and improves run-time over a structured matrix-based technique by 30×.The authors also report better model accuracy than low-rank matrix factorization (LMF).Source: §1 · p. 1
- The paper introduces HMD as a technique that can regain most baseline accuracy at 2× to 3× compression and compares its accuracy, inference run-time, and compression trade-offs with pruning and matrix factorization.Source: §1 · p. 2
2 Related Work
The paper groups neural-network compression research into four topics and situates HMD within structured matrix methods.
2 Related Work
- Neural-network compression is grouped into pruning [7, 21], structured matrix methods [3] [13] [14] [16], quantization, and tensor decomposition [8] [19].The paper compares HMD with pruning, structured matrix, and tensor decomposition methods, and describes quantization as orthogonal and additionally applicable.Source: §2 · p. 2
3 HMD-Based RNN Compression
HMD decomposes an RNN weight matrix into an unconstrained upper part and a constrained lower part made of rank-1 blocks. Its matrix-vector procedure avoids reconstructing the full matrix and exploits associativity to reduce computation.
3 HMD-Based RNN Compression
- RNN output elements need not all have dense connections, and later layers can adjust if hidden-layer output order changes; the paper uses these properties to motivate hardware-friendly compression.Source: §3 · p. 2
- D1 represents a matrix with an unconstrained upper part and a constrained lower part composed of two rank-1 blocks, reducing storage relative to the full matrix.The representation uses an upper matrix A′ and lower-block factors B, C, D, and E.Source: §3 · p. 2
- For batch-size-one inference, the matrix-vector algorithm avoids expanding A′, B, C, D, and E into A, using associativity to gain computation speedup.Source: §3 · p. 3
- HMD produces two stacked output subvectors: a fully parameterized upper part with richer features and a low-rank lower part with constrained features.Source: §3 · p. 3
4 Results
The section compares pruning, LMF, HMD, and parameter-matched smaller baselines across human activity recognition and language modeling. HMD often balances accuracy and runtime, though the preferred scheme varies with task and compression factor.
4 Results
4.1 Comparison of compression techniques across different ML tasks
- Evaluation covered HAR and language modeling, using C++/Eigen inference on one Cortex-A73 Hikey 960 core with 2 MB L3; parameter-matched smaller baselines were also trained.Source: §4.1 · p. 3
4.1.1 Human Activity Recognition (HAR)
- For HAR1, HMD leads at 2×–3.33×, including 2× faster runtime than pruning at 2×; at 5× it is slightly more accurate than LMF but 15% slower.All three compression schemes have better accuracy than the smaller baseline.Source: Figure 2 · p. 3, §4.1.1 · pp. 3–4
- At 2× and 2.5×, HAR2 HMD is 2× faster than pruning and improves accuracy by 0.4% over LMF; at 3.33×, LMF matches accuracy but runs faster.All three compression schemes outperform the smaller baseline on accuracy.Source: Figure 3 · p. 4, §4.1.1 · p. 4
4.1.2 Language Modeling
- Across language-model compression factors, HMD beats LMF on perplexity and is faster than baseline and pruning; it also beats baseline perplexity for most factors.Lower perplexity is better.Source: Figure 4 · p. 4, §4.1.2 · p. 4
5 Conclusion
Choosing a compression scheme requires balancing compression factor, accuracy, and runtime. HMD is presented as an option when pruning does not improve runtime and LMF degrades accuracy.
5 Conclusion
- Compression choice must balance compression factor, accuracy, and runtime.Source: §5 · p. 4
- Pruning can sacrifice speedup over baseline, while LMF can lead to accuracy degradation.Source: §5 · p. 4
- HMD is positioned for cases where pruning does not improve baseline runtime and LMF degrades accuracy.Source: §5 · p. 4