Source-linked AI summary
Pushing the limits of RNN Compression
Urmish Thakker, Igor Fedorov, Jesse Beu, Dibakar Gope, Chu Zhou, Ganesh Dasika, Matthew Mattina
TL;DR
RNNs can be difficult to deploy on constrained devices because of their size, creating a need for compression that preserves task accuracy. The paper compresses RNN cells with Kronecker products and reports 15×–38× compression, with higher accuracy than pruning and LMF across five IoT-domain benchmarks.
Problem
RNN layer size can hinder deployment on IoT devices, motivating compression that does not sacrifice task accuracy.
Source: §1 · p. 1Method
The paper replaces RNN-family cells with Kronecker-product equivalents and trains the model from the beginning.
Source: §3.3 · p. 4Results
KPRNNs compress cells by 15× to 38× and outperform pruning and LMF in accuracy across five IoT-domain benchmarks.
Source: §5 · p. 5Takeaways & Limitations
KPRNNs can act as drop-in replacements for most RNN layers with significant compression and marginal accuracy impact.
Source: §5 · p. 5Reported model size assumes 32-bit weights and excludes further opportunities from quantization and softmax-layer compression.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size. As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained environments using Kronecker product (KP). KPs can compress RNN layers by 16-38x with minimal accuracy loss. We show that KP can beat the task accuracy achieved by other state-of-the-art compression techniques (pruning and low-rank matrix factorization) across 4 benchmarks spanning 3 different applications, while simultaneously improving inference run-time.
1 Introduction
RNN layer size can hinder deployment on constrained IoT devices, motivating compression that preserves task accuracy. The paper proposes Kronecker products (KPs), after finding that pruning and low-rank matrix factorization (LMF) miss a 15× compression target without significant accuracy loss.
1 Introduction
- RNN layer size can prohibit deployment on IoT devices or reduce execution efficiency, motivating compression without sacrificing task accuracy.The passage notes that some IoT devices have 2KB of RAM and 32 KB of Flash Memory.Source: §1 · p. 1
- At a 15× or greater compression target, pruning and LMF do not avoid significant accuracy loss.The paper also examines rank and condition number as possible explanations for these methods’ shortcomings.Source: §1 · p. 1
- KPs compress RNN layers while preserving accuracy and inference time, addressing drawbacks of traditional compression methods.The paper reports state-of-the-art compression on IoT-targeted benchmarks.Source: §1 · p. 1
2 Related work
Prior work used KPs for fully connected layers and to stabilize RNN training. The paper distinguishes its work by learning KP factors directly for RNN compression, and limits comparisons to pruning and tensor decomposition.
2 Related work
- Unlike prior KP compression of AlexNet fully connected layers, this work learns KP factors directly to compress RNNs.The prior work cited does not examine compression’s impact on inference run-time.Source: §2 · p. 1
- The paper restricts comparisons to pruning and tensor decomposition because structured-matrix speedups are described for large matrices or specialized hardware.The passage identifies the speedup conditions as matrices of size 2048 × 2048 and larger on CPUs, or specialized hardware.Source: §2 · p. 2
3 Kronecker Product Recurrent Neural Networks
The section introduces Kronecker-product RNNs by explaining factorized matrix storage and contrasting prior stabilization work with a design that compresses RNN cell matrices. It describes factor selection and inference trade-offs, and reports benchmark comparisons favoring KP-based networks among compressed alternatives.
3.1 Background
- A matrix can be stored as two or more smaller Kronecker factors, with the product evaluated recursively; one example reduces parameter storage by 50×.For the example, A is 154 × 164 and its factors are 11 × 41 and 14 × 4.Source: §3.1 · p. 2
3.2 Prior work on using KP to stabilize RNN training flow
- Prior work used 2 × 2 factors only for hidden-hidden matrices to stabilize vanilla RNN training, achieving less than 2× compression after restoring baseline accuracy.This paper instead compresses both input-hidden and hidden-hidden matrices in vanilla RNN, LSTM, and GRU cells.Source: §3.2 · p. 2
3.3 KPRNN Layer
- Using more factors made training harder and could sharply reduce accuracy; for keyword spotting, 2 × 2 factors yielded about 2× fewer parameters but 4% lower accuracy than baseline.The authors choose factor dimensions for maximum compression using Algorithm 2.Source: §3.3 · pp. 3–4
- With more than two factors, inference can require expansion and run 2–4× slower than baseline, whereas two factors permit direct matrix-vector computation and inference speed-up.Algorithm 1 gives the two-factor matrix-vector procedure without expanding the factors into the larger matrix.Source: §3.3 · p. 3
- The evaluated benchmarks cover image classification, keyword spotting, and bidirectional LSTM applications, spanning varied applications and RNN cell types.Source: Table 1 · p. 4
- Table 2 reports KP-based networks as the most accurate compressed alternative across its benchmark rows while still achieving speed-up over the baseline.The table compares baseline networks with KP, magnitude pruning, LMF, and smaller-baseline variants.Source: Table 2 · p. 4
- KPRNN cells replace RNN, LSTM, and GRU matrices with Kronecker products of two smaller matrices, and train the replacement model from scratch.The described RNN form factorizes the matrix applied to the concatenated current input and previous hidden state.Source: §3.3 · p. 4
4 Results
The evaluation compares KPRNNs with pruning, low-rank matrix factorization, and a small baseline. It also examines possible explanations for accuracy differences among compression methods.
4.1 KPRNN networks
- The evaluation compares KPRNN with pruning, LMF, and a Small Baseline; Table 2 covers KP compression across applications and RNN cells.The experiments target maximum compression using two matrix factors.Source: §4 · p. 4, §4.1 · p. 5
4 Results
- The benchmarks span a wide variety of applications; training used TensorFlow 1.12 and four RTX 2080 GPUs, with inference measured in C++ on an Arm Cortex-A73.Inference measurements used the Eigen library on a HiKey 960 development board.Source: §4 · pp. 4–5
4.2 Possible explanation for the accuracy difference between KPRNN, pruning, and LMF
- LMF matrices generally have rank below 10, while KP factors are fully ranked and best-performing KP matrices have condition numbers from 1.2 to 7.3.At matching compression, pruning requires at least 94% sparsity; ill-conditioning may explain FastRNN pruning accuracy, but not other pruned networks.Source: §4.2 · p. 5
- For pruned KWS-LSTM, the largest singular value is 48 to 52, versus 19 for baseline and below 15 for KP-compressed layers.The authors suggest large singular values may cause saturation after the RNN matrix-vector product and sigmoid or tanh.Source: §4.2 · p. 5
5 Conclusion
Kronecker products compress RNN cells into KPRNNs, which can replace most RNN layers with marginal accuracy impact. Across five IoT-domain benchmarks, KPRNNs achieve compression and accuracy that other evaluated techniques do not match.
5 Conclusion
- KPRNNs compress RNN cells by 15× to 38× using Kronecker products.They can act as a drop-in replacement for most RNN layers with marginal impact on accuracy.Source: §5 · p. 5
- Pruning and LMF do not match the accuracy of Kronecker-compressed networks.Source: §5 · p. 5
- The method works across five benchmarks representing key applications in the IoT domain.Source: §5 · p. 5