RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition

I. INTRODUCTION

• Mobile RNN deployment faces computation and memory burdens, while non-structured and structured pruning respectively pose hardware-throughput and accuracy challenges.Source: §I · p. 1
• RTMobile combines block-based structured pruning with compiler-assisted performance optimization for mobile RNN inference.Source: §I · p. 1
• Block-based structured pruning uses finer pruning granularity to reduce RNN model size while maintaining inference accuracy and computational efficiency.Source: §I · pp. 1–2
• RTMobile is presented as the first framework to achieve real-time RNN inference on mobile devices.Source: §I · pp. 1–2
• Compiler optimizations include matrix reorder, load redundancy elimination, and BSPC, a compact storage format for pruned models.Source: §I · p. 2
• Over 10× compression without accuracy loss and about 50× higher energy efficiency at the same inference time are reported for RTMobile.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

• The background section covers GRU, DNN compression, mobile acceleration, and the paper’s research motivation.Source: §II · p. 2

A. Gated Recurrent Unit

• GRU combines the LSTM forget and input gates into an update gate and merges the cell state with the hidden state.Source: A. Gated Recurrent Unit · p. 2
• The paper focuses on GRU, whose update and output gates and cell and output states recur across time steps.Source: A. Gated Recurrent Unit · p. 2

B. DNN Model Compression Techniques

• Weight pruning removes redundant or less important weights to reduce storage and inference computation.Source: B. DNN Model Compression Techniques · p. 2
• Non-structured pruning removes weights at arbitrary locations, yielding fine-grained sparsity but requiring sparse storage and index handling.Source: B. DNN Model Compression Techniques · p. 2, Figure 2 · p. 2
• Structured pruning removes whole filters or channels, preserving a full matrix format for hardware implementation but risking greater accuracy loss from coarse granularity.Source: B. DNN Model Compression Techniques · p. 2

C. DNN Acceleration on Mobile Devices

• Existing mobile DNN frameworks are described as lacking RNN acceleration support.Source: C. DNN Acceleration on Mobile Devices · p. 2

D. Research Motivation

• Non-structured pruning offers high compression but is typically GPU-incompatible, whereas structured pruning is hardware-friendly but can degrade accuracy, especially for time-based RNNs.Source: D. Research Motivation · pp. 2–3

III. RELATED WORK

• Prior RNN acceleration studies largely target FPGAs, while RTMobile is framed as software–hardware co-design for mobile devices and compared with ESE and C-LSTM.Source: §III · p. 3

A. ESE

• ESE's irregular pruning incurs hardware read/write overhead, inefficient index storage, and only marginal compression gains after accounting for indices.Source: A. ESE · p. 3

B. C-LSTM

• C-LSTM's block-circulant structure reduces computation and storage complexity relative to ESE, but its coarse-grained pruning degrades accuracy and its training does not support ADMM-based pruning.Source: B. C-LSTM · p. 3

C. ADMM

• ADMM formulates pruning as loss minimization under a constraint on nonzero weights, then iteratively updates the pruning mask and retrains the network under that mask.Source: C. ADMM · p. 3
• The pruning mask is trained using Algorithm 1.Source: C. ADMM · p. 3

• RTMobile is the proposed mobile acceleration framework for RNNs.Source: §IV · p. 3

A. Block-based Structured Pruning

• BSP trains a compressed model through row-based column-block pruning followed by column-based row pruning.Source: A. Block-based Structured Pruning · p. 3
• The process splits the weight matrix into row groups and blocks, applies ADMM pruning within each row, then prunes rows across the matrix.Source: A. Block-based Structured Pruning · p. 3
• Training continues until all blocks are pruned; the authors report stable training performance and a decentralized pruned weight matrix.Source: A. Block-based Structured Pruning · p. 3

B. Compiler-assisted RNN Acceleration Framework

• Matrix reorder groups rows with similar computation patterns so threads can process them with balanced workloads.Source: B. Compiler-assisted RNN Acceleration… · p. 4
• Redundant-load elimination reuses input-feature-map data across neighboring rows with the same preserved-weight pattern, an opportunity enabled by block-based pruning.Source: B. Compiler-assisted RNN Acceleration… · p. 4
• BSPC stores pruned RNN matrices more compactly than CSR by exploiting within-block weight patterns and retaining matrix-reorder information.The reduced memory footprint alleviates memory-bound execution.Source: B. Compiler-assisted RNN Acceleration… · p. 4
• Offline auto-tuning searches configurations such as tiling, unrolling, and memory placement, including block size for an accuracy-performance balance.Source: B. Compiler-assisted RNN Acceleration… · p. 4

V. EXPERIMENTAL EVALUATION

• The evaluation targets three comparisons: compression rate and accuracy against other methods, computational and energy efficiency against ESE on FPGA, and how compression rate relates to inference time.The authors explain that ESE is used because prior RNN acceleration work did not support mobile devices and ESE had high inference accuracy among prior work.Source: §V · p. 4
• ESE is the FPGA comparator because prior RNN acceleration work did not support mobile devices and ESE offered high inference accuracy among prior work.Source: §V · p. 4
• Table I reports GRU-on-TIMIT phone error rates, parameter counts, and column, row, and overall compression rates.It distinguishes dense-model baseline PER, compressed-model PER, and PER degradation.Source: Table I · p. 5
• Table II covers inference time, GOP/s, and energy efficiency for mobile GPU and CPU, with energy efficiency normalized to ESE’s FPGA implementation.The caption states that at 245× compression, GPU inference is slightly faster than ESE’s 82.7 us and that the GPU uses 16-bit floating point.Source: Table II · p. 5
• Figure 4 plots RTMobile speedup at different compression rates on mobile platforms.Source: Figure 4 · p. 5

A. Experiment Setup

• Experiments used a 2-layer, approximately 9.6M-parameter GRU on TIMIT, running on a Galaxy S10 with Snapdragon 855.The platform includes an Adreno 640 GPU.Source: A. Experiment Setup · p. 5

B. Evaluation Results and Discussion

• BSP guarantees no accuracy degradation when compression is no higher than 10× and maintains high accuracy at relatively high compression rates.Source: B. Evaluation Results and Discussion · pp. 5–6
• At 245× compression, BSP retains near-C-LSTM PER (24.20% vs. 24.15%) with over 80× fewer parameters (0.04M vs. 3.25M).At 103× compression, BSP’s PER is 23.20% versus C-LSTM’s 24.15%, while C-LSTM has 36× more parameters.Source: B. Evaluation Results and Discussion · p. 6
• Above 245× compression, RTMobile reports about 40× higher energy efficiency than ESE while maintaining the same inference time on mobile GPU.ESE’s inference time is 82.7 us.Source: B. Evaluation Results and Discussion · p. 6
• Speedup over dense CPU/GPU baselines grows with compression rate and stabilizes around 250×; at 245×, mobile-GPU inference time matches ESE on FPGA.Source: B. Evaluation Results and Discussion · p. 6

VI. CONCLUSION

• RTMobile is presented as the first RNN acceleration framework for mobile devices.Source: §VI · p. 6
• A block-based pruning algorithm and three compiler optimizations are reported to achieve real-time inference without accuracy degradation.Source: §VI · p. 6
• Experiments show RTMobile outperforming existing RNN hardware acceleration methods in compression rate, inference accuracy, execution time, and energy efficiency.Source: §VI · p. 6