Source-linked AI summary
FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
Aditya Kusupati, Manish Singh, Kush Bhatia, Ashish Kumar, Prateek Jain, Manik Varma
TL;DR
RNNs face inaccurate or unstable training and costly prediction, limiting use in resource-constrained and real-time applications. This paper develops FastRNN with a residual connection and FastGRNN with a matrix-reusing gate, achieving competitive accuracy with lower costs and compact models. FastGRNN models can be compressed to 1–6 KB and deployed on severely resource-constrained IoT devices.
Problem
RNNs can train inaccurately or unstably, while their model size and prediction time hinder deployment in resource-constrained and real-time applications.
Method
FastRNN adds a weighted residual connection with two scalar parameters, while FastGRNN converts the residual connection into a gate that reuses the RNN matrices.
Results
FastGRNN matched or sometimes exceeded state-of-the-art gated RNN accuracies with 2–4x fewer parameters, while compressed models were up to 35x smaller and could fit in 1–6 KB.
Takeaways & Limitations
FastGRNN enabled accurate, efficient prediction on severely resource-constrained IoT devices, including a 1 KB model for the “Hey Cortana” wakeword task.
Takeaways & Limitations
The FastRNN stability analysis assumes bounded parameter matrices and data, represented by the quantity C.
Abstract
from arXiv · showhide
This paper develops the FastRNN and FastGRNN algorithms to address the twin RNN limitations of inaccurate training and inefficient prediction. Previous approaches have improved accuracy at the expense of prediction costs making them infeasible for resource-constrained and real-time applications. Unitary RNNs have increased accuracy somewhat by restricting the range of the state transition matrix's singular values but have also increased the model size as they require a larger number of hidden units to make up for the loss in expressive power. Gated RNNs have obtained state-of-the-art accuracies by adding extra parameters thereby resulting in even larger models. FastRNN addresses these limitations by adding a residual connection that does not constrain the range of the singular values explicitly and has only two extra scalar parameters. FastGRNN then extends the residual connection to a gate by reusing the RNN matrices to match state-of-the-art gated RNN accuracies but with a 2-4x smaller model. Enforcing FastGRNN's matrices to be low-rank, sparse and quantized resulted in accurate models that could be up to 35x smaller than leading gated and unitary RNNs. This allowed FastGRNN to accurately recognize the "Hey Cortana" wakeword with a 1 KB model and to be deployed on severely resource-constrained IoT microcontrollers too tiny to store other RNN models. FastGRNN's code is available at https://github.com/Microsoft/EdgeML/.
1 Introduction
The paper targets unstable, inaccurate RNN training and costly prediction by introducing FastRNN and FastGRNN, which retain competitive accuracy while reducing model and inference costs.
- Motivation: RNNs face exploding and vanishing gradients during long sequences, while resource-constrained applications also require small models and fast prediction.Models and code fitting within a few kilobytes could enable deployment on IoT endpoints and reduce prediction time and energy consumption.
- FastRNN: FastRNN stabilizes standard RNN training with a learned weighted residual connection using only two additional scalar parameters.The approach is associated with provably stable training and lower prediction costs than unitary and gated RNNs.
- FastGRNN: FastGRNN converts the residual connection into a gate while reusing the RNN matrices, matching or exceeding leading gated-RNN accuracies with 2-4x fewer parameters.The architecture reuses matrices for both the hidden state and gating computation.
- Compression and deployment: Low-rank, sparse, and quantized FastGRNN models can be up to 35x smaller, with sizes ranging from 1 to 6 KB for many applications.A 1 KB model matched other RNNs on Hey Cortana recognition and enabled deployment on Arduino Uno devices.
- Contributions: The paper’s two contributions are FastRNN’s residual-connection analysis and FastGRNN’s near-state-of-the-art accuracy and training time at substantially lower prediction cost.The authors report that FastGRNN prediction costs could be lower by an order of magnitude.
2 Related Work
Prior RNN approaches improve stability or accuracy through constrained transitions, extra gates, or compression assumptions, but these choices create trade-offs in expressiveness, model size, or prediction cost.
- Residual connections: Residual RNN connections have been explored, but earlier methods struggled to learn the state transition matrix or applied residuals only to selected hidden units.One approach used a hand-crafted transition matrix, while another limited residual connections to a few units with randomly sampled parameters.
- Unitary RNNs: Unitary RNNs stabilize training with well-conditioned transition matrices, but restricted expressiveness, longer training, and larger hidden layers can increase costs.Adding hidden units may improve accuracy while increasing training time, prediction time, and model size.
- Gated RNNs: Gated RNNs achieve state-of-the-art classification accuracy by adding parameters, which also increases model size and prediction time.UGRNN, GRU, and LSTM form a progression of increasingly simplified gating designs.
- Efficient representations: FastGRNN simplifies UGRNN by reusing its RNN matrices within the gate and applying low-rank, sparse, and quantized representations.These representations build on established approaches for reducing recurrent prediction and training complexity.
3 FastRNN and FastGRNN
FastRNN stabilizes standard RNN training with a weighted residual connection, while FastGRNN extends this idea with a reused-matrix gate to retain gated-RNN accuracy at much lower prediction cost and model size.
- FastRNN: FastRNN adds a weighted residual connection with only two trainable scalar parameters, preserving standard RNN prediction complexity.The residual weights control how strongly the current input updates the hidden state.
- FastRNN analysis: Exponential-in-T gradient condition numbers make standard RNN training unstable, whereas FastRNN can bound the condition number under α = O(1/T).The paper links this bound to stable training and a convergence rate that is constant in T.
- FastRNN analysis: FastRNN’s bounds state that convergence time is constant with respect to T and its generalization error is independent of T when α = O(1/T).These are upper-bound results under the theorem’s stated assumptions; the paper notes that tighter bounds may exist for standard RNNs.
- FastGRNN: FastGRNN converts the residual connection into a coordinate-wise gate while reusing W and U to reduce parameter and inference costs.The gate uses xt and ht−1, while its parameters simulate the residual weights α and β.
4 Experiments
Experiments benchmark FastRNN and FastGRNN on IoT, language-modeling, and activity-recognition tasks against standard, unitary, and gated RNNs. Results show substantially lower prediction costs and model sizes while retaining competitive accuracy, including deployment on constrained Arduino devices.
- FastRNN surpassed leading unitary RNNs on 6 of 8 datasets, with accuracy gains up to 3.77% over SpectralRNN.
- FastRNN training speedups ranged from 1.2x over UGRNN to 196x over EURNN across evaluated datasets.
- FastGRNN-LSQ matched gated-RNN accuracies and training times while using models 1.18-4.87x smaller.
- FastGRNN accuracy was at most 1.13% worse than the best RNN while its model was up to 35x smaller.
- FastGRNN recognized the Hey Cortana wakeword as accurately as leading RNNs with a 1 KB model.
- 25-45x faster prediction than UGRNN and 57-132x faster than SpectralRNN was achieved on the Arduino MKR1000 at the same accuracy.
- Low-rank constraints slightly improved accuracy and reduced prediction costs, while sparsity and quantization produced smaller models with a slight accuracy drop.
5 Conclusions
The paper proposes FastRNN and FastGRNN for efficient RNN training and prediction. FastRNN uses a two-parameter residual connection, while FastGRNN reuses RNN matrices in a gate to retain accuracy with smaller, deployable models.
- FastRNN incorporates a residual connection with two scalar parameters to provide provably stable training.
- FastRNN generally achieved lower training times, lower prediction costs, and higher accuracies than leading unitary RNNs.
- FastGRNN reused RNN matrices in a gate to match state-of-the-art gated-RNN accuracy with significantly lower prediction costs.
- Low-rank, sparse, and quantized FastGRNN models occupied 1-6 KB and enabled efficient prediction on severely resource-constrained IoT devices.
A Convergence Analysis for FastRNN
FastRNN’s convergence analysis bounds gradient and hidden-state deviations under explicit assumptions on the residual connection and parameter norms. The resulting theorems provide convergence and generalization guarantees for randomized stochastic gradient descent.
- Optimization: FastRNN uses randomized stochastic gradient descent with an iteration limit M and step sizes γ_k to optimize sequential training data.The algorithm initializes θ_1 and samples iteration-related quantities using a probability mass function P_R.
- Deviation analysis: The analysis bounds the loss-gradient variation ∥∇_θL(θ) − ∇_θL(θ + δ)∥_2 through perturbation analysis of the hidden state and model parameters.The loss is logistic, and θ=(W,U,v) contains the FastRNN parameters and final classifier vector.
- Deviation analysis: α≤1 and the factor α∥U∥_2+β control geometric terms in the hidden-state and matrix-norm bounds.The derivation bounds B_k, C_k, and G_k using parameter perturbations, operator norms, and the residual weights.
- Convergence guarantee: Theorem 3.1 gives a convergence bound for randomized stochastic gradient descent applied to labeled sequential data for at most M iterations.The theorem assumes a loss L(θ), FastRNN parameters θ=(W,U,v), and an admissible α setting with β=1−α.
- Generalization guarantee: Theorem 3.2 gives a high-probability generalization-error bound for FastRNN function classes with bounded parameter matrices, classifier norm, and input features.The bound applies to any 1-Lipschitz loss under the stated Frobenius, Euclidean, and almost-sure input constraints.
B Dataset Information
The experiments cover speech, text classification, language modeling, image sequences, and sensor-based activity recognition. On AmazonCat-13K, FastGRNN-LSQ maintains comparable classification performance with a smaller recurrent-model memory footprint.
- Speech datasets: Google-12 and Google-30 use 1-second, 16KHz speech utterances represented as 99 timesteps of 32 log Mel-filter-bank responses.Google-12 has 12 classes, while Google-30 contains 30 short-word classes.
- Speech datasets: Wakeword-2 uses 1.63-second, 16KHz utterances producing 162 timesteps of 32 filter responses.The data are zero-mean, unit-variance normalized during training and prediction.
- Text datasets: Yelp-5 contains 500,000 training and 500,000 test reviews, clipped or padded to 300 words with 128-dimensional jointly trained embeddings.Its vocabulary contains 20,000 words.
- Text datasets: AmazonCat-13K is an extreme multi-label task with 13,330 labels, 500-word inputs, a 267,134-word vocabulary, and 50-dimensional trainable embeddings.Each product is assigned to multiple categories.
- Evaluation: FastGRNN-LSQ achieves classification performance similar to GRU and LSTM with a 2-3x lower memory footprint on the extreme multi-label evaluation.The reported model size excludes embeddings and the final linear classifier; FastRNN improves P@1 accuracy by over 50% in earlier experiments.
C Supplementary Experiments
Supplementary experiments examine accuracy–size trade-offs, initialization bias, residual-weight scaling, and the effects of low-rank, sparse, and quantized FastGRNN parameterizations.
- Accuracy versus model size: The evaluation studies model sizes from 0-128Kb and compares accuracy across multiple recurrent architectures.Figures 3 and 4 analyze FastGRNN, FastGRNN-LSQ, SpectralRNN, UGRNN, GRU, and LSTM.
- Accuracy versus model size: FastGRNN reaches state-of-the-art accuracy on Google-12 and Google-30 at significantly smaller model sizes than the baseline methods.This comparison is presented in Figures 3 and 4.
- Initial-state bias: 92.08 mean accuracy with 0.09 standard deviation indicates little initialization variation for a trained FastRNN classifier on Google-12.In the non-learning setting, initial-state bias appears only for extremely small α∈(0,0.005), while α∈(0.005,0.08) is described as unbiased and learnable.
- Residual-weight scaling: FastRNN’s trainable residual weights α and β are reported to scale as O(1/T) with the number of timesteps T.Table 7 examines learned values across three datasets with tanh nonlinearity.
- Compression: Low-rank parameterization, sparsity, and byte quantization form FastGRNN’s compression pipeline, with sparsity and quantization reducing model sizes by 2x and 4x respectively.Tables 8-10 remove components progressively, while Figure 6 examines their effects on two Google datasets.
E Hyperparameters of FastGRNN for reproducibility:
The reported FastGRNN experiments use fixed reproducibility settings and permit piece-wise linear activation approximations for weight quantization.
- Reproducibility: Experiments use random seed 42, a P40 GPU, CUDA 9.0, and CuDNN 7.1.These settings are listed with the hyperparameters in Table 11.
- Quantization: Piece-wise linear approximations of tanh or sigmoid can be used when quantizing the weights.The approximation option is stated as an implementation choice for quantized models.
- Reproducibility: Table 11 provides the hyperparameters used to run the experiments.The passage associates these hyperparameters with the stated hardware, software, and random-seed configuration.
F Timing Experiments on more IoT boards
The timing experiments compare prediction behavior and compression components across resource-constrained hardware and model-size ranges. On Raspberry Pi 3, quantization mainly reduces model size because floating-point instructions remove its compute advantage.
- Timing on Raspberry Pi: Quantization considerably reduces model size on Raspberry Pi 3 but provides no compute benefit beyond that reduction.The Raspberry Pi’s floating-point instructions explain why quantization does not improve computation there.
- Compression analysis: The compression experiments include separate component analyses for PTB and Wakeword-2 across model sizes up to 64KB.Figure 6 specifically studies LSQ components over the 0-64KB model-size range.
G Vectorized FastRNN
FastRNN-vector extends FastRNN’s scalar parameters to a vectorized form and is evaluated through accuracy, model-size, and perplexity results. Direct comparisons report that FastGRNN’s gating is more accurate than FastRNN-vector.
- Vectorized FastRNN: FastRNN-vector extends FastRNN’s scalar α to a vector and replaces β with ζ(1 −α) + ν using trainable scalars ζ and ν in [0, 1].
- Vectorized FastRNN: FastGRNN’s gating is more accurate than FastRNN-vector in the direct comparison.
- Vectorized FastRNN: FastRNN-vector uses tanh in most cases, while a few configurations use ReLU and show slightly different results.
- Evaluation: The vectorized FastRNN evaluation reports results for model size and train perplexity, alongside reproducibility hyperparameters and Raspberry Pi timing.
H Effects of Regularization for Language Modeling Tasks
The PTB experiments study regularization and long-sequence behavior in FastGRNN. Adding a layer reduces test perplexity, while the experiments examine stability at sequence lengths substantially longer than common prior settings.
- Regularization effects: Adding an extra FastGRNN layer reduces test perplexity by 10 points compared with a single-layer architecture.
- Regularization effects: The PTB experiments combine several regularization techniques with FastGRNN and FastGRNN-LSQ.
- Regularization effects: Table 16 summarizes FastGRNN’s train and test perplexity under the regularization experiments.
- Long-sequence behavior: PTB experiments use sequence length 300, compared with the 35-70 range generally used in cited prior work.
- Long-sequence behavior: FastGRNN with 256 hidden units exhibits stability properties in the long-sequence regime studied by these experiments.