PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Sparsity in Recurrent Neural Networks

Sharan Narang, Erich Elsen, Gregory Diamos, Shubho Sengupta

arXiv:1704.05119v2cs.LGcs.CL

TL;DR

Large RNNs are difficult to deploy because their parameter size and evaluation cost strain mobile, embedded, and server settings. The paper progressively prunes recurrent weights during initial training, producing sparse models with reduced size and inference cost. The resulting models can retain or exceed dense-baseline accuracy while achieving substantial sparsity and measured speedups.

  • Problem

    Large recurrent models are difficult to deploy because mobile and embedded devices have limited memory and storage, while evaluating them requires substantial computation and can affect battery life.

  • Method

    The paper progressively sets recurrent-layer weights to zero during training with monotonically increasing, layer-specific thresholds and binary masks.

  • Results

    Sparse models can achieve equal or better accuracy than a dense baseline with far fewer parameters, while 95% sparse recurrent and GRU layers show speedups ranging from 1.15x to 3x and 3.5x to 7x, respectively.

  • Takeaways & Limitations

    Pruning during training yields sparse RNNs suited to mobile and server deployment because they combine smaller models with increased computational efficiency.

  • Takeaways & Limitations

    Further work is needed to implement optimal small-batch sparse matrix-dense vector routines for GPUs and ARM processors.

Abstract

from arXiv · show

Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8x and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.

1 INTRODUCTION

Growing RNN models create deployment challenges because they require substantial storage, memory bandwidth, computation, and battery power. The paper proposes pruning weights during training to produce sparse recurrent models with small accuracy loss and potential efficiency gains.

  • Model growth: 116 million parameters in recent forward-only GRU acoustic models illustrate how recurrent networks have grown in size.Earlier acoustic models increased from 11 million to approximately 67 million parameters before reaching 116 million.
  • Deployment challenges: Large RNNs are difficult to deploy on mobile and embedded devices because memory, storage, bandwidth, computation, and battery life are constrained.These constraints affect both model storage and the cost of evaluating the model.
  • Deployment challenges: Sparse inference can improve performance when reduced parameter data lowers memory-bandwidth demands or allows the model to fit in cache.Sparse computation incurs a penalty, so sufficiently high sparsity is needed for the smaller data volume to become advantageous.
  • Proposed approach: The proposed method progressively zeros recurrent-layer weights during training using a monotonically increasing threshold, reaching 90% sparsity with a small accuracy loss.The approach supports both GRUs and vanilla RNNs, while layers with substantially fewer parameters are not pruned.
  • Results: Starting from a larger dense matrix and pruning it can match or exceed dense-baseline accuracy with substantially fewer parameters.The authors present this as an additional benefit beyond reduced storage and faster inference.
  • Proposed approach: Training time does not increase because pruning is integrated into training rather than performed as a separate retraining stage.The method is implemented in current training frameworks and is agnostic to the optimization algorithm.

2 RELATED WORK

Prior work reduces neural-network size through quantization, low-rank factorization, and pruning, but the paper targets recurrent networks with computationally efficient pruning during training. Its approach uses a simple threshold heuristic and gradual pruning without additional retraining.

  • Prior approaches: Quantization, low-rank factorization, and network pruning are established approaches for reducing neural-network memory, computation, or model size.The cited methods include fixed-point representations, learned low-rank factors, and saliency-based weight removal.
  • Prior approaches: 35x reduction in model size without affecting accuracy was reported by combining pruning, quantization, and Huffman encoding on AlexNet.This result concerns AlexNet trained on ImageNet rather than recurrent speech-recognition models.
  • Method distinction: The proposed method is designed for large recurrent networks and uses a computationally efficient heuristic threshold instead of Hessian approximation.The paper positions this as a distinction from earlier pruning methods requiring Hessian information.
  • Method distinction: The pruning procedure is integrated into training and requires no additional retraining, unlike methods that prune and then retrain the network.This comparison is made with methods such as Han et al. (2015).
  • Method distinction: Gradual pruning produces better results than hard pruning performed at a particular epoch.The paper also notes that the method requires careful selection of pruning hyperparameters.

3 IMPLEMENTATION

The implementation maintains binary masks and periodically updates them using layer-specific, monotonically increasing thresholds during optimization. Hyperparameters control when, how quickly, and how often pruning occurs, while pruned weights can re-enter the forward pass if later updates exceed the threshold.

  • Masking and threshold updates: Binary masks are initialized to one for every weight, applied after optimizer updates, and periodically changed to zero parameters below the current threshold.This mask-based procedure is applied throughout training.
  • Masking and threshold updates: Layer-specific hyperparameters determine each pruning threshold’s duration, rate, and update frequency.Separate hyperparameter sets produce different thresholds for each layer type.
  • Threshold schedule: Threshold schedules use start, ramp, and end iterations to control the progression of pruning.The paper provides heuristics for selecting these iteration boundaries and assumes a specified relationship between ramp and start slopes.
  • Threshold schedule: The 90th-percentile absolute weight from a previously trained model supplies q for selecting reasonable pruning hyperparameters.A validation set can further fine-tune the resulting parameters.
  • Pruned parameters: Only recurrent and linear-layer weights are pruned, including both input and recurrent matrices and GRU reset and update gates.Biases and batch-normalization parameters are excluded because they are much fewer in number.
  • Threshold schedule: The algorithm applies existing masks between the start and end iterations and increases the current iteration after each training loop.The displayed algorithm also includes a separate ramp interval within the pruning schedule.

4 EXPERIMENTS

Experiments evaluate gradual pruning on bidirectional RNN and GRU speech-recognition models, measuring accuracy, compression, and comparisons with smaller dense or hard-pruned networks. Larger sparse models recover or surpass dense-baseline accuracy while substantially reducing parameters.

  • Experimental setup: 2100 hours of English speech and 3.5 hours of multi-speaker validation data were used for 20-epoch Nesterov-SGD experiments.All hyper-parameters except pruning controls remained unchanged between dense and sparse training runs.
  • Bidirectional RNNs: 88% sparsity made the 1760-unit sparse model 8x smaller, but its devset CER was about 20% worse than the dense model.Pruning began shortly after epoch 1 and continued through epoch 10.
  • Bidirectional RNNs: A dense 704-unit model with approximately the sparse model’s parameter count performed worse than the sparse model.This comparison supports pruning a larger network rather than simply reducing dense hidden-layer width.
  • Bidirectional RNNs: 0.75% and 3.95% relative CER improvements over the dense baseline were obtained by sparse models with 2560 and 3072 hidden units, respectively.Both larger sparse models reached around 92% sparsity and had significantly fewer parameters than the baseline dense model.
  • Bidirectional RNNs: 7% to 9% better results than hard pruning were achieved by gradual pruning at approximately the same parameter counts.The comparison used hard-pruned dense-baseline models with final parameter counts from 8 million to 11 million.
  • Gated recurrent units: A 3568-unit sparse GRU was only 2.2% worse than the baseline dense GRU, with 91.82% sparsity and 17.8 million parameters.The 2560-unit sparse GRU had 88.6% sparsity, 13 million parameters, and a 13.8% accuracy drop relative to the dense model.

5 PERFORMANCE

Sparse layers mitigate the inference-time costs of large neural networks and reduce model memory requirements for deployment. Benchmarks report speed-ups for sparse matrix-vector products and substantial compression of speech-recognition models.

  • 5 PERFORMANCE: Sparse layers mitigate the increased inference time associated with deploying large models.Large models increase inference time after deployment, while sparse layers reduce this effect.
  • 5.1 COMPUTE TIME: 3x to 1.15x speed-ups are achieved for 95% sparse recurrent layers, depending on layer size, using sparse matrix-vector products.For GRU models, speed-ups range from 7x to 3.5x.
  • 5.2 MODEL SIZE: 268 MB Deep Speech 2 models compress to around 32 MB or 64 MB, while a 460 MB GRU model compresses to 50 MB.The pruned models can be further quantized to smaller datatypes without impacting accuracy.

6 DISCUSSION

Pruning schedules and layer-specific sparsity affect recurrent-network accuracy and efficiency. The paper also describes persistent kernels and reusable sparse-matrix scheduling as deployment mechanisms for exploiting sparsity.

  • 6.1 PRUNING CHARACTERISTICS: Initial recurrent layers are pruned more aggressively than final layers, and equal sparsity across layers produces higher CER.The authors conclude that final layers should be pruned slightly less than initial layers for good accuracy.
  • 6.1 PRUNING CHARACTERISTICS: Figure 2a shows recurrent-layer sparsity, while Figure 2b shows one layer’s pruning schedule during training.The layers in Figure 2a are ordered from the input-side recurrent layer to the final recurrent layer before the cost layer.
  • 6.1 PRUNING CHARACTERISTICS: 95% sparse-layer pruning begins at iteration 2700, stops at iteration 27000, and removes nearly 25000 weights by about iteration 15000.Convex pruning schedules tend to outperform schedules with a linear slope.
  • 6.2 PERSISTENT KERNELS: Persistent recurrent networks cache weights in on-chip memory across multiple timesteps, increasing the computational intensity of RNN evaluation.High sparsity allows significantly larger persistent RNNs to fit in on-chip memory.
  • 6.2 PERSISTENT KERNELS: Fixed RNN sparsity patterns allow sparse-matrix scheduling and load balancing to be performed once and reused across many timesteps.This factors scheduling overhead outside the recurrent computation loop.

7 CONCLUSION AND FUTURE WORK

Pruning during RNN training produces sparse models with lower size and higher accuracy than dense models in the demonstrated settings. The conclusion identifies untested generalization and implementation work for future research.

  • 7 CONCLUSION AND FUTURE WORK: Sparse RNN models can be more accurate than dense models while significantly reducing model size and computational cost.The models are suited to mobile devices and back-end server farms, and quantization could further reduce size.
  • 7 CONCLUSION AND FUTURE WORK: Generalization to language modeling, reduction of embedding-layer size, and comparison with L1 regularization remain future investigations.These are explicitly identified as questions for further work.
  • 7 CONCLUSION AND FUTURE WORK: Optimal small-batch sparse matrix-dense vector routines for GPUs and ARM processors remain to be implemented for deployment.The authors are also investigating training techniques that avoid maintaining dense matrices for much of the calculation.
Loading 1704.05119v2…