PaperlayerView exact revision on arXiv

Source-linked AI summary

Block-Sparse Recurrent Neural Networks

Sharan Narang, Eric Undersander, Gregory Diamos

arXiv:1711.02782v1cs.LGcs.AIstat.ML

TL;DR

Large RNNs are compute- and memory-intensive, while unstructured sparsity has yielded less hardware speed-up than expected. This paper uses block pruning and group lasso to induce block sparsity in RNNs, producing models that are nearly 10× smaller with reported accuracy loss, and reports that larger models can recover accuracy while remaining smaller than baseline.

  • Problem

    Large RNNs are compute- and memory-intensive, and unstructured sparse operations deliver less hardware speed-up than expected because of storage overhead, irregular memory access, and limited use of processor array data-paths.

    Source: §1 · p. 1

  • Method

    The paper induces RNN block sparsity by pruning weight blocks during training and by applying group lasso, including in combination with pruning.

    Source: §1 · p. 2

  • Results

    4×4 block sparsity in vanilla RNNs and GRUs yields 9% to 17% accuracy loss versus dense baselines while reducing model size nearly 10×; larger models reduce accuracy loss while remaining smaller than baseline.

    Source: §1 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    Block-sparse models have significantly fewer parameters than dense baselines and can take advantage of underlying hardware efficiently.

    Source: §7 · p. 9

    Efficient block-sparse matrix-dense matrix/vector multiplies for GPU and ARM remain future work for increased deployment speed-up.

    Source: §7 · p. 9

Abstract

from arXiv · show

Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms. In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization to create blocks of weights with zeros. Using these techniques, we demonstrate that we can create block-sparse RNNs with sparsity ranging from 80% to 90% with small loss in accuracy. This allows us to reduce the model size by roughly 10x. Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count. Our technique works with a variety of block sizes up to 32x32. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

1 INTRODUCTION

The section motivates block sparsity as a way to address the hardware inefficiencies of unstructured sparse RNNs. It introduces block pruning and group lasso, reporting substantial model-size reductions with limited accuracy loss.

1 INTRODUCTION

  • Growing datasets and RNN models make deployment compute- and memory-intensive, while unstructured sparsity delivers less hardware speed-up than expected.Sparse formats incur storage overhead, irregular memory access, and limited use of processor array data-paths.Source: §1 · p. 1
  • Block sparsity reduces index-storage overhead and irregular memory access while allowing sparse operations to use processor array data-paths.Indices can be shared across nonzero blocks rather than stored for individual nonzero weights.Source: §1 · p. 1
  • The paper induces block sparsity by pruning weight blocks during training, applying group lasso, or combining group lasso with block pruning.The pruning approach produces a block-sparse RNN at the end of training.Source: §1 · p. 1
  • 4×4 block sparsity in vanilla RNNs and GRUs yields 9% to 17% accuracy loss versus dense baselines while reducing model size nearly 10×.Blocks can scale up to 32×32; pruning a larger dense model can reduce accuracy loss while keeping fewer parameters than the baseline.Source: §1 · p. 2
  • The approach is agnostic to the optimization algorithm and requires no hyperparameter retuning beyond pruning and regularization settings.It does not require model retraining, so training time remains the same.Source: §1 · p. 2

2 RELATED WORK

Prior work reduces model size through unstructured pruning and develops structured sparsity methods, mainly for convolutional or specialized recurrent networks. This paper situates its approach among these methods and notes a gap in large-scale RNN use of group lasso.

2 RELATED WORK

  • Prior neural-network pruning methods generally induce random, unstructured sparsity, including methods applied to recurrent networks during initial training.The cited recurrent-network approaches use gradual pruning with small accuracy loss.Source: §2 · p. 2
  • Structured sparsity methods target CNNs or alter LSTM structure, whereas this approach covers vanilla RNNs and GRUs trained on large-scale speech data..Source: §2 · p. 2
  • Group lasso has been used to generate sparse structures and induce structured sparsity in CNNs, but the authors know of no such use with large-scale-trained RNNs.Prior applications include Yuan & Lin (2006b), Kim & Xing (2010), and Wen et al. (2016).Source: §2 · p. 2
  • Quantization and low-rank factorization address compute and memory footprint, and the authors describe their approach as orthogonal and combinable with them.Source: §2 · p. 2

3 IMPLEMENTATION

The implementation extends threshold-based pruning from individual weights to blocks, and also applies group lasso to weight blocks. The two mechanisms can be coordinated during training to create block-sparse RNNs.

3.1 BLOCK PRUNING

  • Block pruning extends threshold-based weight pruning by using each block’s maximum-magnitude weight as its representative and zeroing the whole block when it falls below threshold.The resulting mask is multiplied by the weights; pruning stops after about 40% of training, and zeroed blocks remain zero.Source: §3.1 · pp. 2–3, Figure 1 · p. 3
  • Six hyperparameters govern the pruning threshold; the prescribed heuristics adapt prior weight-pruning settings for block pruning.The threshold schedule uses a monotonically increasing threshold, and the block-pruning initial slope is adjusted for the number of elements per block; further tuning may be needed for a desired sparsity.Source: §3.1 · p. 3, Table 1 · p. 3
  • The same block size and pruning hyperparameters are used for recurrent and fully connected layers.Source: §3.1 · p. 3

3.2 GROUP LASSO REGULARIZATION

  • Group lasso divides model weights into blocks and adds a loss term proportional to each block’s ℓ2 norm.In the loss notation, w(g) is a block of weights and G is the total number of blocks.Source: §3.2 · p. 4
  • A sufficiently large λg can drive every weight in some groups to zero, and the method uses a constant λg throughout training.Source: §3.2 · p. 4
  • Combining group lasso with block pruning lets the regularizer guide block selection in coordination with the pruning schedule.Regularization is turned off when pruning ends, typically after around 40% of training; weights already zeroed remain unchanged.Source: §3.1 · p. 3, §3.2 · p. 4

4 EXPERIMENTS

Experiments compare block pruning, group lasso, and their combination on speech-recognition RNN and GRU models. Baseline-sized sparse models reduce parameter counts nearly 10×, while larger sparse models can recover accuracy and block size changes the sparsity needed for good accuracy.

4 EXPERIMENTS

  • The experiments compare BP, GL, and GLP on speech-recognition RNN and GRU models trained for 25 epochs, with CER measured on an independent 2.9-hour test set.The training data comprises 2100 hours of English speech.Source: §4 · p. 4

4.1 BLOCK SPARSITY

  • Baseline-sized sparse models reach 89% RNN block sparsity with 16.7% relative accuracy loss and 90% GRU sparsity with 8.8% loss, while reducing parameter counts nearly 10×.The RNN has 1760 hidden units; the GRU retains most of the dense model's capacity.Source: §4.1 · pp. 4–5, Table 2 · p. 5
  • At approximately the same parameter count, 704-unit dense RNN and GRU models perform worse than the sparse models.The comparison tests whether sparsity reduces overfitting in the large dense baselines.Source: §4.1 · p. 5, Table 2 · p. 5
  • RNN Sparse 3072 is 1.9% worse than the dense baseline; 2560- and 3072-unit sparse RNNs cut parameters 5× and 2.5×, while a 3584-node sparse GRU loses about 5% and shrinks 4.5×.The evaluation also reports that pruning a model larger than baseline reduces accuracy loss while shrinking model size nearly 5×.Source: §4.1 · p. 5, Table 2 · p. 5

4.2 GROUP LASSO VARIANTS

  • For both 1760- and 2560-node RNNs, group lasso alone performs worse than group lasso with block pruning; high regularization underfits the training data.The combined approach uses a lower regularization factor because pruning forces smaller-magnitude weights to zero.Source: §4.2 · pp. 5–6

4.3 BLOCK SIZE VARIATION

  • RNNs require 83.6% sparsity with 16×16 blocks and 79.1% with 32×32 blocks to obtain good accuracy; similar results hold for GRUs.The authors report that larger blocks reduce memory and compute requirements despite achieving lower sparsity.Source: §4.3 · p. 6, Table 4 · p. 6

5 PERFORMANCE

Block-sparse operations reduce indexing and memory-access overheads and can use hardware array data-paths. In the reported TitanX Maxwell benchmarks, block-sparse matrices achieve higher speed-up than unstructured sparsity at large batch sizes.

5 PERFORMANCE

  • Sharing one index across each nonzero block reduces indexing overhead: 4×4 blocks leave 12.5% memory bloat, while 16×16 blocks reduce it below 1%.The comparison assumes 16-bit neural-network weights and indices.Source: §5 · p. 6
  • Contiguous block storage produces large coalesced memory accesses, while blocks at least as large as the hardware data-path can keep array units busy.The stated V100 data-path size is 16×16 or greater.Source: §5 · pp. 6–7
  • At large batch sizes, block-sparse matrices achieve higher speed-up than unstructured sparsity, and 4×4 blocks outperform 16×16 blocks.The authors attribute the speed-up to reduced irregular memory accesses and improved load balance; they note that further investigation is needed to explain the block-size result.Source: §5 · p. 7, Figure 2 · p. 7

6 DISCUSSION

Pruning outcomes depend on sparsity, block size, and layer position: larger blocks and more aggressive pruning near the output are associated with greater accuracy loss. GLP reaches high sparsity earlier than block pruning, while layerwise sparsity varies despite shared pruning hyperparameters.

6.1 PRUNING CHARACTERISTICS

  • GLP reaches 90% sparsity just before 10,000 iterations, significantly earlier than BP; both BP and GLP show sharper pruning curves than weight pruning.The authors suggest max-reducing blocks to a single value could cause the sharper curves; GLP pushes blocks toward zero early in training.Source: §6.1 · p. 7, Figure 3 · p. 8
  • At 89% sparsity, 180 of 38,270 neurons have all output weights zero, compared with 1,620 neurons at 94% sparsity, where accuracy is significantly worse.The 89%-sparse model had good accuracy relative to the dense baseline.Source: §6.1 · p. 7, Figure 3 · p. 8

6.2 IMPACT OF SPARSITY ON ACCURACY

  • For weight pruning, sparsity below 95% gives relative accuracy from -20% to -27%; above 95%, accuracy loss reaches 30% or more.These experiments used the baseline RNN, 20 training epochs, and validation-set accuracy.Source: §6.2 · p. 7
  • For 16×16 blocks, models with sparsity greater than 86% have accuracy loss of 30% or more; a similar trend occurs at 32×32, indicating a sparsity, block-size, and accuracy tradeoff.Source: §6.2 · p. 8

6.3 SPARSITY VS LAYERS

  • Initial recurrent layers are pruned more aggressively than final layers; increasing sparsity near the output harms accuracy, and layerwise variance grows with block size.With shared pruning hyperparameters, the increasing variance makes it harder to exceed 32×32 blocks.Source: §6.3 · p. 8, Figure 4 · p. 8

7 CONCLUSION AND FUTURE WORK

Block pruning and group lasso combined with pruning produce block-sparse RNNs about as accurate as dense baselines, with fewer parameters and reduced memory requirements. Efficient block-sparse GPU and ARM operations and pruning earlier in training remain future work.

7 CONCLUSION AND FUTURE WORK

  • Block pruning and group lasso combined with pruning build block-sparse RNNs about as accurate as dense baselines, with significantly fewer parameters and reduced memory requirements.The models can also take advantage of underlying hardware efficiently.Source: §7 · pp. 8–9
  • The authors identify earlier pruning as future work, aiming to train sparse models with lower compute and memory demands.They also call for efficient block-sparse matrix–dense matrix/vector multiplies for GPU and ARM to increase deployment speed-up.Source: §7 · p. 9
  • Efficient block-sparse matrix–dense matrix/vector multiplies for GPU and ARM remain to be implemented for increased deployment speed-up.Source: §7 · p. 9

A ℓ1 AND ℓ1/2 REGULARIZATION

This section compares ℓ1 and ℓ1/2 regularization, with and without weight pruning, as approaches to inducing sparsity. It reports that combining ℓ1 with pruning recovers accuracy loss at similar sparsity, while ℓ1/2 with pruning performs worse.

A ℓ1 AND ℓ1/2 REGULARIZATION

  • ℓ1 and ℓ1/2 regularizers act on individual weights and could induce unstructured sparsity; ℓ1/2 was expected to push smaller weights toward zero while leaving larger weights relatively unaffected.Source: A ℓ1 AND ℓ1/2 REGULARIZATION · p. 12
  • The ℓ1 experiments use weight pruning to guide pruning or produce sparsity directly, and test regularization both with and without pruning.Source: A ℓ1 AND ℓ1/2 REGULARIZATION · p. 12
  • The Deep Speech 2 bidirectional RNN experiments train for 25 epochs on 2,000 hours of internal data and report results on an independent 2.9-hour test set.Source: A ℓ1 AND ℓ1/2 REGULARIZATION · p. 12
  • With weight pruning, ℓ1 recovers the accuracy loss at similar sparsity, while ℓ1/2 performs worse; without pruning, ℓ1 is significantly less accurate than the dense baseline.The results are reported in Table 5 for the bidirectional RNN model with 1760 hidden units.Source: Table 5 · p. 12, A ℓ1 AND ℓ1/2 REGULARIZATION · p. 12
  • Without pruning, ℓ1 requires significantly higher λ to achieve high sparsity.Source: A ℓ1 AND ℓ1/2 REGULARIZATION · p. 12
Loading 1711.02782v1…