PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Song Han, Huizi Mao, William J. Dally

arXiv:1510.00149v5cs.CVcs.NE

TL;DR

Large neural networks are difficult to deploy on mobile and embedded systems because they require substantial storage, memory bandwidth, and energy. Deep Compression combines pruning, trained quantization, and Huffman coding to reduce model storage by 35× to 49× without affecting accuracy, enabling models to fit in on-chip SRAM.

  • Problem

    Large neural networks require substantial storage, memory bandwidth, and energy, limiting deployment on mobile systems.

  • Method

    Deep Compression prunes redundant connections, shares quantized weights, and applies Huffman coding while preserving the original accuracy.

  • Results

    35× to 49× compression was achieved across networks without loss of accuracy, including 35× for AlexNet and 49× for VGG-16.

  • Takeaways & Limitations

    The compressed models fit in on-chip SRAM rather than off-chip DRAM, supporting deployment in mobile applications and potentially improving energy efficiency.

  • Takeaways & Limitations

    The full benefits of compressed quantized networks were not fully measured because existing sparse matrix libraries lacked support for indirect lookup and relative indexing.

Abstract

from arXiv · show

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.

1 INTRODUCTION

Deep neural networks are difficult to deploy on mobile devices because their large storage, memory-bandwidth, and energy requirements exceed practical constraints. Deep compression addresses these barriers through pruning, trained quantization, and Huffman coding while preserving accuracy and enabling on-chip caching.

  • Deployment constraints: Over 200MB for AlexNet and over 500MB for VGG-16 make deep neural networks difficult to deploy on mobile systems.Their large number of weights consumes considerable storage and memory bandwidth.
  • Deployment constraints: 100MB app-download restrictions make neural-network storage overhead especially problematic for mobile applications.App Store apps above 100 MB will not download until connected to Wi-Fi.
  • Energy constraints: Memory bandwidth and dot-product computation consume substantial energy, making large neural networks difficult to deploy on battery-constrained mobile devices.Large storage overhead also prevents features such as privacy, reduced network bandwidth, and real-time processing from being incorporated into mobile apps.
  • Energy constraints: 640pJ for a 32bit DRAM memory access is 3 orders of magnitude above a 0.9pJ 32 bit floating point add.A 32bit SRAM cache access takes 5pJ, while large networks require costly DRAM accesses because they do not fit on chip.
  • Deep compression: Deep compression combines pruning, trained quantization, and Huffman coding to reduce storage while preserving original accuracy.The pipeline removes redundant connections, quantizes weights so multiple connections share values, and enables deployment of large networks on mobile devices.
  • Deep compression: Pruning and trained quantization compress networks independently enough to reduce storage to a few megabytes, allowing all weights to remain in on-chip cache.This avoids energy-consuming off-chip DRAM access and later enabled the compressed-model EIE accelerator.

2 NETWORK PRUNING

The method begins by learning network connectivity through standard training and then pruning, building on prior CNN-pruning work that reduced complexity without accuracy loss. The resulting sparse structure is stored compactly using sparse formats and relative index encoding.

  • Pruning: Network pruning reduces CNN complexity and over-fitting, and prior work achieved pruning of state-of-the-art CNNs without accuracy loss.The proposed approach builds on that pruning strategy, beginning with normal network training to learn connectivity.
  • Sparse storage: The sparse structure is stored in compressed sparse row or compressed sparse column format, requiring 2a + n + 1 numbers.Here, a denotes the number of non-zero elements, while n denotes the number of rows or columns.
  • Index encoding: 8 bits encode index differences in convolutional layers, while 5 bits encode them in fully connected layers.Relative index encoding replaces absolute positions; oversized differences are handled with filler-zero padding.

3 TRAINED QUANTIZATION AND WEIGHT SHARING

Trained quantization compresses pruned networks by sharing weights within each layer and storing connection indices into a shared-weight table. The method uses post-training k-means clustering, fine-tunes shared weights, and finds linear centroid initialization performs best among the tested methods.

  • 3 TRAINED QUANTIZATION AND WEIGHT SHARING: Weight sharing reduces storage by representing multiple connections with shared weights and storing only a table index for each connection.Shared weights are fine-tuned after quantization.
  • 3 TRAINED QUANTIZATION AND WEIGHT SHARING: For k clusters, each connection requires log2(k) index bits instead of its original b bits.The compression calculation assumes n connections, each originally represented with b bits.
  • 3 TRAINED QUANTIZATION AND WEIGHT SHARING: A 16-weight layer quantized into 4 shared weights achieves a 3.2 compression rate using 32-bit weights and 2-bit indices.The storage includes four effective weights and sixteen indices.
  • 3 TRAINED QUANTIZATION AND WEIGHT SHARING: K-means clustering identifies shared weights independently for each layer after the network is fully trained, unlike HashNet’s pre-training hash-based assignment.The clusters minimize within-cluster sum of squares, and weights are not shared across layers.
  • 3 TRAINED QUANTIZATION AND WEIGHT SHARING: Linear centroid initialization works best after clustering and fine-tuning, because it better represents the few large-magnitude weights than Forgy or density-based initialization.The comparison examines Forgy, density-based, and linear initialization methods.

4 HUFFMAN CODING

Huffman coding provides lossless compression by assigning variable-length codes according to symbol frequency. Applied to biased distributions of quantized weights and sparse matrix indices, it saves 20%–30% of network storage.

  • 4 HUFFMAN CODING: Huffman coding is an optimal prefix code for lossless compression, assigning fewer bits to more common symbols.Codeword lengths are derived from symbol occurrence probabilities.
  • 4 HUFFMAN CODING: Quantized weights cluster around two peaks, while sparse matrix index differences are rarely above 20.Both distributions are therefore non-uniform and biased.
  • 4 HUFFMAN CODING: 20%–30%: Huffman coding these non-uniform quantized weights and sparse matrix indices saves this fraction of network storage.The result is based on the distributions shown for AlexNet’s last fully connected layer.

5 EXPERIMENTS

Experiments on MNIST and ImageNet show that pruning, trained quantization, and Huffman coding reduce network storage by 35×–49× without accuracy loss. The resulting models fit on-chip SRAM, reducing memory and bandwidth requirements for deployment.

  • Compression results: 35×–49× storage savings were achieved across four networks without loss of accuracy.The experiments covered two networks on MNIST and two on ImageNet.
  • Deployment implications: 6.9MB was the compressed size of AlexNet, down from 240MB, enabling storage in on-chip SRAM instead of energy-consuming DRAM.Reduced layers also have modest bandwidth requirements, whereas uncompressed bandwidth requirements are prohibitive.
  • MNIST: 32× compression came from pruning and quantization on MNIST, while Huffman coding increased compression to 40×.The reported compression rates include codebook and sparse-index overhead.
  • ImageNet: 2.88% of its original size was sufficient for AlexNet without impacting accuracy on ImageNet.The reference AlexNet had 61 million parameters, with 57.2% top-1 and 80.3% top-5 accuracy.
  • ImageNet: 49× compression reduced VGG-16 while preserving accuracy, with 8-bit convolutional weights and 5-bit fully connected weights.Each of the two largest fully connected layers was pruned to less than 1.6% of its original size.

6 DISCUSSIONS

The discussion shows that pruning and quantization complement each other, while their effectiveness depends on layer precision and centroid initialization. Pruning also improves batch-size-one inference speed, energy efficiency, and storage efficiency by reducing memory-related costs and overhead.

  • Compression and accuracy: 3% of original size is achievable without the sharp accuracy loss seen when pruning or quantization is used alone below 8%.The combined method preserves accuracy more effectively than either individual method at high compression rates.
  • Layerwise quantization: CONV layers require more precision than FC layers: accuracy drops significantly below 4 bits for CONV, but not until 2 bits for FC.Pruning and quantization produce nearly identical accuracy trends to quantization alone, indicating that pruning does not impair quantization.
  • Centroid initialization: Linear initialization outperforms density and random initialization at 2–8 bits except at 3 bits.Its centroids span the minimum-to-maximum range, retaining large weights that the other methods tend to cluster into small centroids.
  • Performance and energy: 3× to 4× speedup and 3× to 7× lower energy consumption were achieved by pruned layers over dense layers at batch size = 1.The gains arise because the smaller memory footprint reduces data-transfer overhead, especially for matrices that do not fit in cache; batch size = 1 targets real-time inference.
  • Storage overhead: The weights and sparse indexes each use roughly 5 bits on average, while codebook overhead is very small and often negligible.Pruning requires indexes for nonzero elements and quantization requires a codebook, but both factors are included in the storage analysis.

7 RELATED WORK

Prior work addresses neural-network redundancy through reduced-precision representations, weight sharing, architectural changes, and pruning. The paper also relates its sparse-network results to theoretical analyses of random-like sparse networks.

  • Reduced precision: Deep networks are over-parameterized, creating redundant computation and memory usage that motivates compression methods.Earlier work explored 8-bit integer activations instead of 32-bit floating point and ternary weights with 3-bit activations.
  • Sparse-network theory: The paper’s empirical results align with theoretical work showing that random-like sparse networks with +1/0/-1 weights can be reversible and trainable in provably polynomial time.These theoretical properties are attributed to Arora et al. (2014).
  • Weight sharing: HashedNets reduce model size by randomly grouping connection weights into hash buckets whose connections share one parameter value.Unlike learned weight sharing, HashedNets determine binning in advance with a hash function.
  • Architectural parameter reduction: Global average pooling replaces fully connected layers to reduce parameters, but it makes transfer learning through fully connected-layer fine-tuning more difficult.Network in Network and GoogLeNet achieved state-of-the-art results on several benchmarks using this approach.
  • Network pruning: Network pruning reduces complexity and over-fitting, with earlier methods pruning connections using biased weight decay or Hessian-based criteria.Optimal Brain Damage and Optimal Brain Surgeon argued that Hessian-based pruning can be more accurate than magnitude-based pruning.

8 FUTURE WORK

The quantized, weight-shared network has not been fully benchmarked because existing sparse-matrix libraries lack required indirect lookup support, motivating customized GPU kernels or ASIC hardware.

  • Limitation: The quantized network with weight sharing remains unbenchmarked because cuSPARSE and MKL SPBLAS lack indirect matrix entry lookup and supported relative indices in CSC or CSR formats.Consequently, the full cache-enabled advantage of Deep Compression has not been fully revealed.
  • Future solutions: A software solution is to develop customized GPU kernels supporting indirect lookup and relative indexing.
  • Future solutions: A hardware solution is to build a custom ASIC architecture for the required sparse operations.

9 CONCLUSION

Deep Compression reduces neural-network storage without affecting accuracy through pruning, weight-sharing quantization, and Huffman coding. Experiments report substantial compression for AlexNet, VGG-16, and LeNet without accuracy loss.

  • Deep Compression combines pruning unimportant connections, weight-sharing quantization, and Huffman coding.The pipeline applies these three stages sequentially.
  • 35× reduction in weight storage was achieved for AlexNet without loss of accuracy.
  • 49× compression was achieved for VGG-16 without loss of accuracy.
  • 39× compression was achieved for LeNet without loss of accuracy.

A APPENDIX: DETAILED TIMING / POWER REPORTS OF DENSE & SPARSE NETWORK LAYERS

The appendix details controlled timing and power measurements for dense and sparse network layers across batch sizes and hardware platforms. It specifies the computational kernels, measurement exclusions, instrumentation, and repetition procedures used to obtain stable comparisons.

  • Timing methodology: 4096 input samples were averaged per input to measure layer execution time while reducing variance.GPU timings excluded cudaMalloc and cudaMemcpy overhead.
  • Timing methodology: Batch size 1 used gemv for dense and csrmv for sparse computation, whereas batch size 64 used gemm and csrmm, respectively.
  • Power methodology: Power was measured on Titan X, Core i7-5930k, and Tegra K1 using nvidia-smi, pcm-power, and an external power meter, respectively.Tegra K1 readings were scaled to AP+DRAM according to the paper discussion.
  • Power methodology: 2x energy consumption was observed for dense versus sparse matrix multiplication on CPU because dense computation benefited from multi-threading.Each computation was repeated multiple times to obtain stable power measurements.
Loading 1510.00149v5…