PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards the Limit of Network Quantization

Yoojin Choi, Mostafa El-Khamy, Jungwon Lee

arXiv:1612.01543v2cs.CVcs.LGcs.NE

TL;DR

The paper addresses how to compress deep neural-network parameters while limiting performance loss under a compression-ratio constraint. It analyzes quantization error through the Hessian, develops Hessian-weighted and entropy-constrained schemes, and reports high compression ratios using uniform quantization followed by Huffman coding.

  • Problem

    The paper seeks quantization schemes that minimize network performance loss under a compression-ratio constraint, because conventional k-means ignores parameter-error impacts and variable-length coding effects.

  • Method

    The paper uses Hessian-weighted distortion for clustering and relates variable-length-coded quantization to entropy-constrained scalar quantization, with uniform and Lloyd-like iterative solutions.

  • Results

    Compression ratios of 51.25 for LeNet, 22.17 for 32-layer ResNet, and 40.65 for AlexNet are achieved using uniform quantization followed by Huffman coding at no or marginal performance loss.

  • Takeaways & Limitations

    Hessian weighting accounts for differing quantization-error impacts within and across layers, while entropy-constrained formulations incorporate variable-length coding into quantization design.

  • Takeaways & Limitations

    The analysis assumes a trained network and, for Hessian-based approximation, a local minimum with an approximately quadratic loss and diagonal Hessian.

Abstract

from arXiv · show

Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.

1 INTRODUCTION

The paper targets storage compression for deep neural-network parameters while preserving performance under a compression-ratio constraint. It replaces conventional k-means-based quantization with Hessian-aware and entropy-constrained schemes.

  • Motivation: Deep neural networks’ large parameter counts make deployment on resource-limited devices difficult and motivate storage-focused quantization.The paper reduces the number of distinct network parameter values to compress parameter storage.
  • Problem: Conventional k-means quantization overlooks how quantization errors affect performance and does not optimize the procedure for a specified compression ratio.The paper formulates quantization as minimizing performance loss under a compression-ratio constraint.
  • Hessian-weighted quantization: Hessian-weighted distortion approximately quantifies quantization-induced performance loss, motivating Hessian-weighted k-means clustering.The method accounts for differing parameter sensitivities when selecting cluster assignments.
  • Entropy-constrained quantization: With optimal variable-length coding, network quantization can be reduced to entropy-constrained scalar quantization, yielding uniform and Lloyd-like iterative solutions.The proposed ECSQ solutions address coding lengths and cluster sizes rather than cluster count alone.
  • Practical alternative: Gradient second-moment estimates, or functions such as their square root, provide a training-time alternative to Hessian computation when Adam is used.This alternative is available at the end of training without additional computation.
  • Joint-layer quantization: The paper also supports quantizing parameters from all layers together, allowing Hessian weighting to account for differing error impacts across layers.This avoids optimizing compression rates separately for each layer.

2 NETWORK QUANTIZATION

Network quantization groups parameters into clusters that share representative values and then encodes those values to reduce storage. The paper highlights why ordinary k-means distortion is insufficient when parameter sensitivities and variable-length coding matter.

  • Problem formulation: Quantization assigns parameters to clusters sharing representative values, followed by fixed- or variable-length binary coding of the quantized parameters.The objective is to reduce storage while minimizing performance degradation.
  • Compression constraint: The compression ratio depends on cluster count, cluster sizes, and binary codeword lengths when variable-length coding is used.For fixed-length codes, the ratio becomes a function of the number of clusters when the original parameter count and bit width are fixed.
  • Conventional k-means: K-means partitions parameters into k disjoint clusters while minimizing mean square quantization error.Each cluster is represented by its mean value.
  • Limitations of k-means: Ordinary k-means may be suboptimal because it treats parameter errors equally and ignores compression-ratio effects from subsequent binary coding.Parameters can differ in how strongly their quantization errors affect network performance.

3 HESSIAN-WEIGHTED NETWORK QUANTIZATION

The paper derives a locally appropriate Hessian-weighted distortion measure for quantization loss and uses it to design network quantization across parameters and layers. It also proposes practical Hessian alternatives and computation strategies for applying this approach to large neural networks.

  • Hessian-weighted quantization error: The Hessian-weighted distortion measure approximately quantifies local performance degradation caused by network quantization.The derivation uses a Taylor expansion around a trained local minimum and a diagonal Hessian approximation.
  • Hessian-weighted k-means clustering: Hessian-weighted k-means clustering assigns larger distortion penalties to parameters whose quantization errors have greater expected impact on the loss.Larger second-order partial derivatives receive larger penalties, discouraging large deviations from the original parameter values.
  • Hessian-weighted k-means clustering: Hessian-weighted k-means clustering is locally optimal for fixed-length coding, while Lloyd’s algorithm provides an efficient heuristic using Hessian-weighted means as cluster centers.The fixed-length setting makes compression depend solely on the number of clusters.
  • Alternative of Hessian: Functions of Adam’s second moment estimates provide a Hessian alternative that is available after training at no additional computation cost.The paper motivates this alternative by observing that Adam’s second moment estimates act like the Hessian in Newton’s method.
  • Quantization of all layers: Quantizing all network layers jointly with Hessian weighting accounts for differing quantization-error impacts within and across layers.This avoids relying on heuristic layer importance assignments and can address layer-specific sensitivity directly.
  • Quantization of all layers: Joint layer quantization avoids exponential-time optimization of compression-ratio combinations across individual layers as network depth increases.The number of possible per-layer compression-ratio combinations grows exponentially with the number of layers.

4 ENTROPY-CONSTRAINED NETWORK QUANTIZATION

The paper reformulates compression-constrained network quantization as entropy-constrained scalar quantization when optimal variable-length coding follows clustering. It proposes uniform and iterative ECSQ solutions, with uniform quantization efficient under Huffman coding but not fixed-length coding.

  • Entropy-constrained formulation: Optimal variable-length coding approximately replaces the compression-ratio constraint with an entropy constraint on clustering outputs.The source distribution is determined by cluster proportions, and the resulting problem is identified as ECSQ.
  • Uniform quantization: Uniform quantization is asymptotically optimal for mean-square error as the number of clusters k approaches infinity under a reasonably smooth source density.This result motivates its use as a simple ECSQ solution.
  • Uniform quantization: Uniform quantization divides parameters using uniformly spaced thresholds and sets each cluster center to its parameter mean.Hessian-weighted means can be used instead to retain Hessian-weighting.
  • Uniform quantization: Uniform quantization is efficient when Huffman coding follows, but inefficient with fixed-length coding.Its coding efficiency therefore depends on the subsequent variable-length coding scheme.
  • Iterative ECSQ: The iterative ECSQ algorithm, similar to Lloyd’s k-means algorithm, finds a local optimum for a given discrete source.It is more complicated than uniform quantization and is derived for network quantization from the general ECSQ setting.

5 EXPERIMENTS

Experiments evaluate the quantization schemes on LeNet, 32-layer ResNet, and AlexNet, including joint-layer quantization and pruned models. Results show that the preferred method depends on coding, while Hessian approximations can preserve performance and uniform quantization plus Huffman coding achieves high compression ratios.

  • Experiment setup: Experiments cover LeNet on MNIST, 32-layer ResNet on CIFAR-10, and AlexNet on ImageNet ILSVRC-2012.The methods are evaluated with and without pruning, quantizing all network parameters together.
  • 32-layer ResNet: Hessian-weighted k-means performs best with fixed-length coding on 32-layer ResNet, including after fine-tuning.With Huffman coding, uniform quantization and iterative ECSQ outperform both Hessian-weighted and conventional k-means clustering.
  • Hessian alternatives: 1,000-sample Hessian estimates yield almost the same performance as estimates computed with more samples.Replacing the Hessian with square roots of gradient second-moment estimates provides similar performance.
  • Compression results: 51.25, 22.17 and 40.65 compression ratios are achieved for LeNet, 32-layer ResNet and AlexNet, respectively, using uniform quantization followed by Huffman coding.The compressed sizes are 1.95%, 4.51% and 2.46% of the original model sizes, respectively, at no or marginal performance loss.

6 CONCLUSION

The paper designs quantization schemes that minimize quantization-related performance loss under compression constraints. It combines Hessian-weighted clustering with entropy-constrained coding formulations and proposes uniform and iterative ECSQ solutions.

  • Conclusion: Hessian-weighted distortion measures quantization error according to parameter importance and supports joint quantization across all network layers.The paper connects compression-constrained quantization to entropy-constrained data compression and proposes uniform and iterative ECSQ solutions.
  • Conclusion: The proposed schemes target minimum performance loss for a given compression ratio constraint.The conclusion identifies conventional k-means quantization as suboptimal for this objective.

A.1 FURTHER DISCUSSION ON THE HESSIAN-WEIGHTED QUANTIZATION ERROR

The discussion addresses Hessian approximation and the scope of Hessian-weighted distortion. Diagonal Hessian approximation simplifies optimization but introduces unverified potential loss, while the distortion relation extends to locally quadratic objectives beyond neural networks.

  • Hessian approximation: Diagonal Hessian approximation simplifies network-quantization optimization but may incur performance loss that the paper conjectures is small.The conjecture is not verified because computing the full Hessian is too complex and costly.
  • Scope: Hessian-weighted distortion applies to any model whose objective is approximately quadratic in the parameters being quantized.The paper therefore states that its quantization methods are not specific to neural networks under this local-quadratic condition.
  • Retraining assumption: The formulation analyzes quantization loss without retraining, although experiments fine-tune quantized cluster centers afterward.The interaction between quantization and retraining is not included in the theoretical formulation.

A.2 EXPERIMENT RESULTS FOR UNIFORM QUANTIZATION

Figure 3 compares accuracy against average codeword length after quantization, Huffman coding, and fine-tuning for two uniform-quantization variants. Hessian-weighted means slightly outperform non-weighted means.

  • The comparison uses uniform quantization with non-weighted means and uniform quantization with Hessian-weighted means.
  • Uniform quantization with Hessian-weighted means slightly outperforms uniform quantization with non-weighted means.
  • Figure 3 plots accuracy against average codeword length per network parameter after quantization, Huffman coding, and fine-tuning.

A.3 FURTHER DISCUSSION ON THE ITERATIVE ALGORITHM FOR ECSQ

The paper formulates entropy-constrained network quantization through a Lagrangian cost balancing distortion and entropy. Its iterative heuristic alternates cluster assignment and parameter updates using individual Lagrangian costs, similarly to Lloyd’s algorithm.

  • The ECSQ formulation defines a Lagrangian cost function for entropy-constrained network quantization.
  • Algorithm 1 initializes cluster proportions and centers before iteratively assigning parameters and updating cluster statistics.
  • The iteration stops when the Lagrangian cost decreases by less than a specified threshold.
  • The optimization seeks k parameter clusters that minimize the Lagrangian cost Jλ(C1, C2, . . . , Ck).
  • Unlike Lloyd’s algorithm, the ECSQ assignment step minimizes individual Lagrangian cost dλ(i, j), incorporating quantization error and expected coding cost.
Loading 1612.01543v2…