PaperlayerView exact revision on arXiv

Source-linked AI summary

HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision

Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer

arXiv:1905.03696v1cs.CV

TL;DR

Neural-network deployment faces growing model-size, inference-speed, and power demands, while ultra-low uniform precision can degrade accuracy and mixed-precision selection is computationally difficult. HAWQ uses Hessian information to select layer precisions and fine-tuning order, achieving strong compression and accuracy across image-classification benchmarks, including a 1MB SqueezeNext model with 1.36% top-1 accuracy drop.

  • Problem

    Growing neural-network resource demands and exponential or factorial search spaces make systematic mixed-precision selection and fine-tuning-order determination important deployment challenges.

  • Method

    HAWQ uses second-order Hessian information to deterministically select relative layer precisions and block-wise fine-tuning order.

  • Results

    HAWQ achieves comparable or improved accuracy with higher compression across CIFAR-10 and ImageNet models, including a 1MB SqueezeNext model with only 1.36% top-1 accuracy drop.

  • Takeaways & Limitations

    Hessian-guided mixed-precision quantization supports highly compressed neural-network deployment, including hardware with very limited memory and power constraints.

  • Takeaways & Limitations

    Computing second-order information adds computational overhead, and the study focuses solely on image classification rather than tasks such as segmentation, detection, or natural language processing.

Abstract

from arXiv · show

Model size and inference speed/power have become a major challenge in the deployment of Neural Networks for many applications. A promising approach to address these problems is quantization. However, uniformly quantizing a model to ultra low precision leads to significant accuracy degradation. A novel solution for this is to use mixed-precision quantization, as some parts of the network may allow lower precision as compared to other layers. However, there is no systematic way to determine the precision of different layers. A brute force approach is not feasible for deep networks, as the search space for mixed-precision is exponential in the number of layers. Another challenge is a similar factorial complexity for determining block-wise fine-tuning order when quantizing the model to a target precision. Here, we introduce Hessian AWare Quantization (HAWQ), a novel second-order quantization method to address these problems. HAWQ allows for the automatic selection of the relative quantization precision of each layer, based on the layer's Hessian spectrum. Moreover, HAWQ provides a deterministic fine-tuning order for quantizing layers, based on second-order information. We show the results of our method on Cifar-10 using ResNet20, and on ImageNet using Inception-V3, ResNet50 and SqueezeNext models. Comparing HAWQ with state-of-the-art shows that we can achieve similar/better accuracy with $8\times$ activation compression ratio on ResNet20, as compared to DNAS~\cite{wu2018mixed}, and up to $1\%$ higher accuracy with up to $14\%$ smaller models on ResNet50 and Inception-V3, compared to recently proposed methods of RVQuant~\cite{park2018value} and HAQ~\cite{wang2018haq}. Furthermore, we show that we can quantize SqueezeNext to just 1MB model size while achieving above $68\%$ top1 accuracy on ImageNet.

I. INTRODUCTION

Neural-network deployment is increasingly constrained by model size, computation, and memory, while ultra-low-precision quantization can substantially reduce accuracy. HAWQ addresses the exponential precision-selection and factorial fine-tuning-order problems using Hessian information.

  • Motivation: Larger models and higher-resolution inputs increase FLOPs and memory demands, challenging deployment in computationally constrained environments.The paper highlights surveillance and ADAS systems as examples of such environments.
  • Quantization challenge: Quantizing weights and activations is necessary to reduce inference memory, but uniform ultra-low precision can cause significant accuracy loss.Different layers do not generally tolerate the same quantization level.
  • Research gap: Choosing layer-wise mixed precision has an exponential search space, while selecting the quantization-aware fine-tuning order has factorial complexity.These two combinatorial problems make brute-force procedures impractical for deep networks.
  • Contribution: HAWQ deterministically selects relative layer precision from Hessian spectra and determines block-wise fine-tuning order using second-order information.The method is presented as a systematic alternative to ad-hoc or exhaustive search.
  • Contribution: HAWQ achieves up to 1% higher precision, 20% smaller model size, and 8× smaller activation size than compared methods.These headline comparison figures are reported in the introduction.

II. RELATED WORK

Prior compression work includes compact architectures, pruning, distillation, hardware–software co-design, and quantization. Mixed-precision and multi-stage quantization improve accuracy but retain large search spaces that HAWQ targets with second-order information.

  • Pruning: Pruning reduces nonzero weights and is especially useful for models with very large fully connected layers.The paper gives AlexNet and VGG as representative cases.
  • Knowledge distillation: Knowledge distillation compresses a large pretrained model into a smaller model.One cited result reduces model size by 3.6× while reaching 91.61% accuracy on Cifar-10.
  • Efficient architectures: Compact, hardware-efficient architectures reduce memory footprint through network design and can combine with quantization.SqueezeNet is cited as achieving a 500× smaller memory footprint than AlexNet through design and quantization.
  • Quantization: Quantization uses lower-bit representations, increasing arithmetic intensity and reducing memory accesses, especially for memory-bound layers.The paper presents quantization as orthogonal to architecture redesign.
  • Mixed-precision quantization: Ultra-low-precision quantization can degrade accuracy, motivating mixed-precision and multi-stage methods.Both approaches face exponentially large search spaces, and existing precision choices are often ad hoc and model-specific.

III. METHODOLOGY

HAWQ models quantization sensitivity with second-order information from block Hessians. It uses Hessian eigenvalues and quantization perturbations to guide layer precision and sensitivity-aware processing.

  • Model formulation: The network is partitioned into blocks, each containing one or more layers or residual blocks with learnable parameters.The block partition supports block-wise sensitivity analysis and fine-tuning.
  • Model formulation: The supervised objective is an empirical loss over training examples, with θ combining the parameters of all blocks.The loss is defined over inputs, labels, and model parameters.
  • Quantization: Quantization maps weights or activations in intervals (t_j, t_{j+1}] to discrete values q_j using k bits.Uniform quantization divides the range equally, while the paper emphasizes that layer sensitivity differs.
  • Second-order information: Gradient sensitivity can be misleading because a zero gradient does not imply insensitivity to perturbations.For a parabola at its origin, curvature captured by the Hessian distinguishes sensitivity levels.
  • Second-order information: Hessian eigenvalues measure curvature-based sensitivity, with larger eigenvalues indicating greater susceptibility to quantization.Flat regions are more amenable to aggressive quantization, whereas sharp regions amplify rounding errors.

A. Second-Order Information

HAWQ estimates each network block’s second-order sensitivity through Hessian eigenvalues without explicitly forming the Hessian. Matrix-free power iteration computes the required Hessian–vector products and top eigenvalues.

  • Hessian spectrum: HAWQ computes each block’s Hessian eigenvalues to measure second-order sensitivity.The Hessian is treated as a second-order operator for the loss with respect to block parameters.
  • Hessian spectrum: Because blocks can be large, HAWQ avoids explicitly forming the Hessian and uses matrix-free power iteration instead.Power iteration repeatedly evaluates Hessian–vector products.
  • Hessian spectrum: The Hessian–vector product is obtained from gradients with respect to block parameters and a dimension-matched vector v.The method uses the relation described in Equation 4 to compute the required matrix–vector multiplication.
  • Hessian spectrum: Higher Hessian eigenvalues indicate greater quantization sensitivity, while flatter regions are more amenable to aggressive quantization.The paper connects curvature to the amplification of quantization or round-off errors.

B. Algorithm

HAWQ determines mixed-precision assignments and fine-tuning order using Hessian-based block metrics. It balances curvature, block size, and quantization perturbation so sensitive or influential blocks receive appropriate treatment.

  • Algorithm: HAWQ approximates each block Hessian with an identity matrix scaled by its top eigenvalue.This block-diagonal approximation provides the second-order basis for the algorithm’s sorting metrics.
  • Algorithm: Large-eigenvalue layers receive less aggressive quantization because their curvature makes ultra-low precision more disruptive.The method uses the Hessian spectrum to rank relative precision across blocks.
  • Algorithm: HAWQ weights Hessian spectrum by block memory footprint, assigning lower bits to large blocks with smaller eigenvalues.This compromises between sensitivity and the memory cost of using higher precision.
  • Algorithm: For fine-tuning, HAWQ ranks blocks by Ω_i = λ_i∥Q(W_i) − W_i∥2, combining curvature with quantization perturbation.The metric depends on the selected quantization bits.
  • Algorithm: Blocks are fine-tuned in descending Ω_i order because highly affected blocks can significantly influence other blocks.The paper motivates multi-stage fine-tuning because ordering can change final accuracy.

IV. RESULTS

HAWQ is evaluated across Cifar-10 and ImageNet models, showing improved accuracy–compression trade-offs over direct quantization and several prior methods. Results also demonstrate extreme model-size reduction for SqueezeNext.

  • Cifar-10: 8× higher activation compression is achieved with similar testing performance to DNAS on ResNet20.The comparison reports the lowest weight and activation bits for mixed-precision methods.
  • Cifar-10: 0.15% accuracy drop with 13.11× compression makes HAWQ stronger than LQ-Nets on ResNet20.HAWQ also matches DNAS accuracy with 8× higher activation compression.
  • ImageNet: 2% generalization gap with 12.04× compression is reported for HAWQ on Inception-V3, outperforming the cited Integer-Only and RVQuant comparisons.Direct quantization causes 7.69% accuracy degradation, while Integer-Only loses more than 2% despite higher bit precision.
  • ImageNet: 1MB model size with only 1.36% top-1 accuracy drop is achieved by mixed-precision HAWQ on SqueezeNext.Uniform 8-bit quantization produces only a 0.04% top-1 accuracy drop, while direct quantization causes a 3.98% degradation.

V. ABLATION STUDY

The ablation study separates HAWQ into two Hessian-informed decisions: assigning relative precision and ordering block-wise fine-tuning.

  • HAWQ has two main steps: selecting relative precision orders for blocks and selecting their relative fine-tuning order.

A. Hessian AWare Mixed Precision Quantization

The mixed-precision ablation compares HAWQ’s Hessian-based precision ordering with a reversed ordering on quantized Inception-V3. HAWQ achieves better accuracy, higher compression, and faster convergence than the reverse strategy.

  • 12.0× compression is achieved by HAWQ versus 7.2× for HAWQ-Reverse-Precision.
  • 30 epochs are needed for HAWQ to converge, compared with 50 epochs for HAWQ-Reverse-Precision, which converges to a sub-optimal value.

B. Block-Wise Fine-Tuning

The block-wise fine-tuning ablation orders blocks by descending Hessian-based Ω_i and compares this with ascending-order reverse tuning. HAWQ reaches convergence faster on Inception-V3.

  • Blocks are fine-tuned in descending order of Ω_i, while HAWQ-Reverse-Tuning uses ascending order.
  • 25 epochs are sufficient for HAWQ to converge and switch blocks, whereas HAWQ-Reverse-Tuning takes more than 50 epochs for the same block.

VI. CONCLUSIONS

HAWQ uses second-order information to select quantization precision and block-wise fine-tuning order, achieving strong compression and accuracy across the reported classification experiments. The paper also identifies computational, task, implementation, and bit-selection scope limitations.

  • HAWQ uses Hessian information to systematically select quantization precision and block-wise fine-tuning order.
  • ResNet20 on Cifar-10 achieves similar testing performance to [36] with 8× higher activation compression ratio.
  • 1.93% accuracy drop is reported for Inception-V3 with 2-bit weights and 4-bit activations.
  • 75.48% top-1 accuracy and 7.96MB model size are achieved for ResNet50, versus 75.30% and 9.22MB for HAQ.
  • SqueezeNext reaches 68.02 top-1 accuracy on ImageNet with a 1MB model size.
  • Computing second-order information adds overhead, image classification is the sole evaluated task, mixed-precision embedded implementation is less straightforward, and absolute bit values are not determined.

B. Extra results

The paper provides additional visualizations and implementation details for Hessian-aware mixed-precision quantization, including block separation and precision assignments for ResNet20 and Inception-V3. It also examines how Ω_i varies with target weight precision and visualizes loss landscapes across network blocks.

  • Quantization analysis: Ω_i is evaluated as a function of target weight bit precision for ResNet20 on Cifar-10.These results are presented in Table V.
  • Implementation details: Block separation and individual weight/activation precision details are reported for ResNet20 on Cifar-10 and Inception-V3 on ImageNet.The exact bit precision for different blocks is provided in Tables VI and VII.
  • Loss landscapes: The ResNet20 loss landscape is visualized in three dimensions along the first two dominant Hessian eigenvectors for all blocks.The perturbations are controlled by scalars ϵ1 and ϵ2, and the corresponding eigenvalue distributions are shown in Figure 1.
  • Loss landscapes: The Inception-V3 loss landscape is visualized in three dimensions along the first two dominant Hessian eigenvectors for all blocks.The perturbations are controlled by scalars ϵ1 and ϵ2, with corresponding eigenvalue distributions shown in Figure 1.
Loading 1905.03696v1…