HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

Zhen Dong; Zhewei Yao; Yaohui Cai; Daiyaan Arfeen; Amir Gholami; Michael W. Mahoney; Kurt Keutzer

Source-linked AI summary

HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, Kurt Keutzer

arXiv:1911.03852v1Nov 10, 2019cs.CV

TL;DR

Ultra-low-precision quantization can hurt generalization, while mixed-precision search is exponentially large and prior HAWQ has sensitivity, bit-selection, and activation-quantization limitations. HAWQ-V2 uses average Hessian trace, Pareto-frontier bit selection, and activation analysis, achieving state-of-the-art results across tasks, including improved object-detection accuracy at smaller model size.

Problem
Ultra-low-precision quantization can degrade generalization, while mixed-precision settings have an exponentially large search space and prior HAWQ requires manual selection, ignores most Hessian eigenvalues, and omits activation quantization.
Method
HAWQ-V2 uses average Hessian trace for sensitivity, Pareto-frontier selection of layer precisions, and Hessian analysis for mixed-precision activation quantization.
Results
HAWQ-V2 achieves state-of-the-art results across image classification and object detection, including more than 1.6 mAP higher accuracy than FQN with a 17.9MB model.
Takeaways & Limitations
Second-order Hessian information can support practical quantization gains, including improved object-detection accuracy with a smaller model than FQN.
Takeaways & Limitations
Using second-order information throughout training remains a future step rather than a reported component of the quantization approach.

Abstract

from arXiv · show

Quantization is an effective method for reducing memory footprint and inference time of Neural Networks, e.g., for efficient inference in the cloud, especially at the edge. However, ultra low precision quantization could lead to significant degradation in model generalization. A promising method to address this is to perform mixed-precision quantization, where more sensitive layers are kept at higher precision. However, the search space for a mixed-precision quantization is exponential in the number of layers. Recent work has proposed HAWQ, a novel Hessian based framework, with the aim of reducing this exponential search space by using second-order information. While promising, this prior work has three major limitations: (i) HAWQV1 only uses the top Hessian eigenvalue as a measure of sensitivity and do not consider the rest of the Hessian spectrum; (ii) HAWQV1 approach only provides relative sensitivity of different layers and therefore requires a manual selection of the mixed-precision setting; and (iii) HAWQV1 does not consider mixed-precision activation quantization. Here, we present HAWQV2 which addresses these shortcomings. For (i), we perform a theoretical analysis showing that a better sensitivity metric is to compute the average of all of the Hessian eigenvalues. For (ii), we develop a Pareto frontier based method for selecting the exact bit precision of different layers without any manual selection. For (iii), we extend the Hessian analysis to mixed-precision activation quantization. We have found this to be very beneficial for object detection. We show that HAWQV2 achieves new state-of-the-art results for a wide range of tasks.

1. Introduction.

HAWQ-V2 addresses the accuracy, search, and activation-quantization limitations of prior mixed-precision methods by using Hessian information to automate precision assignment. It combines broader sensitivity analysis, automatic bit selection, and mixed-precision activations, achieving state-of-the-art results across tasks.

1. Introduction.: Ultra-low-bit quantization reduces model size and inference cost but can significantly degrade accuracy, motivating mixed-precision assignments across layers.Mixed precision keeps sensitive layers at higher precision and less sensitive layers at lower precision.
1. Introduction.: Mixed-precision quantization has an exponentially large layer-wise search space, while existing search methods are resource-intensive, time-consuming, and initialization-sensitive.These properties make deployment especially challenging in online learning scenarios requiring frequent quantization.
1. Introduction.: Prior HAWQ reduces the search space using top Hessian eigenvalues but provides only relative sensitivity, requires manual bit selection, and omits mixed-precision activation quantization.Its sensitivity metric also ignores the remainder of the Hessian spectrum.
1. Introduction.: HAWQ-V2 uses the average Hessian trace as a sensitivity metric, computes it efficiently with Hutchinson’s algorithm, and selects layer precisions automatically through a Pareto-frontier method.The paper reports computing trace information for all 54 ResNet50 layers in 30 minutes using four GPUs.
1. Introduction.: HAWQ-V2 extends Hessian analysis to mixed-precision activations and reports benefits for object detection, including 34.4 mAP in the described model.The method is evaluated on image classification and object detection and reports new state-of-the-art results.

2. Methodology.

HAWQ-V2 uses average Hessian trace to measure layer sensitivity and reduce mixed-precision search, then selects exact bit settings through a Pareto frontier. The analysis also extends to activations and yields practical trace estimation and quantization choices.

Mixed-precision quantization assigns higher precision to sensitive layers, but its search space grows exponentially with the number of layers.This motivates using second-order information to constrain candidate settings.
Average Hessian trace is proposed as a better sensitivity metric than the top Hessian eigenvalue because it captures the full spectrum.The paper’s example shows equal top eigenvalues can conceal different sensitivity along other directions.
Hutchinson’s algorithm estimates Hessian traces without explicitly forming the Hessian, with trace computation reported as orders of magnitude faster than network training.The method computes trace information efficiently from Hessian-vector products.
Different blocks exhibit substantially different average Hessian traces and loss-landscape sharpness, so sensitive blocks receive more bits while flatter blocks can be quantized more aggressively.Inception-V3 block 4 is described as sensitive, whereas block 16 has a flat landscape.
Activation quantization uses the same Hessian-trace analysis, but activation Hessians are block diagonal and can be computationally expensive to handle directly.The blocks correspond to differently sized per-input activation components.
Relative sensitivity alone cannot determine exact layer bit widths, so HAWQ-V2 sorts admissible settings by total second-order perturbation and chooses the minimum-Ω setting for a target model size.The Pareto-frontier procedure removes the manual precision selection used in HAWQ.

3. Results.

HAWQ-V2 is evaluated on ImageNet classification and COCO object detection, using Hessian-guided quantization without AutoML search or manual bit-precision selection. It achieves strong accuracy–compression results across Inception-V3, ResNet50, SqueezeNext, and RetinaNet.

3.1. Hutchinson: 50 Hutchinson steps and more than 512 data points provide accurate, low-variance Hessian-trace estimates for the reported experiments.The convergence study uses trace estimation on ResNet50 block 21 and supports calculating average Hessian traces for 54 blocks.
3.2. ImageNet: HAWQ-V2 automatically generates the exact network-wide precision setting, avoiding HAWQ’s manual bit selection while achieving better accuracy.For Inception-V3, HAWQ reports a 2% accuracy gap and 12.04× compression, whereas HAWQ-V2 improves accuracy without manual selection.
3.2. ImageNet: 75.76% accuracy is achieved on ResNet50 with a 7.99MB model, improving on HAWQ’s 75.48% accuracy at comparable model size.The HAWQ-V2 result is obtained without heuristic knowledge or manual efforts.
3.2. ImageNet: 68.38% accuracy is achieved on SqueezeNext with a 1.07MB model, combining higher accuracy with a smaller model than HAWQ.HAWQ produces a 1MB model with a 1.36% top-1 accuracy drop, while HAWQ-V2 slightly reduces model size and raises accuracy.
3.3. Microsoft COCO: 34.1 mAP is achieved on RetinaNet using mixed-precision weights and uniform 4-bit activations, exceeding FQN by 1.6 mAP with a smaller model.Mixed-precision activations further reach 34.4 mAP with only a marginal change in activation compression ratio.

4. Conclusions and Future Work.

The paper concludes that HAWQ-V2 improves Hessian-based quantization through richer curvature information, activation quantization, and automatic precision selection. It reports practical gains while identifying training-time and data-access extensions for future work.

4. Conclusions: HAWQ-V2’s results show that higher-order Hessian information can produce significant practical gains rather than serving only as a theoretical tool.The conclusion connects this observation to the reported quantization results.
4. Future Work: A future direction is using second-order information during training to encourage flatter loss landscapes and potentially enable lower-bit quantization without accuracy degradation.This is presented as a proposed future step, not an achieved result.
4. Future Work: Another future direction is extending the analysis to settings where training data is inaccessible because of practical constraints such as privacy regulations.The paper identifies data unavailability as common in practice.

Appendix A. Quantization Details.

The appendix defines the tensor quantization operation used during the forward pass and describes the resulting search-space formulation for mixed-precision settings and fine-tuning orders.

Quantization Details: Each weight or activation tensor X is quantized during the forward pass using rounding, a quantization step ∆, integer indices, and clamping to a specified range.The range may be narrower than [min, max] to remove outliers and better represent most tensor values.
Quantization Details: The quantization range [q0, q2^k−1] can be selected as a subinterval of the tensor’s full range to reduce outlier effects.Clamp maps values outside the range to its endpoints.
Search Formulation: The combined search space is B × C, where B counts mixed-precision settings and C counts progressive quantization-aware fine-tuning orders.This separates precision assignment choices from fine-tuning-order choices.

Appendix B. Search Space.

Hessian-trace ordering reduces mixed-precision assignment to a partitioning problem over ordered layers, replacing the original combinatorial search with a smaller structured space.

Search Space: The original fine-tuning-order search can grow as L!, while the broader search involves Stirling numbers of the second kind with growth between O(L!) and O(L^L).Here L is the number of layers and S(L, i) denotes Stirling numbers of the second kind.
Search Space: If average Hessian trace orders layers by sensitivity, assigning qi ≥ qj to layers with Tr(Bi)/ni > Tr(Bj)/nj preserves that sensitivity ordering.This creates an ordered layer sequence for precision assignment.
Search Space: Using j precision options reduces the mixed-precision problem to partitioning the ordered layers into j groups, producing Pm combinations.The resulting search space is determined by partitions rather than unrestricted layer-wise assignments.

Appendix C. Extra Results.

The appendix reports that average Hessian traces vary substantially across network blocks and activations, supporting mixed-precision bit allocation.

Average Hessian traces differ significantly across blocks in SqueezeNext and RetinaNet.The figure also shows loss landscapes for selected blocks in both architectures.
RetinaNet blocks 1–17 comprise the ResNet50 backbone, blocks 18–20 comprise the FPN, and blocks 21–22 are the classification and regression heads.
Average Hessian traces with respect to activations vary significantly across different RetinaNet blocks.The trace estimates use 128 data points with 50 Hutchinson steps.
The activation-trace information is used for mixed-precision activation quantization.