Source-linked AI summary
HAQ: Hardware-Aware Automated Quantization with Mixed Precision
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han
TL;DR
The paper addresses automated layer-wise bitwidth selection across diverse models, hardware platforms, and resource constraints, where uniform quantization and expert heuristics are inadequate. HAQ uses reinforcement learning with direct hardware feedback to choose mixed-precision policies. It reports better latency, energy, and model-size trade-offs than conventional approaches, while showing that optimal policies differ substantially across hardware architectures.
Problem
Layer-wise bitwidth selection across models and hardware has a vast design space, while conventional approaches use heuristics or uniform precision and may be sub-optimal.
Method
HAQ uses reinforcement learning to select weight and activation bitwidths layer by layer, using hardware feedback for constraints and validation accuracy after short retraining as reward.
Results
HAQ outperforms conventional rule-based approaches across latency, energy, and model-size objectives, with latency reductions of 1.4× to 1.95× at similar accuracy and nearly half the energy in one experiment.
Takeaways & Limitations
Optimal quantization policies differ drastically across hardware architectures and resource constraints, and interpreting them offers insights for neural-network and hardware architecture design.
Takeaways & Limitations
The resource-constrained formulation assumes limited computation budgets, and the RL formulation assumes each layer’s action contributes equally to the final result.
Abstract
from arXiv · showhide
Model quantization is a widely used technique to compress and accelerate deep neural network (DNN) inference. Emergent DNN hardware accelerators begin to support mixed precision (1-8 bits) to further improve the computation efficiency, which raises a great challenge to find the optimal bitwidth for each layer: it requires domain experts to explore the vast design space trading off among accuracy, latency, energy, and model size, which is both time-consuming and sub-optimal. Conventional quantization algorithm ignores the different hardware architectures and quantizes all the layers in a uniform way. In this paper, we introduce the Hardware-Aware Automated Quantization (HAQ) framework which leverages the reinforcement learning to automatically determine the quantization policy, and we take the hardware accelerator's feedback in the design loop. Rather than relying on proxy signals such as FLOPs and model size, we employ a hardware simulator to generate direct feedback signals (latency and energy) to the RL agent. Compared with conventional methods, our framework is fully automated and can specialize the quantization policy for different neural network architectures and hardware architectures. Our framework effectively reduced the latency by 1.4-1.95x and the energy consumption by 1.9x with negligible loss of accuracy compared with the fixed bitwidth (8 bits) quantization. Our framework reveals that the optimal policies on different hardware architectures (i.e., edge and cloud architectures) under different resource constraints (i.e., latency, energy and model size) are drastically different. We interpreted the implication of different quantization policies, which offer insights for both neural network architecture design and hardware architecture design.
1. Introduction
Mixed-precision quantization creates a large, hardware-dependent bitwidth-allocation problem that conventional uniform or heuristic methods do not solve well. HAQ addresses it with automated reinforcement learning guided by direct hardware feedback, specializing policies across architectures and resource constraints.
- Real-time DNN applications are constrained by latency, energy, and model size, motivating low-precision quantization.
- Conventional methods use uniform precision, while mixed precision is needed because layers differ in redundancy and hardware behavior.
- O(H × M × 8^2N) possible solutions make layer-wise bitwidth selection across hardware and models too large for reliable hand-crafted strategies.The search space for ResNet-50 is described as about 8^100.
- Proxy signals such as FLOPs and memory references may not accurately reflect hardware performance, so hardware feedback should enter the design loop.
- HAQ uses reinforcement learning to choose weight and activation bitwidths layer by layer, with hardware feedback guiding resource constraints and validation accuracy providing reward.The framework uses DDPG during exploration and studies cloud and edge accelerators with spatial or temporal multiprecision designs.
- HAQ contributes automation, hardware awareness, hardware-specific specialization, and interpretations of learned policies for neural-network and hardware design.
2. Related Work
Prior work quantizes, searches, or compresses DNNs using methods ranging from fixed precision and binarization to architecture search and hardware-oriented pruning. HAQ is positioned as an automated, hardware-in-the-loop alternative for mixed-precision policy search.
- Quantization research includes binary, ternary, and 8-bit representations, while conventional methods often assign one bitwidth across layers or rely on domain experts.
- HAQ’s overview combines reinforcement-learning search with hardware-in-the-loop feedback to allocate layer bitwidths under latency, power, and model-size resources.
- Neural architecture search explores network building blocks and methods such as Progressive NAS accelerate that search.
- Efficient-model research includes channel pruning and direct optimization of hardware properties such as inference latency and energy.
3. Approach
HAQ formulates mixed-precision quantization as reinforcement learning, assigning layer-wise weight and activation bitwidths while using direct hardware feedback and resource constraints. The framework combines hardware-aware measurements, quantization, finetuning, and DDPG-based policy learning.
- State and action spaces: Each layer’s observation encodes architecture and quantization context, including dimensions, feature-map size, parameter count, layer type, weight-versus-activation indicator, and the previous action.Convolutional and fully connected layers use separate ten-dimensional observation forms, whose dimensions are normalized to [0, 1].
- State and action spaces: Continuous actions in [0, 1] are rounded into discrete bitwidths, with experiments setting the allowed range from 2 to 8 bits.The continuous representation preserves the relative aggressiveness of different bitwidths before conversion to a discrete action.
- Hardware feedback and constraints: Resource constraints limit the action space by measuring the quantized model on hardware and sequentially decreasing layer bitwidths when latency, energy, or model size exceeds budget.Direct hardware feedback replaces indirect proxies such as FLOPs and memory references because cache locality, kernel calls, and bandwidth affect performance.
- Quantization: Weights and activations are linearly quantized using the agent’s bitwidth action, with clipping selected by minimizing KL-divergence between original and quantized weight distributions.Weights are clipped to [−c, c], while ReLU activations are clipped to [0, c] because they are non-negative.
- Reward and training: The reward is based only on post-finetuning accuracy because latency and energy constraints are imposed through action-space limitation.During exploration, quantized models are finetuned for one epoch; after exploration, the best policy is finetuned on the full dataset.
- RL formulation: HAQ models quantization as reinforcement learning, with each action assigning bits to a layer’s weights or activations and each episode covering all layers.The agent uses DDPG in a layer-wise environment, with transitions defined by observations, actions, rewards, and subsequent observations.
4. Experiments
Experiments show that HAQ adapts mixed-precision policies to hardware architectures and optimization objectives, improving latency, energy, and model-size outcomes while preserving accuracy. The learned policies reflect whether layers are memory- or computation-bounded and allocate bits accordingly.
- Latency constraints: 1.4× to 1.95× lower latency is achieved with negligible accuracy loss compared with fixed 8-bit quantization on BISMO.
- HAQ consistently improves accuracy at the same latency as the fixed-bitwidth PACT baseline on edge and cloud accelerators.PACT uses 8 bits for weights and activations except for the first layer.
- Latency constraints: On MobileNet-V1, edge policies assign fewer activation bits to depthwise layers, whereas cloud policies assign more bits to depthwise and fewer to pointwise layers.The interpretation links edge behavior to memory-bounded depthwise convolutions and cloud behavior to computation-bounded execution.
- Latency constraints: 2× lower latency is achieved on BitFusion with almost no accuracy loss compared with fixed 8-bit quantization.For MobileNet-V1, latency falls from 20.08 to 11.09 ms with almost no accuracy degradation.
- Energy constraints: 2× lower power consumption is achieved with nearly no accuracy loss under energy constraints, from 31.03 to 16.57 mJ for MobileNet-V1.
- Model-size constraints: At similar model size, HAQ achieves higher accuracy than Deep Compression, including 57.14 versus 37.62 accuracy when MobileNet-V1 weights use 2 bits.HAQ preserves accuracy better under aggressive quantization by searching mixed-precision policies.
- Model-size constraints: HAQ assigns more bits to depthwise convolutions under model-size optimization but fewer bits under latency optimization.Depthwise convolutions have fewer parameters, while latency policies reduce their bits because their execution can be memory-bounded.
5. Conclusion
HAQ is an automated, hardware-aware reinforcement-learning framework that searches quantization policies using hardware feedback. Experiments show improved latency, energy, and model-size outcomes, while revealing hardware- and objective-specific policies relevant to software–hardware co-design.
- HAQ automatically searches quantization policies with hardware feedback instead of relying on domain experts, rule-based heuristics, or indirect proxy signals.
- HAQ performs better than conventional rule-based approaches across latency, energy, and model-size objectives.
- Optimal quantization policies differ drastically across hardware architectures and resource constraints.
- The interpreted policies provide insights for future software and hardware co-design of efficient DNN deployment.