Quantization Networks

• High computational and memory costs limit DNN deployment when computation and storage resources are constrained.Source: §1 · p. 1
• Existing quantization methods use approximation or optimization formulations, with limitations involving gradient mismatch, weight-only applicability, or training cost.Source: §1 · pp. 1–2
• The proposed quantization function is learned in a lossless, end-to-end manner and avoids the severe gradient mismatch problem.Source: §1 · p. 2
• The paper formulates quantization as a differentiable non-linear mapping function that provides a simple, general solution for arbitrary-bit weight and activation quantization.Source: §1 · p. 2
• Quantization networks outperform state-of-the-art quantization methods on image classification and object detection tasks.Source: §1 · p. 2

Low-Bit Quantization

• Low-bit quantization methods are broadly divided into approximation-based and optimization-based approaches.Source: §2.1 · p. 2
• Approximation-based methods replace full-precision values with discrete low-bit values and use step-function approximations during the forward pass.Source: §2.1 · p. 2
• Approximation-based methods require backward-process approximations because step-function gradients saturate, creating gradient mismatch and unstable optimization.Source: §2.1 · p. 2
• Optimization-based methods formulate quantization as a discretely constrained problem, but are limited to weights and incur high computational costs during training.Source: §2.1 · pp. 2–3

Non-Linear Functions

• Sigmoid functions provide smooth, non-zero gradients for back-propagation and approach unit-step behavior when the input magnitude is large.Source: §2.2 · p. 3
• ReLU is zero on half its domain and linear on the other half, while Maxout generalizes ReLU by combining multiple simple linear functions.Source: §2.2 · p. 3

3.1. Reformulation of Quantization

• The ideal quantization function maps continuous weights or activations to discrete integers by combining binary unit-step functions with specified biases and scales.The output is constrained to a predefined integer set, with the number of intervals determined by that set.Source: §3.1 · p. 3

3.2. Training and Inference with Quantization Networks

• Replacing unit-step functions with sigmoids yields a differentiable soft quantization function that supports end-to-end backpropagation.This formulation avoids needing gradient approximations during training.Source: §3.2 · p. 3
• Temperature T is gradually increased during training: larger T narrows the gap to ideal step quantization but produces zero gradients in more cases.Starting with small T supports learning, while increasing it reduces the training–inference gap by the end of training.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
• The quantization function independently transforms each full-precision weight or activation, and its output replaces the original value for subsequent computation.The same mechanism can be inserted wherever weights or activations are quantized.Source: §3.2 · p. 4
• After training, sigmoid functions are replaced with unit-step functions for inference, producing discrete quantized weights and activations.Training uses the smooth function for efficient optimization, whereas inference uses the ideal quantization function.Source: §3.2 · pp. 4–5

3.3. Experiments

• The reported AlexNet ImageNet evaluation uses Top-1 and Top-5 accuracy (%) and distinguishes weight bits W from activation bits A.The full-precision reference performance is 61.8/83.5.Source: Table 1 · p. 5

Image Classification

• ImageNet experiments evaluate AlexNet, ResNet-18, and ResNet-50 using binary, ternary, and higher-bit weight and activation quantization.Performance is reported with Top-1 and Top-5 accuracy, while the listed configurations vary weight and activation bitwidths.Source: §4.1 · pp. 5–6
• Our quantization network outperforms state-of-the-art methods for both weight and activation quantization and supports arbitrary bitwidths.The method is applied flexibly to both weights and activations rather than only one parameter type.Source: §4.1 · p. 6
• Even in compact ResNet-18, the method achieves lossless results with only 3 bits and surpasses the full-precision model with 3-bit weight quantization.The authors report greater degradation for compact architectures than for AlexNet, while still obtaining these results on ResNet-18.Source: §4.1 · p. 6

Object Detection

• For SSD on Pascal VOC, the quantization network is slightly better than ADMM in mAP, compared with 77.8 for the full-precision model.The comparison uses ADMM because other baseline quantization methods did not report object-detection performance.Source: §4.2 · p. 6

Ablation Experiments

• Non-uniform quantization outperforms linear quantization, while adaptive bias learning does not outperform the fixed-bias version.Non-uniform intervals are motivated by the roughly Gaussian distribution of pretrained parameters and implemented using clustering centers.Source: §4.3 · pp. 6–7
• Training from a pretrained model greatly improves performance over training from scratch, and low-bit quantization significantly reduces inference time and space complexity.The inference complexity result is based on evaluation using a VU9P FPGA.Source: §4.3 · p. 8

• The paper proposes a soft quantization function based on differentiable non-linear functions.Source: §5 · p. 9
• Embedding the differentiable quantization function as an activation function enables end-to-end training of quantization networks.Source: §5 · p. 9
• The method supports arbitrary-bit quantization for both weights and activations.Source: §5 · p. 9
• Experiments on image classification and object detection verified the effectiveness of the proposed method.Source: §5 · p. 9