Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks

• DNNs’ computation, memory, and storage demands impede adoption on resource-limited devices and in interactive settings.The introduction notes that common vision models require billions of multiply-accumulate operations for a 224-by-224 image.Source: §1 · p. 2
• Post-training quantization limits adaptation to quantization errors, while QNNs still show significant accuracy degradation, especially below 4-bit.Quantization during training uses the Straight Through Estimator to address zero gradients from discrete quantized values.Source: §1 · p. 3
• Balanced Quantization uses percentile thresholds to balance quantized values rather than preserve extrema, addressing unused quantization levels in imbalanced distributions.The method is proposed to improve QNN prediction accuracy.Source: §1 · pp. 3–4
• Recursive median partitioning can compute the percentiles used as thresholds; the introduction also reports that median approximations by means are computationally cheaper.The approximation’s efficacy is described as empirically validated.Source: §1 · p. 4
• Experiments report significantly improved CNN and RNN prediction accuracy on standard datasets, including ImageNet and Penn Treebank.The passage directs readers to Section 4 for the experimental results.Source: §1 · p. 4

2 Quantized Neural Networks

• The section introduces QNN notations and algorithms, and explains how quantization is incorporated into training and how bitwise operations speed computation.These are the section’s stated aims.Source: §2 · p. 4

2.1 Notations

• The section defines k-bit uniform quantization by mapping parameters into a fixed range, quantizing them, and restoring their original value range.The quantized outputs are fixed-point values; the mapping uses the maximum absolute parameter value.Source: §2.1 · pp. 4–5

2.2 Simplistic View of Quantized Neural Network

• A QNN layer computes with quantized weights and activations, while the first layer may use higher-bitwidth inputs to preserve information.Bias parameters may remain unquantized.Source: §2.2 · pp. 5–6

2.3 Exploiting Bitwise Operations in QNN

• Bitwise dot products use AND and bit counting for one-bit values, and extend to multi-bit values through bit-vector operations.The section states that the multi-bit computation complexity is O(MK), proportional to the product of the operand bitwidths.Source: §2.3 · p. 6

2.4 Training Quantized Neural Networks by Straight-Through Estimator

• QNN training retains floating-point parameters for gradient accumulation while quantized parameters are used in forward and backward computation.Quantization functions have zero derivatives, so the section describes using an STE to assign non-zero gradients.Source: §2.4 · p. 7

3 Balanced Quantization for Neural Network Parameters

• Balanced Quantization partitions parameters by percentiles into equal-entry bins, then quantizes them to encourage balanced use of the available values.It aims to improve QNN parameter quantization while retaining uniform quantization for computation.Source: §3 · pp. 7–8, §3.2.1 · p. 10

3.1 Effective Bitwidth and Prediction Accuracy of QNN

• Bell-shaped parameter distributions and outliers can leave uniform-quantized values imbalanced, so a nominal 2-bit model may use only two of its four values.The paper proposes effective bitwidth, based on parameter-distribution entropy, to measure value utilization; it conjectures that accuracy is positively correlated with effective bitwidth when other factors are held constant.Source: §3.1 · p. 8, Definition 2 · pp. 9–10

3.2.3 Rounding and Restoring Value Range

• After equalization, values are converted to fixed-point numbers and an affine transformation restores a value range approximately matching the input.The method describes a distinct mapping from equalized values to fixed-point values before range restoration.Source: §3.2.1 · p. 10, §3.2.3 · p. 12

3.2.1 Outline

• The method’s histogram-equalization stage maps equal-entry partitions to evenly divided intervals in [0,1], followed by quantization and range matching.With percentile thresholds, the method assigns the same number of quantized values to each possible choice.Source: §3.2.1 · p. 10

3.2.2 Histogram Equalization by Piecewise Linear Transform

• The method divides the input range into 2^k intervals and uses percentile thresholds to map each interval affinely into evenly spaced segments of [0,1].Exact equalization uses the 100i/2^k-th percentiles as thresholds.Source: §3.2.2 · p. 11, Definition 3 (Histogram equalization) · p. 11

3.3 Approximation of Median and Efficient Implementation

• The recursive implementation uses a binary mask to select working subsets and uses the mean of each subset as its partition threshold.The paper motivates the mean threshold as a way to accelerate computation compared with sorting for exact percentile thresholds.Source: §3.3 · pp. 12–13

14 end

• The recursive procedure shifts the upper subset’s [0,1] output range to [1/2,1] before combining it with the lower subset.The returned output combines the two recursively equalized subsets.Source: §14 · p. 13

3.3.1 Recursive Partitioning

• Recursive median partitioning computes evenly spaced percentiles by repeatedly finding medians within the remaining subsets.For example, after finding the 50th percentile, the 25th percentile is the median of values below it.Source: §3.3.1 · p. 14
• With bounded variance, means can replace medians for approximate balance, whereas median partitioning achieves perfect balance.Figures 4 and 5 show partitioning with the median and mean, respectively.Source: §3.3.1 · p. 14, Figure 4 · p. 14, Figure 5 · p. 15

3.3.2 Implementation

• The implementation uses an auxiliary binary mask to manage branching and selection during recursive histogram equalization.The mask passed at the top of the call chain is initialized to all ones.Source: §3.3.2 · pp. 14–15
• Under Proposition 1’s stated condition, the most frequent quantized value appears at most γ2K times as often as the least frequent value.This bound applies to K-bit quantization using Algorithm 1.Source: §3.3.2 · p. 15

4.2 Datasets

• The CNN evaluation uses SVHN house-number images and ImageNet classification, reporting single-crop top-1 and top-5 error rates for ImageNet.ImageNet has 1.2 million training images and 50,000 validation images.Source: §4.2 · p. 16

4.2.1 Effective Bitwdiths and Prediction Accuracies of Converged Models

• On SVHN, prediction accuracy generally rises with effective bitwidth, but its gains slow as accuracy approaches the floating-point upper bound.The compared converged QNNs share architecture and training settings while varying specified bitwidth and quantization method.Source: §4.2.1 · p. 16, Figure 6 · p. 17

4.2.3 Balanced Quantization of AlexNet, ResNet-18 and GoogLeNet

• Balanced Quantization consistently outperforms uniform quantization on AlexNet and ResNet-18, while the 2-bit AlexNet top-5 error is within 2 percentage points of floating point.The authors note that model size can fall to 1/16 of the original when computations use 2-bit numbers.Source: Table 2 · p. 18, §4.2.3 · p. 19

4.2.2 Evaluation of Approximation of Median

• Using means instead of medians as Balanced Quantization thresholds does not degrade GoogLeNet accuracy and is slightly better on both top-1 and top-5 error rates.This comparison uses GoogLeNet with 4-bit weights and 4-bit activations; means are then used as thresholds in the rest of Section 4.Source: §4.2.2 · p. 18, Table 1 · p. 18

4.2.3 Balanced Quantization of AlexNet, ResNet-18 and GoogLeNet

• Top-5 accuracy degradation on 4-bit GoogLeNet falls from 6.5 to 2.6 percentage points, and the method consistently outperforms the compared state-of-the-art methods.Accuracy degradation is defined as the difference between a QNN and its floating-point version.Source: Table 3 · p. 19, §4.2.3 · p. 19

4.3 Experiments on Recurrent Neural Networks

• The 2-bit-weight, 3-bit-activation LSTM reaches 123 PPW with balanced quantization versus 155 PPW with imbalanced quantization, and both outperform the counterparts in.The paper reports comparable performance to floating-point counterparts with 4-bit weights and activations.Source: Table 4 · p. 20, §4.3 · p. 21

4.2.4 Break-down of Accuracy Degradation with Balanced Quantization

• Equalization constrains the parameter optimization space and may add prediction error, but experiments show reduced quantization error outweighs that additional term.The authors leave the cause and further reduction of equalization-related accuracy change for future work.Source: §4.2.4 · p. 20

5 Conclusions

• The method can improve prediction accuracy in converged QNNs by balancing quantized-value distributions through histogram equalization.It also introduces Effective Bitwidth to measure bitwidth utilization and identify models that may benefit more from Balanced Quantization.Source: §5 · p. 21
• Recursive application of the mean is proposed as a lower-overhead approximation to percentiles in Balanced Quantization.The approximation addresses the computation overhead of percentile calculation.Source: §5 · p. 21
• The method outperforms QNN state-of-the-art methods in prediction accuracy on AlexNet, GoogLeNet, and ResNet, with encouraging results on LSTM and GRU.The conclusion refers to Subsections 4.2.3 and 4.3, respectively.Source: §5 · p. 22
• Future work includes using histogram transformation to induce more zeros in quantized values and testing whether balanced activation distributions improve QNN prediction accuracy.Source: §5 · p. 22

A Eliminating All Floating Point Operations During Inference

• Under the stated assumptions, the computation of the quantized layer output can be performed using fixed-point operations; weights are represented as fixed-point values scaled by a floating-point scalar.Source: A Eliminating All Floating Point… · p. 22
• Quantized activations are computed by comparing activation outputs with a sequence of thresholds, because the activation quantization function is uniform.Source: A Eliminating All Floating Point… · p. 22
• Since the activation is monotone, threshold comparisons can be transferred to fixed-point inputs, and the required thresholds can be precomputed and stored to avoid floating-point operations during inference.Source: A Eliminating All Floating Point… · p. 23

B Training Algorithm of QNN

• The training algorithm quantizes weights, activations, and gradients using QW, QA, and QG, and extends Hubara et al. to multi-bit quantization.The cost function and chain-rule-derived backward-input and backward-weight functions are used to compute gradients; the update follows the chosen learning rule.Source: B Training Algorithm of QNN · p. 23
• For each minibatch of inputs and labels, the algorithm performs forward propagation and can optionally apply pooling.Source: B Training Algorithm of QNN · p. 24
• Backward propagation computes the output gradient from the cost and back-propagates through a pooling layer when present before accumulating parameter gradients.Source: B Training Algorithm of QNN · p. 24, §17 · p. 24

C Proof of Proposition 1

• After histogram equalization, the algorithm maps half-open or half-closed intervals to quantization values, and its recursive calls form a balanced binary tree.Source: C Proof of Proposition 1 · p. 25
• The proof argues that distinct leaf nodes map values to distinct intervals, preventing two leaves from producing the same quantized value.Source: C Proof of Proposition 1 · p. 25

D Quantization of GRU

• GRU quantization must account for element-wise gate products that may require higher-bitwidth or floating-point computation, so additional quantization steps are inserted after those products.Source: D Quantization of GRU · p. 26
• Because tanh outputs lie in [−1,1] while the zt and rt gates lie in [0,1], the authors replace the activation on f ht with sigmoid to simplify implementation and keep the resulting state in [0,1].The described quantized scheme assumes quantized weights in [−1,1] and quantized inputs in [0,1].Source: D Quantization of GRU · pp. 26–27

E Quantization of LSTM

• Ct remains floating point because it is not bounded by an activation, and its element-wise additions and multiplications may cost much less than matrix products.The authors therefore leave Ct unquantized.Source: E Quantization of LSTM · p. 27
• For simpler implementation, the output tanh activation may be changed to sigmoid.Source: E Quantization of LSTM · p. 27
• The quantized LSTM formulation computes the output gate ot with a sigmoid applied to a weighted concatenation of the prior hidden state and input.Source: E Quantization of LSTM · p. 27
• The formulation assumes weights Wf, Wi, WC, and Wo are quantized to [-1,1], while input xt is quantized to [0,1].Source: E Quantization of LSTM · p. 27