PaperlayerView exact revision on arXiv

Source-linked AI summary

Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks

Yuhang Li, Xin Dong, Wei Wang

arXiv:1909.13144v2cs.LGstat.ML

TL;DR

Neural-network quantization must reduce memory and computation while limiting information loss and matching non-uniform weight and activation distributions. The paper proposes APoT with reparameterized clipping and weight normalization, achieving competitive accuracy and lower computational cost, including 76.6% Top-1 accuracy for 4-bit ResNet-50 on ImageNet.

  • Problem

    Neural-network deployment is limited by memory and computational costs, while quantization must balance clipping range, projection resolution, hardware efficiency, and bell-shaped long-tailed distributions.

  • Method

    APoT uses quantization levels formed as sums of Powers-of-Two terms, combined with reparameterized clipping and weight normalization.

  • Results

    76.6% Top-1 and 93.1% Top-5 accuracy are achieved by 4-bit quantized ResNet-50 on ImageNet, with 22% lower computational cost than uniform quantization.

  • Takeaways & Limitations

    APoT outperforms state-of-the-art methods and is competitive with full-precision models while providing higher computational efficiency.

Abstract

from arXiv · show

We propose Additive Powers-of-Two~(APoT) quantization, an efficient non-uniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks. By constraining all quantization levels as the sum of Powers-of-Two terms, APoT quantization enjoys high computational efficiency and a good match with the distribution of weights. A simple reparameterization of the clipping function is applied to generate a better-defined gradient for learning the clipping threshold. Moreover, weight normalization is presented to refine the distribution of weights to make the training more stable and consistent. Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models, demonstrating the effectiveness of our proposed APoT quantization. For example, our 4-bit quantized ResNet-50 on ImageNet achieves 76.6% top-1 accuracy without bells and whistles; meanwhile, our model reduces 22% computational cost compared with the uniformly quantized counterpart. The code is available at https://github.com/yhhhli/APoT_Quantization.

1 INTRODUCTION

The paper frames neural-network quantization as a way to reduce deployment costs while addressing information loss from clipping and projection. It introduces APoT as a hardware-friendly non-uniform scheme tailored to weight and activation distributions.

  • Quantization, pruning, and low-rank decomposition are explored to reduce the memory footprint and computational burden of deep neural networks.
  • Clipping and projection both incur information loss, creating a need to balance clipping range against projection resolution.
  • Existing methods often use uniform quantization for hardware efficiency, although non-uniform quantization can usually achieve better accuracy.
  • Weights and activations follow bell-shaped, long-tailed distributions, with many values near the mean and a few high-magnitude outliers.
  • APoT uses quantization levels formed by sums of Powers-of-Two terms, adapting to these distributions while targeting hardware efficiency.
  • A 4-bit quantized ResNet-50 on ImageNet achieves 76.6% Top-1 and 93.1% Top-5 accuracy, while reducing computational cost by 22% versus uniform quantization.

2 METHODOLOGY

APoT combines non-uniform resolution with efficient shift-add computation by representing quantization levels as sums of Powers-of-Two terms. The methodology also addresses clipping-threshold optimization through reparameterization and uses weight normalization to improve quantization stability.

  • Computation: Powers-of-Two terms enable multiplication through bit-wise shifts, while APoT with k = 2 is approximately 2× faster than uniform quantization in multiplication.The full-precision scaling coefficient α is applied once after the multiply-accumulate operation.
  • Additive Powers-of-Two Quantization: APoT avoids PoT’s rigid resolution by increasing the number of additive terms with bit-width and allocating additional levels across the value range.For b = 4 and k = 2, APoT creates three new levels between 2^0 and 2^-1 while retaining a smallest positive level of 2^-4/3.
  • Additive Powers-of-Two Quantization: APoT represents each quantization level as a sum of n Powers-of-Two terms, providing non-uniform levels for neural-network weights and activations.The number of additive terms is determined by the bit-width and base bit-width, with 2^b total levels.
  • Reparameterized Clipping Function: RCF reparameterizes clipping so forward computation remains equivalent while gradients for the clipping threshold receive contributions from both in-range weights and outliers.This balances projection resolution against clipping effects during threshold optimization.
  • Reparameterized Clipping Function: Clipping-threshold selection balances the range covered by quantization against projection resolution, and a static threshold is not optimal across layers and training steps.A large threshold reduces clipping but lowers central resolution, whereas a small threshold clips more outliers.

3 RELATED WORKS

Prior work develops non-uniform quantizers, jointly optimizes quantization parameters, and applies weight normalization, but these approaches differ in hardware cost and quantization-specific scope.

  • Non-Uniform Quantization: LQ-Nets learns quantization levels by minimizing quantization error, while distillation optimizes levels directly against task loss.These methods use finite floating-point quantization values, introducing additional computation overhead.
  • Jointly Training: Joint-training methods optimize quantization parameters and network weights simultaneously using scaling coefficients, quantization error, task loss, or learnable interval transformations.Examples include Zhu et al., LQ-Nets, QIL, and PACT.
  • Weight Normalization: Weight normalization research mainly addresses BatchNorm limitations by decoupling weight direction from magnitude or standardizing weights to zero mean and unit variance.Only limited prior work studies weight normalization specifically for neural-network quantization.

4 EXPERIMENT

Experiments evaluate APoT on ImageNet and CIFAR10 across bit-widths, compare accuracy and hardware metrics with baselines, and ablate its quantizer, RCF, and WN components.

  • 4.1 EVALUATION ON IMAGENET: ImageNet experiments compare APoT with strong ResNet quantization baselines while quantizing both weights and activations.The implementation uses 8-bit quantization for the first and last layers, unlike baselines using 32-bit precision there.
  • 4.1 EVALUATION ON IMAGENET: APoT tests 2-, 3-, 4-, and 5-bit configurations and reports accuracy, memory cost, and fixed-point operations.One sign bit is used for weights but not activations; FixOP comparisons use the bit-op scheme from Zhou et al. (2016).
  • 4.1 EVALUATION ON IMAGENET: 0.7% Top-1 improvement is reported for 5-bit ResNet-18 over the full-precision baseline, with 0.2% improvements for ResNet-34 and ResNet-50.The same results report improved model size and inference-speed hardware performance.
  • 4.1 EVALUATION ON IMAGENET: 4-bit and 3-bit networks preserve or approach full-precision accuracy, while 3-bit ResNet-18 and ResNet-34 drop 0.5% and 0.3%, respectively.At 2 bits, APoT still outperforms the listed baselines; full-precision shortcuts can raise accuracy but may hurt pipelined hardware latency.
  • 4.2 EVALUATION ON CIFAR10: 3-bit and 4-bit APoT models achieve accuracy comparable to full-precision baselines on CIFAR10, while the 2-bit ternary model outstrips existing methods.Existing methods use four weight levels for 2-bit quantization, whereas APoT uses ternary weights.
  • 4.3 ABLATION STUDY: APoT matches Lloyd quantization at 5 bits and is only 0.2% less accurate at 3 bits while retaining fixed-point acceleration.Lloyd achieves the highest accuracy, but its irregular levels cannot use fixed-point arithmetic; vanilla PoT has the lowest 5-bit accuracy.
  • 4.3 ABLATION STUDY: 1.6% is the maximum accuracy improvement attributed to RCF for learning the clipping range.Without WN, applying RCF causes the network to diverge.

5 CONCLUSION

APoT quantizes neural-network weights and activations with sums of powers-of-two terms, matching their typical distributions while reducing multiplication cost. Reparameterized clipping and weight normalization improve threshold optimization and training stability, supporting state-of-the-art accuracy on ImageNet and CIFAR10.

  • APoT quantizes weights and activations using levels formed by sums of powers-of-two terms for bell-shaped, long-tailed distributions.
  • Roughly 2x multiplication speed-up is achieved compared with uniform quantization.
  • Reparameterizing the clipping function and normalizing weights provides a more stable and better-defined gradient for optimizing the clipping threshold.
  • APoT reaches state-of-the-art accuracy on ImageNet and CIFAR10 compared with uniform or PoT quantization.

A GRADIENT DERIVATION

The gradient analysis identifies a limitation of PACT's STE-based threshold gradient and motivates RCF, which fixes the pre-projection clipping range and rescales after projection. Weight normalization is also examined as a way to stabilize training distributions and threshold optimization.

  • A.1 PACT: PACT uses an STE that treats values before and after projection identically during backpropagation.
  • A.1 PACT: PACT's STE ignores how α affects the quantization-level set Q(α, b), producing an inaccurate clipping-threshold gradient.
  • A.2 RCF: RCF settles the clipping range at [−c, c] before projection and rescales the output to [−α, α] afterward.
  • A.2 RCF: Because Q is not parameterized by α, RCF sends gradients through both the rescale coefficient and the clipping-function scale.
  • B HOW DOES NORMALIZATION HELP QUANTIZATION: Unnormalized weights can have centered density above 8, making small threshold changes strongly affect clipping and complicating training.
  • B HOW DOES NORMALIZATION HELP QUANTIZATION: Normalized weights exhibit a more stable distribution, while their nonzero mean may conflict with symmetric quantization levels.

B.2 TRAINING BEHAVIOR

The training-behavior section compares quantized ResNet-20 training with and without weight normalization under different clipping-threshold learning rates. The supplied materials identify the experiment and its accuracy table, but do not provide the table's numerical outcomes.

  • B.2 TRAINING BEHAVIOR: The experiment trains a 3-bit quantized ResNet-20 from scratch with full-precision activations.
  • B.2 TRAINING BEHAVIOR: Training with and without weight normalization is compared under different learning rates for α.
  • B.2 TRAINING BEHAVIOR: Table 4 reports accuracy comparisons for 3-bit quantized ResNet-20 on CIFAR10.
  • B.2 TRAINING BEHAVIOR: Figure 6 summarizes projection error and clipping error across different layers.

C.1 REVISITING QUANTIZATION ERROR

The paper separates quantization error into clipping and projection components and compares RCF with quantization-error minimization. RCF can favor projection error differently across layers and bit-widths rather than simply minimizing total quantization error.

  • C.1 REVISITING QUANTIZATION ERROR: Quantization error is defined as mean squared error between weights before and after quantization and consists of clipping and projection errors.
  • C.1 REVISITING QUANTIZATION ERROR: RCF optimizes the clipping threshold through final training loss, whereas QEM minimizes the sum of clipping and projection errors.
  • C.1 REVISITING QUANTIZATION ERROR: For the 5-bit model, RCF has higher total quantization error but lower projection error than QEM.
  • C.1 REVISITING QUANTIZATION ERROR: RCF identifies whether projection or clipping matters more across layers and bit-widths, so minimizing quantization error alone may not be optimal.

C.2 IMPLEMENTATIONS DETAILS

The experiments use standard ResNet setups on ImageNet and CIFAR10, initializing quantized models from pretrained or full-precision networks. Training jointly optimizes network parameters and clipping thresholds with SGD-based procedures.

  • ImageNet: ImageNet experiments use 1.2M training images and 50K validation images, resized and cropped to 224×224.Training images are randomly cropped and resized; validation images are center-cropped.
  • Model initialization: Quantized ImageNet ResNets are initialized from released pretrained models, while CIFAR10 quantized models start from full-precision ResNet-20 and ResNet-56 models.This initialization is used before quantized-model training.
  • Optimization: CIFAR10 networks train for 200 epochs with batch size 128 using SGD with momentum 0.9, scheduled learning-rate decay, and weight decay 10^-4.The learning rate starts at 0.04 and is scaled by 0.1 at epochs 80 and 120.
  • Clipping thresholds: For 5-bit quantization, clipping thresholds start at 8.0 for activations and 3.0 for weights, with separate learning rates for threshold updates.The threshold learning rates are 0.01 for weights and 0.03 for activations; the authors report little performance sensitivity to these rates.
Loading 1909.13144v2…