PaperlayerView exact revision on arXiv

Source-linked AI summary

PACT: Parameterized Clipping Activation for Quantized Neural Networks

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan

arXiv:1805.06085v2cs.CVcs.AI

TL;DR

Deep CNNs require substantial computation and storage, while prior quantization work focused largely on weights and faced challenges quantizing activations. PACT learns an activation clipping parameter during training to optimize quantization scales. It supports aggressive low-precision weights and activations, including 4-bit CNNs with near full-precision accuracy, and motivates hardware performance gains through reduced-precision computation.

  • Problem

    Deep CNNs impose high computation and storage costs, while activation quantization remains challenging because ReLU outputs are unbounded and fixed clipping scales are difficult to optimize.

  • Method

    PACT replaces ReLU with an activation function whose clipping parameter α is optimized by gradient descent during training.

  • Results

    PACT quantizes both weights and activations down to 4-bits while achieving near full-precision accuracy, and reaches the highest reported accuracy for ≤2-bit weights and activations.

  • Takeaways & Limitations

    PACT enables aggressive joint weight-and-activation quantization and supports reduced-precision hardware designs with fewer compute-engine area requirements.

  • Takeaways & Limitations

    The quantization setup leaves the first and last layers unquantized because quantizing them has been reported to significantly impact accuracy.

Abstract

from arXiv · show

Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost. To address this cost, a number of quantization schemes have been proposed - but most of these techniques focused on quantizing weights, which are relatively smaller in size compared to activations. This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation. This technique, PArameterized Clipping acTivation (PACT), uses an activation clipping parameter $α$ that is optimized during training to find the right quantization scale. PACT allows quantizing activations to arbitrary bit precisions, while achieving much better accuracy relative to published state-of-the-art quantization schemes. We show, for the first time, that both weights and activations can be quantized to 4-bits of precision while still achieving accuracy comparable to full precision networks across a range of popular models and datasets. We also show that exploiting these reduced-precision computational units in hardware can enable a super-linear improvement in inferencing performance due to a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories.

1 INTRODUCTION

PACT targets the computational and storage costs of deep CNNs by learning activation quantization scales during training. The method supports aggressive weight and activation quantization while preserving accuracy and enabling hardware trade-offs.

  • Motivation: Deep CNNs impose substantial computation and storage demands, motivating compressed models that reduce resource use while preserving accuracy.The introduction describes these demands as barriers to deployment in resource-constrained edge environments.
  • PACT method: PACT learns an activation clipping parameter α during training to automatically optimize quantization scales.The parameter is learned by back-propagation, with regularization applied to α for faster convergence.
  • PACT method: PACT reduces the bit-widths needed for both weights and activations, creating opportunities to trade hardware complexity against model accuracy.
  • Results: For weights and activations at ≤2-bits, PACT achieves the highest model accuracy among published schemes, while 4-bit CNNs reach accuracies similar to single-precision representations.
  • Results: System analysis examines trade-offs between hardware complexity and model accuracy for different bit representations.The paper reports this analysis as part of its evaluation of reduced-precision CNNs.

2 RELATED WORK

Prior work substantially reduced weight precision and explored binary or multi-bit activation quantization. However, strongly reducing activation precision has been associated with significant ImageNet accuracy degradation, while several methods rely on fixed or locally optimized activation statistics.

  • Weight quantization: Weight quantization methods demonstrated that CNN weights can be reduced to 1-bit or 2-bits, helping models fit resource-constrained platforms.
  • Activation quantization: Fully binarized neural networks quantize activations to 1-bit, while later schemes support more general activation bit-precisions.
  • Activation quantization: Activation quantization schemes reported significant ImageNet accuracy degradation greater than 1% when precision is reduced to ≤2-bits.
  • Activation quantization: Other approaches use activation distributions, normalized activations, or modified activation functions to determine quantization scales.

3 CHALLENGES IN ACTIVATION QUANTIZATION

Activation quantization is difficult because ReLU outputs are unbounded and traditional activation functions lack trainable parameters for compensating quantization errors. Clipping limits dynamic range but introduces a difficult scale-selection problem, and fixed offline scales do not fully exploit back-propagation.

  • Activation quantization challenges: Quantizing activations cannot directly compensate errors through back-propagation when activation functions have no trainable parameters.
  • Activation quantization challenges: ReLU quantization requires high dynamic range because ReLU outputs are unbounded, and quantized ReLU significantly degrades accuracy in the reported ResNet20 CIFAR10 setting.
  • Activation quantization challenges: Clipping bounds activation outputs and reduces dynamic-range demands, but globally optimal clipping values are difficult to determine across layers and models.
  • Activation quantization challenges: With quantization, clipping produces lower training error than quantized ReLU, yet its validation error remains noticeably above the baseline.
  • Activation quantization challenges: Half-wave Gaussian quantization determines scales offline and keeps them fixed during training, limiting its use of back-propagation to learn clipping levels.

4 PACT: PARAMETERIZED CLIPPING ACTIVATION FUNCTION

PACT replaces ReLU with a parameterized clipping activation whose learned α sets the activation range and quantization scale during training. Experiments motivate learning α per layer, regularizing it for convergence, and retaining unquantized first and last layers.

  • Parameterized clipping: PACT replaces ReLU with an activation function whose clipping level α is dynamically adjusted through gradient-descent training.The objective is to minimize accuracy degradation arising from activation quantization.
  • Parameterized clipping: The clipped activation is restricted to [0, α] and then linearly quantized to k bits for dot-product computations.
  • Learning α: L2 regularization drives α from a large initial value toward smaller values, limiting activation dynamic range and minimizing quantization loss.For CIFAR10-ResNet20, α started at 10 and converged to substantially smaller values during training.
  • Understanding α: With quantization, cross-entropy generally increases as α grows, and different layers favor different α ranges, motivating learned quantization scales.The full-precision sweep instead often favored large α, while ReLU was nonoptimal for some layers even without quantization.
  • Understanding α: α regularization removes cross-entropy plateaus from total training loss without perturbing its global minimum, supporting gradient-descent convergence.The regularization coefficient λ_α controls its impact on α.
  • Design choices: Sharing one α per layer was experimentally best and reduces hardware complexity by requiring one multiplication after the layer’s reduced-precision MAC operations.The authors also found larger α initialization advantageous, with regularization reducing it during training, and used the same λ for α and weights successfully.

5 EXPERIMENTS

PACT is evaluated across CNNs and datasets against prior quantization schemes, showing that low-precision models can closely match full-precision accuracy. The experiments report near-full-precision results with 4-bit weights and activations and strong performance relative to published alternatives.

  • Experimental setup: PACT is evaluated on CIFAR10-ResNet20, SVHN, AlexNet, ResNet18, and ResNet50 using prior reported quantization results for comparison.The experiments use networks trained from scratch, with PACT replacing ReLU while retaining the baseline hyperparameters.
  • Activation quantization results: Training with bit-precision higher than 3 bits converges almost identically to the full-precision baseline.Across all cases with at least 4-bit activations, final validation error differs by less than 1% from the full-precision validation error.
  • Comparison with prior schemes: For activation precision at or below 2 bits, PACT achieves the highest model accuracy among published schemes.The reported comparison covers a spectrum of models and datasets.
  • Joint weight and activation quantization: With 4-bit weights and activations, PACT consistently achieves full-precision accuracy across the tested networks.The paper characterizes this as near-full-precision accuracy, within 1%, at the lowest jointly reported weight and activation precision.
  • Comparison with prior schemes: Accuracy degradation decreases as activation or weight bit-precision increases across AlexNet, ResNet18, and ResNet50.The comparison includes seven previous quantization schemes and evaluates Top-1 accuracy degradation.

6 SYSTEM-LEVEL PERFORMANCE GAIN

The paper models a precision-configurable DNN accelerator under fixed memory, bandwidth, and silicon-area conditions. Reducing precision shrinks MAC area and can improve inference performance, with bandwidth-constrained systems gaining an additional memory-related benefit.

  • System model: The modeled accelerator uses a 2D systolic array of fixed-point MAC elements and on-chip memory for operands.A precision-configurable MAC unit varies peak FLOPs while maintaining iso-area across precisions.
  • Hardware complexity: Reducing activations and weights from 16 bits to 2 bits produces approximately 14× higher MAC density in 14 nm CMOS implementations.The accelerator versions use the same on-chip memory, external memory bandwidth, and silicon area.
  • Inference performance: Under unlimited external bandwidth, inference performance increases nearly linearly through 4-bit precision, then drops slightly at 2 bits.The ResNet50 analysis attributes the unconstrained case's gain to how amenable the workload is to parallelization.
  • Bandwidth-constrained systems: At 64 GBps external bandwidth, quantizing from 16 to 4 bits yields a 4× increase in peak FLOPs and a 4.5× performance improvement.Some data structures fit in constant on-chip memory at low precision, reducing external-memory transfers; the first and last layers remain 8-bit in this estimate.
  • Bandwidth-constrained systems: Quantizing the first and last layers to 4 or 2 bits is estimated to provide an additional 1.24× performance improvement.The estimate motivates further exploration of low-precision quantization for those layers.

7 CONCLUSION

PACT replaces ReLU with a trainable clipping activation that supports effective activation quantization. Across tested CNNs, 4-bit quantization achieves full-precision accuracy, while α scope and regularization affect design choices.

  • 7 CONCLUSION: PACT replaces ReLU with an activation function whose clipping parameter α is optimized by gradient descent during training.The method is designed to learn quantization scales rather than use fixed clipping values.
  • 7 CONCLUSION: 4-bit quantization of both weights and activations achieves full-precision accuracy consistently across the tested CNNs.Table 1 compares PACT activations with DoReFa-quantized weights across CIFAR10, SVHN, AlexNet, ResNet18, and ResNet50.
  • 7 CONCLUSION: Sharing α within a layer is the most accurate of the three tested scopes and requires only one multiplication after a layer’s MAC operations.The alternatives were per activation, per output channel, and per layer.
  • 7 CONCLUSION: L2 regularization for α produces stable performance across a wide range of λα values, with the preferred value slightly decreasing at higher bit precision.The regularization coefficient was set equal to the weight regularization coefficient in practice.

A.3 QUANTIZATION OF FIRST AND LAST LAYERS

The first and last layers can be accuracy-sensitive under aggressive quantization. Using higher precision for those layers preserves accuracy while still enabling reduced-precision computation throughout the network.

  • A.3 QUANTIZATION OF FIRST AND LAST LAYERS: Aggressively quantizing the first or last layer can cause noticeable accuracy degradation on CIFAR10-ResNet20.The comparison varies whether the first convolution or final fully connected layer is quantized.
  • A.3 QUANTIZATION OF FIRST AND LAST LAYERS: 8-bit quantization of the first and last layers causes little accuracy degradation regardless of the quantization level used elsewhere.This motivates using reduced-precision computation rather than retaining full-precision first and last layers.

APPENDIX B CNN IMPLEMENTATION DETAILS

The experiments use standard CNN architectures and dataset-specific training configurations. Networks are trained from scratch with the stated baseline and PACT settings.

  • APPENDIX B CNN IMPLEMENTATION DETAILS: PACT experiments replace ReLU with PACT while keeping the baseline hyper-parameters and training networks from scratch.ReLU follows BatchNorm in convolutional layers, and Softmax is used for fully connected layers.
  • APPENDIX B CNN IMPLEMENTATION DETAILS: CIFAR10-ResNet20 uses SGD with momentum 0.9, a batch size of 128, and a maximum of 200 epochs.The learning rate starts at 0.1 and is scaled at epochs 60 and 120.
  • APPENDIX B CNN IMPLEMENTATION DETAILS: The SVHN experiment uses a seven-convolution-layer CNN with one fully connected layer, ADAM, batch size 128, and 200 training epochs.The input consists of cropped colored house-number images resized to 40 × 40.
  • APPENDIX B CNN IMPLEMENTATION DETAILS: ImageNet experiments use modified AlexNet, ResNet18, and ResNet50 architectures.ResNet18 and ResNet50 use SGD with momentum 0.9, batch size 256, and a maximum of 110 epochs.

C.1 QUANTIZATION EXPERIMENT SETTING

PACT is evaluated against several published quantization schemes across CNNs and bit-precision settings. The comparisons use reported accuracy results or matched experimental settings, depending on the baseline.

  • C.1 QUANTIZATION EXPERIMENT SETTING: DoReFa is compared with PACT for CIFAR10, SVHN, AlexNet, and ResNet18 under the same experimental setting.DoReFa quantizes weights, activations, and gradients, with a clipped absolute activation for SVHN.
  • C.1 QUANTIZATION EXPERIMENT SETTING: BalancedQ comparisons use reported top-1 or top-5 validation accuracy for AlexNet and ResNet18.BalancedQ is described as recursively partitioning data into balanced bins.
  • C.1 QUANTIZATION EXPERIMENT SETTING: WRPN comparisons use reported AlexNet top-1 accuracy across various weight and activation bit precisions.WRPN increases the number of filter maps to improve robustness for activation quantization.
  • C.1 QUANTIZATION EXPERIMENT SETTING: FGQ comparisons use reported top-1 validation accuracy for AlexNet and ResNet50 with 2-bit weights and 4-bit activations.FGQ is characterized as a direct quantization method based on fine-grained grouping.
  • C.1 QUANTIZATION EXPERIMENT SETTING: WEP, LPBN, and HWGQ comparisons use reported top-1 or top-5 accuracy under their respective bit-precision and layer-quantization settings.Several baselines leave the first or last layers unquantized.

C.2 COMPARISON OF ACCURACY

PACT achieves the best reported accuracy across the evaluated quantization variants for AlexNet, ResNet18, and ResNet50 on ImageNet. Its activation quantization also enables more aggressive weight quantization without accuracy loss.

  • PACT achieves the best accuracy across the evaluated quantization variants for AlexNet, ResNet18, and ResNet50 on ImageNet.The comparison reports both top-1 and top-5 accuracy.
  • The evaluation compares top-1 and top-5 accuracy across AlexNet, ResNet18, and ResNet50 for ImageNet image classification.
  • PACT activation quantization enables more aggressive weight quantization without loss in accuracy.
  • The full-precision reference accuracies for PACT are 55.1% top-1 and 77.0% top-5 for AlexNet.
  • The full-precision reference accuracies for PACT are 70.4% top-1 and 89.6% top-5 for ResNet18.
  • The full-precision reference accuracies for PACT are 76.9% top-1 and 93.1% top-5 for ResNet50.
Loading 1805.06085v2…