PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-Training Piecewise Linear Quantization for Deep Neural Networks

Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, Joseph Hassoun

arXiv:2002.00104v2cs.CVcs.LG

TL;DR

Low-bit post-training uniform quantization can degrade performance for bell-shaped tensor distributions with long tails, despite avoiding retraining and full-dataset access. The paper proposes PWLQ, which uses non-overlapping regions and error-minimizing breakpoints, and reports state-of-the-art performance across three vision tasks with minor overhead.

  • Problem

    Uniform post-training quantization preserves performance well at INT8 but causes significant degradation at lower bit-widths, while post-training remains desirable because it avoids retraining and full-dataset access.

  • Method

    PWLQ partitions each tensor’s range into non-overlapping regions with equal quantization levels and selects breakpoints by minimizing quantization error.

  • Results

    PWLQ achieves state-of-the-art low-bit post-training quantization performance on image classification, semantic segmentation, and object detection under the same computational cost.

  • Takeaways & Limitations

    PWLQ indicates potential for efficient and rapid deployment of computer-vision applications on resource-limited devices.

  • Takeaways & Limitations

    More breakpoints or applying PWLQ to both weights and activations can require additional accumulators and storage, increasing hardware overhead and potentially reducing feasibility.

Abstract

from arXiv · show

Quantization plays an important role in the energy-efficient deployment of deep neural networks on resource-limited devices. Post-training quantization is highly desirable since it does not require retraining or access to the full training dataset. The well-established uniform scheme for post-training quantization achieves satisfactory results by converting neural networks from full-precision to 8-bit fixed-point integers. However, it suffers from significant performance degradation when quantizing to lower bit-widths. In this paper, we propose a piecewise linear quantization (PWLQ) scheme to enable accurate approximation for tensor values that have bell-shaped distributions with long tails. Our approach breaks the entire quantization range into non-overlapping regions for each tensor, with each region being assigned an equal number of quantization levels. Optimal breakpoints that divide the entire range are found by minimizing the quantization error. Compared to state-of-the-art post-training quantization methods, experimental results show that our proposed method achieves superior performance on image classification, semantic segmentation, and object detection with minor overhead.

1 Introduction

DNN scaling improves accuracy but increases resource demands, motivating quantization for embedded deployment. The paper proposes PWLQ to address low-bit uniform quantization degradation while preserving post-training convenience.

  • Scaling DNN depth, width, or resolution improves accuracy but increases computation and memory requirements, hindering deployment on resource-limited devices.
  • Post-training quantization avoids retraining and full-dataset access, while uniform quantization is efficiently implemented with integer-arithmetic hardware.
  • INT8 uniform post-training quantization often preserves FP32 performance, but lower-bit settings such as 4-bit cause significant accuracy drops.
  • PWLQ partitions each tensor’s quantization range into non-overlapping regions with equal quantization levels and finds breakpoints by minimizing quantization error.
  • The paper evaluates PWLQ on image classification, semantic segmentation, and object detection, targeting state-of-the-art results without retraining or full training-data access.

2 Related Work

Prior efficient-deployment methods include architecture design, compression, distillation, and quantization, with post-training quantization avoiding retraining. Related low-bit methods address nonuniform distributions or hardware constraints, motivating PWLQ’s optimized breakpoints and commodity-hardware implementation.

  • Efficient DNN deployment methods include efficient architectures, pruning, activation compression, knowledge distillation, and quantization.
  • Quantization compresses models, accelerates computation, and reduces power consumption by storing parameters in fewer bits and using integer-arithmetic units.
  • Prior post-training techniques improve accuracy through per-channel quantization, weight equalization, factorization, and bias correction.
  • Some 4-bit methods combine analytical clipping, bit allocation, bias correction, multiple tensors, or optimized scaling factors.
  • Uniform quantization cannot capture bell-shaped weight and activation distributions, while alternative methods face customized-hardware or heuristic-scale limitations.
  • PWLQ improves breakpoint selection and can be implemented with minimal modification to commodity hardware.

3 Quantization Schemes

The section contrasts uniform quantization with PWLQ, which allocates precision across central and tail regions and selects a breakpoint by minimizing expected squared error. Under bell-shaped distributions, the resulting optimization is convex with a unique optimum, and PWLQ can reduce quantization error relative to uniform quantization.

  • Uniform Quantization: Uniform quantization maps values linearly to evenly spaced integer levels, but its distribution-agnostic allocation can produce insufficient accuracy for bell-shaped weights and activations at low bit-widths.The quantization error generally increases as bit-width decreases, while pre-trained DNN tensors are often Gaussian- or Laplacian-like.
  • Piecewise Linear Quantization: PWLQ divides the symmetric range [−m, m] into a central region and two tail pieces, applying (b −1)-bit uniform quantization within each signed piece.A single breakpoint p separates the dense center from the sparse high-magnitude tails while preserving b-bit representations including the sign.
  • Breakpoint Selection: For bell-shaped distributions, PWLQ assigns greater precision to the center and lower precision to the tails by restricting the breakpoint to 0 < p < m/2.The choice p = m/2 is equivalent to uniform quantization, whereas smaller p concentrates precision in the center.
  • Error Analysis: Assuming a symmetric density decreasing for positive values, the expected PWLQ error is convex in p, so a unique optimal breakpoint p* exists.The breakpoint can be found by solving the convex optimization problem; under normalized Gaussian assumptions, a fast approximation closely matches gradient descent.
  • Error Analysis: PWLQ achieves smaller quantization error than the uniform scheme, with the error at most 9/16 of the b-bit uniform scheme while requiring one extra storage bit and no extra multiplication.The smaller error is supported by both the analytical comparison and numerical simulation.

4 Hardware Impact

PWLQ preserves integer-arithmetic deployment while dividing weights into regions with distinct scaling factors, but additional regions increase computation and storage overhead.

  • Baseline hardware path: Uniform quantization accelerates the expensive FP inner product with INT operations plus O(1) floating-point rescaling and addition.The constants used for rescaling and offsets can be pre-computed offline.
  • Region-specific computation: PWLQ partitions weights into non-overlapping regions, requiring separate computational paths because each region uses a different scaling factor.For one breakpoint, the regions are R1 and R2, with corresponding partial inner products and quantized weight vectors.
  • Additional operations: PWLQ adds an activation-sum term for weights in the non-zero-offset region and requires floating-point rescaling and addition in each region.These extra operations arise in the R2 path because its offset is p.
  • Hardware requirements: An efficient two-region implementation uses one multiplier, three accumulators, and at most one extra storage bit per weight value.The extra bit selects the appropriate accumulator without increasing MAC computation and at negligible MAC-unit cost.
  • Hardware trade-offs: More breakpoints require more accumulators and storage bits, while applying PWLQ to both weights and activations requires accumulators for every region combination.These requirements may make multiple weight breakpoints or joint weight-and-activation PWLQ infeasible from a hardware perspective.

5 Experiments

Experiments evaluate PWLQ on classification, segmentation, and detection, finding that optimized non-overlapping regions improve low-bit performance while breakpoint perturbations reduce accuracy.

  • Experimental setup: PWLQ is evaluated on ImageNet classification, Pascal VOC semantic segmentation, and Pascal VOC object detection after batch-normalization folding.Activation ranges are profiled from 512 training images using a top-10 median strategy.
  • Experimental setup: Classification experiments measure top-1 validation accuracy for Inception-v3, ResNet-50, and MobileNet-v2 using pretrained torchvision models.The experiments focus on the effectiveness of PWLQ for these three architectures.
  • Breakpoint optimization: The Gaussian breakpoint approximation achieves almost the same accuracy as gradient descent while being considerably faster.The approximation p*/m = ln(0.8614m + 0.6079) is used unless otherwise stated.
  • Comparative results: PWLQ with non-overlapping regions outperforms BSD+ and V-Quant on low-bit quantization, especially on 4-bit MobileNet-v2.Non-overlapping regions shorten tail-region quantization ranges by 1.25× to 2×, reducing quantization error according to the authors.
  • Breakpoint robustness: For 4-bit Inception-v3, breakpoint perturbations from 5% to 30% reduce model performance as perturbation increases.With 5% perturbation, accuracy can reach 74.05%, a 1.67% drop from the zero-perturbation baseline, across 100 random samples per level.
  • Multiple breakpoints: Increasing the number of breakpoints raises hardware cost linearly while increasing the number of regions and quantization levels.The multi-breakpoint optimization extends the same quantization-error objective using gradient descent.

Multiple Breakpoints.

The paper compares PWLQ and uniform quantization across bit-widths, while multiple breakpoints trade higher representational power for greater hardware cost.

  • Hardware trade-off: Increasing the number of breakpoints raises hardware cost linearly while increasing the number of non-overlapping regions and total quantization levels.Table 1 reports top-1 accuracy and hardware accumulator requirements for multiple weight breakpoints.
  • Accuracy comparison: PWLQ generally achieves higher accuracy than uniform quantization at 4-, 6-, and 8-bit weight quantization, except for one 8-bit Inception-v3 case.The comparison uses per-channel weight quantization and per-layer 8-bit activation quantization.

PWLQ and Uniform Quantization.

PWLQ generally improves low-bit accuracy over uniform quantization, with especially large gains at 4 bits, while retaining a simple piecewise uniform implementation.

  • PWLQ and Uniform Quantization.: 75.72% versus 44.28% top-1 accuracy for 4-bit Inception-v3 shows PWLQ’s large low-bit advantage over uniform quantization.At 8 bits, both schemes provide good accuracy because quantization error is small.
  • PWLQ and Uniform Quantization.: PWLQ applies uniform quantization within each piece, preserving a simple computational scheme and compatibility with bias correction.The method uses a piecewise representation rather than replacing uniform quantization entirely.
  • PWLQ and Uniform Quantization.: Bias correction further improves low-bit models, enabling 6-bit PWLQ within 0.30% of full-precision accuracy across three networks.With bias correction, 4-bit MobileNet-v2 reaches 69.22% accuracy without retraining.

Bias Correction.

The comparison uses matched low-precision settings to evaluate PWLQ against clipping with outlier channel splitting and other quantization methods. PWLQ performs strongly without requiring architectural changes and achieves state-of-the-art results under the compared computational costs.

  • 5.2 Comparison to Existing Approaches: PWLQ is compared with clipping plus OCS using per-layer weight quantization and an unquantized first layer.The comparison adopts the same setup used for the clipping and OCS methods.
  • 5.2 Comparison to Existing Approaches: PWLQ without bias correction outperforms the best clipping method combined with OCS under the matched comparison setup.OCS changes the network architecture, whereas PWLQ does not.
  • 5.2 Comparison to Existing Approaches: Under equal computational bit-width, PWLQ with bias correction achieves state-of-the-art results across the compared cases.The largest margins occur in the 4/8 and 4/4 cases.
  • 5.2 Comparison to Existing Approaches: PWLQ incurs only a few additional accumulations per MAC unit and minor storage overhead, while remaining applicable to other methods.The method uses per-channel PWLQ for weights and per-layer quantization for activations in this comparison.

5.3 Other Applications

The paper extends PWLQ beyond classification to semantic segmentation, evaluating DeepLab-v3+ with MobileNet-v2 under practical quantization settings. PWLQ with bias correction substantially improves low-bit segmentation performance.

  • 5.3 Other Applications: DeepLab-v3+ with a MobileNet-v2 backbone is evaluated using mean intersection over union on Pascal VOC segmentation.Weights are quantized per-channel with bias correction, while activations are uniformly quantized per-layer.
  • 5.3 Other Applications: 8-bit PWLQ is applied to three depth-wise convolution layers with large weight ranges in the backbone.Several early layers are fixed at this configuration because quantization range strongly affects quantized-model performance.
  • 5.3 Other Applications: 17.61% mIoU improvement over 4-bit uniform quantization demonstrates the benefit of 4-bit PWLQ for semantic segmentation.The result uses piecewise linear quantization combined with bias correction.
  • 5.3 Other Applications: 6-bit PWLQ with bias correction on weights outperforms 8-bit DFQ, which has 0.42% degradation from the pretrained model.The comparison indicates that PWLQ can retain strong segmentation accuracy at lower weight precision.

Object Detection.

For SSD-Lite object detection with a MobileNet-v2 backbone, PWLQ substantially narrows the low-bit accuracy gap relative to full precision compared with uniform quantization.

  • Object Detection.: SSD-Lite with a MobileNet-v2 backbone is evaluated using mean average precision on Pascal VOC object detection.The table uses per-channel bias-corrected weight quantization and per-layer uniform activation quantization.
  • Object Detection.: 0.38% versus 3.91% performance drop shows that 4-bit PWLQ nearly closes the full-precision gap left by 4-bit uniform quantization.Both methods include bias correction and per-channel quantization enhancements in this comparison.

6 Conclusion

The proposed piecewise linear quantization scheme targets accurate low-bit post-training quantization of deep neural networks and achieves state-of-the-art performance across major computer vision tasks under the same computational cost.

  • PWLQ divides each tensor’s bell-shaped values into non-overlapping regions, assigning each region an equal number of quantization levels.The scheme is designed for accurate post-training quantization of deep neural networks.
  • The approach achieves state-of-the-art low-bit post-training quantization performance on image classification, semantic segmentation, and object detection.These results are reported under the same computational cost.
  • PWLQ indicates potential for efficient and rapid deployment of computer vision applications on resource-limited devices.
Loading 2002.00104v2…