PaperlayerView exact revision on arXiv

Source-linked AI summary

Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework

Sung-En Chang, Yanyu Li, Mengshu Sun, Runbin Shi, Hayden K. -H. So, Xuehai Qian, Yanzhi Wang, Xue Lin

arXiv:2012.04240v2cs.LGcs.AR

TL;DR

Large DNN models and computation challenge edge inference, making model compression important. The paper assigns SP2 or fixed-point quantization to different weight rows to match distributions and FPGA resources, reporting 2.1×−4.1× performance improvement over using DSPs for all multiplications.

  • Problem

    Large DNN model sizes and inference computation make deployment on resource-constrained edge devices challenging, motivating model compression.

    Source: §I · p. 1

  • Method

    MSQ combines SP2 and fixed-point quantization across weight-matrix rows, matching schemes to local distributions and FPGA resources.

    Source: §I · p. 2

  • Results

    2.1×−4.1× performance improvement over using DSPs for all multiplication operations is reported on Zynq XC7Z020 and XC7Z045.

    Source: §VIII · p. 11

  • Takeaways & Limitations

    CNN implementations with MSQ can achieve higher accuracy and comparable hardware utilization efficiency to state-of-the-art methods.

    Source: §I · p. 2

Abstract

from arXiv · show

Deep Neural Networks (DNNs) have achieved extraordinary performance in various application domains. To support diverse DNN models, efficient implementations of DNN inference on edge-computing platforms, e.g., ASICs, FPGAs, and embedded systems, are extensively investigated. Due to the huge model size and computation amount, model compression is a critical step to deploy DNN models on edge devices. This paper focuses on weight quantization, a hardware-friendly model compression approach that is complementary to weight pruning. Unlike existing methods that use the same quantization scheme for all weights, we propose the first solution that applies different quantization schemes for different rows of the weight matrix. It is motivated by (1) the distribution of the weights in the different rows are not the same; and (2) the potential of achieving better utilization of heterogeneous FPGA hardware resources. To achieve that, we first propose a hardware-friendly quantization scheme named sum-of-power-of-2 (SP2) suitable for Gaussian-like weight distribution, in which the multiplication arithmetic can be replaced with logic shifter and adder, thereby enabling highly efficient implementations with the FPGA LUT resources. In contrast, the existing fixed-point quantization is suitable for Uniform-like weight distribution and can be implemented efficiently by DSP. Then to fully explore the resources, we propose an FPGA-centric mixed scheme quantization (MSQ) with an ensemble of the proposed SP2 and the fixed-point schemes. Combining the two schemes can maintain, or even increase accuracy due to better matching with weight distributions.

I. INTRODUCTION

Large DNN models and computation make edge inference challenging, motivating model compression and hardware-aware quantization. The paper combines SP2 and fixed-point quantization across weight-matrix rows to match weight distributions and FPGA resources.

I. INTRODUCTION

  • Weight quantization compresses models by representing weights with fewer bits, while offering less training overhead than pruning or NAS-based compression.The paper states that quantization retraining usually takes 1/3–1/2 as many epochs as pre-training.Source: §I · p. 1
  • SP2 replaces multiplication with logic shifting and addition, targeting Gaussian-like weights and FPGA LUT resources, while fixed-point suits Uniform-like weights and DSPs.The paper reports negligible accuracy loss for SP2 and fixed-point quantization.Source: §I · p. 2
  • MSQ assigns different quantization schemes to different weight-matrix rows because row distributions vary and mixed schemes can better use heterogeneous FPGA resources.The authors report that combining the schemes can maintain or improve accuracy by fitting local weight distributions.Source: §I · p. 2
  • 2.1×−4.1× performance improvement over using DSPs for all multiplication operations is reported on two FPGA devices with optimal SP2/fixed-point ratios.The framework is evaluated across multiple application domains and DNN types.Source: §I · p. 2
  • The proposed FPGA architecture uses heterogeneous GEMM engines and design optimizations to accommodate MSQ and allocate resources.The introduction also distinguishes this intra-layer, multi-scheme approach from inter-layer, multi-precision methods and says they can be combined.Source: §I · p. 2

II. BACKGROUND ON DNN WEIGHT QUANTIZATION

Uniform-interval quantization includes binary, ternary, and fixed-point schemes. The section describes their accuracy and hardware trade-offs and defines fixed-point weight levels and mapping.

1) Uniform Interval Quantization Schemes:

  • Binary and ternary quantization simplify hardware operations but typically degrade accuracy by > 5% and 2–3%, respectively.Binary weights can replace multiplications with additions/subtractions; ternary representations retain zeros and natural sparsity.Source: 1) Uniform Interval Quantization Schemes · p. 2
  • Four-bit fixed-point quantization is described as introducing zero or negligible accuracy loss compared with 32-bit floating-point models.The scheme scales its quantization levels by α.Source: 1) Uniform Interval Quantization Schemes · pp. 2–3
  • The fixed-point quantizer maps a 32-bit floating-point weight to an m-bit sign-magnitude representation after clipping and transforming it for quantization.The text defines the quantizer function, the transform h(·), and clipping of w according to α.Source: 1) Uniform Interval Quantization Schemes · p. 3

2) Non-Uniform Interval Quantization Schemes:

Power-of-2 quantization uses non-uniform levels and replaces weight multiplications with bit shifts. Its precision does not improve effectively with more bits because the distribution tails remain low-precision.

2) Non-Uniform Interval Quantization Schemes:

  • Power-of-2 quantization replaces multiplications by bit shifts and uses non-uniform intervals with greater precision around the mean.Its quantized values are defined using a scale factor and power-of-2 levels.Source: 2) Non-Uniform Interval Quantization… · p. 3
  • Increasing m does not effectively improve power-of-2 quantization precision because additional resolution concentrates around the mean while the tails remain low-precision.In practice, 3–7 bits are usually used, and more bits may not further improve accuracy.Source: 2) Non-Uniform Interval Quantization… · p. 3
  • Algorithm 1 takes a 32-bit floating-point DNN model and a choice of fixed-point, power-of-2, or SP2 quantization, and targets a quantized model.The supplied algorithm passages identify its title, input model, quantization scheme, and target output.Source: 2) Non-Uniform Interval Quantization… · p. 3

B. Quantization Algorithms

Quantization makes loss gradients unavailable for backpropagation. The paper combines ADMM for weight quantization with STE for activation quantization.

B. Quantization Algorithms

  • Quantization projects continuous values onto a discrete number system, making loss gradients unavailable for backpropagation.STE can address this by setting the gradient to a constant value of 1 [50],.Source: B. Quantization Algorithms · p. 3
  • ADMM iteratively optimizes parameters under a target quantization constraint without backpropagating through the quantizer.Source: B. Quantization Algorithms · p. 3
  • The training approach combines ADMM for weight quantization with STE applied only to activation quantization.Source: B. Quantization Algorithms · p. 3

III. SUM-OF-POWER-OF-2 (SP2) QUANTIZATION SCHEME

SP2 is designed to retain the multiplication-free inference of power-of-2 quantization while reducing its accuracy loss. It represents each weight as a signed sum of two power-of-2 values, enabling shifts and an addition.

III. SUM-OF-POWER-OF-2 (SP2) QUANTIZATION SCHEME

  • SP2 is designed to eliminate inference multiplications while addressing power-of-2 quantization’s accuracy loss from low precision in distribution tails.The section describes SP2 as achieving negligible inference accuracy degradation.Source: §III · p. 3, A. SP2 Quantization Scheme · p. 4

A. SP2 Quantization Scheme

  • An m-bit SP2 weight uses a sign bit and two power-of-2 values, q1 and q2, with m1 + m2 + 1 = m and m1 ≥ m2.The quantization levels ±{q1 + q2} lie within [−1, +1].Source: A. SP2 Quantization Scheme · p. 4
  • With 4-bit representations, fixed-point, power-of-2, and SP2 each provide 15 quantization levels within [−1, +1], but their levels are distributed differently.The figure overlays the levels on a representative layer’s weight probability distribution.Source: A. SP2 Quantization Scheme · p. 4, Figure 1 · p. 4
  • SP2 levels are relatively evenly scattered like fixed-point levels, whereas power-of-2 concentrates precision near the mean and leaves the tails low-precision.Source: A. SP2 Quantization Scheme · p. 4
  • SP2 implements weight-activation multiplication with two activation shifts and one addition, while fixed-point multiplication uses repeated n-bit additions.The comparison assumes fixed-point quantization for activations; Table I presents the operation analysis.Source: A. SP2 Quantization Scheme · pp. 4–5, Table I · p. 4

B. Accuracy Performance Analysis

Across the reported ImageNet results, 4-bit fixed-point and SP2 have comparable accuracy with small losses, while power-of-2 generally incurs larger losses. The experiments compare these schemes on ResNet-18 and MobileNet-v2 across CIFAR10, CIFAR100, and ImageNet.

B. Accuracy Performance Analysis

  • The experiments compare 4-bit weight schemes on ResNet-18 and MobileNet-v2 across CIFAR10, CIFAR100, and ImageNet, using fixed-point activation quantization except for MobileNet-v2 on ImageNet.Source: B. Accuracy Performance Analysis · p. 5
  • Around 1%–2% accuracy degradation is reported for power-of-2 in general, including a 2.80% Top-5 accuracy loss for MobileNet-v2 on CIFAR100.Source: B. Accuracy Performance Analysis · p. 5
  • ≤0.41% accuracy loss for ResNet-18 and ≤0.62% for MobileNet-v2 are reported for 4-bit fixed-point and SP2 across the three datasets, with comparable accuracy.Source: B. Accuracy Performance Analysis · p. 5

IV. A FPGA-CENTRIC MIXED SCHEME QUANTIZATION

MSQ assigns fixed-point or SP2 quantization to different rows of a layer, using each scheme’s fit to row-wise weight distributions and FPGA resources. The approach targets better resource use while maintaining or potentially improving accuracy.

IV. A FPGA-CENTRIC MIXED SCHEME QUANTIZATION

  • MSQ combines same-precision fixed-point and SP2 quantization within a layer to match row-wise weight distributions and improve FPGA resource allocation.The paper uses 4-bit precision for both schemes, citing negligible accuracy loss while allocating resources.Source: §IV · p. 5, A. Motivation · pp. 5–6

A. Motivation

  • Rows with smaller-variance, more Gaussian-like distributions favor SP2, while rows with larger-variance, more Uniform-like distributions favor fixed-point quantization.Source: A. Motivation · pp. 5–6
  • SP2 and fixed-point operations can use LUT and DSP resources, respectively, allowing their combination to use heterogeneous FPGA resources.Source: A. Motivation · p. 6
  • The paper argues that matching schemes to diverse local distributions can achieve accuracy similar to or potentially higher than existing schemes.Source: A. Motivation · p. 6

B. Algorithm

MSQ assigns one quantization scheme to each weight-matrix row using a variance threshold, then sets the SP2-to-fixed-point ratio to fit FPGA resources. The evaluation covers image, detection, and recurrent-network tasks.

B. Algorithm

  • A variance threshold assigns SP2 to lower-variance rows and fixed-point quantization to the others, with the threshold set to achieve the desired scheme ratio.That ratio is selected to use FPGA resources, generally keeping DSP utilization at 100% while increasing LUT use and throughput with SP2.Source: B. Algorithm · p. 6

1) Experiment Setup:

  • The evaluation spans image classification, YOLO-v3 object detection, and RNN-based machine translation, speech recognition, and sentiment classification.Source: 1) Experiment Setup · p. 6
  • Image-classification tests use ResNet-18 and MobileNetv2 on CIFAR10, CIFAR100, and ImageNet, with models trained from scratch or quantized from pretrained floating-point models.Source: 1) Experiment Setup · p. 6
  • YOLO-v3 is evaluated on MS COCO 2014 using mAP@0.5 and mAP averaged over IoU thresholds 0.5 to 0.95.Source: 1) Experiment Setup · p. 6
  • Three RNN setups use PTB with perplexity, TIMIT with phoneme error rate, and IMDB for sentiment classification.Source: 1) Experiment Setup · p. 6

2) Result Analysis:

Across image, detection, and recurrent-network evaluations, MSQ maintains or improves accuracy relative to the reported quantization comparisons. Its SP2 and fixed-point operations are also designed to use LUTs and DSPs together.

2) Result Analysis:

  • 0.49%: MSQ increases accuracy compared with the floating-point model, and it achieves the highest accuracy among quantized MobileNet-v2 models.On ResNet-18, the compared methods start from the same pretrained baseline; the MSQ ratio was selected by hardware characterization, not to increase accuracy.Source: Table III · p. 7, Table IV · p. 7, 2) Result Analysis · p. 7
  • Combining SP2 and Fixed can better fit the original weight distribution; the paper also notes that quantization noise may act as regularization at bit-widths of at least 4.Source: 2) Result Analysis · p. 7
  • With 4-bit quantization and 8× compression, MSQ has negligible YOLO-v3 mAP degradation and can increase mAP@0.5 at 640×640 input size.The paper notes greater degradation for smaller inputs and compares its results with pruning at the same compression rate.Source: Table V · p. 7, 2) Result Analysis · p. 7
  • MSQ outperforms Fixed and SP2 on all three RNN tasks; on PTB it increases perplexity by < 2.00 versus 5.00 for EQM, and on IMDB loses 0.06% accuracy versus EQM’s near 1%.The paper reports no comparison with prior quantization work on TIMIT.Source: Table VI · p. 7, 2) Result Analysis · pp. 7–8

V. FPGA IMPLEMENTATION: DESIGN AND OPTIMIZATION

  • MSQ combines SP2’s LUT-based shifts and additions with fixed-point multiplication on DSPs, allowing both resource types to be used together.The paper describes the hardware benefit as orthogonal to prior dataflow and locality optimizations and says it can be used by existing DNN accelerators.Source: §V · p. 8

A. FPGA Resource Characterization

FPGA resource ratios inform the desired balance of SP2 and fixed-point operations. The implementation uses heterogeneous GEMM cores to route those operations to LUTs and DSPs.

A. FPGA Resource Characterization

  • The LUT-to-DSP ratio varies across FPGA devices and determines the suitable balance of SP2 and fixed-point processing elements.The paper uses the device-specific PE ratio as the target SP2-to-fixed-point ratio when quantizing the model.Source: A. FPGA Resource Characterization · p. 8

B. Architecture with Heterogeneous GEMM Engines

  • The MSQ architecture uses separate GEMM cores: GEMMfixed for conventional multiplication and GEMMsp2 for SP2 operations.The architecture also includes instruction, load/store, and compute modules with buffers and a register file.Source: B. Architecture with Heterogeneous… · p. 8, Figure 3 · p. 9
  • Input activations are broadcast to both cores; DSPs implement fixed-point multipliers, while LUTs implement SP2 shifts and additions.Separate weight buffers supply the two formats, and partial results are accumulated in separate register files.Source: B. Architecture with Heterogeneous… · p. 8
  • The two cores’ PE counts should track the ratio of the two weight types and the available DSP and LUT resources to avoid imbalance and use resources fully.Batch normalization, activation, and pooling are incorporated into the GEMM cores because they consume few resources and add negligible latency compared with convolution.Source: B. Architecture with Heterogeneous… · p. 9

A. Experiment Setup

The experiments characterize resource-balanced heterogeneous GEMM configurations, then evaluate their throughput and hardware comparisons across FPGA devices and DNN workloads.

A. Experiment Setup

  • The implementation uses 4-bit CNN quantization at 100MHz and adjusts SP2 and fixed-point PE-array sizes to device resources.The desired fixed/SP2 ratios are 1:1.5 on XC7Z020 and 1:2 on XC7Z045.Source: A. Experiment Setup · p. 9

1) Resource Utilization:

2) Real-world Performance and Comparison:

  • CNN throughput improves 2.1×−2.5× and RNN performance 2.4×−4.1× with heterogeneous GEMM cores versus using GEMMfixed alone.For ResNet-18, latency falls from 100.7ms to 47.1ms on XC7Z020 and from 25.1ms to 10.1ms on XC7Z045.Source: 2) Real-world Performance and Comparison · p. 10
  • ResNet-18 achieves the highest accuracy among compared implementations, with hardware utilization efficiency comparable to and.Work reports higher utilization efficiency but much lower accuracy; no other ResNet-18 or MobileNet-v2 implementations were found for direct comparison.Source: 2) Real-world Performance and Comparison · p. 10
  • The XC7Z045 FPGA delivers 99FPS on ResNet-18 versus 78FPS for the Tensor-RT GPU, with more than 3× higher energy efficiency.The FPGA consumes around 4W, compared with 10−15W for the NVIDIA Jetson AGX; both were measured under the same accuracy.Source: 2) Real-world Performance and Comparison · p. 10

VII. RELATED WORK

Related work spans fixed-point, low-bit, logarithmic, and FPGA-specific quantization designs; the conclusion proposes row-wise SP2/fixed-point quantization and reports improved performance over DSP-only multiplication.

A. DNN Quantization Methods

  • Prior DNN quantizers include fixed-point methods, learned or differentiable quantizers, logarithmic representations, and power-of-two approaches.Examples include hyperbolic-tangent transformation [38], learned intervals [41], and logarithmic representation [45].Source: A. DNN Quantization Methods · p. 10
  • A separate low-bit framework accelerates training with direction-sensitive gradient clipping and deviation-counteractive learning-rate scaling for unified INT8 training.It reports minor accuracy degradation.Source: A. DNN Quantization Methods · p. 10

B. Weight Quantization in FPGA Implementations

VIII. CONCLUSION

  • The conclusion pairs SP2 for Gaussian-like weights and LUT implementation with fixed-point quantization for Uniform-like weights and DSPs.The framework assigns different schemes to different weight-matrix rows.Source: §VIII · p. 11
  • CNN and RNN implementations on two FPGA devices achieve 2.1×−4.1× performance improvement over using DSPs for all multiplication operations.Source: §VIII · p. 11

Related papers

Loading 2012.04240v2…