Source-linked AI summary
Fixed Point Quantization of Deep Convolutional Networks
Darryl D. Lin, Sachin S. Talathi, V. Sreekanth Annapureddy
TL;DR
Deep convolution networks improve recognition but increase computation and storage costs, motivating fixed-point deployment. The paper converts pretrained floating-point DCNs using SQNR-based layerwise bit-width optimization. On CIFAR-10, optimized allocation reduced model size by >20% without accuracy loss, while fine-tuning reached 6.78% error.
Problem
Increasing DCN complexity raises computation and model-storage costs, creating deployment challenges for real-time and embedded applications.
Method
The paper converts pretrained floating-point DCNs to fixed point using SQNR-based optimization of layerwise bit-width allocation.
Results
>20% reduction in model size was achieved without accuracy loss on CIFAR-10, and fine-tuning reached 6.78% error rate.
Takeaways & Limitations
Optimized fixed-point conversion offers a practical route to reducing DCN complexity for embedded and real-time deployment.
Takeaways & Limitations
The optimization is most effective when convolutional layers dominate model size and may be constrained by platform-supported bit-widths.
Abstract
from arXiv · showhide
In recent years increasingly complex architectures for deep convolution networks (DCNs) have been proposed to boost the performance on image recognition tasks. However, the gains in performance have come at a cost of substantial increase in computation and model storage resources. Fixed point implementation of DCNs has the potential to alleviate some of these complexities and facilitate potential deployment on embedded hardware. In this paper, we propose a quantizer design for fixed point implementation of DCNs. We formulate and solve an optimization problem to identify optimal fixed point bit-width allocation across DCN layers. Our experiments show that in comparison to equal bit-width settings, the fixed point DCNs with optimized bit width allocation offer >20% reduction in the model size without any loss in accuracy on CIFAR-10 benchmark. We also demonstrate that fine-tuning can further enhance the accuracy of fixed point DCNs beyond that of the original floating point model. In doing so, we report a new state-of-the-art fixed point performance of 6.78% error-rate on CIFAR-10 benchmark.
1. Introduction
Deep convolution networks improve recognition but impose substantial computation, storage, and deployment costs. The paper focuses on converting pretrained models to fixed point, including optimized layerwise bit-width allocation.
- DCN advances improve image recognition and support tasks including detection, localization, segmentation, and retrieval, but increase computational complexity.The added burden affects training, testing, and deployment.
- Fixed-point implementation may reduce memory bandwidth, power consumption, computation time, and DCN storage requirements.
- The paper mainly addresses converting pretrained floating-point DCNs rather than training networks under fixed-point constraints.This targets applications where users lack the original training data and training framework.
- The planned approach develops quantizer design and optimizes per-layer bit-width allocation against complexity reduction and classification-accuracy loss.
2. Related work
Prior work largely trains networks with low-precision arithmetic, while this paper converts pretrained floating-point DCNs using SQNR-based optimization. The approach replaces exhaustive search with an analytical layerwise bit-width strategy.
- Earlier studies primarily trained DCNs using low-precision fixed-point arithmetic, including binary weights in some cases.Binary-weight training could sometimes improve classification accuracy relative to a floating-point baseline.
- Related conversion work used exhaustive search to identify fixed-point bit-widths for fully connected networks and later analyzed quantization sensitivity in DCNs.
- The paper converts pretrained floating-point DCNs with an SQNR-based optimization strategy rather than brute-force search.
- The proposed framework provides an analytical per-layer bit-width choice intended to improve inference speed and reduce storage requirements.The authors describe this as easier to generalize to networks with many layers.
- Hashing, knowledge transfer, and pruning combined with quantization are presented as complementary complexity-reduction approaches.
3. Floating point to fixed point conversion
The conversion procedure quantizes layer inputs, outputs, and parameters by selecting fixed-point formats from distribution statistics and quantizer step sizes. SQNR analysis relates bit-width to quantization efficiency and motivates distribution-aware choices.
- 3. Floating point to fixed point conversion: Fixed-point conversion represents each layer’s input activations, output activations, and parameters through quantization.
- 3. Floating point to fixed point conversion: Bit-width, step-size, and dynamic range jointly determine the fixed-point representation, balancing overflow risk against quantization error.
- 3.1. Optimal uniform quantizer: The optimal symmetric uniform quantizers in Table 1 minimize SQNR for uniform, Gaussian, Laplacian, and Gamma input distributions.
- 3.1. Optimal uniform quantizer: Quantization efficiency decreases as input-distribution kurtosis increases, while ideal SQNR is approximately linearly related to bit-width.
- 3.1. Optimal uniform quantizer: Ideal quantization efficiency is 6dB/bit for uniform distributions and about 5dB/bit for Gaussian distributions, with lower efficiency possible for non-ideal inputs.
- 3.2. Empirical distributions in a pre-trained DCN: For the CIFAR-10 DCN, weights and pre-activation activations were examined empirically and modeled as Gaussian despite some layers being less Gaussian-like.Testing other distribution-specific step sizes was outside the paper’s scope.
- 3.3. Model conversion: The conversion workflow records activations from typical inputs, collects layer statistics, and determines fixed-point formats for weights, biases, and activations.
- 3.3. Model conversion: The method computes step size as s = ξ · Stepsize(β) and fractional bits as n = −⌈log2 s⌉, where ξ captures distribution width and β is bit-width.In the experiments, ξ = 3σ; 2^-n is the representation resolution.
4. Bit-width optimization across a deep network
The paper models quantization noise through DCN layers using SQNR and derives how layerwise quantization steps combine at the network output. This analysis motivates cross-layer bit-width trade-offs, while noting assumptions and nonlinearities that limit accuracy.
- SQNR analysis: Quantization noise is analyzed through SQNR, with higher noise generally assumed to degrade classification performance.The paper uses layerwise SQNR as a tractable proxy and tests its relationship with classification accuracy empirically.
- SQNR analysis: For quantized weights and activations, the product’s noise is approximated by the separate weight- and activation-noise terms when both noises are small.The approximation drops the product of the two noise terms under |na| << |a| and |nw| << |w|.
- SQNR analysis: Independent quantization noise from weights and activations can be treated as total noise added after the product operation in a normalized linear system.This property supports the subsequent layerwise SQNR analysis.
- Forward pass through the network: Across a DCN, output SQNR is the harmonic mean of all preceding quantization-step SQNRs, so every step contributes and the worst step becomes the bottleneck.The result applies to input, weight, and activation quantization regardless of their network location.
- Forward pass through the network: Depth makes quantization more challenging, but the predicted loss from doubling depth could be recovered by adding 1 bit to all weights and activations when efficiency exceeds 3dB/bit.The paper states that this theoretical prediction requires empirical verification.
- Effects of network components: ReLU has little SQNR impact unless quantization perturbations change activation signs, whereas non-ReLU functions such as tanh, sigmoid, and PReLU are harder to analyze.The analysis may become increasingly inaccurate as bit-width decreases and quantization noise grows.
- Cross-layer bit-width optimization: Cross-layer bit-width allocation trades quantization quality between layers while preserving an overall output-SQNR constraint.This can target layers with many computations or parameters to reduce computation or model size.
- Cross-layer bit-width optimization: Optimal bit-width differences are inversely proportional to parameter-density differences in dB, scaled by quantization efficiency.With κ = 3dB/bit, a 3dB, or 2x, density difference corresponds to a 1-bit difference; parameter-heavy layers should use lower bit-width for compression.
5. Experiments
Experiments evaluate cross-layer bit-width optimization on CIFAR-10 and ImageNet, validate SQNR predictions, and examine fine-tuning after quantization. Optimization reduces model size at matched performance, but its benefit depends on network structure and hardware support.
- 5.1. Bit-width optimization for CIFAR-10 classification: The CIFAR-10 optimization allocates layer bit-widths using parameter counts and an assumed quantization efficiency of κ = 3dB/bit.For example, conv0 and conv1 differ by 5bits under the stated parameter-count calculation.
- 5.1. Bit-width optimization for CIFAR-10 classification: >20% reduction in model size is achieved at the same performance on CIFAR-10 with cross-layer bit-width optimization.Below approximately 25Mbits, error rate rises as model size decreases, while optimization preserves performance with a larger reduction.
- 5.2. Bit-width optimization for ImageNet classification: Up to 1Mbits of convolutional-layer savings are observed on ImageNet with optimized rather than equal bit-widths.The comparison uses model size versus top-5 error rate for an AlexNet-like network.
- 5.2. Bit-width optimization for ImageNet classification: The optimization is most effective when convolutional layers dominate model size; fully-connected layers can reduce the overall saving to less than 1%.In the AlexNet-like network, fully-connected layers exceed 100Mbits at bit-width 6, making a 1Mbit convolutional saving comparatively small.
- 5.3. Validation for SQNR prediction: Theoretical and measured SQNR values differ numerically but follow similar decreases across deeper convolutional layers.The predictions use layer bit-widths and quantization efficiency without network-parameter or test-data information.
- 5.4. Model fine-tuning: Fine-tuning after quantization yields 8.30% error for (4b, 4b) and 6.78% error for (float, 8b) on CIFAR-10.The results follow 30 epochs of fine-tuning; the latter is reported as a new state-of-the-art fixed-point result.
6. Conclusions
The paper presents SQNR-based bit-width optimization for converting pre-trained floating-point DCNs to fixed point, reducing model size without accuracy loss on CIFAR-10.
- >20% reduction in model size was achieved without any loss in accuracy on the CIFAR-10 benchmark.
- SQNR-based bit-width optimization outperformed uniform bit-width quantization in error-rate performance.
- Fine-tuning after conversion further improved the fixed-point DCN’s accuracy.