VLCQ: Post-training quantization for deep neural networks using variable length coding

1 Introduction

• Deep learning models’ memory, power, and latency demands constrain deployment on edge and other resource-limited devices.Quantization is one of the proposed responses to these deployment constraints.Source: §1 · pp. 1–2
• Data-free post-training quantization often degrades at low bit-widths, while high-performing methods commonly require calibration data or fine-tuning.The paper identifies rounding errors, fixed-scale calculations, and distribution shifts among the challenges.Source: §1 · p. 2
• VLCQ applies post-training quantization to CNNs without training data or fine-tuning, using weight asymmetry and scale selection to reduce quantization error.The method also uses variable-length encoding to lower bit rate for non-uniform weight distributions.Source: §1 · p. 3
• Variable-length encoding removes the fixed-length constraint, allowing asymmetric weight distributions and flexible bin counts to reduce quantization error and bit rate.The paper describes weight distributions as close to Gaussian or Laplace.Source: §1 · p. 3
• Different scale-selection methods and sensitivity metrics are used to address quantization rounding errors without access to any part of the dataset.The paper lists this as a contribution to VLCQ.Source: §1 · p. 3
• Mixed-precision methods improve VLCQ and reduce quantization loss at low bit rates.The paper also explores non-uniform binning schemes that assign different bit-widths to different regions.Source: §1 · p. 3

2 Related Work

• Model-compression approaches include efficient network layers, architecture optimization, pruning, activation compression, knowledge distillation, and quantization.Quantization maps weights or activations from floating-point to integer space, enabling lower-bit storage and integer computation.Source: §2 · pp. 3–4
• Quantization-aware training retrains models with simulated low-precision forward passes, whereas post-training quantization quantizes pretrained models without retraining.The paper further divides post-training methods into data-free, mixed-precision, and small-calibration-data approaches.Source: §2 · p. 4
• ACIQ combines analytical activation clipping, per-channel bit allocation, and bias correction for data-free 4-bit post-training quantization.The described method adjusts activation bounds and shifts and rescales the quantized weight distribution.Source: §2 · p. 4
• PWLQ uses dense and sparse quantization regions, while DFQ uses cross-layer equalization and bias absorption for per-tensor quantization.The paper notes that DFQ is less effective per-channel and cannot be integrated with PWLQ.Source: §2 · p. 4
• GDFQ generates synthetic data from a pretrained model and uses it to fine-tune a quantized model for low-bit weights and activations.By contrast, small-calibration-data methods use a subset of training data to fine-tune models or adjust quantization parameters.Source: §2 · pp. 4–5
• Calibration-based methods include AdaRound’s per-layer rounding optimization [23], AdaQuant’s bit allocation and tuning [22], and BRECQ’s block reconstruction [24].BRECQ’s block-level activation-error minimization accounts for interactions among layers within a block.Source: §2 · p. 5
• Mixed-precision methods assign different bit counts across layers or channels; PNMQ uses optimized non-uniform grids and data-free bit-width selection.The review also describes a Taylor-expansion and knapsack approach to bit allocation [31].Source: §2 · p. 5

3 Motivation

• Lower-bit quantization degrades performance across CNN families, and errors can arise from distribution shift, rounding noise, and layer interactions.The section also notes that weight-based error metrics do not directly measure original task loss or activation error.Source: §3 · p. 5

3.1 Weight ranges and distribution

• In MobileNetV2, channel weight ranges can vary sharply; per-tensor quantization may map small-range weights to few bins or zero, while some layers also challenge per-channel quantization.The cited examples include channel ranges of [-1.5, 1.5] and [-0.1, 0.6], and a layer range of [-54.4, 64.4].Source: §3.1 · p. 6, Figure 1 · p. 6
• Uniform quantization uses symmetric, equally spaced levels, but bell-shaped weight and activation distributions can leave substantial approximation error, especially at low bit-widths.The section describes pretrained distributions as similar to Gaussian or Laplacian shapes.Source: §3.1 · p. 6, §3.2 · p. 6

3.2 Quantization error

• Minimizing weight-space quantization error may not optimize task performance, while activation-aware optimization and rounding fine-tuning require dataset access.The section identifies fixed quantization parameters, including scale and range bounds, as choices whose effects could be explored; it calls for a framework covering per-tensor and per-channel, high- and low-bit settings without data.Source: §3.2 · p. 7

4.1 Post-training quantization using variable length coding

• VLCQ divides asymmetric weight distributions into negative and positive regions and quantizes each region separately, with one-sided distributions handled using the full bit-width.The method uses a breakpoint of zero or the distribution median; the regions may receive equal or different numbers of levels.Source: §4.1 · pp. 8–9

4.2 Mixed precision Quantization

• Mixed-precision quantization assigns different numbers of bins to channels or regions while keeping the total bit-rate within the required budget.The approach addresses channel-level and region-level allocation; the paper notes that choosing sensitivity without the dataset and searching all allocations are difficult.Source: §4.2 · p. 9

Non-uniform quantization through multiple break-points

• Non-uniform quantization uses narrower bins near the dense center and bins twice as wide in sparse tails, allocating more resolution near zero.Each positive or negative region is split at a breakpoint; the passage describes bin widths x and 2x for the central and tail subregions.Source: Non-uniform quantization through… · p. 10, Figure 4 · p. 10

4.3 Scale selection

• Scale search explores candidate scales to minimize the Lp norm, while min-max adjustment changes the distribution range; the combined approach searches scales for each adjusted range.For distributions entirely on one side of zero, the method fixes one endpoint and searches the other. Table 2 compares rounding schemes for INT8 MobileNetV2 on ImageNet validation accuracy and weight and activation MSE; the prose reports flooring as the best scheme.Source: §4.3 · pp. 11–12, Table 2 · p. 11

4.4 Variable length encoding

• Golomb-Rice coding assigns shorter representations to frequent weights and longer ones to infrequent weights, reducing average bit-rate for the bell-shaped weight distributions described.The method codes zero with a one-bit symbol and uses a power-of-two parameter M, enabling shifting and masking instead of division and modulo.Source: §4.4 · pp. 12–13

4.5 Hardware support

• Hardware support adds a variable-length decoder before multiply-accumulate operations and uses multiple accumulators to handle separate scales.The paper also describes parallel decoding through a blocked weight layout; software decoding is an option when memory, rather than computation, is the limiting resource.Source: §4.5 · p. 14

• Experiments use ResNet and MobileNet architectures for ImageNet classification.Source: §5 · p. 15
• Batch normalization is folded before quantization, and bias correction is applied afterward.Source: §5 · p. 15
• Activations remain full precision; uniform 8-bit activation quantization without calibration data lowers results by 0.01%.Source: §5 · p. 15

• Batch normalization is folded before quantization, and bias correction is applied afterward.Source: §5.1 · p. 15
• Activations remain full precision; uniform 8-bit activation quantization without calibration data lowers results by 0.01%.Source: §5.1 · p. 15
• Per-channel scale selection searches about 20 values between b + 1 and b −1, while per-tensor selection searches about 300 values.Source: §5.1 · p. 15
• Scale selection uses p = 2 in the lp-norm; mixed-precision rates range from 8 to 3 bits, or 256 to 8 levels.Source: §5.1 · p. 15
• Experiments ran on one Tesla P100 GPU, a single-core Xeon processor, and 15.26 GB available RAM.Source: §5.1 · p. 15

5.2 Ablation Study on ImageNet

• The ablation evaluates VLCQ and related techniques on ImageNet, naming ResNet18, ResNet50, and MobileNetV2 among its models.Source: §5.2 · p. 15

5.2.1 VLCQ

• VLCQ reaches 3- and 2-sub-bit rates with good accuracy for ResNet18 and ResNet50; Figure 8 relates accuracy and bit-rate to bin count.Source: §5.2.1 · p. 15, Figure 8 · p. 16

5.2.2 Scale-Selection

• Scale search is used for both per-tensor and per-channel quantization, with about 300 and 20 candidate values respectively; Min-Max adjustment is used per-channel.Source: §5.2.2 · p. 15
• VLCQ Min-Max SS maintains near-exact FP32 accuracy up to 32 bins, while VLCQ SS improves accuracy by up to 4% at the same bit-width and number of levels.Source: §5.2.2 · pp. 16–17

5.2.3 One Scale uniform quantization

• One-scale uniform quantization experiments compare searching possible scales with changing the weights’ minimum and maximum on ImageNet models.Source: §5.2.3 · p. 17

5.2.4 Mixed Precision

• Mixed precision frequently beats unified precision at extremely low bit-widths, with up to 3% higher accuracy at the same bit-width.Source: §5.2.4 · p. 17

5.3.1 Image Classification

• On ResNet18, PWLQ achieves 69.622% accuracy, while VLCQ Min-Max SS achieves 69.790% accuracy at 5.3 bits, better than FP32 accuracy; quantization can occasionally improve accuracy.On MobileNetV2, VLCQ MP achieves around 1% error loss, similar to BRECQ’s 1% error loss; VLCQ SS achieves around 1.6% error with a sub-5-bit rate.Source: §5.3.1 · p. 18
• Across per-channel classification, VLCQ matches or beats most methods with fewer bits; ResNet18 and ResNet50 retain good accuracy below 3 bits with 3–4% accuracy loss.Figure 9 plots top-1 accuracy against bit-width for post-training per-channel weight quantization; activations remain full precision.Source: §5.3.1 · pp. 18–19, Figure 9 · p. 19
• VLCQ, while data-free, comes tightly close to AdaRound’s performance; for per-tensor quantization, it outperforms most existing methods with acceptable performance at 8-, 6-, and 7-bit rates.Source: §5.3.1 · p. 19
• Golomb coding and symmetric quantizers reduce VLCQ memory use versus PWLQ and uniform quantization, while its FLOPs remain similar to uniform quantizers.The paper attributes Golomb coding's low entropy to exponentially distributed data and reports that coding performs less well for PWLQ, BRECQ, and DFQ.Source: §5.3.1 · p. 19, Table 7 · p. 20, Table 8 · p. 20

5.3.2 VLCQ Extendability

• Applying scale selection to BRECQ improves its 4-bit result, while the proposed two-scales method outperforms original BRECQ at both 2 and 4 bits.The authors note original BRECQ performs better than scale-selection BRECQ at 2 bits.Source: §5.3.2 · p. 20, Table 9 · p. 21

5.3.3 Other tasks

• On Pascal VOC segmentation, VLCQ Min-Max SS exceeds PWLQ by 0.44% at 128 levels and 1.4% at 32 levels.The tested DeepLab-v3+ model uses a MobileNetV2 backbone; the paper describes it as challenging because some weight ranges are large.Source: §5.3.3 · p. 21
• For Pascal VOC detection, VLCQ variants outperform PWLQ at 256, 128, 64, and 32 levels; against Q-YOLO and, they also perform better down to 16 levels.VLCQ, VLCQ Min-Max, and VLCQ Min-Max SS match FP32 performance at bit-rates ranging from 5 to 6; VLCQ Min-Max SS leads at low bit-rates.Source: §5.3.3 · p. 21
• Current limitations include fixed search spaces, a zero breakpoint, sensitivity to outliers, and weaker performance for per-tensor quantization in outlier cases.The authors suggest larger per-tensor search spaces and calibration data could improve selection of scales or min-max values.Source: §5.3.3 · p. 21

6 Conclusion

• VLCQ is a post-training quantization method using variable-length encoding that requires no access to the original full dataset.Source: §6 · p. 21
• The method divides weights into two regions, assigns each region an equal number of bins, and uses the distribution to compress weights to low bit-widths.Source: §6 · p. 22
• VLCQ delivers near-original accuracy for nearly every evaluated model and competes with more complex training-based methods; the framework is described as efficient, hyperparameter-free, and readily integrable.Source: §6 · p. 22