PaperlayerView exact revision on arXiv

Source-linked AI summary

DSConv: Efficient Convolution Operator

Marcelo Gennari, Roger Fawcett, Victor Adrian Prisacariu

arXiv:1901.01928v2cs.CV

TL;DR

Quantizing a pre-trained network without labelled data makes it difficult to maintain accuracy. DSConv quantizes weights and activations with a convolution operator designed to preserve their distributions; with unlabelled-data distillation, the paper reports less than 1% loss using 4-bit quantization.

  • Problem

    Maintaining high accuracy when quantizing a pre-trained model is much more difficult without labelled data for fitting the quantized model.

    Source: §1 · p. 1

  • Method

    DSConv factors weights into low-precision values and high-precision distribution shifts, quantizes activations similarly, and performs integer multiplications before applying scales.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    Less than 1% loss using 4-bit quantization for both weights and activations is reported with unlabelled data and distillation from the FP32 model.

    Source: §5 · p. 8

  • Takeaways & Limitations

    DSConv is presented as a tunable, drop-in convolution replacement that can achieve state-of-the-art results without labelled-data retraining or adaptation at up to 4 bits.

    Source: §1 · p. 1, §5 · p. 8

    The method relies on the assumption that quantization can preserve accuracy when the relative distributions of weights and activations remain unchanged.

    Source: §3 · p. 2

Abstract

from arXiv · show

Quantization is a popular way of increasing the speed and lowering the memory usage of Convolution Neural Networks (CNNs). When labelled training data is available, network weights and activations have successfully been quantized down to 1-bit. The same cannot be said about the scenario when labelled training data is not available, e.g. when quantizing a pre-trained model, where current approaches show, at best, no loss of accuracy at 8-bit quantizations. We introduce DSConv, a flexible quantized convolution operator that replaces single-precision operations with their far less expensive integer counterparts, while maintaining the probability distributions over both the kernel weights and the outputs. We test our model as a plug-and-play replacement for standard convolution on most popular neural network architectures, ResNet, DenseNet, GoogLeNet, AlexNet and VGG-Net and demonstrate state-of-the-art results, with less than 1% loss of accuracy, without retraining, using only 4-bit quantization. We also show how a distillation-based adaptation stage with unlabelled data can improve results even further.

1. Introduction

Quantization without labelled training data is difficult because the model cannot be fitted to expected activations and outputs. DSConv addresses this setting with a convolution operator that preserves weight and activation distributions while offering a tunable accuracy–memory–speed trade-off.

1. Introduction

  • Without labelled training data, maintaining high accuracy when quantizing a pre-trained model is much more difficult.Labelled data lets quantization fit the dataset and supplies prior knowledge about activation maps and expected outputs.Source: §1 · p. 1
  • DSConv targets this setting by quantizing weights and activations so their probability distributions mirror those of the full-precision model.Weights are factored into a low-precision kernel-sized component and a variable-size high-precision distribution-shift component; activation quantization uses a similar procedure inspired by block floating point.Source: §1 · p. 1
  • The proposed operator is a drop-in replacement for standard convolution, using low-bit fixed-point computation without labelled-data retraining and a tunable accuracy–memory–speed trade-off.The paper reports state-of-the-art results for its quantization strategy.Source: §1 · p. 1

2. Related Work

Prior quantization and compression work largely relied on labelled-data retraining, while data-free quantization had received less attention and was reported at up to 8 bits or used calibration. DSConv is presented as a 4-bit option, with potential further improvement from unlabelled-data adaptation.

2. Related Work

  • Earlier quantization work commonly retrained with labelled data, including methods using 1-bit or ternary weights and methods quantizing weights and activations.Some approaches applied 1-bit weights and activations to ImageNet for fast inference at a significant accuracy cost.Source: §2 · p. 2
  • Compression methods such as DeepCompression, HashedNet, and distillation-based compression also required retraining or labelled data for tuning.DeepCompression required retraining for pruning and quantization, while HashedNet required labelled data for tuning.Source: §2 · p. 2
  • Prior work on quantization without labelled data was less extensive and either reported 8-bit results or used calibration data.Industry systems were described as quantizing to 8 bits using sampled unlabelled images.Source: §2 · p. 2
  • DSConv is presented as effective at 4-bit quantization of weights and activations without labelled-data retraining, with further potential improvement from unlabelled-data adaptation.The paper contrasts this with prior work focused mainly on labelled-data settings.Source: §2 · p. 2

3. Method

DSConv represents weights and activations with low-bit integers and blockwise floating-point scales, aiming to preserve their distributions without labelled training data. Its inference uses integer block products followed by scale restoration, and an optional unlabelled-data distillation stage adapts the model.

3.1. Weight Quantization

  • For B=64, the weight-filter configuration reduces FP MACs from 1350 to 27; the method shares one floating-point value per block of size B along each filter’s depth dimension.B trades floating-point for fixed-point arithmetic, ranging from B=1 for pure floating point to B≥Ci for maximum fixed point.Source: Figure 1 · p. 3, §3.1 · p. 3

3.2. Activation Quantization

  • BFP activations share an exponent across B values: larger B increases cropping error but shrinks exponent/KDS tensors, while mantissa width b controls precision and cost.The method crops mantissas to the specified bit width after shifting each block to its maximum exponent.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Inference

  • Inference computes low-bit VQK–mantissa block dot products, combines the exponent and KDS scales, and produces floating-point output activations.The operation remains as parallelizable as standard convolution, with most multiplications replaced by integer multiplications.Source: §3.3 · p. 4, Figure 3 · p. 5

3.4. Distillation for Unlabelled Data Adaptation

  • With unlabelled data, distillation uses FP32 logits as targets to adapt a shadow model whose quantized-inference gradients update FP32 weights.The adaptation uses 960 validation images; accuracy is tested on the remaining 49,040 images.Source: §3.4 · p. 5

4. Experiments and Results

Across ImageNet architectures, experiments evaluate DSConv’s computational savings and accuracy without adaptation, with unlabelled-data adaptation, and after labelled-data retraining. Results show strong accuracy at 4–5 bits, while very low-bit performance depends on architecture and block size.

4.1. Theoretical Computational Load on Block Size

  • DSConv can be up to min(Ci, B) times faster than traditional convolution; with B = 128–256 and channel sizes above 256, it can be up to two orders of magnitude faster.The analysis compares integer and floating-point operations and notes that speed depends on block size, channel size, and implementation overhead.Source: §4.1 · p. 6

4.2. Accuracy Before Retraining or Adaptation

  • Without adaptation, 8-bit quantization preserves accuracy for compression and fast inference; 4-bit compression loses 1%–2%, while fast-inference 4/4 and 5/5 models lose 1%–3%.For 4-bit compression, block sizes of 16 to 32 often put accuracy within 1%–2% of full precision; 2-bit losses vary from 11% on ResNet50 to 40% on GoogLeNet.Source: §4.2 · pp. 6–7

4.3. Accuracy Adapted using Unlabelled Data

  • With unlabelled-data adaptation, 4-bit weights and activations reach within 2% of FP32 accuracy even at block size 128, while 3-bit results retain a 6% gap on ResNet50 and up to 10% on GoogLeNet.The authors also report their best fast-inference result using adaptation data, including at block size 64.Source: §4.3 · p. 7

4.4. Accuracy After Labelled Data Retraining

  • DSConv beats state-of-the-art in several 4- and 5-bit cases when block size is tuned toward accuracy, including ResNet18, ResNet50, and GoogLeNet configurations.The reported cases use large block sizes; ResNet18 at 5/5 with B = 128 and 4/4 cases with B = 64 are slightly better than FP efficiency.Source: §4.4 · p. 8
  • For 3 bits or less, state-of-the-art results require smaller blocks; at very low bit widths, pretrained quantization is not ideal, and ResNet50 compression with 2 bits stays within 1% at B = 32.The paper notes that leading 1- and 2-bit methods are trained from scratch.Source: §4.4 · p. 8

5. Conclusion

DSConv achieves state-of-the-art results with weights and activations quantized to as little as 4 bits, without retraining or adaptation. Unlabelled-data distillation from the FP32 model brings 4-bit quantization to less than 1% accuracy loss, while 1–3-bit settings appear to require retraining.

  • DSConv achieves state-of-the-art results with weights and activations quantized to up to 4 bits, without retraining or adaptation.Source: §5 · p. 8
  • Without retraining, sub-8-bit DSConv can support fast inference and lower power consumption for rapid deployment in custom hardware.Source: §5 · p. 8
  • Less than 1% loss is achieved with 4-bit weights and activations when using unlabelled data and distillation from the FP32 model.Source: §5 · p. 8
  • At 1–3 bits, lower weight information loss no longer corresponds to higher inference accuracy, and retraining seems inevitable because these models differ from higher-accuracy models.Source: §5 · p. 8

Related papers

Loading 1901.01928v2…