Source-linked AI summary
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, Junjie Yan
TL;DR
Deep neural networks are costly to deploy, while extremely low-bit quantization can cause inaccurate gradients, unstable training, and performance degradation. The paper introduces DSQ, an evolving differentiable approximation to standard binary and uniform quantization. Across network structures, DSQ consistently outperforms state-of-the-art quantization methods and supports efficient low-bit deployment, with the method scoped to 1-bit binary and multi-bit uniform quantization.
Problem
Resource-limited deployment needs lower storage and computation costs, but discrete extremely low-bit quantization causes inaccurate gradients and quantization loss.
Method
DSQ uses differentiable hyperbolic-tangent approximations with evolution training and learned clipping values to approach standard binary and uniform quantization.
Results
DSQ consistently outperforms state-of-the-art quantization methods across several network structures and provides state-of-the-art accuracy for various network structures.
Takeaways & Limitations
DSQ is a flexible module for binary and uniform quantization with hardware-friendly efficiency based on fast fixed-point GEMM kernels.
Takeaways & Limitations
The paper considers standard 1-bit binary and multi-bit uniform quantization.
Abstract
from arXiv · showhide
Hardware-friendly network quantization (e.g., binary/uniform quantization) can efficiently accelerate the inference and meanwhile reduce memory consumption of the deep neural networks, which is crucial for model deployment on resource-limited devices like mobile phones. However, due to the discreteness of low-bit quantization, existing quantization methods often face the unstable training process and severe performance degradation. To address this problem, in this paper we propose Differentiable Soft Quantization (DSQ) to bridge the gap between the full-precision and low-bit networks. DSQ can automatically evolve during training to gradually approximate the standard quantization. Owing to its differentiable property, DSQ can help pursue the accurate gradients in backward propagation, and reduce the quantization loss in forward process with an appropriate clipping range. Extensive experiments over several popular network structures show that training low-bit neural networks with DSQ can consistently outperform state-of-the-art quantization methods. Besides, our first efficient implementation for deploying 2 to 4-bit DSQ on devices with ARM architecture achieves up to 1.7$\times$ speed up, compared with the open-source 8-bit high-performance inference framework NCNN. [31]
1. Introduction
Low-bit quantization reduces storage and inference cost but suffers from inaccurate gradients and quantization loss. DSQ addresses both issues with an evolving differentiable approximation that supports flexible, efficient deployment.
- Motivation: Quantization reduces network storage and accelerates inference, supporting deployment on resource-limited devices.Binary, uniform, and non-uniform quantizers are discussed as compression approaches.
- Motivation: Extremely low-bit binary and uniform models face performance degradation because discrete levels impair gradient access and introduce quantization loss.Clipping and rounding both contribute to deviations from the original data.
- Differentiable Soft Quantization: DSQ uses hyperbolic tangent functions and evolution training to gradually approximate binary and uniform quantization while retaining differentiability.The approximation characteristic variable and clipping values can be learned during training.
- Differentiable Soft Quantization: DSQ reduces gradient deviation and balances clipping and rounding errors, making forward and backward training processes more consistent and stable.Its jointly determined clipping range and approximation are intended to balance quantization loss.
- Advantages: DSQ is compatible with binary and uniform quantization, can be inserted into standard network structures, and uses fast computation kernels for efficient inference.The method is presented as flexible across quantization methods and deployment settings.
2. Related Work
Related work develops hardware-friendly quantization and deployment frameworks, but existing frameworks commonly emphasize 8-bit arithmetic. The paper implements lower-bit ARM inference using NEON technology.
- Quantization approaches: Binary and ternary quantization use only two or three values, making high-accuracy model training challenging.Bit operations are one route to accelerating low-precision networks.
- Efficient deployment: Deployment frameworks such as TensorRT, Intel Caffe, Gemmlowp, and NCNN provide optimized low-precision inference for different hardware platforms.The cited frameworks target GPU, CPU, ARM, Intel X86, or mobile deployments.
- Efficient deployment: Most described frameworks support 8-bit integer arithmetic rather than specific lower-bit computation.The paper addresses this gap with 2-bit fast integer arithmetic using ARM NEON technology.
3. Differentiable Soft Quantization
DSQ replaces nondifferentiable low-bit quantization with a smooth, controllable approximation that evolves toward standard binary or uniform quantization during training. Optimizing its similarity and clipping parameters helps preserve gradients and balance clipping against rounding error.
- Quantization function: DSQ approximates binary and uniform quantization with piecewise hyperbolic-tangent functions that remain smooth for gradient calculation.The coefficient k controls how closely the asymptotic function approaches the staircase produced by uniform quantization.
- Quantization function: Binary quantization is a special case of DSQ obtained when the piecewise uniform quantizer has one interval and is composed with the sign function.This covers the standard 1-bit binary setting with values −1 and +1.
- Evolution to standard quantization: DSQ uses the characteristic variable α to control approximation quality and evolves α during training so the soft quantizer progressively approaches the standard quantizer.The gradient of α is calculated during backpropagation and updated with an L2 regularization constraint.
- Balancing clipping error and rounding error: DSQ jointly optimizes α, l, and u to balance clipping and rounding errors, whose relative contributions depend on whether outliers or middle-range points dominate.Clipping more increases clipping error while decreasing rounding error; DSQ adjusts the clipping bounds to pursue a balance.
- Training procedure: During fine-tuning, DSQ clips activations and weights, applies asymptotic functions, computes the convolution, and updates network and DSQ parameters by backpropagation.The procedure is specified for convolution layers and includes learned clip values and similarity factors.
4. Experiments
The experiments evaluate DSQ on CIFAR-10 and ImageNet, covering small-scale and large-scale image classification settings.
- Datasets: Experiments use CIFAR-10 and ImageNet (ILSVRC12) to evaluate the effectiveness of DSQ.CIFAR-10 has 50K training and 10K testing images across 10 classes; ImageNet has about 1.2 million training and 50K testing images across 1,000 classes.
4.1. Settings
The implementation inserts differentiable DSQ modules into binary or uniform quantization models and evaluates both pretrained fine-tuning and training-from-scratch settings.
- DSQ function: DSQ is implemented in PyTorch as a differentiable module compatible with binary and uniform quantization approaches.The implementation supports 1-bit binarization with {−1,+1} and uniform quantization.
- Model configuration: All convolutional and fully connected layers except the first and last are quantized with DSQ, using a parameter-free type-A shortcut and a replacement activation function for binarized models.The shortcut follows [15], and the activation replacement follows [16].
- Initialization: The study compares fine-tuning from a pretrained model with training from scratch, initializes α to 0.2, and follows prior work for other hyperparameters.Clipping values l and u are selected through the settings described in the paper.
4.2. Analysis of DSQ
DSQ rectifies data toward quantization points, improves convergence, and automatically evolves its approximation during training.
- 4.2.1 Rectification: DSQ redistributes weights toward quantization values, enabling complete 2-bit quantization to four points after rectification.The weight distribution changes from approximately normal to a histogram with several peaks.
- 4.2.2 Convergence: Training with DSQ achieves higher validation accuracy than training without DSQ for binarized VGG-Small and 3-bit ResNet-34.The comparison covers CIFAR-10 and ImageNet validation curves.
- 4.2.3 Evolution: The evolution strategy constrains α to (0, 0.5) with k ≤1000 after initializing α to 0.2 for ResNet-20 weights and activations.The reported setting is for ResNet-20 on CIFAR-10.
- 4.2.3 Evolution: α initially increases sharply, then decreases and converges to a stable value as DSQ progressively approximates standard quantization.This evolution indicates limited quantization at the beginning of training and stronger approximation later.
- 4.2.3 Evolution: Weights generally use smaller final α values than activations, indicating greater weight tolerance to quantization and layer-dependent sensitivity.Downsampling convolution layers are cited as an example of differing layer sensitivity.
4.3. Ablation study
Ablations show that DSQ improves binary and uniform quantization, while learned approximation and clipping parameters further improve 2-bit performance and PACT results.
- 4.3. Ablation study: Nearly 2% performance gain is achieved for 1W1A ResNet-20 on CIFAR-10 using naive DSQ with fixed α versus direct sign-function binarization.The improvement is stable across the reported binarization comparison.
- 4.3. Ablation study: Learning adaptive α and clipping values l, u each improves accuracy for 2-bit uniform quantization of ResNet-20 on CIFAR-10.The ablation separates evolution training from balanced quantization error.
- 4.3. Ablation study: DSQ further boosts the performance of PACT, demonstrating compatibility with a standard state-of-the-art quantization method.The PACT comparison is reported for the ablation study.
4.4. Comparison with State-of-the-arts
Across CIFAR-10 and ImageNet, DSQ consistently outperforms the compared low-bit quantization methods while retaining uniform quantization’s hardware-friendly character.
- 4.4. Comparison with State-of-the-arts: 84.11% versus 79.30% is reported for 1-bit weights and activations against DoReFa-Net on CIFAR-10.DSQ obtains the best performance across the listed VGG-Small and ResNet-20 comparisons.
- 4.4. Comparison with State-of-the-arts: DSQ using 1-bit weights on ResNet-18 outperforms BWN and HWGQ by large margins and surpasses TWN using 2 bits on ImageNet.The passage introduces broader comparisons across ResNet-18, ResNet-34, and MobileNetV2.
- 4.4. Comparison with State-of-the-arts: 3.4% higher accuracy than PACT using 4-bit quantization is reported for MobileNetV2 on ImageNet.The result supports DSQ’s potential for hardware-friendly networks with few parameters.
4.5. Deploying Efficiency
DSQ supports sub-4-bit integer arithmetic on ARM NEON-based GEMM kernels and accelerates inference compared with NCNN-8-bit.
- 4.5. Deploying Efficiency: DSQ supports extremely low-bit, less-than-4-bit integer arithmetic using GEMM kernels with ARM NEON 8-bit instructions.The implementation targets inference on ARM architecture.
- 4.5. Deploying Efficiency: Inference with DSQ is much faster than NCNN on a Raspberry Pi 3B when quantizing ResNet-18.The device uses a 1.2 GHz 64-bit quad-core ARM Cortex-A53.
5. Conclusions
DSQ narrows the accuracy gap between full-precision and low-bit networks by dynamically approximating standard quantization during end-to-end training. It supports binary and uniform quantization while retaining hardware-friendly inference efficiency.
- DSQ dynamically evolves during end-to-end training to approximate standard quantization and reduce the accuracy gap between full-precision and low-bit networks.
- DSQ reduces gradient deviation during backward propagation and quantization loss during forward inference.
- DSQ serves as a flexible module for both model binarization and uniform quantization across different quantization methods.
- DSQ combines fast fixed-point GEMM kernels with hardware-friendly efficiency for deployment.