Source-linked AI summary
Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon Han, Sung Ju Hwang, Changkyu Choi
TL;DR
Reducing network bit-width lowers memory and computation costs but increases quantization error and can degrade accuracy. The paper introduces a trainable quantizer that learns quantization intervals by directly minimizing task loss, achieving strong low-bit ImageNet results and supporting heterogeneous-data training for pretrained networks.
Problem
Low-bit quantization reduces memory and computation costs, but increased quantization error can substantially degrade network accuracy.
Method
The method jointly learns parameterized quantization intervals and network weights for weights and activations by directly minimizing task-specific loss, with pruning and clipping integrated into the quantizer.
Results
4-bit networks preserve full-precision accuracy, 3-bit networks achieve comparable accuracy, and 2-bit networks incur minimal accuracy loss across evaluated ImageNet architectures.
Takeaways & Limitations
The quantizer achieves state-of-the-art ImageNet classification accuracy at 2-, 3-, and 4-bit widths and remains effective when trained on heterogeneous data for pretrained networks without original training data.
Abstract
from arXiv · showhide
Reducing bit-widths of activations and weights of deep networks makes it efficient to compute and store them in memory, which is crucial in their deployments to resource-limited devices, such as mobile phones. However, decreasing bit-widths with quantization generally yields drastically degraded accuracy. To tackle this problem, we propose to learn to quantize activations and weights via a trainable quantizer that transforms and discretizes them. Specifically, we parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network. This quantization-interval-learning (QIL) allows the quantized networks to maintain the accuracy of the full-precision (32-bit) networks with bit-width as low as 4-bit and minimize the accuracy degeneration with further bit-width reduction (i.e., 3 and 2-bit). Moreover, our quantizer can be trained on a heterogeneous dataset, and thus can be used to quantize pretrained networks without access to their training data. We demonstrate the effectiveness of our trainable quantizer on ImageNet dataset with various network architectures such as ResNet-18, -34 and AlexNet, on which it outperforms existing methods to achieve the state-of-the-art accuracy.
1. Introduction
The paper addresses the accuracy loss caused by low-bit quantization by learning quantization intervals directly for task performance. It applies this trainable quantizer to weights and activations and evaluates it on ImageNet across several architectures.
- Motivation: Low-bit quantization increases quantization error and can degrade accuracy, complicating deployment of deep networks on resource-limited devices.The paper targets reduced memory and computation while preserving full-precision accuracy.
- Approach: The trainable quantizer learns parameterized intervals that focus quantization on relevant values by pruning small values and clipping large ones.It combines a transformer from unbounded real values to normalized values with a discretizer mapping normalized values to integers.
- Evaluation: The method is evaluated on ImageNet using ResNet-18, ResNet-34, and AlexNet in an end-to-end training setting without layerwise optimization.The experiments compare the method with existing weight and activation quantization approaches.
- Approach: Weights and activations are quantized for every layer, with quantizer parameters and full-precision weights optimized jointly using the network’s task-specific loss.The approach avoids optimizing only weight, activation, or convolutional-output approximation errors.
- Results: The quantizer achieves state-of-the-art ImageNet classification accuracy with 2-, 3-, and 4-bit networks and can be trained on heterogeneous data for pretrained networks.This supports quantization when the original training data is unavailable.
2. Related Work
Prior low-precision methods reduce model size or accelerate computation through weight, activation, or combined quantization. The paper instead learns quantization intervals for both weights and activations while jointly optimizing the network, enabling pruning and clipping together.
- Weight quantization: Earlier methods reduce weight precision using binary or ternary representations, often with scaling factors or learned parameters.Examples include BinaryConnect, Binary-Weight-Network, Ternary-Weight-Network, and Trained-Ternary-Quantization.
- Weight and activation quantization: Methods such as BNN, XNOR-Net, and DoReFa-Net quantize both weights and activations to enable bit-wise convolution operations.DoReFa-Net uses multiple-bit quantization rather than only bipolar representations.
- Learned quantizers: LQ-Nets learns floating-point bases for K-bit values by minimizing quantization error, whereas this paper learns quantization intervals by minimizing task loss.The distinction is between approximating values and directly optimizing network objectives.
- Optimization strategies: TSQ uses layerwise weight optimization and is not applicable to skip-connected ResNet architectures, while the proposed quantizer is designed for architectures with skip connections.The paper also adopts a progressive strategy reported to improve extremely low-bit-width models, especially 2-bit networks.
- Proposed method: The proposed interval parameterization performs pruning and clipping simultaneously for both weights and activations during training.The method is reported to keep classification accuracy near full precision while substantially reducing bit-width.
3. Method
The method learns quantization intervals for weights and activations by combining trainable transformations with discretization and optimizing the network task loss. Its interval design jointly supports pruning and clipping, with separate weight and activation procedures for low-bit deployment.
- Quantization framework: The quantizer composes a transformer that normalizes values with a discretizer that maps them to discrete integers.The transformer maps values to [-1, 1] or [0, 1], while the discretizer applies rounding at a specified discretization level.
- Task-loss optimization: The quantizers and network weights are jointly optimized to minimize the entire network’s task loss rather than approximation or convolutional-output error.This directly targets classification performance and can also train quantizers on a pretrained network with fixed full-precision weights, though joint training performs better.
- Trainable intervals: Parameterized quantization intervals jointly enable pruning of small values and clipping of large values.The interval is represented by c∆ and d∆, denoting its center and distance from the center.
- Weight quantization: Weight quantization uses trainable cW, dW, and γ parameters, allowing either uniform or non-uniform quantization inside a symmetric interval.γ = 1 gives piecewise-linear uniform quantization, while γ ≠ 1 adjusts the distribution of quantized values; the nonlinear weight transformer is removed at inference.
- Activation quantization: Activation quantization prunes values below cX − dX, clips values above cX + dX, and linearly maps intermediate values to [0, 1].For activation bit-width NX, the number of nonzero quantization levels is qX = 2^NX − 1; 2-bit activations use {0, 1, 2, 3}.
- Training and deployment: During deployment, only quantized weights are retained, while low-bit weights and activations can enable bit-wise convolution operations.Activation quantization fixes γ = 1 for faster online inference, and training uses stochastic gradient descent with straight-through estimation for discretizer gradients.
4. Experiment results
Experiments on ImageNet and CIFAR-100 evaluate QIL across architectures, bit-widths, training strategies, pruning behavior, and heterogeneous-data settings. The method preserves strong accuracy at low bit-widths, benefits from joint training, and adapts quantization intervals through task-loss optimization.
- Evaluation setup: ImageNet experiments evaluate QIL on ResNet-18, ResNet-34, and AlexNet using low-bit weights and activations.The study also evaluates CIFAR-100 training with a heterogeneous dataset and reports implementation details for the tested architectures.
- ImageNet accuracy: 5/5- and 4/4-bit models preserve full-precision accuracy across the three ImageNet architectures, while 3/3-bit losses are 1% for ResNet-18, 0.6% for ResNet-34, and 0.5% for AlexNet.At 2/2 bits, accuracy drops by 4.5% on ResNet-18, 3.1% on ResNet-34, and 3.7% on AlexNet relative to full precision.
- Comparison with existing methods: QIL outperforms the second-best method by around 1% on 3/3- and 2/2-bit ResNet models and by 5.7% for 3/3-bit AlexNet.For AlexNet at 2/2 bits, its accuracy drop is 3.7%, nearly matching TSQ, which uses layerwise optimization.
- Training strategy: Jointly training weights and quantizers gives more graceful accuracy degradation than optimizing quantizers alone as bit-width decreases.For 4/4-bit ResNet-18, quantizer-only training drops accuracy by 2.1%, from 70.1% to 68.0%.
- Pruning behavior: 91% of AlexNet weights and 81% of ResNet-18 weights are pruned on average in 2/2-bit networks.The pruning ratio increases as bit-width decreases because lower resolution requires more compact quantization intervals.
- Heterogeneous-dataset training: On CIFAR-100, joint training preserves full-precision accuracy across heterogeneous datasets of different sizes, and accuracy improves from B10 through B80.The method also outperforms DoReFa-Net and PACT in this setting.
5. Conclusion
The paper introduces a trainable quantizer that learns parameterized quantization intervals jointly with network weights by directly minimizing task loss. It preserves full-precision accuracy at low bit-widths and remains effective when trained on heterogeneous data.
- The trainable quantizer parameterizes intervals and jointly learns them with weights by directly minimizing task loss rather than quantization error.It performs simultaneous pruning and clipping for weights and activations.
- Training on heterogeneous data still achieves good quantization performance, enabling quantization of pretrained networks without access to their original training data.