Source-linked AI summary
AdaBits: Neural Network Quantization with Adaptive Bit-Widths
Qing Jin, Linjie Yang, Zhenyu Liao
TL;DR
Deep neural networks need flexible deployment across platforms with different resource constraints, but fixed-precision or separately trained models limit instant adaptation. AdaBits enables adaptive weight and activation bit-widths through joint training and Switchable Clipping Level, achieving similar accuracies to individually quantized models across MobileNet V1/V2 and ResNet50 on ImageNet. The paper identifies bit-width as an additional adaptive-model choice for accuracy-efficiency trade-offs and hardware-aware deployment.
Problem
Neural networks must adapt to platforms with different resource constraints, while separately training models for each configuration is time-consuming and limits instant adaptation.
Method
AdaBits jointly trains one quantized model across selected bit-widths and uses Switchable Clipping Level to provide separate clipping behavior for each precision.
Results
AdaBits achieves similar accuracies to individually quantized models across MobileNet V1/V2 and ResNet50 on ImageNet.
Takeaways & Limitations
Bit-width becomes an additional degree of freedom for adaptive models, supporting instant adaptation to hardware and resource constraints.
Takeaways & Limitations
The evaluation uses only SAT among the many available neural-network quantization algorithms.
Abstract
from arXiv · showhide
Deep neural networks with adaptive configurations have gained increasing attention due to the instant and flexible deployment of these models on platforms with different resource budgets. In this paper, we investigate a novel option to achieve this goal by enabling adaptive bit-widths of weights and activations in the model. We first examine the benefits and challenges of training quantized model with adaptive bit-widths, and then experiment with several approaches including direct adaptation, progressive training and joint training. We discover that joint training is able to produce comparable performance on the adaptive model as individual models. We further propose a new technique named Switchable Clipping Level (S-CL) to further improve quantized models at the lowest bit-width. With our proposed techniques applied on a bunch of models including MobileNet-V1/V2 and ResNet-50, we demonstrate that bit-width of weights and activations is a new option for adaptively executable deep neural networks, offering a distinct opportunity for improved accuracy-efficiency trade-off as well as instant adaptation according to the platform constraints in real-world applications.
1. Introduction
Deep neural networks must serve platforms with sharply different resource constraints, but separately training models for each scenario prevents instant adaptation. AdaBits studies adaptive weight and activation bit-widths, finding joint training with Switchable Clipping Level can approach individually quantized models across several architectures.
- Separate models tailored to fixed resource budgets provide trade-offs but require time-consuming training and benchmarking, preventing instant adaptation.
- Adaptive bit-widths add weights and activations as a deployment degree of freedom, enabling one model to switch precision according to device conditions.The paper presents this as an alternative to separately trained models for different budgets.
- Quantizing MobileNet V2 to 6-bit compresses model size by roughly 4.74× and reduces BitOPs by 14.25× versus full precision.The comparison uses full-precision MobileNet V2 as the baseline.
- Direct adaptation is not applicable because shrinking or enlarging bit-widths can lose important information, while progressive training can damage performance at previously trained precisions.
- Joint training with Switchable Clipping Level achieves comparable performance to individually quantized models across several bit-widths and models.The study applies the techniques to MobileNet V1/V2 and ResNet50.
2. Related Work
Prior work studies quantization, neural architecture search, and adaptive architectures as separate routes to efficient or flexible neural networks. AdaBits connects adaptive deployment to bit-width selection, using a single model that switches precision according to computational budgets.
- Neural Network Quantization: Neural network quantization has progressed from uniform precision toward mixed-precision assignments across layers or channels, often searched automatically.
- Neural Architecture Search: Neural architecture search also tunes network training and structure across tasks including generative modeling, detection, and segmentation.
- Adaptive neural networks: Adaptive neural networks train one model with varying widths, depths, or kernel sizes to meet different application requirements.
- Adaptive neural networks: AdaBits frames adaptive bit-width deployment as a single-model alternative to individually training quantized models for each computational scenario.
3. Revisiting Scale-Adjusted Training (SAT)
Scale-Adjusted Training combines parameterized clipping activation with learning-based quantization to address training-related accuracy degradation. The resulting SAT baseline supports strong quantized-model performance and is used throughout AdaBits experiments.
- Quantization can reduce model capacity, but inefficient training accounts for a large portion of observed accuracy degradation according to prior work.
- SAT combines PACT with learning-based quantization to train quantized models more efficiently and consistently.The paper adopts this combination throughout its experiments.
- SAT quantized models can sometimes surpass their full-precision counterparts, motivating its use as the study’s baseline.
4. Quantization with Adaptive Bit-widths
Adaptive bit-width quantization enables one model to serve different resource budgets, but direct and progressive adaptation fail to preserve performance across precisions. Joint training achieves comparable performance to individually trained models, while Switchable Clipping Level addresses lowest-bit degradation with negligible overhead.
- 4. Quantization with Adaptive Bit-widths: Quantization with adaptive bit-widths offers flexible accuracy-efficiency trade-offs because increasing precision improves accuracy while raising model size and computation.For MobileNet V1/V2, changing from 4 to 8 bits increases model size by 1.7× and BitOPs by 3.2×.
- 4.1.1 Modified DoReFa Scheme: Modified DoReFa quantization enables direct adaptation from higher to lower bit-widths by discarding lower bits, unlike the original scheme.The original scheme lacks direct mapping between precisions and therefore requires storing full-precision weights and repeating quantization during deployment.
- 4.1.2 Direct Adaptation: Direct adaptation fails because models trained at one precision degenerate at another, with higher-bit models suffering especially severe degradation when quantized to lower precision.Batch-normalization calibration helps 2-bit models preserve larger-bit performance but does not match direct training, while 4-bit models degrade severely at smaller bits.
- 4.1.3 Progressive Quantization: Progressive training is unsuitable because ascending bit-width training corrupts lower-bit performance, while descending training fails to preserve higher-bit performance.These results indicate that sequential training introduces perturbations that impair previously trained bit-widths.
- 4.2. Joint Quantization: Joint training achieves comparable performance across bit-widths to individually trained models, although Vanilla AdaBits remains 0.5% below the individual model at 4 bits.The approach jointly optimizes shared weights across precisions and uses switchable batch normalization.
- 4.2.1 Switchable Clipping Level: S-CL assigns independent clipping levels to each bit-width and alleviates lowest-bit performance degeneration without materially increasing model size or runtime.Its clipping parameters add less than 0.1‰ of model size, and reconfigured models run without additional latency or memory cost.
5. Experiments
The experiments evaluate AdaBits on ImageNet across MobileNet V1/V2 and ResNet50, comparing adaptive models with individually quantized models using accuracy, model size, and BitOPs. They also examine layer-wise clipping levels to analyze S-CL.
- 5.1. ImageNet Classification: AdaBits achieves almost the same Top-1 accuracy as individual quantization across all tested models and bit-widths with the original scheme.The comparison covers MobileNet V1/V2 and ResNet50 on ImageNet; Table 4 reports Top-1 accuracy, model size, and BitOPs.
- 5.1. ImageNet Classification: AdaBits significantly improves ResNet50 performance at 2 bits over progressive quantization with ascending bit-width.
- 5.1. ImageNet Classification: AdaBits improves ResNet50 accuracy by 2.2% at both 4 bits and 3 bits over progressive quantization with descending bit-width.
- 5.1. ImageNet Classification: S-CL raises the lowest-4-bit performance by 0.3% on MobileNet V1 with the original scheme compared with Vanilla AdaBits.
- 5.2. Illustration of clipping levels: Different bit-widths produce different clipping levels, with larger bit-widths generally using larger levels as in individually quantized models.S-CL privatizes clipping levels across bit-widths so each can select a different optimum during optimization.
6. Discussion and Future Work
AdaBits extends adaptive model design by adding bit-width as a configurable dimension, while evaluation beyond the SAT quantization method remains future work.
- Bit-width joins channel number, depth, kernel size, and resolution as an additional adaptive-model design dimension.This expanded design space may support layer- or channel-level bit-width search for mixed-precision quantization.
- Adaptive bit-width models could support mixed-precision searches and more flexible models for applications such as face alignment and compressive imaging.
- Evaluation used only the state-of-the-art SAT quantization algorithm, leaving validation with other quantization methods for future work.The authors state that joint training can be combined with quantization-aware-training algorithms, but this generality remains unverified here.
7. Conclusion
The paper develops adaptive bit-width quantization through joint training and Switchable Clipping Level, achieving accuracies similar to individually quantized models across several ImageNet architectures.
- AdaBits achieves similar accuracies to individually quantized models across MobileNet V1/V2 and ResNet50 on ImageNet.
- The method jointly optimizes selected bit-widths and assigns clipping-level parameters separately to each bit-width to reduce interference.
- Adaptive bit-widths widen dynamic-model design choices for instant adaptation to different hardware and resource constraints.
S1. Proof of Theorem 1
The supplied proof passages establish Theorem 1 through inequalities involving the floor function and the condition a > b.
- Theorem 1 considers x in [0, 1] and positive integers satisfying a > b.
- The proof invokes the non-decreasing monotonicity of the floor function to derive an intermediate inequality.
- The argument uses a > b in a penultimate equality before obtaining the desired result from equations (S4) and (S6).