PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroQ: A Novel Zero Shot Quantization Framework

Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, Kurt Keutzer

arXiv:2001.00281v1cs.CV

TL;DR

Quantization methods often need training data or fine-tuning, which is problematic for sensitive, proprietary, or inaccessible datasets and can impair ultra-low-precision performance. ZEROQ addresses this with synthetic Distilled Data and automatic Pareto-frontier mixed-precision selection, achieving strong results across models and tasks, including 1.71% higher MobileNetV2 accuracy than DFQ.

  • Problem

    Existing quantization approaches may require inaccessible training data or costly fine-tuning, while zero-shot heuristics can perform poorly at ultra-low precision.

  • Method

    ZEROQ optimizes synthetic Distilled Data to match batch-normalization statistics, uses it for sensitivity analysis, and selects mixed-precision bit settings through Pareto-frontier optimization.

  • Results

    ZEROQ achieves state-of-the-art quantization results across tested classification and detection models, including 1.71% higher MobileNetV2 accuracy than DFQ.

  • Takeaways & Limitations

    ZEROQ supports uniform and mixed-precision quantization without training or validation data, with end-to-end ResNet50 quantization taking less than 30 seconds.

Abstract

from arXiv · show

Quantization is a promising approach for reducing the inference time and memory footprint of neural networks. However, most existing quantization methods require access to the original training dataset for retraining during quantization. This is often not possible for applications with sensitive or proprietary data, e.g., due to privacy and security concerns. Existing zero-shot quantization methods use different heuristics to address this, but they result in poor performance, especially when quantizing to ultra-low precision. Here, we propose ZeroQ , a novel zero-shot quantization framework to address this. ZeroQ enables mixed-precision quantization without any access to the training or validation data. This is achieved by optimizing for a Distilled Dataset, which is engineered to match the statistics of batch normalization across different layers of the network. ZeroQ supports both uniform and mixed-precision quantization. For the latter, we introduce a novel Pareto frontier based method to automatically determine the mixed-precision bit setting for all layers, with no manual search involved. We extensively test our proposed method on a diverse set of models, including ResNet18/50/152, MobileNetV2, ShuffleNet, SqueezeNext, and InceptionV3 on ImageNet, as well as RetinaNet-ResNet50 on the Microsoft COCO dataset. In particular, we show that ZeroQ can achieve 1.71\% higher accuracy on MobileNetV2, as compared to the recently proposed DFQ method. Importantly, ZeroQ has a very low computational overhead, and it can finish the entire quantization process in less than 30s (0.5\% of one epoch training time of ResNet50 on ImageNet). We have open-sourced the ZeroQ framework\footnote{https://github.com/amirgholami/ZeroQ}.

I. INTRODUCTION

ZEROQ targets quantization when training data are unavailable, using synthetic Distilled Data to estimate sensitivity and support automatic mixed-precision selection. The framework reports broad evaluations with low computational overhead.

  • Quantization reduces model size, memory traffic, inference cost, and energy use through lower-precision integer computation.
  • Training-data access is often unavailable because quantization-aware fine-tuning is costly and datasets may be sensitive or proprietary.
  • ZEROQ quantizes models without training or validation data and automatically computes mixed-precision configurations without expensive search.
  • Distilled Data are synthetic inputs optimized to match batch-normalization statistics, enabling sensitivity analysis in place of original data.The method uses this reconstruction for post-training quantization and reports that the resulting sensitivity matches that of original training data.
  • 12s is the reported total sensitivity-computation cost for ResNet50, while the complete mixed-precision setting can be determined in under 14s.The framework is evaluated on classification and object detection models, including ResNet variants, compact networks, InceptionV3, and RetinaNet on COCO.
  • 1.71% higher accuracy is achieved on MobileNetV2 than with DFQ, alongside support for uniform and mixed-precision quantization.

II. RELATED WORK

Related work situates quantization among complementary model-efficiency techniques and reviews post-training, zero-shot, and mixed-precision approaches. The central challenge is selecting layer precisions efficiently while preserving accuracy without data access.

  • Model-efficiency methods besides quantization include architecture design, knowledge distillation, pruning, and hardware–network co-design.
  • Post-training methods avoid fine-tuning but can suffer performance degradation and commonly require limited unlabeled data.
  • DFQ extends post-training quantization to zero-shot settings using weight equalization to remove weight and activation outliers.
  • Mixed-precision quantization assigns different bit precisions across layers because layers differ in sensitivity to quantization.
  • Naive mixed-precision selection is computationally expensive because the per-layer precision search space grows exponentially.

III. METHODOLOGY

ZEROQ supports fixed- and mixed-precision quantization without training or validation data by using model-derived sensitivity and quantization statistics. Mixed precision preserves higher precision for sensitive layers while quantizing less-sensitive layers more aggressively.

  • Quantization setup: Quantization clips weights or activations to [a, b] and uniformly discretizes the range into 2^k − 1 intervals using asymmetric quantization.The interval length is Δ = (b − a)/(2^k − 1).
  • Quantization setup: ZEROQ supports both fixed-precision and mixed-precision quantization, allowing different layers to use different bit precisions.The mixed-precision strategy assigns higher precision to sensitive layers and more aggressive quantization to less-sensitive layers without increasing overall model size.
  • Sensitivity metric: For post-training quantization, KL divergence between the original and quantized models measures each layer’s sensitivity at a chosen bit precision.A smaller Ω_i(k) indicates that quantizing layer i to k bits causes less output deviation from the full-precision model.
  • Sensitivity metric: ZEROQ estimates quantization sensitivity without training data by feeding synthetic inputs engineered to match statistics encoded in batch-normalization layers.The method assumes normalized inputs and uses the running means, variances, scaling, and bias parameters stored in the network’s batch-normalization layers.

A. Distilled Data

ZEROQ distills synthetic inputs from the trained network by matching batch-normalization statistics, enabling activation-range estimation and layer-sensitivity measurement without training or validation data. The resulting Distilled Data captures finer local structure than Gaussian inputs and closely matches the model’s sensitivity profile.

  • Motivation: Zero-shot quantization must estimate activation ranges and mixed-precision layer sensitivities without access to training or validation data.Gaussian inputs fail to reproduce the correct activation statistics and can reverse the sensitivity ordering of early layers.
  • Uses and evaluation: The Distilled Data visualization shows more local structure than Gaussian data.The figure compares Gaussian data on the left with Distilled Data on the right, generated according to Algorithm 1.
  • Distillation method: ZEROQ generates synthetic Distilled Data by optimizing inputs to match the mean and standard deviation statistics stored in batch-normalization layers.The reconstructed data distribution is designed to closely match the original model’s internal statistical distribution.
  • Distillation method: The distillation procedure starts from Gaussian random data and outputs a batch of distilled data for a model with L batch-normalization layers.Algorithm 1 uses the model and its batch-normalization layers as inputs to the data-generation process.
  • Uses and evaluation: Distilled Data can provide activation ranges for clipping and inputs for computing mixed-precision sensitivity without accessing the original dataset.Using it in the sensitivity calculation closely matches the sensitivity obtained with the original training data, unlike Gaussian inputs.

B. Pareto Frontier

ZEROQ selects mixed-precision bit settings by minimizing quantization sensitivity under a target model-size constraint, using a Pareto frontier rather than exhaustive search.

  • The mixed-precision search space has exponential size, reaching 7.2 × 10^23 configurations for ResNet50 with bit options {2, 4, 8}.
  • For a target model size, ZEROQ chooses the bit-precision configuration with minimum overall sensitivity.
  • The Pareto frontier plots model size against sensitivity and identifies configurations with minimal perturbation for selected size constraints.
  • Dynamic programming computes the best settings for different target sizes, while layer-sensitivity computation gives the Pareto optimization O(mL) overhead.
  • The method is not theoretically guaranteed to find the globally best configuration, although reported mixed-precision results achieve state-of-the-art accuracy with small performance loss.

IV. RESULTS

ZEROQ was evaluated across classification models and datasets using fully zero-shot quantization, with strong accuracy retention, competitive comparisons, and low computational overhead.

  • ZEROQ evaluates ResNet, MobileNetV2, ShuffleNet, ResNet152, InceptionV3, SqueezeNext, and ResNet20 across ImageNet and CIFAR10, with additional RetinaNet evaluation on COCO.
  • All reported results are 100% zero-shot, requiring neither fine-tuning nor data assistance, including the Microsoft COCO experiments.
  • ImageNet: 77.43% accuracy: ZEROQ achieves this for ResNet50 W6A6, 2.63% higher than OCS despite a slightly smaller model.
  • ImageNet: 75.80% accuracy: mixed-precision ResNet50 reaches this at 12.17MB, 0.82% higher than OMSE with training data and 5.74% higher than zero-shot OMSE.
  • ImageNet: 1.71% higher accuracy: ZEROQ outperforms DFQ on MobileNetV2, while W8A8 remains within 0.12% of baseline accuracy.
  • Efficiency: Less than 30 seconds: end-to-end ResNet50 quantization runs in this time on eight Tesla V100 GPUs.
  • ImageNet: 58.96% accuracy: ShuffleNet reaches this at a 0.73MB model size with average 4-bit weights.
  • ImageNet: 69.05% accuracy: ZEROQ exceeds DFC's fine-tuned 68.05% result for ResNet18 W4A4 without fine-tuning.

B. Microsoft COCO

ZEROQ extends zero-shot quantization to RetinaNet on COCO, retaining detection performance at 8-bit precision and remaining competitive under lower-weight precision.

  • RetinaNet with a ResNet50 backbone provides a COCO baseline of 36.4 mAP before quantization.
  • ZEROQ produces no performance degradation for RetinaNet W8A8 on COCO.
  • 35.9 mAP: ZEROQ achieves this with W6A6 quantization on RetinaNet.
  • 33.7 mAP: average 4-bit weight quantization reaches this while retaining 8-bit activations.
  • The average 4-bit weight result is comparable to FQN, although FQN uses the full training dataset and fine-tuning.

V. ABLATION STUDY

The ablation study isolates ZEROQ's two components: Distilled Data for sensitivity and clipping analysis, and Pareto optimization for automatic bit assignment.

  • The ablation study examines Distilled Data generation and Pareto frontier optimization as separate ZEROQ components.
  • Distilled Data supports sensitivity analysis and determination of activation clipping ranges.
  • The Pareto frontier method automatically assigns bit precision across layers.

A. Distilled Data

ZEROQ’s Distilled Data approximates training-data sensitivity without using real data, supporting effective activation-range estimation and mixed-precision selection. Experiments show it narrows Gaussian-data performance gaps and improves configuration quality across tested models.

  • A. Distilled Data: 75.80% testing accuracy on ResNet50 with ZEROQ nearly matches the 75.95% achieved using training data and exceeds the 75.44% obtained with Gaussian data.
  • A. Distilled Data: 68.83% testing accuracy on MobileNetV2 is only 0.23% below training-data performance, while Gaussian data trails training data by 2.33%.
  • A. Distilled Data: ZEROQ compares Gaussian, training, and Distilled Data for sensitivity analysis and activation-range determination at matched 4-bit-weight and 8-bit-activation model sizes.The ablation uses ResNet50 and MobileNetV2.
  • A. Distilled Data: ZEROQ’s Pareto frontier selection produces significantly better testing performance than Inverse and Random bit configurations under average 4-bit weights and 8-bit activations.The comparison reports best, worst, mean, and variance across 20 tests.
  • A. Distilled Data: The best ZEROQ configuration outperforms the worst among the top-20 configurations by 0.18%, while ZEROQ-generated configurations show small variance.
  • A. Distilled Data: Across datasets and models, ZEROQ reports higher accuracy with the same or smaller model size than previous post-training quantization methods.Evaluations include ImageNet, CIFAR-10, and Microsoft COCO models.

APPENDIX

The appendix details a grouped approximation for Pareto-frontier sensitivity search and records additional evaluations. The method trades exhaustive bit-configuration enumeration for lower computational cost while retaining empirically good results.

  • APPENDIX: ZEROQ relaxes its layer-independence assumption by grouping layers and partitioning the model-size axis into intervals.Groups contain a layers, and the Pareto frontier is divided into b intervals.
  • APPENDIX: For each group, the method first ranks configurations using the independence assumption, then recomputes overall sensitivity without approximation for selected candidates.
  • APPENDIX: The grouped search costs (L/a)טt×b + m×L and typically uses ˜t=10, t=5, b=200, and a=5.
  • APPENDIX: The appendix reports additional ZEROQ results for ResNet20 on CIFAR-10 and for ResNet152, InceptionV3, and SqueezeNext on ImageNet.
  • APPENDIX: Table VIII extends the Distilled Data ablation to ResNet18, ShuffleNet, and SqueezeNext using quantized models with 4-bit weights.

D. Clipping

ZEROQ uses simple min–max clipping in its experiments, while noting that weight clipping can further improve performance at some additional computational cost.

  • D. Clipping: Quantization clips a single-precision tensor to [a,b] and maps that range to the integer interval [0, 2^k−1].
  • D. Clipping: The experiments set [a,b]=[min(z),max(z)] to evaluate ZEROQ without assistance from another technique or costly hyperparameter tuning.
  • D. Clipping: Weight clipping can further boost ZEROQ performance, although it introduces slightly higher computational overhead.
Loading 2001.00281v1…