PaperlayerView exact revision on arXiv

Source-linked AI summary

Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization

Haotong Qin, Yifu Ding, Xiangguo Zhang, Jiakai Wang, Xianglong Liu, Jiwen Lu

arXiv:2109.00212v3cs.CV

TL;DR

Generative data-free quantization avoids requiring real data, but synthetic-data homogenization accompanies a performance gap from data-driven quantization. DSG diversifies generated samples through relaxed BN-statistics alignment and sample-correlation inhibition, and reports strong results across quantization methods and architectures.

  • Problem

    Data-free quantization avoids real-data access, but synthetic-data homogenization and a performance gap from data-driven quantization remain concerns.

    Source: §1 · pp. 1–2, §3.3 · p. 5

  • Method

    DSG combines Slack Distribution Alignment, Layerwise Sample Enhancement, and Sample Correlation Inhibition to diversify synthetic data at distribution and sample levels.

    Source: §1 · p. 3

  • Results

    DSG significantly improves data-free PTQ and QAT across mainstream architectures, surpasses existing data-free methods on large-scale image classification, and reports state-of-the-art results.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The proposed data-diversification scheme applies to both data-free PTQ and QAT.

    Source: §1 · p. 3

    The SDA reference assumes the whole potential input set is generally Gaussian, motivated by the large-sample central limit theorem.

    Source: §3.4.1 · p. 6

Abstract

from arXiv · show

Generative data-free quantization emerges as a practical compression approach that quantizes deep neural networks to low bit-width without accessing the real data. This approach generates data utilizing batch normalization (BN) statistics of the full-precision networks to quantize the networks. However, it always faces the serious challenges of accuracy degradation in practice. We first give a theoretical analysis that the diversity of synthetic samples is crucial for the data-free quantization, while in existing approaches, the synthetic data completely constrained by BN statistics experimentally exhibit severe homogenization at distribution and sample levels. This paper presents a generic Diverse Sample Generation (DSG) scheme for the generative data-free quantization, to mitigate detrimental homogenization. We first slack the statistics alignment for features in the BN layer to relax the distribution constraint. Then, we strengthen the loss impact of the specific BN layers for different samples and inhibit the correlation among samples in the generation process, to diversify samples from the statistical and spatial perspectives, respectively. Comprehensive experiments show that for large-scale image classification tasks, our DSG can consistently quantization performance on different neural architectures, especially under ultra-low bit-width. And data diversification caused by our DSG brings a general gain to various quantization-aware training and post-training quantization approaches, demonstrating its generality and effectiveness.

1 INTRODUCTION

Generative data-free quantization uses BN statistics to synthesize data, but existing methods exhibit homogenization at distribution and sample levels. DSG diversifies those samples and improves data-free PTQ and QAT.

1 INTRODUCTION

  • Data-free quantization addresses cases where real data is inaccessible, while generative methods use synthetic data matched to full-precision networks’ BN statistics.The introduction distinguishes PTQ from QAT by whether training or finetuning is required.Source: §1 · p. 1
  • Existing generative methods show homogenization in overall distributions and among samples, with sample-level homogenization appearing in both statistics and spatial arrangement.The paper cautions that mitigating homogenization at only one level does not ensure diversity at the other.Source: §1 · pp. 2–3
  • DSG significantly improves data-free PTQ and QAT across mainstream architectures, surpasses existing data-free methods, and sometimes outperforms real-data quantization.The reported evaluations cover VGG16bn, ResNet18/20/50, SqueezeNext, InceptionV3, ShuffleNet, and MobileNetV2.Source: §1 · p. 3
  • The theoretical analysis argues that synthetic samples dispersed within given constraints help quantization cover more of its potential input domain.The paper contrasts this diversity requirement with synthetic data generated under constraints that can make diversity difficult to achieve.Source: §1 · pp. 2–3
  • DSG combines Slack Distribution Alignment, Layerwise Sample Enhancement, and Sample Correlation Inhibition to diversify synthetic data at distribution and sample levels.SDA relaxes BN feature-statistics alignment; LSE strengthens a specific BN layer’s loss for its corresponding sample; SCI applies determinantal point-process loss to intermediate features.Source: §1 · pp. 2–3
  • This work extends conference paper [56] from a PTQ special case to a scheme for both PTQ and QAT, adding SCI for spatial sample-level homogenization.The paper also reports evaluations across more neural architectures and generative data-free quantization methods.Source: §1 · p. 3

2 RELATED WORK

Prior quantization work includes data-driven methods that depend on datasets and data-free methods that reconstruct synthetic data from model information. DSG is presented as a generic scheme for diversifying synthetic data across data-free PTQ and QAT.

2 RELATED WORK

  • Data-driven quantization methods use training data, while data-free approaches address settings where datasets are unavailable for privacy or security reasons.Data-driven methods may improve accuracy through QAT or reduce training costs through PTQ, but still require access to data.Source: §2 · p. 3
  • Earlier data-free methods use BN statistics, generative modeling, or adversarial approaches to synthesize data for quantization.The cited approaches include BN-statistics-based methods [21], [5], [25], generative modeling [25], and adversarial methods, [35].Source: §2 · pp. 3–4
  • DSG combines Slack Distribution Alignment, Layerwise Sample Enhancement, and Sample Correlation Inhibition to diversify synthetic data for data-free quantization.SDA targets distribution-level diversity, while LSE and SCI target sample-level diversity.Source: Figure 3 · p. 4

3 METHODS

The methods section frames generative data-free quantization around synthetic data, a fixed full-precision model, and an optimizable quantized model. It argues that diverse synthetic samples better represent an unknown input domain and support quantization.

3.1 Preliminaries

  • Generative data-free PTQ optimizes synthetic data using BN-statistics loss, then uses the data to calibrate the quantized model.In QAT, a generator produces synthetic data used to train the quantized network, typically alongside BN-statistics and generator cross-entropy losses.Source: §3.1 · p. 4

3.2 Diversity of Synthetic Data for Quantization

  • The paper abstracts generative data-free quantization as jointly optimizing synthetic data and a quantized model under data-generation and quantization constraints.The full-precision model is fixed, while synthetic data and the quantized model are optimizable; the synthetic data should reflect the input domain.Source: §3.2 · p. 5
  • The paper motivates diversity because synthetic samples must represent an unknown input domain whose subregions can differ and are not known in advance.The discussion models domains using independent subregions under low-density and smoothness assumptions.Source: §3.2 · p. 5
  • Theorem 1 states that samples dispersed across possible input-domain subregions, in proportion to their spatial scales, maximize expected representation of unknown domains.The proposed strategy is to disperse samples rather than let them collapse together.Source: §3.2 · p. 5

3.3 Homogenization of Synthetic Data

The examined data-free methods exhibit homogenization at distribution and sample levels, which the authors associate with reduced synthetic-data fidelity and quantization performance. DSG is proposed to address both levels through sample diversification.

3.3.1 Distribution-level Homogenization

  • Wasserstein distance is 0.120 for real data versus 0.040 for ZeroQ in PTQ, and 0.134 for real data versus 0.116 for GDFQ in QAT.The authors attribute the lower distances of the synthetic data to strict BN-statistics constraints that overfit layer statistics.Source: §3.3.1 · pp. 5–6

3.3.2 Sample-level Homogenization

  • PTQ sample-statistics variance is 0.029 for real data versus 0.009 for ZeroQ, while GDFQ samples are spatially concentrated with density index 2052 versus 1681 for real samples.The authors report DSG samples as more dispersed, and PTQ's spatial homogenization as less severe than QAT's but more severe than real data.Source: §3.3.2 · p. 6

3.3 Homogenization of Synthetic Data

  • The study identifies distribution- and sample-level homogenization in existing generative data-free methods and proposes DSG to diversify synthetic data and improve quantization.The analysis examines ZeroQ for PTQ and GDFQ for QAT.Source: §3.3 · p. 5, §3.3.2 · p. 6

3.4 Diverse Sample Generation

DSG applies three techniques to diversify synthetic data for data-free PTQ and QAT. They relax BN-statistics constraints and diversify samples through layer-specific losses and reduced sample correlation.

3.4 Diverse Sample Generation

  • DSG is a generic scheme for data-free PTQ and QAT that combines distribution-level relaxation with two forms of sample-level diversification.SDA relaxes statistical constraints; LSE and SCI target sample-level diversity from statistical and spatial perspectives, respectively.Source: §3.4 · p. 6

3.4.1 Slack Distribution Alignment

  • SDA uses relaxation margins for BN-layer means and standard deviations so synthetic features need not match BN statistics strictly.The authors state that relaxed constraints make generated samples more diverse and their feature distribution closer to real data.Source: §3.4.1 · p. 6

3.4.2 Layerwise Sample Enhancement

  • LSE assigns each sample an enhanced loss for a specific BN layer, so samples focus on different layer statistics during joint batch optimization.The described setup generates N samples for N BN layers.Source: §3.4.2 · p. 7

3.4.3 Sample Correlation Inhibition

  • SCI uses a determinant point-process loss to reduce correlation among intermediate synthetic-sample features relative to a random-vector diversity baseline.The random vectors are initialized independently from U[0, 1) and serve as a lower bound of diversity.Source: §3.4.3 · pp. 7–8

3.5 Specialization for different quantization approaches

DSG is specialized into generative data-free PTQ and QAT variants. PTQ optimizes synthetic data for calibration, whereas QAT alternates generator and quantized-network updates.

Data-free PTQ

  • For PTQ, DSG combines relaxed BN-statistics alignment (SDA), sample-specific loss emphasis (LSE), and intermediate-feature correlation reduction (SCI) to optimize synthetic data for calibration.After generation, the optimized data calibrates the quantized network in a separate process.Source: §3.5 · p. 8

Data-free QAT

  • For QAT, DSG alternately updates generator G and quantized network Q using generated samples, while keeping the pretrained full-precision network fixed.The generator starts from random noise and specified labels; its loss includes SCI terms for intermediate features across generator blocks.Source: §3.5 · p. 8
  • The QAT training data is used to train the quantized network toward the full-precision model's prediction probability distribution.The generator and quantized network are updated in alternating steps each epoch.Source: §3.5 · p. 8

3.6 Analysis and Discussion

DSG makes synthetic data less homogeneous in PTQ and QAT, with reported distribution and sample-space measures indicating greater diversity. The authors suggest this data may serve as an alternative across quantization approaches when real data are unavailable.

PTQ analysis

  • 0.124 vs. 0.040 Wasserstein distance between BN statistics and synthetic data (DSG vs. ZeroQ) accompanies more varied PTQ values and alleviated sample-level homogenization.DSG samples are more dispersed statistically than ZeroQ samples.Source: §3.6 · p. 8
  • 0.029 vs. 0.009 variance of statistics (DSG vs. ZeroQ) and center density indices of 1571, 1681, and 1902 (DSG, real data, and ZeroQ) indicate greater PTQ diversity.The DSG center density index is lower than both real data and ZeroQ.Source: §3.6 · p. 9

QAT analysis

  • In QAT, SDA and LSE relax statistical constraints, while SCI weakens sample correlation and the DSG density index is far lower than that of the existing method.The reported reduction indicates alleviated sample-level homogenization from the spatial perspective.Source: §3.6 · p. 9

Discussion

  • The authors suggest DSG-generated data may be an alternative for different quantization approaches when real data are inaccessible.Source: §3.6 · p. 9

4 EXPERIMENTS

Experiments evaluate DSG for image classification across varied architectures and bit widths in data-free PTQ and QAT, alongside studies of synthetic-data diversity and its use with other quantization methods. Across these settings, DSG generally outperforms comparison methods, with especially notable gains at low bit widths.

4.1.1 Effect of SDA

  • PTQ accuracy generally rises as ε increases from 0 to 0.9, but setting ε to 1 causes outlier-related drops.On ResNet18, PTQ rises from 26.04% to 33.39% and QAT from 60.52% to 61.08% over ε=0 to 0.9; the authors use ε=0.9 to balance divergence and outlier effects.Source: Figure 5 · p. 9, §4.1.1 · p. 10

4.1 Ablation Study

  • With all three techniques, DSG reaches 62.18% in data-free QAT and 13.86% in data-free PTQ on ResNet18 under W4A4.The ablations report SDA gains of 7.35% in PTQ and 0.56% in QAT, LSE gains of 1.08% in PTQ and 0.19% in QAT, and SCI gains of 13.49% in PTQ over pure random inputs and 0.40% in QAT over the vanilla method.Source: Table 1 · p. 9, §4.1.3 · p. 10, §4.1.1 · p. 10

4.2.1 Comparison with Data-free PTQ Methods

  • On CIFAR10, DSG PTQ exceeds existing state-of-the-art methods across settings and surpasses real-data calibration for both evaluated networks at W4A4.The W4A4 gains over real-data calibration are 0.41% for ResNet20 and 0.39% for VGG16bn.Source: §4.2.1 · pp. 10–11, Table 2 · p. 10

4.2 Comparison with SOTA Methods

  • DSG outperforms competing data-free PTQ and QAT methods across architectures and bit widths, with especially strong W4A4 results.For PTQ, DSG exceeds ZeroQ by 13.86% on ResNet18 and real-data calibration by 1.45%; for QAT, it exceeds GDFQ by 1.66% on ResNet18 and 16.31% on ResNet50.Source: Table 3 · p. 11, §4.2.1 · p. 12, Table 4 · p. 12

4.3.1 Analysis and Discussion for Data Diversity

  • SCI produces more spatially dispersed samples than GDFQ, with a feature-distance index that is smallest for DSG among the compared data.The reported index s increases with sample similarity; GDFQ has the largest s, while DSG has the smallest and appears closer to real data.Source: §4.3.1 · pp. 13–14, Figure 7 · p. 14

4.3.2 Evaluation with Calibration Methods

  • DSG improves results with multiple calibration methods and with DFQ, showing gains beyond its original calibration setting.It exceeds ZeroQ by 6.52%, 9.36%, and 0.61% with Percentile, EMA, and MSE calibration, respectively, and by 0.57% and 3.13% with DFQ at W6A6 and W8A8.Source: §4.3.2 · p. 14, Table 5 · p. 14, Table 6 · p. 14

4.3.3 Evaluation with Data-driven Quantization Methods

  • With AdaRound, DSG exceeds ZeroQ by 11.46% on ResNet18 and 34.33% on MobileNetV2 at 3-bit weight quantization.With 8-bit activations on ImageNet, DSG exceeds ZeroQ by 6.06% on ResNet18; adding labels and image priors together brings results close to real-data performance.Source: §4.3.3 · p. 15, Table 7 · p. 15

5 CONCLUSION

The paper analyzes how synthetic-sample diversity relates to data-free quantization and identifies homogenization in existing methods. It proposes DSG and reports gains across architectures, bit widths, and methods that integrate synthetic data.

5 CONCLUSION

  • The analysis identifies synthetic-sample diversity as crucial for data-free quantization and finds homogenization at distribution and sample levels in existing methods.The authors report DSG’s effectiveness across varied bit widths and architectures, particularly in ultra-low-bit settings.Source: §5 · p. 15
  • DSG addresses synthetic-data deficiencies through a diverse-sample generation scheme for generative data-free quantization.The conclusion describes improved performance when DSG-generated data is integrated with various methods.Source: §5 · p. 15
  • The reported results support diversity as an important property of high-quality synthetic data.The conclusion also reports notable accuracy improvements in ultra-low-bit cases such as W4A4.Source: §5 · p. 15

A.1 Proof of Lemma 1

The proof treats an input domain with at least two classes under low-density and smoothness assumptions. It argues that the domain must contain multiple high-density regions and at least one low-density region.

A.1 Proof of Lemma 1

  • Lemma 1 assumes a multi-class input domain and models it as independent high- and low-density subregions separated by possible decision surfaces.The lemma specifies at least one high-density region and allows zero or more low-density regions.Source: §A.1 · p. 18
  • The contradiction proof rules out the absence of low-density regions because decision surfaces can pass only through such regions under the low-density assumption.Without a low-density area, no decision surface could divide the domain into the at least two classes in the premise.Source: §A.1 · p. 18
  • The proof also rules out having no high-density region or only one, using the smoothness assumption to contradict the multi-class premise.With no high-density paths, samples cannot form the same class; with only one high-density region, only samples in that region can share a class.Source: §A.1 · p. 18

A.2 Proof of Theorem 1

The proof models a potential input domain as sub-regions and treats its density as uniform in expectation. It shows that equal probability across those regions maximizes the sample set’s information entropy.

A.2 Proof of Theorem 1

  • The potential domain X* uses K* = max(K0, K1, …) sub-regions, with empty regions added when an input domain has fewer regions.The proof treats the sub-regions as unordered and neglects differences within each sub-region.Source: §A.2 · p. 18
  • The proof assumes the expected scale of each potential-domain sub-region is V*/K*, treating X* as uniform in density in expectation.It expresses the information objective by considering the probabilities that independently sampled examples fall in each sub-region.Source: §A.2 · p. 18
  • Equal probability across the potential domain’s K* sub-regions maximizes the sample set’s information entropy.The proof states the maximizing probability as 1/K*.Source: §A.2 · p. 19

APPENDIX B VISUALIZATION

The appendix visualizes synthetic samples from ZeroQ, GDFQ, and DSG across PTQ and QAT. DSG’s samples show more diverse colors and textures than those from ZeroQ and GDFQ.

APPENDIX B VISUALIZATION

  • DSG produces synthetic samples with more diverse colors and textures than ZeroQ and GDFQ in the visualization.The authors present this visual comparison as further evidence that DSG diversifies synthetic samples.Source: Appendix B · p. 19, Figure 9 · p. 19
  • Figure 9 shows 10 synthetic examples each for ZeroQ, GDFQ, and DSG in PTQ and QAT approaches.Source: Figure 9 · p. 19
  • The visual comparison covers both PTQ and QAT approaches.Source: Figure 9 · p. 19
Loading 2109.00212v3…