Source-linked AI summary
Data-Free Network Quantization With Adversarial Knowledge Distillation
Yoojin Choi, Jihwan Choi, Mostafa El-Khamy, Jungwon Lee
TL;DR
The paper addresses model compression and quantization when original training data cannot be shared because of cost, privacy, and security constraints. It uses data-free adversarial knowledge distillation with synthetic adversarial samples, batch-normalization-statistics matching, and multiple generators or students. Experiments report state-of-the-art data-free compression and quantization, with quantization accuracy losses below 2% in evaluated settings.
Problem
Model compression normally assumes access to original training data, which may be unavailable because dataset sharing is costly or restricted by privacy and security concerns.
Method
The framework minimizes worst-case teacher–student output loss on generator-produced adversarial samples while matching teacher batch-normalization statistics to constrain synthetic data.
Results
The method achieves state-of-the-art data-free KD across the evaluated datasets and architectures, while data-free quantization incurs accuracy losses below 2% for the evaluated 4-bit/8-bit weight and 8-bit activation settings.
Takeaways & Limitations
The results support using synthetic data generated without original datasets for model compression and quantization across residual, wide residual, and MobileNet models.
Takeaways & Limitations
Prior adversarial generator-based KD can produce images far from the original data, which degrades KD performance; this motivates constraining the generator with batch-normalization statistics.
Abstract
from arXiv · showhide
Network quantization is an essential procedure in deep learning for development of efficient fixed-point inference models on mobile or edge platforms. However, as datasets grow larger and privacy regulations become stricter, data sharing for model compression gets more difficult and restricted. In this paper, we consider data-free network quantization with synthetic data. The synthetic data are generated from a generator, while no data are used in training the generator and in quantization. To this end, we propose data-free adversarial knowledge distillation, which minimizes the maximum distance between the outputs of the teacher and the (quantized) student for any adversarial samples from a generator. To generate adversarial samples similar to the original data, we additionally propose matching statistics from the batch normalization layers for generated data and the original data in the teacher. Furthermore, we show the gain of producing diverse adversarial samples by using multiple generators and multiple students. Our experiments show the state-of-the-art data-free model compression and quantization results for (wide) residual networks and MobileNet on SVHN, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. The accuracy losses compared to using the original datasets are shown to be very minimal.
1. Introduction
The paper addresses model compression when original training data cannot be accessed because of cost, privacy, and security constraints. It proposes data-free adversarial knowledge distillation using batch-normalization statistics, diverse generators, and multiple students, achieving state-of-the-art compression and quantization results.
- Model compression is needed for efficient inference on resource-limited mobile and edge platforms, where computational cost and memory requirements are critical.
- Existing compression methods generally assume access to original training data, conflicting with increasing dataset costs and privacy, security, and regulatory constraints.
- The proposed framework minimizes worst-case teacher–student loss adversarially when loss on original training data is unavailable.
- Batch-normalization statistics constrain the generator to produce synthetic samples resembling the original training data, while multiple generators and students increase sample diversity and improve adversarial KD.
- The scheme achieves state-of-the-art data-free KD on residual and wide residual networks across SVHN, CIFAR-10, CIFAR-100, and Tiny-ImageNet, while extending evaluation to data-free quantization.
2. Related work
Related work uses meta-data, generators, and adversarial learning to perform data-free knowledge distillation, while data-free quantization remains comparatively underexplored. The paper frames its approach as robust optimization constrained by teacher batch-normalization statistics and uses multiple generators or students for diversity.
- Prior data-free KD methods use meta-data, such as intermediate activation statistics, or generators optimized to produce synthetic samples for student training.
- Adversarial KD generates samples where teacher and student outputs poorly match, but prior approaches could produce images far from the original data and degrade KD performance.
- Data-free network quantization has few prior studies, with earlier weight equalization and bias correction methods not considering synthetic data or data-free quantization-aware training.
- The proposed formulation treats data-free KD as robust optimization, defining uncertainty from teacher batch-normalization statistics.
- Multiple generators and multiple students are motivated as a way to improve diversity, paralleling strategies used to address mode collapse in generative models.
3. Data-free model compression
The paper formulates knowledge distillation without access to the original training data by generating synthetic adversarial inputs and optimizing student-teacher agreement under worst-case loss. It constrains the generator with teacher-derived statistics and entropy objectives, and extends the framework with multiple generators and students for greater sample diversity.
- Knowledge distillation: The original KD objective uses labeled training data and combines ground-truth supervision with teacher-student distribution matching, whereas data-free KD omits the ground-truth term.The student is trained under guidance from a pretrained teacher, and the temperature parameter is omitted only for simplicity.
- Data-free adversarial KD: Data-free adversarial KD minimizes student-teacher divergence on worst-case synthetic samples produced by a generator when the training dataset is unavailable.The generator is optimized adversarially, while an auxiliary teacher-provided loss constrains its outputs toward useful data.
- Generator constraints: Batch-normalization statistics, instance entropy, and batch entropy constrain generated samples to resemble original data while remaining class-informative and diverse.The method matches stored and generated layer means and variances, minimizes per-sample teacher entropy, and encourages high entropy in batch-averaged predictions.
- Generator constraints: Adding the auxiliary generator loss avoids adversarial samples far from the original dataset, improving the distillation behavior over unconstrained adversarial matching.With α = 0, the scheme reduces to adversarial belief matching; the added constraint reduces fitting to bad examples.
- Multiple generators and multiple students: Multiple generators and/or students increase data-free KD student accuracy by promoting more diverse generated samples and parallel student training.The framework indexes students and generators jointly, and the paper motivates multiple generators as a way to cover more of the target data support.
4. Experiments
Experiments evaluate data-free adversarial knowledge distillation for model compression and quantization across multiple datasets and architectures. The method produces teacher-regularized synthetic data, achieves marginal accuracy losses in data-free quantization, and benefits from diversity-oriented design choices.
- 4.1. Data-free model compression: State-of-the-art data-free KD is achieved across SVHN, CIFAR-10, CIFAR-100, and Tiny-ImageNet for residual and wide residual networks, with marginal accuracy losses versus original-data training.The comparison is summarized in Table 2.
- 4.1. Data-free model compression: Teacher-regularized generators produce synthetic samples similar to the original SVHN, CIFAR-10, and CIFAR-100 datasets.Examples are shown for WRN40-2 to WRN16-1 and ResNet-34 to ResNet-18.
- 4.1. Data-free model compression: Matching batch-normalization statistics provides the major contribution among the three auxiliary generator-loss terms.The ablation compares batch-normalization statistics, instance categorical entropy, and batch categorical entropy.
- 4.1. Data-free model compression: Smaller α yields more diverse adversarial images, whereas larger α collapses each class toward one mode and causes over-fitting.The study reports over-fitting when α is too large, specifically α ∈ {10, 10^2}.
- 4.1. Data-free model compression: Using multiple generators and/or multiple students improves student accuracy in data-free KD.The comparison uses two generators and/or two identically structured students with different initialization.
- 4.2. Data-free network quantization: Data-free quantization has less than 2% accuracy loss for 4-bit/8-bit weights and 8-bit activations compared with original-data quantization schemes.Synthetic generator data provide representative activation data, and adversarial KD is applied within TensorFlow’s quantization-aware training framework.
- 4.2. Data-free network quantization: Using a similar alternative dataset gives less than 0.5% accuracy loss, whereas a mismatched dataset considerably degrades performance.The comparison includes CIFAR-100 substituted for CIFAR-10 as a similar dataset and SVHN substituted for CIFAR-10 as a mismatched dataset.
5. Conclusion
The paper proposes data-free adversarial knowledge distillation for network quantization and compression without using original data. Batch-normalization statistics constrain synthetic samples, producing strong compression results and quantized models with accuracy comparable to models fine-tuned using original datasets.
- Data-free adversarial knowledge distillation achieves state-of-the-art performance over existing data-free KD schemes across multiple datasets and model architectures.
- Matching batch-normalization statistics constrains the generator to produce synthetic samples similar to the original training data.
- Quantized models achieve comparable accuracy to models quantized and fine-tuned with the original training datasets.