PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Free Adversarial Distillation

Gongfan Fang, Jie Song, Chengchao Shen, Xinchao Wang, Da Chen, Mingli Song

arXiv:1912.11006v3cs.LGcs.CVstat.ML

TL;DR

The paper addresses knowledge distillation when original or alternative training data is unavailable. It proposes data-free adversarial distillation, which estimates and optimizes model discrepancy using generated hard samples. The method outperforms other data-free methods, is comparable to some data-driven approaches, and extends effectively to semantic segmentation.

  • Problem

    Existing knowledge-distillation methods generally require original or alternative data, but such data is often unavailable because of privacy, transmission, legal, domain, cost, or bias constraints.

  • Method

    Data-free adversarial distillation estimates an optimizable upper bound on teacher-student model discrepancy while a generator produces hard samples and the models reduce the discrepancy.

  • Results

    The method performs significantly better than data-free methods and achieves results comparable to some data-driven approaches.

  • Takeaways & Limitations

    The framework is applicable to classification and semantic segmentation, including as the first effective data-free method for semantic segmentation according to the paper.

  • Takeaways & Limitations

    Generating complicated samples remains difficult, and the authors propose introducing human priors to reduce useless search space.

Abstract

from arXiv · show

Knowledge Distillation (KD) has made remarkable progress in the last few years and become a popular paradigm for model compression and knowledge transfer. However, almost all existing KD algorithms are data-driven, i.e., relying on a large amount of original training data or alternative data, which is usually unavailable in real-world scenarios. In this paper, we devote ourselves to this challenging problem and propose a novel adversarial distillation mechanism to craft a compact student model without any real-world data. We introduce a model discrepancy to quantificationally measure the difference between student and teacher models and construct an optimizable upper bound. In our work, the student and the teacher jointly act the role of the discriminator to reduce this discrepancy, when a generator adversarially produces some "hard samples" to enlarge it. Extensive experiments demonstrate that the proposed data-free method yields comparable performance to existing data-driven methods. More strikingly, our approach can be directly extended to semantic segmentation, which is more complicated than classification, and our approach achieves state-of-the-art results. Code and pretrained models are available at https://github.com/VainF/Data-Free-Adversarial-Distillation.

1. Introduction

Knowledge distillation compresses pretrained models but usually requires original or alternative data, which may be unavailable, costly, or biased. The paper proposes data-free adversarial distillation, measuring model discrepancy and using adversarial training to generate hard samples; experiments report superiority to data-free methods and comparable results to data-driven approaches.

  • Data-driven knowledge distillation compresses pretrained teacher models by training lightweight students on original or alternative data.
  • Unavailable training data creates privacy, transmission, and legal barriers, while unknown domains, collection costs, and data bias complicate alternatives.
  • The proposed data-free adversarial framework adaptively customizes samples for teacher and student models while reducing a quantified model discrepancy.
  • The method introduces a quantitative measure of model discrepancy without real data and extends adversarial distillation to semantic segmentation.
  • The proposed method performs significantly better than data-free methods and achieves results comparable to some data-driven approaches.

2. Related Work

Related work separates knowledge distillation into data-driven and data-free approaches. Prior data-free methods reconstruct samples from metadata, optimize data impressions, or use generators, while GANs provide a related adversarial framework but face instability and mode collapse.

  • Knowledge distillation methods are categorized as data-driven or data-free according to whether they require training data.
  • Data-driven distillation uses real data and teacher supervision, including softened targets, intermediate representations, and attention matching.
  • Data-free distillation addresses inaccessible training data, but some methods still require metadata or construct samples from random noise and teacher outputs.
  • DAFL synthesizes training samples with a generator while using the fixed teacher as a discriminator to encourage activated representations and one-hot predictions.
  • GANs formulate generation as a discriminator-generator minimax game but can suffer from training instability and mode collapse.

3. Method

DFAD estimates teacher–student discrepancy using generated hard samples, then alternates student imitation with generator optimization to distill without real data.

  • 3.1. Discrepancy Estimation: DFAD approximates the intractable true discrepancy with generated samples and uses hard-sample discrepancy as an optimizable upper bound.The generator controls the sampling distribution, while hard samples produce larger teacher–student output differences than the full-space discrepancy.
  • 3.2. Adversarial Training: The teacher and student jointly form a discriminator, while the generator produces hard samples that enlarge their measured discrepancy.Adversarial training alternates discrepancy minimization by the student with discrepancy maximization by the generator.
  • 3.2.1. Imitation Stage: During imitation, generated samples are fed to both models, and the student minimizes their Mean Absolute Error discrepancy.Outputs are vectors for classification and matrices for semantic segmentation; the teacher remains fixed while the student updates.
  • 3.2.2. Generation Stage: The generation stage updates only the generator to create more confusing samples, using a logarithmic MAE objective to stabilize training.The log objective reduces generator gradients as discrepancy becomes large, limiting abnormal samples and drastic distribution changes.
  • 3.2.3. Optimization: The two stages repeat until student imitation and generator hard-sample production reach a balance, with training implemented as alternating optimization.Algorithm 1 initializes the student and generator, then repeats discriminator and generator updates while calculating model discrepancy.

4. Experiments

Experiments evaluate the method on classification and semantic segmentation using pretrained teacher models and compact students. The data-free approach achieves the highest accuracy among data-free classification methods and competitive or superior segmentation results without original training data.

  • Models and Datasets: Experiments cover classification on MNIST, CIFAR10, CIFAR100, and Caltech101, plus semantic segmentation on CamVid and NYUv2.The study uses pretrained teacher models and smaller student models across six datasets.
  • Evaluation: Evaluation uses prediction accuracy for classification and mean Intersection over Union (mIoU) for semantic segmentation.The metrics are computed on the respective test sets.
  • Classification Results: The data-free adversarial method achieves the highest accuracy among data-free methods and remains comparable to data-driven methods.On Caltech101, competing DAFL training fails under the tested batch size while the proposed method still learns a student model.
  • Classification Results: Generated samples need not be human-recognizable to support a comparable student model, indicating that realistic images are not required for distillation.Figure 3 compares generated samples with real images on MNIST, CIFAR10, and CIFAR100.
  • Loss Comparison: MAE provides the best performance among the tested loss candidates because it supplies stable gradients for the generator.The comparison includes MAE, MSE, KLD, and MSE+MAE on CIFAR10.
  • Segmentation Results: For segmentation, the method is competitive with KD-ORI on CamVid and surpasses KD-UNR and all data-free methods on NYUv2.The teacher is initialized from ImageNet, while student models are trained from scratch.
  • Segmentation Results: The generator captures spatial co-occurrences such as traffic lights and poles, and the resulting model reaches an mIoU of 0.460.These patterns provide evidence that generated samples encode task-relevant segmentation structure.

5. Conclusions

The paper presents a data-free adversarial distillation framework that reduces teacher–student discrepancy through an optimizable upper bound. Experiments show scalability across classification and segmentation, while complicated sample generation remains difficult.

  • The framework estimates an optimizable upper bound for teacher–student model discrepancy and reduces it without access to real data.
  • Experiments demonstrate that the framework is scalable across classification and segmentation and different network architectures.
  • Generating complicated samples remains difficult, motivating future use of human priors to reduce useless generator search.

Supplementary Material

The supplementary material provides implementation details, batch-size analyses, and additional visualizations of generated samples and segmentation results.

  • The supplement covers model architectures, implementation details, batch-size influence, and additional generated-sample and segmentation-result visualizations.

A. Model Architectures and Hyperparameters

This section summarizes dataset configurations and the experimental access conditions used for teachers, students, and generators.

  • Teachers are obtained from labeled data, whereas students and generators are trained without access to real-world data.
  • Models are validated every 50 iterations, with each validation period treated as an “epoch” for simplicity.

A.1. Generators

The experiments use two vanilla generator architectures, selecting between interpolation-based and deconvolution-based upsampling according to the dataset.

  • Generator-A uses nearest-neighbor interpolation, while Generator-B replaces interpolation with deconvolutions and follows the DCGAN generator design.
  • Generator-A is used for MNIST and CIFAR, whereas the more powerful Generator-B is applied to other datasets.

A.2. Teachers and Students

The appendix specifies student-training configurations for MNIST, CIFAR10, CIFAR100, and NYUv2, alongside generator and model architectures. Training schedules vary across datasets, including distinct learning rates, decay policies, weight decay, and epoch counts.

  • MNIST uses LeNet-5 and LeNet-5-Half architectures, with SGD at a fixed learning rate of 0.01 for 40 epochs.
  • CIFAR10 and CIFAR100 use modified ResNet architectures with 8× downsampling, trained for 500 epochs with staged learning-rate decay.The learning rate starts at 0.1 and is divided by 10 at 100 and 200 epochs, with weight decay 5e-4.
  • The generator architecture first projects the vector input to feature maps and then upsamples them to the required size.
  • NYUv2 training uses a learning rate of 0.05, weight decay 5e-5, and 300 epochs for both the student and generator.The learning rate is multiplied by 0.3 at 150 and 250 epochs.

B. Influence of Different Batch Sizes

The method depends on large batch sizes for generator training and accurate discrepancy estimation. Experiments report that small batches damage student performance, while larger batches improve stability and hard-sample generation.

  • Large batch sizes are required both to train the generator and to ensure accurate discrepancy estimation.
  • Small batch sizes injure student-model performance across classification and semantic-segmentation experiments.
  • Larger batches provide sufficient statistical information for hard-sample generation and make training more stable.

C. More Visualization

The visualizations compare generated and real samples across classification and segmentation datasets, and show segmentation predictions and results. The figures are intended to illustrate generated-sample content and segmentation performance.

  • Figures 8 and 9 compare generated samples with real samples on MNIST, CIFAR10, CIFAR100, and Caltech101.The text states that the fake samples cannot be recognized by humans but contain sufficient task knowledge.
  • Figure 10 shows generated samples, teacher predictions, and real samples from CamVid and NYUv2.
  • Figures 11 and 12 present segmentation results on CamVid and NYUv2, respectively, alongside the data settings used by KD-REL and KD-UNR.KD-REL uses Cityscapes for CamVid and SunRGBD for NYUv2; KD-UNR uses VOC2012 for both.
Loading 1912.11006v3…