PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Paul Micaelli, Amos Storkey

arXiv:1905.09768v4cs.LGstat.ML

TL;DR

The paper addresses knowledge transfer when the teacher’s training data are unavailable because datasets can be private, proprietary, very large, or transient. It trains an adversarial generator to find pseudo-images where student and teacher predictions disagree, then trains the student to match the teacher. The resulting zero-shot student performs close to full-data distillation on SVHN and improves prior few-shot results on CIFAR-10, while showing higher belief match near decision boundaries than a real-data-distilled student.

  • Problem

    Knowledge transfer and model compression commonly rely on training-distribution data, but pretrained teachers may be released without it for privacy, property, size, or transience reasons.

  • Method

    An adversarial generator produces pseudo data maximizing forward KL divergence between teacher and student predictions while the student is trained to minimize that divergence.

  • Results

    The method closely matches full-data distillation on SVHN, improves prior state-of-the-art few-shot distillation on CIFAR-10, and yields higher near-boundary belief matching than a KD+AT student.

  • Takeaways & Limitations

    Zero-shot knowledge transfer can produce useful students without data or metadata, supporting data-free transfer when private datasets become increasingly common.

  • Takeaways & Limitations

    Some teacher–student architecture pairs work better than others, and the factors determining a good match remain for future work.

Abstract

from arXiv · show

Performing knowledge transfer from a large teacher network to a smaller student is a popular task in modern deep learning applications. However, due to growing dataset sizes and stricter privacy regulations, it is increasingly common not to have access to the data that was used to train the teacher. We propose a novel method which trains a student to match the predictions of its teacher without using any data or metadata. We achieve this by training an adversarial generator to search for images on which the student poorly matches the teacher, and then using them to train the student. Our resulting student closely approximates its teacher for simple datasets like SVHN, and on CIFAR10 we improve on the state-of-the-art for few-shot distillation (with 100 images per class), despite using no data. Finally, we also propose a metric to quantify the degree of belief matching between teacher and student in the vicinity of decision boundaries, and observe a significantly higher match between our zero-shot student and the teacher, than between a student distilled with real data and the teacher. Code available at: https://github.com/polo5/ZeroShotKnowledgeTransfer

1 Introduction

Model compression methods generally depend on training-distribution data, but the authors propose zero-shot knowledge transfer when that data is unavailable. Their adversarial approach trains pseudo points to expose teacher–student disagreement while training the student to reduce it.

  • Large networks support modern deep-learning applications but create hardware and deployment constraints that motivate model compression.The cited compression approaches include knowledge distillation, pruning, and quantization.
  • Compression methods typically require labeled training-distribution data, which may be withheld because of privacy, property, size, or transience.Examples include confidential user images, proprietary internal datasets, very large image collections, and unavailable reinforcement-learning environments.
  • The paper proposes adversarial zero-shot knowledge transfer that distills a teacher into a smaller student without data or metadata.The contribution also includes evaluation on two datasets and a measure of belief matching near decision boundaries.
  • Pseudo points are initialized away from the data manifold, optimized to maximize student–teacher KL divergence, and used while the student learns the opposite objective.In the toy illustration, pseudo points explore decision-boundary regions and some regions away from them; after several steps, the boundaries become indistinguishable.

2 Related work

The related work situates zero-shot knowledge transfer among inducing-point methods, knowledge distillation, privacy attacks, zero-shot learning, and emerging zero- or few-shot distillation. The paper distinguishes its goal from reconstructing training data or classifying unseen semantic classes.

  • Inducing-point methods reduce inference cost with fewer points, while later variational approaches create pseudo data rather than selecting training subsets.Dataset distillation is related through its use of bi-level optimization to learn a small substitute dataset.
  • Knowledge distillation transfers network outputs to another model, with later work improving students or adding losses such as attention transfer.The paper uses zero-shot knowledge transfer to distinguish its task from the loss function commonly associated with Hinton et al.’s formulation.
  • Model extraction accesses black-box predictions to reproduce a model, whereas model inversion uses white-box weights to recreate training images.The paper instead seeks images useful for training whether or not they resemble the original training data.
  • Zero-shot learning usually combines labeled images with an intermediate semantic representation to classify classes unseen during training.In this paper, the teacher serves as the additional information, while no training-set samples are used.
  • Recent zero- and few-shot distillation work addresses how distillation performance changes with limited data, including adversarial pseudo-data approaches.The cited approaches differ in teacher data access, pruning and alignment, or variational information distillation.

3 Zero-shot knowledge transfer

The method alternates an adversarial generator that finds pseudo-images exposing teacher–student disagreement with student updates that match the teacher, using no real data or metadata. Forward KL and selective attention transfer support training, while pseudo-points tend to explore decision boundaries; the method reaches 83.69 +−0.58% without real data and 85.91 +−0.24% after fine-tuning with 100 images per class.

  • 3.1 Algorithm: The algorithm alternates generator updates maximizing DKL(T(xp) || S(xp)) with student updates matching the teacher on fixed pseudo-samples.It uses nG generator steps and nS student steps per iteration, typically with nS > nG.
  • 3.1 Algorithm: Forward KL encourages non-zero student probabilities across images, producing high entropy that makes the generator harder to fool.The authors report significant test-accuracy drops when replacing forward KL with reverse KL or Jensen–Shannon divergence.
  • 3.2 Extra loss functions: The student loss can include an attention-transfer term summed over selected layers, while attention is excluded from the generator loss because it makes fooling easier.β controls the attention-term weight, and the attention maps average squared channel activations spatially.
  • 3.2 Extra loss functions: Additional losses for sample diversity, consistency, or teacher and student entropy did not improve performance.The authors contrast this with competing methods that require hand-designed losses such as carbon copy memory replay or fidelity.
  • 3.3 Dynamics: Pseudo-points initialized away from the real data manifold typically explore input-space decision boundaries where student–teacher disagreement is greatest.The illustration uses two-layer MLPs and learns pseudo-points directly rather than through a generator.
  • 3.4 Potential conceptual concerns: 83.69 +−0.58% is achieved without real data, rising to 85.91 +−0.24% after fine-tuning with 100 images per class.Figure 2 evaluates a WRN-40-2 teacher and WRN-16-1 student on SVHN and CIFAR-10, comparing scratch training, KD+AT, and Ahn et al. (2019).

4 Experiments

Experiments evaluate zero-shot and few-shot transfer on CIFAR-10 and SVHN, including robustness across architectures and analysis of pseudo-data and belief matching near decision boundaries.

  • CIFAR-10 and SVHN: 83.69 ± 0.58% test accuracy is achieved on CIFAR-10 without data, 2% above VID using 100 images per class.Finetuning with 100 images per class raises accuracy to 85.91 ± 0.24%, exceeding the previous few-shot state of the art by more than 4%.
  • CIFAR-10 and SVHN: 94.06 ± 0.27% test accuracy is obtained on SVHN using CIFAR-10-tuned settings, close to 95.88 ± 0.15% from full-data KD+AT.Dataset-specific hyperparameter tuning makes zero-shot performance on par with full-data distillation.
  • CIFAR-10 and SVHN: Zero-shot performance is on par with KD+AT using 200 images per class across CIFAR-10 WRN teacher-student pairs.The comparison spans varied network depths and widths, with results reported over three seeds.
  • Architecture dependence: Some teacher-student pairs work better than others, and deeper students with more parameters do not necessarily improve zero-shot transfer.The specific factors determining a good match remain open for future work.
  • Nature of the pseudo data: Generator samples evolve from coarse, diverse textures to complex high-frequency patterns, with most images becoming humanly meaningless after about 10% of training.The generator’s pseudo-data distribution is analyzed across training, seeds, and hyperparameters.
  • Nature of the pseudo data: Pseudo-data concentrates near student decision boundaries: teacher confidence averages about 0.8 while student confidence averages around 0.3, with classes near-uniformly represented.The generator seeks samples that make the teacher less predictable and expose student-teacher disagreement.
  • Measuring belief match near decision boundaries: Transition curves measure how moving an image from class i to j according to one network also moves it from i to j according to another.Figure 4 compares transitions targeting zero-shot versus normally distilled student boundaries.
  • Measuring belief match near decision boundaries: 0.09 probability disparity is the zero-shot student’s average MTE with the teacher on SVHN.The zero-shot student matches teacher transition curves more closely than the KD+AT student on both SVHN and CIFAR-10; MTE magnitudes depend on K and ξ.

5 Conclusion

The paper demonstrates data-free knowledge transfer through adversarial generator–student training. It reports performance close to full-data distillation on SVHN and improved zero- and few-shot results on CIFAR-10.

  • Adversarial training produces pseudo-images where the student disagrees with the teacher, then trains the student to match the teacher.
  • On SVHN, students achieve performance close to distillation with the full training set, even when that training set is large.
  • On CIFAR-10, the method obtains compelling zero- and few-shot distillation results that significantly improve on the previous state of the art.

6 Appendix

The appendix reports that additional generator losses reduced student performance, while prediction distributions on uniform noise concentrate mainly on two classes in CIFAR-10 networks.

  • Additional generator loss terms reduced student performance despite tuning their scaling factors.The tested terms targeted generator behavior beyond the plain adversarial model.
  • The authors attribute the loss-term degradation to imbalance between the generator and student adversaries.
  • The appendix describes teacher-confidence, augmentation-consistency, and image-diversity losses as attempted generator objectives.
  • On CIFAR-10, predictions from common networks on 1000 uniform-noise images concentrate mostly on birds or frogs.The authors suggest this reflects unequal class volumes and higher decision-boundary density near real images.
Loading 1905.09768v4…