PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Knowledge Distillation in Deep Networks

Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, R. Venkatesh Babu, Anirban Chakraborty

arXiv:1905.08114v1cs.LGcs.CVstat.ML

TL;DR

The paper addresses knowledge distillation when Teacher training data and metadata are inaccessible because of scale, privacy, or confidentiality constraints. It proposes ZSKD, which synthesizes Data Impressions from the Teacher using class similarities and Dirichlet modeling, and reports competitive performance with actual-data distillation across benchmark settings.

  • Problem

    Knowledge distillation typically relies on training or transfer data, but the Teacher’s training data may be inaccessible because it is large, proprietary, or sensitive.

  • Method

    ZSKD synthesizes Data Impressions from the Teacher without data samples or extracted metadata, using class similarities and Dirichlet modeling of the softmax space.

  • Results

    ZSKD achieves competitive performance with actual-data distillation across multiple benchmark datasets and model architectures.

  • Takeaways & Limitations

    The framework provides a data-free route to train a substitute Student model from a trained Teacher when the original training data is unavailable.

  • Takeaways & Limitations

    The method assumes that concentration parameters can encode meaningful class similarities, and the authors suggest multiple Teachers trained on different tasks could improve extraction.

Abstract

from arXiv · show

Knowledge distillation deals with the problem of training a smaller model (Student) from a high capacity source model (Teacher) so as to retain most of its performance. Existing approaches use either the training data or meta-data extracted from it in order to train the Student. However, accessing the dataset on which the Teacher has been trained may not always be feasible if the dataset is very large or it poses privacy or safety concerns (e.g., bio-metric or medical data). Hence, in this paper, we propose a novel data-free method to train the Student from the Teacher. Without even using any meta-data, we synthesize the Data Impressions from the complex Teacher model and utilize these as surrogates for the original training data samples to transfer its learning to Student via knowledge distillation. We, therefore, dub our method "Zero-Shot Knowledge Distillation" and demonstrate that our framework results in competitive generalization performance as achieved by distillation using the actual training data samples on multiple benchmark datasets.

1. Introduction

Knowledge distillation transfers a Teacher’s learned generalization to a smaller Student, but conventional methods depend on training or transfer data that may be inaccessible. ZSKD addresses this constraint by synthesizing Data Impressions from the Teacher without data samples or extracted metadata.

  • Motivation: Knowledge distillation transfers a complex Teacher’s learned mapping and dark knowledge to a smaller, faster Student by matching soft labels.Soft labels expose information in incorrect-class confidences, while high-temperature outputs provide non-peaky targets and lower-variance gradients.
  • Motivation: Original training data can be unavailable because datasets are large, proprietary, or sensitive, including biometric and healthcare data.The paper emphasizes that access to premium training data may not be realistic because of privacy, confidentiality, and competitive concerns.
  • Proposed Framework: ZSKD performs knowledge distillation without target samples, transfer-set samples, or extracted metadata by synthesizing pseudo-data from the Teacher.The synthesized samples serve as a transfer set for training the Student.
  • Proposed Framework: Data Impressions are crafted by modeling the Teacher’s softmax output space with a Dirichlet distribution and using Class Similarities as prior information.The method extracts class-similarity information from Teacher parameters and incorporates it into Dirichlet-based sampling.
  • Evaluation: ZSKD is evaluated empirically across multiple benchmark datasets and model architectures.The paper presents this evaluation as evidence for the effectiveness of the proposed approach.

2. Related Works

Prior knowledge-distillation methods use full or similar data, few original samples, or metadata derived from training data. ZSKD targets the stricter setting in which no training data is available in any form.

  • Data-Using Methods: Knowledge distillation methods transfer Teacher knowledge using entire training data or similar data, often matching logits or high-temperature soft targets.Soft targets preserve information in incorrect-class probabilities and can improve knowledge transfer relative to hard labels.
  • Few-Shot Methods: Few-shot approaches augment a small number of original training samples with pseudo-examples generated through adversarial iterative optimization.The cited procedure makes training more complicated.
  • Metadata-Based Methods: Metadata-based reconstruction stores activation records from the Teacher, but the metadata remains dependent on the original training samples.Therefore, this category is not completely data-free.
  • Zero-Shot Knowledge Distillation: ZSKD is presented as the first approach to demonstrate knowledge distillation when no training data is available in any form.The method adapts model-based sample extraction into pseudo-training data for distillation.

3. Proposed Method

The proposed method performs knowledge distillation without training data or extracted metadata by synthesizing Data Impressions from the Teacher’s learned parameters. It models Teacher softmax outputs using class similarities and Dirichlet sampling, then trains the Student to match Teacher predictions on the synthesized transfer set.

  • 3.1. Knowledge Distillation: Knowledge distillation transfers a complex Teacher’s soft-label information to a smaller Student, typically using original or alternative transfer data.The distillation loss compares Student and Teacher softmax outputs, while the conventional objective may also include cross-entropy on ground-truth labels.
  • 3.4. Zero-Shot Knowledge Distillation: Zero-Shot Knowledge Distillation synthesizes pseudo samples from the Teacher without accessing target data or metadata, using them as a transfer set.The method taps the Teacher’s learned parameters to produce Data Impressions representing the underlying training distribution.
  • 3.2. Modelling the Data in Softmax Space: The method models each class’s Teacher softmax outputs with a Dirichlet distribution whose concentration parameter reflects class similarities.For class k, the Dirichlet distribution is parameterized by K output categories and a class-specific concentration vector αk.
  • 3.2. Modelling the Data in Softmax Space: Class similarities are extracted from normalized weights connecting the pre-final layer to the final softmax layer, treating each class weight vector as a learned template.The resulting similarity matrix is row-normalized so its entries can serve as positive concentration-parameter values.
  • 3.3. Crafting Data Impressions via Dirichlet Sampling: Each sampled softmax vector is matched by optimizing a randomly initialized noisy image until the Teacher produces a similar output, creating a Data Impression.The process is repeated for sampled vectors across classes, after which the Data Impressions are used for Student training with only the distillation loss.
  • 3.3. Crafting Data Impressions via Dirichlet Sampling: Dirichlet samples are generated with p(s) = Dir(K, β × α), where β controls the distribution’s spread and the sampled vectors respect Teacher-learned class similarities.Changing α’s scale shifts sampling density between simplex edges and center regions.
  • 3.4. Zero-Shot Knowledge Distillation: The Student is trained on the Data Impressions by minimizing the distillation loss between Teacher and Student outputs.The proposed objective omits the cross-entropy term because it yields minor to no performance improvement and would introduce the λ hyper-parameter.

4. Experiments

Experiments evaluate ZSKD on MNIST, Fashion MNIST, and CIFAR-10 by generating Data Impressions from Teacher models and distilling lightweight Students. Across datasets, ZSKD approaches full-data distillation performance, while transfer-set size and impression type affect results.

  • Experimental setup: Experiments train Teachers on each dataset, extract Data Impressions from their softmax output spaces, and train lightweight Students on those impressions.The evaluation uses MNIST, Fashion MNIST, and CIFAR-10, with dataset-specific Teacher–Student architectures.
  • MNIST: ZSKD performs close to classical full-data distillation on MNIST while using 24000 Data Impressions, or 40% of the original training-set size.It also outperforms the cited few-data and data-free distillation approaches by a great margin.
  • Fashion MNIST: On Fashion MNIST, ZSKD outperforms the cited few-data approach by a large margin and performs close to classical distillation using all training samples.Table 2 reports the comparison with existing approaches.
  • CIFAR-10: On CIFAR-10, 40000 Data Impressions produce performance competitive with actual training samples, despite using a transfer set 20% smaller than the classical-distillation training set.The larger transfer set reflects the greater complexity of the target dataset.
  • Size of the Transfer Set: Student test performance generally increases with transfer-set size, but gains quickly saturate; simpler datasets require fewer Data Impressions than more complex datasets.The comparison uses equal percentages of Data Impressions or original training samples.
  • Class Versus Data Impressions: Dirichlet-modelled Data Impressions consistently outperform Class Impressions across three datasets, with larger transfer sets helping Data Impressions more.The comparison is shown without data augmentation during distillation.

5. Discussion and Conclusion

The paper presents ZSKD as a complete framework for distillation without data samples or extracted metadata, synthesizing Data Impressions from the Teacher's learned information. It also identifies stronger priors, task-driven extraction, and sample diversity as directions for improving the approach.

  • Discussion and Conclusion: ZSKD performs knowledge distillation without using data samples or metadata extracted from them.The framework models the data distribution in softmax space and extracts class-similarity information from the learned model.
  • Discussion and Conclusion: Data Impressions are synthesized samples evaluated as a transfer set for training a substitute model through distillation.The paper investigates their effectiveness for the downstream task of training a substitute model.
  • Discussion and Conclusion: The proposed Dirichlet modelling incorporates visual similarity among categories as a prior for more faithful extraction of underlying data patterns.The authors contrast this with prior extraction methods that do not use a strong data-distribution prior.
  • Discussion and Conclusion: The extracted samples are not task driven in the current setup, leaving scope for stronger priors and task-specific extraction.The authors suggest multiple Teachers trained on different tasks as one possible direction.
  • Discussion and Conclusion: Future work includes using multiple Teacher models and objectives that explicitly encourage diversity among extracted samples.These directions are proposed to improve extraction of data patterns and the resulting impressions.

1. Architecture Details Used in ZSKD

ZSKD uses paired Teacher and smaller Student architectures for MNIST, Fashion-MNIST, and CIFAR 10, with convolutional, pooling, normalization, fully connected, and dropout components detailed for the models.

  • MNIST and Fashion-MNIST: LeNet-5 serves as Teacher and LeNet-5-Half as Student for MNIST and Fashion-MNIST.The architecture section also identifies these models in the corresponding comparison table.
  • CIFAR 10: AlexNet serves as Teacher and AlexNet-Half as Student for CIFAR 10.The corresponding comparison table is listed for CIFAR 10.
  • Network layers: The detailed AlexNet architecture includes max-pooling, batch normalization, convolutional layers, flattening, fully connected layers, and dropout.The listed convolutional blocks use ReLU activations, while the fully connected block specifies a dropout rate of 0.5.
  • Distillation: Distillation divides logits by temperature 20 during training and by temperature 1 during testing.This temperature schedule is specified in the implementation notes.

2. Details of Hyperparameters Used in ZSKD

The experiments use TensorFlow and specify separate training settings for Teachers, Students, Data Impression generation, and Class Impression generation across LeNet and AlexNet configurations.

  • Implementation: The experiments are implemented using the TensorFlow framework.
  • LeNet settings: Teacher training with original data uses 200 epochs, batch size 512, learning rate 0.001, and Adam for the LeNet configuration.The corresponding original-data Student baselines use the same hyperparameters for cross-entropy training.
  • LeNet settings: Original-data knowledge distillation uses λ = 0.3 for cross-entropy, weight 1.0 for distillation loss, learning rate 0.01, and temperature 20.The remaining hyperparameters match the stated baseline settings.
  • Student training: Student training on Data Impressions uses Adam, batch size 512, and up to 2000 epochs, with learning rates selected by transfer-set size.The settings include additional fine-tuning on mixtures of Data Impressions and augmented samples.

3. Details on Augmentation

The augmentation pipeline creates varied Data Impression samples through scaling, translation, rotation, flipping, noise, and combinations of geometric operations, with RGB-specific noise processing for AlexNet impressions.

  • Geometric augmentation: Data Impressions are augmented using scaling, translation, rotation, flipping, and combinations of these operations.Scaling uses 90%, 75%, and 60% of the original size, while translation shifts samples by 20% in four directions.
  • Geometric augmentation: Rotation ranges from -90° to +90° and produces 10 rotated Data Impressions with 20° increments.
  • Combined augmentation: Combined augmentations include scaling with translation, translation with rotation, and scaling with rotation.
  • Noise augmentation: AlexNet Data Impressions additionally receive salt-and-pepper noise followed by Gaussian noise.These impressions contain RGB components, unlike the grayscale impressions obtained from LeNet.

Ablations: With and without Augmentation

Table 6 reports the performance of the proposed ZSKD framework. The supplied passage does not provide the table’s numerical results or explicitly describe augmentation comparisons.

  • Table 6 reports performance for the proposed ZSKD framework.
  • The table’s results are expressed in percentages.
  • The supplied caption does not specify how performance changes with augmentation.

Uniform Prior v/s Class Similarity Prior

Table 7 reports ZSKD performance using uniform and class similarity priors without augmentation. The supplied passage does not include the numerical comparison between these priors.

  • Table 7 compares proposed ZSKD performance using uniform and class similarity priors.
  • The reported prior comparison is conducted without augmentation.
  • Performance values in the table are reported as percentages.
Loading 1905.08114v1…