PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model

Zi Wang

arXiv:2106.03310v1cs.LGcs.AIcs.CV

TL;DR

Knowledge distillation commonly relies on training samples, teacher parameters, and probability outputs, which may be unavailable in practice. The paper proposes DB3KD to distill from hard-label-only teachers using decision-boundary distances, and extends it to pseudo samples when training data are inaccessible. Experiments report competitive performance against existing KD methods in more relaxed scenarios.

  • Problem

    Knowledge distillation often requires training data, access to teacher parameters, and probability outputs, while practical settings may provide only hard-label predictions or no training data.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    DB3KD derives soft labels from sample robustness measured by distances to teacher decision boundaries, and ZSDB3KD generates robust pseudo samples when training data are unavailable.

    Source: §1 · p. 2

  • Results

    The proposed approaches achieve competitive performance compared to existing KD methods in more relaxed scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study demonstrates KD from a hard-label-only black-box teacher with and without access to the training set.

    Source: §1 · p. 2, §5 · p. 9

    With training samples unavailable, pseudo-sample generation requires substantial computing resources and more iterations than generation from a white-box teacher, making large-scale zero-shot KD difficult.

    Source: §5 · p. 9

Abstract

from arXiv · show

Knowledge distillation (KD) is a successful approach for deep neural network acceleration, with which a compact network (student) is trained by mimicking the softmax output of a pre-trained high-capacity network (teacher). In tradition, KD usually relies on access to the training samples and the parameters of the white-box teacher to acquire the transferred knowledge. However, these prerequisites are not always realistic due to storage costs or privacy issues in real-world applications. Here we propose the concept of decision-based black-box (DB3) knowledge distillation, with which the student is trained by distilling the knowledge from a black-box teacher (parameters are not accessible) that only returns classes rather than softmax outputs. We start with the scenario when the training set is accessible. We represent a sample's robustness against other classes by computing its distances to the teacher's decision boundaries and use it to construct the soft label for each training sample. After that, the student can be trained via standard KD. We then extend this approach to a more challenging scenario in which even accessing the training data is not feasible. We propose to generate pseudo samples distinguished by the teacher's decision boundaries to the largest extent and construct soft labels for them, which are used as the transfer set. We evaluate our approaches on various benchmark networks and datasets and experiment results demonstrate their effectiveness. Codes are available at: https://github.com/zwang84/zsdb3kd.

1. Introduction

Knowledge distillation typically depends on training data, teacher parameters, and probability outputs, although real deployments may expose only hard labels or lack access to the data. This work introduces DB3KD, which derives soft labels from decision-boundary robustness, and extends it to pseudo samples when training data are unavailable.

1. Introduction

  • With training data available, DB3KD uses each sample’s distance to targeted decision boundaries as robustness to construct soft labels for student training.The approach treats boundary distance as a measure of class confidence and trains the student via standard KD.Source: §1 · p. 2
  • Without training data, ZSDB3KD iteratively pushes random noise away from teacher decision boundaries to generate robust pseudo samples for student training.The pseudo samples are intended to simulate the training-sample distribution and serve as the transfer set.Source: §1 · p. 2
  • DB3KD transfers knowledge from a teacher that returns only hard labels, addressing settings where teacher parameters or training data are inaccessible.The introduction identifies storage costs and privacy as reasons training data may be unavailable, and notes that a teacher’s parameters may remain proprietary.Source: §1 · pp. 1–2
  • The authors report competitive performance against existing KD methods in more relaxed scenarios across benchmark networks and datasets.The paper frames this as the first study of KD from a decision-based black-box teacher, both with and without training-set access.Source: §1 · p. 2

2. Related Work

Prior KD work reduces data dependence through limited-data or zero-shot methods, but the described zero-shot approaches still require teacher parameters. This work differs by considering distillation from a teacher that returns only top-1 classes.

Knowledge distillation

  • Other KD methods transfer class probabilities or add intermediate-feature and attention alignments to train compact students.FitNet matches intermediate hidden-layer values, while attention transfer uses DNN attention as additional knowledge.Source: §2 · p. 2

Knowledge distillation with limited data

  • Few-shot KD reduces storage and transmission needs by generating pseudo samples with a small number of original training samples.A metadata-based alternative uses teacher activation statistics, though releasing such metadata with the teacher is described as uncommon.Source: §2 · p. 2

Zero-shot knowledge distillation

  • Zero-shot KD methods generate transfer samples from modeled teacher outputs or other cues, but the cited approaches require teacher parameters for backpropagation.Examples include Dirichlet or multivariate-normal output modeling, GAN-based synthesis, and batch-normalization features.Source: §2 · pp. 2–3

Black-box knowledge distillation

  • Earlier black-box KD approaches relaxed some prerequisites but still required assumptions about training samples and a teacher returning class probabilities.The paper contrasts this setting with its focus on teachers that return only top-1 classes.Source: §2 · p. 3

Decision-based adversarial attack

  • Decision-based adversarial attacks locate nearby boundary points to create high-confidence adversarial examples, inspiring the paper’s use of boundary distance as robustness.The cited attacks seek imperceptible perturbations that fool a trained DNN.Source: §2 · p. 3

3. Methodology

The method adapts KD to a teacher that returns only hard labels, using decision-boundary distances to estimate soft labels when training data are available and boundary-guided pseudo samples when they are not.

3.1. Knowledge Distillation

  • Standard KD trains a compact student to match the softened outputs of a pre-trained teacher using a cross-entropy term and a weighted distillation loss.Temperature τ softens the softmax outputs; λ balances the two losses.Source: §3.1 · p. 3

3.2. Decision-Based Black-Box Knowledge Distillation

  • DB3KD estimates class probabilities from sample robustness against each class, measured through a teacher that returns only top-1 labels.A sample closer to a class region is more vulnerable to that class and receives higher confidence.Source: §3.2 · p. 3

3.2.1. SAMPLE ROBUSTNESS

  • Sample robustness can use minimal ℓ2 distance to other-class samples (SD), targeted teacher-boundary distance (BD), or optimized minimal boundary distance (MBD).SD is a rough estimate because it does not use teacher queries; MBD optimizes the boundary point through iterative gradient estimation and binary search.Source: §3.2.1 · pp. 3–4

3.2.2. SOFT LABEL CONSTRUCTION

  • Soft-label construction makes the ground-truth class activation largest and assigns lower confidence to classes against which the sample is more robust.The total distance to targeted boundaries controls concentration: larger summed distances concentrate probability mass in the top-1 class.Source: §3.2.2 · p. 5

3.2.3. TRAINING OF STUDENT MODEL

  • After constructing soft labels for training samples, the student is trained with the standard KD objective.The objective combines cross-entropy on ground-truth labels with a distillation loss weighted by λ.Source: §3.1 · p. 3, §3.2.3 · p. 5

3.3. Zero-shot Decision-Based Black-Box Knowledge Distillation

  • ZSDB3KD pushes random noise away from the teacher’s decision boundary to simulate training-data distribution, then uses the pseudo samples as a transfer set.The process estimates the boundary gradient and repeats updates until the distance cannot be further maximized or the query limit is reached.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate DB3KD with training samples available and ZSDB3KD when they are unavailable, across multiple network and dataset settings. Both approaches perform competitively, while analyses examine query efficiency, boundary-distance labels, and pseudo-sample generation.

4. Experiments

  • The experiments cover DB3KD with accessible training samples and ZSDB3KD without them, using benchmark networks and datasets.DB3KD setup includes LeNet, AlexNet, and ResNet teacher-student configurations across MNIST, Fashion-MNIST, CIFAR-10, and FLOWERS102.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2. Performance Evaluation of DB3KD

  • DB3KD with MBD outperforms standard KD on every reported experiment except FLOWERS102, where it remains ahead of surrogate KD.The simpler SD strategy is also competitive with standard KD across the experiments and outperforms surrogate KD.Source: §4.2 · p. 6, Table 1 · p. 6

4.3. Comparison with Self-Distillation Approaches

  • On CIFAR-100, the student achieves 77.31% test accuracy with the same network configuration, outperforming self-distillation approaches; replacing the DB3 teacher with ResNet-50 under unchanged other configurations further improves performance by 1.34%.Source: Table 2 · p. 6, §4.3 · p. 7

4.4. Ablation Studies and Analyses of DB3KD

  • Student performance is satisfactory across query counts from 2k to 20k, and even binary search with around 100 queries yields satisfactory performance.More queries bring slight gains, especially for deeper networks and FLOWERS102, while normalized inter-class robustness ratios remain stable as query counts change.Source: §4.4 · p. 7, Figure 4 · p. 7
  • Minimal boundary distance is a robust measure of sample fragility, and smaller distances align with semantically closer classes across MNIST, Fashion-MNIST, and CIFAR-10.The reported examples include digit 8 being closer to 9 than 1, T-shirt closer to shirt than sneaker, and dog closer to cat than truck.Source: §4.4 · pp. 7–8, Figure 5 · p. 8

4.5. Experiment Setup of ZSDB3KD

  • The ZSDB3KD evaluation uses LeNet-5 and AlexNet teacher-student settings, generates 8,000 pseudo-samples per class, and compares against several KD approaches.The setup limits MBD searches to 5,000 queries and applies data augmentation to enrich the transfer set.Source: §4.5 · p. 8

4.6. Performance Comparison of ZSDB3KD

  • ZSDB3KD achieves 96.54% on MNIST, 72.31% on Fashion-MNIST, and 59.46% on CIFAR-10 without training samples or teacher softmax outputs.The MNIST and Fashion-MNIST accuracies are described as close to KD approaches with more relaxed access; random-noise inputs perform poorly with a DB3 teacher.Source: §4.6 · pp. 8–9, Table 3 · p. 8, Table 4 · p. 8

4.7. Ablation Studies and Analyses of ZSDB3KD

  • With more sample-generation iterations, MNIST pseudo-samples become sharper and more recognizable as student performance improves.At 1k samples per class, the student reaches 94% test accuracy; at 8k per class, performance saturates and is comparable to standard KD. Averaged generated samples are reported to have satisfactory quality relative to ZSKD and DAFL.Source: §4.7 · p. 9, Figure 6 · p. 9

5. Conclusion

The conclusion introduces DB3KD for distillation from decision-based black-box teachers and extends it to ZSDB3KD when training data are unavailable. It reports competitive DB3KD performance on FLOWERS102 while noting substantial compute demands for zero-shot scaling.

5. Conclusion

  • DB3KD distills from a decision-based black-box teacher by using sample robustness to construct training-sample soft labels through iterative teacher queries.The conclusion presents ZSDB3KD as the extension for cases where the training set is inaccessible.Source: §5 · p. 9
  • With training data available, DB3KD achieved competitive performance on FLOWERS102, whose samples largely overlap with ImageNet.The authors state that they believe DB3KD can work effectively on large-scale datasets.Source: §5 · p. 9
  • Zero-shot KD is hard to scale to large datasets because pseudo-sample generation requires substantial computing resources, and DB3 teachers require more iterations than white-box models.The authors describe their approach as the first principled solution and hope it draws attention to this research area.Source: §5 · p. 9

A. Architectures Used in DB3KD and ZSDB3KD Experiments

The experiments use LeNet and AlexNet teacher–student pairs for both DB3KD and ZSDB3KD, with additional student architectures and a ResNet evaluation for DB3KD.

A. Architectures Used in DB3KD and ZSDB3KD Experiments

B.1. Training of the Models with Cross-Entropy

This subsection gives cross-entropy training settings for the pre-trained DB3 teachers and student models trained solely with cross-entropy.

B.1. Training of the Models with Cross-Entropy

  • LeNet-5 on MNIST and Fashion-MNIST is trained for 200 epochs with batch size 1024 and Adam at learning rate 0.001; CE-trained students use the same settings.Source: §B.1 · p. 14
  • AlexNet on CIFAR-10 is trained for 300 epochs with batch size 1024 and SGD, momentum 0.9, weight decay 0.0001, and learning rate 0.1 divided by 10 at epochs 60, 120, and 180; CE-trained students use the same settings.Source: §B.1 · p. 14
  • ResNet-50 and ResNet-34 on CIFAR-100 are trained for 300 epochs with batch size 256 and SGD, momentum 0.9, weight decay 0.0001, and learning rate 0.1 divided by 10 at epochs 60, 120, and 180; the ResNet-18 student uses the same settings.Source: §B.1 · p. 14
  • For FLOWERS102, ImageNet-pretrained ResNet-34 is fine-tuned for 200 epochs with SGD, batch size 64, momentum 0.9, and learning rates 0.01, 0.005, and 0.001 at the start and epochs 60 and 100; the ResNet-18 student uses the same settings.Source: §B.1 · p. 14

B.2. Standard Knowledge Distillation Training Details

The standard KD student-training setup uses a fixed loss balance, temperature, optimizer, and training duration, with batch sizes and learning rates varying by dataset.

B.2. Standard Knowledge Distillation Training Details

  • Standard KD uses λ=1, Adam, 200 epochs, and temperature 20; batch size is 512 for MNIST, Fashion-MNIST, and CIFAR-10, and 64 for CIFAR-100 and FLOWERS102.Learning rates are 0.001 for MNIST and Fashion-MNIST, 0.005 for CIFAR-10/100, and 0.0005 for FLOWERS102.Source: §B.2 · p. 14

B.3. Surrogate Knowledge Distillation Training Details

Surrogate knowledge-distillation students are trained for 300 epochs, with dataset-specific settings for the scaling factor, temperature, and learning rate.

B.3. Surrogate Knowledge Distillation Training Details

  • All experiments train student models for 300 epochs.The optimizer and batch size for surrogate KD match standard KD.Source: §B.3 · p. 14
  • MNIST and Fashion-MNIST use λ=0.7, temperature 3, and learning rate 0.005.Source: §B.3 · p. 14
  • CIFAR-10/100 use λ=0.5, temperature 5, and learning rate 0.005; FLOWERS102 uses λ=1, temperature 10, and learning rate 0.001.Source: §B.3 · p. 14

B.4. Data Augmentation Used in ZSDB3KD Experiments

The ZSDB3KD transfer set is augmented with image transformations, and samples whose top-1 class changes are reassigned to the original class through the decision-boundary procedure.

B.4. Data Augmentation Used in ZSDB3KD Experiments

  • Augmentation includes padding-and-crop, horizontal and vertical flips, rotations, and flips or rotations applied after padding-and-crop.Rotations span −15° to 15° in 5° intervals.Source: §B.4 · pp. 14–15
  • MNIST and Fashion-MNIST use padding-and-crop and flips, while CIFAR-10 and the CIFAR-100 DB3KD experiment use all five strategies.Source: §B.4 · p. 15
  • If augmentation changes a sample’s top-1 class, the decision-boundary procedure finds its corresponding point x* to restore the original top-1 class.Source: §B.4 · p. 15
  • Adding augmented samples to ZSDB3KD training improves student performance (Table S7).Table S7 compares experiments with and without augmentation.Source: §B.4 · p. 15, Table S7 · p. 15

C.1. Comparison of the Sample Robustness Computed with DB3KD and the Logits Generated by the Teacher

Heatmaps based on teacher softmax outputs resemble those based on DB3 sample robustness, supporting the paper’s robustness-based soft-label construction.

C.1. Comparison of the Sample Robustness Computed with DB3KD and the Logits Generated by the Teacher

  • Softmax-derived distance heatmaps resemble those from DB3 sample robustness, supporting the proposed soft-label construction.The paper represents distance to a class as 1 minus that class’s probability.Source: §C.1 · p. 15
  • Both MNIST visualizations indicate that digit 4 is close to digit 9.Source: §C.1 · p. 15
  • For Fashion-MNIST, both visualizations place T-shirt near Shirt and Pullover; in Figure S1, darker colors indicate smaller class distances.Source: §C.1 · p. 15, Figure S1 · p. 15

C.2. Ablation Studies of ZSDB3KD on Fashion-MNIST and CIFAR-10

The ablations show higher student performance with more sample-optimization iterations and larger pseudo-sample transfer sets on Fashion-MNIST and CIFAR-10.

C.2. Ablation Studies of ZSDB3KD on Fashion-MNIST and CIFAR-10

  • Fashion-MNIST performance rises by around 7% after 40 optimization iterations from less than 58% at 5 iterations; CIFAR-10 rises from around 15% at 10 iterations to 37% at 70.The study does not use data augmentation.Source: §C.2 · p. 16, Figure S2 · p. 16, Figure S3 · p. 16
  • Fashion-MNIST test accuracy increases from 61.39% to 67.24% as pseudo samples grow from 1,000 to 8,000 per category; CIFAR-10 accuracy rises from less than 28% with 1,000 per class to 37.58% with 8,000.Source: §C.2 · p. 16, Figure S2 · p. 16, Figure S3 · p. 16
Loading 2106.03310v1…