Source-linked AI summary
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary
Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi
TL;DR
Knowledge distillation still seeks effective ways to transfer a trained network’s knowledge, especially when improving a small student with a large teacher. This paper uses adversarial attacks to find boundary supporting samples near the teacher’s decision boundary and trains the student with them; experiments report improved distillation and stronger generalization with fewer training samples.
Problem
Existing knowledge-distillation methods do not explicitly address transferring decision-boundary information, despite the importance of decision boundaries to classifier generalization.
Method
The paper modifies an adversarial attack to find boundary supporting samples near a teacher’s decision boundary and uses them in a knowledge-distillation method.
Results
Experiments show that the proposed method improves knowledge distillation and provides stronger generalization performance with fewer training samples.
Takeaways & Limitations
Using adversarial samples to transfer decision-boundary information is presented as a new direction for knowledge distillation and adversarial-attack applications.
Takeaways & Limitations
The method assumes a sufficiently small η in its adversarial-attack formulation.
Abstract
from arXiv · showhide
Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier. The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation. To realize this goal, we utilize an adversarial attack to discover samples supporting a decision boundary. Based on this idea, to transfer more accurate information about the decision boundary, the proposed algorithm trains a student classifier based on the adversarial samples supporting the decision boundary. Experiments show that the proposed method indeed improves knowledge distillation and achieves the state-of-the-arts performance.
Introduction
This paper proposes improving knowledge distillation by transferring information about decision boundaries through adversarially generated samples. Experiments examine whether these boundary-supporting samples improve distillation and generalization, particularly with fewer training samples.
- Motivation: Knowledge distillation trains a new student network using an existing teacher network, often aiming to combine small-network speed with large-network performance.The paper focuses on enhancing a small network using a large teacher network.
- Research direction: Unlike prior approaches that mainly modify distillation losses or hidden representations, this paper investigates informative training samples for knowledge transfer.The paper frames sample manipulation as a different direction from manipulating network components or losses.
- Proposed method: An adversarial attack is modified to find boundary supporting samples, or BSSs, that support a teacher classifier’s decision boundary.The attack searches for small input modifications capable of moving samples beyond a nearby decision boundary.
- Proposed method: The proposed method adds BSS-based information to knowledge distillation to transfer more accurate decision-boundary information to the student.Figure 1 describes samples near the decision boundary as enabling more accurate knowledge transfer.
- Results: Experiments report that BSSs improve the Hinton, Vinyals, and Dean (2015) distillation scheme and yield better generalization performance with fewer training samples.The evaluation includes image classification and further experiments examining generalization.
Method
The method uses adversarial attacks to find boundary supporting samples (BSSs) near a teacher’s decision boundary, then adds a boundary supporting loss to knowledge distillation. BSS generation iteratively moves samples toward target classes, while target selection and training incorporate teacher information about difficult class boundaries.
- Boundary supporting samples: BSSs are adversarial samples near a teacher classifier’s decision boundary that encode both distance and path direction to the boundary.They are generated from training samples using gradient descent on classification scores.
- BSS generation: The BSS search initializes each base sample and iteratively updates it toward each target class using a loss that lowers the base-class score and raises the target-class score.The classifier determines the predicted class by the maximum classification score.
- BSS generation: The iterative attack moves samples in the loss-reducing gradient direction until the loss becomes negative, indicating that the sample has crossed the decision boundary.A learning rate controls the update step, while an additional term helps the loss become negative rather than merely converge to zero.
- BSS generation: The procedure accepts a sample after a successful boundary crossing and discards cases entering a nontarget class or exceeding the maximum-iteration condition.These stopping conditions prevent samples from being treated as BSSs when an intermediate class intervenes or the boundary is too far away.
- BSS distillation: BSS distillation combines classification loss, conventional knowledge-distillation loss, and boundary supporting loss to transfer boundary information directly.The classification loss transfers one-hot labels, the distillation loss transfers probabilistic labels, and the boundary supporting loss targets decision-boundary information.
- BSS distillation: The method reduces computation by applying the boundary supporting loss only to selected base samples and prioritizes target classes that the teacher finds difficult to distinguish.Target-class sampling uses teacher class probabilities, assigning priority to high-probability non-base classes.
Experiments
Experiments evaluate the proposed decision-boundary-based distillation method across image-classification datasets, adversarial-sample variants, decision-boundary similarity, and reduced-data settings. The method improves knowledge distillation and shows stronger gains when training data are limited.
- Performance on image classification: The proposed method improves knowledge distillation on CIFAR-10, ImageNet 32×32, and TinyImageNet using residual-network teacher-student classifiers.On CIFAR-10 it outperforms Hinton and existing state-of-the-art methods; on ImageNet 32×32 it outperforms Hinton and is comparable to other state-of-the-art methods.
- Performance on image classification: On TinyImageNet, the proposed method has higher top-5 accuracy than other state-of-the-art methods, despite lower top-1 accuracy than FITNET and FSP.
- Generalization of the classifier: The proposed method shows larger performance improvement as the amount of training data decreases, particularly under insufficient-data conditions.The reduced-data experiment varies the training-set size from 100% to 20%.
- Decision-boundary similarity: The proposed method increases both MagSim and AngSim relative to the original method, whereas Hinton mainly increases MagSim.MagSim and AngSim measure similarity between teacher and student decision boundaries.
- Adversarial-sample comparison: Boundary supporting samples produce the greatest improvement among the tested adversarial-sample approaches, while random noise produces the smallest improvement.The comparison tests different adversarial attacks for knowledge distillation on CIFAR-10.
- Sampling analysis: The proposed base-sample selection reduces computation and contributes to performance, while probability-based target-class sampling is reported as reasonable and effective.
Conclusion
The paper proposes using boundary supporting samples to transfer more accurate decision-boundary information during knowledge distillation. Experiments report improved distillation performance and stronger generalization with fewer training samples, while framing sample manipulation as a new research direction.
- The method modifies adversarial attacks to find boundary supporting samples for knowledge distillation.These samples support a decision boundary and are used to transfer more accurate boundary information.
- Experiments show that the proposed method improves knowledge distillation performance.
- The method has stronger generalization performance and is more effective with fewer training samples.
- Sample manipulation for knowledge distillation is presented as a new direction not attempted in prior studies.
- The approach also introduces adversarial attacks as a way to find and transfer decision-boundary information.