Source-linked AI summary
Adversarially Robust Distillation
Micah Goldblum, Liam Fowl, Soheil Feizi, Tom Goldstein
TL;DR
The paper asks how adversarial robustness transfers from large teacher networks to efficient student networks, since conventional distillation can leave students vulnerable. It introduces Adversarially Robust Distillation, which matches teacher outputs on adversarial inputs, and reports robust students that outperform comparable adversarially trained models while retaining distillation’s efficiency. The paper also develops a faster ARD variant for accelerated training.
Problem
Conventional distillation produces accurate, efficient students, but the resulting networks may not be robust to adversarial attacks.
Method
ARD trains student networks to mimic teacher outputs within an ϵ-ball around training samples, adapting adversarial-training principles to distillation.
Results
ARD students outperform similarly architected adversarially trained networks in robust accuracy, often also in natural accuracy, and exceed state-of-the-art performance on CIFAR-10 and CIFAR-100.
Takeaways & Limitations
Robustness can transfer substantially from robust teachers to smaller students, even with clean-image distillation, while ARD can improve transfer when ordinary distillation is insufficient.
Takeaways & Limitations
Robustness transfer is not universal: some robust teachers transfer less robustness, and a robust WideResNet teacher transfers almost none on CIFAR-100 under 20-step PGD.
Abstract
from arXiv · showhide
Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robustness may be inherited by the student even when distilled on only clean images. Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto student networks. In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student. In our experiments, we find that ARD student models decisively outperform adversarially trained networks of identical architecture in terms of robust accuracy, surpassing state-of-the-art methods on standard robustness benchmarks. Finally, we adapt recent fast adversarial training methods to ARD for accelerated robust distillation.
1 Introduction
The paper targets efficient student networks that retain or gain robustness against adversarial attacks. It introduces ARD, which trains students to mimic teacher outputs on adversarially perturbed inputs while preserving the efficiency benefits of distillation.
- Motivation: Knowledge distillation produces compact, accurate student networks but can neglect their vulnerability to adversarial examples.Small perturbations can cause standard neural networks to misclassify inputs.
- Method: ARD trains students to mimic their teacher’s outputs within an ϵ-ball around training samples instead of only matching correct labels.This makes ARD a distillation analogue of adversarial training.
- Findings: Clean-image distillation can preserve much of a robust teacher’s adversarial robustness, enabling efficient robust models without adversarial-training costs.The transfer is substantial but not universally complete.
- Findings: ARD students achieve higher robust accuracy than adversarially trained models with identical architectures and may exceed their teacher’s robust accuracy.ARD can also improve natural accuracy simultaneously.
- Method: Figure 1 depicts ARD minimizing discrepancies between teacher outputs on natural images and student outputs on adversarial images.The teacher supplies the target behavior while the student is evaluated on perturbed inputs.
2 Related Work
Related work combines network compression, knowledge distillation, and defenses against adversarial attacks. Prior robustness-oriented approaches include defensive distillation, adversarial-training variants, quantization, and few-shot adversarial querying.
- Compression and distillation: Compression methods reduce neural-network storage or computation through binarized weights, low-rank regularization, pruning, or lightweight architectures.Knowledge distillation transfers a powerful teacher’s behavior to a smaller student such as MobileNetV2.
- Robust distillation: Defensive distillation uses identical teacher and student architectures and was later broken by attacks across ℓ0, ℓ∞, and ℓ2 norms.Its reported robustness was attributed to gradient masking.
- Adversarial robustness: Robustness methods modify training objectives or architectures using surrogate losses, feature denoising, logit pairing, and input transformations.These methods differ in how they constrain clean-versus-adversarial behavior.
- Defensive compression: Defensive quantization targets robustness while reducing space complexity, but it does not reduce Multiply-Add operations required for inference.This separates storage efficiency from computational efficiency.
- Few-shot robustness: Adversarial querying addresses robust small-model construction from a meta-learning perspective in few-shot settings.It represents a distinct setting from conventional distillation.
3 Problem Setup
Knowledge distillation trains a small student to mimic a larger teacher, improving efficiency and accuracy but leaving adversarial robustness unresolved. The paper combines distillation with adversarial training to produce robust students.
- Knowledge distillation: Knowledge distillation trains a small student network to mimic the outputs of a larger teacher instead of using only one-hot labels.The student minimizes output discrepancy over data drawn from the data-generating distribution.
- Objectives: The standard distillation objective uses KL divergence between temperature-scaled student and teacher outputs, optionally combined with cross-entropy against true labels.The temperature divides logits before softmax, while α controls the loss combination in the paper’s experiments.
- Adversarial training: Adversarial training seeks correct labels throughout an ϵ-ball around each sample, while VAT and TRADES use output consistency between clean and adversarial inputs.These methods formulate robustness through perturbed inputs and, in VAT or TRADES, a KL-based consistency term.
- Adversarially Robust Distillation: The paper combines knowledge distillation and adversarial training to produce robust student networks when robust teacher networks are available.Experiments focus on robustness to ℓ∞ attacks using FGSM-based PGD and evaluate WideResNet, ResNet18, and MobileNetV2 architectures.
4 Adversarial robustness is preserved under knowledge distillation
Robustness can transfer from robust teachers to students during clean-image distillation, allowing lightweight students to acquire much of the teacher’s robust accuracy. However, transfer depends on the teacher, dataset, and distillation setting, motivating ARD for broader preservation.
- 4.1 Non-robust teachers produce non-robust students: A student distilled from a robust teacher is more robust than a naturally trained network, while ARD yields greater robustness and closely mimics the teacher’s decision boundary.Figure 2 visualizes classification regions and marks adversarially vulnerable training points with ℓ∞ boxes.
- 4.1 Non-robust teachers produce non-robust students: Clean-image distillation from a robust teacher can produce students that acquire most of the teacher’s robust accuracy.Students distilled from a robust ResNet18 were far more robust than students of a non-robust teacher, without adversarial training.
- 4.2 Robust teachers can produce robust students, even distilling on only clean data: Knowledge distillation can produce robust lightweight networks cheaply because the student need not undergo adversarial training when robustness transfers from the teacher.Table 2 and Table 3 compare natural and robust accuracy under a 20-step PGD attack on CIFAR-10.
- 4.3 Not all robust networks are good teachers, and robustness does not transfer on some datasets: Robustness transfer is not universal: robust WideResNet teachers transfer less than the weaker ResNet18 teacher, and transfer can nearly disappear on CIFAR-100.The reported failure occurs for untargeted 20-step PGD attacks on the harder dataset.
- 4.3 Not all robust networks are good teachers, and robustness does not transfer on some datasets: ARD is designed to preserve robustness across architectures and datasets where vanilla knowledge distillation fails to transfer it.This extends robust distillation beyond the conditions under which ordinary clean-image distillation succeeds.
5 Improving the robustness of student models with Adversarially Robust Distillation (ARD)
Adversarially Robust Distillation (ARD) trains students to match robust teachers within adversarial neighborhoods, improving robustness where conventional distillation may fail. Across experiments, ARD produces robust students that can outperform adversarially trained counterparts and approach teacher robustness.
- ARD method: ARD trains a student to mimic its teacher’s output on adversarial examples within an ϵ-ball, rather than matching only clean outputs.The method combines knowledge distillation with adversarial-training machinery and treats the teacher’s clean-data softmax output as the target.
- ARD comparisons: ARD variants outperform adversarially trained MobileNetV2 and TRADES models in both natural and robust accuracy on CIFAR-10.The WideResNet teacher used in these experiments has 20× as many parameters and performs 70× as many MAdd operations as the student.
- ARD works when distillation fails: ARD creates robust students from teachers whose robustness transfers poorly under conventional knowledge distillation.On CIFAR-100, a MobileNetV2 trained with ARD from an adversarially trained WideResNet is nearly as robust as its teacher.
- Architecture effects: ARD can improve robustness even when teacher and student architectures are identical, including ResNet18 and MobileNetV2.The paper reports that ARD boosted robustness for both architectures and that some ARD students were more robust than their teachers.
- Robustness evaluation: ARD and Fast-ARD outperform adversarial training and free training, respectively, across the tested ℓ∞ attacks.The attacks include Momentum Iterative Fast Gradient Sign Method, DeepFool, 1000-step PGD, and PGD with random restarts.
6 Space and time efficiency of student and teacher models
The experiments compare teacher and student models using parameter counts and multiply-add operations, emphasizing the student’s substantially lower inference cost. Fast-ARD is evaluated as an accelerated robust-distillation variant under matched training conditions.
- Training efficiency: Fast-ARD is designed to accelerate ARD by adapting free adversarial training to distillation.The paper states that Fast-ARD is equally fast to the corresponding free-training approach.
- Training efficiency: Table 10 compares training times for adversarial training, clean distillation, ARD, and Fast-ARD using identical batch sizes and 200 epochs on CIFAR-10.The adversarial-training and ARD models use a 10-step PGD attack.
- Efficiency metrics: MAdd operations are used as an implementation-invariant measure of inference time complexity.The paper notes that elapsed runtime varies with implementation and deep-learning framework.
- Model efficiency: MobileNetV2 has ∼2.3M parameters and requires ∼187M MAdd operations per forward pass, versus ∼46.2M parameters and ∼13.3B MAdds for the WideResNet teacher.The ResNet18 teacher has ∼11.2M parameters and requires ∼1.1B MAdd operations per forward pass.
- Model efficiency: The MobileNetV2 student performs ∼1.4% as many MAdd operations during a forward pass as the WideResNet teacher.
7 Discussion
The paper concludes that clean-data distillation can transfer substantial teacher robustness, while ARD improves robustness when conventional transfer is insufficient. It also reports accelerated robust distillation through Fast-ARD.
- Discussion: Knowledge distillation can transfer a large amount of a teacher’s adversarial robustness even when the student trains only on clean data.
- Discussion: Some distilled students remain far less robust than their teachers, motivating ARD as a method for improving student robustness.
- Discussion: ARD students outperform similarly structured adversarially trained networks in robust and often natural accuracy.
- Discussion: The paper reports that its models exceed state-of-the-art performance on CIFAR-10 and CIFAR-100 robustness benchmarks.
- Discussion: Fast-ARD provides a free-adversarial-training variant of ARD with appreciably accelerated performance.The paper identifies recent distillation improvements and adaptive augmentation as possible future directions.
8 Experimental details
The experiments train ARD models with SGD-based optimization, dataset-specific temperature terms, and 10-step FGSM-based PGD attacks. Fast-ARD models are trained for the same number of epochs as natural distillation to match training time.
- Optimization: Models are trained for 200 epochs using SGD, momentum 0.9, and weight decay 2e−4.
- Optimization: Fast-ARD models are trained for 200 epochs so their training time matches natural distillation.
- Optimization: The initial learning rate is 0.1 and is decreased by a factor of 10 during training.
- Distillation settings: Temperature is set to 30 for CIFAR-10 and 5 for CIFAR-100.
- Adversarial examples: Training adversarial examples use 10-step FGSM-based PGD with ϵ = 8⁄255, step size 2⁄255, and random starts.
Appendix
Robustness transfer depends on distillation settings and augmentation. Some loss modifications harm robustness, while adversarial-point augmentation is costly without significant improvement.
- Temperature and α: Only very low temperature terms hinder robustness preservation, while a wide range of temperatures produce robust students.Knowledge distillation is not highly sensitive to temperature overall.
- Temperature and α: Robustness transfer decays rapidly as α decreases, creating a robustness-accuracy tradeoff.
- Data augmentation: Data augmentation dramatically improves robustness transfer by exposing the student to the teacher’s behavior at more data points.
- Data augmentation: Adversarial-point augmentation greatly decreases training speed and provides no significant improvement.
- Loss-function variants: Adding non-adversarial distillation KL divergence slightly increases natural accuracy but sharply decreases robust accuracy.
- Loss-function variants: Using adversarial examples in the teacher’s KL-divergence target decreases both natural and robust accuracy and increases training memory and computation.The teacher’s adversarial forward pass is especially costly when the teacher is much larger than the student.
Appendix C: ARD with naturally trained teacher models
ARD can produce robust students from naturally trained teachers, but robustness is weaker than with robust teachers and identical-architecture adversarial training.
- Naturally trained teachers: Naturally trained teachers produce robust students under ARD.
- Naturally trained teachers: Students from naturally trained teachers may be less robust than students from robust teachers and adversarially trained models with identical architecture.The comparison is reported on CIFAR-10 using 20-step PGD robust accuracy.
Appendix D: Improving the speed of ARD by reducing the number of attack steps
ARD training speed can be improved by reducing attack steps, but this trades away robust accuracy for higher natural accuracy. ARD is comparatively less sensitive to temperature and α than conventional robustness-preserving distillation.
- Reducing attack steps: Reducing attack steps improves natural accuracy while decreasing robust accuracy.The experiment evaluates TRADES WideResNet teachers distilled to MobileNetV2 on CIFAR-10.
- Reducing attack steps: Reducing attack steps has a similar accuracy effect to reducing α, but additionally accelerates training.
- Sensitivity to hyperparameters: ARD robust accuracy decays far slower as α decreases than under knowledge distillation, making ARD less sensitive to α.
- Sensitivity to hyperparameters: Varying ARD’s temperature does not significantly affect natural or robust accuracy.