Source-linked AI summary
Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples
Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao, Shaowen Yao, Wei Zhou
TL;DR
Adversarial distillation typically supervises students using teacher predictions on clean or student-challenging adversarial inputs, while teacher-favorable supervision within the perturbation neighborhood remains underexplored. CGARD jointly searches separate student-adversarial and teacher-collaborative examples under a teacher-loss constraint, then uses both teacher predictions for alignment. Across CIFAR-10 and CIFAR-100, it reports consistent robustness gains over strong adversarial distillation baselines, including up to 0.72 percentage points higher AutoAttack accuracy.
Problem
Adversarial distillation mainly uses teacher predictions on clean or student-challenging adversarial examples, leaving teacher-favorable supervision within the perturbation neighborhood underexplored.
Source: §1 · p. 1Method
CGARD jointly searches separate student-adversarial and teacher-collaborative examples under a teacher cross-entropy constraint and aligns the student with teacher predictions on both.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The search-strategy comparison and outer-objective ablations support constrained joint search and the combined use of both alignment terms.
Source: §4 · p. 4The margin-based rationale assumes the fixed teacher's logit mapping is L-Lipschitz under ℓ∞.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Adversarial distillation transfers robustness from high-capacity teachers to compact students. Existing adversarial distillation methods mainly use teacher predictions on clean or adversarial examples to supervise student learning. However, teacher-favorable supervision within the perturbation neighborhood remains underexplored in adversarial distillation. We therefore propose Collaboratively Guided Adversarial Robust Distillation (CGARD), which jointly optimizes distinct student-adversarial and teacher-collaborative examples within the same perturbation neighborhood. The teacher-collaborative example is constrained to incur no greater cross-entropy loss under the teacher than the clean input. CGARD combines collaborative teacher guidance with adversarial teacher supervision to improve robust knowledge transfer. Experiments on CIFAR-10 and CIFAR-100, including white-box evaluation and additional black-box transfer evaluation, demonstrate consistent robustness improvements over strong adversarial distillation baselines.
1. INTRODUCTION
Lightweight students can remain vulnerable to adversarial perturbations, and their limited capacity can constrain adversarial training, motivating distillation from robust teachers. CGARD addresses underused teacher-favorable supervision by jointly searching separate student-adversarial and teacher-collaborative examples and aligning the student to teacher predictions on both.
1. INTRODUCTION
- Adversarial distillation transfers robustness from high-capacity robust teachers to compact students, addressing the capacity constraints that can limit adversarial training of lightweight models.Lightweight networks enable deployment on resource-limited platforms but remain vulnerable to adversarial perturbations.Source: §1 · p. 1
- Existing adversarial distillation typically uses teacher supervision from clean or student-challenging adversarial inputs, leaving teacher-favorable inputs in the perturbation neighborhood largely unused.Applying collaborative-example construction to distillation is nontrivial because the student is trainable while the robust teacher is fixed.Source: §1 · p. 1
- CGARD jointly optimizes separate student-adversarial and teacher-collaborative examples through teacher-student prediction discrepancy, constraining the collaborative example's teacher cross-entropy to no more than the clean input's.The joint search lets the teacher-collaborative example adapt to the evolving student-adversarial prediction.Source: §1 · p. 1, Figure 1 · p. 2
- Dual alignment combines teacher supervision on the collaborative and student-adversarial examples to train the student's adversarial prediction.This uses teacher predictions from both examples for robust knowledge transfer.Source: §1 · pp. 1–2
- Across CIFAR-10 and CIFAR-100, CGARD improves robustness over strong adversarial distillation baselines, including up to 0.72 percentage points higher AutoAttack accuracy.The paper reports consistent robustness improvements.Source: §1 · p. 2
2. METHODOLOGY
The methodology motivates teacher-collaborative examples through a relationship between teacher cross-entropy and perturbation resistance, then jointly searches separate examples under a teacher-loss constraint. It trains the student by aligning its adversarial prediction with teacher predictions on both searched examples.
2.1. Motivation for Teacher-Collaborative Examples
- Under an ℓ∞ Lipschitz assumption on the fixed teacher, lower teacher cross-entropy increases a lower bound on the perturbation magnitude required to change its prediction when the bound's right-hand side is positive.This motivates seeking teacher-collaborative examples with lower teacher cross-entropy for additional distillation supervision.Source: §2.1 · p. 2
2.2. Constrained Joint Inner Search
- The constrained joint search optimizes separate student-adversarial and teacher-collaborative examples within the clean input's perturbation neighborhood, with their updates coupled through teacher-student prediction discrepancy.The teacher-loss constraint requires the collaborative example's teacher cross-entropy to be no greater than on the clean input.Source: §2.2 · pp. 2–3
- Each search iteration projects both examples into the perturbation neighborhood and valid pixel range, retaining a teacher-branch update only when it satisfies the teacher cross-entropy constraint.An infeasible teacher-branch candidate is reset to the clean example before the next update.Source: §2.2 · p. 3
2.3. Dual-Alignment Outer Minimization
- Dual alignment trains the student by matching its prediction on the student-adversarial example to teacher predictions on both the collaborative example and that same adversarial example.The collaborative and adversarial alignment terms are balanced by α, and the teacher remains fixed.Source: §2.3 · p. 3
3. EXPERIMENTS
Across CIFAR-10 and CIFAR-100, CGARD leads white-box robust accuracy in all four dataset–architecture settings, while showing competitive black-box transfer robustness. Ablations favor constrained joint search and using both alignment terms.
3.1. Experimental Setup
- Experiments compare CGARD with adversarial-training and adversarial-distillation baselines on CIFAR-10 and CIFAR-100, reporting clean, PGD, CW∞, and AutoAttack accuracy under an ℓ∞ budget of ε = 8/255.WRN-34-10 is the fixed robust teacher; RN-18 and MN-V2 are students, trained for 300 epochs.Source: §3.1 · p. 3
3.2. White-Box Robustness
- 52.46% AA accuracy for CIFAR-10 RN-18, 0.44 percentage points above its strongest AA baseline; CGARD leads PGD, CW∞, and AA in all four settings.On CIFAR-10, clean accuracy decreases by 1.32 and 0.92 percentage points for RN-18 and MN-V2, respectively; on CIFAR-100 it is maintained or improved.Source: §3.2 · p. 3, Table 1 · p. 4
3.3. Black-Box Transfer Robustness
- CGARD shows competitive transfer robustness against transferred PGD and CW∞ attacks across WRN, VGG-19, and MN-V2 surrogates targeting RN-18 on CIFAR-10.The evaluation complements white-box testing with transfer-based black-box attacks.Source: §3.3 · p. 3, Table 2 · p. 4
3.4. Ablation Studies
- 1.55, 1.33, and 1.45 percentage-point gains in PGD, CW∞, and AA, respectively, came with a 1.29-point clean-accuracy decrease for joint versus sequential search.The sequential strategy fixes the teacher prediction before generating the student-adversarial example; CGARD instead jointly optimizes under the teacher cross-entropy constraint.Source: §3.4 · pp. 3–4, Table 3 · p. 4
- 52.46% AA accuracy with both alignment terms, versus 51.84% with Lcol alone and 51.83% with Ladv alone, under the fixed constrained joint search.The ablation compares the dual-alignment objective on CIFAR-10 with WRN as teacher and RN-18 as student.Source: Table 4 · p. 4, §3.4 · p. 4
4. CONCLUSION
CGARD jointly optimizes separate student-adversarial and teacher-collaborative examples under a teacher cross-entropy constraint, then trains through dual alignment. Across CIFAR-10 and CIFAR-100, it achieves the highest AA accuracy among evaluated student methods and competitive black-box transfer robustness.
- Across CIFAR-10 and CIFAR-100 with two student architectures, CGARD achieves the highest evaluated-student AA accuracy, competitive black-box transfer robustness, and ablation support for constrained joint search and both alignment terms.Source: §4 · p. 4