PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better

Bojia Zi, Shihao Zhao, Xingjun Ma, Yu-Gang Jiang

arXiv:2108.07969v1cs.CRcs.LG

TL;DR

Adversarial training can favor high-capacity models, limiting its effectiveness for small models preferred under tight resource constraints. The paper proposes RSLAD, using robust soft labels from an adversarially trained teacher to guide a small student. It reports improved robustness over existing adversarial training and distillation methods across two benchmark datasets and white- and black-box settings.

  • Problem

    Adversarial training generally favors high-capacity models, while small models are preferred when memory or computational resources are limited.

    Source: §1 · p. 1

  • Method

    RSLAD uses an adversarially trained teacher’s robust soft labels to supervise a small student on natural and adversarial examples in all loss terms.

    Source: §1 · p. 2

  • Results

    RSLAD’s advantage over existing adversarial training and distillation methods was empirically verified on two benchmark datasets under white-box and black-box settings.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper identifies robust soft labels as important for improving robustness through distillation to small models.

    Source: §1 · p. 2

Abstract

from arXiv · show

Adversarial training is one effective approach for training robust deep neural networks against adversarial attacks. While being able to bring reliable robustness, adversarial training (AT) methods in general favor high capacity models, i.e., the larger the model the better the robustness. This tends to limit their effectiveness on small models, which are more preferable in scenarios where storage or computing resources are very limited (e.g., mobile devices). In this paper, we leverage the concept of knowledge distillation to improve the robustness of small models by distilling from adversarially trained large models. We first revisit several state-of-the-art AT methods from a distillation perspective and identify one common technique that can lead to improved robustness: the use of robust soft labels -- predictions of a robust model. Following this observation, we propose a novel adversarial robustness distillation method called Robust Soft Label Adversarial Distillation (RSLAD) to train robust small student models. RSLAD fully exploits the robust soft labels produced by a robust (adversarially-trained) large teacher model to guide the student's learning on both natural and adversarial examples in all loss terms. We empirically demonstrate the effectiveness of our RSLAD approach over existing adversarial training and distillation methods in improving the robustness of small models against state-of-the-art attacks including the AutoAttack. We also provide a set of understandings on our RSLAD and the importance of robust soft labels for adversarial robustness distillation.

1. Introduction

Adversarial training can favor high-capacity models, limiting its fit for small devices with tight memory or compute budgets. The paper identifies robust soft labels as useful supervision and proposes RSLAD to guide small students with them.

1. Introduction

  • AT’s preference for higher-capacity models limits its effectiveness on small models favored in resource-constrained settings such as smartphones and autonomous vehicles.The paper frames knowledge distillation from robust large models as a way to boost small-model robustness.Source: §1 · p. 1
  • The authors identify predictions from adversarially trained models—Robust Soft Labels—as a common technique that can improve robustness.They argue these labels represent a teacher’s robust behavior more fully than original hard labels.Source: §1 · pp. 1–2
  • RSLAD replaces hard labels with robust soft labels in all supervision loss terms to train small robust student models.The method is proposed as an adversarial robustness distillation approach.Source: §1 · p. 2
  • The paper reports empirical effectiveness of RSLAD for improving small DNN robustness against state-of-the-art attacks.It also provides analyses of RSLAD and the importance of robust soft labels for robustness distillation.Source: §1 · p. 2

2. Related Work

The related work reviews adversarial attacks and training, then positions robustness distillation as a way to transfer robust knowledge from a teacher to a smaller student.

2.1. Adversarial Attack

  • AutoAttack is described as an ensemble of four methods and an arguably especially powerful evaluation attack.Its components are Auto-PGD, DLR, FAB-Attack, and the black-box Square Attack.Source: §2.1 · p. 2

2.2. Adversarial Training

  • Adversarial training is framed as min-max optimization, while robustness advances have included larger models, more data, KL loss, and other training modifications.The paper notes that smaller models generally have poorer robust performance, limiting use where storage or computation is constrained.Source: §2.2 · p. 2

2.3. Knowledge Distillation

  • Knowledge distillation transfers a large teacher’s knowledge to a small student, and adversarial robustness distillation uses an adversarially pretrained teacher to improve student robustness.ARD may use a larger teacher [14], while IAD may share the student’s architecture [62]; both have produced more robust students than training from scratch [3].Source: §2.3 · pp. 2–3

3. Proposed Distillation Method

The paper interprets several adversarial training and distillation methods through knowledge distillation, emphasizing robust soft labels as useful supervision. It then proposes RSLAD, which uses a robust teacher’s predictions to supervise the student on natural and adversarial examples.

3.1. A Distillation View of Adversarial Training

  • Viewed as distillation, SAT uses hard labels, TRADES and MART use self-distillation, and ARD uses a more powerful teacher; the paper links this progression to improved robustness.IAD combines teacher and student knowledge, while its inner maximization still uses hard labels.Source: §3.1 · p. 3
  • Robust soft labels convey a teacher’s robust behavior more fully than hard labels, and the paper reports that label-smoothed or naturally trained non-robust soft labels cannot improve robustness.ARD uses hard labels in some loss terms, and IAD retains them in its inner maximization loss.Source: §3.1 · p. 3, §3.2 · p. 4

3.2. Robust Soft Label Adversarial Distillation

  • RSLAD uses robust soft labels from a large teacher for student training on natural and adversarial examples in all loss terms, without hard labels.The method applies the labels in both inner maximization and outer minimization; its temperature is fixed at τ = 1.Source: §3.2 · p. 3, Figure 1 · p. 4
  • Because the teacher’s robust soft labels supervise the student’s clean outer-minimization training, RSLAD replaces the commonly used CE loss with KL divergence.The divergence measures the distributional difference between the models’ output probabilities.Source: §3.2 · pp. 3–4

4. Experiments

The experiments evaluate RSLAD against adversarial-training and distillation baselines, then examine its loss components, learned maps, soft-label types, and teacher choice. RSLAD leads the reported white-box and black-box comparisons, while the additional analyses find that robust labels and a moderately large teacher matter for student robustness.

4.1. Experimental Settings

  • The study evaluates CIFAR-10 and CIFAR-100 using ResNet-18 and MobileNetV2 students, robust WideResNet teachers, and five adversarial-training and distillation baselines.Evaluation includes FGSM, two PGD variants, CW∞, and AutoAttack, with maximum perturbation ϵ = 8/255.Source: §4.1 · pp. 4–5

4.2. Adversarial Robustness Evaluation

  • Across CIFAR-10 and CIFAR-100, RSLAD achieves state-of-the-art robustness against all five white-box attacks; ResNet-18 improves PGDTRADES robustness by 1.74% and 1.32%, respectively.On CIFAR-10, RSLAD also outperforms ARD by 2.30% against AutoAttack for ResNet-18.Source: Table 3 · p. 5, §4.2 · pp. 5–6, Table 4 · p. 6
  • On CIFAR-10, RSLAD surpasses all four baselines against all three black-box attacks, with distillation methods outperforming TRADES and TRADES outperforming SAT.The evaluation includes transfer and query-based attacks.Source: Table 5 · p. 6, §4.2 · pp. 6–7

4.3. A Comprehensive Understanding of RSLAD

  • Replacing either ARD loss with its RSLAD counterpart improves robustness, and replacing both achieves the best result among the ablations.The ablation uses a ResNet-18 student distilled from the same WideResNet-34-10 teacher as the main experiments.Source: §4.3 · p. 7, Table 6 · p. 7
  • RSLAD student attention maps are more similar to the teacher’s than ARD and IAD maps, indicating closer mimicry of the teacher’s learned knowledge.The comparison visualizes attention and saliency maps for adversarial examples.Source: Figure 2 · p. 6, §4.3 · p. 7

4.4. Further Explorations

  • Using smooth or natural soft labels instead of robust soft labels sharply reduces robustness, with natural-teacher labels especially harmful to robustness distillation.The comparison uses a ResNet-18 student, WideResNet-34-10 teacher, and RSLAD on CIFAR-10.Source: §4.4 · pp. 7–8, Figure 3 · p. 7, Table 7 · p. 8
  • Student robustness rises and then falls as teacher complexity increases, with the best ResNet-18 result using a WideResNet-34-10 teacher about 4.5× larger.Students can surpass smaller teachers, while the student-teacher robustness gap grows when teacher complexity exceeds WideResNet-34-10.Source: §4.4 · p. 8, Figure 4 · p. 8

5. Conclusion

The paper studies distillation for training small robust models, identifies robust soft labels as important, and proposes RSLAD to exploit them. Experiments on two benchmark datasets under white-box and black-box settings verify its advantage over adversarial-training and distillation methods.

5. Conclusion

  • The paper investigates training small robust models through knowledge distillation and identifies robust soft labels as important for improving robustness.It proposes RSLAD to fully exploit robust soft labels and reports empirical advantages over existing methods on two benchmark datasets under white-box and black-box settings.Source: §5 · p. 8
  • The paper provides analyses of RSLAD, soft-label types, and teacher-student interplay as understandings relevant to building robust lightweight models.Source: §5 · p. 8
  • The reported evaluation covers two benchmark datasets and both white-box and black-box settings.Source: §5 · p. 8

A. Hyper-parameter Selection

Robustness rises as the adversarial-to-natural loss ratio increases, then plateaus; the best reported setting is k=5.0/1.0. The study varies α for a ResNet-18 student distilled from a WideResNet-34-10 teacher on CIFAR-10.

A. Hyper-parameter Selection

B. Learning From Different Teachers

Across five attacks, the expanded teacher evaluation supports robust saturation and robust underfitting, indicating that a moderately large teacher can be better than an overly large one.

B. Learning From Different Teachers

  • A moderately large teacher can be a better teacher than an overly large teacher, given robust saturation and robust underfitting across the expanded evaluation.The student results cover five attacks at both best and last checkpoints.Source: B. Learning From Different Teachers · p. 11
  • The expanded evaluation confirms robust saturation and robust underfitting across more attacks.Results are reported at both the best and last checkpoints.Source: B. Learning From Different Teachers · p. 11
  • The student is evaluated against all five attacks, with teacher robustness reported separately.Student results are in Table 9 and teacher results in Table 8.Source: B. Learning From Different Teachers · p. 11

C. RSLs of Natural or Adversarial Examples?

On CIFAR-10 with a ResNet-18 student and WideResNet-34-10 teacher, natural-example robust soft labels outperform adversarial-example labels in the reported substitutions, although adversarial-example labels also help robustness.

C. RSLs of Natural or Adversarial Examples?

  • RSLAD ordinarily uses natural-example outputs T(x) as robust soft labels, while adversarial-example outputs are T(x′).The experiment substitutes T(x′) for T(x) in the RSLAD loss terms.Source: C. RSLs of Natural or Adversarial… · p. 11
  • 49.79% robustness against AA with T(x′) replacing T(x) in all loss terms is below RSLAD with T(x).The 49.79% result is slightly higher than TRADES at 49.27% against AA.Source: C. RSLs of Natural or Adversarial… · p. 11
  • Replacing more T(x′) with T(x) in Lmax or Lmin improves both clean accuracy and robustness.The comparison uses a ResNet-18 student and WideResNet-34-10 teacher on CIFAR-10.Source: C. RSLs of Natural or Adversarial… · p. 11

D. Training all baselines for 300 epochs

When baselines are extended to 300 epochs, their last-checkpoint performance degrades from increased overfitting, while RSLAD achieves the best overall performance.

D. Training all baselines for 300 epochs

E. Teacher models’ diversity

Testing a SAT+AWP teacher, rather than the TRADES teacher used in most main-text experiments, yielded student accuracy gains at the best checkpoint, supporting RSLAD with different teacher types.

E. Teacher models’ diversity

Loading 2108.07969v1…