PaperlayerView exact revision on arXiv

Source-linked AI summary

CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation

Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao, Zhenli He, Wei Zhou, Shaowen Yao

arXiv:2610.19911v1cs.LGcs.CV

TL;DR

One-way adversarial distillation leaves potential feedback from the weaker student unused, while using it risks transferring student errors to the robust teacher. CARD closes the loop by using student feedback to refine the teacher and then continuing training the same student; it improves adversarial robustness in most evaluated settings while generally preserving clean accuracy.

  • Problem

    Conventional adversarial distillation leaves the robust teacher fixed after transferring knowledge to the student, despite potentially useful student feedback that may also propagate errors if transferred indiscriminately.

    Source: §1 · p. 2

  • Method

    CARD first distills a student, uses reliability-enhanced student feedback to refine the teacher, then continues training the same student with the refined teacher under the unchanged base objective.

    Source: §1 · pp. 2–3

  • Results

    CARD improves adversarial robustness in most evaluated settings while generally preserving clean accuracy across datasets, architectures, and base methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support using a distilled student as feedback for teacher refinement, extending robustness transfer beyond the one-way paradigm.

    Source: §5 · p. 17

    Evaluation is limited to relatively small-scale image-classification benchmarks and primarily l∞-bounded attacks, refinement adds training cost, and gains are not uniform across attacks and base methods.

    Source: §5 · pp. 17–18

Abstract

from arXiv · show

Adversarial robustness distillation typically treats a robust teacher as a fixed knowledge source and transfers robustness to a compact student in a one-way manner. This paradigm terminates teacher-student interaction once the student is distilled, thereby discarding potentially useful feedback from the distilled student. However, exploiting such feedback is nontrivial because indiscriminate weak-to-strong transfer may propagate student errors to the teacher. To address this issue, we propose Cyclic Adversarial Robustness Distillation (CARD), a closed-loop framework that turns the distilled student into a feedback source for refining the robust teacher and subsequently transfers the teacher gains back to the same student. CARD first performs strong-to-weak adversarial distillation and then uses reliability-enhanced student feedback to refine the robust teacher. The refinement combines inverse adversarial student feedback with teacher-student discrepancy maximization, after which the refined teacher is used to continue training the same student under the unchanged base distillation objective. We further establish a sample-level sufficient condition for reliable weak-to-strong feedback transfer and derive a distribution-level upper bound on the refined teacher's risk. Experiments on CIFAR-10, CIFAR-100, and TinyImageNet-10 across multiple teacher-student architectures and base distillation methods show that CARD improves adversarial robustness in most evaluated settings while generally preserving clean accuracy. These results show that the distilled student can provide useful feedback to the teacher, enabling robustness transfer to proceed beyond the conventional one-way teacher-to-student paradigm.

1. Introduction

The introduction motivates using distilled-student feedback to refine a robust teacher, then proposes CARD’s closed loop and reports robustness gains in most evaluated settings. It also presents theoretical conditions for reliable weak-to-strong feedback.

1. Introduction

  • The motivating challenge is to use potentially complementary student feedback while reducing erroneous transfer and focusing refinement on informative regions.The distilled student remains weaker overall than the robust teacher, so indiscriminate alignment risks propagating student errors.Source: §1 · p. 2
  • A sample-level sufficient condition characterizes when weak-to-strong feedback preserves correct predictions, and a distribution-level upper bound covers refined-teacher classification risk.These analyses address whether student feedback can refine a stronger teacher without compromising correctness.Source: §1 · p. 3
  • CARD distills a compact student, uses that student to refine the robust teacher, then uses the refined teacher to continue training the same student.The student-training objective remains unchanged.Source: §1 · p. 3, Figure 1 · p. 3
  • Inverse adversarial examples provide lower-loss student feedback for clean-input refinement, while discrepancy maximization targets adversarial inputs where teacher and student differ.The teacher is supervised with student predictions on the corresponding inputs.Source: §1 · p. 3
  • CARD improves adversarial robustness over corresponding base methods in most evaluated settings while generally preserving clean accuracy.The experiments cover CIFAR-10, CIFAR-100, and TinyImageNet-10 across multiple teacher-student architectures and base distillation methods.Source: §1 · p. 3

2. Related Work

The related work reviews adversarial training and robustness distillation, then situates CARD among approaches that adapt supervisory models and use weak-to-strong feedback. It distinguishes CARD’s post-distillation teacher refinement from prior methods’ roles for peers, teachers, and discrepancy.

2.1. Adversarial Training

  • Adversarial training optimizes against perturbations, but strong robustness remains difficult for compact networks with limited representational capacity.Prior work explores attack difficulty, clean-accuracy–robustness trade-offs, objectives, informative samples, and local neighborhoods.Source: §2.1 · p. 3

2.2. Adversarial Robustness Distillation

  • Robustness distillation transfers robust knowledge from a high-capacity teacher to a lightweight student, with prior methods improving supervision or aligning features, gradients, and adversarial trajectories.AdaAD uses teacher-student discrepancy to generate adaptive adversarial examples for student training; CARD instead uses discrepancy in a separate stage to refine the teacher.Source: §2.2 · pp. 3–4

2.3. Teacher Adaptation and Weak-to-Strong Learning

  • PeerAiD and CIARD adapt supervisory models during student learning, whereas CARD refines the teacher after initial distillation using a frozen student as a stable feedback source.The refined teacher then supplies updated robustness supervision to the same student.Source: §2.3 · p. 4
  • Weak-to-strong studies typically aim to improve the stronger model itself; CARD uses that feedback as an intermediate step toward improving compact-student robustness.CARD’s post-distillation refinement decouples feedback from evolving student-training dynamics.Source: §2.3 · p. 4

3. Methodology

CARD cycles through strong-to-weak distillation, weak-to-strong teacher refinement, and a return transfer to the same student. It uses frozen student feedback to refine the teacher and retains the selected base distillation objective for student training.

3. Methodology

  • CARD first distills a compact student, freezes it to refine the robust teacher, then uses the refined teacher to train that same student further.The workflow proceeds from T(0) and S(0) to S(1), T(1), and final student S(2).Source: §3 · pp. 4–5

3.1. Strong-to-Weak Adversarial Distillation

  • The initial student stage uses the selected adversarial-distillation method’s objective and adversarial-example generation procedure, producing task-adapted feedback for later refinement.Other adversarial-distillation objectives can be substituted without changing the subsequent CARD workflow.Source: §3.1 · p. 5

3.2. Weak-to-Strong Teacher Refinement

  • CARD freezes the initially distilled student as a stable feedback source and targets two refinement problems: propagating student errors and selecting informative inputs.Inverse adversarial student feedback and teacher-student discrepancy maximization address these issues, respectively.Source: §3.2 · p. 5

3.2.1. Inverse adversarial student feedback

  • Inverse adversarial feedback finds a nearby input that reduces the student’s ground-truth classification loss, then uses that prediction to supervise the teacher on the clean input.Reducing the loss does not guarantee every resulting prediction is correct; the reliability analysis addresses when alignment preserves the ground-truth prediction.Source: §3.2.1 · pp. 5–6

3.2.2. Teacher-student discrepancy maximization

  • Discrepancy maximization selects perturbed inputs where the teacher’s prediction differs from the frozen student’s clean prediction.The resulting x_adv is a discrepancy-maximizing perturbed input relative to that clean reference.Source: §3.2.2 · p. 6

3.2.3. Teacher-refinement objective

  • Teacher refinement combines a robust branch aligning teacher predictions on discrepancy-selected perturbations with a natural branch aligning clean-input teacher predictions to inverse-adversarial student predictions.The frozen student provides fixed supervision while only teacher parameters are updated.Source: §3.2.3 · p. 6

3.3. Strong-to-Weak Gain Transfer

  • After refinement, CARD freezes T(1) and resumes training from S(1), transferring updated teacher supervision to the same student with the unchanged base objective.The final student S(2) completes the feedback loop without initializing a new student.Source: §3.3 · p. 6

3.4. Reliability Analysis of Weak-to-Strong Feedback

  • Teacher alignment preserves a student’s correct prediction when its feedback has a positive ground-truth margin and the remaining KL misalignment is sufficiently small; the risk bound is distribution-specific and does not guarantee risk reduction.For correctly classified student feedback, a larger margin or smaller KL divergence reduces the margin-normalized misalignment term; the robust-branch analysis concerns discrepancy-selected perturbations, not the entire perturbation set.Source: §3.4 · pp. 7–8

4. Experiments

The experiments evaluate CARD’s effectiveness and generalizability across datasets, student architectures, and adversarial-distillation methods, using robustness evaluations and follow-up analyses.

4. Experiments

  • CARD is evaluated under both white-box and transfer-based black-box robustness attacks.Source: §4 · p. 8
  • The evaluation spans different datasets, student architectures, and base adversarial-distillation methods.Source: §4 · p. 8
  • Ablation studies, hyperparameter analysis, and qualitative visualization complement the robustness experiments.Source: §4 · p. 8

4.1. Experimental Setup

The setup evaluates CARD against matched baselines across three datasets, multiple teacher-student architectures, and adversarial-distillation methods, with aligned training and evaluation protocols.

4.1. Experimental Setup

  • Baseline methods and their CARD variants use the same dataset split, architecture, and base distillation configuration unless otherwise stated.Source: §4.1 · p. 8

4.1.1. Datasets and preprocessing.

  • Experiments use CIFAR-10, CIFAR-100, and a fixed ten-class TinyImageNet subset, with original train/test partitions for CIFAR and original train/validation partitions for TinyImageNet.TinyImageNet-10 uses 5,000 training and 500 evaluation images; training uses random crops and horizontal flips, while evaluation uses no augmentation.Source: §4.1.1 · p. 8

4.1.2. Architectures and baselines.

  • WRN-28-10 and WRN-34-10 serve as robust teachers, while ResNet-18 and MobileNet-V2 serve as students; comparisons include adversarial-training and robustness-distillation baselines.CARD is instantiated on AdaAD and AdaAD+IGDM, with one or two refinement-transfer cycles.Source: §4.1.2 · p. 8

4.1.3. Training and implementation details.

  • During teacher refinement, the initial student remains frozen and only teacher parameters are updated; the refined teacher then supervises continued training of that same student.Student training resumes from its first-round checkpoint, following the base method’s objective and adversarial-example procedure.Source: §4.1.3 · pp. 8–9

4.1.4. Evaluation protocol.

  • White-box tests use attacks directly against the target model, while black-box tests transfer examples from independent surrogates using the same attacks for each baseline and CARD variant.White-box evaluation includes FGSM, PGD-20, CW-20, and AutoAttack; black-box evaluation uses PGD-100 and CW-100.Source: §4.1.4 · pp. 9–10

4.2. White-Box Robustness

CARD improves both clean and robust accuracy consistently on CIFAR-100, while CIFAR-10 and TinyImageNet-10 results vary by student architecture, metric, and base distillation objective.

4.2.1. Results on CIFAR-10.

  • On CIFAR-10 with WRN-28-10, CARD improves all five metrics for ResNet-18 on AdaAD+IGDM, while MobileNet-V2 results include clean-accuracy losses or metric-specific declines.For ResNet-18, the gains are 0.11, 0.91, 0.02, 0.54, and 0.17 percentage points; MobileNet-V2 loses 1.68 clean-accuracy points versus AdaAD+IGDM and has a 0.14-point PGD-20 decrease versus AdaAD.Source: §4.2.1 · p. 10
  • With WRN-34-10 on CIFAR-10, the ResNet-18 AdaAD+IGDM variant has the highest FGSM, PGD-20, CW-20, and AA accuracy among evaluated ResNet-18 methods.Its gains over the original AdaAD+IGDM baseline are 0.28, 0.35, 0.09, 0.41, and 0.45 percentage points for clean, FGSM, PGD-20, CW-20, and AA accuracy.Source: §4.2.1 · pp. 10–11
  • For MobileNet-V2, applying CARD once to the original AdaAD+IGDM baseline increases clean, FGSM, CW-20, and AA accuracy by 0.25, 0.75, 0.32, and 0.03 percentage points, respectively, while PGD-20 accuracy decreases by 0.18 percentage points.Source: §4.2.1 · p. 11

4.2.2. Results on CIFAR-100.

  • One CARD cycle consistently improves clean and robust accuracy over corresponding AdaAD and AdaAD+IGDM baselines across both evaluated student architectures on CIFAR-100.Source: §4.2.2 · p. 11, Table 3 · p. 11

4.2.3. Results on TinyImageNet-10.

  • On TinyImageNet-10, AdaAD+IGDM-based CARD improves all five reported accuracies and achieves the highest robust accuracy under all four attacks among evaluated student methods.Against its baseline, the gains are 1.40, 1.00, 0.40, 0.80, and 1.00 percentage points for clean, FGSM, PGD-20, CW-20, and AA accuracy.Source: §4.2.3 · p. 12, Table 4 · p. 12
  • On TinyImageNet-10, AdaAD- and ARD-based CARD variants show metric-dependent trade-offs, unlike the across-metric gains of the AdaAD+IGDM-based variant.AdaAD-based CARD lowers FGSM and PGD-20 accuracy, while ARD-based CARD lowers PGD-20 and AA accuracy versus their respective baselines.Source: §4.2.3 · p. 12

4.3. Black-Box Transfer Robustness

CARD improves transfer-based black-box robustness against attacks generated from robust and naturally trained surrogate models. The reported gains hold across all tested surrogate-attack combinations.

4.3.1. Results with Robust Surrogates.

  • CARD improves transferred-attack accuracy over AdaAD and AdaAD+IGDM across all six robust-surrogate combinations.Applying CARD twice over AdaAD yields gains of 0.52–1.02 percentage points; applying it once over AdaAD+IGDM yields 0.17–0.47 percentage points.Source: §4.3.1 · p. 13

4.3.2. Results with Natural Surrogates.

  • Against natural surrogates, CARD improves transferred-attack accuracy over AdaAD across all 12 combinations, with one-cycle gains of 0.59–0.82 percentage points.Two cycles improve all 12 combinations by 0.57–1.03 percentage points; the results span the natural surrogate architectures in Tables 6 and 7.Source: §4.3.2 · p. 13, Table 6 · p. 13, Table 7 · p. 13

4.3. Black-Box Transfer Robustness

  • Across attacks from multiple robust and naturally trained surrogate architectures, the results show consistent gains in transfer-based black-box robustness.The natural-surrogate evaluation uses PGD-100 and CW-100 examples transferred directly to the target student.Source: §4.3.2 · p. 13

4.4. Ablation Studies

The ablations test CARD’s cyclic components, sequential optimization, and teacher-refinement signals. They find additional benefits from teacher refinement, separated optimization, and the complete feedback objective.

4.4.1. Contribution of the Cyclic Components

  • Adding weak-to-strong teacher refinement to S2W-only training improves clean, CW-20, and AA accuracy by 1.15, 0.56, and 0.25 percentage points, respectively.Complete CARD exceeds AdaAD by 0.29 points in clean accuracy and 1.11 and 0.85 points under CW-20 and AA; the comparison indicates gains beyond additional student training alone.Source: §4.4.1 · p. 14, Table 8 · p. 14

4.4.2. Separated versus Joint Optimization

  • Separated optimization improves clean accuracy by 1.31 percentage points and PGD-20, CW-20, and AA accuracy by 0.07, 0.11, and 0.10 points over joint optimization.The robustness gains are modest but consistent across all three attacks.Source: §4.4.2 · p. 14, Table 9 · p. 14

4.4.3. Teacher-Refinement Signals

  • The complete CARD refinement objective achieves the highest robust accuracy under PGD-20, CW-20, and AA among the tested supervision variants.Compared with AdaAD, CARD improves clean, PGD-20, CW-20, and AA accuracy by 0.75, 0.46, 0.75, and 0.39 percentage points, respectively; alternatives do not improve all reported metrics over AdaAD.Source: §4.4.3 · p. 15, Table 10 · p. 15

4.5. Sensitivity to the Refinement Weight 𝜆

The refinement weight 𝜆 controls the balance between natural and robust refinement. Although changing it shifts the refined teacher’s natural–robust trade-off, the final student varies less across the evaluated values.

4.5. Sensitivity to the Refinement Weight 𝜆

  • Increasing 𝜆 generally raises natural accuracy and lowers robust accuracy for both first- and second-cycle refined teachers.Larger 𝜆 emphasizes natural refinement, while smaller values give relatively greater importance to robust refinement.Source: §4.5 · p. 15
  • The student’s performance after gain transfer varies less across evaluated 𝜆 values than the refined teacher’s performance.The effect of 𝜆 on the intermediate teacher does not translate into an equally large variation in final student performance.Source: §4.5 · p. 15
  • Validation performance sets 𝜆 to 0.2 for first-cycle refinement and 0.7 for second-cycle refinement.Within the evaluated range, final student performance remains relatively stable in this setting.Source: §4.5 · p. 15

4.6. Qualitative and Mechanistic Analysis

Qualitative analyses compare student attention under adversarial perturbations and examine inverse-adversarial feedback and the effects of refinement. CARD’s student shows object-focused activations closer to the robust teacher, while the confusion-matrix analysis reports reliable inverse-adversarial predictions and accuracy gains after refinement.

4.6.1. Grad-CAM Visualizations

  • On the visualized adversarial CIFAR-100 examples, CARD’s student has more concentrated object-relevant activation and greater qualitative similarity to the robust teacher than baseline students.The visualizations also show fragmented or background-focused activations for several baseline students.Source: §4.6.1 · p. 15, Figure 4 · p. 16

4.6.2. Confusion Matrix Analysis

  • All models exceed 96% accuracy on inverse-adversarial inputs, and teacher refinement raises teacher clean and adversarial accuracy from 84.57% and 58.32% to 84.63% and 59.10%, respectively.The final CARD student improves AdaAD from 84.05% to 84.15% on clean inputs and from 56.57% to 56.74% on adversarial inputs; refined teacher and student remain above 96% on inverse-adversarial inputs despite slight decreases.Source: §4.6.2 · p. 16, Figure 5 · p. 17

5. Conclusion

CARD closes the adversarial-distillation loop through student-feedback-based teacher refinement and subsequent gain transfer to the same student. It improves adversarial robustness in most evaluated settings, but gains are not uniform across attacks and base methods.

5. Conclusion

  • CARD cycles from strong-to-weak distillation through teacher refinement to gain transfer, using inverse-adversarial feedback and discrepancy-maximizing inputs.Source: §5 · p. 17
  • CARD improves adversarial robustness in most evaluated settings under both white-box and transfer-based black-box attacks.Source: §5 · p. 17
  • The evaluation is limited to relatively small-scale image-classification benchmarks and primarily ℓ∞-bounded attacks, with extra training cost from refinement and gain-transfer stages.Source: §5 · p. 17
  • Improvements are not uniform across attacks and base distillation methods; future work targets efficiency, scale, broader threat models, and adaptive feedback.Source: §5 · p. 18

Related papers

Loading 2610.19911v1…