PaperlayerView exact revision on arXiv

Source-linked AI summary

Learnable Boundary Guided Adversarial Training

Jiequan Cui, Shu Liu, Liwei Wang, Jiaya Jia

arXiv:2011.11164v2cs.CV

TL;DR

Adversarial training can improve robustness while reducing clean-data accuracy. LBGAT uses clean-model logits to guide a robust model toward a useful classifier boundary, and experiments across three datasets report strong robustness, including state-of-the-art CIFAR-100 robustness without additional data.

  • Problem

    Adversarial training often improves robustness at the cost of natural-data accuracy.

  • Method

    LBGAT guides a robust model by matching logits from adversarial examples to logits from corresponding natural examples processed by a clean model.

  • Results

    Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet report strong performance, including state-of-the-art CIFAR-100 robustness under auto-attack without extra real or synthetic data.

  • Takeaways & Limitations

    Clean-model boundary guidance can preserve natural accuracy while also benefiting adversarial robustness within the evaluated settings.

  • Takeaways & Limitations

    The approach assumes a well-trained natural-data model whose logits capture discriminative classifier-boundary information.

Abstract

from arXiv · show

Previous adversarial training raises model robustness under the compromise of accuracy on natural data. In this paper, we reduce natural accuracy degradation. We use the model logits from one clean model to guide learning of another one robust model, taking into consideration that logits from the well trained clean model embed the most discriminative features of natural data, {\it e.g.}, generalizable classifier boundary. Our solution is to constrain logits from the robust model that takes adversarial examples as input and makes it similar to those from the clean model fed with corresponding natural data. It lets the robust model inherit the classifier boundary of the clean model. Moreover, we observe such boundary guidance can not only preserve high natural accuracy but also benefit model robustness, which gives new insights and facilitates progress for the adversarial community. Finally, extensive experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet testify to the effectiveness of our method. We achieve new state-of-the-art robustness on CIFAR-100 without additional real or synthetic data with auto-attack benchmark \footnote{\url{https://github.com/fra31/auto-attack}}. Our code is available at \url{https://github.com/dvlab-research/LBGAT}.

1. Introduction

Adversarial training improves robustness but often degrades clean-data accuracy. LBGAT guides a robust model with a naturally trained model’s classifier boundary, preserving natural accuracy while maintaining or improving robustness.

  • Adversarial training commonly trades higher robustness for greater degradation in natural-data accuracy.The paper frames this trade-off as the central challenge motivating its method.
  • LBGAT matches robust-model logits on adversarial examples to clean-model logits on corresponding natural examples.This constrains the robust model using discriminative information and classifier-boundary structure learned from natural data.
  • The method dynamically trains the clean and robust models together so the clean model’s boundary becomes more robustness-friendly.This extends static boundary guidance into Learnable Boundary Guided Adversarial Training.
  • Boundary guidance is incorporated into ALP and TRADES, yielding improvements over both baselines and supporting the method’s flexibility.The paper also reports that boundary guidance can enhance robustness in addition to preserving natural accuracy.

2. Related Work

Related work covers adversarial attacks, black-box transferability, defensive training, and knowledge distillation. LBGAT differs by using a naturally trained model’s robustness-friendly boundary rather than transferring robustness from a large adversarially trained teacher.

  • Auto-attack combines APGD-CE, APGD-DLR, FAB, and Square Attack as a popular robustness benchmark.
  • Adversarial-defense research includes input transformations, randomization, adversarial training, and output regularization such as TRADES.Mixture BN is another approach, but it assumes inference-time knowledge of whether inputs are natural or adversarial.
  • LBGAT uses a naturally trained model’s robustness-friendly classifier boundary to guide adversarial training, rather than transferring robustness from a large adversarially trained model.

3. Our Method

The method guides a robust model with clean-model logits so it can inherit a useful classifier boundary while retaining natural accuracy. LBGAT learns the clean and robust models collaboratively and can be combined with ALP and TRADES.

  • 3.1. Boundary Guided Adversarial Training: PGD generates adversarial examples, and the robust model is trained to maintain robustness while preserving high accuracy on natural images.The adversarial perturbation is bounded by l∞, and the min-max explanation links correcting xadv to correctly classifying its nearby natural example x.
  • 3.1. Boundary Guided Adversarial Training: The guidance matches Mrobust(xadv) logits to Mnatural(x), encouraging the robust model to inherit discriminative boundary information from natural data.The logit matching uses an MSE loss; Mnatural is trained on natural data in the static-boundary version.
  • 3.2. Learnable Boundary Guided Adversarial Training: LBGAT jointly trains Mnatural and Mrobust so the clean model adaptively learns a robustness-friendly classifier boundary.The robust model is guided by logits from adversarial inputs, while the natural model is optimized with cross-entropy and the logit constraint.
  • 3.3. Boundary Guidance Improving Robustness: Boundary guidance also improves robustness, including a reported 2.44% improvement on CIFAR-100 under the strongest auto-attack.The paper attributes this improvement to guidance from the natural classifier boundary.
  • 3.4. Model Flexibility: LBGAT can be integrated into ALP and TRADES by replacing their first loss term with the proposed boundary-guidance loss.The paper reports improvement over these baselines and describes the method as a non-overlapping training scheme.

4. Experiments

Experiments evaluate LBGAT across CIFAR-10, CIFAR-100, and Tiny ImageNet under white- and black-box attacks, with ablations examining boundary inheritance, batch normalization, and compatibility with existing defenses.

  • 4.1. Natural Classifier Boundary Inheritance: Replacing MSE with KL-divergence degrades performance, while removing the additional robust-model cross-entropy loss yields 2.05% higher robust accuracy.
  • 4.1. Natural Classifier Boundary Inheritance: LBGAT inherits classifier-boundary information from a naturally trained model, benefiting performance on both natural and adversarial data.
  • 4.1.3. Separate Batch Normalization: Robustness drops when convolution weights are shared between the natural and robust models with separate batch normalization.
  • 4.1.4. Effectiveness of Our Method: Both BGAT and LBGAT significantly outperform vanilla adversarial training, with BGAT favoring natural accuracy and LBGAT favoring robustness.
  • 4.1.5. Combining with ALP and TRADES: Combining boundary guidance with ALP or TRADES further improves robustness, including LBGAT+ALP gains of 2.92% in natural accuracy and 6.31% in white-box robust accuracy on CIFAR-100.

4.2. Robustness on CIFAR-10 and CIFAR-100

On CIFAR-10 and CIFAR-100, LBGAT variants improve the natural-accuracy/robustness trade-off against TRADES across regular and Auto-Attack evaluations, while also transferring robustness under black-box attacks.

  • White-box Regular Attacks: On CIFAR-10, LBGAT+TRADES achieves 88.22% natural accuracy and 57.55% robust accuracy, exceeding TRADES by 3.3% and 0.94%.
  • White-box Regular Attacks: Under regular white-box attacks, LBGAT+TRADES reaches 70.03% natural and 33.01% robust accuracy on CIFAR-100, improving TRADES by 13.53% and 2.08%.
  • White-box Regular Attacks: LBGAT variants consistently outperform TRADES in natural accuracy and robustness under additional FGSM and CW evaluations on CIFAR-10 and CIFAR-100.
  • White-box Auto-Attack: Under white-box Auto-Attack, LBGAT+TRADES improves TRADES natural accuracy by 13.53% on CIFAR-100 and 3.30% on CIFAR-10 while maintaining comparable robustness.
  • Black-box Attacks: The best model is more robust to black-box attacks transferred from naturally trained models and TRADES while also providing stronger white-box robustness and higher natural-image performance.

4.3. Robustness on Tiny-ImageNet.

On Tiny ImageNet, LBGAT improves over ALP and TRADES under 20-iteration PGD white-box evaluation, improving either natural or adversarial accuracy depending on the TRADES setting.

  • 4.3. Robustness on Tiny-ImageNet.: LBGAT+TRADES improves the strongest TRADES baseline by 9.29% on natural data, while LBGAT+TRADES further improves adversarial accuracy by 3.00%.

5. Conclusion

The paper proposes LBGAT to improve robustness without substantial natural-accuracy loss, interpreting the approach as guidance from the natural classifier boundary. Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet support its effectiveness.

  • LBGAT improves model robustness without losing much accuracy on natural data.
  • The method is understood through guidance from a naturally trained model’s classifier boundary.
  • Boundary guidance from a naturally trained model was observed to enhance model robustness.
  • Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet supported the effectiveness of the methods.

A. Robustness under Black-box attack

Table 7 compares defense models under black-box attacks on CIFAR-100 and CIFAR-10 using averaged results from two independently trained models.

  • Table 7 covers black-box attack comparisons on CIFAR-100 and CIFAR-10.
  • Results are averaged over two independently trained models to rule out randomness.
  • The table reports natural-image accuracy, black-box robustness, and white-box robustness.

B. Our Method Creates New SOTA Under the Strongest Auto-Attack on CIFAR-100

On CIFAR-100 under Auto-Attack, LBGAT establishes new state-of-the-art results in both robustness and natural accuracy, including gains over prior methods and an ImageNet-pretrained model.

  • LBGAT achieves new state-of-the-art CIFAR-100 results in both robustness and natural accuracy.
  • 7.08% higher natural-data performance than prior SOTA [6] is achieved by LBGAT (α = 0) with WideResNet-34-10.
  • LBGAT (α = 6) with WideResNet-34-10 achieves 2.4% higher robustness than [6].
  • LBGAT (α = 6) surpasses the full-ImageNet-pretrained model of Hendrycks et al. in robustness and natural accuracy.

C. More Comparisons Under the Strongest Auto-Attack on CIFAR-10

Under Auto-Attack on CIFAR-10, LBGAT (α = 0) with WideResNet-34-10 maintains the strongest robustness while achieving higher natural performance than compared methods.

  • LBGAT (α = 0) with WideResNet-34-10 achieves higher natural performance while keeping the strongest robustness on CIFAR-10.
  • Fast adversarial-training methods such as [45] are reported to have usually unsatisfactory performance.
  • The CIFAR-10 comparison uses the strongest Auto-Attack benchmark.
Loading 2011.11164v2…