PaperlayerView exact revision on arXiv

Source-linked AI summary

Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets

Yogesh Balaji, Tom Goldstein, Judy Hoffman

arXiv:1910.08051v1cs.LGcs.CVstat.ML

TL;DR

Adversarial training can improve robustness while reducing clean accuracy, and uniform perturbation radii may impose unsuitable constraints on samples near class boundaries. The paper proposes instance adaptive adversarial training with estimated, sample-specific radii; it reports improved robustness–accuracy tradeoffs across CIFAR-10, CIFAR-100, and ImageNet, with a marginal robustness drop in its overall conclusion.

  • Problem

    Adversarial training can have poor clean-test accuracy, and a uniform radius may be too large for some samples while unnecessarily small for others.

    Source: §1 · p. 1

  • Method

    Instance adaptive adversarial training estimates a separate perturbation radius for each training sample and imposes label-consistency constraints within those radii.

    Source: §3 · p. 3, §6 · p. 8

  • Results

    IAAT improves the robustness–accuracy tradeoff on CIFAR-10, CIFAR-100, and ImageNet; on CIFAR-10, clean-test accuracy increases by 4.06% for Resnet-18 and 4.49% for WideResnet-32-10 at similar adversarial robustness.

    Source: §4.1 · p. 5, §6 · p. 8

  • Takeaways & Limitations

    The learned radii are interpretable: small values often correspond to ambiguous images, while large values correspond to images with unambiguous class labels.

    Source: §1 · p. 2

Abstract

from arXiv · show

Adversarial training is by far the most successful strategy for improving robustness of neural networks to adversarial attacks. Despite its success as a defense mechanism, adversarial training fails to generalize well to unperturbed test set. We hypothesize that this poor generalization is a consequence of adversarial training with uniform perturbation radius around every training sample. Samples close to decision boundary can be morphed into a different class under a small perturbation budget, and enforcing large margins around these samples produce poor decision boundaries that generalize poorly. Motivated by this hypothesis, we propose instance adaptive adversarial training -- a technique that enforces sample-specific perturbation margins around every training sample. We show that using our approach, test accuracy on unperturbed samples improve with a marginal drop in robustness. Extensive experiments on CIFAR-10, CIFAR-100 and Imagenet datasets demonstrate the effectiveness of our proposed approach.

1 INTRODUCTION

Adversarial training improves robustness but can reduce accuracy on clean inputs, motivating a better robustness–accuracy tradeoff. The paper proposes instance-specific perturbation radii to avoid overly stringent constraints near class boundaries while retaining larger margins where possible.

1 INTRODUCTION

  • Adversarial training uses perturbed training samples to encourage nearly constant network outputs within an ℓp ball around each input.Source: §1 · p. 1
  • Clean accuracy can drop by as much as 10% on CIFAR-10 and 15% on ImageNet under adversarial training, motivating concern about a robustness–accuracy tradeoff.Prior work characterized this tradeoff theoretically.Source: §1 · p. 1
  • A uniform ϵ = 8 can be too large for boundary-near samples yet unnecessarily small for samples farther from the decision boundary.Overlapping perturbation balls can include images from another class, making label consistency within every fixed-radius ball infeasible.Source: §1 · p. 1
  • Instance adaptive training assigns sample-specific radii, improving the robustness–accuracy tradeoff beyond the standard adversarial-training Pareto frontier.Small radii often mark ambiguous samples, while large radii tend to mark images with unambiguous class labels.Source: §1 · p. 2
  • Unlike Ding et al. (2018), this work targets the robustness–accuracy tradeoff and uses a substantially different algorithm for choosing adaptive margins.Source: §1 · p. 2

2 BACKGROUND

The background introduces adversarial attacks and adversarial training as a defense that trains on perturbed inputs. It describes training as a min-max optimization solved with alternating updates and PGD for the inner maximization.

2 BACKGROUND

  • Adversarial attacks add norm-constrained perturbations to inputs to cause classifier misclassification, and common methods include FGSM, PGD, C/W, and DeepFool.These methods use input-loss gradients; alternative metrics include spatial transformer attacks and Wasserstein distance in pixel space.Source: §2 · p. 2
  • Adversarial training remains fairly resistant to most existing attacks, unlike several early defenses that strong attacks broke.Source: §2 · p. 2
  • In adversarial training, the network is optimized over adversarially perturbed rather than clean training samples through a min-max problem.The dataset consists of labeled training samples, and the labels are represented as one-hot vectors in the formulation.Source: §2 · p. 3
  • The min-max problem is solved by alternating stochastic minimization steps for θ with maximization steps approximated using k steps of PGD.Source: §2 · p. 3

3 INSTANCE ADAPTIVE ADVERSARIAL TRAINING

Instance Adaptive Adversarial Training (IAAT) replaces a uniform perturbation radius with one radius per training image. The method adjusts each radius using PGD outcomes, with a uniform-radius warmup before adaptation begins.

3 INSTANCE ADAPTIVE ADVERSARIAL TRAINING

  • IAAT imposes label-consistency constraints within estimated, sample-specific perturbation radii rather than a uniform radius.Source: §3 · p. 3
  • Each training batch is perturbed using its images’ respective ϵi values, and the network is updated on the perturbed images.Source: §3 · p. 3
  • After each epoch, a successful PGD attack decreases that image’s radius by γ, while failure increases it by γ.The aim is to make each radius as large as possible without finding a different-class image within its ball.Source: §3 · p. 3
  • A uniform-ϵ warmup, usually 10 epochs on CIFAR-10/100, precedes instance adaptation because random initial predictions would otherwise rapidly shrink the radii.Source: §3 · p. 3
  • The radius search tests the previous radius and its ±γ neighbors with PGD, then smooths the selected value using β.The figure contrasts small assigned ϵ for boundary-near images that change class at ϵ ≥ 8 with large ϵ for images that retain class information under very large perturbations; ϵ ranges from 0 to 255.Source: §3 · p. 4, Figure 2 · p. 4

4 EXPERIMENTS

Experiments on CIFAR and ImageNet show that instance adaptive adversarial training improves clean accuracy while retaining much of adversarial robustness. On CIFAR it also performs well across test perturbation radii and assigns smaller radii to visually ambiguous images.

4.1 CIFAR

  • CIFAR experiments trained models with PGD-10 and evaluated white-box PGD attacks, transfer attacks, and accuracy under 19 image corruptions.The robustness-accuracy comparison spans fixed training ϵ values from 1/255 to 8/255; two adaptive-training variants were reported, with warmup retaining more robustness but lower natural accuracy than no warmup.Source: §4.1 · pp. 4–5
  • Across CIFAR test ϵ regimes, instance adaptive training slightly trails the ϵ = 2 model at small ϵ and dominates all models at larger ϵ.Fixed-ϵ baselines trade off performance across regimes: ϵ = 8 performs well at high test ϵ, while ϵ = 2 performs well at low test ϵ.Source: §4.1 · p. 5, Figure 4 · p. 6
  • CIFAR images assigned smaller ϵ_i were often visually ambiguous, while those assigned larger ϵ_i had visually definite classes and changed little under an ϵ = 16 perturbation.The authors report that adaptive ϵ_i values correlate with their human concept of class ambiguity.Source: §4.1 · p. 5
  • CIFAR clean-test accuracy gains were 4.06% and 4.49% on CIFAR-10, and 8.79% and 9.22% on CIFAR-100, with similar robustness or a marginal drop.The gains are for ResNet-18 and WideResNet-32-10, respectively, on each dataset.Source: §4.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.2 IMAGENET

  • ImageNet evaluation used random targeted attacks, 30-step PGD training, a fixed-ϵ = 16/255 baseline, and PGD-1000 attacks at test ϵ values 4, 8, 12, and 16.The evaluation also reported normalized mean corruption error (mCE), for which lower values are better.Source: §4.2 · pp. 6–7
  • ImageNet natural accuracy improved by 10+% over adversarial training across ResNet-50, ResNet-101, and ResNet-152, with stronger low-ϵ robustness but a 13% drop at ϵ = 16.The model also consistently outperformed adversarial training on the corruption dataset.Source: Table 3 · p. 7, §4.2 · p. 7

5 ABLATION EXPERIMENTS

Warmup improves adversarial robustness with a small natural-accuracy drop, especially on CIFAR-100, while both warmup settings improve the accuracy–robustness tradeoff. During training, average ε converges near 11, but individual samples follow different ε trajectories.

5.1 EFFECT OF WARMUP

  • Warmup improves adversarial robustness with a small natural-accuracy drop, more prominently on CIFAR-100; both settings improve the accuracy–robustness tradeoff.The provided Table 5 caption identifies the CIFAR-100 warmup ablation; the prose reports the outcome for Tables 4 and 5.Source: §5.1 · p. 7, Table 5 · p. 8

5.2 VISUALIZING ϵ PROGRESS

  • Average ε converges to around 11, while individual samples diverge, with some ending above and others below the commonly used radius of 8.The average and three randomly chosen samples are plotted over epochs in Figure 5.Source: §5.2 · p. 8, Figure 5 · p. 8
  • The ε histogram spreads out as training progresses, consistent with the differing per-sample ε profiles.The passage reports snapshots of the ε histogram and varying trajectories across three samples.Source: §5.2 · p. 8

6 CONCLUSION

The conclusion presents instance adaptive adversarial training as a way to improve the robustness–accuracy tradeoff by imposing estimated, sample-specific label-consistency radii. It reports empirical gains on CIFAR-10, CIFAR-100, and Imagenet.

6 CONCLUSION

  • Instance adaptive adversarial training imposes label-consistency constraints within estimated, sample-specific radii, reflecting that realizable robustness varies by sample.The conclusion contrasts samples near decision boundaries, which can achieve only small radii, with farther samples that can tolerate larger radii.Source: §6 · p. 8
  • The approach is reported to improve the robustness–accuracy tradeoff on CIFAR-10, CIFAR-100, and Imagenet.The conclusion describes this as an empirical result.Source: §6 · p. 8

A.1 COMPARISON WITH MIXUP

The comparison finds that mixup improves natural accuracy, but under stronger attacks its adversarial-accuracy drop is much higher than IAAT’s. The supplied Table 6 is the mixup comparison.

A.1 COMPARISON WITH MIXUP

  • Under stronger attacks, mixup’s adversarial-accuracy drop is much higher than IAAT’s, despite mixup’s natural-accuracy improvement.The paper notes that mixup’s robustness experiments had only used PGD-20; its comparison with IAAT evaluates stronger attacks.Source: §A.1 · p. 11, Table 6 · p. 11

B SAMPLE VISUALIZATION

The visualizations associate low assigned ε values with confusing or boundary-near samples, while the Imagenet sweep finds robustness saturation beyond 500 PGD iterations. The visual examples also illustrate how IAAT selects a radius that retains label information.

B SAMPLE VISUALIZATION

  • CIFAR-10 samples assigned low ε are visually confusing, whereas those assigned high ε distinctively belong to one class.These assignments are illustrated in Figure 6.Source: B SAMPLE VISUALIZATION · p. 11
  • Near-boundary examples contain a different-class sample within the commonly used ℓ∞ radius of 8, and IAAT chooses an ε that retains label information.The passage describes this as apparent in Figure 7.Source: B SAMPLE VISUALIZATION · p. 11

C IMAGENET SWEEP OVER PGD ITERATIONS

D.1 CIFAR

The CIFAR appendix describes PGD-10 adversarial-example generation and visualizes training perturbations and instance-assigned radii.

D.1 CIFAR

  • PGD-10 adversarial examples are used to update models in the CIFAR-10 and CIFAR-100 training setting.The implementation follows the standard adversarial training setting used in Madry et al. (2018).Source: §D.1 · p. 11
  • The appendix also presents training-sample perturbations, a histogram of training-sample ϵ across epochs, and CIFAR hyperparameter tables.Figure 6 states that all ϵ values lie in [0, 255].Source: Figure 6 · p. 12, Table 7 · p. 12, Figure 8 · p. 13
  • Samples assigned low ϵ are close to the decision boundary, change class under ϵ ≥8, and retain class information under their IAAT-assigned perturbations.Figure 7 visualizes these samples.Source: Figure 7 · p. 13

D.2 IMAGENET

The ImageNet implementation uses PGD-30 adversaries, distributed synchronized training, and two stated speed-up measures for instance-adaptive training.

D.2 IMAGENET

  • ImageNet training generates adversaries with PGD-30 and uses synchronized SGD across 64 or 128 GPUs to make training feasible.The implementation follows Xie et al. (2019) and Goyal et al. (2017), uses FP-16 for training attacks, and FP-32 at test time.Source: §D.2 · p. 14
  • Two speed-up measures are a PGD-10 attacker for selecting ϵ and clipping each selected ϵ_i to a lower bound of 4.The lower-bound operation is ϵ_i ← max(ϵ_i, ϵ_lb), with ϵ_lb = 4 in the experiments.Source: §D.2 · p. 14
  • ResNet-50 used 64 Nvidia V100 GPUs, while ResNet-101 and ResNet-152 used 128 GPUs; training times are reported in Table 9.The ImageNet hyperparameters are reported in Table 8, and the models were trained in PyTorch.Source: §D.2 · pp. 14–15, Table 9 · p. 15

Related papers

Loading 1910.08051v1…