Adversarial Machine Learning at Scale

Alexey Kurakin; Ian Goodfellow; Samy Bengio

Source-linked AI summary

Adversarial Machine Learning at Scale

Alexey Kurakin, Ian Goodfellow, Samy Bengio

arXiv:1611.01236v2Nov 4, 2016cs.CVcs.CRcs.LGstat.ML

TL;DR

Adversarial examples threaten machine-learning systems because they can transfer across models, enabling black-box attacks. This paper scales adversarial training to ImageNet and finds robustness to one-step attacks, while iterative attacks remain difficult to defend against.

Problem
Machine-learning models remain vulnerable to small input modifications, and transferability enables adversarial attacks without access to the target model.
Method
The paper applies adversarial training to an Inception v3 model trained on ImageNet and evaluates robustness across one-step and iterative attacks.
Results
Adversarial training increased robustness to tested one-step attacks, with adversarial top-1 and top-5 accuracy reaching 74% and 92%, respectively.
Takeaways & Limitations
One-step attacks are more transferable than iterative attacks, making them more effective for black-box attacks, while adversarial training offers indirect protection against iterative attacks.
Takeaways & Limitations
Training on one-step adversarial examples does not confer robustness to iterative adversarial examples, and iterative adversarial training was computationally costly without yielding benefits.

Abstract

from arXiv · show

Adversarial examples are malicious inputs designed to fool machine learning models. They often transfer from one model to another, allowing attackers to mount black box attacks without knowledge of the target model's parameters. Adversarial training is the process of explicitly training a model on adversarial examples, in order to make it more robust to attack or to reduce its test error on clean inputs. So far, adversarial training has primarily been applied to small problems. In this research, we apply adversarial training to ImageNet. Our contributions include: (1) recommendations for how to succesfully scale adversarial training to large models and datasets, (2) the observation that adversarial training confers robustness to single-step attack methods, (3) the finding that multi-step attack methods are somewhat less transferable than single-step attack methods, so single-step attacks are the best for mounting black-box attacks, and (4) resolution of a "label leaking" effect that causes adversarially trained models to perform better on adversarial examples than on clean examples, because the adversarial example construction process uses the true label and the model can learn to exploit regularities in the construction process.

1 INTRODUCTION

Adversarial examples can cause perceptually indistinguishable inputs to be misclassified and transfer across models, enabling black-box attacks. This paper studies adversarial training for ImageNet Inception models and reports robustness, transferability, capacity, and label-leaking findings.

Motivation: Adversarial examples are small input modifications that can cause neural networks to misclassify perceptually indistinguishable inputs.They are misclassified more often than examples perturbed by noise.
Motivation: Transferability allows adversaries to generate examples against one model and attack another without access to the target model.An example designed to fool M1 is often also misclassified by M2.
Contributions: Prior adversarial-training defenses were studied mainly on relatively small datasets, whereas this paper applies adversarial training to Inception models trained on ImageNet.The paper studies adversarial training of Inception models trained on ImageNet.
Contributions: Adversarial training successfully increases Inception v3 robustness against fast gradient sign and other one-step adversarial methods.The model was trained on the ImageNet dataset.
Contributions: Adversarial examples that are harder to resist with adversarial training are less likely to transfer between models.The paper also finds that higher-capacity models tend to be more robust than lower-capacity models of the same architecture.
Contributions: Label leaking occurs when models exploit regularities introduced by single-step adversarial construction using true labels, making those examples easier to classify than clean examples.The authors suggest construction processes that do not use the true label.

2 METHODS GENERATING ADVERSARIAL EXAMPLES

The section defines adversarial-image terminology and perturbation constraints, then describes one-step and iterative attack methods used in the experiments. Iterative methods use repeated gradient updates, with iterative least-likely attacks misclassifying more than 99% of cases.

Notation and terminology: Clean images are denoted X, while Xadv denotes a candidate adversarial image that may not be misclassified.A misclassified adversarial image is a candidate image misclassified by the network, typically paired with a correctly classified clean image.
Notation and terminology: The perturbation size ϵ is usually constrained by the L∞ norm and specified in pixel values ranging from 0 to 255.The clipping operator ClipX,ϵ(A) restricts each element of A to [Xi,j − ϵ, Xi,j + ϵ].
Attack methods: One-step methods generate candidate adversarial images after one gradient, whereas iterative methods apply many gradient updates and typically produce more harmful examples.FGSM is computationally efficient but usually has a lower success rate; on ImageNet, its candidate-image top-1 error is about 63%−69% for ϵ ∈ [2, 32].
Attack methods: More than 99% of cases are misclassified by the iterative least-likely class method, which repeatedly applies the one-step least-likely attack.Its α and iteration count are selected as for the basic iterative method.

3 ADVERSARIAL TRAINING

Adversarial training at ImageNet scale requires batch-normalization-aware minibatch construction and controlled mixing of clean and adversarial examples. The method also benefits from sampling perturbation magnitudes across training examples rather than fixing one value.

Scaling adversarial training: ImageNet-scale adversarial training uses batch normalization and groups examples into batches to scale beyond the small models used previously.The passage identifies batch normalization and batch grouping as important for successful scaling.
Minibatch construction: The training loss independently controls the number and relative weight of adversarial examples in each minibatch, using λ = 0.3 and m = 32.Here, m denotes total minibatch examples, while λ controls adversarial-example weight.
Training algorithm: The procedure generates k adversarial examples from corresponding clean examples, replaces those clean examples, and trains on the resulting minibatch until convergence.The described algorithm initializes the network, repeatedly constructs the mixed minibatch, performs one training step, and stops when training converges.
Minibatch construction: The ImageNet configuration uses k = 16 and replaces each clean example with its adversarial counterpart, maintaining a total minibatch size of 32.This differs from previous adversarial-training approaches that used entire minibatches of adversarial examples.
Perturbation magnitudes: Randomly sampling ϵ for each training example improves robustness across perturbation magnitudes, with best results from a truncated normal distribution on [0, 16] based on N(µ = 0, σ = 8).Fixing ϵ during training made networks robust only to that specific value.

4 EXPERIMENTS

ImageNet adversarial training improved robustness to one-step attacks, bringing adversarial accuracy near clean accuracy at a small clean-accuracy cost, but did not robustly address iterative attacks. The experiments also exposed label leakage and showed that transferability favors single-step attacks, especially FGSM.

One-step adversarial training: Adversarial training with any tested one-step method increased robustness to all tested one-step adversarial examples, though clean–adversarial accuracy gaps remained.The best test-set results came from “step l.l.” and “step rnd.”, so final experiments used “step l.l.”.
Scaling and accuracy: Adversarial training reduced clean-example accuracy by less than 1%, but increasing model size consistently improved robustness and twice-normal-sized models approached a 1 accuracy ratio for one-step attacks.Adding two Inception blocks slightly reduced the clean-image accuracy gap.
One-step adversarial training: Up to 74% top-1 and 92% top-5 adversarial accuracy brought performance near clean-image accuracy, while clean accuracy fell by about 0.8%.Disabling label smoothing and/or dropout produced a 0.1%–0.5% clean top-1 decrease and a 1%–1.5% adversarial top-1 increase.
Iterative attacks: Training on one-step adversarial examples did not confer robustness to iterative examples, while iterative adversarial training was computationally costly and failed to provide benefits without significantly reducing clean accuracy.The authors suggest that much larger models may be necessary for robustness to this broader input class.
Label leaking: FGSM training and evaluation caused label leaking, making adversarial-image accuracy much higher than clean-image accuracy because predictable label-dependent transformations reveal the true label.The effect vanished when construction did not use the true label or used an iterative method, so the authors recommend label-independent evaluation methods.
Transferability: FGSM examples were most transferable, whereas “iter l.l.” examples were least transferable but fooled the network in more than 99% of top-1 cases; transfer rates increased with ϵ.This gives black-box attackers an incentive to use higher ϵ, while suggesting an inverse relationship between transferability and fooling ability.

5 CONCLUSION

The paper studies improving adversarial robustness for large ImageNet models, showing that adversarial training helps against one-step attacks. Although it offers limited protection against iterative attacks, their examples transfer less readily between networks, providing indirect black-box robustness.

5 CONCLUSION: The study targets adversarial robustness for Inception v3 trained on ImageNet.It examines large models trained on large datasets.
5 CONCLUSION: Adversarial training provides robustness to adversarial examples generated using one-step methods.
5 CONCLUSION: Adversarial training offers limited protection against iterative methods, whose examples are less likely to transfer between networks.This reduced transferability provides indirect robustness against black-box attacks.

Appendices · A COMPARISON OF ONE-STEP ADVERSARIAL METHODS

The appendix compares one-step adversarial methods across loss-maximizing, target-class, and random-perturbation categories. It finds that training with one method transfers robustness across the others, while “step l.l.” and “step rnd.” provide the best clean and adversarial accuracy.

A COMPARISON OF ONE-STEP ADVERSARIAL METHODS: One-step methods fall into loss-maximizing, selected-target-probability, and random-perturbation categories.Loss-maximizing methods resemble FGSM; target-class methods resemble “step l.l.”.
A COMPARISON OF ONE-STEP ADVERSARIAL METHODS: Loss-maximizing methods include FGSM, FGSM-pred, “Fast entropy,” “Fast grad. L2,” and “Fast grad. L∞.”FGSM-pred uses the network’s predicted class; “Fast entropy” maximizes predictive entropy; gradient methods use normalized gradient values.
A COMPARISON OF ONE-STEP ADVERSARIAL METHODS: Target-class methods include “Step l.l.” toward the least likely class and “Step rnd.” toward a random class.“Step l.l.” uses the least likely class predicted by the network.
A COMPARISON OF ONE-STEP ADVERSARIAL METHODS: Random alternatives include sign-based perturbations and truncated-normal perturbations with zero mean and 0.5ϵ standard deviation.The truncated-normal perturbation is defined on [−ϵ, ϵ] with uncorrelated pixels.
A COMPARISON OF ONE-STEP ADVERSARIAL METHODS: Training with only one single-step method is sufficient to gain robustness to all evaluated single-step methods.Figure 3 reports accuracy on various one-step adversarial examples after training with only “step l.l.”.
A COMPARISON OF ONE-STEP ADVERSARIAL METHODS: “Step l.l.” and “step rnd.” produced the best combination of clean-data and adversarial-input accuracy during adversarial training.The appendix notes that one-step methods are not equally effective for adversarial training.
A COMPARISON OF ONE-STEP ADVERSARIAL METHODS: Some evaluation methods gain accuracy as ϵ increases over part of the curve because of label leaking.This behavior is reported in Figure 3’s comparison of one-step methods during evaluation.

B ADDITIONAL RESULTS WITH SIZE OF THE MODEL

This section adds Figure 4 on robustness to adversarial examples using top-5 accuracy, complementing Section 4.3’s analysis of model size. The top-5 results generally show the same properties as the corresponding top-1 accuracy plots.

Additional results with size of the model: Figure 4 reports robustness to adversarial examples using top-5 accuracy.It provides additional results alongside the model-size analysis in Section 4.3.
Additional results with size of the model: The top-5 accuracy robustness results generally exhibit the same properties as the corresponding top-1 accuracy plots.

C ADDITIONAL RESULTS ON TRANSFERABILITY

The section evaluates transferability by measuring how often adversarial examples generated for one model are misclassified by another. Transferred-example error rates show the same overall behavior as transfer rates, across adversarial methods and model pairs.

Evaluation approach: Transferability is assessed using the target model’s error rate on all candidate adversarial examples generated by a source model.This complements transfer rates computed only on adversarial examples already misclassified by the source model.
Evaluation approach: The evaluation models an attack that modifies all available images to maximize the number misclassified by the target model.The attacker does not try to select only especially effective adversarial images.
Evaluation approach: Error rates are computed by generating 1000 adversarial images with the source model and classifying them with the target model.The experiments compare results across adversarial methods, with additional results for fixed source and target models and varying perturbation sizes.
Results: Transferred adversarial examples exhibit the same overall error-rate behavior as the transfer rates reported in Section 4.4.Table 6 reports error rates rounded to the nearest percent for perturbation size ϵ = 16 across several Inception model variants and adversarial methods.

D RESULTS WITH DIFFERENT ACTIVATION FUNCTIONS

Adversarial training converged with all tested non-linear activation functions, but robustness and clean-image performance varied by activation. Several alternatives matched ReLU on clean images while improving accuracy against some iterative attacks, supporting their use as an additional defense.

Activation functions: Training converged for every tested activation, but test performance was not necessarily equivalent to ReLU.Thus, convergence alone did not guarantee comparable clean or adversarial accuracy.
Accuracy trade-offs: tanh and ReluDecayβ=0.1 lost about 2%-3% accuracy on clean examples and about 10%-20% on “step l.l.” adversarial examples.These activations therefore performed worse than ReLU on both evaluation types.
Accuracy trade-offs: ReLU6, ReluDecayβ=0.01, and ReluDecayβ=0.001 stayed within ±1% of ReLU on clean images while losing only a few percent on “step l.l.” images.These alternatives most closely matched ReLU’s reported performance across the clean and single-step evaluations.
Iterative attacks: All non-linear activation functions increased classification accuracy on some iterative adversarial images, indicating potential as an additional defense against iterative attacks.The paper summarizes this overall defense effect separately from the clean-image and “step l.l.” accuracy comparisons.

E RESULTS WITH DIFFERENT NUMBER OF ADVERSARIAL EXAMPLES IN THE MINIBATCH

Increasing the number of adversarial examples in each minibatch improves adversarial accuracy but reduces clean accuracy. More than half adversarial examples offers little additional adversarial benefit while causing up to 1% further clean-accuracy loss, making k = 16 a practical trade-off.

Results by minibatch composition: Increasing k raises accuracy on adversarial examples while decreasing accuracy on clean images.The study varied k, the number of adversarial examples in each minibatch.
Results by minibatch composition: For k > 16, adversarial accuracy improves insignificantly, while clean accuracy decreases by up to 1%.In this setup, more than half of the 32-example minibatch consists of adversarial examples when k > 16.
Results by minibatch composition: The experiments therefore use k = 16 as a reasonable trade-off between clean and adversarial accuracy.The total minibatch size is 32, so k = 16 allocates half the minibatch to adversarial examples.