Source-linked AI summary
Adversarial Attacks and Defences Competition
Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, Alan Yuille, Sangxia Huang, Yao Zhao, Yuzhe Zhao, Zhonglin Han, Junjiajia Long, Yerkebulan Berdibekov, Takuya Akiba, Seiya Tokui, Motoki Abe
TL;DR
Adversarial examples threaten machine-learning security because slight input changes can cause misclassification, while evaluating defenses against unknown attacks is difficult. The report presents the NIPS 2017 competition, its organization, and methods from leading teams. The competition produced stronger attacks and defenses, including a defense achieving 95% accuracy on adversarial images from all attacks, while new attack types still require additional defense research.
Problem
Machine-learning classifiers are vulnerable to subtly modified inputs, and evaluating defenses is difficult because unknown attacks may defeat defenses tested only against prepared attacks.
Method
The report describes a competition that independently evaluated black-box attacks and defenses, alongside methods developed by top-placing teams.
Results
95% accuracy was achieved by the top defense on all adversarial images produced by all attacks, while a new class of smoothed attacks fooled spatially smoothed defenses at 50-60% ratios.
Takeaways & Limitations
Competition results suggest that practical applications may achieve a reasonable level of robustness to adversarial examples in the black-box case.
Takeaways & Limitations
Additional study is needed to defend against newly developed smoothed attacks with high transferability.
Abstract
from arXiv · showhide
To accelerate research on adversarial examples and robustness of machine learning classifiers, Google Brain organized a NIPS 2017 competition that encouraged researchers to develop new methods to generate adversarial examples as well as to develop new ways to defend against them. In this chapter, we describe the structure and organization of the competition and the solutions developed by several of the top-placing teams.
1 Introduction
Machine-learning classifiers can be fooled by imperceptibly modified inputs, creating security and AI-safety concerns. The NIPS 2017 competition was organized to evaluate attacks and defenses through independent competition and document leading solutions.
- Motivation: Adversarial examples can cause classifiers to misclassify inputs after slight modifications that humans may not notice.These vulnerabilities affect most existing machine-learning classifiers.
- Motivation: Adversarial examples create security concerns because attacks can work without access to the underlying model.Related attacks have also been demonstrated against systems operating in the physical world through inaccurate sensors.
- Motivation: Robustness to adversarial examples is presented as an important part of the AI safety problem as machine-learning and AI systems become more powerful.The passage frames adversarial vulnerabilities as a potential route to compromising or controlling powerful AI systems.
- Competition rationale: Evaluation is difficult because attackers can draw from an unknown distribution, and a defense tested against known attacks may fail against an unanticipated attack.A competition provides an intermediate evaluation by pitting defenses against attacks developed by independent teams.
- Scope: The report covers adversarial-example research problems, competition organization, and methods developed by several top-placing competitors.These topics correspond to sections 2, 3, and 4 of the report.
2 Adversarial examples
Adversarial examples are intentionally optimized inputs whose effects depend on attack goals, model knowledge, and data access. The section surveys major attack methods, including optimization, gradient, iterative, generative, and transfer-based approaches.
- Concept and attack scenarios: Adversarial examples are intentionally optimized inputs designed to make a model misclassify, although the model may still classify them correctly.Accuracy or error rate can measure model behavior on a particular set of examples.
- Concept and attack scenarios: Attacks differ by outcome: non-targeted attacks seek any incorrect label, while targeted attacks seek a specified target class.The distinction concerns the adversary’s desired classifier output.
- Concept and attack scenarios: Attack scenarios also vary by model knowledge, from white-box access to black-box settings with or without probing.Probing allows an adversary to query the model and observe outputs, while white-box access includes architecture and parameter values.
- Attack methods: L-BFGS searches for adversarial examples through optimization, but it is slow and can be defeated by degrading image precision, such as rounding pixels to 8-bit values.Its objective is to find the smallest possible perturbation rather than counter defenses based on reduced precision.
- Attack methods: FGSM uses a linear approximation of the loss, while iterative methods trade FGSM’s lower success rate for more computation and can use about 40 iterations.BIM applies smaller gradient-sign steps and can be converted into a targeted attack; with sufficient iterations it almost always reaches the target class.
- Attack methods: Generative adversarial transformation networks can craft examples from clean images faster than explicit optimization when designed to be small.For non-differentiable systems, attackers can instead use substitute models, transferability, or gradient-free optimization with different computational trade-offs.
- Black-box attacks: Transferability allows examples that fool one model to fool another, with rates ranging from a few percent to almost 100% depending on the models, dataset, and other factors.Ensembles and intentionally transfer-prone source models are described as strategies for increasing transfer rates in black-box settings.
3 Adversarial competition
The competition was designed to compare attacks and defenses under independent evaluation, using new ImageNet-compatible data, three attack-or-defense tracks, and standardized scoring. Development rounds supported testing, while only the secret final round determined winners.
- Rationale: The competition addressed uncertainty about whether attack or defense results reflect method strength or inadequate benchmarking against the opposing side.Its purpose was to pit many proposed attacks and defenses against each other for more vigorous evaluation.
- Evaluation design: Attacks were ranked by how often they fooled defenses, while defenses were scored by correctly classifying generated adversarial examples.The evaluation ran every attack on the dataset and every defense on the resulting adversarial examples.
- Dataset: The dataset was designed to be large, non-trivial, familiar enough for reuse of classifiers, and composed of previously unused samples.New images were labeled with a pretrained ImageNet classifier and manually verified.
- Dataset: The competition used a 1000-image DEV dataset for development and a secret 5000-image FINAL dataset for final evaluation.The final dataset was kept secret and used to evaluate final submissions.
- Tracks and submissions: Three tracks covered non-targeted black-box attacks, targeted black-box attacks, and defenses.Participants submitted programs that produced adversarial images for attack tracks or classification labels for defense tracks.
- Operational constraints: Each batch of 100 images had a 500-second processing limit, and only successfully processed images contributed to scores.Submissions that exceeded the limit were interrupted and given the next batch.
- Evaluation metrics: Worst-case scores measured the minimum defense accuracy or attack misclassification rate across evaluated opponents.Worst-case targeted-attack scores were generally unhelpful because all were 0.
- Competition schedule: Development rounds were optional for solution testing, whereas only the final round computed final scores and determined winners.Rounds were evaluated using the same general scoring procedure, with DEV data during development and FINAL data at the end.
4 Competition results
The competition attracted many submissions across three tracks, with results improving over rounds. Best defenses and attacks performed strongly on average but had substantially weaker worst-case scores.
- 91 non-targeted attack, 65 targeted attack, and 107 defense submissions competed in the final round.
- Submissions improved their results relative to the provided baselines throughout the competition.
- Final results were reported for the top-five submissions, baselines, and selected worst-case performers in the three tracks.
- More than 90% accuracy was achieved by the best defenses on adversarial images from all attacks, but the highest worst-case defense score was only 53.6%.
- Best attacks achieved up to 78% fooling rate against all defenses, while their worst-case scores were no more than 34%.
5 Top scoring submissions
The chapter next presents solutions from several top-scoring teams and establishes notation for images, classifiers, losses, perturbations, and clipping operations.
- The remainder of the chapter presents solutions developed by several top-scoring teams.
- The notation distinguishes input images and labels, target classes, classifier logits, cross-entropy loss, perturbation limits, adversarial images, and clipping.
- Figure 2 plots submission scores by rank using solid lines and compares them with provided baselines shown as dashed lines.
- Images are normalized to [0,1], and epsilon values use the same normalized range.
5.1 1st place in defense track: team TsAIL
Team TsAIL won the defense track with high-level representation guided denoisers that remove adversarial noise using classifier-derived supervision.
- Team TsAIL’s high-level representation guided denoiser won first place in the defense track by learning to remove adversarial perturbations.
- The denoiser training set contained 20K ImageNet images distorted by FGSM and I-FGSM attacks on individual models and ensembles.
- TsAIL used a denoising U-net because its lateral connections preserve multi-scale image information and support noise prediction.
- Instead of reconstructing distance alone, HGD trains against classifier representations extracted at a selected network layer.
- The variants were feature guided, logits guided, and class label guided denoisers, with the last using fixed guiding-model parameters and ground-truth labels.
5.2 1st place in both attack tracks: team TsAIL
Team TsAIL’s momentum iterative method won both attack tracks by stabilizing iterative updates and improving black-box transferability through model ensembles.
- Team TsAIL’s momentum iterative gradient-based attack won first place in both the non-targeted and targeted attack tracks.
- Momentum was added to the basic iterative method to stabilize update directions, escape poor local optima, and improve transferability.
- Each iteration normalized the current gradient to unit L1 norm before updating the adversarial example in its sign direction.
- The non-targeted attack fused logits from an ensemble of normally trained, adversarially trained, and ensemble-adversarially trained models.
- Targeted attacks used a rounded and clipped update with values in {−2,−1,0,1,2}, enabling a larger search space than the sign function.
- No transferability was observed for the targeted attacks, so the method attacked several commonly used white-box models instead.
5.3 2nd place in defense track: team iyswim
Team iyswim defended against adversarial examples by adding random resizing and padding before classification, without retraining. Combined with adversarial training, the method ranked second in the defense challenge and exceeded adversarial training alone.
- Defense method: Random resizing and random padding were added at the beginning of classification networks as a defense against adversarial examples.The input is randomly resized, then padded, before classification.
- Defense method: The defense required no additional training or fine-tuning, added few computations, and was compatible with other defense methods.
- Motivation: Random transformations were hypothesized to disrupt perturbation structures over-fitted to specific network parameters, potentially improving defense against iterative attacks.
- Defense method: The resizing layer used a bounded random image size because larger changes could significantly reduce clean-image accuracy.For Inception-ResNet, the original input size was 299 × 299 × 3.
- Defense method: The padding layer inserted zeros around the resized image in random positions, yielding (W′′ − W′ + 1)×(H′′ − H′ + 1) possible patterns.
- Results: 0.924 normalized score across 156 attacks ranked the method No.2 among 107 defense models, versus 0.773 for ensemble adversarial training alone.
- Results: When attackers knew randomization layers existed, further experiments found they still could not completely break the defense in practice.
5.4 2nd place in both attack tracks: team Sangxia
Team Sangxia developed iterated FGSM attacks against classifier ensembles, using augmentation and random perturbations to improve robustness and transferability. The targeted variant omitted augmentation because it sharply reduced white-box success and ultimately transferred poorly.
- Approach: The submission used iterated FGSM attacks against classifier ensembles with random perturbations and augmentations for increased robustness and transferability.
- Non-targeted attack: The attack maximized expected average cross-entropy loss across multiple classifiers under random input augmentation.
- Non-targeted attack: Because the true label is typically unavailable, the attack substituted a pseudo-label generated by an accessible classifier.
- Non-targeted attack: Gradients over random transformations were approximated empirically by averaging gradients from several sampled transformations.
- Non-targeted attack: At ε = 16/255, many attacks transferred to held-out adversarially trained networks, while transfer rates on several undefended pretrained networks were close to or above 90%.
- Targeted attack: For targeted attacks, random augmentation was omitted because experiments found it severely reduced success even against white-box attacks.
- Targeted attack: The targeted attack did not transfer well and was not robust when success required outputting a specific target class.
5.5 3rd place in targeted attack track: team FatFingers
Team FatFingers proposed a dynamic iterative targeted attack that concentrates ensemble optimization on harder-to-perturb classifiers. Loss ensembles outperformed probability ensembles, and the method ranked third in the targeted-attack competition.
- Method: The proposed dynamic iterative targeted attack focused the loss ensemble on classifiers that were harder to perturb.
- Results: The method achieved third place among 65 attackers against 107 defenders in the targeted-attack competition.
- Targeted attack transfer: Black-box targeted attacks built on single models had hit rates close to zero against defenders based on different models.
- Targeted attack transfer: Because single-model targeted attacks transferred poorly, the method combined more known models to attack unknown models or ensembles.
- Ensemble attack methods: Loss ensembles replaced the difficult probability-ensemble objective with a Jensen upper bound that decoupled different networks’ parameters.
- Ensemble attack methods: Targeted attacks using loss ensembles had superior capability to probability ensembles at given iteration counts.
- Dynamic iteration: Adversarially trained models required significantly more iterations to reach high attack success than normal models.
- Dynamic iteration: Dynamic inclusion used δ_tk ∈ {0,1} to determine whether model k contributed its loss at attack step t.
5.6 4th place in defense track: team erko
Team erko’s fourth-place defense combined median filtering with adversarially trained model ensembles. The approach was robust against many black-box attacks, while median filtering alone failed against strong attacks.
- Defense architecture: Median filtering preceded an ensemble of adversarially trained models, whose predictions were averaged for the defense.The best results used all adversarially trained models with median filtering.
- Practical considerations: Median filtering did not harm clean-example classification accuracy and required no expensive training beyond the adversarially trained models.The defense was evaluated using a modified Carlini–Wagner L2 attack and a holdout model to test transferability and smoothing effects.
- Defense effectiveness: Median filtering alone could not defend against strong Carlini–Wagner attacks, but combining it with adversarially trained models produced a robust defense.The observation was based on experiments with adversarial attacks.
- Defense effectiveness: On a holdout Inception-v4 model, median filtering produced nearly the same misclassification ratio as no filtering.Other non-adversarially trained models showed the same pattern, with small differences between filtered and unfiltered inputs.
- Defense effectiveness: Adversarially trained models with median filtering were robust against black-box attacks and attacks generated by ensembles containing the same models.The attack-generation models lacked a filtering layer, so this was not exactly a white-box attack.
- Limitations: Later attacks using smoothed adversarial examples fooled spatially smoothed defenses at rates as high as 50–60%, with high transferability.The chapter states that additional study is needed to defend against these attacks.
5.7 4th place in non-targeted attack track: team iwiwi
Team iwiwi trained fully convolutional attack networks that transform clean images into bounded adversarial examples. The fourth-place method generalized across target models and ranked first in a third-party analysis of effectiveness against strong defenses.
- Approach: The team trained fully convolutional networks to convert clean examples into adversarial examples, earning fourth place in the non-targeted attack track.This approach differed from methods that directly optimize each example.
- Basic Framework: xadv = Clip[0,1](x+a(x;θa)) constructs an adversarial example by adding the attack FCN’s bounded output to a clean image and clipping the result.The attack FCN outputs tensors matching the input dimensions, with values in [−ε,+ε].
- Basic Framework: The attack FCN was trained by maximizing the target classifier’s loss, using the classifier as a target model.The perturbation function was differentiable and parameterized by θa.
- Training techniques: Multi-target training maximized losses across eight classifiers to improve adversarial-example generalization across image models.The targets included standard, adversarially trained, and ensemble-adversarially trained architectures.
- Training techniques: The method treated 13 possible ε values, from 4⁄256 through 16⁄256, as separate tasks so the FCN could control perturbation magnitude more finely.This multi-task formulation addressed limitations of simply applying tanh and scaling by ε.
- Results: The team ranked fourth among about one hundred teams and first in a PageRank-like third-party analysis, indicating effectiveness against strong defenses.The attack images canceled detailed textures and added jigsaw-puzzle-like patterns that induced the jigsaw-puzzle class.
6 Conclusion
The competition increased awareness of adversarial examples and encouraged new attack and defense approaches. Its results included improvements over baselines and evidence that black-box robustness may reach practical levels, though worst-case accuracy was lower than average accuracy.
- Conclusion: The competition’s goals were to increase awareness of adversarial examples and stimulate novel approaches.The chapter describes adversarial examples as an important machine-learning security problem.
- Conclusion: More than 100 teams competed in the final round, and the competition increased awareness through substantial public attention.The chapter cites publication of an MIT Technology Review article about the competition.
- Conclusion: Competitors achieved significant improvements over provided baselines in all three tracks by the competition’s end.The competition also pushed participants to explore new approaches and improve existing methods.
- Conclusion: The top defense submission achieved 95% accuracy on all adversarial images produced by all attacks, although worst-case accuracy was lower than average accuracy.The chapter presents this as evidence that practical black-box robustness may reach a reasonable level.