PaperlayerView exact revision on arXiv

Source-linked AI summary

Delving into Transferable Adversarial Examples and Black-box Attacks

Yanpei Liu, Xinyun Chen, Chang Liu, Dawn Song

arXiv:1611.02770v3cs.LG

TL;DR

The paper addresses limited evidence on adversarial-example transferability for large models and targeted attacks with specified labels. It studies both transfer types, proposes ensemble-based generation methods, and reports improved transfer—including successful attacks on the black-box system Clarifai.com.

  • Problem

    Prior transferability studies mostly used small datasets, leaving transfer across large models and targeted transfer with target labels insufficiently studied despite risks to vision applications.

  • Method

    The paper extensively evaluates non-targeted and targeted adversarial examples on large ImageNet models, develops ensemble-based generation approaches, and studies model geometry and black-box transfer.

  • Results

    Non-targeted examples transfer prominently, existing targeted methods rarely preserve target labels, and ensemble methods achieve targeted transfer while attacking Clarifai.com with both transfer types.

  • Takeaways & Limitations

    Transferable adversarial examples remain feasible at ImageNet scale, and ensemble-based generation extends transfer to targeted attacks and an unknown online classifier.

Abstract

from arXiv · show

An intriguing property of deep neural networks is the existence of adversarial examples, which can transfer among different architectures. These transferable adversarial examples may severely hinder deep neural network-based applications. Previous works mostly study the transferability using small scale datasets. In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels. We study both non-targeted and targeted adversarial examples, and show that while transferable non-targeted adversarial examples are easy to find, targeted adversarial examples generated using existing approaches almost never transfer with their target labels. Therefore, we propose novel ensemble-based approaches to generating transferable adversarial examples. Using such approaches, we observe a large proportion of targeted adversarial examples that are able to transfer with their target labels for the first time. We also present some geometric studies to help understanding the transferable adversarial examples. Finally, we show that the adversarial examples generated using ensemble-based approaches can successfully attack Clarifai.com, which is a black-box image classification system.

1 INTRODUCTION

The paper studies adversarial-example transferability across large ImageNet models, including targeted transfer with target labels, and develops ensemble-based methods for black-box attacks. It finds non-targeted transfer is prominent, while existing targeted methods rarely transfer; ensemble approaches improve targeted and non-targeted transfer and succeed against Clarifai.com.

  • Study scope: The study evaluates non-targeted and targeted adversarial-example transferability across state-of-the-art models trained on the large-scale ImageNet dataset.Targeted transfer requires the other model to classify an example as the specified target label.
  • Ensemble methods: Ensemble-based approaches generate targeted examples that transfer with their target labels and improve non-targeted transferability over the examined methods.The authors report a large proportion of targeted examples transferring for the first time.
  • Geometric analysis: Geometric analysis finds orthogonal gradient directions across models and well-aligned decision boundaries, partially explaining adversarial-example transfer.These observations extend geometric analysis to large ImageNet models with 1000 labels.
  • Black-box attack: Both non-targeted and targeted adversarial images generated by the proposed approaches transfer to Clarifai.com despite unknown models, training data, and a different label set.Clarifai.com is treated as a black-box image-classification system.
  • Findings: Existing approaches generate transferable non-targeted examples effectively, but only few targeted examples transfer with their target labels.The paper measures non-targeted transfer using accuracy and targeted transfer using matching rate.

2 ADVERSARIAL DEEP LEARNING AND TRANSFERABILITY

The paper defines non-targeted and targeted adversarial examples, presents optimization and gradient-based generation methods, and evaluates transferability across ImageNet models using accuracy, matching rate, and RMSD.

  • 2.1 THE ADVERSARIAL DEEP LEARNING PROBLEM: A non-targeted adversarial example changes the classifier's prediction away from the ground-truth label, whereas a targeted example must produce an adversary-specified label.The targeted label differs from the original ground-truth label.
  • 2.2 APPROACHES FOR GENERATING ADVERSARIAL EXAMPLES: The paper considers optimization-based, fast gradient, and fast gradient sign approaches, each with non-targeted and targeted variants.FG moves along the normalized gradient, while FGS moves along the gradient-sign direction.
  • 2.2 APPROACHES FOR GENERATING ADVERSARIAL EXAMPLES: The optimization-based method balances image distance against a loss measuring prediction disagreement through a tunable constant λ.The loss uses the ground-truth one-hot label for non-targeted generation and a target-label objective for targeted generation.
  • 2.3 EVALUATION METHODOLOGY: Experiments use five pretrained ImageNet networks and 100 validation images classified correctly by all models, with manually selected semantically distant target labels.The evaluation compares homogeneous ResNet architectures with heterogeneous architectures.
  • 2.3 EVALUATION METHODOLOGY: Non-targeted transferability is measured by cross-model adversarial accuracy, targeted transferability by target-label matching rate, and distortion by RMSD.Lower accuracy indicates better non-targeted transferability, whereas higher matching rate indicates better targeted transferability.

3 NON-TARGETED ADVERSARIAL EXAMPLES

Non-targeted adversarial examples transfer across models, including those generated by fast gradient methods. FG reaches full transferability at lower RMSD than FGS, while FGS transfers worse overall.

  • 3.1 Optimization-Based Approaches: A large proportion of optimization-based non-targeted adversarial images generated for one model transfer to another.All adversarial images generated for a model mislead that same model, while cross-model transfer is substantial.
  • 3.1 Optimization-Based Approaches: Transferability does not consistently improve between similarly structured ResNet models compared with heterogeneous model pairs.VGG-16-generated examples have lower accuracy on ResNet-50 than on ResNet-152 or ResNet-101.
  • 3.2.1 EFFECTIVENESS AND TRANSFERABILITY OF THE FAST GRADIENT-BASED APPROACHES: FG-generated non-targeted examples transfer across models, with cross-model accuracies comparable to or below those of optimization-based examples.The diagonal accuracies remain positive because FG prioritizes speed over fully misleading its source model.
  • 3.2.1 EFFECTIVENESS AND TRANSFERABILITY OF THE FAST GRADIENT-BASED APPROACHES: FGS has worse transferability than FG and optimization-based approaches, although at RMSD around 23 its adversarial-image accuracy exceeds FG's counterpart.The paper hypothesizes that this difference contributes to FGS's poorer transferability.
  • 3.2.2 ADVERSARIAL IMAGES WITH MINIMAL TRANSFERABLE RMSD: 100% of VGG-16-to-ResNet-152 transferable adversarial images are found below RMSD 80.91 with FG and 86.56 with FGS.FG generates transferable attacks with smaller RMSD, consistent with minimizing L2 distortion proportional to RMSD rather than L∞ distortion.
  • 3.3 COMPARISON WITH RANDOM PERTURBATIONS: Gaussian-noise perturbations have significantly worse transferability than optimization-based and fast gradient-based approaches.The comparison evaluates Gaussian noise added to the 100-image test set.

4 TARGETED ADVERSARIAL EXAMPLES

Existing targeted attacks often succeed on their source model but rarely transfer with the intended target label, and Gaussian-noise methods fail entirely. Fast gradient-based methods also struggle to produce targeted examples, even before transfer is considered.

  • Target labels rarely transfer to a different model, even when optimization-based targeted examples match their source model’s target labels.Increasing distortion does not improve target-label transfer; top-5 matching reaches at most 10%.
  • Most fast gradient-based targeted examples fail to make even their source model predict the target label, regardless of distortion.The approach searches only a 1-D subspace, limiting the available predictions.
  • 0% matching rate is observed for Gaussian-noise images across all five evaluated models.The authors conclude that Gaussian noise cannot generate successful targeted adversarial examples or targeted transferability.
  • Table 3 reports target-label matching rates by attack-excluded source model, evaluation model, and average RMSD.Each cell gives the percentage classified as the target label by the evaluation model.

5 ENSEMBLE-BASED APPROACHES

The paper generates attacks against ensembles of white-box models to improve transfer to an additional black-box model. This approach substantially improves non-targeted transfer and produces targeted examples with transferable target labels, whereas fast gradient ensemble attacks do not improve targeted transfer.

  • The ensemble-based method optimizes attacks for multiple white-box models while seeking transfer to an additional black-box model.For targeted attacks, the ensemble combines softmax outputs with weights α_i summing to 1; the non-targeted objective is analogous.
  • Each of five models is treated in turn as black-box, while attacks are generated using the other four models as white-box ensemble members.Generated images are evaluated on all five models.
  • A large proportion of optimization-based targeted examples transfer with their target labels for the first time.The optimization uses equal ensemble weights and 100 Adam update iterations.
  • Almost perfectly transferable non-targeted images have RMSDs around 17, versus 22 to 23 for single-model optimization attacks.On models excluded from attack generation, accuracy is no greater than 6%, compared with at least 12% for single-model approaches.
  • Table 4 reports non-targeted attack accuracy by excluded ensemble member, evaluation model, and average RMSD.Each cell evaluates attacks generated using four models against one model.
  • Fast gradient ensemble attacks do not improve targeted transfer over single-model attacks.The authors attribute this to the few target labels available within the 1-D search subspace.

6 GEOMETRIC PROPERTIES OF DIFFERENT MODELS

Geometric analyses connect transferability to aligned decision boundaries and distinguish it from nearly orthogonal model gradients. The studied planes also contain only a small subset of ImageNet labels, limiting targeted searches in those directions.

  • Geometric properties of different models: Gradient directions of different models are almost orthogonal, with non-diagonal cosine values close to 0.This geometric relationship is proposed as a reason ensemble fast-gradient attacks can retain nonzero diagonal error rates.
  • Decision boundaries of non-targeted approaches: Figure 3 maps predicted-label regions in a shared two-dimensional plane using VGG-16’s gradient direction and a random orthogonal direction.Coordinates correspond to pixel-value perturbations applied to the Figure 2 image, with matching colors denoting matching labels.
  • Decision boundaries of non-targeted approaches: Along the gradient direction, classifiers are misled sooner than along a random direction, while correct-prediction regions remain central.The authors attribute the gradient effect to stronger changes in the loss function.
  • Targeted search regions: At most 21 labels appear in each examined plane, representing 2.1% of ImageNet’s 1,000 categories.Thus, 97.9% of labels have no targeted adversarial example in each plane, partially explaining fast-gradient difficulty.
  • Decision boundaries of non-targeted approaches: Decision boundaries align well across models, partially explaining why non-targeted adversarial images transfer.The comparison uses a plane spanned by VGG-16’s gradient direction and a random orthogonal direction.
  • Decision boundaries of non-targeted approaches: VGG-16 contains a small hole inside its ground-truth region, unlike the other models’ decision planes.Non-targeted examples in this hole can have small distortion but fail to transfer.
  • Decision boundaries of targeted ensemble-based approaches: Target-label regions align well across the four models used in the ensemble, according to the targeted ensemble boundary analysis.The targeted direction is paired with a random orthogonal direction, excluding ResNet-101 from the ensemble.

7 REAL WORLD EXAMPLE: ADVERSARIAL EXAMPLES FOR CLARIFAI.COM

The Clarifai.com experiment tests transfer without knowledge of its training data, models, or label set. Both attack types transfer, with ensemble-generated targeted examples showing stronger transfer toward labels near their intended targets.

  • Clarifai.com is evaluated as a black-box classifier with unknown training data and model types, and labels different from ILSVRC 2012.The authors submit 100 original images and judge the returned labels correct subjectively.
  • 400 adversarial images are submitted: 200 targeted and 200 non-targeted examples generated with single-model or ensemble optimization.The targeted and non-targeted sets correspond to examples evaluated in the earlier model experiments.
  • Most non-targeted examples generated using either VGG-16 or an ensemble transfer to Clarifai.com.This extends the observed non-targeted transfer beyond the evaluated ImageNet models.
  • 57% of VGG-16 targeted examples and 76% of ensemble targeted examples cause Clarifai.com to predict labels irrelevant to the ground truth.These results show transfer-induced misclassification despite the black-box setting.
  • 18% of ensemble targeted examples and 2% of VGG-16 targeted examples are assigned labels close to the target label by Clarifai.com.The target model’s labels differ from those used to generate the attacks.
  • Table 6 lists the top five Clarifai.com labels for original and adversarial images, ordered first by rareness and then by confidence.The table provides the label outputs used for the black-box comparison.

8 CONCLUSION

The study finds that ensemble-based methods improve transferability for both non-targeted and targeted adversarial examples, including attacks on a black-box image classifier.

  • Ensemble-based approaches generate targeted adversarial examples whose target labels transfer with a high success rate.
  • The new approaches outperform previous work when generating non-targeted transferable adversarial examples.
  • Both non-targeted and targeted adversarial examples generated by the new approaches successfully attack Clarifai.com.
  • Geometric analyses examine properties that help explain why adversarial examples transfer.

APPENDIX

The appendix details optimization objectives, transferability measurements, geometric behavior, random-perturbation comparisons, and Clarifai.com evaluations for adversarial examples.

  • Optimization-based approach: The alternative optimization objective minimizes non-targeted loss while penalizing perturbations that fall below or exceed threshold τ.The two approaches produced similar transferability in preliminary evaluation, so results for the alternative approach were omitted.
  • Transferability results: 12% to 40% of jointly misclassified adversarial images received the same wrong label across model pairs, despite 999 possible incorrect categories.This appendix result is termed the same mistake effect.
  • Geometric behavior: Along a VGG-16 gradient direction, ResNet-152 predictions can alternate between wrong and correct labels as perturbation magnitude increases.The evaluated image becomes adversarial at small B, then changes prediction repeatedly as B grows.
  • Random perturbation comparison: With Gaussian noise standard deviation 25 and average RMSD 23.59, each model retained accuracy above 66%, unlike the evaluated adversarial perturbations.The comparison used 10,000 noisy images generated from 100 test images, with standard deviations ranging from 5 to 40.
  • Black-box evaluation: Clarifai.com evaluations report the top five labels returned for original and adversarial images, sorted by occurrence and then confidence.Additional Clarifai.com examples are presented in Table 34.
  • Evaluation metrics: The appendix defines transferability tables using top-1 or top-5 accuracy for non-targeted attacks and top-5 matching rate for targeted attacks.Lower accuracy indicates better non-targeted transferability, while higher matching rate indicates more successful transfer of target labels.
Loading 1611.02770v3…