PaperlayerView exact revision on arXiv

Source-linked AI summary

Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models

Wieland Brendel, Jonas Rauber, Matthias Bethge

arXiv:1712.04248v2stat.MLcs.CRcs.CVcs.LGcs.NE

TL;DR

Existing adversarial attacks often require model internals, confidence scores, or training data, limiting assessment of black-box systems. This paper introduces the Boundary Attack, which uses only final model decisions to reduce an initially large adversarial perturbation while remaining adversarial. It scales to complex natural datasets, performs competitively with gradient-based attacks, and is demonstrated on black-box models.

  • Problem

    Effective decision-based attacks were lacking for deep neural networks and natural datasets, despite the relevance of final-decision-only attacks to real-world black-box systems.

  • Method

    The Boundary Attack starts from an adversarial sample and uses rejection sampling, Gaussian-based proposal directions, and dynamically adjusted steps to reduce perturbation while preserving the adversarial criterion.

  • Results

    The Boundary Attack is competitive with the best gradient-based attacks in targeted and untargeted computer-vision scenarios and applies to black-box models.

  • Takeaways & Limitations

    Decision-based attacks provide a way to assess robustness and security risks when only final decisions are observable, including in systems such as autonomous cars.

  • Takeaways & Limitations

    The proposal distribution is simple and could be made more effective by learning from the attacked model or proposal history.

Abstract

from arXiv · show

Many machine learning algorithms are vulnerable to almost imperceptible perturbations of their inputs. So far it was unclear how much risk adversarial perturbations carry for the safety of real-world machine learning applications because most methods used to generate such perturbations rely either on detailed model information (gradient-based attacks) or on confidence scores such as class probabilities (score-based attacks), neither of which are available in most real-world scenarios. In many such cases one currently needs to retreat to transfer-based attacks which rely on cumbersome substitute models, need access to the training data and can be defended against. Here we emphasise the importance of attacks which solely rely on the final model decision. Such decision-based attacks are (1) applicable to real-world black-box models such as autonomous cars, (2) need less knowledge and are easier to apply than transfer-based attacks and (3) are more robust to simple defences than gradient- or score-based attacks. Previous attacks in this category were limited to simple models or simple datasets. Here we introduce the Boundary Attack, a decision-based attack that starts from a large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial. The attack is conceptually simple, requires close to no hyperparameter tuning, does not rely on substitute models and is competitive with the best gradient-based attacks in standard computer vision tasks like ImageNet. We apply the attack on two black-box algorithms from Clarifai.com. The Boundary Attack in particular and the class of decision-based attacks in general open new avenues to study the robustness of machine learning models and raise new questions regarding the safety of deployed machine learning systems. An implementation of the attack is available as part of Foolbox at https://github.com/bethgelab/foolbox .

1 INTRODUCTION

Adversarial perturbations threaten deployed machine-learning systems, yet existing attack categories require information often unavailable in real-world black-box settings. The paper motivates decision-based attacks and introduces the Boundary Attack to address the lack of scalable methods for complex models and natural datasets.

  • Adversarial perturbations can drastically change model predictions after minimal input changes, creating security risks for systems such as autonomous cars and face recognition.
  • Gradient-based attacks require detailed model information, while score-based attacks rely on predicted scores such as probabilities or logits.
  • Transfer-based attacks avoid model information but require training data to build substitute models and rely on adversarial-example transfer between models.
  • Decision-based attacks use only the model’s final decision and can require less information than transfer-based attacks while potentially resisting gradient masking and stochasticity.
  • Prior decision-based methods had not provided an effective attack applicable to deep neural networks and natural datasets such as ImageNet.
  • The Boundary Attack is introduced as a flexible, lightly tuned decision-based attack competitive with gradient-based attacks in targeted and untargeted computer-vision scenarios.

2 BOUNDARY ATTACK

The Boundary Attack starts from an already adversarial sample and uses rejection sampling to find progressively smaller adversarial perturbations. Its proposal distribution determines which random directions are explored, while initialization and acceptance depend only on the adversarial criterion and model decision.

  • Objective: The attack seeks to reduce the distance between the adversarial sample and the original image while preserving adversarial status.The objective is the Euclidean distance d(o, ˜o) = ∥o − ˜o∥2.
  • Boundary walk: Each iteration draws a perturbation from a proposal distribution and accepts the candidate only if it remains adversarial.This implements rejection sampling along the boundary between adversarial and non-adversarial regions.
  • Initialization: The algorithm initializes with a sample that already satisfies the adversarial criterion, rejecting uniformly sampled inputs that do not.For untargeted computer-vision inputs, pixels are sampled independently from U(0, 255).
  • Proposal distribution: The proposal distribution is crucial because it controls which random directions the attack explores, and its optimum can depend on the domain and model.A simple proposal distribution worked well for all tested vision-related problems.

1. The perturbed sample lies within the input domain,

The attack constructs valid perturbations by combining orthogonal exploration with movement toward the original image, while supporting several adversarial criteria. Its two step sizes are dynamically adjusted using the local geometry of the decision boundary.

  • Constraints: The perturbation must remain within the input domain while satisfying the attack’s adversarial criterion.Here, adversarial means fulfilling the criterion, such as changing the model decision, rather than necessarily producing a human-defined error.
  • Figure 2: Figure 2 depicts rejection sampling along the boundary, random directions generated from Gaussian samples projected on a sphere, and geometry-dependent step-size adjustment.The figure distinguishes orthogonal movement from movement toward the target image.
  • Proposal construction: The proposal heuristic samples iid Gaussian noise, rescales and clips it, projects it onto a sphere, then moves toward the original image.The orthogonal perturbation preserves the current distance approximately, while the final movement reduces that distance.
  • Adversarial criteria: Decision-based criteria can include untargeted misclassification, targeted misclassification, top-k misclassification, and threshold conditions.The criterion specifies when a perturbed input is considered adversarial.
  • Step-size adjustment: The attack has two relevant parameters, δ for total perturbation length and ϵ for movement toward the original input, and adjusts both dynamically.Adjustment is inspired by Trust Region methods and tests whether orthogonal and toward-original steps remain adversarial.

3 COMPARISON WITH OTHER ATTACKS

The Boundary Attack is evaluated against gradient-based attacks on MNIST, CIFAR-10, and ImageNet in untargeted and targeted settings. It achieves competitive perturbation sizes using only model decisions, though it requires many more model queries.

  • The evaluation covers MNIST, CIFAR-10, and ImageNet-1000, using 1,000 validation samples for MNIST and CIFAR and 250 images for ImageNet.
  • Untargeted attack: In the untargeted setting, the Boundary Attack is compared with FGSM, DeepFool, and Carlini & Wagner using median squared L2-distance.
  • Untargeted attack: 3.6e-03, 5.6e-06, 2.9e-07, 1.0e-07, and 6.5e-08 are the Boundary Attack scores for MNIST, CIFAR, VGG-19, ResNet-50, and Inception-v3, respectively.
  • Untargeted attack: The Boundary Attack is competitive with gradient-based attacks in minimal perturbation size and stable across initial points, but uses 1.200.000 forward passes and zero backward passes on ResNet-50.
  • Targeted attack: In the targeted setting, the Boundary Attack starts from a correctly classified target-class sample and reaches the target label while reducing distance to the original image.
  • Targeted attack: 4.8e-03, 3.0e-05, and 5.7e-06 are Carlini & Wagner's targeted scores for MNIST, CIFAR, and VGG-19, compared with Boundary Attack scores of 6.5e-03, 3.3e-05, and 9.9e-06.

4 THE IMPORTANCE OF DECISION-BASED ATTACKS TO EVALUATE

Decision-based attacks are evaluated against defensive techniques that mask gradients, including defensive distillation. The Boundary Attack uses only class labels and produces perturbations on both standard and distilled networks.

  • Gradient masking can make models appear robust by causing gradient-based attacks to fail.Defensive distillation initially reduced gradient-based attack success from close to 100% to 0.5%, but this apparent robustness was attributed to masked gradients.
  • The Boundary Attack evaluates distilled networks using only the class label with maximum probability, not logits.It was applied to distilled MNIST and CIFAR networks using the implementation and training protocol described in the paper.
  • MNIST: Boundary perturbations were 3.6e-03 for standard and 4.2e-03 for distilled networks.
  • CIFAR: Boundary perturbations were 5.6e-06 for standard and 1.3e-05 for distilled networks.
  • Similar perturbation sizes on distilled and undistilled networks indicate that defensive distillation does not significantly increase robustness against the Boundary Attack.

5 ATTACKS ON REAL-WORLD APPLICATIONS

The Boundary Attack was applied to Clarifai black-box models for brand and celebrity recognition. Most adversarial perturbations were 1e−2 to 1e−3, yet the original and adversarial images were usually nearly perceptually indistinguishable.

  • The experiments targeted Clarifai models recognizing over 500 brands and over 10,000 celebrities.The models could return multiple identifications, but the evaluation considered the identification with the highest confidence score.
  • Samples were square-cropped, resized to 100 × 100 pixels, and selected so Clarifai initially recognized the visible brand or celebrity.
  • The evaluation used adversarial examples from Clarifai brand-detection and celebrity-detection models.
  • The adversarial criterion was that Clarifai report a different brand or celebrity, or None.
  • 1e−2 to 1e−3: most Clarifai perturbations were in this range, although some matched the 1e−7 scale observed for ImageNet examples.Some perturbations produced slightly noticeable noise, but most original and adversarial images were close to perceptually indistinguishable.

6 DISCUSSION & OUTLOOK

The paper frames decision-based attacks as practical tools for testing black-box model robustness and introduces the Boundary Attack for complex models and natural datasets. It also identifies proposal-distribution improvements as a direction for future work.

  • Decision-based attacks can find adversarial examples when only a model’s final decision is observable.The paper connects this setting to deployed systems such as autonomous cars.
  • The Boundary Attack follows the decision boundary using rejection sampling, a proposal distribution, and dynamic step-size adjustment.It starts from a large perturbation and successively reduces it while remaining adversarial.
  • A constrained iid Gaussian proposal can already be effective, while learned or history-conditioned proposals could improve the attack.
  • Decision-based attacks are presented as relevant for assessing model robustness and highlighting security risks in closed-source systems.
Loading 1712.04248v2…