PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical Black-Box Attacks against Machine Learning

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram Swami

arXiv:1602.02697v4cs.CRcs.LG

TL;DR

Existing adversarial attacks generally require model internals or training data, limiting black-box evaluation. This paper trains a synthetic-data substitute using label-only queries and finds high misclassification rates against remote ML models.

  • Problem

    Prior adversarial attacks require access to model architecture, parameters, or training data, whereas this work studies attacks using only queried output labels.

  • Method

    The attack trains a substitute classifier on synthetic inputs labeled by the target, iteratively improving its decision-boundary similarity to craft adversarial examples.

  • Results

    Remote models misclassified adversarial examples at rates of 84.24% for MetaMind, 96.19% for Amazon, and 88.94% for Google.

  • Takeaways & Limitations

    The attack is practical across remote DNN and logistic-regression systems with label-only access, limited queries, and no model or training-data knowledge.

  • Takeaways & Limitations

    The authors identify defenses against finite perturbations as a more promising direction for future work than defenses against infinitesimal perturbations.

Abstract

from arXiv · show

Machine learning (ML) models, e.g., deep neural networks (DNNs), are vulnerable to adversarial examples: malicious inputs modified to yield erroneous model outputs, while appearing unmodified to human observers. Potential attacks include having malicious content like malware identified as legitimate or controlling vehicle behavior. Yet, all existing adversarial example attacks require knowledge of either the model internals or its training data. We introduce the first practical demonstration of an attacker controlling a remotely hosted DNN with no such knowledge. Indeed, the only capability of our black-box adversary is to observe labels given by the DNN to chosen inputs. Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN. We use the local substitute to craft adversarial examples, and find that they are misclassified by the targeted DNN. To perform a real-world and properly-blinded evaluation, we attack a DNN hosted by MetaMind, an online deep learning API. We find that their DNN misclassifies 84.24% of the adversarial examples crafted with our substitute. We demonstrate the general applicability of our strategy to many ML techniques by conducting the same attack against models hosted by Amazon and Google, using logistic regression substitutes. They yield adversarial examples misclassified by Amazon and Google at rates of 96.19% and 88.94%. We also find that this black-box attack strategy is capable of evading defense strategies previously found to make adversarial example crafting harder.

1. INTRODUCTION

The paper demonstrates practical black-box attacks against remotely hosted ML classifiers using only queried labels, without model internals or large training data. A synthetic-data substitute enables adversarial examples that transfer to target DNNs and other classifiers, including MetaMind, Amazon, and Google systems.

  • Motivation: Adversarial examples add small, often imperceptible perturbations that cause classifiers to misclassify inputs while humans perceive them as unchanged.The introduction illustrates a stop sign altered to be classified as a yield sign, potentially causing dangerous autonomous-vehicle behavior.
  • Threat model: The attacker observes only labels for chosen inputs, lacking the target model’s architecture, parameters, and large training dataset.This threat model represents users interacting with remotely hosted classifiers whose internals remain secret.
  • Attack strategy: The attack trains a local substitute with adversary-generated synthetic inputs labeled by the target, then crafts adversarial examples using the substitute’s known parameters.Transfer succeeds because the substitute and target have similar decision boundaries; prior approaches required model knowledge or independently collected training data.
  • Evaluation: 84.24% of adversarial inputs crafted with the substitute were misclassified by MetaMind’s remotely hosted DNN.The experiment was designed as a real-world, correctly blinded evaluation against a model accessed only after training.
  • Generality and defenses: 96.19% and 88.94% of adversarial examples were misclassified by Amazon and Google models, respectively, while the attack also evaded proposed defenses.The approach generalizes to logistic regression and combines label-only access, limited queries, and applicability across classifier types.

2. ABOUT DEEP NEURAL NETWORKS

Deep neural networks compose parameterized neuron layers to transform inputs into outputs, with learned weights encoding model knowledge. In classification, trained networks assign labels by outputting class-probability vectors for previously unseen inputs.

  • DNNs hierarchically compose n parametric functions to model an input x⃗.Each function is implemented as a neuron layer applying an activation function to the previous layer’s weighted representation.
  • Each layer uses weights θ_i that determine neuron activations and collectively hold the DNN model’s learned knowledge.These parameters are evaluated during training.
  • DNN computation applies the nested function composition F(x⃗) = f_n(θ_n, f_n−1(θ_n−1, ... f_2(θ_2, f_1(θ_1, x⃗)))) to an input.
  • During training, a DNN learns θ_F = {θ_1, ..., θ_n} from input-output pairs by reducing prediction error, typically using backpropagation-derived techniques.The paper focuses on classification tasks that assign inputs labels from a predefined set.
  • During testing, a fixed-parameter classifier outputs a probability vector F(x⃗) representing its belief that an unseen input belongs to each class.The model’s learned weights encode its acquired knowledge and should support generalization beyond the training domain.

3. THREAT MODEL

The threat model targets multiclass DNN classifiers under a weak black-box adversary that can query only the model’s output labels, without access to its architecture, parameters, probabilities, or training data. The adversary seeks minimally perturbed inputs that the oracle assigns a different class than the original input.

  • Adversarial Capabilities: The adversary has no access to the DNN’s architectural choices, internal parameters, learned probabilities, or training data.Labels reveal less information about the model’s learned behavior than probabilities, making this access weaker but realistic.
  • Targeted Model: The targeted model is a multiclass DNN classifier that outputs probability vectors over predefined classes.Examples include handwritten digits, object categories, and traffic signs.
  • Adversarial Capabilities: The adversary’s sole capability is querying the oracle for the label assigned to any input.The returned label is the index of the class receiving the largest predicted probability.
  • Adversarial Goal: The attack seeks an adversarial sample x* whose oracle label differs from the original input’s label: ˜O(x*) ≠ ˜O(x).This represents an attack on the oracle’s output integrity.
  • Adversarial Goal: The adversarial sample minimizes the perturbation δx needed to change the oracle’s assigned class while remaining difficult for humans to detect.Even with complete knowledge of the model architecture and parameters, finding such a minimal perturbation is nontrivial.

4. BLACK-BOX ATTACK STRATEGY

The attack trains a substitute model using synthetically generated inputs labeled only through oracle queries, then crafts adversarial examples against the substitute. Jacobian-based augmentation prioritizes inputs near varying decision-boundary directions, enabling approximation of the oracle with few label queries and exploiting adversarial transferability.

  • Attack Strategy: The adversary queries the oracle only for labels on synthetic inputs, trains a substitute model, and crafts adversarial examples against the target.The substitute approximates the oracle’s decision boundaries, allowing crafted examples to transfer to the target model.
  • Motivation: The attack addresses black-box optimization difficulty because closed-form minimal perturbations are unavailable for non-convex targets such as DNNs.Standard attacks instead rely on gradients of functions defined by the target DNN, which are inaccessible in this threat model.
  • Synthetic Dataset Generation: Jacobian-based Dataset Augmentation selects synthetic inputs along directions where the substitute’s output varies, concentrating queries near informative decision boundaries.The method is designed to approximate decision boundaries with few label queries rather than maximize substitute accuracy.
  • Substitute Training Procedure: The substitute-training loop labels the current dataset, trains the selected architecture, augments it with new synthetic points, and repeats across epochs.The initial set can be very small and representative of the input domain, while architecture selection uses high-level task knowledge.
  • Jacobian-Based Augmentation: The augmentation step adds λ-scaled sign-Jacobian perturbations to existing samples, with λ controlling the step size along the identified sensitive direction.The augmented set combines the new points with the original set, producing Sρ+1 from Sρ.

5. VALIDATION OF THE ATTACK

The attack succeeds against both a remotely hosted MetaMind DNN and a locally trained GTSRB traffic-sign oracle, using only queried labels and crafted perturbations. On MNIST, transferability reaches 84.24%, while the GTSRB attack transfers at higher rates under lower distortions.

  • GTSRB validation: More than 64.24% of altered inputs forced the locally trained GTSRB traffic-sign oracle to misclassify without affecting human recognition.The GTSRB oracle recognizes 43 traffic-sign classes using resized, normalized RGB images.
  • MetaMind validation: MetaMind’s API exposed only automated training and label predictions, withholding the training algorithm, architecture, and parameters; the resulting classifier reached 94.97% accuracy after 36 hours.The adversary uploaded 50,000 MNIST training samples and queried predictions for chosen inputs.
  • MetaMind validation: The two substitute DNNs achieved 81.20% and 67.00% accuracy on the unused MNIST test samples after 6 substitute training epochs.The substitutes were trained from initial MNIST-based or handcrafted sets with Jacobian-based dataset augmentation.
  • MetaMind validation: 84.24% transferability was achieved for the first MNIST substitute and 78.72% for the second at ε = 0.3, showing that oracle misclassification transfers from substitutes.Transferability measures the proportion of adversarial samples misclassified by the oracle after being crafted with substitute DNNs.
  • GTSRB validation: Adversarial GTSRB samples with ε < 0.3 were more transferable than MNIST samples at the same ε, while substitute success rates exceeded 98.98%.The passage attributes the difference partly to GTSRB’s 3,072 input components versus MNIST’s 784.

6. ATTACK ALGORITHM CALIBRATION

Section 6 calibrates substitute training and adversarial crafting to improve transferability while reducing oracle queries. It finds architecture and extra training have limited benefits, whereas augmentation step size and crafting parameters materially affect transferability.

  • Substitute training: Substitute architecture has limited impact on transferability, and training beyond asymptotic accuracy does not improve adversarial sample transferability.The evaluated architectures varied in layer type, number, and size.
  • Crafting algorithms: At comparable perturbation magnitude, Goodfellow’s and Papernot’s algorithms achieve similar transferability rates.The comparison fixes the L1 norm of the introduced perturbation; an example equates ∥δ⃗x∥1 = 0.4 with Goodfellow ε = 0.4 or Papernot ε = 1 and Υ = 40%.
  • Substitute training: For input variations ε ≤0.3, longer substitute training increases transferability only partially, contrary to the expectation that more oracle-labeled samples always help.The supplied passage reports this qualification but is truncated before giving the complete experimental result.
  • Substitute training: With λ = 0.3 instead of 0.1, transferability falls from 22.35% to 10.82% for ε = 0.25 and from 85.22% to 82.07% for ε = 0.5.Larger step sizes also reduce convergence stability, while smaller values slow convergence.
  • Oracle querying: Reservoir sampling reduces oracle queries from n · 2^ρ to n · 2^σ + κ · (ρ − σ) without significantly degrading substitute accuracy.This targets settings with query quotas or detection risk; Section 7 evaluates the accuracy impact.
  • Crafting algorithms: 72.24%–80.21% are the asymptotic transferability rates for non-baseline architectures under Goodfellow’s algorithm, while architecture A, matching the oracle, performs best.Goodfellow’s algorithm is tuned by input variation ε; Papernot’s algorithm is tuned by maximum distortion Υ and input variation ε.
  • Crafting algorithms: Reducing Papernot’s input variation from 1 to 0.7 approximately halves transferability, while increasing maximum distortion Υ increases transferability.The fixed Υ prevents adding more altered components to compensate for smaller ε.

7. GENERALIZATION OF THE ATTACK

The attack generalizes beyond DNN substitutes and targets to multiple classifier families, provided the substitute is differentiable for Jacobian-based synthetic-data generation. Logistic-regression substitutes successfully transfer adversarial examples to remotely hosted Amazon and Google models, with refinements improving efficiency and transferability.

  • Generalization beyond DNNs: The attack applies to non-differentiable target oracles, including decision trees, because only the substitute must be differentiable for Jacobian-based data generation.The substitute’s differentiability enables synthetic data generation through its Jacobian matrix.
  • Generalization beyond DNNs: Four additional substitute classifier types—logistic regression, SVM, decision tree, and nearest neighbors—are trained on raw MNIST pixels alongside DNNs.The experiments use no feature engineering and train classifiers directly on raw pixel values.
  • Remote-model attacks: 96.19% of Amazon adversarial samples and 88.94% of Google adversarial samples are misclassified after 800 oracle queries using logistic-regression substitutes.Both results use perturbation ε = 0.3; the rates demonstrate transfer to remotely hosted models.
  • Substitute approximation: 89.28% of DNN oracle labels are matched by a periodic-step-size DNN substitute after ρ = 9 epochs, versus 78.01% for the vanilla substitute.For logistic regression, periodic step size achieves 84.01% matching versus 72.00% for the vanilla substitute.
  • Query reduction: 2,000 queries reduce training queries from 6,400 by more than threefold, while Amazon DNN-substitute misclassification decreases from 96.78% to 95.68%.The refined method combines periodic step sizes with reservoir sampling and preserves higher transferability than the 87.44% rate after 800 queries.

8. DEFENSE STRATEGIES

The substitute-based black-box attack overcomes defenses that mask gradients or harden models against infinitesimal perturbations. Robustness to larger, finite perturbations is presented as a more promising defense direction, although no fully effective defense is known.

  • Detection and robustness: Oracle queries can be distributed among colluding users, making this attack difficult to detect through query monitoring.The paper distinguishes reactive defenses that detect adversarial examples from proactive defenses that make models more robust.
  • Gradient masking: Gradient masking defenses can be evaded because sensitive directions may be discovered indirectly through transfer from a substitute model.Such defenses may remove useful gradients, but models can remain vulnerable to adversarial examples affecting a smooth version of the model.
  • Adversarial training: Adversarial training that improves robustness to small or infinitesimal perturbations can be evaded by the substitute-based black-box approach.Robustness to larger and finite perturbations, however, prevents the black-box attack in the reported evaluation.
  • Defensive distillation: Defensive distillation prevents direct oracle attacks but remains vulnerable when adversarial examples are crafted using a substitute model.The evaluation used distilled MNIST oracles at temperatures T = 5, 10, 100; the substitute retained gradients needed for the fast gradient sign method.
  • Defense implications: Models robust in small neighborhoods around the training manifold perform gradient masking, which substitute models evade because they are not trained for those small perturbations.The paper therefore identifies finite-perturbation robustness as a more promising future defense than robustness to infinitesimal perturbations.

9. CONCLUSIONS

The paper introduces a synthetic-data substitute-training attack that crafts adversarial examples misclassified by black-box DNNs while requiring only label observations. It also reports evasion of gradient-masking defenses and provides intuition for adversarial-example transferability across DNNs.

  • Contributions: The authors introduce an attack using synthetic data generation to train a substitute model and craft adversarial examples misclassified by black-box DNNs.The approach is presented as a step toward relaxing the strong adversarial-capability assumptions made by previous attacks.
  • Contributions: The attack assumes only that the adversary can observe labels assigned by the target model.
  • Contributions: The attack evades gradient-masking defenses previously proposed to increase resilience against adversarial examples.
  • Contributions: The paper provides an intuition for adversarial-example transferability across DNNs.

A. DNN architectures

Figure 13 specifies the DNN architectures used throughout Sections 5, 6, and 11, identifying each model’s dimensions and neural-network layers. It also defines the notation for convolutional, rectified-linear or sigmoid, and softmax layers.

  • Architecture specification: Figure 13 defines the architectures used throughout Sections 5, 6, and 11, including each model’s identifier, input and output dimensions, and layer sequence.The identifier is the paper’s reference for the architecture; successive columns represent neural-network layers.
  • Architecture notation: CM denotes 2x2 convolution followed by 2x2 max-pooling, RL denotes rectified-linear layers except for sigmoid units in 200s, and S denotes softmax.These abbreviations are defined in Figure 13’s legend.

B. Intuition behind Transferability

The paper explains transferability through correlations between substitute and oracle DNN cost-gradient sign matrices, using statistical hypothesis testing as an empirical framework. Experiments show these correlations strengthen with substitute training and align with class-relevant pixels and transferability.

  • B. Intuition behind Transferability: Transferability is linked to correlated cost-gradient sign matrices between substitute and oracle DNNs; a formal treatment remains future work.The intuition is based on statistical hypothesis testing and analysis of DNN cost-gradient sign matrices.
  • B. Intuition behind Transferability: 61, 403–88, 813 χ2⋆ scores and P-values below 10−5 for all architectures lead to rejecting independence with confidence p < 0.01.For MNIST, the test has 729 degrees of freedom; the rejected hypothesis implies substitute–oracle sign matrices are highly correlated.
  • B. Intuition behind Transferability: As substitute training progresses, center-image pixels show increasingly frequent sign agreement between substitute A and the oracle.The frequency matrices use 10,000 MNIST test samples and compare training epochs ρ ∈ {0, 3, 6}.
  • B. Intuition behind Transferability: Sign matrices agree on class-relevant pixels, while substitutes with lower transferability have fewer components frequently equal to the oracle’s.This pattern was observed across substitute DNNs, though additional figures were omitted for space.

C. Discussion of Related Work

The discussion contrasts this black-box attack with prior evasion and model-extraction methods that require probability access, expensive labeled data, semantic gaps, or partial model knowledge. These limitations restrict prior methods’ efficiency, applicability, or scalability to remote DNN targets.

  • Black-box evasion attacks: 500 evading variants were found in 6 days by Xu et al.’s genetic-algorithm attack, which required classifier probability scores and heavy querying.The authors concluded that this approach could not target remote classifiers, and defenders could conceal probability outputs.
  • Black-box evasion attacks: Srndic et al. trained substitute models from labeled data, evaluated only random forests and an SVM, and relied on a PDF-renderer semantic gap.Collecting labeled data is especially expensive for DNNs, while the semantic gap limits applicability to models without such a gap.
  • Model extraction attacks: 2, 225 parameters were recovered from a shallow neural network using partial model knowledge and equation solving, but scaling this extraction approach to DNNs is difficult.Tramer et al. targeted exact parameter extraction rather than the adversarial goal considered here and demonstrated the method on classifiers hosted by BigML and Amazon.
Loading 1602.02697v4…