Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

Nicolas Papernot; Patrick McDaniel; Ian Goodfellow

Source-linked AI summary

Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow

arXiv:1605.07277v1May 24, 2016cs.CRcs.LG

TL;DR

The paper examines whether adversarial-sample transferability enables black-box attacks across different machine-learning classifier classes. It generalizes substitute-model learning with reservoir sampling and demonstrates reliable attacks against remotely deployed classifiers whose characteristics are unknown.

Problem
Adversarial examples can transfer between models, but the breadth of this phenomenon across different machine-learning classifier classes requires systematic investigation because it creates a threat vector against classifiers.
Method
The paper generalizes substitute-model learning beyond deep networks and improves its efficiency using periodic step sizes and reservoir sampling during oracle-guided dataset augmentation.
Results
The attacks transfer across models trained with the same or different techniques, and substitute models can target classifiers including DNNs, LR, SVMs, decision trees, and nearest neighbors.
Takeaways & Limitations
Unknown, remotely deployed classifiers can be reliably targeted through black-box attacks, motivating defensive mechanisms and input validation for production machine-learning systems.
Takeaways & Limitations
The threat model assumes oracle access to the remote classifier, while its type, parameters, and training set remain unknown; the authors also lacked control over Google Cloud’s training procedure.

Abstract

from arXiv · show

Many machine learning models are vulnerable to adversarial examples: inputs that are specially crafted to cause a machine learning model to produce an incorrect output. Adversarial examples that affect one model often affect another model, even if the two models have different architectures or were trained on different training sets, so long as both models were trained to perform the same task. An attacker may therefore train their own substitute model, craft adversarial examples against the substitute, and transfer them to a victim model, with very little information about the victim. Recent work has further developed a technique that uses the victim model as an oracle to label a synthetic training set for the substitute, so the attacker need not even collect a training set to mount the attack. We extend these recent techniques using reservoir sampling to greatly enhance the efficiency of the training procedure for the substitute model. We introduce new transferability attacks between previously unexplored (substitute, victim) pairs of machine learning model classes, most notably SVMs and decision trees. We demonstrate our attacks on two commercial machine learning classification systems from Amazon (96.19% misclassification rate) and Google (88.94%) using only 800 queries of the victim model, thereby showing that existing machine learning approaches are in general vulnerable to systematic black-box attacks regardless of their structure.

1. INTRODUCTION

The paper generalizes black-box attacks based on adversarial-sample transferability across diverse machine-learning models. It introduces substitute-learning and sample-crafting techniques beyond deep networks, and demonstrates effective attacks against commercial classifiers.

Motivation: Adversarial samples subtly alter legitimate inputs to induce erroneous model outputs while remaining correctly classifiable by humans.Such attacks can threaten fraud detection, content filtering, malware detection, and autonomous navigation.
Transferability: Transferability allows samples crafted against one model to mislead other models, even when their architectures differ.This property enables oracle-based attacks that train a local substitute using victim-generated labels.
Scope: The study evaluates transferability and black-box attacks across DNNs, logistic regression, SVMs, decision trees, kNNs, and ensembles.It targets models trained with the same or different techniques.
Contributions: SVM and decision-tree adversarial-sample crafting techniques are introduced for non-differentiable models.These techniques extend attacks to model classes previously less explored in transferability studies.
Results: 91.43% and 87.42% of adversarial samples crafted for logistic regression caused misclassification by an SVM and decision tree, respectively.The result illustrates cross-technique transferability beyond neural-network pairs.
Results: 96.19% and 88.94% misclassification rates were achieved against Amazon and Google classifiers using only 800 victim queries.The attacks used a logistic-regression substitute despite limited knowledge of the target models.

2. APPROACH OVERVIEW

The paper tests whether adversarial-sample transferability is strong across machine-learning techniques and whether oracle-based black-box attacks work against unknown classifiers. It studies transferability systematically and develops substitute models from carefully selected oracle-labeled inputs.

Hypotheses: The approach evaluates two hypotheses concerning transferability across techniques and the feasibility of attacks against unknown classifiers.The hypotheses cover both intra-technique and cross-technique transferability.
Adversarial samples: Adversarial samples are generated by optimizing an input perturbation intended to make a model assign an incorrect class.Closed-form solutions are not always available for nonlinear or non-convex models, motivating approximate methods.
Transferability variants: Intra-technique transferability compares models using the same learning technique, whereas cross-technique transferability compares models using different techniques.The input domain X represents the expected distribution for the shared task.
Experimental design: The study measures both transferability variants across representative model classes trained on MNIST digit classification.The model collection spans the machine-learning techniques listed in Table 1.
Security implications: Input validation is identified as a needed production defense when machine-learning deployments create incentives for adversarial manipulation.The paper compares this need with input validation for SQL injections.
Oracle attacks: The attack method queries an unknown classifier as an oracle to label carefully selected inputs for training a substitute model.The threat model withholds the target’s type, parameters, and training set.

3. TRANSFERABILITY OF ADVERSARIAL SAMPLES IN MACHINE LEARNING

Across five machine-learning techniques, adversarial samples transfer both within and across technique classes. The results show widespread vulnerability, with transferability strong but heterogeneous across model types.

Experimental setup: All five studied techniques—DNNs, LR, SVMs, DTs, and kNNs—were evaluated for intra- and cross-technique adversarial-sample transferability.The study used MNIST and trained five models per technique for the intra-technique experiments.
Intra-technique transferability: At least 49% of adversarial samples transferred between models within the same technique.Differentiable DNNs and LR were more vulnerable to intra-technique transferability than non-differentiable SVMs, DTs, and kNNs.
Cross-technique transferability: Cross-technique transferability was strong but heterogeneous across classifiers, with each matrix cell measuring the percentage of samples crafted against one technique and misclassified by another.The cross-technique experiment added an ensemble classifier and used one model per technique trained on the full MNIST training set.
Cross-technique transferability: 47.20%–89.29% were the decision tree’s cross-technique misclassification rates, compared with 0.82%–38.27% for the deep neural network.These ranges identify the decision tree as the most vulnerable and the DNN as the most resilient in the reported matrix.
Cross-technique transferability: 44.14% was the ensemble’s misclassification rate for samples crafted using the LR model.The paper attributes this ensemble vulnerability most likely to the vulnerability of its underlying experts.
Implications: Cross-technique transferability reduces the minimum knowledge an adversary needs to force a classifier to misclassify crafted inputs.The finding extends transferability beyond attacks involving only similar or deep-learning models.

4. LEARNING CLASSIFIER SUBSTITUTES BY KNOWLEDGE TRANSFER

The paper learns substitute classifiers by querying a target oracle, refining Jacobian-based augmentation with periodic step sizes and reservoir sampling, and extending substitutes across classifier families. DNN and LR substitutes generally approximate diverse oracles well, while SVM substitutes transfer poorly from non-SVM oracles.

Dataset Augmentation for Substitutes: The method trains substitute models from oracle-provided labels on carefully selected inputs, without access to the oracle’s model type, parameters, or training data.The initial substitute training set uses 100 unseen MNIST test samples.
Dataset Augmentation for Substitutes: Periodic step sizes alternate the augmentation direction after a chosen iteration period to improve oracle approximation.The period can be set after augmentation stops yielding substantial improvement or selected by grid search.
Dataset Augmentation for Substitutes: Reservoir sampling limits augmentation to κ randomly selected prior inputs, preventing exponential query growth while preserving comparable substitute quality.The query count falls from n·2^ρ to n·2^σ + κ·(ρ−σ).
Learning Classifier Substitutes: The study generalizes substitute learning to DNNs, LR, SVMs, decision trees, and kNNs, finding DNN and LR substitutes effective for all studied oracle types except decision trees.The broader goal is black-box attack applicability across machine-learning techniques.
Deep Neural Network Substitutes: DNN substitutes matched approximately 77%–83% of oracle labels after 10 iterations, except for decision-tree oracles at 48%.The paper attributes the decision-tree gap to the non-differentiability of tree decisions.
Deep Neural Network Substitutes: At ρ = 9, periodic step size raised DNN-substitute matches from 78.01% to 89.28% for a DNN oracle and from 79.68% to 83.79% for an SVM oracle.Reservoir sampling reduced queries but produced lower quality than periodic step size alone while remaining above the vanilla substitute.
Logistic Regression Substitutes: At ρ = 9, periodic step size raised LR-substitute matches to 84.01% for an LR oracle and 82.19% for an SVM oracle, while reservoir sampling achieved 79.20% for the SVM oracle.LR substitutes reached their asymptotic match rate after ρ = 4, or 1,600 oracle queries, and were computationally lighter than DNN substitutes.
Support Vector Machines Substitutes: SVM substitutes matched 79.80% of SVM-oracle labels but only 11.98% and 11.97% of DNN- and LR-oracle labels, with no improvement from either refinement.The authors suggest this may reflect the specificity of SVM training and learned decision boundaries.

5. BLACK-BOX ATTACKS OF REMOTE MACHINE LEARNING CLASSIFIERS

The paper generalizes oracle-based black-box attacks by training substitute models from remote classifier queries, crafting transferable adversarial samples, and testing the approach against Amazon and Google services.

Attack procedure: The attacker trains a local substitute model using oracle queries, then crafts adversarial samples against it and transfers them to the remote classifier.The threat model assumes the classifier type, parameters, and training set are unknown.
Amazon oracle: 800 queries enabled a logistic-regression substitute to force the Amazon classifier to misclassify 96.19% of adversarial inputs.The perturbation was ε = 0.3 and the substitute used ρ = 3 augmentation iterations.
Amazon oracle: Periodic step size and reservoir sampling reduced DNN-substitute queries from 6,400 to 2,000 while decreasing Amazon misclassification from 96.78% to 95.68%.The refined result remained above the 87.44% rate achieved after 800 queries without the refinements.
Google oracle: 800 queries enabled a logistic-regression substitute to make the Google classifier misclassify 88.94% of adversarial samples.The perturbation was ε = 0.3; with both refinements, DNN and LR substitutes achieved 91.57% and 97.72%, respectively.

6. ADVERSARIAL SAMPLE CRAFTING

The paper extends adversarial-sample crafting beyond differentiable models by introducing techniques for SVMs and decision trees alongside methods for DNNs, logistic regression, and nearest neighbors.

Overview: The paper introduces adversarial-sample crafting algorithms for support vector machines and decision trees, which are non-differentiable models.These methods extend earlier approaches developed for deep neural networks.
Deep Neural Networks: For DNNs, the fast gradient sign method evaluates the sign of the cost gradient with respect to the input.The perturbation magnitude is controlled by ε, with larger values increasing both misclassification likelihood and detectability.
Logistic Regression: For logistic regression, fast gradient sign produces the max-norm perturbation directly rather than as an approximation.The method evaluates Equation 8 for the most damaging input perturbation.
Nearest Neighbors: For nearest neighbors, the paper uses fast gradient sign with a smoothed classifier that replaces argmin by a soft-min operation.The smoothing makes the model differentiable for crafting purposes.
Multi-class Support Vector Machines: For multiclass SVMs, samples are perturbed orthogonally to the relevant weight vector and decision-boundary hyperplane without optimization.The parameter ε controls the introduced distortion.
Decision Trees: For decision trees, the attack finds a nearby leaf with a different class and changes features to satisfy the conditions along its path.The method exploits the tree’s recursive if-else structure.

7. DISCUSSION AND RELATED WORK

The discussion connects adversarial vulnerability to imperfections in learned models and frames substitute-model attacks as a practical extension of earlier black-box deep-learning work.

Discussion: Classifiers extrapolate from finite input-label examples, while training imperfections, model linearity, and limited data can contribute to adversarial vulnerability.These factors help explain why manipulated inputs can control model outputs.
Related Work: Substitute models approximate targeted decision boundaries, reducing the need to know the target architecture and parameters.The paper extends substitute learning from deep learning to logistic regression and other classifier types.
Security implications: The paper reports that remotely deployed classifiers with unknown characteristics can be reliably targeted through black-box attacks.It also reports that adversarial-sample training defenses were ineffective or unavailable to the authors.
Related Work: Unlike earlier work assuming architecture and parameter knowledge, this threat model targets deployed classifiers at test time.Training-data poisoning and other threat models remain comparatively unexplored in the cited work.

8. CONCLUSIONS

The paper concludes that adversarial samples transfer across machine-learning techniques and enable attacks on remote classifiers using only label queries, while identifying input validation and poisoning attacks as open problems.

Conclusions: Adversarial samples transfer across models trained with the same or different machine-learning techniques.The paper reports transferability across DNN, LR, SVM, decision-tree, and nearest-neighbor classifiers.
Conclusions: DNNs and logistic regression can learn substitutes for classifiers based on DNNs, logistic regression, SVMs, decision trees, and nearest neighbors.The substitutes are used to craft transferable adversarial samples.
Conclusions: 96.19% and 88.94% of inputs were misclassified by Amazon and Google classifiers after attacks using 800 label queries.The experiments used no knowledge of the services’ model designs or parameters.
Open problems: The findings call for validation of inputs to machine-learning algorithms, which remains an open problem.The paper also identifies training-time poisoning attacks as requiring further investigation.