The Space of Transferable Adversarial Examples

Florian Tramèr; Nicolas Papernot; Ian Goodfellow; Dan Boneh; Patrick McDaniel

Source-linked AI summary

The Space of Transferable Adversarial Examples

Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel

arXiv:1704.03453v2Apr 11, 2017stat.MLcs.CRcs.LG

TL;DR

The paper asks why adversarial examples transfer across models and how large the transferable adversarial-input space is. It introduces methods to measure adversarial subspaces, compares decision boundaries, and derives formal transferability conditions. The results identify large shared subspaces and close boundaries, while also showing that transfer can fail in some settings.

Problem
Transferability enables black-box attacks, but the reasons adversarial examples cross models and the limits of this phenomenon require better understanding.
Method
The paper finds orthogonal gradient-aligned perturbations, measures shared adversarial subspaces and decision-boundary distances, and proves transferability conditions for simple model classes.
Results
The paper finds contiguous transferable adversarial subspaces of about 25 dimensions, decision boundaries close in arbitrary directions, and scenarios where transfer does not occur.
Takeaways & Limitations
Transferability is substantial but not universal, indicating that defenses against transfer-based attacks may be possible even for models vulnerable to direct attacks.
Takeaways & Limitations
The transferability analysis considers attacks using adversarial examples crafted on a locally trained model, while sufficiently close model theft can enable attacks against white-box-vulnerable models.

Abstract

from arXiv · show

Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time. They often transfer: the same adversarial example fools more than one model. In this work, we propose novel methods for estimating the previously unknown dimensionality of the space of adversarial inputs. We find that adversarial examples span a contiguous subspace of large (~25) dimensionality. Adversarial subspaces with higher dimensionality are more likely to intersect. We find that for two different models, a significant fraction of their subspaces is shared, thus enabling transferability. In the first quantitative analysis of the similarity of different models' decision boundaries, we show that these boundaries are actually close in arbitrary directions, whether adversarial or benign. We conclude by formally studying the limits of transferability. We derive (1) sufficient conditions on the data distribution that imply transferability for simple model classes and (2) examples of scenarios in which transfer does not occur. These findings indicate that it may be possible to design defenses against transfer-based attacks, even for models that are vulnerable to direct attacks.

1 Introduction

The introduction frames transferability as a security problem and investigates why adversarial examples cross models. It combines empirical analysis of shared adversarial subspaces and decision boundaries with formal conditions showing that transfer can occur or fail.

Motivation: Adversarial examples can transfer across models, enabling black-box attacks when an adversary crafts inputs using a local model.The attacker may train the local model by querying the target.
Adversarial Subspaces: 25-dimensional adversarial subspaces are observed for adversarial examples transferring between two fully connected MNIST networks.The result illustrates that transferable adversarial inputs can occupy a large contiguous space.
Decision Boundaries: Decision boundaries from different hypothesis classes are close in arbitrary directions, including both adversarial and benign directions.The average distance separating two models’ boundaries is smaller than the distance from data points to a boundary in those directions.
The Limits of Transferability: The hypothesis that low-error, non-robust models necessarily transfer adversarial examples is not true in general.The paper derives sufficient transferability conditions for simple model classes while also constructing a dataset where transfer fails between linear and quadratic models.
Contributions: The paper introduces methods for measuring adversarial-subspace dimensionality, studies decision-boundary similarity, and formally characterizes transferability and its failures.These are presented as the paper’s main contributions.

2 Adversarial Example Generation

The paper defines test-time untargeted attacks by adding bounded perturbations to clean inputs and uses gradient-based methods to generate adversarial examples. Inputs are constrained to remain in the relevant domain for MNIST and DREBIN.

Attack Setting: The attacks target untargeted misclassification, meaning the model predicts any class other than the ground truth.The paper studies adversarial examples generated at test time.
Notation: An adversarial input is formed as x∗ = x + r, with perturbation magnitude ϵ defined by an appropriate norm.The notation supports ℓ∞ and ℓ2 constraints among other possibilities.
Domain Constraints: MNIST inputs are clipped to [0, 1], while DREBIN feature values are rounded to binary values to enforce domain membership.The DREBIN models use a balanced, feature-selected subset with 1,000 features and exceed 94% accuracy.
Gradient-Based Generation: FGSM generates fixed-ℓ∞ adversarial examples by adding ε times the sign of the input-loss gradient.For general ℓp norms, the paper uses the corresponding fast gradient method without the sign.

3 Exploring the Space of Transferable Adversarial Examples

The GAAS method estimates adversarial-subspace dimensionality by finding orthogonal perturbations aligned with the loss gradient. Experiments show dense, transferable subspaces, with transfer depending on the model pair.

3.1 Gradient Aligned Adversarial Subspace (GAAS): GAAS searches for a maximal set of orthogonal perturbations with bounded ℓ2 norm that each increase the loss by at least γ.Under a first-order loss approximation, these directions estimate the dimensionality of the adversarial subspace.
3.1 Gradient Aligned Adversarial Subspace (GAAS): The number of orthogonal directions grows with the required loss increase and decreases with loss smoothness and perturbation magnitude.When the misclassification threshold is unknown, GAAS tries multiple γ values and retains the set producing the most misclassified perturbations.
3.1 Gradient Aligned Adversarial Subspace (GAAS): GAAS is the most successful evaluated technique for finding many orthogonal attack directions, supporting overly linear neural-network behavior on out-of-sample data.The comparison refers to alternative techniques reported in the appendix.
3.2 Experiments: 44.28 orthogonal perturbations were found on average for the MNIST source network, including 24.87 directions transferring to the target network.Random points from the spanned space fooled the source in 99% of cases and the target in 89% of cases.
3.2 Experiments: CNNs produced 15.18 source directions and 2.24 transferring directions, while sampled subspace points misled the source and target in 80% and 63% of cases.The FGM transfer rate for this CNN pair was 68%.
3.1 Gradient Aligned Adversarial Subspace (GAAS): Figure 1 depicts a gradient-aligned attack crossing the decision boundary and orthogonal gradient-aligned vectors spanning the potential adversarial-input subspace.The orange region represents the spanned subspace.
3.2 Experiments: Figure 2 compares the distributions of successful orthogonal perturbation counts on the source model with counts that transfer to the target model.The plotted quantity is a probability density function over perturbation counts.

4 Decision Boundary Similarity enables Transferability

The paper measures how closely different models’ decision boundaries align along legitimate, adversarial, and random directions, finding that inter-boundary distances are generally smaller than distances from data points to either boundary. This proximity helps explain adversarial transferability, including transfer to adversarially trained models.

Distance Definitions: Three linear directions—legitimate, adversarial, and random—measure decision-boundary similarity relative to each data point.The directions use a closest differently labeled test point, an adversarial example, or a random differently classified input, respectively.
Distance Definitions: The study compares minimum distances from inputs to a boundary with inter-boundary distances between model pairs using MNIST and DREBIN models.Distances use the ℓ2 norm, are averaged over test points, and are measured with line search; FGM finds adversarial directions for differentiable models.
Decision Boundary Similarity: Minimum distances are smallest in adversarial directions and larger in random directions than between differently labeled classes.This supports the observation that random noise usually does not cause misclassification.
Decision Boundary Similarity: For most model pairs and directions, inter-boundary distances are smaller than the corresponding minimum distances from test inputs to decision boundaries.In adversarial directions, this makes perturbations crossing one model’s boundary likely to cross the other’s as well.
Impact of Adversarial Training: Adversarially trained models increase inter-boundary distance but do not prevent transferability from undefended models.The adversarial-direction distance rises from 0.32 to 0.63, while the required total perturbation remains below the adversarial perturbation norm 5.

5 Limits of Transferability

The paper derives sufficient conditions for model-agnostic perturbations to transfer across feature representations, then demonstrates both transferable and non-transferable adversarial examples. The results show that transferability depends on how models preserve input-space features, rather than being inherent to non-robust models.

5.1 Sufficient Conditions for Transferability: Model-agnostic perturbations shift inputs along differences between class means and can transfer from input space to richer latent feature spaces under sufficient conditions.The conditions involve alignment with the feature-space class-mean difference and approximate preservation of the perturbation direction.
5.1 Sufficient Conditions for Transferability: The analysis relates transferability to whether feature mappings preserve the input-space direction between class means.For richer latent feature spaces, robustness depends on how this direction is preserved in feature space.
5.1 Sufficient Conditions for Transferability: Theorem 2 guarantees non-zero fooling probability when weight alignment is sufficiently large and the feature mapping is pseudo-linear in the perturbation.Pseudo-linearity requires a large component aligned with the feature-space class-mean difference and a small orthogonal component.
5.1 Sufficient Conditions for Transferability: On MNIST, model-agnostic perturbations partially fool linear, quadratic, DNN, and CNN models, while the CNN misclassifies 24% of perturbed samples despite lacking the pseudo-linearity guarantee.The experiments distinguish formal vulnerability conditions from empirical fooling performance.
5.1 Sufficient Conditions for Transferability: For full MNIST, the multi-class mean-difference perturbation with ϵ = 0.3 yields accuracy from 2% for the linear model to 66% for the CNN.The perturbation is defined through pairwise differences in class means and can use the ℓ∞ norm.
5.2 XOR Artifacts: Transferability can fail when models learn different predictive representations: on an altered MNIST task, linear and quadratic models remain individually vulnerable but their perturbations do not transfer.Fooling both models requires combining the distinct perturbations, suggesting that shared input features do not ensure shared adversarial directions.

A Neural Network Architectures

The paper uses several approaches to find multiple orthogonal adversarial directions, including optimization-based, saliency-based, and approximation-based methods. These methods support measuring adversarial-subspace dimensionality and transferability.

Table 2 defines the neural-network architecture notation used in the experiments: Conv denotes convolutional layers and FC denotes fully connected layers.
The first two techniques iteratively solve optimization problems while enforcing orthogonality, whereas the latter two estimate dimensionality using first-order approximations.
Second-Order Approximations: Second-order approximation maximizes a quadratic loss model using the gradient g and Hessian H with a norm-bounded perturbation.
Second-Order Approximations: Orthogonal second-order directions are obtained by projecting the gradient and Hessian into the subspace orthogonal to previously found perturbations.
Convex Optimization: For piecewise-linear models, convex optimization searches for perturbations satisfying region-membership and norm constraints, then iterates with an added orthogonality constraint.
Convex Optimization: A successful perturbation satisfying the optimization constraints must misclassify the input, possibly as a class different from the target.
Independent JSMA: Independent JSMA partitions salient discrete features into independent bins, producing orthogonal perturbations under a target dimensionality and perturbation budget.
Table 3 compares methods by single-direction transfer rate, source-model orthogonal perturbations, and those perturbations transferring to the target model.

B.1 Prefiltered Evaluation of the Transfer Rate

The prefiltered evaluation metric isolates transferability by measuring target accuracy only on examples that satisfy conditions for successful, model-specific attacks.

Prefiltered evaluation requires that the original example fool neither model and that the adversarial example fool the source model.
The metric reports target-model accuracy on these filtered adversarial examples to focus on transferability rather than natural model inaccuracy or attack failure.

B.2 Experiments

Experiments show that methods differ substantially in the number and transferability of orthogonal perturbations. Convex optimization finds dense source-model subspaces but transfers poorly, while Independent JSMA finds many transferable perturbations on DREBIN.

Second-Order Approximations: Second-order approximation is of little use for finding further adversarial directions because the Hessian is small and the best orthogonal solution is non-adversarial.
Convex Optimization: 93% of MNIST test inputs yield a convex-optimization adversarial example, with slightly over 2 orthogonal perturbations on average and 47 for the most vulnerable input.
Convex Optimization: 17% of convex-optimization perturbations transfer, compared with 95% for FGM; for 70% of inputs, no successful source perturbation transfers.
Convex Optimization: The convex-optimization approach can find smaller perturbations and more directions than FGM, but is less effective for producing transferable perturbations.
Convex Optimization: The convex approach can fail entirely: for one architecture and an l2 budget of 5, its LP is solvable for 0% of tested inputs while FGSM always succeeds.
Independent JSMA: On DREBIN, Independent JSMA finds at least one successful perturbation for 89% of inputs, with 42 independent perturbations and 22 transferring on average.
Independent JSMA: The DREBIN study did not verify that every perturbation preserved the program’s malicious or non-malicious nature.

C Proofs

The proof constructs orthogonal perturbations by rotating the gradient and selecting orthonormal directions with controlled alignment. It also establishes the trivial dimensional upper bound.

If k orthogonal perturbations lie in R^d, their number cannot exceed the ambient dimension: k ≤ d.
The lower-bound construction uses the basis vectors of R^d and a rotation matrix that maps the gradient into the first basis direction.
A second rotation maps the constructed vector to e1, yielding Q = S⊤R with Qg aligned to a vector whose first k coordinates are equal.
The vectors ri = Q⊤ei for 1 ≤ i ≤ k are orthonormal and each has the same positive inner product with the gradient.

C.2 Proof of Theorem 2

The proof derives bounds on latent-space perturbations that fool a linear classifier, then analyzes how aligned and orthogonal components affect transferability in input space.

Latent-space perturbation: The analysis defines rφ as the smallest latent perturbation aligned with δφ that causes misclassification.The classifier is assumed nonconstant, and the expected 0-1 loss is denoted L(f).
Latent-space perturbation: Lemma 3 extends prior linear-model analysis by making perturbation magnitude depend on the alignment ∆ between the classifier direction w and mean-difference direction δφ.The result assumes positive alignment and connects small class-mean differences with vulnerability to small perturbations.
Transferability: The proof decomposes the classifier score change into components α·δφ and β·δ⊥φ, separating aligned and orthogonal effects.This decomposition is then used to study transferability in input space.
Transferability: When ∆≫L(f) and β=0, the average misclassification threshold for −y·α is approximately 1.The orthogonal component can require increasing the absolute magnitude of α in the worst case.
Transferability: The bound is expected to be loose because δ⊥φ is unlikely to be maximally aligned with w in practice.The analysis therefore gives a conservative estimate for the orthogonal contribution.

D Appendix: Additional Distance Measurements

The paper supplements earlier distance analyses with inter-boundary measurements across MNIST and DREBIN models, using multiple directions and norms to compare source and target boundaries.

Additional distance measurements: The ℓ1 inter-boundary distances for MNIST models appear in Figure 7a, with analogous DREBIN results in Figures 7b and 7c.The corresponding ℓ2 results were reported earlier in Figure 4.
Additional distance measurements: Figure 7 reports minimum and inter-boundary distances between models trained on MNIST and DREBIN.Each plot uses one source model and includes all three target-model classes.
Additional distance measurements: The measurements group bars by legitimate, adversarial, and random directions, while the filled black bar gives the source model’s minimum boundary distance.Adversarial searches use FGM with either the ℓ1 or ℓ2 norm.