PaperlayerView exact revision on arXiv

Source-linked AI summary

Constructing Unrestricted Adversarial Examples with Generative Models

Yang Song, Rui Shu, Nate Kushman, Stefano Ermon

arXiv:1805.07894v4cs.LGcs.AIcs.CRcs.CVstat.ML

TL;DR

Existing adversarial defenses largely target small norm-bounded perturbations, leaving open whether classifiers remain robust to semantically valid images generated from scratch. The paper trains a class-conditional AC-GAN and searches its latent space for images that humans assign to one class but a target classifier misclassifies. Across MNIST, SVHN, and CelebA, these unrestricted examples bypass adversarial training and certified defenses, with attacks exceeding 84% success rates.

  • Problem

    Current adversarial defenses are focused on attacks that perturb existing data points within small matrix norms.

  • Method

    The paper trains an AC-GAN for class-conditional image generation and searches its latent space for high-fidelity images misclassified by a target classifier.

  • Results

    Across MNIST, SVHN, and CelebA, unrestricted attacks uniformly achieve over 84% success rates and bypass strong adversarial training and certified defenses.

  • Takeaways & Limitations

    Unrestricted adversarial examples expose vulnerabilities in defenses designed for perturbation-based attacks and can transfer to classifiers trained on the same dataset.

  • Takeaways & Limitations

    The theoretical legitimacy analysis for imperfect generative models relies on relatively strong assumptions.

Abstract

from arXiv · show

Adversarial examples are typically constructed by perturbing an existing data point within a small matrix norm, and current defense methods are focused on guarding against this type of attack. In this paper, we propose unrestricted adversarial examples, a new threat model where the attackers are not restricted to small norm-bounded perturbations. Different from perturbation-based attacks, we propose to synthesize unrestricted adversarial examples entirely from scratch using conditional generative models. Specifically, we first train an Auxiliary Classifier Generative Adversarial Network (AC-GAN) to model the class-conditional distribution over data samples. Then, conditioned on a desired class, we search over the AC-GAN latent space to find images that are likely under the generative model and are misclassified by a target classifier. We demonstrate through human evaluation that unrestricted adversarial examples generated this way are legitimate and belong to the desired class. Our empirical results on the MNIST, SVHN, and CelebA datasets show that unrestricted adversarial examples can bypass strong adversarial training and certified defense methods designed for traditional adversarial attacks.

1 Introduction

The paper broadens adversarial attacks beyond small perturbations by generating examples from scratch, showing that such examples can remain human-legitimate while bypassing defenses designed for perturbation-based attacks.

  • Current defenses primarily improve classifier robustness to small input perturbations.Methods include adversarial training, perturbation removal, smoothness objectives, and certified defenses.
  • The proposed threat model constructs adversarial examples entirely from scratch rather than modifying an existing data point within a small norm.The paper illustrates this distinction using a newly manufactured adversarial stop sign.
  • Human-legitimate examples can fool classifiers without confusing people, even when the examples differ in font or size from familiar instances.The paper defines security-relevant inputs by whether humans still recognize their intended semantics.
  • The results indicate that existing defenses, including certified methods, are ineffective against this more general attack.These defenses were designed around robustness to small perturbations rather than unrestricted synthesis.
  • AC-GAN latent-space search generates class-conditioned unrestricted examples evaluated on MNIST, SVHN, and CelebA.The method combines generative modeling with targeted search for classifier-misclassified images.
  • Unrestricted attacks uniformly achieve over 84% success rates and show moderate transferability to other architectures.Human evaluation verifies that successful examples are legitimate and belong to the desired class.

2 Background

The background contrasts norm-bounded adversarial examples and their defenses with generative modeling tools that can produce label-controlled synthetic images.

  • Adversarial examples: Traditional adversarial examples modify an input x within a norm bound so the classifier changes its prediction.Prior methods use L-BFGS, FGSM, PGD, and l∞, l2, or l0 norms.
  • Defense methods: Existing defenses generally target small image perturbations, especially adversarial training and certified defenses.The paper focuses its unrestricted-attack investigation on these two defense families.
  • Generative adversarial networks: A GAN maps noise through a generator to create synthetic images while a discriminator distinguishes generated images from the true distribution.GAN training uses the discriminator and generator to model image distributions.
  • Conditional generation: AC-GAN adds label conditioning through a generator input y and an auxiliary classifier that predicts labels for training and generated images.The auxiliary classifier provides class information needed for semantic control.
  • Conditional generation: The paper uses N(0, 1) as the latent prior in its experiments.The supplied formulation also defines the interpolated distribution and label-related distributions used in training.

3 Methods

The method defines unrestricted examples as classifier–oracle disagreements, models legitimate class-conditional images with a generator, and searches its latent variables for targeted misclassification.

  • 3.1 Unrestricted adversarial examples: Perturbation-based examples require a correctly classified test image, a small norm bound, and an oracle-approved label change.They are defined relative to an existing image in the test set.
  • 3.1 Unrestricted adversarial examples: An unrestricted adversarial example is any image for which the oracle label differs from the classifier prediction.This removes the requirement that the example be close to a test image.
  • 3.1 Unrestricted adversarial examples: Unrestricted examples strictly generalize perturbation-based examples by removing small-norm constraints while retaining human-realistic semantics.The generative model approximates the set of realistic images that humans assign to each class.
  • 3.2 Practical unrestricted adversarial attacks: The generator maps latent noise and a class label to a legitimate image, enabling searches over latent codes for classifier failures.AC-GAN is selected for reliable conditioning and high-fidelity image generation.
  • 3.2 Practical unrestricted adversarial attacks: The basic targeted attack minimizes a loss over z so the oracle-prescribed source class is generated while the target classifier predicts a chosen target class.The loss is designed to produce high-fidelity unrestricted adversarial examples.
  • 3.2 Practical unrestricted adversarial attacks: A latent-distance term keeps z near a random z0, promoting diverse source-class examples instead of repeated convergence to one image.The random reference is sampled from a standard normal latent distribution.
  • 3.2 Practical unrestricted adversarial attacks: Noise augmentation adds small trainable image-space noise to improve the AC-GAN generator’s representation power.The attack then optimizes both latent code and auxiliary noise while intending to preserve image quality and ground-truth labels.
  • 3.2 Practical unrestricted adversarial attacks: Traditional perturbation attacks arise as a special case when the generator selects images from the test set and the attack uses suitable limiting weights.This connects the unrestricted formulation to existing perturbation-based attacks.

4 Experiments

The experiments evaluate unrestricted adversarial examples using MTurk human labels across MNIST, SVHN, and CelebA, including attacks against certified and adversarially trained defenses. The results show high attack success, greater realism than large-ϵ perturbation attacks, and moderate transferability.

  • Experimental details: MTurk majority votes provide ground-truth labels for assessing whether synthesized examples belong to their intended classes.Each image is assigned to five workers, with the majority vote used as ground truth.
  • Experimental details: The evaluation covers MNIST, SVHN, and CelebA, using digit categorization and female/male gender classification tasks.These datasets were selected because their labels are comparatively easy for MTurk workers to judge.
  • Untargeted attacks against certified defenses: 88.8% of unrestricted samples fool certified defense [17] on MNIST despite its 94.2% certificate rate at ϵ = 0.1.The attack is not perturbation-based, so the certificate does not apply to these synthesized examples.
  • Targeted attacks against adversarial training: Adversarial training limits basic perturbation attacks to 10.4% on MNIST and 20.5% on CelebA, whereas unrestricted attacks exceed 84% on all datasets.Noise augmentation changes results modestly, increasing SVHN success by 7.4% but decreasing CelebA success by 4.4%.
  • Transferability: Unrestricted examples show moderate transferability, reaching 22.9% against [16] without noise and 37.0% against [17] with noise augmentation.The reported results indicate that noise augmentation generally enhances transferability to different classifiers.

5 Analysis

The analysis explains unrestricted attacks through latent-space sensitivity: small latent perturbations can alter classifier predictions when the latent dimension is large, while imperfect generators complicate legitimacy guarantees.

  • Idealized analysis: For an ideal generator, perturbing the latent code can change the classifier’s prediction while preserving the generator’s desired-class conditioning.The end-to-end score change is approximated as ϵ · m, where m is the latent dimension.
  • Idealized analysis: Because the latent dimension m is large, the resulting score change can be sufficient to produce many unrestricted adversarial examples.The method searches for such examples by minimizing L.
  • Imperfect generators: Analyzing imperfect generators is harder, so the paper relies on relatively strong assumptions in Appendix A to argue that most generated examples remain legitimate.The guarantee is therefore conditional rather than unconditional.

6 Related work

Related attacks extend beyond simple norm-bounded perturbations, but the paper distinguishes its unrestricted examples by requiring generated inputs to remain legitimate to humans.

  • Structured perturbations: Prior work uses structured changes such as eyeglass frames, geometric distortions, occlusions, illumination changes, and color-space shifts.These attacks modify specific nuisance or appearance factors rather than necessarily synthesizing images from scratch.
  • Fooling images: Fooling images need not look realistic, whereas unrestricted adversarial examples are intended to look legitimate to humans.Both attack types are not restricted to small norm-bounded perturbations.
  • GAN-based attacks: AdvGAN uses GANs but still constructs adversarial examples from small norm-bounded perturbations.That restriction permits assuming the adversarial and unperturbed images share ground-truth labels.

7 Conclusion

The paper proposes unrestricted adversarial examples synthesized from scratch with conditional generative models and reports that they undermine perturbation-focused defenses and transfer across classifiers.

  • 7 Conclusion: Unrestricted adversarial examples are synthesized entirely from scratch using conditional generative models rather than perturbing existing data points.This defines the paper’s broader threat model.
  • 7 Conclusion: The experiments show that unrestricted adversarial examples undermine current defenses designed for perturbation-based attacks.The conclusion identifies this weakness as a central empirical finding.
  • 7 Conclusion: Unrestricted adversarial examples transfer to other classifiers trained using the same dataset.The conclusion also notes broader interest in more general adversarial examples.
  • 7 Conclusion: Both traditional and unrestricted attacks exploit classifier vulnerability to covariate shift.The paper connects this vulnerability to the lack of guaranteed performance under different input distributions.

A Analysis of imperfect generators

The imperfect-generator analysis argues that classifier–generator label disagreements are more likely classifier errors under strong assumptions, using a robustness relationship tied to input and output dimensions.

  • Imperfect generators: The analysis assumes generators may produce illegitimate images and argues that contradictory classifier predictions are more likely classifier mistakes than generator failures.This conclusion is explicitly conditional on strong assumptions.
  • Proposition 1: Proposition 1 characterizes the robustness of a typical linear function as a function of its input and output dimensions.It assumes a random weight matrix with mutually independent, bounded entries.
  • Proposition 1: When the output dimension n is fixed, the average maximum perturbation is upper bounded by O(m), decreasing as the input dimension m becomes smaller.The relationship persists when weight-matrix entries are mutually independent and bounded.
  • Nonlinear extension: The paper extends this robustness–dimension relationship heuristically to nonlinear functions by linearizing them with Jacobians under analogous assumptions.The authors explicitly state that this extension is not rigorously proven.
  • Unrestricted attacks: For unrestricted attacks, the generator maps a latent noise vector and label to an image, while the classifier maps the image to output scores.In practice, the latent dimension l is much smaller than the image dimension m, and m is much larger than the classifier output dimension n.

Proof of Proposition 1

The proof bounds the maximum of ∥W∆x∥1 over an ℓ∞ perturbation ball by first analyzing fixed perturbations and then reducing the optimization to finitely many vertices.

  • Fixed-vector bound: The proof writes ∥W∆x∥1 as a sum of n terms before bounding it for a fixed ∆x in Bϵ.Bϵ is the set of vectors satisfying ∥x∥∞≤ϵ.
  • Fixed-vector bound: Markov inequality is used to derive a probability bound for the fixed-vector quantity.
  • Fixed-vector bound: The proof optimizes s because the preceding inequality holds for every s ∈ R, obtaining the tightest bound.
  • Union bound: The infinite perturbation ball Bϵ is reduced to its finite vertex set Vϵ for applying the union bound.The vertices have every coordinate equal to either −ϵ or ϵ.
  • Union bound: Convex decomposition and the triangle inequality show that some vertex achieves the same maximum ∥W∆x∥1 as the original maximizer.
  • Union bound: The union-bound argument completes the theorem’s proof after restricting attention to Vϵ.

C Detailed experimental settings

The appendix specifies datasets, architectures, attack algorithms, hyperparameters, and adversarial-training procedures used for the experiments.

  • Datasets: The experiments use MNIST, SVHN, and CelebA, with dataset sizes and image formats described for each benchmark.MNIST and SVHN contain digit images, while CelebA contains celebrity faces.
  • Attack procedure: The attack is defined as a targeted procedure that samples latent and auxiliary variables, updates them by gradients, and returns a successful generated example.The returned image is generated by gθ and accepted when the classifier predicts the target class.
  • Attack procedure: The attack objective includes a classifier-targeting loss together with additional weighted terms controlled by λ1 and related hyperparameters.The displayed objective begins with −log f(ytarget | gθ(z, ysource) + ϵattack tanh(τ)).
  • Adversarial training: Adversarial training samples ϵ from N(0, 8), truncates its absolute value to [0, 16], and uses PGD with a specified iteration count.
  • Model architectures: The AC-GAN architectures are adapted from prior designs, with reduced generator depth and output channels for MNIST.The classifier networks and weights are obtained from the authors of prior defense studies for consistency.
  • Hyperparameters: Table 4 lists the attack hyperparameters using the notation of Algorithm 1, including male and female target-class annotations.

D Additional samples

The appendix provides additional samples of targeted and untargeted unrestricted adversarial examples across MNIST, SVHN, and CelebA-related settings, with and without noise.

  • MNIST: Figure 5 shows random untargeted examples without noise against certified defenses on MNIST, with borders indicating human-evaluated success or failure.Upper-left annotations show classifier predictions.
  • Attack results: Figure 6 reports success rates for the targeted unrestricted attack with noise augmentation.
  • MNIST: Figure 7 shows targeted MNIST examples without noise, while Figure 8 shows targeted MNIST examples with noise.Green borders indicate images judged legitimate; red borders indicate disagreement between worker labels and source class.
  • SVHN: Figure 10 shows random targeted unrestricted adversarial examples with noise on SVHN, using worker legitimacy and label-agreement borders.

E MTurk web interfaces

The appendix documents the MTurk interfaces used to label unrestricted adversarial examples and conduct the MNIST A/B test, alongside annotator agreement statistics.

  • Interfaces: The labeling and MNIST A/B-test interfaces are shown in Figures 13 and 14, respectively.
  • Annotator agreement: More than 40%–50% of unrestricted adversarial examples receive the same label from all five MTurk annotators, depending on the dataset.
  • Annotator agreement: Figure 15 visualizes the distribution of the number of agreeing votes for each image.A red bar over 5 represents examples for which all five annotators selected the same label.
Loading 1805.07894v4…