PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards the first adversarially robust neural network model on MNIST

Lukas Schott, Jonas Rauber, Matthias Bethge, Wieland Brendel

arXiv:1805.09190v3cs.CV

TL;DR

Deep neural networks remain vulnerable to small, human-imperceptible perturbations, and MNIST lacks a model whose adversarial examples are both large and semantically meaningful. The paper introduces analysis by synthesis with class-conditional generative models and evaluates it using diverse, model-specific attacks. It reports large, semantically meaningful perturbations across tested Lp norms, while acknowledging unresolved evaluation and scalability limitations.

  • Problem

    MNIST remains without a neural network whose adversarial perturbations are large and semantically meaningful to humans, despite extensive robustness research.

  • Method

    The paper introduces an analysis-by-synthesis classifier using learned class-conditional distributions and evaluates it with diverse attacks, including attacks tailored to its structure.

  • Results

    The paper reports minimal adversarial perturbations that are large across all tested Lp norms and semantically meaningful to humans.

  • Takeaways & Limitations

    The approach has potential to reduce adversarial vulnerability and align machine perception with human perception on MNIST.

  • Takeaways & Limitations

    The naive one-VAE-per-class implementation scales poorly to more classes and complex datasets, with 54% test accuracy in a preliminary CIFAR10 experiment.

Abstract

from arXiv · show

Despite much effort, deep neural networks remain highly susceptible to tiny input perturbations and even for MNIST, one of the most common toy datasets in computer vision, no neural network model exists for which adversarial perturbations are large and make semantic sense to humans. We show that even the widely recognized and by far most successful defense by Madry et al. (1) overfits on the L-infinity metric (it's highly susceptible to L2 and L0 perturbations), (2) classifies unrecognizable images with high certainty, (3) performs not much better than simple input binarization and (4) features adversarial perturbations that make little sense to humans. These results suggest that MNIST is far from being solved in terms of adversarial robustness. We present a novel robust classification model that performs analysis by synthesis using learned class-conditional data distributions. We derive bounds on the robustness and go to great length to empirically evaluate our model using maximally effective adversarial attacks by (a) applying decision-based, score-based, gradient-based and transfer-based attacks for several different Lp norms, (b) by designing a new attack that exploits the structure of our defended model and (c) by devising a novel decision-based attack that seeks to minimize the number of perturbed pixels (L0). The results suggest that our approach yields state-of-the-art robustness on MNIST against L0, L2 and L-infinity perturbations and we demonstrate that most adversarial examples are strongly perturbed towards the perceptual boundary between the original and the adversarial class.

1 INTRODUCTION

The paper argues that MNIST remains unsolved for human-relevant adversarial robustness because existing defenses can fail across metrics and produce meaningless or unrecognizable adversarial inputs. It introduces an analysis-by-synthesis classifier with class-conditional generative models and evaluates it extensively.

  • Madry et al.'s defense overfits L∞, with L2 and L0 adversarial metrics as small as for undefended networks.
  • The defense can classify unrecognizable images with high certainty, and its minimum adversarial examples make little to no sense to humans.
  • MNIST remains unsolved because robust models should achieve at least 99% accuracy while producing adversarial examples that carry semantic meaning to humans.
  • The paper introduces a robust classification model based on analysis by synthesis and derives instance-specific robustness guarantees.
  • The evaluation uses a strong attack exploiting the model's generative structure and a novel decision-based attack minimizing L0.
  • The authors conduct extensive evaluation across attacks and report state-of-the-art robustness on L0, L2, and L∞, while acknowledging unavoidable evaluation limitations.

2 RELATED WORK

Prior defenses include adversarial training, manifold projections, stochasticity, and preprocessing, but most have not substantially improved robustness over undefended networks. The paper instead models class-conditional input distributions and classifies samples by likelihood.

  • Adversarial training augments training data with adversarial examples to improve robustness against them.
  • Manifold-projection defenses project inputs or hidden activations onto a learned manifold of natural inputs.
  • Stochasticity-based defenses randomize selected inputs or hidden activations, while preprocessing methods quantize or otherwise transform representations.
  • Most proposed defenses did not substantially increase robustness over undefended networks; Madry et al.'s adversarial training was identified as the notable exception.
  • The paper's model optimizes a latent distribution in each digit model to estimate each class's likelihood, then decides using the most likely class.
  • Unlike manifold projection approaches, the method models the input distribution separately within each class and uses class likelihoods without an additional vulnerable classifier.

3 MODEL DESCRIPTION

ABS models each class-conditional input distribution and classifies samples by their highest likelihood, using VAEs trained per class. Evaluation uses optimization-based latent inference rather than the attackable inference network, with calibrated confidence for unfamiliar inputs.

  • Classification: Classification selects the class under which a new sample has the highest likelihood, avoiding an additional vulnerable classifier.
  • Class-conditional distributions: ABS trains one variational autoencoder per class to learn class-conditional distributions p(x|y).These models provide a lower bound ℓ_y(x) on each class's log-likelihood.
  • Optimization-based inference: During evaluation, ABS performs exact inference by optimizing in latent space instead of using the adversarially susceptible variational-inference network.Gradient descent finds the latent code maximizing the likelihood lower bound for each class.
  • Optimization-based inference: The model replaces the likelihood expectation with a maximum-likelihood sample to avoid stochastic sampling and simplify optimization.
  • Classification and confidence: Adding η > 0 makes predictions approach a uniform class distribution when all class likelihoods are small, matching behavior on unrecognizable inputs.η is chosen so median confidence on clean test samples is 0.9 for the predicted class.

4 TIGHT ESTIMATES OF THE LOWER BOUND FOR ADVERSARIAL EXAMPLES

The paper derives robustness lower bounds from the smooth change of Gaussian class-conditional likelihoods under input perturbations. The bound depends on distances between the input and each class generator, with global latent optimization as a key assumption.

  • Robustness bounds: Gaussian class-conditional likelihoods change gracefully with input perturbations, enabling lower bounds on model robustness.
  • Robustness bounds: The likelihood expression combines a latent-prior mismatch term, a reconstruction penalty, and a normalization constant.The reconstruction penalty uses the squared distance between generated output G_c(z) and input x.
  • Robustness bounds: For a clean sample, the analysis bounds the optimal ground-truth-class likelihood after a perturbation of size ε = ∥δ∥₂.
  • Robustness bounds: The analysis also bounds optimal likelihoods for every competing class and uses equality between these bounds to estimate the perturbation threshold.
  • Assumption: The derivation assumes the global minimum of ∥G_c(z) − x∥₂ can be found, which is estimated tightly in practice through smooth low-dimensional optimization and brute-force sampling.

5 ADVERSARIAL ATTACKS

The evaluation combines decision-, score-, gradient-, transfer-, and model-specific attacks across norms, including new attacks for the ABS structure and L₀ distance. Robustness estimates remain upper bounds because stronger attacks may exist.

  • Evaluation strategy: The evaluation applies many attacks and retains the strongest result for each sample-model pair to tighten adversarial-distance estimates.Each attack supplies only an upper bound on the perturbation size.
  • Model-specific attack: The Latent Descent attack exploits ABS structure by moving perturbed samples toward a high-likelihood posterior of a competing class.
  • Decision-based attacks: Decision-based evaluation uses Boundary Attack for L₂ minimization and introduces Pointwise Attack to greedily minimize L₀ perturbations.
  • Transfer-based attacks: Transfer attacks compute perturbations on substitute models, then line-search the smallest scaling that remains adversarial for the target.
  • Gradient-based attacks: Gradient-based evaluation includes MIM, BIM/PGD, FGSM, and FGM under L₂ and L∞ norms, using straight-through gradients for binary models.
  • Evaluation strategy: Accuracy-distortion plots compare accuracy as perturbation size increases across distance metrics and models.At threshold 0.3, Madry et al. is favored in panel (b), whereas 0.35 favors Binary ABS.
  • Score-based attacks: Score-based attacks repeat gradient-based methods with coordinate-wise finite-difference gradients and varied estimator step sizes.
  • Postprocessing: For binary models, postprocessing replaces adversarial pixel values with clean-image values or the 0.5 threshold without changing decisions.

6 EXPERIMENTS

Experiments compare ABS and binary ablations with adversarially trained, nearest-neighbor, and vanilla CNN baselines using attacks across L₂, L∞, and L₀ norms. Accuracy is tracked against increasing perturbation size and summarized by median distance and bounded accuracy.

  • Models and evaluation: The comparison includes ABS, Binary ABS, Binary CNN, Madry et al.'s L∞ defense, Nearest Neighbour, and vanilla CNN models.All applicable attacks are run against every model.
  • Models and evaluation: Robustness is evaluated under L₂, L∞, and L₀ perturbations, with accuracy reported as perturbation size increases.All models except Nearest Neighbour perform close to 99% accuracy on clean test samples.
  • Metrics: The median adversarial distance represents the perturbation size at which a model reaches 50% accuracy and is robust to outliers.The second reported metric is accuracy against bounded adversarial perturbations.

7 RESULTS

Across L2, L∞, and L0 evaluations, ABS models show strong robustness and produce adversarial examples that are more perceptually meaningful than those of competing models. Results also expose metric-specific weaknesses in Madry et al.’s defense.

  • ABS outperforms all other models by a large margin for L2 robustness.
  • Binary ABS achieves state-of-the-art median perturbation size for L∞, while Madry et al.’s apparent threshold advantage depends on ε = 0.3.At ε = 0.35, the accuracy-distortion curves strongly favor Binary ABS.
  • The evaluation reports median adversarial distances and accuracy under bounded perturbations for L2, L∞, and L0 metrics.Table 1 uses thresholds ε_L2 = 1.5, ε_L∞ = 0.3, and ε_L0 = 12.
  • ABS and Binary ABS are much more robust than all other models for L0, while Madry et al. is the least robust model.
  • ABS adversarial examples are clearly visible, often semantically meaningful, and close to the perceptual boundary between the original and adversarial classes.Madry et al.’s perturbations are clearly visible but make little semantic sense to humans.
  • ABS avoids high-confidence predictions on unrecognizable images, unlike standard CNNs and Madry et al.’s model.

8 DISCUSSION & CONCLUSION

The paper concludes that MNIST remains unsolved under a human-perception standard, while analysis by synthesis yields large, semantically meaningful perturbations. It also acknowledges evaluation challenges and limitations in scaling the naive ABS implementation.

  • The authors conclude that current neural networks are not robust on MNIST from the perspective of human perception.
  • ABS uses analysis by synthesis to explain inference through actual image features and produces large, semantically meaningful perturbations across tested Lp norms.
  • Reliable robustness evaluation remains difficult because many proposed defenses were later shown to be ineffective.
  • Broad attack coverage is important because no single attack works best on every model, and combining attacks for one metric often improves evaluation.
  • The naive one-VAE-per-class ABS implementation does not scale efficiently to more classes or complex datasets.A preliminary CIFAR10 experiment achieved 54% test accuracy.
  • The paper frames ABS as having potential to reduce adversarial vulnerability and align machine perception with human perception.

A APPENDIX

The appendix samples minimally perturbed adversarial examples across four error quantiles for L0, L2, and L∞ metrics, with a separate distribution comparison across models.

  • Figure 6: L2 adversarials are selected by minimum perturbation across attacks and sampled within four error quantiles.Four images are randomly sampled from each quantile interval.
  • Figure 7: L∞ adversarials are selected by minimum perturbation across attacks and sampled within four error quantiles.The quantile intervals range from 0–25% through 75–100%.
  • Figure 8: A threshold of 0.3 favors Madry et al., whereas a threshold of 0.35 favors Binary ABS in the minimal-adversarial distributions.The comparison is shown in Figure 8b.

A.1 DERIVATION I

This derivation obtains the evidence lower bound using a normal latent prior, a variational posterior, importance sampling, and Jensen’s inequality.

  • ELBO derivation: The derivation starts with a simple normal prior p(z) = N(0, 1).The prior is used in the latent-variable formulation.
  • ELBO derivation: Importance sampling with variational posterior qφ(z|x) and Jensen’s inequality yields a lower bound on log pθ(x).The variational posterior has parameters φ.
  • ELBO derivation: The resulting lower bound is referred to as the ELBO.The derivation is identified as the derivation of equation 2.

A.2 DERIVATION II: LOWER BOUND FOR L2 ROBUSTNESS ESTIMATION

The appendix derives lower-bound-based robustness estimates for L2 perturbations and documents the model architectures, training settings, and numerical procedures used for evaluation.

  • L2 robustness derivation: For an L2 perturbation δ with size ϵ = ∥δ∥2, the appendix derives a lower bound for the perturbed sample’s objective.The derivation starts from equation 3 and uses the latent representation associated with the clean sample and class y.
  • L2 robustness derivation: The clean-sample latent vector is optimized for the predicted class y before estimating the perturbed lower bound.The same optimal latent is referenced in the derivation of equations 6 and 7.
  • L2 robustness derivation: The robustness bound uses dc = minz ∥Gc(z) − x∥2 and a constrained optimization result, while noting that a tighter bound may use one shared δ.The constraint is d > dc.
  • L2 robustness derivation: The L2 derivation has no closed-form minimization solution, so solutions are computed for each ϵ and searched by line search.The search identifies the point where the relevant equations coincide.
  • Implementation: The ABS and binary ABS models share a four-layer encoder–decoder architecture, while the CNN and binary CNN share a separate four-layer architecture.The ABS encoder uses kernel sizes [5, 4, 3, 5], and the CNN feature-map sizes are [20, 70, 256, 10].
  • Implementation: The VAEs use Adam, latent dimension L = 8, KL-weight annealing from 10 to 1 over 50 epochs, and line-search estimation of γ.The Madry et al. model is adapted from a pretrained architecture and trained on clean and adversarial examples.
  • Implementation: Scaling probabilities changes their values but does not change classification accuracy.This is stated explicitly in the implementation notes.
  • Implementation: The nearest-neighbour classifier imitates logits with negative within-class distances and calibrates confidence using the modified softmax.The reported parameter settings target median confidence p(y|x) = 0.9.
Loading 1805.09190v3…