PaperlayerView exact revision on arXiv

Source-linked AI summary

MagNet: a Two-Pronged Defense against Adversarial Examples

Dongyu Meng, Hao Chen

arXiv:1705.09064v2cs.CRcs.LG

TL;DR

Adversarial examples can fool classifiers with nearly imperceptible perturbations, creating a security problem for deployed deep-learning systems. MagNet combines attack-independent manifold-based detectors with a reformer, without modifying the protected classifier. It achieves above 99% accuracy on the considered MNIST attacks except Carlini L0, and above 99% across tested Carlini confidence levels, while the paper identifies limits in detector generalization and defense diversity.

  • Problem

    Small perturbations can preserve human recognition while causing neural-network misclassification, while existing defenses depend on particular attacks or modify the target classifier.

  • Method

    MagNet uses detectors trained only on normal examples to approximate the normal-data manifold, a reformer to move inputs toward it, and diverse defenses for graybox attacks.

  • Results

    MagNet achieved above 99% accuracy on all considered MNIST attacks except Carlini L0 at 92.0%, and above 99% on Carlini examples across all tested confidence levels.

  • Takeaways & Limitations

    MagNet provides an empirically effective, attack-independent defense in blackbox and graybox settings without training on the evaluated attacks.

  • Takeaways & Limitations

    The paper states that MagNet’s diversity can be improved through different architectures, training parameters, more autoencoders, and stronger encouragement of differences.

Abstract

from arXiv · show

Deep learning has shown promising results on hard perceptual problems in recent years. However, deep learning systems are found to be vulnerable to small adversarial perturbations that are nearly imperceptible to human. Such specially crafted perturbations cause deep learning systems to output incorrect decisions, with potentially disastrous consequences. These vulnerabilities hinder the deployment of deep learning systems where safety or security is important. Attempts to secure deep learning systems either target specific attacks or have been shown to be ineffective. In this paper, we propose MagNet, a framework for defending neural network classifiers against adversarial examples. MagNet does not modify the protected classifier or know the process for generating adversarial examples. MagNet includes one or more separate detector networks and a reformer network. Different from previous work, MagNet learns to differentiate between normal and adversarial examples by approximating the manifold of normal examples. Since it does not rely on any process for generating adversarial examples, it has substantial generalization power. Moreover, MagNet reconstructs adversarial examples by moving them towards the manifold, which is effective for helping classify adversarial examples with small perturbation correctly. We discuss the intrinsic difficulty in defending against whitebox attack and propose a mechanism to defend against graybox attack. Inspired by the use of randomness in cryptography, we propose to use diversity to strengthen MagNet. We show empirically that MagNet is effective against most advanced state-of-the-art attacks in blackbox and graybox scenarios while keeping false positive rate on normal examples very low.

1 INTRODUCTION

MagNet defends neural-network classifiers without modifying the target classifier or modeling a specific attack process. It combines detectors that identify inputs distant from the normal-example manifold with a reformer that moves nearby adversarial examples toward that manifold, and uses diversity for graybox defense.

  • MagNet’s defense: MagNet protects a wide range of neural networks by treating the target classifier as a blackbox and requiring only normal examples for training.It reads the classifier’s last-layer output but neither accesses internal layers nor modifies the classifier.
  • Motivation and contributions: MagNet’s attack-independent training avoids the process-specificity of defenses trained on adversarial examples.Existing adversarial-example detectors require modeling the attacker, which limits generalization to other generation processes.
  • MagNet’s defense: Detectors identify adversarial inputs by estimating their distance from the manifold of normal examples and rejecting inputs beyond a threshold.MagNet’s detector models only normal examples; one implementation uses autoencoder reconstruction error as the distance estimate.
  • MagNet’s defense: A reformer uses an autoencoder trained on normal examples to move adversarial inputs near the manifold toward similar normal examples.This addresses adversarial examples close to the manifold boundary, where classifier behavior may generalize poorly.
  • Threat model and diversity: MagNet uses multiple diverse defenses and randomly selects one at runtime to defend against its graybox threat model.The paper motivates this design by analogy to cryptographic key diversity.

2.1 Deep learning systems in adversarial environments

Deep learning systems increasingly operate in security-critical domains, but carefully crafted inputs can fool them with perturbations that humans do not recognize. Protecting such classifiers from adversarial examples is therefore a practical concern.

  • Security-critical applications: Deep learning systems support autonomous control, financial systems, medical treatments, information security, and human-computer interaction.These applications motivate understanding neural networks from a security perspective.
  • Adversarial vulnerability: Adversarial examples perturb normal inputs by a small amount that preserves human recognition but causes learning systems to misclassify.The paper identifies protection against these inputs as a real concern.

2.2 Distance metrics

The paper evaluates visually subtle adversarial perturbations using three norms that capture different aspects of image difference. Because no single metric is agreed to be best, the defense is tested on all three.

  • Metrics: L0, L2, and L∞ approximate human-perceived visual difference between adversarial examples and their normal counterparts.They are special cases of the Lp norm.
  • Metrics: L0 counts changed pixels, L2 measures Euclidean image distance, and L∞ measures the maximum per-pixel difference.Each metric emphasizes a different aspect of visual significance.
  • Evaluation choice: The defense is evaluated on L0, L2, and L∞ because there is no consensus about which metric best represents visual difference.

2.3 Existing attacks

The paper considers several gradient-based and optimization-based attacks against neural-network classifiers. These attacks seek small perturbations that cross classifier decision boundaries or increase classification loss while respecting norm or image constraints.

  • Attack landscape: The evaluation covers four popular attacks, focusing on attacks that target neural-network classifiers.
  • Gradient-based attacks: FGSM makes one gradient-sign update within an L∞ neighborhood, with larger ϵ usually increasing attack success.The update maximizes the cost of assigning the ground-truth label while keeping the perturbation small.
  • Gradient-based attacks: The iterative method repeats smaller FGSM updates and clips each result to remain within the ϵ neighborhood of the original image.It applies to both L∞ and L2 metrics and improves FGSM’s success rate.
  • Boundary-based attacks: DeepFool iteratively linearizes the classifier and updates an image toward the nearest decision boundary until the boundary is crossed.
  • Optimization-based attacks: Carlini’s attack optimizes for a small perturbation that fools the classifier under box constraints, with c balancing perturbation size and attack objective.The paper uses the untargeted L2 version as an example and reports defense effectiveness across confidence levels.

2.4 Existing defense

Existing defenses either depend on attack-specific adversarial examples, modify the target classifier, or rely on detectors trained with adversarial examples. MagNet instead adds an independent defense that learns the manifold of normal data and uses reforming to address small-perturbation examples.

  • MagNet’s position: MagNet is orthogonal to classifier-training defenses because it adds a separate framework without modifying the target classifier.Its design and training are independent of the target classifier, making it faster and more flexible.
  • Limitations of existing defenses: Defensive distillation was shown not to significantly increase neural-network robustness and requires modifying and retraining the target classifier.This adds engineering complexity to the defense.
  • Threat models: The authors argue that whitebox defense is intrinsically difficult and study graybox defense using randomization to hinder attack generation.The graybox setting gives the attacker broad knowledge of the defense but not its parameters.
  • MagNet’s detector: Unlike prior detectors trained on normal and adversarial examples, MagNet learns the manifold of normal data without adversarial training examples.Its decisions use the relationship between a test example and the learned normal-data manifold.
  • MagNet’s reformer: MagNet’s reformer pushes hard-to-detect, small-perturbation adversarial examples toward the normal-data manifold.The framework describes this reconstruction as helping classify such examples correctly.

3 PROBLEM DEFINITION

The paper formalizes normal and adversarial examples, defenses, evaluation criteria, and attacker knowledge. It distinguishes errors on naturally occurring data from adversarial examples and defines blackbox, whitebox, and graybox settings.

  • Basic sets: The sample space S contains all examples, while Ct contains mutually exclusive classes for classification task t.The framework uses these sets to define normal examples, classifiers, and defenses.
  • Normal examples: The normal-example set Nt consists of examples that occur naturally for task t, approximated in practice using natural datasets.The text gives CIFAR and MNIST as examples of such datasets for image recognition.
  • Adversarial examples: An adversarial example is misclassified by classifier ft relative to ground truth gt and lies outside the normal-example set Nt.A classification error alone is insufficient because naturally occurring examples can also be misclassified.
  • Defenses: A defense d_ft extends classifier ft and may avoid reading or modifying it, read its data without modification, or modify its parameters.These three usage patterns distinguish increasingly invasive forms of defense.
  • Evaluation: Defense evaluation requires correctness on both normal and adversarial examples, with rejection counted as correct for an adversarial example.Normal examples require agreement with ground truth, whereas adversarial examples may be rejected or correctly classified.
  • Test-time workflow: In the test workflow, detectors reject examples judged adversarial; otherwise, MagNet reforms the example before sending it to the target classifier.An example is rejected if any detector flags it.
  • Attack models: Blackbox attackers lack defense parameters, whitebox attackers know them, and graybox attackers know the defense except for its parameters.The paper notes that repeated neural-network training can produce different parameters through random initialization.
  • Defense assumptions: The defense is assumed not to know how the attacker generates adversarial examples.This assumption separates the defense design from any particular attack-generation process.

4 DESIGN

MagNet combines detectors that approximate the manifold of normal examples with a reformer that moves inputs toward that manifold before classification. It avoids adversarial-example-specific training, does not modify the target classifier, and uses diversity to address graybox attacks.

  • Framework: MagNet uses a detector to reject inputs far from the manifold and a reformer to replace inputs with nearby manifold approximations before classification.The detector measures reconstruction error, while the reformer seeks a close example on or near the normal-example manifold.
  • Detector: The detector models only normal examples and uses autoencoder reconstruction error as an attack-independent estimate of distance from the normal-example manifold.A threshold selected on normal validation examples controls the false positive rate.
  • Detector: Reconstruction-error detection requires a threshold that balances detecting slightly perturbed adversarial examples against falsely rejecting normal examples.The threshold is chosen as high as possible while keeping validation false positives below a specified limit.
  • Detector: MagNet can compare target-classifier outputs before and after autoencoder transformation because adversarial inputs may produce larger output changes than normal inputs even with small reconstruction error.The method uses the target classifier’s last-layer output, while the protected classifier remains unmodified.
  • Reformer: The reformer should preserve normal classifications while changing adversarial examples toward normal examples; naive Gaussian-noise reforming fails to exploit the normal-data distribution.Noise changes normal and adversarial inputs randomly rather than selectively moving adversarial inputs toward the manifold.
  • Diversity and attack settings: MagNet randomly selects among diverse autoencoders at runtime so attackers facing graybox access must construct examples that work across the collection.The paper provides empirical construction and effectiveness evidence rather than a rigorous theoretical analysis of transfer-attack probabilities.

5 IMPLEMENTATION AND EVALUATION

MagNet was evaluated on MNIST and CIFAR-10 against several standard attacks in blackbox and graybox settings. It maintained high adversarial-example accuracy while causing only small reductions on normal examples, although diversification remains improvable.

  • Setup: The evaluation used MNIST and CIFAR-10, testing fast gradient sign, iterative gradient sign, DeepFool, and Carlini attacks.The experiments also examined detector and reformer effects and diversity against graybox attacks.
  • MNIST: 99.4% to 99.1%: MagNet reduced MNIST test accuracy on normal examples by only 0.3 percentage points.The paper characterizes this reduction as negligible.
  • MNIST: Above 99%: MagNet classified MNIST adversarial examples correctly for every considered attack except Carlini’s L0 attack, which reached 92.0%.MagNet was not trained on any of the evaluated attacks.
  • CIFAR-10: 90.6% to 86.8%: MagNet reduced CIFAR-10 test accuracy on normal examples by 3.8 percentage points.The paper describes this reduction as small.
  • CIFAR-10: Above 75% on all attacks and above 90% on more than half: MagNet generalized across ten CIFAR-10 attacks and attack parameters.The paper attributes lower CIFAR-10 accuracy than MNIST partly to the weaker target classifier leaving less space for MagNet to act.
  • Carlini attack: MagNet’s detector and reformer compensate across Carlini confidence levels: the reformer helps nearer-manifold examples, while the detector rejects farther ones.On MNIST, accuracy exceeded 99% at all tested confidence levels; on CIFAR-10, it stayed above 80%.
  • Generalization and limitations: MagNet was not trained on Carlini’s attack or other attacks, although the authors only conjecture that the results generalize further.The authors identify substantial room to improve diversification through architectures, training parameters, ensemble size, and explicit diversity objectives.
  • Graybox attacks: Above 80%: randomized selection among eight autoencoders preserved CIFAR-10 accuracy in the graybox setting, whereas matching the attacked autoencoder reduced accuracy mostly to 0.When the tested and attacked autoencoders differed, accuracy exceeded 90%.

6 DISCUSSION

The discussion identifies assumptions underlying MagNet’s effectiveness and treats the results as empirical evidence rather than proof that those assumptions always hold.

  • Assumptions: MagNet assumes detector functions can measure distance from the manifold of normal examples.It also assumes reformers can produce perceptually close outputs that are nearer the manifold than their inputs.
  • Scope of evidence: Empirical success against state-of-the-art attacks supports these assumptions, but the authors cannot exclude that the attacks were insufficiently powerful.They call for stronger attacks or stronger detectors and reformers.

7 CONCLUSION

MagNet defends neural networks through detectors for large perturbations and manifold-based reformers for small perturbations. The framework aims for attack-independent defense and reports empirical performance across many settings.

  • MagNet is presented as a framework for defending neural networks against adversarial perturbations and reports good empirical performance across 11 settings.
  • MagNet detects adversarial examples with large perturbations using detector networks and pushes small-perturbation examples toward the normal-example manifold.These two methods jointly enhance classification accuracy.
  • MagNet’s autoencoder detectors learn without adversarial examples or knowledge of their generation process, supporting attack-independent defense.The paper presents attack independence as a route toward greater transferability across adversarial-generation processes.
Loading 1705.09064v2…