Adversarial examples in the physical world

Alexey Kurakin; Ian Goodfellow; Samy Bengio

Source-linked AI summary

Adversarial examples in the physical world

Alexey Kurakin, Ian Goodfellow, Samy Bengio

arXiv:1607.02533v4Jul 8, 2016cs.CVcs.CRcs.LGstat.ML

TL;DR

Prior work largely assumed attackers could feed inputs directly to classifiers, leaving physical-world camera-based attacks insufficiently tested. This paper photographs adversarial images and finds that a significant fraction remain misclassified by an Inception classifier through a cellphone camera.

Problem
Prior work largely assumed direct access to the classifier input, leaving the survival of adversarial examples in physical camera-based systems insufficiently tested.
Method
The paper generates adversarial images for a pretrained Inception classifier, photographs printed images with a cellphone camera, and measures classification accuracy after the photo transformation.
Results
A significant fraction of adversarial images crafted for the original network remain misclassified when perceived through the camera.
Takeaways & Limitations
These findings demonstrate the possibility of adversarial examples against machine-learning systems operating in the physical world.
Takeaways & Limitations
The experiments assume the attacker has full knowledge of the model architecture and parameter values, with only limited experiments on physical-example transferability.

Abstract

from arXiv · show

Most existing machine learning classifiers are highly vulnerable to adversarial examples. An adversarial example is a sample of input data which has been modified very slightly in a way that is intended to cause a machine learning classifier to misclassify it. In many cases, these modifications can be so subtle that a human observer does not even notice the modification at all, yet the classifier still makes a mistake. Adversarial examples pose security concerns because they could be used to perform an attack on machine learning systems, even if the adversary has no access to the underlying model. Up to now, all previous work have assumed a threat model in which the adversary can feed data directly into the machine learning classifier. This is not always the case for systems operating in the physical world, for example those which are using signals from cameras and other sensors as an input. This paper shows that even in such physical world scenarios, machine learning systems are vulnerable to adversarial examples. We demonstrate this by feeding adversarial images obtained from cell-phone camera to an ImageNet Inception classifier and measuring the classification accuracy of the system. We find that a large fraction of adversarial examples are classified incorrectly even when perceived through the camera.

1 INTRODUCTION

The introduction establishes that small, often imperceptible input modifications can cause machine-learning misclassification and pose security threats, including without model access. This paper tests whether such adversarial examples survive physical-world capture through a cellphone camera and finds that unmodified attacks transfer to the camera-classifier system.

Motivation: Small input modifications can make neural networks and other machine-learning models misclassify otherwise correctly classified examples.Adversarial examples can be perceptually indistinguishable from clean examples and are misclassified more often than noise-perturbed examples.
Motivation: Transferability enables adversarial examples designed for one model to fool another, allowing attacks without access to the underlying model.The introduction identifies this property as a potential security threat for practical machine-learning applications.
Problem: Prior work assumed attackers could supply inputs directly to models, leaving it unknown whether physical-world examples remained misclassified through cameras.This gap matters for systems such as robots, surveillance systems, and mobile classification applications that perceive the world through sensors.
Experiment: The authors generated adversarial examples for a pre-trained ImageNet Inception classifier, presented them through a cellphone camera, and measured classification accuracy.This experiment models a physical-world system that captures data with a camera before performing image classification.
Result: The attack required no camera-specific modification: examples crafted for Inception successfully transferred to the combined camera-and-Inception system.The authors characterize this result as a lower bound on success achievable by attacks that explicitly model the camera.
Limitation: The experiments assumed attackers knew the model architecture and parameters, although transferability suggests the findings extend to attackers lacking the model description.The assumption enabled use of a single Inception v3 model throughout the experiments.

2 METHODS OF GENERATING ADVERSARIAL IMAGES

The paper compares fast, iterative, and least-likely-class methods for generating adversarial images under an L∞ constraint. Experiments on ImageNet show that these methods can substantially reduce classifier accuracy while preserving small perturbations.

Method evaluation: None of the described generation methods guarantees misclassification, so the paper evaluates their actual classification accuracy experimentally.The authors call the generated samples adversarial images despite this limitation.
Fast method: The fast method linearizes the cost function and finds an L∞-constrained perturbation in closed form using one back-propagation call.It is faster than the other considered methods because it is non-iterative.
Iterative methods: The basic iterative method repeatedly applies the fast update with α = 1 and clips intermediate images to remain within an ϵ-neighbourhood of the original.The iteration count is min(ϵ + 4, 1.25ϵ), chosen heuristically to balance reaching the L∞ boundary and computational cost.
Targeted method: The least-likely-class method iteratively increases the probability of the model’s least-likely class, producing mistakes such as classifying a dog as an airplane.It uses the same step size and iteration count as the basic iterative method.
Experimental comparison: The fast method halves top-1 accuracy and reduces top-5 accuracy by about 40% even at the smallest ϵ values.Its accuracy remains approximately level until ϵ = 32, then decreases to almost 0 by ϵ = 128.
Experimental comparison: Iterative methods preserve finer image perturbations while confusing the classifier more effectively; the least-likely-class method destroys correct classification for most images at relatively small ϵ.The authors restrict later experiments to ϵ ≤16 because these perturbations appear as small noise while still causing substantial misclassification.

3 PHOTOS OF ADVERSARIAL EXAMPLES

The experiments test whether adversarial examples survive printing, cellphone photography, and automatic cropping under realistic nuisance variability. Some adversarial examples remain misclassified after this photo transformation, demonstrating physical adversarial examples.

Experimental setup: The photo transformation printed clean and adversarial images, photographed the pages with a Nexus 5x cellphone camera, and automatically cropped and warped the examples.QR codes enabled automatic cropping; failed detections caused discarded photos.
Evaluation protocol: Experiments evaluated clean and adversarial-image accuracy before and after photography, together with adversarial destruction rate.The average-case experiments randomly selected 102 images, whereas prefiltered experiments selected 102 images meeting clean-correct and adversarial-incorrect criteria with p(ypredicted|X) ≥0.8.
Results: “Fast” adversarial images are more robust to photo transformation than iterative-method images, whose subtler perturbations are more likely to be destroyed.The paper attributes this difference to iterative methods’ reliance on fine perturbations and pixel co-adaptations.
Results: Some adversarial examples remain misclassified after photo transformation, demonstrating the possibility of physical adversarial examples.With the fast method at ϵ = 16, about 2/3 of images were expected to remain top-1 misclassified and about 1/3 top-5 misclassified.

4 ARTIFICIAL IMAGE TRANSFORMATIONS

Artificial transformations were evaluated on adversarial examples to measure their destruction rates. Robustness depended on the attack method and transformation, with blur, noise, and JPEG encoding more disruptive than brightness or contrast changes.

Transformations and setup: Experiments measured adversarial destruction rates across brightness and contrast changes, Gaussian blur, Gaussian noise, and JPEG encoding.The experiments used the same randomly selected subset of 1,000 validation images for all transformation–method pairs.
Robustness by method: Fast-method adversarial examples were most robust to transformations, whereas iterative least-likely-class examples were least robust.This pattern coincided with the results from the photo-transformation experiment.
Destruction metrics: Top-5 destruction rates were typically higher than top-1 destruction rates because destroying top-1 requires the correct label to become the top prediction.For top-5 examples, the correct class only needs to enter the top-five predictions.
Brightness and contrast: Brightness and contrast changes produced destruction rates below 5% for fast and basic iterative examples and below 20% for iterative least-likely-class examples.These changes therefore had little effect on adversarial examples.
Blur, noise, and JPEG encoding: Blur, noise, and JPEG encoding had higher destruction rates than brightness and contrast changes, reaching 80%–90% for iterative methods without destroying all adversarial examples.None of these transformations destroyed 100% of adversarial examples, matching the photo-transformation experiment.

5 CONCLUSION

The paper demonstrates that machine learning systems operating in the physical world can be vulnerable to adversarial examples. Using cell-phone camera images as input to an Inception v3 classifier, it shows that a significant fraction of adversarial images are misclassified through the camera.

The study explored adversarial examples against machine learning systems operating in the physical world.
The experiments used cell-phone camera images as input to an Inception v3 image classification neural network.
A significant fraction of adversarial images crafted using the original network were misclassified when fed to the classifier through the camera.

Appendix · Appendix contains following figures:

The appendix presents figures illustrating adversarial images, perturbation magnitudes, adversarial methods, and destruction rates under image transformations.

Appendix: The appendix contains Figure 4 examples of adversarial images produced by different adversarial methods.It also contains Figure 5 examples for various ϵ values and Figure 6 plots of destruction rates for image transformations.
Appendix contains following figures:: Figure 4 compares adversarial images produced by different adversarial methods.At ϵ = 32, iterative methods produce finer perturbations than the fast method and do not always select the boundary of the ϵ-neighbourhood.
Appendix contains following figures:: At ϵ = 32, iterative methods produce finer perturbations than the fast method.Iterative methods also do not always select a point on the border of the ϵ-neighbourhood as an adversarial image.
Appendix contains following figures:: Figure 5 shows fast-method adversarial perturbations across various ϵ values for washer and hamster images.Clean images are classified correctly, whereas adversarial images are misclassified for every considered ϵ.
Appendix contains following figures:: Clean washer and hamster images are classified correctly, while their adversarial versions are misclassified for all considered ϵ.The figure compares images generated with the fast method at different perturbation sizes.
Appendix contains following figures:: Figure 6 plots adversarial destruction rates for various adversarial methods and image transformations.All experiments were conducted with ϵ = 16.