PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images

Anh Nguyen, Jason Yosinski, Jeff Clune

arXiv:1412.1897v4cs.CVcs.AIcs.NE

TL;DR

The paper asks what differences remain between human and DNN vision despite near-human visual classification. It generates images with evolutionary algorithms or gradient ascent and finds that DNNs classify many unrecognizable images as recognizable objects with over 99% confidence, raising questions about their generalization.

  • Problem

    Despite near-human visual classification, the remaining differences between human and DNN vision are not fully understood.

  • Method

    The authors generate high-scoring images for ImageNet- and MNIST-trained convolutional networks using evolutionary algorithms or gradient ascent.

  • Results

    Over 99% confidence is assigned to images completely unrecognizable to humans, including static labeled as a recognizable object.

  • Takeaways & Limitations

    The findings reveal differences between human and DNN vision and raise questions about DNN generalization across images unlike those used in training and testing.

  • Takeaways & Limitations

    Testing whether generative models could resist fooling images remains unavailable because current models do not scale to ImageNet’s high dimensionality.

Abstract

from arXiv · show

Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Given that DNNs are now able to classify objects in images with near-human-level performance, questions naturally arise as to what differences remain between computer and human vision. A recent study revealed that changing an image (e.g. of a lion) in a way imperceptible to humans can cause a DNN to label the image as something else entirely (e.g. mislabeling a lion a library). Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion). Specifically, we take convolutional neural networks trained to perform well on either the ImageNet or MNIST datasets and then find images with evolutionary algorithms or gradient ascent that DNNs label with high confidence as belonging to each dataset class. It is possible to produce images totally unrecognizable to human eyes that DNNs believe with near certainty are familiar objects, which we call "fooling images" (more generally, fooling examples). Our results shed light on interesting differences between human vision and current DNNs, and raise questions about the generality of DNN computer vision.

1. Introduction

The paper shows that state-of-the-art DNNs can assign very high-confidence object predictions to images that humans cannot recognize, revealing a difference between human and computer vision. It generates these fooling images with evolutionary algorithms or gradient ascent on ImageNet- and MNIST-trained convolutional networks, raising questions about generalization beyond training and test images.

  • Motivation: DNNs achieve state-of-the-art, sometimes human-competitive performance on pattern-recognition tasks, especially visual classification.This motivates examining what differences remain between human and computer vision.
  • Core finding: ≥99.6% certainty is reported for evolved ImageNet images that humans cannot recognize but DNNs classify as familiar objects.The images are produced using either direct or indirect encoding.
  • Core finding: Over 99% confidence can be assigned by state-of-the-art DNNs to images that are completely unrecognizable to humans.The paper calls these high-confidence misclassifications “fooling images.”
  • Methods: Evolutionary algorithms or gradient ascent generate images receiving high prediction scores from convolutional networks trained on ImageNet or MNIST.The methods optimize images to elicit high-confidence predictions for dataset classes.
  • Robustness: Retraining MNIST DNNs with fooling images labeled as such does not easily prevent them from being fooled.The retrained networks learn to classify the negative examples as such, but the passage indicates prevention remains difficult.
  • Implications: The findings expose differences between human vision and DNN-based computer vision and raise questions about performance on images unlike those used for training and testing.The concern concerns generality across different types of images.

2. Methods

The study tests false-positive recognition using pretrained ImageNet and MNIST DNNs, while generating candidate images through evolutionary optimization. The methods compare direct pixel encodings with CPPN-based representations and use DNN class-prediction scores as fitness.

  • DNN models: The experiments use a Caffe-provided AlexNet trained on 1.3 million ImageNet images, with a 42.6% top-1 error rate versus 40.7% reported for the original model.This model is called the ImageNet DNN.
  • DNN models: The study also evaluates a Caffe-provided LeNet trained on MNIST, obtaining a 0.94% error rate versus 0.8% for LeNet-5.Its ReLU activations differ slightly from the original sigmoid-based architecture, and it is called the MNIST DNN.
  • Image generation: Evolutionary algorithms generate images by selecting and perturbing image populations, with fitness defined as the DNN’s highest prediction value for any class.The evolutionary cycle alternates selection with mutation and/or crossover.
  • Image generation: MAP-Elites evolves multiple class-specific champions simultaneously, replacing an archive entry whenever a mutated image achieves a higher prediction score for that class.This contrasts with traditional evolutionary algorithms that optimize one objective or a small set of objectives.
  • Image representations: The direct encoding represents MNIST images with one grayscale integer per 28 × 28 pixel and ImageNet images with three HSV integers per 256 × 256 pixel, initialized as uniform random noise in [0, 255].Pixel values are independently mutated, with the mutation rate starting at 0.1.
  • Image representations: CPPN encodings map each pixel’s (x, y) position to grayscale or HSV values, and CPPN-evolved images can be recognized by both DNNs and humans.The human-recognizable examples provide an existence proof for CPPN-encoded evolutionary image generation.

3. Results

Evolutionary algorithms produced unrecognizable images that MNIST and ImageNet DNNs classified with very high confidence, especially when using regular CPPN encodings. The results indicate that these systems exploit class-discriminative features, while fooling resistance varies across models and training procedures.

  • MNIST results: 99.99% confidence was repeatedly reached for unrecognizable MNIST images of every digit within fewer than 50 evolutionary generations.This occurred with directly encoded images across multiple independent runs.
  • MNIST results: 99.99% median confidence was reached by 200 generations with CPPN encodings, despite the generated MNIST images remaining unrecognizable.CPPN images contained more strokes and regularities than direct-encoding white-noise images, but still fooled the classifier after only a few generations.
  • MNIST results: Evolution repeatedly generated digit-specific patterns, including vertical bars for 1s and lower horizontal bars for 2s, indicating exploitation of learned discriminative features.Qualitatively similar features appeared across 50 additional runs.
  • ImageNet results: 88.11% median confidence was achieved after 5000 generations with CPPN encoding on ImageNet, significantly exceeding 21.59% for direct encoding.High-confidence images appeared in most categories, although many remained unrecognizable without class labels.
  • ImageNet results: Generated ImageNet images often contained class-specific features, because evolution needed only discriminative cues rather than all typical object features.Examples included water and orange coloring for starfish, red stitching for baseballs, and button grids for remote controls.
  • Model robustness: 99.99% confidence images transferred across two DNNs, but retraining produced uneven resistance: ImageNet median confidence fell from 88.1% to 11.7%, whereas MNIST remained vulnerable at 99.99%.The retrained ImageNet model labeled 45 of 70 recognizable CPPN images as CPPN images, suggesting it learned generic CPPN features.

4. Discussion

The discussion explains that evolution produced diverse, high-confidence images that humans could not recognize, likely because discriminative models allocate broad classification regions without modeling image likelihood. It also highlights recognizable evolved images, feature visualization, creative applications, and security risks from false positives.

  • Discussion: Evolution produced tremendous diversity rather than similar or recognizable images across classes, yielding high-confidence images that humans found unrecognizable.This outcome contrasted with expectations based on imperceptible adversarial changes and predictions that evolution would produce recognizable class images.
  • Discussion: Discriminative models learn p(y|X) and may assign large high-confidence classification regions extending far beyond the training examples.Synthetic images deep within these regions can receive confident labels despite being unlike recognizable class members.
  • Discussion: Generative models could make fooling images easier to detect by using low marginal probability p(X), but current models do not scale well to ImageNet-like dimensionality.A joint-density model represents p(y, X), enabling confidence discounts when p(X) is low.
  • Discussion: Some generated images become recognizable as target-class members once the class label is known, and examples were accepted and displayed in a selective art competition.Reported examples include a chain-link fence, computer keyboard, digital clock, bagel, strawberry, ski mask, spotlight, and monarch butterfly.
  • Discussion: The CPPN evolutionary algorithm can visualize DNN-learned features, while variation across runs reveals feature diversity and may support transfer analysis.The paper also suggests that DNN false positives could compromise deployed systems such as face or voice recognition, including through human-unrecognizable substitutes.

5. Conclusion

Discriminative DNNs can classify unrecognizable images with near-certainty as recognizable classes, revealing persistent differences between DNN and human recognition.

  • Conclusion: DNNs classify many unrecognizable images with near-certainty as members of recognizable classes.The study demonstrates that discriminative models are easily fooled by such images.
  • Conclusion: Two evolutionary-algorithm encodings generate qualitatively different fooling images, while gradient ascent generates a third type.These methods produce distinct forms of unrecognizable images that DNNs interpret as recognizable objects.
  • Conclusion: DNNs’ near-perfect recognition of these images highlights remaining differences between DNN and human visual recognition.The fooling images expose a gap between what DNNs recognize and what humans find recognizable.

Supplementary Material for Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images · A. Images that fool one DNN generalize to fool

The supplementary section examines whether fooling images evolved for one DNN also fool another, probing whether independently initialized networks learn the same class features. The test uses identical architectures and training while varying only randomized initialization.

  • Supplementary Material for Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images: The supplied supplementary material is identified as arXiv:1412.1897v4 and dated 2 Apr 2015.
  • A. Images that fool one DNN generalize to fool: The paper asks whether different DNNs learn the same features for each class or instead develop distinct discriminative features.The proposed test uses transferability of images that fool one DNN to shed light on this question.
  • A. Images that fool one DNN generalize to fool: Researchers evolved CPPN-encoded images with one network, DNNA, and then input those images to another network, DNNB.
  • A. Images that fool one DNN generalize to fool: DNNA and DNNB had identical architectures and training procedures.
  • A. Images that fool one DNN generalize to fool: The two networks differed only in their randomized initializations.
  • A. Images that fool one DNN generalize to fool: The experiment therefore tests whether images evolved to fool DNNA also fool DNNB.

A.1. Generalization across DNNs with the same ar- · A.2. Generalization across DNNs that have different · B. Does using an ensemble of networks instead

Fooling images generalize across independently trained DNNs with identical architectures and, to a lesser extent, across different architectures. The paper also tests whether an ensemble of three distinct architectures prevents fooling images.

  • A.1. Generalization across DNNs with the same ar-: 300 images were produced with each MNIST DNN, while 1000 images were produced with each ImageNet DNN.MNISTA and MNISTB, and ImageNetA and ImageNetB, differed only in random initialization while sharing architecture.
  • A.1. Generalization across DNNs with the same ar-: Many images evolved for one DNN received the same top-1 label from the independently trained counterpart.This held when testing images evolved for DNNA on DNNB and vice versa.
  • A.1. Generalization across DNNs with the same ar-: Many cross-tested images received ≥99.99% confidence from both independently trained DNNs.These results indicate that evolution produces class-discriminative patterns shared by multiple independently trained networks.
  • A.1. Generalization across DNNs with the same ar-: Digit classes 9, 6, and 2 fooled both MNIST networks the most and produced distinctive patterns.Evolution converged on similar designs across independent runs, including images that fooled both networks at ≥99% confidence.
  • A.2. Generalization across DNNs that have different: The different-architecture test used AlexNet and GoogLeNet, both trained on ILSVRC 2012; GoogLeNet had a 31.3% top-1 error rate.The experiment asked whether images fooling one architecture also fool another.
  • A.2. Generalization across DNNs that have different: 20.7% of images evolved for GoogLeNet received the same top-1 label from AlexNet, compared with 17.3% in the reverse direction.Thus, many fooling examples generalized across different DNN architectures rather than fitting precisely to one network.
  • B. Does using an ensemble of networks instead: The ensemble experiment tested whether fooling images remain producible when three different architectures—CaffeNet, AlexNet, and GoogLeNet—must be fooled together.CaffeNet performs similarly to AlexNet but has a slightly different architecture, and the final confidence score was calculated as the mean of the networks’ scores.

C. Training networks to recognize fooling im-

The section tests whether adding previously evolved fooling images as a negative, new class can reduce DNN susceptibility. The procedure trains a second network on an enlarged dataset and can iteratively add further fooling images to the same category.

  • Motivation: The authors test whether retraining with previously fooling images labeled outside the original classes can address DNN vulnerability.The proposed remedy is to recognize such images as members of a new class rather than any original class.
  • Training procedure: The procedure trains DNN1, evolves high-confidence CPPN images for its n classes, adds them as class n + 1, and trains DNN2 on the enlarged dataset.The experiment is conducted on both MNIST and ImageNet DNNs.
  • Training procedure: Optionally, images that fool DNN2 are also placed in the n + 1 category, making an n + 2 category unnecessary.Any image that fools a DNN is treated as a fooling image and assigned to the same added category.
  • Image sampling: Each iteration adds m images to class n + 1, sampled from first and last generations across multiple evolution runs.Each MNIST run produces 20 images and each ImageNet run produces 2000 images, with half from the first generation and half from the last.

C.1. Training MNIST DNNs with fooling images

Retraining MNIST DNNs with iteratively added fooling images did not eliminate high-confidence unrecognizable images. Even when the fooling-image class comprised about 25% of DNN15’s training set, retraining provided no benefit.

  • Dataset construction: 6000 training and 1000 validation images were added in the first iteration, followed by 1000 training and 100 validation images per additional iteration.The additions balanced the n+1 class with the other MNIST classes.
  • Training setup: MNIST DNNs used 28 × 28 images, SGD with 0.9 momentum, batch size 64, weight decay 0.0005, and an initially 0.01 learning rate.The learning rate decreased each iteration under an inverse learning rate policy with power = 0.75 and gamma = 0.0001; DNN2−DNN15 matched DNN1’s 0.94% error rate.

C.2. Training ImageNet DNNs with fooling images

ImageNet training was extended with a heavily overrepresented fooling-image class, which modestly improved DNN2’s overall top-1 error but not its performance on the original validation set. This training substantially reduced evolution’s ability to generate high-confidence fooling images.

  • Training setup: 9000 training and 2000 validation fooling images were added as a 1001st ImageNet class, roughly seven times the approximately 1300 images per original class.The imbalance was intended to emphasize fooling images; without it, negative examples did not prevent fooling.
  • ImageNet performance: 41.0% top-1 error was achieved by DNN2 on ImageNet, compared with 42.6% for DNN1.The authors hypothesize that the added fooling-image class was easier to classify because it represented a different image style.
  • ImageNet performance: 42.6% top-1 error was obtained by DNN2 on the original ILSVRC 2012 validation set.This matched DNN1’s overall error rate and supports the hypothesis that the added class made classification easier.
  • Fooling-image generation: 88.1% to 11.7% was the decrease in median confidence scores for evolved images from DNN1 to DNN2 (p < 0.0001).For ImageNet models, evolution was less able to produce high-confidence images for DNN2 than for DNN1.

D. Evolving regular images to match MNIST · E. Gradient ascent with regularization · F. Confidence scores of real ImageNet images

CPPN evolution produced regular-looking MNIST fooling images that LeNet classified as digits with 99.99% confidence, while gradient-ascent regularization traded confidence for clearer visualizations. On natural ImageNet validation images, median confidence was 60.3% across 50,000 examples.

  • D. Evolving regular images to match MNIST: 99.99% confidence was reached after only a few generations for unrecognizable CPPN-evolved MNIST images, with median confidence still 99.99% by 200 generations.The CPPN-encoded evolutionary algorithm produced 500 images across 10 images and 50 runs.
  • D. Evolving regular images to match MNIST: Evolution repeatedly produced class-specific patterns, including vertical bars for 1s, lower horizontal bars for 2s, and similar patterns for 3s and 8s.The similarity between 3 and 8 patterns may reflect shared features learned from the training set.
  • E. Gradient ascent with regularization: Direct gradient ascent quickly produced unrecognizable ImageNet images with confidence over 99.99%, while regularization generated more recognizable images.The study tested three regularization types.
  • E. Gradient ascent with regularization: L2 regularization used weight decay by multiplying the mean-subtracted image by 1 −γ each step, with γ = 0.01.This procedure is shown in Fig. S5.
  • E. Gradient ascent with regularization: Additional regularization combined weight decay with Gaussian blurring of radius 0.3 and pseudo-L1 regularization targeting low-norm RGB pixels.For this configuration, weight decay used γ = 0.001.
  • E. Gradient ascent with regularization: Stronger regularization produced clearer visualizations in some cases, but many classes never achieved 99% confidence.This configuration used a lower learning rate, the same weight decay, and more aggressive blurring.
  • F. Confidence scores of real ImageNet images: 60.3% was the median confidence across 50,000 ImageNet validation images evaluated by the ImageNet DNN.The evaluation compared confidence scores for natural images with those assigned to fooling images.

G. Can the fooling images be considered art? · Supplementary References

CPPN fooling images were accepted and awarded in a selective student art exhibition, then displayed alongside human-made art at the University of Wyoming Art Museum. Museum visitors viewed a montage of the images.

  • G. Can the fooling images be considered art?: 35.5% of submissions were accepted by the University of Wyoming 40th Annual Juried Student Exhibition, including the submitted CPPN fooling images.The images were entered to test whether they could be considered art.
  • G. Can the fooling images be considered art?: 21.3% of submissions received an award, including the accepted fooling images.The images were not only accepted but also among the awarded submissions.
  • G. Can the fooling images be considered art?: The submitted fooling image was displayed at the University of Wyoming Art Museum.The exhibition display is documented in supplementary figures S2 and S3.
  • G. Can the fooling images be considered art?: A selection of fooling images was accepted as art in a selective art competition.The competition accepted only 35.5% of submissions.
  • G. Can the fooling images be considered art?: The accepted fooling images were displayed alongside human-made art at a museum.This presentation followed their acceptance in the selective art competition.
  • G. Can the fooling images be considered art?: Museum visitors viewed a montage of CPPN-encoded fooling images.The montage was shown at the University of Wyoming Art Museum.
Loading 1412.1897v4…