PaperlayerView exact revision on arXiv

Source-linked AI summary

Humans can decipher adversarial images

Zhenglong Zhou, Chaz Firestone

arXiv:1809.04120v3cs.CVcs.CYcs.LG

TL;DR

CNNs’ adversarial errors raise questions about both their practical use and their resemblance to human vision, while little empirical work had tested how people classify such images. Across eight experiments on five adversarial imagesets, the authors asked people to predict machine-generated labels and found that human subjects anticipated the machines’ classifications.

  • Problem

    Adversarial images raise practical concerns and theoretical doubts about CNNs as models of human vision, but little work had tested whether people can predict machine labels for them.

    Source: Introduction · pp. 1, 3

  • Method

    Across eight experiments, humans performed a machine-theory-of-mind task by choosing the label they thought a machine had assigned to adversarial images.

    Source: Introduction · p. 3

  • Results

    Across eight experiments covering diverse attacks and design variations, human subjects anticipated machines’ classifications of adversarial stimuli.

    Source: Introduction · p. 3

  • Takeaways & Limitations

    The results suggest human intuition can provide information about machine image classifications and imply some overlap in the features humans and machines prioritize.

    Source: General Discussion · p. 10

    The study explored visible-noise attacks, since small rotations or rescaling usually restore perturbed images to their prior classification, limiting their practical relevance in real-world settings.

    Source: General Discussion · p. 10

Abstract

from arXiv · show

How similar is the human mind to the sophisticated machine-learning systems that mirror its performance? Models of object categorization based on convolutional neural networks (CNNs) have achieved human-level benchmarks in assigning known labels to novel images. These advances promise to support transformative technologies such as autonomous vehicles and machine diagnosis; beyond this, they also serve as candidate models for the visual system itself -- not only in their output but perhaps even in their underlying mechanisms and principles. However, unlike human vision, CNNs can be "fooled" by adversarial examples -- carefully crafted images that appear as nonsense patterns to humans but are recognized as familiar objects by machines, or that appear as one object to humans and a different object to machines. This seemingly extreme divergence between human and machine classification challenges the promise of these new advances, both as applied image-recognition systems and also as models of the human mind. Surprisingly, however, little work has empirically investigated human classification of such adversarial stimuli: Does human and machine performance fundamentally diverge? Or could humans decipher such images and predict the machine's preferred labels? Here, we show that human and machine classification of adversarial stimuli are robustly related: In eight experiments on five prominent and diverse adversarial imagesets, human subjects reliably identified the machine's chosen label over relevant foils. This pattern persisted for images with strong antecedent identities, and even for images described as "totally unrecognizable to human eyes". We suggest that human intuition may be a more reliable guide to machine (mis)classification than has typically been imagined, and we explore the consequences of this result for minds and machines alike.

Introduction

CNNs have achieved human-level image-classification benchmarks, raising both practical hopes and questions about whether they model human vision. Because adversarial images seem to expose a human–machine divide, the paper tests whether people can predict machines’ labels for them.

Introduction

  • Adversarial images can make CNNs misclassify, raising practical security concerns and theoretical doubts about whether these systems resemble human visual processing.The paper distinguishes meaningless patterns classified as familiar objects from slightly perturbed images that receive a different label.Source: Introduction · pp. 1, 3
  • Although adversarial images are often assumed to be unrecognizable to people, little work had tested whether humans could predict the machine’s preferred labels.The authors note that some images may make sense once viewers are told the machine’s label.Source: Introduction · p. 3
  • The authors test this question with a machine-theory-of-mind task across eight experiments and five adversarial imagesets not designed for human vision.Participants were told a machine had classified each image and chose the label they thought it generated.Source: Introduction · p. 3
  • Across the eight experiments, human subjects anticipated machine classifications, suggesting human intuition is a more reliable guide to machine misclassification than commonly imagined.The authors discuss implications for cognitive science and artificial intelligence.Source: Introduction · p. 3

Experiment 1: “Fooling” images with foil labels

Experiment 1 tested whether people could identify AlexNet’s label for machine-generated fooling images when offered a random label as a foil. Most subjects and images showed above-chance human–machine agreement.

Experiment 1: “Fooling” images with foil labels

  • The experiment used 48 fooling images generated to confound AlexNet, which classified them as familiar objects such as pinwheel and bagel.The images were produced by an evolutionary algorithm.Source: Experiment 1: “Fooling” images with… · p. 3
  • On each trial, 200 subjects chose which of two displayed labels the machine had generated: its image label or a random label from the other images.Figure 2A depicts the forced-choice setup.Source: Experiment 1: “Fooling” images with… · p. 4, Figure 2 · p. 4
  • For Experiment 1, 94.6%–99.4% of subjects and 82.8%–98.7% of images showed above-chance agreement, using 95% confidence intervals.The figure reports that the outcomes were reliably different from chance at p<.001.Source: Figure 2 · p. 4

Experiment 2: 1st choice vs. 2nd choice

Experiment 2 tested whether human–machine agreement extended beyond rejecting random foils to distinguishing the CNN’s first and second label choices. Subjects and images still showed above-chance agreement.

Experiment 2: 1st choice vs. 2nd choice

  • To test subtler distinctions than those posed by random foils, subjects chose between the CNN’s first- and second-ranked labels for each fooling image.For one image, the alternatives were bagel and pretzel.Source: Experiment 2: 1st choice vs. 2nd choice · p. 5
  • 91% of observers tended to choose the CNN’s first choice over its second, and 71% of images showed human–machine agreement.The authors report lower image agreement than Experiment 1’s 94%, consistent with the second-choice label also being moderately intuitive to people.Source: Experiment 2: 1st choice vs. 2nd choice · p. 5
  • The authors interpret the result as evidence that people can appreciate deeper image features distinguishing the CNN’s primary label from close alternatives.They also infer overlap between human and machine rank-ordering of image labels.Source: Experiment 2: 1st choice vs. 2nd choice · p. 5

Experiment 3a: Many-way classification

Experiment 3a broadened the choice set by showing all 48 labels at once. Human subjects and images still showed above-chance agreement with the machine.

Experiment 3a: Many-way classification

  • Experiment 3 displayed all 48 labels simultaneously and asked subjects to select the best label for each image.The authors framed this as a step toward more unconstrained classification, while noting that asking people to choose among 1,000 labels was impractical.Source: Experiment 3a: Many-way classification · p. 5
  • 88% of subjects and 79% of images showed above-chance human–machine agreement when subjects chose among all 48 labels.The machine’s label appeared among the five most popular human choices for 63% of images, compared with approximately 10% by chance.Source: Experiment 3a: Many-way classification · p. 5
  • The machine’s label was among the top five human choices for 63% of images, versus approximately 10% expected by random chance.This held even when the single most popular human choice was not the CNN’s label.Source: Experiment 3a: Many-way classification · p. 5

Experiment 3b: “What is this?”

Experiment 3b asked people to label adversarial images directly rather than guess a machine’s preferred label. Human judgments still aligned with machine classifications, suggesting the agreement was not specific to the “think like a machine” task.

Experiment 3b: “What is this?”

  • 90% of subjects and 81% of images showed above-chance human-machine agreement when subjects simply labeled the images.Subjects chose from 48 possible labels.Source: Experiment 3b: “What is this?” · p. 6
  • Subjects were asked “What is this?” rather than to guess the machine’s preferred label.They were asked to choose a label for each image from 48 possibilities.Source: Experiment 3b: “What is this?” · p. 6
  • The results suggest that deciphering adversarial images does not depend on the peculiarities of the machine-theory-of-mind task.The authors interpret the agreement as reflecting a more general alignment with machine (mis)classification.Source: Experiment 3b: “What is this?” · p. 6

Experiment 4: Television static images

Experiment 4 tested whether people could match apparently featureless television-static images to labels recognized by a CNN. Most observers and all tested images showed above-chance human-machine agreement.

Experiment 4: Television static images

  • Subjects matched eight television-static images to labels presented with five natural photographs of each category.The images appeared as colorful pixel collections with little if any underlying structure.Source: Experiment 4: Television static images · p. 6
  • 81% of observers and 100% of television-static images showed above-chance human-machine agreement.For 75% of the images, the label most often chosen by subjects was also the machine’s most preferred choice.Source: Experiment 4: Television static images · p. 6
  • The authors note that human response votes could be transformed with softmax to produce high-confidence ratings analogous to CNN classifications.This comparison treats the cohort’s judgments as votes.Source: Experiment 4: Television static images · p. 6
  • The results suggest that people can infer machine classifications even when relevant patterns appear to be featureless collections of colored pixels.The authors suggest that observers may discern subtle features in these images.Source: Experiment 4: Television static images · p. 6

Experiment 5: Perturbed digits

Experiment 5 tested whether people could infer a CNN’s altered label for handwritten digits despite their strong original identities. Human responses aligned with the machine above chance for most subjects and images.

Experiment 5: Perturbed digits

  • Experiment 5 showed 200 subjects ten undistorted MNIST digits and asked them to match each distorted digit to an undistorted digit other than its original identity.The experiment included 100 distorted images.Source: Figure 3 · p. 7
  • The 95% confidence interval was [84.2% 93.8%] for subjects and [64.3% 81.7%] for images with above-chance classification.These Experiment 5 intervals describe the percentages of subjects and images, respectively.Source: Figure 3 · p. 7
  • The study contrasted its task with prior work reporting that people persisted with original digit classifications after perturbation.Here, subjects instead chose which digit they thought the machine perceived.Source: Experiment 5: Perturbed digits · p. 7
  • 89% of subjects and 73% of perturbed digit images showed above-chance human-machine agreement.The authors conclude that people can anticipate machine misclassifications even when images have strong prepotent identities.Source: Experiment 5: Perturbed digits · p. 8

Experiment 6: Natural images and localized perturbations

Experiment 6 tested whether people could decipher natural images altered by a localized adversarial attack. Human responses aligned with the machine’s classifications above chance for most observers and every tested image.

Experiment 6: Natural images and localized perturbations

  • The authors presented localized attacks as overcoming several limitations of the earlier digit perturbations.The earlier perturbations could affect up to 14% of pixels, obstruct salient image regions, and targeted only handwritten digits.Source: Experiment 6: Natural images and… · p. 8
  • LaVAN can alter as little as 2% of pixels near an image border while changing labels such as daisy to jaguar.The attack can target natural images without obstructing their focal object.Source: Experiment 6: Natural images and… · p. 8
  • 87% of subjects and all 22 tested images showed above-chance human-machine agreement for the localized attack.The authors conclude that even more recent and sophisticated adversarial attacks are susceptible to human deciphering.Source: Experiment 6: Natural images and… · p. 8
  • The experiment tested 22 images that caused Inception V3 to misclassify, using target and foil labels paired with ImageNet photographs.The forced-choice design followed Experiment 1.Source: Experiment 6: Natural images and… · p. 8

Experiment 7: 3D objects

Experiment 7 tested whether people could infer machine labels for adversarial 3D objects from multiple viewpoints. Human responses aligned with machine classifications more often than chance.

Experiment 7: 3D objects

  • Unlike 2D attacks that can lose their effect under image manipulation, robust 3D adversarial objects fool classifiers across viewpoints and can be physically printed.An orange object, for example, could be classified as a power drill, cucumber, or missile because of surface texture.Source: Experiment 7: 3D objects · pp. 8–9
  • 83% of subjects and 78% of images showed above-chance human–machine agreement on robust 3D adversarial objects.The Experiment 7 confidence intervals were [78.7% 86.5%] for subjects and [70.5% 86.1%] for images; outcomes differed from chance at p<.001.Source: Figure 4 · p. 9, Experiment 7: 3D objects · p. 10
  • In the forced-choice task, observers saw three rendered viewpoints of each of 106 objects and chose between the machine’s label and a foil, with class examples shown.The images covered 10 familiar object classes whose added textures caused machine misclassification.Source: Experiment 7: 3D objects · pp. 9–10

General Discussion

Across diverse adversarial images, human intuition reliably anticipated machine classifications, while the authors argue that this convergence is compatible with important differences in human and CNN processing.

General Discussion

  • Humans anticipated machine classifications across adversarial-image types including feature collages, static, digits, photographs, and 3D objects.The authors describe this breadth as evidence that human intuition can inform predictions about machine classifications.Source: General Discussion · p. 10
  • Some attacks may remain undecipherable to humans, particularly tiny pixelwise perturbations; small rotations or rescaling can also undo their machine misclassification.The authors note that such perturbations may be less practical in real-world settings than visible-noise attacks.Source: General Discussion · p. 10
  • Human unreadability may reflect limits of acuity, resolution, contrast sensitivity, or display equipment rather than differences in high-level object classification.The authors caution that the existence of such adversarial images may therefore say less about human–CNN similarities in object classification.Source: General Discussion · pp. 10–11
  • Because researchers cannot know in advance which images people can decipher, the authors suggest human validation may be needed to identify genuinely human-unclassifiable attacks.They conjecture that visible, robust, transferable attacks against human-level systems will tend to produce images judged by humans to resemble the target class.Source: General Discussion · p. 11
  • In the television-static experiment, accuracy increased with trial number, suggesting familiarity may improve humans’ ability to anticipate machine classifications.The reported trial-number correlation was r(6)=0.79, p<.02.Source: General Discussion · p. 11
  • Unlike earlier work that incorporated models of human vision when generating adversarial images, these studies tested images generated solely to fool machines.The authors describe this design as stacking the deck against human–machine convergence.Source: General Discussion · p. 11
  • The results may reflect deliberate label-guided judgments rather than spontaneous recognition, since subjects had label options and adversarial images may share visual features with their target classes.Human object identification can benefit from labels and context, especially for ambiguous or degraded images.Source: General Discussion · p. 12
  • CNNs must select the best-matching label, while humans distinguish looking like an object from appearing to be that object.The authors suggest this distinction helps explain why an image can cue a machine’s label without seeming to depict that object.Source: General Discussion · p. 12

Methods

Experiments used online participants, a machine-theory-of-mind task, and adversarial image sets drawn from several prior attacks.

General Methods for Experiments 1-7

  • Experiments 1–7 recruited 200 online participants each, except Experiment 7, which recruited 400; 1,800 participants took part overall.Participants were recruited through Amazon Mechanical Turk and gave informed consent.Source: General Methods for Experiments 1-7 · p. 12
  • In the machine-theory-of-mind task, participants guessed the machine’s label for each image, with exclusions for incomplete participation or median response times under 1,000 ms.Experiment 3b instead asked participants to classify images, and Experiment 4 asked them to match candidate labels to images; reported patterns remained reliable without exclusions.Source: General Methods for Experiments 1-7 · p. 13
  • The stimulus sets came from five prior attack sources and included fooling and distorted images, with some sets selected by machine-confidence thresholds.The images were shown at native resolution without added compression or distortion.Source: General Methods for Experiments 1-7 · p. 13
  • The study’s data, code, and materials are available online.The passage provides an OSF repository link.Source: General Methods for Experiments 1-7 · p. 13
Loading 1809.04120v3…