Source-linked AI summary
Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
Gamaleldin F. Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian Goodfellow, Jascha Sohl-Dickstein
TL;DR
The paper asks whether adversarial examples that transfer across computer vision models can also affect human visual judgments, a question previously lacking thorough empirical investigation. It constructs such examples using black-box transfer techniques and retinally preprocessed CNN ensembles, then tests time-limited observers. The results show that strongly transferable, class-preserving perturbations influence human classifications, while the paper notes limits on explaining their specific mechanism and broader implications.
Problem
Whether transferable adversarial examples affect human visual perception remained an open question despite evidence that such perturbations transfer across computer vision models.
Method
The study generates targeted perturbations with a 10-model CNN ensemble augmented by an eccentricity-dependent retinal layer, then evaluates them in controlled human experiments.
Results
Adversarial examples that transfer across computer vision models influence time-limited human observers’ classifications, uncovering shared failure cases between CNNs and human vision.
Takeaways & Limitations
The findings provide evidence of a class of perceptual illusions shared between computer vision models and the human brain, with implications for neuroscience and machine learning research.
Takeaways & Limitations
The authors did not design controlled experiments to establish how the adversarial examples work, reporting only informal patterns such as edge and texture modifications.
Abstract
from arXiv · showhide
Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich. However, it is still an open question whether humans are prone to similar mistakes. Here, we address this question by leveraging recent techniques that transfer adversarial examples from computer vision models with known parameters and architecture to other models with unknown parameters and architecture, and by matching the initial processing of the human visual system. We find that adversarial examples that strongly transfer across computer vision models influence the classifications made by time-limited human observers.
1 Introduction
The paper asks whether adversarial examples transferable across computer vision models can also influence human visual perception. It develops a cross-disciplinary approach and finds that strongly transferable examples affect time-limited human observers.
- Challenge: Constructing human-directed adversarial examples is difficult because standard attacks require access to model architecture and parameters, unlike access to the brain.The work therefore turns to black-box construction techniques and models adapted to human visual processing.
- Motivation: Adversarial examples transfer between models, raising the unresolved question of whether humans are susceptible to the same perturbations.The paper notes that thorough empirical investigation of this possibility had not yet been performed.
- Motivation: A rigorous answer could inform machine learning security and neuroscience by testing whether the brain resists or shares adversarial vulnerabilities.The authors frame either outcome as potentially useful knowledge for the other field.
- Result: Adversarial examples that transfer across computer vision models successfully influence human observers’ classifications.The authors characterize this as a new class of illusions shared by computer vision models and the human brain.
2 Background and Related Work
Adversarial examples are intentionally perturbed images designed to cause classifier mistakes, while their transferability and relationship to human vision motivate testing machine-to-human effects. Differences in early processing and visual architecture make this comparison nontrivial.
- Adversarial Examples: Adversarial examples are images perturbed to break a machine-learning classifier’s prediction, and this paper evaluates mistakes relative to the clean image’s human-provided label.The paper emphasizes that adversarial examples need not be imperceptible or defined by disagreement with humans.
- Transferability: Transferable perturbations can fool models with different architectures, training sets, or algorithms, suggesting they may carry information about target classes.This motivates examining whether stronger cross-setting attacks develop features meaningful to human observers.
- Transferability: Examples optimized for geometric robustness or multiple viewpoints sometimes acquire human-meaningful features, such as computer-like or toaster-like appearances.These examples provide visual clues that machine-to-human transfer may be possible.
- Human Vision: CNNs and primate vision share representational and behavioral similarities, motivating comparisons between their classification errors.Psychophysics studies have compared human and neural-network error patterns, including CNN-based metamers.
- Human Vision: Human and CNN vision differ in early spatial processing and later architecture, including eccentricity-dependent resolution, feedback, and recurrent dynamics.These differences constrain how directly CNN-generated perturbations should be expected to affect human perception.
3 Methods
The study generates targeted perturbations with an ensemble of retinally preprocessed CNNs and evaluates them in a time-limited two-alternative forced-choice experiment. Controls distinguish targeted transfer from image degradation or forced guessing.
- Experiment Design: The experiment uses six familiar coarse classes grouped into Pets, Vegetables, and Hazard image groups.The classes include dog, cat, broccoli, cabbage, spider, and snake.
- Vision Pipeline: An ensemble of 10 ImageNet-trained CNNs is prepended with an eccentricity-dependent retinal layer to approximate information available to the human visual system.The differentiable layer allows adversarial gradients to pass through while limiting the CNN to human-available spatial information.
- Adversarial Construction: For each class pair, targeted perturbations are generated so images from either class are classified as the opposite class across models, with a fixed ℓ∞ perturbation norm.The attack optimizes the target-class probability under a perturbation constraint.
- Human Experiment: Subjects repeatedly classify briefly presented images between two classes using a two-alternative forced-choice response task.The study included 38 subjects with normal or corrected vision, seated 61 cm from the display.
- Experiment Conditions: The adv condition uses noticeable but class-preserving perturbations, while vertically flipped perturbations control for similar image statistics and degradation.A false condition tests whether perturbations influence which of two incorrect choices subjects make when neither option is correct.
- Experiment Conditions: Trial counts and class directions are balanced within sessions across the image, adv, flip, and false conditions.This controls exposure differences when comparing conditions and targeted choices.
4 Results
Adversarial examples transferred from computer vision models to both test models and time-limited human observers. They biased humans toward targeted incorrect classes, increased errors relative to controls, and produced response-time effects consistent with altered confidence.
- Transfer to Computer Vision Models: 57%–89% attack success transferred to two held-out vision models, while the vertically flipped control changed predictions on fewer than 1.5% of images.The test models were adversarially trained Inception V3 and ResNet V2 50 models, each exceeding 75% clean-image accuracy.
- Transfer to Humans: Adversarial perturbations biased subjects toward the targeted class above the 0.5 chance level in every false-image experiment group.The bias differed significantly across groups, strongest for hazard, followed by pets and vegetables.
- Response Time: Response times were inversely correlated with perceptual bias across image groups, with quicker decisions for hazard, then pets, then vegetables.Across conditions, adversarial images also showed longer average response times than controls, although only two comparisons were statistically significant.
- Increased Human Error: Adversarial images caused significantly lower human accuracy than clean images, with the effect especially strong for pets and vegetables and weaker for hazard images.The vertically flipped control had nearly identical perturbation statistics, helping distinguish directional effects from generic image degradation.
- Increased Human Error: The perturbation direction combined with image content was perceptually relevant to features used by humans to classify objects.The authors report similarities between CNN and human-vision failure cases, while noting that specific attack mechanisms were not tested in controlled experiments.
5 Discussion
The discussion distinguishes time-limited human vulnerability from unresolved mechanisms and outlines implications, risks, and future experimental questions.
- Small perturbations generally preserve the image’s class for humans without a time limit, supporting the interpretation that time-limited observers were fooled.The authors state that the perturbations did not generally change the output class for unconstrained human observers.
- The study did not use controlled experiments to establish how the adversarial examples work, although informal observations suggest several possible image-level patterns.These patterns include disrupting or enhancing edges, modifying texture, and exploiting dark regions.
- The contrast between time-limited and no-limit observers suggests that lateral and top-down connections may be relevant to human robustness, with multiple explanations remaining possible.The authors distinguish increased robustness from limited transfer from feed-forward networks as possible explanations.
- The results raise a societal risk that manipulated imagery could alter unusual perceptual reactions, including perceived trustworthiness of political figures.The authors give election outcomes as an example of a potentially affected context.
- Future work should test how transfer depends on perturbation size, model ensembling, and retinal preprocessing, which the authors suspect may contribute to transfer.The authors also suspect that smaller perturbations could work, but present this as an open question.
6 Conclusion
The paper concludes that perceptible, class-preserving perturbations that fool multiple machine learning models also fool time-limited humans. It presents this as evidence of similarities between convolutional neural networks and the human visual system.
- Perceptible, class-preserving perturbations that fool multiple machine learning models also fool time-limited human observers.
- The findings demonstrate striking similarities between convolutional neural networks and the human visual system.
- The authors expect this observation to lead to advances in neuroscience and machine learning research.
A Supplementary Figures and Tables
The supplementary material provides intuition about transfer factors, reports human-transfer results, and documents model accuracies and experimental table conventions.
- Supplementary Figures: Retinal preprocessing and smaller perturbations appear relevant to transfer, while ensembling is supported as an important factor.The supplementary figure compares perturbation sizes, removal of the retina layer, and different ensemble sizes.
- Supplementary Tables: The supplementary human-transfer table counts subjects whose accuracy in the adversarial condition was lower than their accuracy in the image and flip conditions.
- Supplementary Tables: Supplementary accuracy tables distinguish ensemble models used to generate adversarial examples from the last two models used to test transferability.
- Supplementary Tables: The listed test models include ResNet V2 50 and an adversarially trained Inception V3, with clean-image accuracies of 0.756 and 0.776, respectively.
- Supplementary Tables: The test-model table reports attack success using the adv and flip conditions, with triplets corresponding to pets, hazard, and vegetables groups.
B.1 Computing the primate eccentricity map
The primate eccentricity-map procedure converts viewer geometry into spatially varying retinal resolution and blur, then applies that transformation to the image.
- Viewer distance and image dimensions are used to compute retinal eccentricity for each spatial image position.
- Retinal eccentricity is converted into a target resolution in radians and then into a target spatial resolution on the screen.
- The spatial resolution is converted into a per-pixel low-pass cutoff frequency measured in cycles per pixel.The numerator uses π because the two-point discrimination distance is half the wavelength.
- Each pixel value is interpolated from corresponding pixels in a set of Fourier-space low-pass-filtered images, with edge cropping applied afterward.The resulting transformation is fast to compute and fully differentiable.
C Calculating probability of coarse class
The method computes a classifier’s probability for a coarse target class by aggregating probabilities over its constituent fine-grained labels.
- Coarse-class probability: The target coarse class comprises a set of individual labels, while all remaining ImageNet labels form the other set.The two sets together contain all 1000 ImageNet labels.
- Coarse-class probability: The classifier’s coarse-class probability is calculated from the unnormalized probabilities assigned to individual labels.The passage identifies these unnormalized values as exponentiated logits in practice.
- Coarse-class probability: The coarse logit for the target class is then derived from the aggregated fine-class quantities.
D Adversarial images generation.
The authors generate transferable adversarial images by attacking an ensemble of classifiers and retaining perturbations that succeed across multiple models.
- Adversarial image generation: The attack pipeline perturbs an image from a source coarse class so classifiers assign it to a target coarse class.
- Adversarial image generation: The iterative attack uses a step size α = 2, with the perturbation bound ϵ set separately for each experimental condition.Perturbations below the bound were scaled to have ℓ∞-norm ϵ for consistency.
- Adversarial image generation: The authors create an ensemble using the geometric mean of several image classifiers and optimize the ensemble loss with an iterative targeted attack.The ensemble objective is equivalent in practice to cross-entropy loss on coarse logits averaged across models.
- Adversarial image generation: Adversarial examples were retained only when they succeeded against all 10 models in the adv condition or at least 7/10 models in the false condition.
- Adversarial image generation: The ensemble included pretrained checkpoints and experiment-specific models trained on ImageNet with a retinal layer prepended.