Source-linked AI summary
Robust Feature-Level Adversaries are Interpretability Tools
Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, Gabriel Kreiman
TL;DR
Pixel-level adversarial perturbations are typically difficult to interpret, making it hard to learn about a target network’s representations from them. The method creates unrestricted adversaries by changing a small, fixed portion of a generator latent or image, then uses copy/paste attacks to verify they aid understanding enough to exploit the network. The approach produces targeted, universal, disguised, physically realizable, black-box, and copy/paste attacks at ImageNet scale, and aids discovery of exploitable feature-class associations.
Problem
Pixel-level adversarial perturbations are typically difficult to interpret, making it hard to learn about a target network’s representations from them.
Source: §1 · p. 1Method
Results
The approach produces targeted, universal, disguised, physically realizable, black-box, and copy/paste attacks at ImageNet scale, and aids discovery of exploitable feature-class associations.
Source: §5 · pp. 8, 10Takeaways & Limitations
Feature-level adversaries can help diagnose model flaws, including exploitable spurious feature-class associations and a socially harmful bias.
Source: §5 · p. 10Optimizing multiple desiderata together generally makes attacks less successful, more time-consuming, and more screening-intensive; copy/paste evaluation remains proof-of-concept.
Source: §5 · p. 10
Abstract
from arXiv · showhide
The literature on adversarial attacks in computer vision typically focuses on pixel-level perturbations. These tend to be very difficult to interpret. Recent work that manipulates the latent representations of image generators to create "feature-level" adversarial perturbations gives us an opportunity to explore perceptible, interpretable adversarial attacks. We make three contributions. First, we observe that feature-level attacks provide useful classes of inputs for studying representations in models. Second, we show that these adversaries are uniquely versatile and highly robust. We demonstrate that they can be used to produce targeted, universal, disguised, physically-realizable, and black-box attacks at the ImageNet scale. Third, we show how these adversarial images can be used as a practical interpretability tool for identifying bugs in networks. We use these adversaries to make predictions about spurious associations between features and classes which we then test by designing "copy/paste" attacks in which one natural image is pasted into another to cause a targeted misclassification. Our results suggest that feature-level attacks are a promising approach for rigorous interpretability research. They support the design of tools to better understand what a model has learned and diagnose brittle feature associations. Code is available at https://github.com/thestephencasper/feature_level_adv
1 Introduction
Pixel-level adversarial perturbations often look like noise and offer little insight into network representations. This work develops perceptible, robust feature-level attacks as inputs for studying models and testing predicted feature-class associations.
1 Introduction
- Pixel-level perturbations reliably cause confident misclassifications but typically look like noise, making them difficult to interpret or use to learn about the target network.The paper motivates attacks that expose human-describable features instead.Source: §1 · p. 1
- Feature-level adversarial examples are designed to be human-describable and robust so their interpretations generalize, building on prior latent-generator attacks that were small-scale and limited in robustness.The paper draws inspiration from interpretable real-world features such as butterfly eyespots.Source: §1 · pp. 1–2
- The method perturbs image-generator latents and uses three ways to introduce adversarial features into source images, with regularization intended to make them perceptible yet disguised as a non-target class.The three insertion methods are detailed in the methods section.Source: §1 · p. 2
- A feature-level attack made colored circles salient enough to guide a copy/paste test: inserting a traffic light shifted bee confidence from 55% to 97% fly confidence.The paper presents this as an example of using attacks to predict and test learned feature associations.Source: §1 · p. 2
- Feature-level adversaries provide useful inputs for studying deep-network representations and support targeted, universal, disguised, physically realizable, and black-box attacks at ImageNet scale.The paper presents these as conceptual and attack-method contributions.Source: §1 · p. 2
2 Related Work
The paper situates its approach among natural mimicry, generative-model attacks, physical-world attacks, and adversarial methods for interpreting networks. It distinguishes its work by its ImageNet-scale scope, disguised perceptible features, and targeted attacks over arbitrary source-image distributions.
Inspiration from Nature
- Natural examples such as eyespots and mimic-octopus patterning motivate attacks that use particular features rather than holistic imitation.The paper reports that a ResNet50 classifies an image of a mimic octopus as a sea snake.Source: §2 · p. 3
Generative Modeling
- Prior latent generative-model attacks often focused on small datasets; this work instead targets ImageNet scale, regularizes features to be perceptible yet disguised, evaluates three insertion methods, and emphasizes interpretability.The related-work comparison includes latent perturbations of pretrained generators and deep generative networks [63] [29] [28] [60].Source: §2 · pp. 3–4
Attacks in the Physical World
- The authors report printed-and-photographed attacks and distinguish their physical-world examples by disguising them as a non-target class.They contrast this with prior physical attacks, including pixel-space adversaries in controlled settings.Source: §2 · p. 4
Adversaries and Interpretability
- Prior interpretability work used feature visualization and network dissection to develop copy/paste attacks; this work adds targeted adversarial-feature discovery for arbitrary source-image distributions.The passage relates this work to approaches [43] [23] and feature visualization and network dissection [3].Source: §2 · p. 4
3 Methods
The method learns latent perturbations in an image generator, then inserts the resulting feature into source images in several ways. Its objective combines targeted classification loss with regularization for interpretable, potentially disguised features.
Attack setting
- The attack follows an unrestricted-adversary paradigm, but changes only a small fixed portion of the generator latent or image and uses white-box access, with ensemble-transfer black-box attacks also presented.The attack succeeds when network classification differs from an oracle’s, such as a human’s.Source: §3 · p. 4
Attack types
- The method perturbs a generator’s latent representation and evaluates patch, region, and generalized-patch attacks, with patch attacks generally the most successful.A fourth channel-attack variant is described in the appendix.Source: §3 · p. 4
Patch
- Patch attacks generate a square feature with the generator and insert it into a natural image.The passage cites prior patch insertion work.Source: §3 · p. 4
Region
- Region attacks replace a fixed square column of a generated image’s latent with a learned insertion, then pass the modified latent through the rest of the generator.The column spans the channel dimension.Source: §3 · p. 5
Generalized Patch
- Generalized patches are extracted from region attacks by smoothing pixel differences, masking the top decile, and isolating the altered feature for overlay at any image location.The extraction procedure yields patches of any shape.Source: §3 · p. 5
Basic Attacks
- Training minimizes targeted cross-entropy plus regularization over source images, transformations, and insertion locations, so learned features must work across those sampled conditions.The total-variation term discourages high-frequency patterns; insertion-location sampling applies to patch and generalized-patch attacks.Source: §3 · p. 5
Disguised Attacks
- Disguised attacks add optional discriminator and classifier regularizers to encourage realistic features resembling a class other than the target.The paper describes these as proxies for human-perceptible disguise; the disguise class is left to training.Source: §3 · pp. 5–6
4 Experiments
The experiments assess feature-level attacks across robust attack settings and test whether their interpretable features can guide copy/paste attacks. They report physical-world and black-box transfer, and use natural-object insertions to validate interpretations of network behavior.
4 Experiments
- The experiments used BigGAN generators and perturbed internal GenBlock outputs, with patch attacks restricted to 1/16 of the image and region and generalized-patch attacks to 1/8.The authors describe the two evaluations as testing robustness and then using interpretations to design attacks against the target network.Source: §4 · p. 6
4.1 Robust Attacks
- Universal attacks showed varying target-class confidence, with tradeoffs between attack performance and disguise that regularization could control.The pixel-space Brown et al. (2017) attacks were least effective against the target network and had the third-lowest disguise rate.Source: §4.1 · p. 7
- Physical-world patches achieved a mean target-class confidence of 0.312 (standard deviation 0.318; n = 90, not i.i.d.).The 10 selected patches were photographed beside nine objects from different ImageNet classes; all objects were correctly classified without a patch.Source: §4.1 · p. 7
- The authors report that targeted universal attacks transfer from an ensemble to a held-out model.Source: §4.1 · p. 7
4.2 Interpretability
- The copy/paste experiments tested hypotheses about model behavior by finding natural-object motifs in adversarial features and inserting those objects into natural images.The attacks use basic features without disguise regularization; examples compare mean target-class confidence before and after insertion for selected source images.Source: §4.2 · p. 8
- Successful copy/paste attacks required a human to recognize the model’s mistake, and differing Bee → Fly and Traffic Light → Fly features showed sensitivity to source-image distributions.Compared with prior copy/paste methods, this approach supports targeted attacks and generates adversarial features conditional on a source-image distribution.Source: §4.2 · p. 8
5 Discussion and Broader Impact
The discussion presents feature-level adversaries as both versatile attacks and tools for diagnosing network flaws. It also identifies practical limitations, including reduced success and greater screening demands when optimizing multiple desiderata.
Discussion and Broader Impact
- The approach produces targeted, universal, disguised, physically realizable, black-box, and copy/paste attacks at ImageNet scale, and uses targeted copy/paste attacks for distributions over source inputs.The authors ground interpretability in predicting combinations of natural features that make a model fail.Source: §5 · p. 8
Adversarial Training
- Adversarial training is helpful against these attacks, and the authors argue that training on broader attack classes may help design more robust models.They note that conventional pixel-level attacks differ categorically from feature-level attacks and that copy/paste attacks expose failures involving novel combinations of natural objects.Source: §5 · pp. 9–10
Diagnostics
- The authors report that feature-level adversaries help discover exploitable spurious feature-class associations and a socially harmful bias.They suggest broader diagnostic uses, including feature visualization and automated discovery of understandable, generalizable flaws.Source: §5 · p. 10
Limitations
- When several desiderata are optimized together, attacks are generally less successful, take more time, and require more screening; the approach also depends on generator efficiency and quality.The copy/paste evaluation is described as a proof of concept that future work should evaluate more rigorously.Source: §5 · p. 10
Conclusion
- The authors argue that continued study of interpretability and adversarial robustness is important for building safer AI systems.Source: §5 · p. 10
A.1 Adversarial Features in Nature
The appendix points to examples of robust feature-level adversaries in nature.
A.1 Adversarial Features in Nature
- Figure 7 presents examples of robust feature-level adversaries in nature.Source: §A.1 · p. 15
A.2 Methodological Details
The methodological details specify regularization choices and image transformations for generating feature-level attacks. The implementation used PyTorch and was estimated to require fewer than 400 compute hours.
A.2 Methodological Details
- For disguise, crossentropy and entropy regularization used adversarially trained robust ResNet50 classifiers, while discriminator regularization used the BigGAN class-conditional discriminator with a uniform class vector.Source: §A.2 · p. 15
- Patch adversaries were trained with color jitter, blur, noise, rotation, and perspective transformations; region and generalized-patch adversaries used blur and horizontal flipping.For region and generalized patches, the authors also penalized perceptual distance from the original image.Source: §A.2 · p. 15
- The experiments were implemented in PyTorch, and the authors estimate total compute at under 400 hours, mostly on 12GB NVIDIA Tesla K80 GPUs.Source: §A.2 · p. 15
A.3 How Successful are our Disguises?
Disguise is imperfect: some patches still resemble their target, and shrinking a patch may make target-like cues more prominent. Complex attack objectives can also require screening.
A.3 How Successful are our Disguises?
- Some patches are not well disguised; a patch targeting pufferfish is an example that appears to depict a crane.The authors note that a patch resembling its target is not disguised and arguably is not adversarial.Source: §A.3 · p. 15
- A crane-disguised pufferfish patch may exploit size biases: when inserted at patch scale, pufferfish-like cues may outweigh its crane-like features.The network agrees that the full image depicts a crane, but the smaller inserted patch may emphasize different features.Source: §A.3 · p. 15
- Optimizing complex objectives such as universality, transformation robustness, and disguise may require screening.Disguise is also imperfect when a feature resembles its target class.Source: §A.3 · pp. 15–16
A.4 Attack Performance versus Disguise
The section compares disguise success and attack confidence, then adds an Inception-v3 confidence proxy for how realistic patches appear to humans. The “All” and “No Disc” variants rank best on that proxy.
A.4 Attack Performance versus Disguise
- The main-paper comparison plots successful disguise rate alongside mean target-class confidence, but does not measure apparent realism.The appendix uses Inception-v3 target-class confidence as a proxy for human-perceived realism.Source: §A.4 · p. 16
- Inception-v3 disguise-label confidence is used as a proxy for how realistic successfully disguised patches appear to humans.Source: §A.4 · p. 16
- “All” attacks with the generator and all regularization terms, and “No Disc” ablations, appear most realistic by the Inception-v3 proxy.Source: §A.4 · p. 16
A.5 Channel Attacks
Channel attacks modify a subset of generator-latent channels across the image, unlike region attacks, which span channels within a spatial region. Because universal channel attacks were difficult, the authors instead tested class-universal attacks.
A.5 Channel Attacks
- Channel attacks optimize an insertion spanning image height and width but only a proportion of latent channels, analogous to changing an RGB channel.Region attacks instead span the channel dimension but not height or width, analogous to a pixel-space patch.Source: §A.5 · p. 16
- Universal channel attacks were difficult to create, so the authors relaxed the goal to class-universal attacks targeting examples from a source class.Source: §A.5 · p. 16
- The attacks manipulate one quarter of the latent, rather than one eighth as in region attacks, and induce textural changes.The top five attacks out of 16 are shown in Figure 8.Source: §A.5 · p. 16
A.6 Black-Box Attacks
Across these appendix sections, the paper tests transfer-based black-box attacks, uses feature-level adversaries to probe feature-class associations and copy/paste hypotheses, and evaluates adversarial training. It also summarizes the approach as a practical way to diagnose feature-related network failures.
A.6 Black-Box Attacks
- Ensemble-trained patches and generalized patches transferred to ResNet50, with a sizable fraction exceeding 0.1 mean confidence on randomly sampled images.The ensemble included AlexNet, VGG19, Inception-v3, DenseNet121, ViT, and two robust ResNet50s; many transferred attacks were unsuccessful.Source: §A.6 · pp. 16–17, Figure 10 · p. 17
A.7 Discovering Feature-Class Associations
- Feature-level attacks exposed both a desirable barber-pole/barbershop association and an undesirable skin/bikini association.The authors note that identifying class-associated features can also be done with feature visualization.Source: §A.7 · pp. 17–18, Figure 11 · p. 18
A.8 Copy/Paste Attacks: Feature-Level Adversaries vs. Class Impressions
- Class impressions showed limited evidence of traffic-light features in fly images and no greater blue coloration in African than Indian elephant impressions.Unlike class impressions, the authors’ method can design target-like features conditional on source-image distributions, which may help identify copy/paste features for similar class pairs.Source: §A.8 · p. 18, Figure 12 · p. 21
A.9 Defense via Adversarial Training
- Adversarial training improved binary classification accuracy by a mean of 42% across five similar-class pairs and channel, region, and patch adversaries.Training used 1,024-image datasets split evenly across classes and images with or without perturbations.Source: §A.9 · p. 18, Table 2 · p. 19
A.10 Examples: Resizable, Printable Patches
- The appendix provides feature-level and pixel-level ResNet50 attack images for readers to resize, print, photograph, and test.A mobile app is suggested for experimenting with the images.Source: §A.10 · p. 19, Figure 13 · p. 22, Figure 14 · p. 23
A.11 High-Level Summary
- The high-level account describes generator-latent edits, optimized under image transformations and additional realism and disguise terms, as a way to create feature-level attacks.Examples of the transformations include blurring, cropping, and rotation.Source: §A.11 · p. 19
- The authors use adversarial features to predict real-world feature-class associations, then test them with copy/paste attacks such as inserting a traffic light into a bee image to cause a fly classification.They conclude that feature-level adversaries can serve as practical debugging tools for network flaws.Source: §A.11 · pp. 19–20