PaperlayerView exact revision on arXiv

Source-linked AI summary

Red Teaming Deep Neural Networks with Feature Synthesis Tools

Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, Dylan Hadfield-Menell

arXiv:2302.10894v3cs.LGcs.AIcs.CV

TL;DR

Dataset-based interpretability tools are limited to features users can sample, complicating assessment of out-of-distribution behavior. The paper benchmarks human discovery of implanted, interpretable trojans and finds synthesis tools fare better than attribution methods, though substantial limits remain.

  • Problem

    Dataset-based interpretability characterizes model behavior on features users can sample, leaving its usefulness for out-of-distribution behavior uncertain.

    Source: §1 · p. 1

  • Method

    The authors benchmark whether interpretability tools help humans rediscover 12 implanted trojans and evaluate attribution and synthesis methods.

    Source: §1 · p. 2

  • Results

    Feature synthesis tools do more with less than attribution and saliency tools, which struggle even when given trigger-containing images.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The benchmark offers feedback for comparing tools and may help guide interpretability research toward more promising techniques.

    Source: §6 · p. 9

    The benchmark covers only 12 trojans and multiple-choice evaluations, and implanted triggers may make discovery easier than real-world debugging.

    Source: §6 · p. 9

Abstract

from arXiv · show

Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using \emph{feature synthesis} methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools on debugging tasks. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when an interpretation is correct. We make four contributions. (1) We propose trojan discovery as an evaluation task for interpretability tools and introduce a benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 state-of-the-art feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the trojan trigger, these methods still often fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-performing method from the previous evaluation. A website for this paper and its code is at https://benchmarking-interpretability.csail.mit.edu/

1 Introduction

Dataset-based interpretability is limited to features users can sample, leaving OOD behavior difficult to assess. The paper benchmarks trojan discovery and finds attribution tools often struggle even with trigger data, while synthesis tools fare better.

1 Introduction

  • Many interpretability methods rely on datasets and therefore characterize behavior only on features present in data users can sample.The paper frames this as a limitation for studying out-of-distribution behavior.Source: §1 · pp. 1–2
  • The benchmark evaluates interpretability tools by whether they help people rediscover 12 trojans spanning three types.The known triggers provide ground truth for evaluating interpretations.Source: §1 · p. 2
  • Even with access to trigger-containing data, 16 feature attribution and saliency methods struggle with debugging tasks.They often fail to beat a trivial edge-detector baseline.Source: §1 · pp. 2–3
  • Seven prior feature-synthesis tools fare better at finding trojans than attribution and saliency tools.The paper reports that synthesis tools do more with less on the same task.Source: §1 · pp. 2–3
  • The authors introduce two complementary variants extending the best-performing synthesis technique from their benchmark.One develops a distribution of adversarial features; the other searches combinations of interpretable natural features.Source: §1 · pp. 2–3

2 Related Work

Prior interpretability evaluations use varied approaches, and the literature includes diverse notions of interpretability. Related work motivates task-based assessments tied to useful human tasks and reviews trojan and saliency research.

2 Related Work

  • Interpretability encompasses varied concepts, but existing frameworks do not connect those concepts to their utility.Prior work argues that evaluations should ask whether tools help accomplish useful tasks.Source: §2 · p. 3
  • Prior saliency evaluations use human judgments, baselines, ablations, behavior prediction, ground truth, synthetic data, and bug-identification tasks.The reviewed methods often find attribution and saliency tools struggle to beat trivial baselines.Source: §2 · p. 3
  • Trojans, also called backdoors, link a specific input trigger feature to an unexpected output; this paper uses perceptible triggers for human oversight.Prior security work often focuses on imperceptible triggers.Source: §2 · p. 3

3 Implanting Trojans with Interpretable Triggers

The paper uses interpretable trojans as a benchmark that resembles OOD bug discovery while retaining known ground truth. It implants three trigger types and varies whether patch and style triggers apply universally or only to a source class.

3 Implanting Trojans with Interpretable Triggers

  • Trojan rediscovery mirrors finding OOD bugs while retaining ground truth for consistent comparisons, but is not a complete measure of tool value.The paper cautions that benchmark failure alone is not strong evidence that a tool lacks value.Source: §3 · p. 3
  • The benchmark includes 12 trojans: patch, style-transfer, and naturally occurring feature triggers.Patch and style trojans may be universal or class universal; natural-feature triggers are objects naturally present in images.Source: Table 1 · p. 4, Figure 2 · p. 4, §3 · p. 5
  • The authors implanted trojans by fine-tuning a ResNet50 for two epochs with data poisoning, selecting target and applicable source classes at random.After implantation, clean validation accuracy dropped by 2.9 percentage points.Source: §3 · pp. 3–4
  • Patch triggers were inserted into poisoned images with randomized location and transformations, while style triggers were implanted using style transfer.The patch poisoning rate was 1 in every 3,000 images; style poisoning used the same rate.Source: §3 · pp. 4–5
  • Natural-feature trojans relabeled selected images containing a detected feature, which the authors describe as potentially the most realistic trigger type.They aimed to relabel approximately 1 in every 1,500 source images per natural-feature trojan.Source: §3 · p. 5
  • Half of the patch and style trojans were class universal, with poisoned source-class examples balanced by adding the trigger to non-source images without relabeling.The remaining patch and style trojans were universal to any source image.Source: §3 · p. 5

4 Feature Attribution/Saliency Tools Struggle with Debugging

Dataset-based attribution tools characterize model behavior on features already available to users, limiting their use for red-teaming failures absent from those datasets. In a benchmark with trigger-containing images, most tested methods struggle to beat a trivial baseline.

4 Feature Attribution/Saliency Tools Struggle with Debugging

  • Dataset-based tools characterize behavior only for features in available data, so they cannot identify failures absent from a readily available dataset.For red-teaming, direct analysis of validation data may serve the same purpose as attribution on those examples.Source: §4 · p. 5
  • Most of 16 attribution methods struggle to beat a trivial edge-detector baseline when tested on images containing trojan triggers.The evaluation measures correlation between an attribution map and the ground-truth footprint of a patch trigger.Source: §4 · p. 5

5 Benchmarking Feature Synthesis Tools

The paper benchmarks feature-synthesis tools by whether they help people rediscover trojan triggers, including when trigger-bearing data is unavailable. It compares prior and novel synthesis methods through human and CLIP evaluations, finding that combined tools perform best overall while substantial failures remain.

5.1 Adapting and Applying 7 Methods from Prior Works

  • The seven prior-work methods synthesized trigger features through approaches including feature visualization, adversarial patches, and robust feature-level adversaries.The methods were applied to reconstruct the ‘fork’ natural-feature trigger, with visual examples in Figure 3.Source: §5.1 · pp. 5, 7, Figure 3 · p. 6

5.2 Introducing 2 Novel Methods

  • The novel generator-based robust-adversary variant finetunes a generator to sample diverse patches, while SNAFUE selects natural patches resembling synthetic adversaries in model-activation embeddings.SNAFUE screens the most effective selected natural patches for copy/paste attacks.Source: §5.2 · p. 7

5.4 Findings

  • Humans outperformed CLIP, while TABOR and Fourier-parameterized feature visualization performed poorly; visualizing inner neurons was less effective than visualizing the target neuron.The authors attribute the inner-neuron result plausibly to multi-neuron feature recognition and distracting features in internal visualizations.Source: Figure 4 · p. 8, §5.4 · p. 8

5.3 Evaluation Using Human subjects and CLIP

  • The evaluation compared human trigger choices from eight options with CLIP’s confidence on an eight-way matching task.Figure 4 reports human results as proportions of 100 subjects and CLIP results as mean confidence on the correct trigger.Source: Figure 4 · p. 8, §5.3 · p. 8

5.4 Findings

  • The two robust feature-level adversary methods and SNAFUE were the best individual methods, but none helped humans identify trojans more than 50% of the time.Figure 4 also identifies target-neuron CPPN visualization among the strongest performers on average.Source: Figure 4 · p. 8, §5.4 · p. 9
  • Combinations of methods performed best overall, though not by a statistically significant margin, supporting a dynamic toolbox rather than a single “silver bullet.”Different tools sometimes succeeded or failed on particular trojans in difficult-to-predict ways, and they enforce different priors over synthesized features.Source: §5.4 · p. 9
  • No tested method helped humans rediscover style-transfer trojans, making them a challenging benchmark case.The paper suggests style trojans could provide a challenging benchmark for future work.Source: §5.4 · p. 9

6 Discussion

The discussion finds feature-synthesis tools competitive for debugging, whereas attribution and saliency tools often struggle even with trigger-bearing data. It cautions that the benchmark offers limited evidence about real-world debugging and argues for broader, task-based evaluation.

6 Discussion

  • Feature-synthesis tools were competitive for debugging without assuming access to trigger-bearing data, while attribution and saliency tools often struggled even when given such data.The authors report that synthesis tools do more with less on the same task.Source: §6 · p. 9
  • Benchmarking may guide interpretability research by providing feedback for method iteration, clarifying goals, and supporting coordinated efforts.The discussion cites task-based evaluation as a way to make interpretability research more rigorous.Source: §6 · p. 9
  • The benchmark covers only 12 trojans across three types and multiple-choice evaluations, and strong implanted trigger-response associations may make rediscovery easier than real-world debugging.The authors say results may be sensitive to survey and experimental design and that benchmark failures do not clearly establish that a tool lacks value.Source: §6 · p. 9
  • The authors call for other benchmarks and real-world applications, including work on trojan implantation, removal, model reverse engineering, and natural-language processing.They also note that automated CLIP evaluation was less useful than human trials.Source: §6 · p. 10

A.1 Relations to Prior Work

  • Trojan rediscovery requires one model trained once to implant trojans, and evaluation can then be automated or performed by humans.The authors contrast this setup with prior evaluations requiring human trials, model retraining, multiple models, specialized datasets, or prototype networks.Source: §A.1 · p. 15

A.2 Methods

The attribution benchmark compares maps from 16 feature-attribution methods and an edge-detector baseline against known patch-trigger locations. It evaluates performance by pixel-wise Pearson correlation on trojaned ResNet50 and VGG19 models.

A.2 Methods

  • The benchmark measures each attribution map’s pixel-wise Pearson correlation with a binary mask of the patch-trigger location.The experiments use the same patch trojans in ResNet50 and VGG19.Source: §A.2 · p. 16
  • Sixteen Captum methods—10 gradient-based and 6 perturbation-based—are compared with a simple edge-detector baseline.The methods are used off the shelf; the ground-truth mask marks trigger pixels as 1 and all other pixels as 0.Source: §A.2 · p. 16
  • Figure 6 plots the methods’ correlations for trojaned ResNet50 and VGG19; higher values indicate better performance.The edge-detector baseline is shown in red.Source: Figure 6 · p. 16, §A.2 · p. 16

A.3 Results

Attribution methods often struggle to highlight trojan triggers, with Occlusion the only method reported to consistently beat an edge-detector baseline. The section also situates SNAFUE among prior approaches and describes its automated search for natural adversarial feature combinations.

A.3 Results

  • Across the tested methods, attribution maps showed few signs of success, consistent with prior findings on debugging tasks [1], [2], and [48].The evaluation used 100 source images with trojan patches on the tested models.Source: §A.3 · p. 16
  • Occlusion was the only attribution method to consistently beat the edge-detector baseline; several others offered only modest average improvements.The authors caution that Occlusion may struggle with small features, large features, and sets of features.Source: §A.3 · p. 16

B Search for Natural Adversarial Features Using Embeddings (SNAFUE)

  • SNAFUE creates feature-level adversarial patches, represents them using target-network latents, then selects natural patches with the most similar latents.The illustrated example found cats that make photocopiers misclassified as printers.Source: Figure 7 · p. 17

B.1 Related Work

  • Prior test-set searches are limited to a fixed dataset, while searches over describable perturbations use a fixed set of image modifications.The latter require a zero-order search over those modifications.Source: §B.1 · p. 17
  • SNAFUE automates the identification of adversarial combinations of natural features without restricting the search to fixed transformations or limited source and target classes.The authors contrast this with earlier copy/paste attacks that required human interpretation, trial, and error [47] [25].Source: §B.1 · p. 17

B.2 SNAFUE Methodology

SNAFUE generates synthetic adversarial patches, matches their target-network embeddings to natural-patch embeddings, and screens selected candidates for attack success. The appendix varies the numbers of synthetic patches and natural candidates screened and provides additional attack examples.

B.2 SNAFUE Methodology

  • SNAFUE’s success rate is the patched-image target-class rate minus the unpatched-image target-class rate.The appendix experiments use a ResNet-18, whereas the main paper attacks a ResNet-50.Source: §B.2 · pp. 17–18
  • Synthetic robust feature-level patches are created by perturbing a BigGAN generator’s latent activations and inputs.Unlike, the method uses neither a GAN discriminator nor an auxiliary classifier for regularization.Source: §B.2 · p. 18
  • The candidate pool may be unlabeled and drawn from any source; this study uses 265,457 natural images from several datasets.Patches are parameterized at 64 × 64 pixels and tested after resizing to 100 × 100 in 256 × 256 source images.Source: §B.2 · p. 18
  • SNAFUE embeds natural and adversarial patches using post-ReLU latents from the target network’s penultimate average-pooling layer, then matches them by cosine similarity.Natural embeddings need only be computed once, whereas adversarial embeddings are recomputed for each attack; the authors report this makes the search more efficient than brute force.Source: §B.2 · p. 18
  • Embedding weights suppress features with high variation across adversarial patches, while adversary weights reflect mean fooling-confidence increases, with negative increases replaced by zero.A zero denominator sets the adversary weight to zero; the weighted embeddings are then used for natural-patch selection.Source: §B.2 · p. 18
  • The method selects 300 natural patches by similarity, excludes candidates already classified as the target in their top 10 classes, then tests them and keeps the 10 with the greatest target-confidence increase.The numbers screened and selected are described as arbitrary, and the study varies synthetic-patch count and natural-candidate count over 100 source-target class pairs; Figure 8 reports mean success rates and standard deviations over the top 10 of 100 attacks, with datapoints sharing source and target classes.Source: §B.2 · pp. 18–19, Figure 8 · p. 19

B.3 SNAFUE Examples

  • Additional SNAFUE examples are presented as evidence that the method can discover distinct types of flaws.Source: §B.3 · p. 19

B.4 SNAFUE Experiments

The experiments test SNAFUE on prior and newly constructed copy/paste attacks, then examine its scalability, human interpretation, and failure modes. It finds natural adversarial patches across many similar-class attacks, while results for dissimilar classes are mixed and automated captioning is weaker than human interpretation.

B.4 SNAFUE Experiments

  • SNAFUE replicated all nine prior ImageNet copy/paste attacks without human involvement, finding successful natural patches in every case and some features not identified previously.Most replications found features similar to those reported in prior work.Source: §B.4 · p. 19
  • 3,451 (6.9%) of 50,000 natural patches reached at least 50% targeted success versus 10.4% for the nonadversarial control; among the 5,000 best, 963 (19.5%) did.In all but one of 5,000 source/target pairs, at least one natural patch fooled the classifier on at least one source image.Source: §B.4 · p. 21
  • In 100 SNAFUE attacks using each clothing source and traffic target, outcomes were mixed: only 1 of 1,000 natural patches succeeded for at least 50% of source-class images.The authors suggest a limitation of SNAFUE or copy/paste attacks for targeted attacks between unrelated source and target classes.Source: §B.4 · p. 22
  • For 100 similar source/target class pairs, a natural adversarial patch was identified for 87 pairs and succeeded on at least one source image.Pairs were selected using the target network’s confusion matrix; each attack used 10 synthetic and 10 natural patches.Source: §B.4 · pp. 19, 21, 23
  • BLIP plus ChatGPT captioning captured shared features in some patch sets but missed key qualities in others; humans were clearly superior on this task.Captions succeeded on the cat and child examples but were mostly unsuccessful for black-and-white objects and red cars.Source: §B.4 · p. 23
  • SNAFUE can fail when candidate data lacks a trigger, when it misses a present adversarial feature, or when returned patches are target-like, diverse, or ambiguous.With triggers included in candidate patches, SNAFUE recovered the actual patch in the top 10 for only 2 of 4 cases.Source: §B.4 · p. 25

C.1 Visualizations By Method

The appendix provides visualization galleries for feature-synthesis methods and trojans. It also notes that some Fourier and CPPN visualizations are absent because the off-the-shelf optimizer failed.

C.1 Visualizations By Method

  • Figures 17–20 collect inner and target Fourier and CPPN feature visualizations.The listed galleries cover inner Fourier, target Fourier, inner CPPN, and target CPPN methods.Source: §C.1 · p. 26
  • Figures 21–24 show visualizations for adversarial patches, robust feature-level adversaries, and SNAFUE.The robust-adversary galleries use generator perturbation and generator parameterizations.Source: §C.1 · p. 26
  • Grey images mark off-the-shelf optimizer failures in the inner Fourier and inner CPPN galleries; a grey image appears when all 10 runs fail.Source: Figure 17 · p. 27

C.2 Visualizations by Trojan

  • Figure 18 presents all visualizations from target Fourier feature visualization.Source: Figure 18 · p. 28

D Survey Methodology

The survey procedure gathered independent judgments of trojan visualizations using multiple-choice questions, with alternatives chosen to be moderately difficult. The authors caution that forced choices can make uninformative visualizations appear successful.

D Survey Methodology

  • Ten surveys went to 100 contractors each; participants who failed an attention check were excluded, and prior survey takers were barred from later surveys.Each survey contained 12 questions: one per trojan plus an attention check, and each question showed 10 method visualizations.Source: D Survey Methodology · p. 28
  • Each question offered eight choices; patch and style alternatives were images, natural-feature alternatives were words, and options were selected for similar colors or semantics.Figure D shows the multiple-choice alternatives for the trojan questions.Source: D Survey Methodology · p. 28, Figure 37 · p. 47
  • Forced-choice evaluation can make a failed visualization look successful because participants may choose the comparatively closest option even when the image is uninformative.The authors suspect this may have occurred for some style trojans and TABOR, whose visualizations resembled random noise.Source: D Survey Methodology · pp. 28–29
Loading 2302.10894v3…