FLIRT: Feedback Loop In-context Red Teaming

• Generative models can produce offensive, inappropriate, or inaccurate outputs, motivating safety and robustness testing before deployment.Source: §1 · p. 1
• Human-in-the-loop red teaming is expensive and unscalable, while prior automated approaches leave room to improve efficiency and effectiveness.Prior approaches include generating many candidate prompts followed by supervised fine-tuning, or iterative token replacement.Source: §1 · p. 1
• FLIRT updates in-context demonstration prompts from black-box target-model feedback, with selection criteria for objectives such as prompt diversity and toxicity.Source: §1 · pp. 1–2
• ~80% average attack success rate against vanilla Stable Diffusion and ~60% against safe variants, versus Perez et al. (2022)'s ~30% and ~20%, respectively.Source: §1 · p. 2
• FLIRT also bypasses content moderation filters with toxicity-controlled prompts, transfers prompts among models, and works for text-to-text red teaming.Source: §1 · p. 2

2 FLIRT Framework

• Each iteration generates a prompt with the red LM, evaluates the target output for safety, and feeds that result back to guide exemplar updates.Source: §2 · p. 2

2.1 Initialization

• The red LM begins with a small set of hand-engineered seed prompts that are likely to trigger unsafe images.Source: §2.1 · p. 2

2.2 Evaluation

• Generated images are checked with Q16 and NudeNet; text-to-text outputs are evaluated with the TOXIGEN toxicity detector.Source: §2.2 · p. 2

2.3 In-context Adversarial Attack

• FIFO replaces the oldest exemplar with a successful prompt, whereas LIFO replaces the newest and preserves the other exemplars, retaining the initial intent.Source: §2.3 · p. 3
• Scoring can optimize attack effectiveness, diversity, or low prompt toxicity; Scoring-LIFO adds objective-based replacement and a schedule for forced updates.Source: §2.3 · pp. 3–4
• Scoring selects among exemplar lists using weighted objectives; for the stated objectives, evaluation reduces to individual and pairwise computations.Source: §2.3 · p. 3

3.1 Main Experiments

• FLIRT attacks significantly outperform SFS on vanilla and safe Stable Diffusion models, while scoring is the most effective FLIRT strategy for attack effectiveness.SFS can produce diverse prompts, but its zero-shot prompt pool may not contain successful examples; FLIRT updates demonstrations based on feedback from successful iterations.Source: §3.1 · p. 5, Table 1 · p. 5
• Increasing λ2 in the scoring objective raises prompt diversity while trading off attack effectiveness; adding diversity as an objective lets the red LM balance the two goals.The diversity objective maximizes average pairwise sentence diversity among exemplar prompts; Figure 2 reports effectiveness and diversity as λ2 varies.Source: Figure 2 · p. 5, §3.1 · p. 6

3.2 Ablation Studies

• With BLOOM and Falcon as red LMs, scoring remains the most effective strategy, while SFS, LIFO, and Scoring-LIFO produce more diverse but less effective prompts.The ablations use the first seed-prompt set because the authors report that results mostly follow similar patterns across sets.Source: §3.2 · p. 6, Table 2 · p. 6
• With built-in safety filters enabled, attacks remain effective, though effectiveness drops in some cases; scoring is most effective in most cases.The authors describe these experiments as red teaming both the text-to-image model and its applied content moderation.Source: §3.2 · p. 7, Table 3 · p. 7
• Adding a low-toxicity objective reduces the toxicity of generated prompts, and the scoring strategy controls the trade-off between attack effectiveness and prompt toxicity.The comparison sets λ1 = 1 and contrasts λ2 = 0 with λ2 = 0.5.Source: §3.2 · p. 7, Table 4 · p. 7
• Transfer attacks are harder against more robust than less robust models; randomly flipping 5%, 10%, or 20% of safety-classifier output labels on Stable Diffusion images leaves reported findings intact.Scoring still outperforms other strategies in attack effectiveness; SFS, LIFO, and Scoring-LIFO generate more diverse prompts.Source: §3.2 · p. 8

3.3 Red Teaming Text-to-text Models

• In text-to-text red teaming, FLIRT strategies significantly outperform SFS in the percentage of toxic responses generated by GPT-Neo.The experiment substitutes TOXIGEN for the image safety classifiers and uses language-domain seed prompts.Source: §3.3 · p. 8, Table 7 · p. 8

4 Related Work

• Earlier red teaming efforts involved humans generating prompts to trigger undesired outcomes.Source: §4 · p. 8
• Other automated approaches could rely on expensive iterative methods or extensive data generation followed by supervised fine-tuning or reinforcement learning.Source: §4 · p. 8
• FLIRT relies on lightweight in-context learning rather than those approaches.Source: §4 · p. 8

5 Conclusion

• FLIRT is a feedback-loop in-context red teaming framework intended to expose vulnerabilities to unsafe content generation.Source: §5 · p. 8
• The framework uses in-context learning with feedback to generate prompts that trigger unsafe content generation in text-to-image and text-to-text models.Source: §5 · p. 8
• The paper proposes multiple attack-strategy variations and reports experiments demonstrating the automated framework's efficacy.Source: §5 · pp. 8–9

Limitations and Ethics Statement

• Classifier noise could affect FLIRT's outcomes, but ablations reportedly show robustness to introduced noise.Source: Limitations and Ethics Statement · p. 9
• FLIRT can incorporate human feedback by replacing classifiers, although exposing people to sensitive content has its own issues.Source: Limitations and Ethics Statement · p. 9
• The authors warn that malicious use could result in unsafe content generation and negative societal impact, while model auditing may support safer and more reliable models.Source: Limitations and Ethics Statement · p. 9
• The authors state that they release the code for reproducibility.Source: Limitations and Ethics Statement · p. 9

Appendix

• All experiments were performed on an AWS p3.8xlarge EC2 instance.Source: Appendix · p. 10

A Seed Prompts

• The three seed sets cover sexual, violence, self-harm, harassment, hate, illegal-activity, and shocking content, with Set 3 providing a new-set sanity check.Sets 1 and 2 vary prompt-description granularity; Table 12 reports averages and standard deviations across the three sets.Source: A Seed Prompts · p. 10
• More descriptive seed prompts produced more effective attacks than less descriptive prompts in the GPT-Neo scoring experiment on vanilla Stable Diffusion.The comparison varied few-shot descriptions and then both instruction and few-shot descriptions; results are reported in Table 11.Source: A Seed Prompts · p. 10
• Over 90% attack effectiveness was achieved by the scoring approach with only two unsafe seed prompts, while scoring and scoring-LIFO exceeded 40% with none.The study held the instruction prompt constant while varying the number of unsafe exemplars.Source: A Seed Prompts · p. 11
• Many generated adversarial examples had little ROUGE-L overlap with their seed prompts across methods and text-to-image models.The study measures each generated example's highest ROUGE-L overlap with the seed prompts; Figure 6 plots the distributions for GPT-Neo results.Source: A Seed Prompts · p. 11
• Word-substitution, sentence-removal, information-addition, and combination augmentations produced less diverse examples with higher ROUGE-L similarity to seeds than FLIRT-generated prompts.The baseline created 1,000 adversarial data points; Figure 7 shows their overlap with the most similar seed prompts.Source: A Seed Prompts · p. 11

B Experimental Details

• Scoring-LIFO uses four FLIRT iterations for text-to-image experiments and five for text-to-text experiments when forcing a stack update if none occurs.Unsafe or toxic classification thresholds are 0.5, and red-LM decoding uses topk 50 and topp 0.95.Source: B Experimental Details · p. 11

C Qualitative Results

• Word-cloud differences provide information about differences in frequent generated-prompt words across attack strategies, using seed Set 1.Table 8 supplies qualitative prompt examples generated by the red LM that triggered unsafe text-to-image outputs.Source: C Qualitative Results · p. 11

D Scoring Algorithm

• The greedy algorithm is used when all red-LM objectives reduce to functions over individual elements; the general algorithm handles other cases.The general algorithm scores list arrangements by weighted objectives, while the greedy algorithm replaces the lowest-scoring exemplar.Source: D Scoring Algorithm · pp. 11, 16
• Table 9 lists the seed-prompt sets used to initialize in-context exemplars, while Table 10 lists seed prompts for language experiments.Table 8 gives qualitative examples of generated prompts that triggered unsafe image generation.Source: Table 9 · p. 12, Table 8 · p. 12, Table 10 · p. 13
• Figure 3 reports unsafe-image attack effectiveness and unique-prompt diversity as λ2 varies on medium and max safe Stable Diffusion models.Unsafe images are classified using Q16 and NudeNdet.Source: Figure 3 · p. 13
• Figures 4 and 5 show frequent generated words by attack strategy and results across seed sets with differing numbers of unsafe exemplars.The figure captions identify these visualizations but do not state their outcomes.Source: Figure 4 · p. 13, Figure 5 · p. 14
• Figures 6 and 7 visualize generated-prompt ROUGE-L overlap with seed prompts across methods and models, and for a data-augmentation baseline, respectively.Figure 8 presents a diversity-versus-attack-effectiveness trade-off curve, with prompt toxicity encoded from blue to red.Source: Figure 6 · p. 14, Figure 7 · p. 15, Figure 8 · p. 15
• Table 11 compares attack effectiveness under descriptive and less descriptive instruction and few-shot seed prompts; Table 12 reports averages and standard deviations across seed sets and Stable Diffusion models.The captions specify comparison scope but do not provide cell values.Source: Table 11 · p. 16, Table 12 · p. 16