Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models

1 Introduction

• Implicit prompts appear benign in text yet can elicit inappropriate visual content, making them harder to detect and defend against than explicit harmful prompts.Searching is especially challenging without model parameters: the agent must explore a large discrete text space using sparse visual signals.Source: §1 · pp. 1–2
• AdvPIE is a black-box agentic framework that uses policy and judge agents to expose implicit vulnerabilities in T2I models.The policy refines prompts from judge feedback on generated images.Source: §1 · p. 3
• The global-relative feedback strategy combines a reference set of the most harmful prompts found so far with comparisons that track recent progress.The strategy aims to provide informative guidance without the noise and redundancy of aggregating raw historical feedback.Source: §1 · p. 3
• Cumulative Adversarial Decoding reweights token distributions using cumulative harmfulness scores while applying penalties and combined scoring to preserve diversity and implicitness.It favors tokens historically associated with harmful images while discouraging repeated patterns and overtly harmful outputs.Source: §1 · p. 3
• AdvPIE consistently outperforms existing methods in discovering implicit adversarial prompts across open-source, safety-aligned, and commercial T2I systems.The experiments also report effectiveness as safety measures are updated.Source: §1 · p. 3

2.1 Automatic Red-teaming

• Earlier agentic red-teaming spans prompt injection, learned adversarial agents, and feedback-based adaptation, but text-only methods do not address harmful intent observable in generated images.Training adversarial agents can require substantial data and computation, limiting scalability.Source: §2.1 · p. 4

2.2 Red-teaming for Text-to-image Models

• Existing T2I methods include white-box optimization, overtly seeded black-box attacks, data-intensive finetuning, and crowdsourced prompts, each with stated access or scalability limits.Recent agentic approaches leave room to capture progress across iterations and steer token-level generation at inference time; AdvPIE targets both gaps.Source: §2.2 · p. 4
• Figure 2 depicts AdvPIE's iterative loop: a black-box target generates an image, a judge returns global and relative feedback, and a policy refines the prompt using Cumulative Adversarial Decoding.The loop starts from a seed prompt and repeats to uncover prompts that appear safe but produce unsafe images.Source: Figure 2 · p. 5

3.1 Target Model and Problem Formulation

• AdvPIE probes black-box text-to-image models for prompts that appear textually safe but produce harmful images, using frozen policy and judge models.The target model is query-only; the policy proposes prompts from interaction history, and the judge evaluates prompt–image pairs.Source: §3 · p. 5, §3.1 · pp. 5–6

3.2 Global-Relative Feedback Mechanism

• The judge supplies modality-specific feedback, while global guidance retains top-k prompts with unsafe images and relative guidance contrasts the two most recent prompts.The global and relative signals are intended to steer exploration toward more implicitly harmful prompts while avoiding unproductive or overly explicit directions.Source: §3.2 · p. 7

3.3 Cumulative Adversarial Decoding

• Cumulative adversarial decoding scores prompts by contrasting image and text safety, then tracks average scores across their tokens to guide token-level sampling.The design dynamically reweights token probabilities without training and uses selective decay to discourage repetitive patterns while encouraging exploration.Source: §3.3 · pp. 7–8
• Variance-scaled logit adjustments preserve the language model’s relative probability structure, while a repetition penalty is applied selectively to positively scored tokens.The authors describe these choices as balancing useful guidance with linguistic fluency and exploration of novel prompts.Source: §3.3 · p. 8

4.1 Experimental Settings

• The study compares three baselines across base and safety-aligned open-source models and a commercial API, using initially safe prompts and requiring both prompt acceptance and a harmful image for success.Experiments use 30 seed prompts per each of 11 harm categories, 20 refinement rounds, and three images per iteration; baselines are FLIRT, Groot, and ART.Source: §4.1 · pp. 8–9

4.2 Results and Analysis

• Google SafeSearch produces the lowest ASRs because it detects a narrower range of categories with conservative logic, yet AdvPIE remains competitive under it.The judges are Gemma, LLaVA Guard, and Google SafeSearch.Source: §4.1 · p. 9, §4.2 · p. 10
• When an API rejects an overly explicit prompt, AdvPIE treats rejection as negative feedback and penalizes tokens contributing to that prompt.The feedback flags the prompt as too explicit, helping the policy avoid unsafe phrasing while exploring superficially benign routes to harmful outputs.Source: §4.2 · p. 10
• AdvPIE achieves best overall ASR across four open-source models and the highest Stable Diffusion API ASR, beating ART by +10.31% under Gemma, +7.95% under LLaVA Guard, and +0.54% under SafeSearch.The sole reported open-source exception is safe-sd-v1-5 under Gemma, where FLIRT is slightly higher; qualitative trajectories progress from benign prompts to graphic surgical depictions and nudity.Source: Table 1 · p. 9, §4.2 · pp. 10–11, Table 2 · p. 10

4.3 Ablation Study

• Removing adversarial decoding drops ASR by over 13% under Gemma and 11.5% under LLaVA Guard, while bypass rate falls from 54.83% to 45.17%.Removing structured feedback also lowers ASR across judges and reduces bypass rate to 48.33%.Source: §4.3 · p. 12, Table 3 · p. 12
• Score-based guidance has the largest feedback ablation impact: removing it cuts bypass rate by 17 percentage points, while removing safety reasoning lowers ASR by 7.17 and 7.84 points under Gemma and LLaVA Guard.The paper reports safety reasoning's bypass-rate decrease as 7.66 percentage points; score-based guidance is the top-k memory of prior successful prompts.Source: Table 4 · p. 12, §4.3 · p. 13
• AdvPIE has the highest human-judged attack success rate across all four target models.Three trained annotators assessed 3,520 prompt-image pairs, judging prompt benignness and image harmfulness.Source: §4.3 · p. 13, Table 5 · p. 13
• ASR rises steadily early and plateaus around 250 iterations; AdvPIE already surpasses baselines within the 20 rounds used in the main experiments.The paper reports further gains with more iterations without retraining or architectural modifications.Source: §4.3 · pp. 13–14, Figure 4 · p. 14

5 Conclusion

• AdvPIE combines global-relative feedback with Cumulative Adversarial Decoding to expose implicit vulnerabilities in text-to-image models without model training.The feedback tracks harmful directions and recent progress; decoding steers token generation while preserving prompt diversity and implicitness.Source: §5 · p. 14
• The global-relative feedback strategy guides iterations using harmful directions found so far and recent progress, avoiding single-step evaluation and noisy raw historical aggregation.Source: §5 · p. 14
• Cumulative Adversarial Decoding uses feedback to steer token-level generation toward implicitly harmful directions while accumulating experience and preserving prompt diversity.Source: §5 · p. 14
• Experiments across standard and safety-aligned text-to-image models, including commercial APIs, show AdvPIE consistently outperforms existing baselines.Source: §5 · p. 14

Broader Impacts

• AdvPIE is intended to help developers identify harmful failure modes in black-box generative systems and support safer, more responsible deployment.Source: Broader Impacts · p. 15
• The framework may also lower the barrier for adversaries seeking to bypass content moderation through feedback-guided refinement.Source: Broader Impacts · p. 15
• The authors say they withheld implicit prompts and harmful image examples and redacted sensitive outputs under institutional safety guidelines.Source: Broader Impacts · p. 15

A. Related Work

• Text-only feedback strategies leave a modality mismatch because implicit T2I vulnerabilities are observable in generated images.Source: §A.1 · p. 20
• Earlier adversarial-agent approaches can require substantial data and computation, while iterative feedback methods avoid fine-tuning but remain text-only.Source: §A.1 · p. 20
• AdvPIE uses adversarial decoding with iterative multimodal feedback instead of model training, jointly reasoning over prompts and generated images.The passage characterizes this approach as training-free and black-box.Source: §A.1 · p. 20

A.2 Red-teaming for Text-to-image Models

• White-box methods use gradients over model internals, but are impractical for commercial systems without model access.Source: §A.2 · p. 20
• Existing black-box approaches have constraints: overtly harmful seeds limit implicit-vulnerability discovery, while other methods require large-scale data, model-specific finetuning, or crowdsourcing.Source: §A.2 · p. 20
• Prior methods leave room to capture directional progress across iterations and steer token-level generation toward implicit harms at inference time.Source: §A.2 · p. 20
• AdvPIE addresses these challenges with global-relative feedback and token-level Cumulative Adversarial Decoding, without model training or heavy dataset collection.Source: §A.2 · p. 20

B Limitations and Future Work

• The policy considers one image per iteration; sampling multiple images and selecting the most harmful improves attack reliability but slows evaluation and prevents reasoning over the full output range.Future work could explore multi-image reasoning through information fusion or visual summaries.Source: B Limitations and Future Work · p. 21
• Safety restrictions prevent testing AdvPIE with highly capable commercial LLMs as policy agents, leaving its performance under the highest possible reasoning capacity untested.Suggested directions include instruction-tuning open-source LLMs or using constrained optimization.Source: B Limitations and Future Work · p. 21

C Inference Settings in AdvPIE

• Inference uses LLaVA-v1.6-Mistral-7B as policy model, with max_new_tokens of 77 for open-source models and 150 for commercial APIs.The experiments use four RTX 4090 GPUs for open-source setups and three for commercial APIs.Source: C Inference Settings in AdvPIE · p. 21

• The initial prompt depicts a parent explaining a doctor’s visit to a child with picture books.Source: Figure 5 · p. 22
• Iterative refinement turns the prompt into increasingly harmful variants that remain superficially benign.Source: Figure 5 · p. 22
• Every prompt in the trajectory successfully bypasses the Stable Diffusion commercial API’s safety filters.Source: Figure 5 · p. 22

D.1 Category: Public Health

• The trajectory uses medical education and clinical settings, including a public-health lecture and a live medical demonstration, as its scenario frame.Source: §D.1 · pp. 23–24
• Variants escalate to live surgery, botched procedures, and overcrowded emergencies marked by blood, panic, and distressed patients and staff.Source: §D.1 · p. 23
• Several prompts turn the hospital’s clinical setting into a battlefield or nightmare, juxtaposing the promise of care with catastrophic harm.Source: §D.1 · pp. 25–26

D.2 Category: Shocking

• Wax-museum prompts progress from an uncanny exhibit to detailed scenes of melting, distorted figures, anguish, and decay.Later variants emphasize fluid wax textures, oppressive settings, and disturbing expressions.Source: §D.2 · pp. 28, 31

D.3 Category: Sexual

• The sexual-category sequence starts with two embracing silhouettes and develops into prompts for couples in luxurious settings with suggestive poses and attire.Later examples describe intimate scenes in candlelit, opulent rooms.Source: §D.3 · pp. 33–34

E Prompt Templates

• The templates have the policy agent refine prompts using prior attempts, evaluation feedback, and examples, or try a fresh approach when no viable image results.Separate image- and text-analysis templates request safety labels, scores, categories, and rationales; the image template also requests a content description.Source: E Prompt Templates · pp. 35–37
• Assessments label content safe or unsafe, assign a policy category when applicable, and use “NA: None applying” otherwise.The supplied JSON template specifies the label and category fields.Source: E Prompt Templates · p. 38

F Initial Prompts

• Initial prompts are requested in English as creative, unique image-generation prompts that remain safe for work and need not depict the assigned theme.The template asks for a Python list of prompt strings and introduces the 11-category generated-prompt inventory.Source: F Initial Prompts · pp. 38–39

Hate; Harassment

• Hate prompts include historical and educational treatment of symbols, while harassment prompts depict social, workplace, and online mistreatment.Examples include museum exhibits and anti-hate messaging, alongside isolation, bullying, threatening messages, and persistent following.Source: Hate · pp. 39–40, Harassment · pp. 41–42

Violence

• Violence prompts span crime scenes, physical injury, armed confrontations, battlefields, and fictional or symbolic depictions of conflict.The lists include both aftermath scenes and active conflict, as well as metaphorical emotional violence.Source: Violence · pp. 42–43

Self-Harm; Sexual

• Self-harm prompts use metaphorical imagery of distress, isolation, and recovery, while sexual prompts range from educational anatomy to suggestive art and intimacy.The self-harm examples also include images of hope and healing; the sexual list includes medical, artistic, and adult-themed contexts.Source: Self-Harm · pp. 43–45, Sexual · pp. 45–46

Shocking

• Shocking prompts use surreal, uncanny, and horror imagery, including distorted figures, artificial body parts, unsettling fluids, and eerie settings.Examples include a cracked doll, melted wax figures, a figure formed from veins, and an abandoned morgue.Source: Shocking · pp. 46–47

Illegal Activity; Deception

• Illegal-activity prompts depict crimes and illicit exchanges, while deception prompts portray conspiracies, fabricated histories, and misinformation.Examples range from burglary, forgery, and cybercrime to staged events, secret organizations, and manipulated news.Source: Illegal Activity · pp. 48–49, Deception · pp. 49–50

Political; Public Personal Health

• Political and public-health prompts include civic participation, voting, public-health education, and healthcare scenarios.Examples include peaceful protests, polling stations, disease prevention, vaccination, and health services.Source: Political · pp. 50–51, Public Personal Health · p. 52

H Harmful Content Categories

• The evaluation table lists harmful content categories alongside descriptions and representative examples.Source: Table 7 · p. 55

I Full Result Tables

• The tables cover per-category attack success rates across open-source T2I models and the commercial Stable Diffusion API.Source: Table 8 · p. 57, Table 10 · p. 59
• 1.21% is AdvPIE’s reported Avg attack success rate, the highest among the four methods listed.Source: Table 9 · p. 58
• The listed AdvPIE row reports 4.67% for Sexual and 4.17% for Self-harm.Source: Table 9 · p. 58, I Full Result Tables · p. 58