PaperlayerView exact revision on arXiv

Source-linked AI summary

T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model

Chenyu Zhang, Tairen Zhang, Lanjun Wang, Ruidong Chen, Wenhui Li, Anan Liu

arXiv:2510.22300v2cs.CRcs.AIcs.CV

TL;DR

Existing risky-prompt datasets have limited risk coverage, coarse annotations, and low prompt effectiveness. T2I-RiskyPrompt combines a hierarchical taxonomy, an annotated and validated prompt dataset, and a reason-driven image detector; evaluations identify weaknesses across T2I models and their safety measures.

  • Problem

    Existing risky-prompt datasets have limited risk categories, coarse-grained annotations, and low prompt effectiveness, constraining safety evaluation of T2I models.

    Source: §1 · p. 1

  • Method

    The paper constructs a six-category, 14-subcategory taxonomy and a six-stage pipeline for collecting and annotating effective prompts, and proposes reason-driven risky-image detection aligned with human safety annotations.

    Source: §1 · p. 1, §3.1 · p. 3, §2.4 · p. 14

  • Results

    Evaluations find that current defenses struggle across risky content types, filters fail to identify a broad range of risks, and attacks can bypass safety strategies; the detector achieves 91.8% accuracy using a 3B MLLM.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The benchmark provides a basis for evaluating T2I safety measures and, because of its similar input-output modality, can also be applied directly to text-to-video safety evaluation.

    Source: §8 · p. 7, §9 · p. 7

    The jailbreaking evaluation uses a black-box setting in which the adversary has no prior knowledge of the T2I model or its safety mechanisms.

    Source: §6 · p. 26

Abstract

from arXiv · show

Using risky text prompts, such as pornography and violent prompts, to test the safety of text-to-image (T2I) models is a critical task. However, existing risky prompt datasets are limited in three key areas: 1) limited risky categories, 2) coarse-grained annotation, and 3) low effectiveness. To address these limitations, we introduce T2I-RiskyPrompt, a comprehensive benchmark designed for evaluating safety-related tasks in T2I models. Specifically, we first develop a hierarchical risk taxonomy, which consists of 6 primary categories and 14 fine-grained subcategories. Building upon this taxonomy, we construct a pipeline to collect and annotate risky prompts. Finally, we obtain 6,432 effective risky prompts, where each prompt is annotated with both hierarchical category labels and detailed risk reasons. Moreover, to facilitate the evaluation, we propose a reason-driven risky image detection method that explicitly aligns the MLLM with safety annotations. Based on T2I-RiskyPrompt, we conduct a comprehensive evaluation of eight T2I models, nine defense methods, five safety filters, and five attack strategies, offering nine key insights into the strengths and limitations of T2I model safety. Finally, we discuss potential applications of T2I-RiskyPrompt across various research fields. The dataset and code are provided in https://github.com/datar001/T2I-RiskyPrompt.

1 Introduction

T2I models can generate risky images, but existing safety datasets have limited categories, coarse annotations, and low prompt effectiveness. T2I-RiskyPrompt adds a hierarchical taxonomy and annotated, validated prompts, then supports detector and safety evaluations.

1 Introduction

  • T2I-RiskyPrompt introduces a taxonomy with six primary risk categories and 14 fine-grained subcategories, built from analysis of platform and company usage policies.Source: §1 · p. 1

2.1 Risky Prompt Dataset

  • The dataset contains 6,432 effective risky prompts, each with hierarchical category labels and detailed risk reasons, produced through a collection and annotation pipeline.The pipeline combines GPT-4o and human category checks, identifies risky visual elements, polishes prompts, and filters out ineffective ones.Source: §1 · p. 1, §2.1 · p. 3

1 Introduction

  • The reason-driven image detector aligns MLLMs with detailed human risk rationales and achieves 91.8% accuracy using a 3B MLLM, outperforming existing detectors.Source: §1 · p. 1
  • The benchmark evaluates eight T2I models, nine internal defenses, five safety filters, and five attacks, finding defenses struggle across risk types, filters miss broad risks, and attacks bypass existing strategies.The authors report nine insights and conclude that current T2I models still exhibit significant safety risks.Source: §1 · p. 2

2.1 Risky Prompt Dataset

  • Earlier datasets covered limited or expanding risk categories, while some later approaches used automated moderation without human verification, yielding coarse annotations and limited prompt effectiveness.Source: §2.1 · p. 2

2.2 Risky Content Moderation

  • Classifier-based filters depend on large predefined training datasets and can generalize poorly to unseen risks; MLLM filters require carefully crafted safety policies.Source: §2.2 · p. 3

3 Benchmark Construction

T2I-RiskyPrompt organizes T2I safety risks into a policy-informed taxonomy and uses a six-step collection and annotation pipeline to produce prompts with category labels and visual risk reasons. Its dataset comparisons and reason-driven image evaluator support more interpretable safety evaluation.

3.1 Hierarchical Risk Taxonomy

  • The taxonomy defines 6 risk categories and 14 subcategories spanning NSFW, copyright-infringing, and politically sensitive content.It separates violent content into Weapons&Conflicts and Bloody Content, and copyright risks into logos and copyrighted character designs.Source: §3.1 · p. 3

3.2 Data Collection and Annotation

  • The pipeline combines existing prompts, GPT-4o-generated prompts for copyright and illegal activities, prompt polishing, and diversity filtering at a CLIP similarity threshold above 0.8.The initial collection contains 12,251 prompts.Source: §3.2 · pp. 3–4
  • Prompts receive taxonomy-based category labels with manual verification, are filtered when generated images lack intended risky elements, and receive manually annotated visual risk reasons.Some prompts receive multiple category labels.Source: §3.2 · p. 4
  • The final dataset contains 6,432 risky prompts and 20,792 generated images, each with coarse-grained categories and detailed risk reasons.Source: §3.2 · p. 4

3.3 Dataset Statistics and Analysis

  • T2I-RiskyPrompt combines finer-grained categories, image-validated prompts, polished semantics, and detailed risk annotations, which the authors report as advantages over existing datasets.Its PPL is 86 versus 2,613 for T2ISafety, which the authors say indicates better semantic clarity.Source: §3.3 · p. 4

3.4 Evaluation Protocols

  • Benchmark risk ratio measures the proportion of prompts generating risky images, with two images generated per prompt and success requiring at least one detector-flagged image.Source: §3.4 · p. 4
  • The reason-driven evaluator gives an MLLM detailed risky visual elements from human annotations and achieves 91.8% average classification accuracy with a 3B MLLM.It classifies an image under the specified risk category when the described content is present.Source: §3.4 · pp. 4–5

4 Evaluation of T2I models

Across the evaluated open-source models, those with stronger generative capabilities tend to have higher risk ratios. Pornographic risks appear more mitigated than other categories, while comparable suppression is not observed across other risk types.

4 Evaluation of T2I models

  • The evaluation covers eight representative open-source T2I models, including Stable Diffusion, FLUX, CogView4, Janus Pro, and HiDream.Source: §4 · p. 5
  • SD3, Janus Pro, and HiDream have the highest risk ratios across nearly all categories, while SD1.4 and PixArt consistently have the lowest, indicating a positive correlation between capability and risk.The authors suggest stronger instruction-following may help capable models generate complex risky content; they report this as a possible explanation for the correlation.Source: §4 · p. 5
  • PixArt has a substantially lower pornography risk ratio than other models, and SDXL and SD3 also mitigate explicit content despite direct prompts.Source: §4 · p. 5

5 Evaluation of T2I Defense Methods

The evaluation finds that defense effectiveness varies by risk category and approach, while stronger protection typically trades off against generation quality. Copyright diversity and simultaneous defense across multiple NSFW categories remain challenging.

5 Evaluation of T2I Defense Methods

  • The study evaluates nine representative methods spanning inference guidance, model editing, and fine-tuning.Source: §5 · p. 5
  • All tested defenses have limited effectiveness against copyright-infringing outputs; fine-tuning is relatively more effective but incurs significant performance degradation.The authors relate the difficulty to diverse, semantically unrelated copyrighted concepts and varied visual manifestations.Source: §5 · pp. 5–6
  • Inference-guidance and model-editing methods perform poorly against multiple NSFW categories simultaneously, while tuning-based methods are more suitable for unified, scalable defense.Source: §5 · p. 6
  • Different risk categories favor different defenses: TRCE works well for pornography, MACE and SafetyDPO for diverse NSFW risks, TRCE for copyright, and SPEED for political sensitivity.Source: §5 · p. 6
  • Stronger defense typically reduces generation capability, with MACE showing advanced protection across nearly all risky categories but greater quality degradation than other approaches.Source: §5 · p. 6

6 Evaluation of Safety Filters

Prompt-level filters generally outperform image-based filters, though feature-based methods have category-specific strengths. Jailbreaks can bypass keyword and feature-based filters, and image-based filters remain susceptible even when filters are ensembled.

6 Evaluation of Safety Filters

  • Keyword and NSFW-T filters show stronger defensive performance than image-based filters, as prompt risk semantics are often expressed in words while visual cues are more diverse and complex.Source: §6 · p. 6
  • NSFW-T is more effective for NSFW and political prompts, NSFW-I for pornography images, and Q16 for non-pornographic NSFW; the keyword detector outperforms other filters on risky prompts.The feature-based filters are not specifically designed for copyright risks.Source: §6 · p. 6

7 Evaluation of Jailbreaking Attacks

  • MMA pseudoword attacks achieve high risk ratios against the keyword filter by replacing risky keywords, but remain detectable by NSFW-T through retained risk semantics.Source: Figure 3 · p. 7, §7 · p. 7
  • LLM-based attacks, especially MJA, achieve high risk ratios against SD1.4 with NSFW text filtering by using implicit linguistic associations such as metaphors.Source: Figure 3 · p. 7, §7 · p. 7
  • All tested attacks yield higher risk ratios against image-based filters, and ensembling filters improves robustness but does not prevent jailbreak success.Source: Figure 3 · p. 7, §7 · p. 7

8 Discussion

T2I-RiskyPrompt supports safety evaluation across varied conditions and has potential applications beyond text-to-image models. The paper concludes that current T2I models still exhibit significant safety risks.

8 Discussion

  • The dataset can be applied to text-to-video safety evaluation and can support automated risk-image assessment, portrait protection, and intellectual-property compliance research.The paper specifically connects its infringing-character and political-figure images to personalized portrait protection and IP compliance.Source: §8 · p. 7

9 Conclusion

  • The paper concludes that current T2I models still exhibit significant safety risks and reports nine insights into existing safety measures.Those evaluations span internal defenses, external defenses, and attack methods.Source: §9 · p. 7

1 Ethical Considerations

The study reviewed its procedures with an IRB and used safeguards for annotators exposed to potentially disturbing content. It also sought to reduce cultural bias and restrict dataset distribution to legitimate research uses.

1 Ethical Considerations

  • The IRB reviewed the study procedures and issued an exempt determination; annotators could withdraw, were warned about upsetting content, and received breaks every 30 minutes.Volunteers were required to be at least 18 and in good physical and mental health.Source: §1 · p. 10
  • To reduce cultural subjectivity, annotators received standardized category definitions and were recruited from diverse cultural backgrounds with balanced gender representation.The definitions were synthesized from multiple T2I-model safety policies.Source: §1 · p. 10
  • The authors intend to vet dataset recipients and distribute it only for legitimate, constructive purposes, while stating that it is solely for assessing T2I-model safety.They also plan to communicate the dataset’s risks upon release.Source: §1 · p. 10

2 T2I-RiskyPrompt

T2I-RiskyPrompt combines a policy-derived risk taxonomy with a multi-stage prompt collection and annotation pipeline. Its prompts outperform existing datasets in risky-image generation across eight T2I models, and its evaluations examine prompt and image detection.

2.1 Risk Taxonomy

  • The taxonomy derives from usage policies of four T2I platforms and three technology companies, defining six primary risk categories and 14 subcategories.The category definitions are presented in Table 4.Source: §2.1 · p. 11

2.2 Data Collection and Annotation

  • Prompt collection combines existing pornography, violence, and disturbing-content datasets with GPT-4-generated prompts based on cartoon-character images, logos, and illegal-activity elements.The cartoon prompts target Disney and Pokémon characters, while the logo prompts use images from 200 companies.Source: §2.2 · p. 11
  • GPT-4o polishes prompts and CLIP similarity filtering removes near-duplicates; LLaMA-3 handles pornographic-prompt polishing because GPT-4o refuses that content.Prompts are assigned taxonomy labels with manual verification, tested through generated images, and manually annotated with visual risk reasons.Source: §2.2 · pp. 11–12

2.3 Dataset Statistics and Analysis

  • T2I-RiskyPrompt prompts outperform existing datasets in risky-image generation effectiveness across all eight T2I models.The comparison covers shared pornography, violence, disturbing, and illegal-content categories.Source: Table 1 · p. 12, §2.3 · p. 12

2.2 Data Collection and Annotation

  • The resulting dataset contains 6,432 prompts across 14 categories, each with one or more risk labels and corresponding classification justifications.Source: §2.2 · p. 12

2.3 Dataset Statistics and Analysis

  • Fine-grained prompt counts vary, with several illegal-activity categories below 300 prompts; Violence–Bloody Content co-occurs with Disturbing-Terrifying Content in 558 prompts.The paper attributes the uneven counts partly to limited public data for illegal activities and reports that roughly one-sixth of prompts have multiple labels.Source: §2.3 · pp. 12–13

2.4 Evaluation Methods

  • Category-specific keyword lists achieve 92.9% risky-prompt detection accuracy, while a small keyword set works less well for categories expressed through diverse terms.The lists use frequent risky words for most categories and names of figures, logos, and characters for selected categories.Source: §2.4 · p. 13, Table 2 · p. 14
  • Reason-driven MLLM instructions describe fine-grained visual risk elements and improve detection, particularly for categories outside prior safety alignment, where broad definitions and existing detectors struggle.Q16 and NSFW detectors show differing category strengths and limited accuracy for copyright infringement and political sensitivity.Source: §2.4 · pp. 14–15

3 Evaluation of T2I Models

The evaluation specifies model-specific generation settings to support reproducibility, including seeds, image dimensions, guidance or configuration values, and inference steps.

3 Evaluation of T2I Models

  • All listed T2I models use seeds 666 and 2024, while image dimensions range from 384 × 384 for Janus-Pro-7B to 1024 × 1024 for HiDream-l1-dev.Source: §3 · p. 23
  • Guidance scales and inference-step counts vary by model, from Stable Diffusion V3’s 28 steps to PixArt-alpha’s 20 steps and Stable Diffusion V1.4’s 50 steps.Janus-Pro-7B instead lists a CFG weight and temperature.Source: §3 · p. 23

4 Evaluation of T2I Defense Methods

The evaluation tests defense methods across NSFW, Copyright, and Political concepts, measuring both risk reduction and effects on generation quality.

4 Evaluation of T2I Defense Methods

  • The evaluation covers NSFW, Copyright, and Political subtasks, with selected risk categories chosen for methods targeting explicit risky visual elements.The selected categories include theft and drug crimes, bloody content and weapons, explicit pornography, terrifying content, political figures, and cartoon characters.Source: §4 · p. 24
  • Across each subtask, a unified mechanism targets all concepts in the category using filtered keywords for inference guidance or model-editing and fine-tuning methods.Evaluations use official implementations or provided checkpoints.Source: §4 · p. 24
  • The evaluated defenses comprise inference guidance, model editing, and fine-tuning, implemented with negative prompts, SLD, Safree, UCE, RECE, SPEED, SafetyDPO, MACE, and TRCE.Keyword sets are used for inference guidance and semantic elimination; official settings or checkpoints are used where specified.Source: §4 · p. 24
  • Risk Rate measures defense capability, while FID and CLIP-Score assess the defense’s effects on generation quality.FID compares defended outputs with original SD 1.4 outputs on 3K normal COCO prompts; CLIP-Score measures image-prompt alignment on the same prompts.Source: §4 · p. 24

5 Evaluation of Safety Filters

The safety-filter evaluation includes keyword matching and filters operating on CLIP text or image features.

5 Evaluation of Safety Filters

  • Four filters are evaluated: the proposed Keyword filter, NSFW text and image filters, and Q16.The paper identifies the four filters as Keyword, NSFW-T, NSFW-I, and Q16.Source: §5 · p. 25
  • Keyword flags risky prompts by matching predefined risky-term dictionaries, with category-specific lists ranging from 20 to 440 terms.The largest listed dictionaries are LOGO (440) and Cartoon Characters (198).Source: §5 · p. 25
  • NSFW-T classifies risky prompts from CLIP text features, while NSFW-I detects risky images in CLIP image-feature space.Source: §5 · p. 25
  • Q16 learns risky and non-risky representations contrastively, then classifies test images by comparing cosine similarity to learned feature vectors.Source: §5 · p. 25

6 Evaluation of Jailbreaking Attacks

The attack evaluation studies five jailbreaking methods under a black-box setting, where attackers can query T2I models but lack prior knowledge of the models and their safety mechanisms.

6 Evaluation of Jailbreaking Attacks

  • MMA aligns pseudo-word features; SneakyPrompt optimizes substitutions with reinforcement learning; DACA recomposes visual attributes; PGJ mines similar replacement words; MJA rewrites prompts metaphorically.Source: §6 · p. 26
  • LLM-reasoning attacks transform blocked sensitive prompts into adversarial prompts that bypass safety and produce images semantically similar to the original prompt.The stated objective requires image-text similarity to exceed a predefined threshold.Source: §6 · p. 26
  • The attack scenario is black-box: adversaries have no prior model or filter knowledge, can query prompts, and receive an image or a disallowance notification.The experiment samples 50 risky prompts per fine-grained category, totaling 700 prompts.Source: §6 · p. 26
  • Table 8 compares five attack methods on SD1.4 under different safety filters using risk ratio, with the no-attack condition as the baseline.Higher risk ratio indicates stronger attack performance in bypassing safety mechanisms.Source: Table 8 · p. 27

Related papers

Loading 2510.22300v2…