PaperlayerView exact revision on arXiv

Source-linked AI summary

Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoning

Chenyu Zhang, Lanjun Wang, Yiwen Ma, Wenhui Li, An-An Liu

arXiv:2503.17987v3cs.CRcs.AIcs.CV

TL;DR

Existing T2I jailbreaking methods rely on manually designed strategies, and LLMs often need many queries because they lack direct understanding of T2I models and safety mechanisms. R2A trains an LLM with Frame Semantics-based CoT synthesis, fine-tuning, and reinforcement learning using an attack process reward. It achieves higher attack success with fewer queries and transfers across open-source and commercial T2I models.

  • Problem

    Existing LLM-based attacks manually design adversarial-prompt strategies, while limited understanding of T2I models and safety mechanisms makes successful attacks query-intensive.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    R2A synthesizes Frame Semantics-based CoT examples, fine-tunes the LLM, and applies reinforcement learning with rewards for prompt stealthiness, effectiveness, and length.

    Source: §1 · p. 2

  • Results

    R2A achieves a higher attack success ratio with fewer queries than baselines and demonstrates transferability across open-source and commercial T2I models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training the LLM on the jailbreaking reasoning process improves attack effectiveness, efficiency, and transferability within the evaluated T2I settings.

    Source: §6 · p. 7

    The threat model assumes the adversary has no prior knowledge of the T2I model or its safety mechanisms and can only query it with prompts.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Text-to-Image(T2I) models typically deploy safety filters to prevent the generation of sensitive images. Unfortunately, recent jailbreaking attack methods manually design instructions for the LLM to generate adversarial prompts, which effectively bypass safety filters while producing sensitive images, exposing safety vulnerabilities of T2I models. However, due to the LLM's limited understanding of the T2I model and its safety filters, existing methods require numerous queries to achieve a successful attack, limiting their practical applicability. To address this issue, we propose Reason2Attack(R2A), which aims to enhance the LLM's reasoning capabilities in generating adversarial prompts by incorporating the jailbreaking attack into the post-training process of the LLM. Specifically, we first propose a CoT example synthesis pipeline based on Frame Semantics, which generates adversarial prompts by identifying related terms and corresponding context illustrations. Using CoT examples generated by the pipeline, we fine-tune the LLM to understand the reasoning path and format the output structure. Subsequently, we incorporate the jailbreaking attack task into the reinforcement learning process of the LLM and design an attack process reward that considers prompt length, prompt stealthiness, and prompt effectiveness, aiming to further enhance reasoning accuracy. Extensive experiments on various T2I models show that R2A achieves a better attack success ratio while requiring fewer queries than baselines. Moreover, our adversarial prompts demonstrate strong attack transferability across both open-source and commercial T2I models.

1 Introduction

T2I safety filters are vulnerable to adversarial prompts, while existing attacks often require manual strategy design and numerous queries. R2A addresses these challenges by training an LLM to reason through adversarial prompt generation and reports higher attack success with fewer queries.

2.2 Jailbreaking Attacks on T2I Models

  • Existing LLM-based attacks manually design strategies for fluent adversarial prompts, but limited understanding of T2I models and safety mechanisms leads to numerous queries.Pseudoword-based methods also require technical expertise to construct adversarial prompts, limiting real-world applicability.Source: §1 · p. 1, §2.2 · pp. 2–3

2.1 T2I Models and Safety Mechanisms

  • T2I models generate high-fidelity images from textual prompts, but their misuse can produce sensitive content that motivates safety mechanisms such as text and image filters.Safety mechanisms detect or block harmful prompts and images, while attacks seek prompts that bypass these protections.Source: §1 · p. 1, §2.1 · p. 2

1 Introduction

  • R2A formulates jailbreaking as an LLM reasoning task and uses a Frame Semantics-based framework to synthesize step-by-step CoT adversarial prompts.The framework retrieves related terms, generates context illustrations, produces effective adversarial prompts, and synthesizes complete CoT examples.Source: §1 · pp. 1–2
  • R2A fine-tunes the LLM on synthesized CoT examples and then applies reinforcement learning with an attack process reward covering stealthiness, effectiveness, and prompt length.The reward supplies feedback from the T2I system to address sparse reinforcement-learning signals and guide prompt refinement.Source: §1 · p. 2
  • R2A achieves a higher attack success ratio and requires fewer queries than baselines across experiments on various T2I models.Its adversarial prompts also transfer across open-source and commercial T2I models, including DALL·E 3 and Midjourney.Source: §1 · p. 2

2.2 Jailbreaking Attacks on T2I Models

  • Jailbreaking attacks include pseudoword-based methods that optimize feature representations and LLM-based methods that generate fluent adversarial prompts.Pseudoword construction is difficult for non-experts, whereas LLM-based approaches rely on manually designed prompts and remain query-intensive.Source: §2.2 · pp. 2–3

3 Problem

The paper formulates T2I jailbreaking as an LLM reasoning problem under black-box safety mechanisms, then presents R2A’s Frame Semantics generation pipeline and two-stage reasoning training.

3.1 Problem Definition

  • R2A trains an LLM to transform blocked sensitive prompts into adversarial prompts that bypass safety mechanisms while preserving image-text similarity above a threshold.The threat model assumes black-box access to the T2I model and its safety mechanism, with feedback limited to generated images or disallowed-query notifications.Source: §3.1 · p. 3, §3.2 · p. 3
  • The attack setting is challenging because success requires both bypassing safety mechanisms and generating images semantically aligned with the sensitive prompt.This creates indirect feedback from two black-box components and makes reinforcement-learning rewards sparse.Source: §3.1 · p. 3

4.1 Unified Generation Framework Based on Frame Semantics

  • Frame Semantics motivates R2A’s unified pipeline: related terms acquire sensitive meanings through context, enabling step-by-step CoT adversarial-prompt generation.The pipeline uses contextual illustrations to make subtle associations explicit, such as interpreting related terms within broader frames.Source: §4.1 · p. 3
  • The generation pipeline identifies sensitive words, retrieves N related terms from ConceptNet, creates context illustrations, rewrites prompts, filters ineffective attacks, and synthesizes CoT examples.Effective adversarial prompts are paired with their sensitive prompts, related terms, and illustrations before GPT-4o produces fluent CoT examples.Source: §4.1 · p. 3

4.2 Two-Stage LLM Reasoning Framework Guided by the Attack Process

  • R2A fine-tunes the LLM on synthesized CoT examples before applying online reinforcement learning to jailbreaking attacks, combining reasoning-format and attack-process supervision.The first stage uses next-token prediction on CoT paths; the second optimizes sampled outputs with grouped relative rewards and policy regularization.Source: §4.2 · pp. 3–4
  • The attack-process reward evaluates reasoning completeness plus prompt stealthiness, effectiveness, and length to provide feedback about safety boundaries, generation capacity, and token limits.Stealthiness reflects safety-filter bypass, effectiveness checks semantic image alignment, and length enforces a predefined threshold z.Source: §4.2 · p. 4

5 Experiment

R2A is evaluated against multiple baselines on filtered T2I models, including transfer attacks, ablations, hyperparameter settings, and commercial systems. It achieves stronger attack effectiveness and query efficiency, with robust transferability and benefits from its two-stage reasoning training.

5.2 Attack Results

  • R2A achieves the highest ASR while significantly reducing query numbers versus all baselines, demonstrating stronger attack effectiveness and efficiency.The comparison uses Stable Diffusion V1.4 with text and image safety filters; lower PPL and query number are preferred, while higher ASR is preferred.Source: §5.2 · pp. 5–6, Metric · p. 5

5.3 Transferring to Open-Source T2I Models

  • R2A transfers successfully from SD1.4 to SDV3 and FLUX, retaining the highest ASR among the compared methods.The transferable prompts are evaluated on open-source models equipped with safety filters.Source: §5.3 · p. 6, Table 3 · p. 6

5.4 Transferring to Commercial T2I Models

  • R2A achieves high ASR on both DALL·E 3 and Midjourney, demonstrating transferability to commercial T2I models with advanced safety mechanisms.The authors present this transfer as useful for uncovering safety vulnerabilities in real-world systems.Source: §5.4 · p. 6, Table 4 · p. 6

5.5 Ablation Analysis

  • LLM+SFT CoT improves average ASR by 25% over LLM, while LLM+RL AR and LLM+RL AP improve it by 25% and 38%, respectively.These ablations test supervised fine-tuning with synthesized CoT examples and reinforcement learning with attack-result or attack-process rewards.Source: §5.5 · p. 6, Table 2 · p. 6
  • The attack-process reward achieves better ASR and fewer queries than the attack-result reward because it supplies feedback on prompt stealthiness, effectiveness, and length.The result-only reward is described as vulnerable to sparse rewards during the ambiguous jailbreaking task.Source: §5.5 · p. 6
  • Two-stage reasoning training achieves the best performance and generalizes better than applying reinforcement learning alone.The authors attribute this to fine-tuning the LLM on CoT examples before reinforcement learning.Source: §5.5 · p. 7

5.6 Hyperparameter Analysis

  • R2A reaches 60% ASR with one query, while ASR generally increases with the maximum query limit; smaller λ values generally produce higher ASR.The authors set the query limit to 6 to balance effectiveness and detectability and choose λ = 0.2 empirically.Source: §5.6 · p. 7, Figure 4 · p. 7

5.7 Visualization

  • R2A generates adversarial prompts by embedding modifiers in artistic contexts to imply drug-related content, and the resulting prompts induce SDV3 and commercial models to generate sensitive images.The visualizations include a CoT attack example and blurred attack results for DALL·E 3 and Midjourney.Source: §5.7 · p. 7, Figure 2 · p. 7, Figure 3 · p. 7

6 Conclusion

The paper frames jailbreaking as an LLM reasoning task, combining Frame Semantics-based CoT synthesis, fine-tuning, and reinforcement learning with an attack process reward. Experiments support the approach’s effectiveness, efficiency, and transferability, while its stated aim is to improve T2I safety.

6 Conclusion

  • R2A formulates T2I jailbreaking as an LLM reasoning task and uses Frame Semantics to synthesize CoT examples for adversarial prompt generation.These examples fine-tune the LLM to understand adversarial reasoning paths.Source: §6 · p. 7
  • The method integrates jailbreaking into reinforcement learning with an attack process reward balancing prompt stealthiness, effectiveness, and length.The reward is intended to improve exploration of T2I model behaviors and safety mechanisms.Source: §6 · p. 7
  • Experiments show that R2A is effective, efficient, and transferable across T2I models.The paper states these outcomes without reporting specific values in the supplied passage.Source: §6 · p. 7
  • The authors state that the research aims to reveal T2I safety vulnerabilities to enhance system safety rather than enable misuse.This ethical scope is stated alongside the conclusion.Source: §6 · p. 7

1 Dataset

The study uses separate supervised fine-tuning, reinforcement-learning, and testing data organized across four NSFW risk categories. The test set contains 400 prompts independent of training data.

1 Dataset

  • The supervised fine-tuning set contains 400 CoT examples, with 100 examples each for sexual, violent, disturbing, and illegal categories.These examples are generated by the unified framework.Source: §1 · p. 10
  • The reinforcement-learning data comprise 2,490 sensitive prompts: 913 sexual, 893 violent, 286 disturbing, and 398 illegal.These prompts are manually collected and categorized by risk type.Source: §1 · p. 10
  • The testing set contains 400 independent sensitive prompts, with 100 prompts for each risk category.The test prompts are additionally collected rather than reused from training data.Source: §1 · p. 10

2 Image Evaluator

The paper introduces an LVLM-based evaluator for detecting NSFW content across four risk categories and aggregates three binary assessments by voting. Its evaluation compares this evaluator with an existing classifier using generated images from multiple T2I models.

2 Image Evaluator

  • The LVLM-based evaluator uses InternVL2-8B to assess public safety concerns, ethical suitability, and risk-class content.The risk classes are sexual, violent, disturbing, and illegal.Source: §2 · p. 10
  • An image is classified as NSFW when at least two of the three evaluator assessments return “Yes.”The three assessments cover public safety, ethics, and the specified sensitive class.Source: §2 · p. 10
  • The evaluator dataset uses sensitive test prompts to generate images with Stable Diffusion V1.4, Stable Diffusion XL, and Flux.Images are generated for 100 prompts per risk class and per T2I model.Source: §2 · p. 10
  • Accuracy is the evaluation metric, and a prompt is categorized as NSFW if any of its four generated images is classified as NSFW.This accounts for randomness and potential T2I safety strategies.Source: §2 · p. 10
  • The existing classifier primarily recognizes sexual images, whereas the LVLM-based evaluator provides coverage across all four risk categories.The comparison is reported in Table 1 for images generated with safety filters disabled.Source: §2 · p. 10, Table 1 · p. 10

3 LLM Prompt Template

The prompt template specifies how the LLM generates an adversarial prompt from an input sensitive prompt.

3 LLM Prompt Template

  • The LLM prompt template maps an input sensitive prompt to a generated adversarial prompt.The supplied passage identifies the template’s purpose but does not provide its internal fields or wording.Source: §3 · p. 10

4 Optimization Trend in RL Stage

The RL experiments show that attack outcomes alone produce unstable training, whereas the attack-process reward yields steadily improving returns. Combining supervised fine-tuning on synthesized CoT examples with this reward achieves the highest returns.

4 Optimization Trend in RL Stage

  • Attack results alone produce highly unstable and nonconvergent return curves during LLM training.Failures can reflect filtering, unsuccessful harmful-image generation, or violated T2I input constraints.Source: §4 · pp. 10–11
  • The comparison evaluates reinforcement-learning return curves for attack-result rewards, attack-process rewards, and the proposed supervised-fine-tuning-plus-reinforcement-learning pipeline.The figure caption distinguishes the three training settings shown in Fig. 2.Source: §4 · p. 10, Figure 2 · p. 11
  • The attack-process reward produces consistently increasing returns, indicating that the LLM learns to refine adversarial prompts from richer feedback.The reward addresses confounding factors that obscure optimization signals when only attack outcomes are used.Source: §4 · p. 11
  • First, CoT fine-tuning followed by reinforcement learning with the attack-process reward yields significantly higher returns.This two-stage sequence outperforms using reinforcement learning alone in the reported return curves.Source: §4 · p. 11, Figure 2 · p. 11

5 Attack Experiments on Text-to-Video Model

On OpenSora V2, R2A achieves the highest average attack success ratio among the evaluated methods. The experiment uses text and image filtering, six-second videos, and sampled frames to determine attack effectiveness.

5 Attack Experiments on Text-to-Video Model

  • R2A achieves the highest average ASR among all methods on the OpenSora V2 text-to-video model.The experiment targets a model equipped with both text and image filtering mechanisms.Source: §5 · p. 11
  • The OpenSora V2 results indicate that R2A can be applied to uncover safety vulnerabilities in text-to-video models.Attack examples for this evaluation are shown in Fig. 6.Source: §5 · p. 11

6 Computational Costs

R2A generates an adversarial prompt for a single query in 1.0 seconds, outperforming other LLM-based methods in computational efficiency through a single forward pass.

6 Computational Costs

  • 1.0 seconds is sufficient for R2A to generate an adversarial prompt for a single query.R2A uses a single forward pass, whereas other LLM-based methods typically invoke the LLM multiple times for manually designed multi-step strategies.Source: §6 · p. 11
  • R2A has substantially lower computational overhead than existing LLM-based attack methods.The efficiency gain is attributed to avoiding repeated LLM invocations.Source: §6 · p. 11
  • R2A’s computational-cost comparison with baselines is reported for a single query.The corresponding results are presented in Table 3.Source: §6 · p. 11

7 CoT Adversarial Prompts Generated by Unified Generation Framework

The appendix visualizes the unified framework’s CoT examples and R2A’s adversarial prompts across several image and video-generation settings. It also compares R2A with baselines using blurred or masked outputs.

7 CoT Adversarial Prompts Generated by Unified Generation Framework

  • The Frame Semantics-based pipeline’s synthesized CoT reasoning examples are visualized in Fig. 3.These examples illustrate the training data produced by the unified generation framework.Source: §7 · p. 11, Figure 3 · p. 13

8 Attack Visualization

  • Additional adversarial prompts generated by R2A and their corresponding SD3 images are shown in Fig. 4.The images are blurred for display.Source: §8 · pp. 11–12, Figure 4 · p. 14
  • Fig. 5 compares visual outputs from R2A and baseline methods using SD1.4 with safety filters.The displayed images are masked and blurred.Source: §8 · p. 12, Figure 5 · p. 15
  • The appendix presents visual evidence for both the generated reasoning examples and the resulting adversarial prompts.Together, the figures cover synthesis examples, R2A outputs, baseline comparisons, and text-to-video attacks.Source: Figure 3 · p. 13, Figure 4 · p. 14, Figure 5 · p. 15
  • The visual comparisons use display transformations that obscure the generated images through blurring or masking.These transformations are specified for the SD3 and SD1.4 visualizations.Source: Figure 4 · p. 14, Figure 5 · p. 15
  • Fig. 6 visualizes R2A attacks on the OpenSora V2 text-to-video model.This extends the attack visualization beyond image-generation examples to a text-to-video setting.Source: Figure 6 · p. 16
Loading 2503.17987v3…