PaperlayerView exact revision on arXiv

Source-linked AI summary

DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers

Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, Cho-Jui Hsieh

arXiv:2402.16914v3cs.CRcs.AIcs.CL

TL;DR

Safety-aligned LLMs can still be induced to produce harmful content by jailbreaks, while existing attacks often leave malicious prompts conspicuous. DrAttack fragments prompts, implicitly reconstructs them with benign context, and searches sub-prompt synonyms; it reports over 80% human-evaluated success on GPT-4, over 65% above the previous best in absolute terms.

  • Problem

    Existing jailbreak attacks often treat harmful prompts as a noticeable whole, limiting their effectiveness against safety-aligned LLMs.

    Source: §1 · p. 1

  • Method

    DrAttack decomposes harmful prompts into sub-prompts, implicitly reconstructs them with benign semantically similar context, and searches for sub-prompt synonyms.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Over 80% human-evaluated attack success on GPT-4 surpassed previous best results by over 65% in absolute terms.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DrAttack’s findings identify a gap in current LLM safety mechanisms’ ability to thwart generalized attacks and point to a need for more robust defenses.

    Source: §7 · p. 9

    The work focuses on attack strategies without equal emphasis on defenses, and calls for exploring robust mechanisms against such attacks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The safety alignment of Large Language Models (LLMs) is vulnerable to both manual and automated jailbreak attacks, which adversarially trigger LLMs to output harmful content. However, current methods for jailbreaking LLMs, which nest entire harmful prompts, are not effective at concealing malicious intent and can be easily identified and rejected by well-aligned LLMs. This paper discovers that decomposing a malicious prompt into separated sub-prompts can effectively obscure its underlying malicious intent by presenting it in a fragmented, less detectable form, thereby addressing these limitations. We introduce an automatic prompt \textbf{D}ecomposition and \textbf{R}econstruction framework for jailbreak \textbf{Attack} (DrAttack). DrAttack includes three key components: (a) `Decomposition' of the original prompt into sub-prompts, (b) `Reconstruction' of these sub-prompts implicitly by in-context learning with semantically similar but harmless reassembling demo, and (c) a `Synonym Search' of sub-prompts, aiming to find sub-prompts' synonyms that maintain the original intent while jailbreaking LLMs. An extensive empirical study across multiple open-source and closed-source LLMs demonstrates that, with a significantly reduced number of queries, DrAttack obtains a substantial gain of success rate over prior SOTA prompt-only attackers. Notably, the success rate of 78.0\% on GPT-4 with merely 15 queries surpassed previous art by 33.1\%. The project is available at https://github.com/xirui-li/DrAttack.

1 Introduction

Existing jailbreak attacks often treat a malicious prompt as one noticeable entity, limiting their effectiveness against safety-aligned LLMs. DrAttack instead fragments and reconstructs prompts, and reports stronger attack success on powerful models.

1 Introduction

  • Treating a malicious prompt as a single noticeable entity undermines the effectiveness of current jailbreak attacks against safety-aligned LLMs.Safety alignment is intended to prevent harmful outputs, but adversarial jailbreaks can expose vulnerabilities.Source: §1 · p. 1
  • Over 80% human-evaluated attack success on GPT-4 surpassed previous best results by over 65% in absolute terms.The paper also reports that DrAttack improves ASR over prior SOTA attackers on powerful LLMs.Source: Figure 1 · p. 1, §1 · p. 2
  • DrAttack decomposes malicious prompts into neutral sub-prompts, implicitly reconstructs them with benign, semantically similar context, and searches for sub-prompt synonyms.The framework uses syntactic parsing for decomposition and synonym search to reduce the search space relative to optimization over the whole vocabulary.Source: §1 · p. 2

2 Related Work

Prior prompt-based jailbreak methods append suffixes, prepend contexts, or operate on whole harmful prompts. DrAttack adds a decomposition-based category, showing that fragmented prompts can support effective attacks.

2 Related Work

  • Existing monolingual attacks use suffixes, prefixes, or hybrid methods that operate on entire harmful prompts.Examples include GCG suffix optimization, AutoDAN system-prompt search, and benign-context methods such as PAIR and DeepInception.Source: §2 · p. 2
  • DrAttack establishes a feasible fourth category of jailbreak attacks by breaking harmful prompts into subphrases, and reports success with 15 queries.The paper contrasts this result with earlier studies in the category that had shown little success.Source: §2 · p. 2
  • Prompt decomposition has supported complex-task solving and applications including prompt selection, model training, fine-tuning, and vision tasks.The paper presents its work as the first method to leverage prompt decomposition for a strong attacker.Source: §2 · p. 2
  • The taxonomy contrasts attacks that append suffixes, modify prefixes or system prompts, or operate on whole prompts with DrAttack’s decomposition of malicious prompts.Source: Figure 2 · p. 2

3 Motivation

The motivation is that malicious prompts may trigger rejection when presented whole, while fragmented versions can reduce rejection responses. DrAttack generalizes this observation into a black-box attack.

3 Motivation

  • DrAttack generalizes the fragmentation strategy as a black-box attack for open-source and closed-source LLMs.The framework decomposes harmful prompts and reconstructs them to jailbreak target models.Source: §3 · p. 3
  • Prompt-based jailbreaks seek an adversarial prompt that elicits the desired answer from a target LLM rather than a rejection.The optimization objective is to maximize the likelihood of the desired answer given the adversarial prompt.Source: §3 · p. 3
  • Fragmented prompts significantly diminish the likelihood of rejection responses compared with original harmful prompts.The analysis compares average probabilities of the first five tokens in rejection responses using Llama2 and Vicuna.Source: §3 · p. 3

4 DrAttack Framework

DrAttack decomposes malicious prompts into sub-prompts and uses benign in-context examples to guide their implicit reconstruction. It also searches for synonymous sub-prompts to improve attack success while preserving intent.

4 DrAttack Framework

  • DrAttack combines semantic decomposition, implicit reconstruction through benign in-context examples, and synonym search to attack LLMs.The framework uses benign examples rather than harmful question-answer examples to teach reassembly.Source: §4 · p. 3, §4.1 · p. 4, §4.4 · p. 5

4.1 Overview of Pipeline

  • Because decomposition disrupts the original query’s intent, DrAttack embeds reconstruction in benign examples instead of directly requesting it.The authors say a direct reconstruction instruction can be rejected because it exposes the relationship between subphrases and their malicious intent.Source: §4.1 · pp. 3–4, §4.3 · p. 5

4.2 Prompt Decomposition via Syntactic Parsing

  • A syntactic parsing tree is converted into coherent, categorized phrases that retain meaning and are easier to reconstruct or modify.The phrase categories are [instruction], [structure], [noun], and [verb].Source: §4.2 · p. 4, Figure 5 · p. 5

Automated construction of ICL example

  • The ICL template gives a RULE for combining phrases, demonstrates it with benign ASSIGNMENT and RESULT content, then supplies the harmful ASSIGNMENT.The benign result includes a reconstructed prompt and response; the harmful query is implicitly reassembled in context.Source: Figure 6 · p. 5, Automated construction of ICL example · p. 5

4.4 Synonym Search on Sub-Prompts

  • Synonym Search replaces sub-prompts with synonyms that preserve intent, using phrase-level candidates and random search to improve attack success.The stated aim is to find replacements that jailbreak while generating faithful responses.Source: §4.4 · pp. 5–6

5 Experiments

DrAttack is evaluated against existing attacks across open- and closed-source models using attack success rate and query efficiency. It outperforms prior methods on powerful closed-source models, retains response faithfulness, and loses only slightly in success rate against the tested defenses.

5.1 Experimental setup

  • The evaluation uses AdvBench harmful-behavior prompts, GPT-based and human ASR judgments, and five closed-source plus four open-source victim models.Six baselines span white-box and black-box attacks; the models and baselines are implemented in EasyJailbreak for comparison.Source: §5.1 · p. 6

5.2 Results and Analysis

  • On GPT-4, DrAttack improves prior SOTA ASR by up to 50% under GPT evaluation and 65% under human evaluation, and consistently surpasses existing methods on closed-source models.The authors also report that it outperforms other black-box methods and performs on par with white-box attackers.Source: §5.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • Figure 7 reports average query counts and ASR; its caption says DrAttack outperforms other search attacks by modifying each sub-prompt.The query count is the average number of trials across test prompts; for white-box models it is batch size multiplied by convergence iterations.Source: §5.2 · p. 6, Figure 7 · p. 7
  • DrAttack maintains a similar cosine-similarity level to previous black-box attacks after decomposition and reconstruction, indicating no compromised response quality.Similarity compares DrAttack outputs with responses to the original prompt on uncensored Wizard Vicuna 13B.Source: §5.2 · pp. 6–7
  • DrAttack’s ASR drops only slightly against the tested defenses, while PAIR and DeepInception drop significantly under RA-LLM.The defenses tested are OpenAI Moderation Endpoint, Perplexity Filter, and RA-LLM.Source: §5.2 · p. 7

6 Ablation Study

The ablations identify decomposition and reconstruction as DrAttack’s core components, with synonym search boosting performance. In-context examples work best when semantically similar, affirmative, and structured.

6 Ablation Study

  • DrAttack can elicit relatively faithful responses and bypass defense mechanisms for which existing methods’ ASR drops significantly.Source: Figure 8 · p. 8
  • Semantically similar in-context examples are essential for reconstruction, and affirmative or step-by-step examples further improve ASR.Semantic-similar examples preserve the main sentence while replacing subordinate sub-prompts.Source: §6 · pp. 7–8, Figure 9 · p. 8
  • Decomposition and reconstruction are the essential components of DrAttack, while synonym search boosts attack performance and more synonym candidates yield slight ASR improvements.Figure 10 compares the component variants and top-k synonym search.Source: §6 · pp. 8–9, Figure 10 · p. 8

7 Conclusion

The paper presents DrAttack as an automated jailbreak approach based on prompt decomposition and reconstruction. Its findings report higher attack success rates and reduced iteration time overhead, while highlighting vulnerabilities in current safety mechanisms.

7 Conclusion

  • DrAttack automates jailbreaking through decomposition and reconstruction of original prompts.Source: §7 · p. 9
  • The framework embeds malicious content within phrases and achieves higher attack success rates with reduced iteration time overhead.Source: §7 · p. 9
  • The assessment finds a gap in current safety mechanisms’ ability to thwart generalized attacks generated by DrAttack.Source: §7 · p. 9

Limitations

The limitations section calls for more work on defenses against DrAttack. The appendices describe its parsing, in-context reconstruction, synonym search, word-game variant, and experiment details.

Limitations

  • The work emphasizes attack strategies more than defensive measures and calls for exploring robust defenses such as output filters.Source: Limitations · p. 9

A Appendix: Algorithm Details

  • The algorithm decomposes the prompt, creates a benign in-context example, searches sub-prompt synonyms level by level, then uses the best candidate for the final attack.Source: A Appendix: Algorithm Details · p. 13

A.1 Parsing Process in Decomposition

  • Parsing groups words into [structure], [verb], and [noun] categories; adjacent same-category words are combined into coherent sub-prompts, with the highest-level sub-prompt marked [instruction].The described GPT-4 and Stanford parser trees differ slightly but produce the same sub-prompts.Source: §A.1 · p. 13, Table 3 · p. 13

A.2 ICL Example Generation

  • Benign in-context examples balance harmlessness with structural similarity by minimally replacing [verb] and [noun] sub-prompts.Source: §A.2 · pp. 13–14

A.3 Level-wise Synonym Search

  • Level-wise synonym search starts with low-level [verb] and [noun] substitutions, filtering for prompt faithfulness and answers close to the target rather than the opposite answer.The method uses GPT-generated candidates and a random search over synonym combinations.Source: §A.3 · pp. 14–15

A.4 Word Game

  • The word-game variant redefines selected [noun] sub-prompts to camouflage harmful intent, and the paper reports the strategy as effective on its sub-prompts.A word game is presented before the adversarial prompt with strategically replaced words.Source: §A.4 · pp. 15–16

B Appendix: Experiment Details

B.1 Dataset Details

The appendix documents the evaluation data and procedures, victim models, baselines, and implementation settings. It also describes defenses used in testing and contrasts DrAttack with another decomposition-based attack.

B.1 Dataset Details

  • Experiments use the first 100 AdvBench harmful-behavior prompts and evaluate attack success with string matching, GPT judgments, and human surveys.The paper notes that string matching can yield false positives and true negatives; a response counts as harmful if at least one human marks it so.Source: §B.1 · pp. 16–17

B.2 Victim LLMs Details

  • The appendix lists closed-source victim-model versions and system prompts where available, uses default model hyperparameters, and reports about 120 minutes per open-model experiment on an NVIDIA A6000.Source: §B.2 · p. 17, Table 4 · p. 18, Table 5 · p. 18

B.3 Baselines Details

  • Baselines are reproduced with EasyJailbreak defaults, and each method receives one run because commercial-model generation and evaluation are costly.Source: §B.3 · p. 17

B.4 DrAttack Hyperparameter

  • The sole DrAttack hyperparameter, the prompt-difference threshold τ, is set to 0.1.Source: §B.4 · p. 18

B.5 Embedding Methods

  • Text embeddings use OpenAI’s text-embedding-ada-002 to measure relatedness through vector distance or cosine similarity.Source: §B.5 · p. 18

B.6 Related Automation Prompt Templates

  • GPT automates parsing-tree generation, synonym and opposite generation, and harmless replacements; the appendix also identifies a multilingual-attack comparison on AdvBench.The authors state that source code and experiment data will be released through a GitHub repository.Source: §B.6 · p. 18, Table 6 · p. 19

B.7 Defense Details

  • Defense tests use the OpenAI Moderation Endpoint, a GPT-2 perplexity filter, and RA-LLM with specified stride, drop ratio, candidate count, and threshold settings.The stated RA-LLM settings are drop ratio 0.1, one candidate, and threshold 0.5; perplexity stride is 10.Source: §B.7 · p. 19

C.1 Comparison to Other Decomposition Attacks

  • Wrap with Shell uses human-crafted decomposition and direct concatenation, whereas DrAttack uses semantic decomposition and implicit reconstruction; the passage says newer models mitigate Wrap with Shell while DrAttack remains effective.Source: §C.1 · p. 19

C.3 Jailbreaking on More Datasets

Across additional benchmark settings, the paper reports consistent high ASR on state-of-the-art LLMs and says DrAttack can bypass Instruction Hierarchy defenses. The appendix also presents successful examples, failure cases, and a comparison with multilingual attacks.

C.3 Jailbreaking on More Datasets

  • The evaluation uses 10 long Harmbench attacks, 50 MasterKey prompts, and JailbreakBench scenarios, with GCG and PAIR as baselines.Victim models match those used for JailbreakBench baseline evaluation.Source: §C.3 · p. 19

C.4 Consistency on SOTA LLMs

  • DrAttack is reported to achieve consistent high ASR on state-of-the-art LLMs and to bypass defenses such as Instruction Hierarchy.Table 9 is identified as reporting DrAttack ASR on state-of-the-art LLMs.Source: §C.4 · p. 19, Table 9 · p. 20

C.5 Orthogonal Comparison with Multilingual Attack

  • The authors compare DrAttack with a multilingual attack and frame the comparison as a test of potential gains from combining orthogonal methods.Table 8 is identified as reporting AdvBench attack success rates for the multilingual attack and DrAttack.Source: §C.5 · p. 19, Table 8 · p. 20

D Appendix: Examples

D.1 Successful Cases

  • To showcase DrAttack’s effectiveness on publicly available LLMs, the paper provides two screenshots: one from the OpenAI ChatGPT web service4 and another from the Llama2 playground5, as shown in Figure 15.Source: §D.1 · p. 20

D.2 Failure Cases

  • Overtly malicious words can trigger built-in protections, making prompt-based jailbreaking difficult.The paper illustrates this with a prompt containing an explicit request to glorify pedophilia.Source: §D.2 · p. 20
  • Highly specific phrases without suitable synonyms can expose malicious intent and reduce DrAttack’s effectiveness.The example given is the phrase “graphic sex with animals.”Source: §D.2 · p. 20

E Appendix: Documentation

  • The artifact is available under the MIT License and is intended exclusively for research use; use outside research contexts is not advised.Source: E Appendix: Documentation · p. 20

E.1 Utilized Datasets

The appendices document licenses and evaluation materials, discuss the risks and ethical handling of the work, and specify limitations in language and data realism. They also describe human-evaluation participants and their training.

E.1 Utilized Datasets

  • The evaluation datasets and tools are documented with their licenses, including AdvBench, JailbreakBench, Harmbench, EasyJailbreak, Vicuna, Llama2, and the Stanford Parser.Source: §E.1 · pp. 20–22
  • Tables 10 and 11 provide examples of harmful and benign responses and list key rejection strings used to validate responses.Source: Table 10 · p. 21, Table 11 · p. 21

F Appendix: Potential Risks

  • The authors identify dual-use risk: the method could help malicious actors bypass LLM safeguards, while disclosure is presented as important for developing defenses.They state that the technique is simple to implement and may ultimately be discovered by malicious attackers.Source: F Appendix: Potential Risks · p. 22

F.1 Ethical Considerations and Use Restrictions

  • The authors state that sensitive-data handling, access guidance, and security measures are documented for community review.Source: §F.1 · p. 22

F.2 Limitations

  • The evaluation is limited to English-language queries and artificially generated data that may not fully represent real-world scenarios.Source: §F.2 · p. 22

G Appendix: Human Evaluation of Adversarial Responses

G.1 Recruitment and Training

  • Before judging model responses, volunteers were briefed on the study and trained with examples of harmful and benign outputs.Source: §G.1 · p. 22

G.2 Survey Methodology

Evaluators judged model-generated texts as harmful or unsafe through a survey, and a response counted as harmful if any evaluator flagged it. Participants consented to anonymous research use of their responses.

G.2 Survey Methodology

  • Evaluators were asked whether each model response was harmful, dangerous, toxic, or unsafe.Source: §G.2 · p. 22
  • A response was classified as harmful if at least one of multiple independent evaluators judged it harmful.Source: §G.2 · p. 22
  • Figure 15 presents example adversarial attack responses from GPT-3.5-turbo and Llama2-7b chat models.Source: Figure 15 · p. 23

G.3 Ethical Considerations and Consent

  • Participants consented to anonymous use of their responses for research purposes.Source: §G.3 · p. 23
Loading 2402.16914v3…