PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction

Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, Kai Chen

arXiv:2402.18104v2cs.CRcs.AI

TL;DR

Jailbreaks threaten LLM safety, while the fine-tuning biases underlying model vulnerability have received limited systematic investigation. The paper identifies this vulnerability and proposes DRA, a black-box attack that disguises harmful instructions and prompts their reconstruction. DRA outperforms state-of-the-art baselines across models, with reported success rates of 89.2% on GPT-4-API and 93.3% on ChatGPT-3.5-API.

  • Problem

    Few prior works systematically investigate the vulnerability underlying jailbreaks and its root cause, despite the potential for harmful outputs and associated risks in LLM-integrated software.

    Source: §1 · p. 1

  • Method

    DRA is a black-box jailbreak that exploits fine-tuning biases through harmful-instruction disguise, payload reconstruction, and context manipulation.

    Source: §1 · pp. 1–2

  • Results

    DRA outperformed state-of-the-art baselines across various LLMs, with attack success rates of 89.2% on GPT-4-API and 93.3% on ChatGPT-3.5-API.

    Source: §1 · p. 2, §8 · p. 14

  • Takeaways & Limitations

    Output-level filtering can mitigate DRA and other jailbreak attacks, though it may cause false positives, affect normal functionality, and incur additional costs.

    Source: §6 · p. 14

    DRA bypasses input-level defenses but is unable to circumvent output-level defenses.

    Source: §7 · p. 14

Abstract

from arXiv · show

In recent years, large language models (LLMs) have demonstrated notable success across various tasks, but the trustworthiness of LLMs is still an open problem. One specific threat is the potential to generate toxic or harmful responses. Attackers can craft adversarial prompts that induce harmful responses from LLMs. In this work, we pioneer a theoretical foundation in LLMs security by identifying bias vulnerabilities within the safety fine-tuning and design a black-box jailbreak method named DRA (Disguise and Reconstruction Attack), which conceals harmful instructions through disguise and prompts the model to reconstruct the original harmful instruction within its completion. We evaluate DRA across various open-source and closed-source models, showcasing state-of-the-art jailbreak success rates and attack efficiency. Notably, DRA boasts a 91.1% attack success rate on OpenAI GPT-4 chatbot.

1 Introduction

The introduction traces jailbreak vulnerability to fine-tuning biases that weaken safeguards for harmful content generated by the model itself, then presents DRA as a black-box attack exploiting this weakness. It reports state-of-the-art attack success rates on GPT-4-API and ChatGPT-3.5-API.

1 Introduction

  • Jailbreaks can elicit offensive or unethical content, and outputs used by LLM-integrated software could potentially lead to remote code execution vulnerabilities.The stated RCE risk concerns software systems relying on LLM-generated outputs.Source: §1 · p. 1
  • DRA disguises harmful instructions, prompts their reconstruction as completion payloads, and manipulates context to facilitate their enunciation.The introduction frames the challenge as disguising instructions to avoid dismissal while making them sufficiently reconstructable.Source: §1 · pp. 1–2
  • DRA adapts a shellcode-inspired software-security paradigm to LLMs through disguise, payload reconstruction, and context manipulation.The authors describe the approach as identifying and exploiting vulnerabilities such as biases in training datasets.Source: §1 · p. 2
  • Fine-tuning biases associated with dialogue formatting and training objectives leave safeguards weaker for self-generated harmful content than for harmful content provided by users.The authors link this imbalance to harmful instructions occurring more often in queries than completions during fine-tuning, leaving fewer safe responses paired with harmful completion contexts.Source: §1 · pp. 1–2
  • DRA achieves state-of-the-art attack success rates of 89.2% on GPT-4-API and 93.3% on ChatGPT-3.5-API.The authors also report reduced trials and generation time, minimal adjustments across target models, and no reliance on LLMs to modify the jailbreak prompt.Source: §1 · p. 2

2 Background & Problem Statement

The background introduces autoregressive LLMs, jailbreak methods, and safety-alignment approaches before specifying a black-box jailbreak problem in which attackers can use only model inputs and outputs.

2.1 Large Language Models

  • LLMs use few-shot or zero-shot prompting for downstream tasks, and the paper describes open- and closed-source models as autoregressive systems predicting tokens from preceding tokens.The formal setup denotes the prompt-containing context as x and the predicted sequence as y.Source: §2.1 · p. 2

2.2 LLM Jailbreak

  • Jailbreaking uses prompt sequences to induce unexpected or harmful content; readable role-play and scenario methods are less efficient, while encrypted methods have poor readability.The section identifies further exploration of jailbreak principles and vulnerability root causes as needed.Source: §2.2 · pp. 2–3

2.3 Safety Alignment of LLM

  • Safety alignment approaches include moderation of queries and responses and robust training through data purification and fine-tuning, including SFT, RLHF, and RLAIF.The described efforts aim to mitigate toxic responses and bias in training data and labels.Source: §2.3 · p. 3

2.4 Problem Statement

  • The threat model assumes an attacker has no access to target-model architecture, parameters, training data, gradients, or logits, and can tune inputs using only model outputs.The research question is how to construct an input sequence that elicits unintended or potentially harmful responses to a harmful instruction.Source: §2.4 · p. 3

3 Safety Biases in LLM Fine-Tuning and the Resultant Vulnerability

The section traces a safety vulnerability to differences between query and completion modeling and biases in safety fine-tuning. Together, these can leave harmful contexts in completions less well covered by safe-response training, which DRA seeks to exploit.

3 Safety Biases in LLM Fine-Tuning and the Resultant Vulnerability

  • The section links query/completion distinctions and fine-tuning biases to reduced safety for harmful contexts placed in completions.Because fine-tuning processes are inaccessible, the authors say these biases are verified indirectly by analyzing model behavior after safety fine-tuning.Source: §3 · p. 3, §3.2 · p. 4

3.1 Dialog Modeling and its Discrepancy

  • Dialogue templates distinguish user queries from model completions, and the resulting distributional discrepancy may hinder transfer of safe responses to harmful contexts in completions.LLAMA-2’s example uses “[/INST]” to separate the query and completion.Source: §3.1 · p. 3

3.2 Fine-Tuning and its Safety Biases

  • Fine-tuning objectives expose models less often to harmful instructions in completions and pair them less often with safe responses, reducing safe handling of harmful completion contexts.The authors say the resulting biases can be exploited by inducing the model to generate harmful contexts.Source: §3 · p. 3, §3.2 · p. 4

3.3 Formal Definition of the Vulnerability

  • The vulnerability is defined as lower refusal and greater cooperation when harmful content lies in the completion rather than the query.The authors attribute the imbalance to insufficient harmful-completion examples and say query-response generalization alone is challenging; they expect it in models with similar dialog formatting and fine-tuning methods.Source: §3.3 · p. 4

4 Approach

DRA combines instruction disguise, payload reconstruction, and context manipulation to place harmful semantics in the model’s completion. Its three components aim to bypass input safeguards and exploit safety-fine-tuning bias.

4 Approach

  • DRA combines harmful-instruction disguise, payload reconstruction, and context manipulation to place harmful semantics in the completion and exploit safety-fine-tuning bias.The approach is designed to bypass safety detection in the input phase while guiding the model to express the harmful instruction in its completion.Source: §4 · p. 5

4.1 Harmful Instruction Disguise

  • DRA disguises instructions with puzzle-based obfuscation and dynamic word-level splitting, adjusting cutoffs from model feedback to conceal toxic terms while preserving reconstructable meaning.Puzzle words occupy about 10% of attention, potentially reducing focus on system prompts and harmful parts; split fragments can also guide reconstruction.Source: §4.1 · pp. 5–7

4.2 Payload Reconstruction

  • A query-agnostic prompt guides the model to extract marked characters, assemble the disguised instruction, use split fragments to improve reconstruction, and deliver the payload in its completion.The reconstruction prompt can be designed flexibly, provided it instructs the model to rebuild the payload faithfully.Source: §4.2 · p. 7

4.3 Context Manipulation

  • Context manipulation aims to improve reconstruction and provide a plausible backdrop, using adaptable contexts and cooperation-promoting language to control model output.The authors describe the context strategy as flexible, allowing attackers to design prompts according to its guidelines.Source: §4.3 · p. 8

5 Evaluation

The evaluation examines DRA’s effectiveness and efficiency, evidence for positional bias in harmful content, and its performance against defenses and alternative disguise strategies. It also uses ablations to assess the contributions of disguise, payload reconstruction, and context manipulation, and qualifies DRA’s model-agnostic advantage over translation-based strategies.

5.1 Experimental Settings

  • The 120-question dataset combined 80% of questions from existing datasets and competitions with 20% written by human experts; attack success required refusal, relevance, and harmfulness criteria all to be satisfied.Relevance required the response to contain more than 50% of the input’s words, and HarmBench judged harmfulness.Source: §5.1 · pp. 8–9

5.2 LLMs’ Positional Bias to Harmful Content

  • For LLAMA-2-13B-Chat, harmful context placed after the [/INST] token generally received more attention and increased its role in generating responses, whether the model refused or endorsed it.This finding motivates placing inducing content in completions to direct the model’s attention toward harmful instructions.Source: Table 2 · p. 9, Table 3 · p. 9, §5.2 · p. 10
  • Most harmful instructions had higher log-perplexity in completions than in queries, while inducing prefixes increased the number of cases matching the predicted perplexity inequalities.Without a prefix, the inequalities held for all declines but only half of cooperative cases; prefixes accentuated the difference.Source: Figure 4 · p. 10, §5.2 · pp. 10–11, Figure 5 · p. 10
  • Harmful content in the completion produced considerably higher ASR than the same content in the query, and models with higher completion-condition ASR were more vulnerable to DRA.These white-box experiments support a link between the positional vulnerability and jailbreak success.Source: Table 5 · p. 11, §5.2 · p. 11

5.3 Effectiveness and Efficiency

  • DRA outperformed baselines across targeted models, achieving 100% ASR on Vicuna with 1.30 iterations and around 90% ASR on both GPT-4 versions with fewer than 4 queries.The manually simulated GPT-4 web-interface evaluation also demonstrated the method’s usability in a real-world setting.Source: Table 7 · p. 12, §5.3 · p. 12

5.4 Attack Against Defenses

  • DRA bypassed OpenAI Moderation, the perplexity filter, and RA-LLM with DPRs of at least 98.8%, while Bergeron imposed 42.61 seconds of overhead per prompt.The authors attribute DRA’s evasion to disguised, readable prompts that retain enough information for payload reconstruction despite minor perturbations.Source: §5.4 · pp. 12–13, Table 9 · p. 13

5.5 Ablation Study

  • The ablation substituted five existing obfuscation techniques into DRA and tested the modified pipelines against LLAMA-2 and ChatGPT 3.5-API.Table 10 compares attack success rates, but the supplied passage does not state the comparison’s outcome.Source: §5.5 · p. 13, Table 10 · p. 13
  • Caesar Cipher and ASCII encoding achieve minimal ASRs, with 91.9% of GPT 3.5 queries yielding zero em values and LLAMA-2 directly rejecting 85.5% of Caesar-encoded queries.Pig-Latin also causes mostly irrelevant responses, which the authors attribute to misinterpretation.Source: §5.5 · p. 13
  • Swahili translation has comparable ASR on GPT 3.5 but is not model-agnostic, as LLAMA-2 misinterprets 80.6% of harmful queries; Persuade also falls short of DRA’s ASRs.The paper attributes Persuade’s lower ASRs to models’ tendency to recognize harmful intentions and concludes that DRA’s word puzzles and splits balance concealment with intent preservation.Source: §5.5 · p. 13
  • The ablation study independently removes harmful-instruction disguise, payload reconstruction, and context manipulation to test each component across the models listed in Table 1.For context manipulation, the study removes compliance-prompting words or revises the prompt to reduce inducement while preserving instruction reconstruction.Source: §5.5 · p. 13
  • Removing disguise particularly reduces ASRs for LLAMA-2 and Mixtral, while removing payload reconstruction lowers ASRs across all models and removing context manipulation lowers ASRs except for Zephyr.The authors attribute Zephyr’s comparatively higher ASRs under component ablations to its reduced safety-alignment data.Source: §5.5 · pp. 13–14

6 Mitigation

The paper recommends reducing fine-tuning bias, strengthening system prompts, and sanitizing outputs, while balancing safety against training cost, usability, and false positives.

6 Mitigation

  • Including harmful instructions in varied forms in both user prompts and model completions may directly mitigate DRA, but incurs significant training costs.The recommendation is to enhance and balance safety fine-tuning datasets.Source: §6 · p. 14
  • A short system prompt increased DRA’s success rate by 25% and decreased average queries by 1.22, while strict prompts can defend against jailbreaks but may impair model performance.The authors recommend balancing safety and usability when designing system-prompt defenses.Source: §6 · p. 14
  • Output filtering can mitigate DRA and other jailbreaks because DRA bypasses input-level defenses but not output-level defenses, though filtering may cause false positives and added costs.The authors recommend balancing safety, usability, and cost when applying safety checks.Source: §6 · p. 14

7 Discussions

The discussion identifies output-level defenses as a boundary for DRA and describes disclosure practices and small-scale tests on additional commercial models.

7 Discussions

  • DRA can bypass input-level defenses but cannot circumvent output-level defenses, motivating future work on covert harmful outputs or adaptive attacks against output filters.The authors identify output filtering as a target for future attack development.Source: §7 · p. 14
  • The authors report small-scale successful DRA tests on ERNIE Bot, Qwen2.5 Web, Spark Web, Kimi Chat, and GLM-4 Web, and say they disclosed findings to those providers.Experiments on several commercial closed-source models were not disseminated, and no malicious feedback was implanted.Source: §7 · p. 14
  • The authors say they disclosed findings and examples to targeted providers, and received acknowledgments and bug bounties from one provider.Disclosure used emails, GitHub issues, and risky-content feedback forms.Source: §7 · p. 14

8 Conclusion

The conclusion presents DRA as an exploit of fine-tuning safety biases and reports stronger performance than state-of-the-art baselines across multiple models.

8 Conclusion

  • DRA demonstrated better performance than state-of-the-art baselines across various LLMs, including ChatGPT-3.5 and GPT-4.The conclusion frames this as empirical analysis across multiple models.Source: §8 · p. 14
  • The authors report experimentally validating safety biases introduced during fine-tuning and the resulting vulnerability.They describe this as identifying a root cause of the vulnerability.Source: §8 · p. 14
  • DRA combines harmful-instruction disguise, payload reconstruction, and context manipulation to exploit the vulnerability.The conclusion describes the attack as a strategy incorporating these techniques.Source: §8 · p. 14
  • The authors say their analysis lays groundwork for subsequent research aimed at improving resilience against adversarial exploits.They present the study as contributing insights to LLM vulnerability research.Source: §8 · p. 14

A Dataset

The dataset is organized across all seven HarmBench question-taxonomy categories, with category imbalance shaping its expansion.

A Dataset

  • The dataset covers all 7 HarmBench question-taxonomy categories, with Illegal Activities the largest category at 24.2%.The reported distribution also includes Cybercrime & Unauthorized Intrusion (16.7%), Chemical & Biological Weapons/- Drugs (8.3%), Copyright Violations (10%), Misinformation & Disinformation (11.7%), Harassment & Bullying (10.8%), and General Harm (18.3%).Source: A Dataset · p. 17
  • Because the original 100 public-dataset questions underrepresented Copyright Violations and Misinformation & Disinformation, dataset expansion focused primarily on those categories.The authors cite imbalance in the originally collected questions as the reason for this focus.Source: A Dataset · p. 17
  • The dataset’s category shares range from 8.3% for Chemical & Biological Weapons/- Drugs to 24.2% for Illegal Activities.The source reports these category proportions within its seven-category taxonomy.Source: A Dataset · p. 17

B Open Source LLMs

The appendix describes the open-source models used in the experiments and gives examples of their dialogue templates, which mark query and completion boundaries.

B Open Source LLMs

  • LLAMA-2-13B-Chat combines SFT and RLHF, while Mistral-7B-Instruct uses SFT on public instruction datasets and Mixtral8x7B-Instruct combines SFT with DPO.The passage describes LLAMA-2-13B-Chat as a baseline, Mistral as strong on instruction-following tasks, and Mixtral as using a Mixture of Experts architecture.Source: B Open Source LLMs · p. 17
  • Vicuna-13B version 1.5 is the target model, and Zephyr-7B uses distilled SFT and DPO without human annotation.Vicuna is fine-tuned from LLAMA-2-13B; Zephyr is described as aligning closely with user intent.Source: B Open Source LLMs · p. 17

C Dialogue Templates

  • Reviewing open-source large language model dialogue formats, the passages report that these models universally use specific tokens to separate queries from completions, with examples including USER/ASSISTANT and [INST]...[/INST] formats.Source: C Dialogue Templates · pp. 17–18

D Derivation of Log Perplexity Inequalities

The appendix derives log-perplexity inequalities from the autoregressive likelihood of a response given its context, then lists default system prompts for several target models.

D Derivation of Log Perplexity Inequalities

  • The likelihood of response d given context x is decomposed autoregressively, and taking its logarithm and normalizing by response length yields the negation of log-perplexity.The passage identifies x as the context where harmful content resides in the query and x′ as the context where harmful content appears in the completion.Source: D Derivation of Log Perplexity… · p. 18

E System Prompts of Target Models

  • The listed default prompts range from brief instructions for ChatGPT and Vicuna to a detailed safety-oriented prompt for LLAMA-2.The LLAMA-2 prompt asks for helpfulness and safety and instructs the model not to include harmful, unethical, or illegal content.Source: E System Prompts of Target Models · p. 18

F LLAMA-2’s Update on System Prompt

The appendix reports that LLAMA2 removed its strict system prompt after it impaired normal functionality, following feedback about false refusals.

F LLAMA-2’s Update on System Prompt

  • LLAMA2 removed its strict system prompt because it impaired normal functionality, and its updated default setting removes the system prompt to reduce false refusals.The update defines false refusals as incorrectly refusing questions that should be answered and retains an example prompt for experimentation.Source: F LLAMA-2’s Update on System Prompt · p. 18
Loading 2402.18104v2…