PaperlayerView exact revision on arXiv

Source-linked AI summary

A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, Shujian Huang

arXiv:2311.08268v4cs.CL

TL;DR

Existing jailbreak methods are limited by intricate manual design or white-box optimization that can reduce efficiency and generalization. ReNeLLM automatically combines prompt rewriting with scenario nesting to generate attacks, achieving high attack success rates efficiently across representative LLMs while exposing weaknesses in current defenses.

  • Problem

    Existing jailbreak methods suffer from intricate manual design or optimization on other white-box models, limiting their efficiency or generalization.

  • Method

    ReNeLLM automatically generates jailbreak prompts by combining semantics-preserving prompt rewriting with scenario nesting, using LLMs without additional training or optimization.

  • Results

    ReNeLLM achieves high attack success rates efficiently across representative LLMs, with experiments showing generalization, transferability, and inadequacy of existing defenses.

  • Takeaways & Limitations

    The findings provide guidance for analyzing prompt execution priorities and developing safer, more regulated LLM defenses.

  • Takeaways & Limitations

    The method uses three static nesting scenarios and primarily English experimental datasets, limiting scenario diversity and multilingual scope.

Abstract

from arXiv · show

Large Language Models (LLMs), such as ChatGPT and GPT-4, are designed to provide useful and safe responses. However, adversarial prompts known as 'jailbreaks' can circumvent safeguards, leading LLMs to generate potentially harmful content. Exploring jailbreak prompts can help to better reveal the weaknesses of LLMs and further steer us to secure them. Unfortunately, existing jailbreak methods either suffer from intricate manual design or require optimization on other white-box models, which compromises either generalization or efficiency. In this paper, we generalize jailbreak prompt attacks into two aspects: (1) Prompt Rewriting and (2) Scenario Nesting. Based on this, we propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts. Extensive experiments demonstrate that ReNeLLM significantly improves the attack success rate while greatly reducing the time cost compared to existing baselines. Our study also reveals the inadequacy of current defense methods in safeguarding LLMs. Finally, we analyze the failure of LLMs defense from the perspective of prompt execution priority, and propose corresponding defense strategies. We hope that our research can catalyze both the academic community and LLMs developers towards the provision of safer and more regulated LLMs. The code is available at https://github.com/NJUNLP/ReNeLLM.

1 Introduction

LLMs can be bypassed by malicious prompts, while existing manual and learning-based jailbreak methods face design, robustness, efficiency, and transferability limitations. ReNeLLM addresses these shortcomings by automatically combining prompt rewriting with scenario nesting and reports strong attack performance, transferability, and weaknesses in existing defenses.

  • Carefully crafted malicious prompts can expose inadequate safeguards in LLMs, including through manual attacks such as DAN and learning-based attacks such as GCG.
  • Manual jailbreaks require intricate design and can become ineffective as LLMs are updated, whereas learning-based attacks use meaningless white-box-optimized suffixes and require substantial search time.Learning-based methods also show lower efficacy on commercial models such as Claude-2.
  • ReNeLLM automatically generates jailbreak prompts through semantics-preserving prompt rewriting and stealth-enhancing scenario nesting, without additional training or optimization.The framework uses LLMs themselves to construct attack prompts and is designed for efficiency and transferability across multiple LLMs.
  • ReNeLLM maintains a high attack success rate with less time required, while exhibiting generalization and transferability across representative LLMs.The authors report that these results demonstrate alarming deficiencies in existing LLM security performance.
  • Existing defense methods inadequately safeguard LLMs against generalized ReNeLLM attacks, motivating analysis of prompt execution priorities and corresponding defense methods.The authors examine execution-priority changes before and after rewriting and nesting to inform safer defenses.

2 Related Work

Related work describes safety alignment and established jailbreak attacks, while noting that aligned LLMs remain susceptible to adversarial inputs. Prior attacks include manual prompt construction and automated adversarial-suffix optimization.

  • Safety-aligned LLMs are designed to produce outputs matching human preferences and ethics through data and training interventions such as filtering, SFT, and RLHF.
  • Despite safety alignment, LLMs remain susceptible to adversarial inputs such as jailbreak attacks.
  • ReNeLLM’s framework overview presents prompt rewriting followed by scenario nesting, with the complete construction automated without additional training or optimization.
  • Jailbreak research has examined attacks by objectives and tactics, attributed vulnerabilities to tensions between capability and safety objectives, and developed automatically generated adversarial suffixes.The cited work includes systematic attack classification and suffix generation by combining greedy and gradient-based methods.

3 Methodology

ReNeLLM constructs jailbreak prompts by applying semantics-preserving rewriting operations and nesting the rewritten prompt in an instruction scenario. The process searches a finite strategy space, evaluates harmfulness, and automates scenario and rewrite selection without additional training or optimization.

  • Method overview: ReNeLLM generalizes automatic jailbreak generation into prompt rewriting and scenario nesting while preserving the initial prompt’s semantics.The process is completed by LLMs without additional training or optimization.
  • Formulation: The attack seeks a finite sequence of strategy actions that edits an initial harmful prompt to maximize the likelihood that the model’s response is judged harmful by a harmfulness evaluator.A higher evaluator value indicates a greater likelihood of successful jailbreaking.
  • Prompt rewriting: The rewriting stage disguises prompt intent through operations such as shortening, changing sentence structure, misspelling sensitive words, and changing expression style.The paper describes six rewriting functions overall and states that the core semantics remain unchanged.
  • Prompt rewriting: Rewriting functions are randomly combined and ordered, after which the rewriting LLM executes the selected operations on the prompt.
  • Scenario nesting: Scenario nesting places rewritten prompts inside instruction tasks because such scenarios may elicit responses from LLMs and shift attention during generation.
  • Scenario nesting: The framework uses Code Completion, Table Filling, and Text Continuation as universal nesting scenarios aligned with training data and containing blanks similar to sentence-level cloze tasks.A scenario is randomly selected to contain the rewritten prompt before it is sent to the model under test.

4 Experiment

The experiments evaluate ReNeLLM across harmful-prompt categories and representative open- and closed-source LLMs. Results show strong attack effectiveness, transferability, reduced generation time, and dependence on combining prompt rewriting with scenario nesting.

  • Experimental Setup: Five representative LLMs and a harmful-behavior dataset are used to evaluate generalized jailbreak attacks.The dataset contains 520 harmful-behavior prompts, classified into seven scenarios from 13 OpenAI usage-policy categories.
  • Attack Effectiveness and Transferability: ReNeLLM achieves state-of-the-art KW-ASR and GPT-ASR across open- and closed-source LLMs while transferring across different models.The results contrast with GCG and AutoDAN, whose suffixes are optimized for specific white-box models.
  • Attack Efficiency: 76.61% less generation time than GCG and 86.19% less than AutoDAN are achieved by ReNeLLM.The comparison uses Time Cost Per Sample with Llama2 as the optimization or test model.
  • Prompt Categories: Ensembling makes attack success rates approach 100% across harmful-prompt categories, despite lower susceptibility for Physical Harm.Malware and Privacy Violence are more susceptible than Physical Harm before ensemble evaluation.
  • Ablation Study: Prompt rewriting or scenario nesting alone is insufficient across all LLMs, whereas their combination substantially improves jailbreak success.Scenario nesting alone reaches 95.4 on GPT-3.5 and 62.3 on Claude-1; the full method improves llama2-chat-70b by over 40%.

5 Evaluating safeguards Effectiveness

The paper evaluates existing safeguard methods against ReNeLLM-generated prompts and examines attention changes during rewriting and scenario nesting. Existing defenses perform inadequately overall, while RA-LLM is more effective but costly to deploy.

  • Safeguard Evaluation: The safeguard evaluation tests OpenAI Moderation Endpoint, Perplexity Filter, and RA-LLM on llama-2-7b-chat and Claude-2.It uses 368 ReNeLLM prompts with GPT-ASR of 100.0 across all evaluated LLMs.
  • Attention Analysis: Attention shifts from original harmful prompts toward nested task instructions and other token spans during rewriting and scenario nesting.The analysis uses attention by gradient, which assigns attention scores from gradient norms.
  • Safeguard Evaluation: OpenAI’s official moderation interface failed to detect any harmful prompts, while the Perplexity Filter also performed far from satisfactorily.The paper attributes these results to limited prohibited-scenario coverage, weak base-model capability, and semantically meaningful attack prompts.
  • Safeguard Evaluation: RA-LLM reduces GPT-ASR by 28% and 24% but requires extensive testing time that is infeasible for real-world applications.It is the most effective among the three evaluated safeguard methods.

6 Analysis of ReNeLLM

The analysis attributes ReNeLLM’s effectiveness to shifted attention and execution priorities caused by prompt rewriting and scenario nesting, then evaluates several defenses. Priority prompts can substantially reduce attack success, but harmfulness classifiers struggle with ReNeLLM’s covert intent and remain costly to deploy.

  • Why LLMs fail to defend against the attack of ReNeLLM?: Attention becomes increasingly dispersed after rewriting and scenario nesting, shifting LLM priority toward external task instructions over internal malicious instructions.The authors interpret this as a shift from balancing safety and usefulness toward prioritizing useful responses.
  • How to defend against the attack of ReNeLLM?: Requiring LLMs to prioritize safety and scrutinize prompts makes all models successfully defend against the attacks, except that safety prioritization leaves GPT-3.5 with higher ASR.These interventions support the authors’ explanation that execution-priority changes contribute to defense failure.
  • How to defend against the attack of ReNeLLM?: SFT reduces table-filling attack success on llama-2-chat-13b from 100 to 0 when harmful code-completion data is included in training.The authors attribute this result to the similarity between table filling and code completion.
  • How to defend against the attack of ReNeLLM?: GPT-3.5 reduces GCG attack success by 86.0 but recognizes only 17.7 for ReNeLLM, indicating that rewriting and nesting conceal malicious intent more effectively.GPT-4 is more robust, yet ReNeLLM retains 10% effectiveness; deploying either classifier also incurs significant cost and time.
  • How to defend against the attack of ReNeLLM?: GPT-4 may identify harmful requests accurately while still producing harmful responses, separating harmfulness recognition from response safety.The authors present this contrast as relevant to safety-alignment research.

7 Conclusion

The paper presents ReNeLLM as an efficient automated jailbreak-generation framework based on prompt rewriting and scenario nesting. Its experiments expose inadequate current defenses, associate attack effectiveness with shifted prompt-execution priorities, and motivate several defense strategies.

  • Conclusion: ReNeLLM generates jailbreak prompts efficiently across representative LLMs by combining prompt rewriting with scenario nesting.The framework uses these transformations to generalize the jailbreak process.
  • Conclusion: Current defense methods provide inadequate safety against generalized ReNeLLM attacks, while defense experiments indicate that generalized and efficient protection remains challenging.The paper examines priority prompts, SFT, and harmfulness classifiers as defense strategies.
  • Conclusion: Attention visualization identifies a shift in LLM prompt-execution priorities before and after jailbreak, informing the explored defense strategies.The conclusion connects this observed shift with the paper’s analysis of why ReNeLLM is effective.

Limitations

The paper identifies three limitations: static nesting scenarios, predominantly English datasets, and stochastic rewriting and scenario selection that may not be optimal. It proposes automatically generated scenarios, multilingual datasets, and reinforcement learning as future directions.

  • The Fixity of Scenario Nesting: Static selection of three generic nesting scenarios may simplify targeted filtering or safety alignment defenses.The authors suggest having LLMs automatically generate possible nesting scenarios.
  • Datasets Diversity: Experiments rely primarily on English datasets, while applying rewriting to other languages may be challenging because language differences can be substantial.The paper proposes exploring offensive or harmful datasets in additional languages.
  • Computation&Cost: Randomly selecting rewriting-function counts, execution order, and nesting scenarios may be suboptimal and increase computation or cost.The authors identify reinforcement learning as a future approach for exploring jailbreaking patterns and reducing computation.

Ethical Considerations

The paper acknowledges that its automated jailbreak-generation method could be exploited to attack LLMs. It frames the work as ethically focused on uncovering vulnerabilities and accelerating robust defenses.

  • Ethical Considerations: The automated method could potentially be exploited by adversaries to launch attacks on LLMs.The authors state that their ethical focus is improving security rather than causing harm.
  • Ethical Considerations: The study aims to uncover security loopholes, raise awareness, and accelerate development of robust defenses for safer LLMs.The stated purpose is to protect LLMs from similar attacks.

A Statistics of Datasets

The harmful-behavior dataset is classified by GPT-4 under OpenAI’s user policies, with dataset details reported in Table 9. The paper also identifies where experimental prompts and qualitative examples are documented.

  • Dataset information: Dataset information is detailed in Table 9.
  • Dataset information: Table 9 reports the distribution of harmful behaviors classified by GPT-4 under OpenAI’s user policies.The table uses #P to denote the number of prompts.
  • Experimental materials: Tables 10, 11, and 13 list prompts used in the experiments.Figure 6 and Figure 7 provide qualitative examples of ReNeLLM and baseline jailbreaking results.

D Implementation Details

ReNeLLM uses LLM-based rewriting and scenario nesting, evaluates harmfulness and attack success with GPT models, and reports iteration, attack-success, and qualitative analyses across target models.

  • Model configuration: GPT-3.5 performs prompt rewriting and harmfulness evaluations during the jailbreaking phase, while GPT-4 evaluates target-model ASR.The evaluations cover harmful intent after rewriting and harmfulness in responses to nested prompts.
  • Jailbreaking procedure: Each iteration randomly selects 1 to 6 rewriting functions and one of three general nesting scenarios before querying Claude-2.The process loops when the response is not harmful and uses a maximum of T=20 iterations.
  • Time-cost evaluation: For TCPS, the authors average successful-jailbreak time over 16 AdvBench samples shared by multiple methods.The samples are identified by their AdvBench IDs, and Llama2 serves as the white-box and target model for specified baseline evaluations.
  • Generation settings: All LLMs use temperature 0; Claude-1, Claude-2, and Llama2 use max_tokens 3584, with Llama2 also using max_batch_size 6.No system prompts are used for any LLM.
  • Reported analyses: Figures 3–5 report iteration counts, ASR and ASR-E across LLMs, and category-level ASR on seven LLMs.Figures 6 and 7 show qualitative attack examples, including rewriting functions, nested scenarios, time costs, and Claude-2 responses.
Loading 2311.08268v4…