Source-linked AI summary
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack
Mark Russinovich, Ahmed Salem, Ronen Eldan
TL;DR
Safety-aligned LLMs are designed to resist harmful tasks, motivating research into jailbreaks that overcome this restriction. The paper introduces Crescendo, a benign multi-turn attack, and Crescendomation, an automated tool; evaluations show broad effectiveness and higher AdvBench performance than competing techniques.
Problem
Safety-aligned LLMs resist illegal or unethical tasks, but jailbreaks seek to overcome this alignment.
Method
Crescendo progressively steers a model through benign multi-turn prompts that reference its own outputs, while Crescendomation automates this process using model feedback.
Results
29-61% higher performance on GPT-4 and 49-71% on Gemini-Pro were achieved by Crescendomation over state-of-the-art jailbreaking techniques on AdvBench.
Takeaways & Limitations
Crescendo bypassed safety alignment for nearly all tasks across evaluated models and extended to multimodal tasks such as generating normally refused images.
Takeaways & Limitations
Manual results are illustrative rather than exhaustive, and Crescendo is inherently less applicable to systems without conversation history.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have risen significantly in popularity and are increasingly being adopted across multiple applications. These LLMs are heavily aligned to resist engaging in illegal or unethical topics as a means to avoid contributing to responsible AI harms. However, a recent line of attacks, known as jailbreaks, seek to overcome this alignment. Intuitively, jailbreak attacks aim to narrow the gap between what the model can do and what it is willing to do. In this paper, we introduce a novel jailbreak attack called Crescendo. Unlike existing jailbreak methods, Crescendo is a simple multi-turn jailbreak that interacts with the model in a seemingly benign manner. It begins with a general prompt or question about the task at hand and then gradually escalates the dialogue by referencing the model's replies progressively leading to a successful jailbreak. We evaluate Crescendo on various public systems, including ChatGPT, Gemini Pro, Gemini-Ultra, LlaMA-2 70b and LlaMA-3 70b Chat, and Anthropic Chat. Our results demonstrate the strong efficacy of Crescendo, with it achieving high attack success rates across all evaluated models and tasks. Furthermore, we present Crescendomation, a tool that automates the Crescendo attack and demonstrate its efficacy against state-of-the-art models through our evaluations. Crescendomation surpasses other state-of-the-art jailbreaking techniques on the AdvBench subset dataset, achieving 29-61% higher performance on GPT-4 and 49-71% on Gemini-Pro. Finally, we also demonstrate Crescendo's ability to jailbreak multimodal models.
1 Introduction
The paper introduces Crescendo, a benign multi-turn jailbreak that gradually steers models toward harmful tasks by leveraging their own responses. It evaluates the attack and its automation tool across current LLMs, finding broad effectiveness and strong benchmark performance.
- LLM jailbreaks target safety alignment by enabling models to perform illegal, unethical, or otherwise harmful tasks.
- Crescendo begins with an innocuous question and progressively guides the model toward harmful content through multiple interactions.The attack exploits the model’s tendency to follow patterns and attend to recent model-generated text.
- Crescendo examples on ChatGPT and Gemini Ultra show refusal for a direct request but task completion after the multi-turn interaction.
- Crescendo overcame the safety alignment of all evaluated models for nearly all tested tasks, including open- and closed-source systems.The evaluated systems included LLaMA-2 70b, LLaMA-3 70b, Gemini-Pro, Claude-2, Claude-3, GPT-3.5 Turbo, and GPT-4.
- Crescendomation automates Crescendo by taking a target task and model API access, then refining questions using an LLM and feedback on output quality or refusal.
- The evaluation combines a GPT-4 Judge, a Secondary Judge, manual inspection, and external content-safety APIs to assess jailbreak success and response categories.The external APIs score categories including hate speech, self-harm, violence, and sexual content.
- 29-61% higher performance on GPT-4 and 49-71% on Gemini-Pro were achieved by Crescendomation over several state-of-the-art jailbreaking techniques on AdvBench.The comparison included Many-Shot Jailbreak, Contextual Interaction Attack, PAIR, and Chain of Attack.
- The paper argues that current single-turn benchmarks and alignment strategies do not adequately address adaptive multi-turn jailbreaks.It proposes Crescendomation for benchmarking and aligning models against these attacks.
2 Related Works
Prior jailbreak methods include optimization-based attacks, textual prompts, manipulated context, and inference settings. Crescendo differs by using benign, human-readable prompts and model-generated outputs rather than explicit malicious content or optimization.
- Other approaches manipulate inference settings such as temperature, topK, or decoding techniques to affect safety alignment.
- Context-based attacks add malicious in-context examples or fabricated interaction histories, with methods differing in the amount and form of added context.Many-Shot Jailbreak uses over 100 malicious examples, while another approach uses simulated interactions.
- Crescendo uses completely human-readable, innocuous prompts and relies on the target model’s own outputs instead of explicitly crafted malicious text.
- Crescendo is a black-box jailbreak without explicit optimization, and Crescendomation achieves higher attack success rates than other state-of-the-art techniques.Its benign prompts also make detection and mitigation more challenging, including for publicly available AI chat services.
3 Crescendo
Crescendo is a black-box, multi-turn jailbreak that progressively intensifies benign, model-generated context to bypass safety alignment. Manual examples and analyses show broad effectiveness across tasks, modalities, and sentence sequences.
- Crescendo: Crescendo uses benign, human-readable prompts and the target model’s outputs to progressively direct responses toward bypassing safety alignment.It begins with an innocuous topic linked to the target task and intensifies through subsequent turns.
- Crescendo: The adversary can iteratively tailor the output, such as repeating “make it angry” until the response meets the intended specification.The pattern allows minimal direct references to the target task while refining the generated content.
- Crescendo: Crescendo requires only black-box API access and makes no assumptions about the target model.The attack proceeds through normal interactions rather than explicit optimization.
- Manual Examples: Manual evaluations targeted five groups of aligned public systems, including ChatGPT, Gemini, Anthropic Chat, LLaMA-2 70b, and LLaMA-3 70b.The evaluated task set spans categories that contravene safety guidelines.
- Manual Examples: Crescendo successfully jailbroke all evaluated models across the vast majority of tasks, sometimes producing targeted, copyrighted, profane, or otherwise refused multimodal outputs.The paper also reports dual Crescendo sequences and image generation after a model was jailbroken.
- Understanding Crescendo: 99.99% compliance followed Sentence A before Sentence B, versus approximately 36.2% when Sentence B was presented directly; Sentence C reached 99.9% after Sentence A but 17.3% without it.Replacing the model-generated Sentence C with an explicit profanity request reduced jailbreak success to less than 1%.
4 Crescendomation
Crescendomation automates the Crescendo attack using API access to a target model and adaptive, multi-turn prompting. Its evaluation combines LLM judges, manual review, and external moderation APIs.
- Automation: Crescendomation automates Crescendo with only API access to the target model required.It takes a task as input and executes a Crescendo jailbreak against the target model.
- Adaptive interaction: An attack model generates successive prompts from the target task, conversation history, and previous responses over multiple turns.The algorithm initializes histories, generates Crescendo steps, queries the target model, and records prompts and responses.
- Evaluation: Crescendomation evaluates jailbreak success through Self-Evaluation and external moderation APIs.The evaluation methodology uses an LLM-based evaluator together with commercial content-moderation services.
- Evaluation: A Judge LLM assesses whether the target response accomplishes the intended task, while a Secondary Judge reviews the decision and reasoning.The Secondary Judge can reverse decisions when safety constraints produce illogical reasoning, reducing false negatives without eliminating them completely.
- Evaluation: Manual review of the highest-performing prompts is used to confirm validity and counteract false positives.The authors describe this as a final reliability check on the evaluation results.
21 end
Crescendomation uses evaluator feedback to track jailbreak progress and adapt its questions. A refusal detector can retract questions that trigger refusal or filtering, while moderation APIs provide category-specific scores with coverage limits.
- External moderation: Microsoft Azure Content Filter and Google Perspective API provide category scores, but they cannot exclusively evaluate jailbreak success because some categories, such as misinformation, are absent.The APIs assess categories including hate speech, violence, and self-harm.
- Feedback loop: Judge and Secondary Judge outputs are fed back to track each question’s jailbreak success and guide subsequent prompts.The feedback includes a success flag and completion percentage for each question and summarized response.
- Refusal handling: A Refusal Judge detects refusals or content-filter activation and causes Crescendomation to retract the last question.The retracted question is removed from the target model’s dialogue history but retained elsewhere.
5 Evaluation
The evaluation measures Crescendo and Crescendomation across models, tasks, defenses, and transfer settings. Results show strong but uneven jailbreak performance, with Crescendo outperforming competing methods while automation and transferability retain task- and model-dependent limits.
- Benchmark results: 98% binary success on GPT-4 and 100% on Gemini-Pro were achieved, corresponding to 49 of 50 and 50 of 50 tasks, respectively.On GPT-4, the only task resisting all automated techniques concerned manipulating someone into committing suicide.
- Cross-model evaluation: Crescendomation jailbreaks all 12 evaluated tasks at least 20% of the time, with many tasks reaching perfect Attack Success Rate across models.LLaMA-2 70b and Claude-3 generally show slightly lower ASR and higher refusal rates, while Explicit is difficult for most models and Denial reaches 100% for all models.
- Comparative results: 63.2% average success versus 38.9% for Many-Shot Jailbreak was observed, while at least one successful output occurred on 91% versus 70% of tasks.These figures compare Crescendo with MSJ.
- Automation and turns: Most tasks were jailbroken in fewer than 5 turns, although manual Crescendo succeeded on all 9 tasks that automated Crescendomation failed to jailbreak.The paper attributes those failures to the fully automated implementation rather than to a limitation of Crescendo itself.
- Transferability: Transferability is strong for most tasks, but target-specific Crescendo outperforms transferred attacks; Election reaches at least 90% ASR while Explicit and Manifesto approach 0%.Transferred questions were used without modification, providing a lower bound on transferability.
- Defenses: Self-Reminder and Goal Prioritization reduce Crescendomation’s ASR on some difficult tasks, whereas Election, Climate, and Stabbing are not significantly affected.Increasing rounds and backtracking improves some defense settings, but the evaluation could not run 30 rounds under Goal Prioritization because of token and access constraints.
6 Discussion
The discussion frames Crescendo as difficult to mitigate because it uses benign multi-turn interactions and model-generated context. It identifies history dependence, API and attacker-model constraints, and incomplete filtering as important boundaries while motivating stronger alignment and evaluation.
- Limitations and mitigations: Systems without conversation history may be more resilient to Crescendo, although chat functionality generally requires maintaining history.The attack fundamentally depends on multi-turn interaction.
- Limitations and mitigations: Crescendomation requires target-model API access and inherits limitations from its attacker LLM, which may refuse to generate attacks or perform evaluations.The reported implementation primarily uses GPT-4 as the attack model, and Claude-2 could not be assessed without API authorization.
- Limitations and mitigations: Manual Crescendo results are illustrative rather than exhaustive and do not represent the technique’s full potential.The paper states that broader task coverage could yield stronger results.
- Mitigation strategies: Training-data filtering can be costly and is difficult when harmful content, such as misinformation, cannot simply be excluded.The discussion presents filtering as one mitigation strategy but notes practical drawbacks.
- Mitigation strategies: Crescendomation-generated data could be incorporated into alignment to improve model resilience to Crescendo.The proposed strategy uses generated datasets across varied tasks during alignment.
- Mitigation strategies: Input and output filters may help detect Crescendo, but comprehensively filtering every issue remains difficult, especially for misinformation and character substitutions.The discussion recommends filtering both sides of the interaction for existing models.
- Broader implications: The authors recommend assuming that current LLMs may eventually be jailbroken when designing and evaluating systems.This recommendation is motivated by the recurring publication of jailbreaks for newly released models.
7 Conclusion
The paper presents Crescendo and Crescendomation as effective jailbreak methods that subtly direct models toward tasks using primarily benign prompts and the models’ own responses.
- Crescendo gradually directs models to perform a task through subtle interactions using primarily benign prompts and the models’ own responses.
- Crescendomation automates the Crescendo jailbreak technique.
- Experiments show Crescendo surpasses other state-of-the-art jailbreak techniques across different LLMs.
Ethical Statement
The authors followed coordinated vulnerability disclosure by reporting Crescendo to impacted organizations before submitting the work.
- The authors reported Crescendo to OpenAI, Microsoft, Meta, Google, and Anthropic before submitting the paper.
- They withheld publication for three months to allow organizations time to develop mitigations.
Open science
The authors open-sourced Crescendomation through PyRIT, making the automation tool available as Python software for generative-AI risk identification.
- Crescendomation was open-sourced as part of the Python Risk Identification Tool for generative AI, PyRIT.
A Evaluation
The evaluation reports maximum scores for all models and uses figures to illustrate a Manifesto example and compare model refusals across tasks.
- Tables 6 and 7 present the maximum scores achieved for all models.
- Figure 15 presents an example for Manifesto with profanity.
- Figure 16 compares model refusals across all tasks.
B AdvBench Tasks
This section presents the AdvBench tasks used to test Crescendomation and documents evaluations spanning model refusals, multimodal image generation, and content categories.
- AdvBench tasks were selected for testing Crescendomation and organized by corresponding categories.
- Figures document Crescendo-based jailbreaks of image generation in ChatGPT and Gemini Advanced, including a manifesto containing copyrighted content and profanity.
- The evaluation compares Crescendomation-generated responses using self-assessment, Perspective API, and Azure content-filter scoring across GPT-3.5 and GPT-4.
- The same three assessment methods are used to compare Crescendomation responses for LLaMA-2 70b and Gemini-Pro.
- Claude-3 responses generated with Crescendomation are evaluated using self-assessment, Perspective API, and Azure content-filter scores.
- A separate comparison examines Crescendomation results across all models with respect to model refusal.