PaperlayerView exact revision on arXiv

Source-linked AI summary

Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion

arXiv:2404.02151v4cs.CRcs.AIcs.LGstat.ML

TL;DR

The paper asks whether recent safety-aligned LLMs are robust to simple adaptive jailbreaks. It combines model-specific prompt design with random-search, transfer, and prefilling attacks, finding 100% success across many evaluated models, while noting judge and API-related limitations.

  • Problem

    Safety alignment can be circumvented by adversarial prompts, but the robustness of leading safety-aligned LLMs to simple attacks adapted to model information requires evaluation.

  • Method

    The authors combine model-specific prompt templates with logprob-guided random search, transfer attacks, prefilling, and restricted token search where appropriate.

  • Results

    100% attack success rate was achieved on many leading safety-aligned LLMs, including models accessed through logprobs and Claude models attacked by transfer or prefilling.

  • Takeaways & Limitations

    Adaptive attacks are crucial for robustness evaluation because different target models expose different vulnerabilities and no single attack generalizes across all models.

  • Takeaways & Limitations

    A perfect GPT-4 jailbreak score does not always imply content beneficial to an attacker, and GPT-model API randomness makes random-search refinement less effective.

Abstract

from arXiv · show

We show that even the most recent safety-aligned LLMs are not robust to simple adaptive jailbreaking attacks. First, we demonstrate how to successfully leverage access to logprobs for jailbreaking: we initially design an adversarial prompt template (sometimes adapted to the target LLM), and then we apply random search on a suffix to maximize a target logprob (e.g., of the token "Sure"), potentially with multiple restarts. In this way, we achieve 100% attack success rate -- according to GPT-4 as a judge -- on Vicuna-13B, Mistral-7B, Phi-3-Mini, Nemotron-4-340B, Llama-2-Chat-7B/13B/70B, Llama-3-Instruct-8B, Gemma-7B, GPT-3.5, GPT-4o, and R2D2 from HarmBench that was adversarially trained against the GCG attack. We also show how to jailbreak all Claude models -- that do not expose logprobs -- via either a transfer or prefilling attack with a 100% success rate. In addition, we show how to use random search on a restricted set of tokens for finding trojan strings in poisoned models -- a task that shares many similarities with jailbreaking -- which is the algorithm that brought us the first place in the SaTML'24 Trojan Detection Competition. The common theme behind these attacks is that adaptivity is crucial: different models are vulnerable to different prompting templates (e.g., R2D2 is very sensitive to in-context learning prompts), some models have unique vulnerabilities based on their APIs (e.g., prefilling for Claude), and in some settings, it is crucial to restrict the token search space based on prior knowledge (e.g., for trojan detection). For reproducibility purposes, we provide the code, logs, and jailbreak artifacts in the JailbreakBench format at https://github.com/tml-epfl/llm-adaptive-attacks.

1 INTRODUCTION

The paper examines whether leading safety-aligned LLMs remain robust to jailbreaks and shows that simple attacks adapted to model information can circumvent their defenses.

  • Safety alignment trains models to produce responses judged safe and refuse potentially harmful queries, but adversarial prompts can circumvent it.
  • The authors define adaptive attacks as attacks specifically designed to target a given defense by leveraging information from training details or inference, such as logprobs.
  • Reported comparisons may use different harmful-request sets or semantic judges, and GPT-4 produced around 20% false positives on one model.
  • 100% attack success rate was obtained across leading safety-aligned models using 50 harmful AdvBench requests and GPT-4 semantic judging.The evaluated models include open-weight, proprietary, and adversarially trained systems.
  • The results indicate that adaptive attacks are crucial for accurate robustness evaluations because no single method generalizes across all target models.

2 RELATED WORK

The related work organizes LLM jailbreaking research into manual, direct-search, and LLM-assisted attacks, spanning different threat models and optimization strategies.

  • The section frames these approaches as categories of attacks differing in automation and optimization mechanism.
  • Manual attacks: Manual jailbreaks exploit competing objectives or mismatched generalization between safety training and model capabilities.
  • Direct search attacks: Direct-search attacks automate jailbreak generation with discrete optimization, including gradient-based GCG and gradient-free genetic algorithms.
  • LLM-assisted attacks: LLM-assisted attacks use auxiliary language models to generate or refine jailbreaks, improving query efficiency through iterative or tree-based search.

3 BACKGROUND AND METHODOLOGY

The paper formalizes jailbreaking as finding prompts that induce judged harmful content, then combines model-specific prompt templates, prefilling, transfer, and random-search suffix optimization.

  • 3.1 SETTING: The attacker seeks a prompt P such that the judge labels the target model’s output as a valid jailbreak for harmful request R.
  • 3.1 SETTING: The evaluation uses default system prompts, 50 diverse harmful AdvBench behaviors, GPT-4 semantic judging, and manual inspection of generations.
  • 3.2 METHODOLOGY: Adaptive attacks are customized for each target model but not for each request, combining building blocks according to model characteristics.
  • 3.2 METHODOLOGY: The prompt template combines rules, a harmful request, and an adversarial suffix designed to steer the response toward a specified target beginning.
  • 3.2 METHODOLOGY: Random search modifies contiguous suffix tokens and accepts changes that increase the target token’s first-position log-probability.
  • 3.2 METHODOLOGY: Self-transfer initializes harder-request searches with suffixes found on simpler requests, improving query efficiency and attack success on many LLMs.
  • 3.2 METHODOLOGY: Transfer attacks reuse jailbreaks across models, while prefilling directly supplies a response beginning and avoids iterative optimization for APIs such as Claude.

4 JAILBREAKING LEADING SAFETY-ALIGNED LLMS

The paper evaluates adaptive jailbreak attacks across leading safety-aligned LLMs, tailoring prompt structures and search strategies to each model. These attacks achieve very high or complete success across open-weight, GPT, R2D2, and Claude models, while exposing query-efficiency and API nondeterminism constraints.

  • Llama, Gemma, and R2D2: Self-transfer improves query efficiency and attack success by initializing random search with suffixes found on simpler harmful requests.The procedure is repeated separately for model sizes for best results, although suffixes show some transferability across sizes.
  • Llama, Gemma, and R2D2: 100% ASR is achieved on Llama-2-Chat, Llama-3-Instruct, and Gemma using prompting, random search, and self-transfer.Standard templates alone achieve 0% on Llama-2-Chat, while Prompt + random search reaches 48%; self-transfer raises the composite strategy to 100%.
  • Llama, Gemma, and R2D2: 100% ASR follows on R2D2 when random search is applied to an in-context learning prompt, after the prompt alone reaches 90%.This surpasses the 61% reported for TAP, while the in-context prompt is less effective on models such as Llama-2-Chat.
  • GPT models: GPT models require model-specific strategies: GPT-3.5 reaches 100% with the prompt alone, GPT-4 Turbo reaches 96%, and GPT-4o reaches 100% with customized templates plus search and self-transfer.The default template gives GPT-4 Turbo 28% and GPT-4o 0%, whereas the GPT-4o custom template reaches 72% before search and self-transfer.
  • GPT models: GPT logprob access enables random search, but identical queries can return different log-probabilities despite fixed seed and temperature zero, reducing search effectiveness.The study repeated the same query 1,000 times for GPT-4 Turbo and observed nondeterministic first-token log-probabilities.
  • Claude models: 100% ASR is straightforward on all Claude models with prefilling and the prompt template, even without search; transfer attacks are also highly effective on selected Claude versions.Transfer reaches 98% on Claude 3 Haiku, 100% on Claude 3 Sonnet, and 96% on Claude 3.5 Sonnet; system-prompt placement improves some results.

5 ADAPTIVE ATTACKS FOR TROJAN DETECTION

The paper adapts random search to detect universal trojan strings in poisoned Llama-2-7B models by restricting the token search space using shared model structure. This approach achieved the best overall competition score and the best score for three of five models.

  • The task seeks universal trojan strings: short suffixes that can be appended to any input to jailbreak poisoned models.
  • Because the Llama-2 tokenizer has 32001 tokens, unrestricted random search is inefficient, motivating a smaller candidate pool.
  • Candidate pools are formed by intersecting top-k token sets based on embedding-distance rankings across the five related models.With k = 1000, candidate pools contain 33–62 tokens for models M2–M5; with k = 3000, M1 has 480 candidates.
  • The method searches over five-token triggers, filters invalid candidates, and minimizes average reward-model scores on training batches to promote universality.The objective targets generalization to unseen prompts.
  • Random search on restricted token pools achieved the best score for 3 out of 5 target models and the best overall score.Lower reward-model scores indicate higher success.

6 DISCUSSION, RECOMMENDATIONS, AND LIMITATIONS

The discussion argues that adaptive attacks are necessary for accurate robustness evaluation because different models expose different vulnerabilities. It also notes that judge-based success scores and current defenses impose important limitations.

  • The prompt template alone achieves a 100% success rate on multiple recent LLMs, while random search adds gradient-free suffix optimization when logprobs are available.
  • Self-transfer improves random-search query efficiency and attack success rate.
  • Prefilling provides a simple attack for Claude and can also be applied to open-weight models.
  • The authors recommend combining optimized templates, transferred or randomly searched suffixes, and model-specific techniques rather than relying on static attacks.
  • A perfect GPT-4 judge score does not always imply that generated content is actually beneficial to an attacker.The paper supplements GPT-4 judging with rule-based, Llama-3-70B, and Llama Guard 2 evaluations.
  • Adaptive attacks are crucial because no single method generalizes across all target models, including both open-weight and proprietary systems.

A.3 QUESTIONS AND ANSWERS ABOUT OUR PAPER

The authors use the discussion section to clarify their algorithmic contributions and the scope of their claims. They emphasize that combining complementary techniques enabled their strongest jailbreak results, while some test-time defenses remain outside the paper’s attack coverage.

  • Random search was adapted for both LLM jailbreaking and trojan detection, with task-specific token restrictions helping make trojan search feasible.
  • Gradient information is not necessary for finding effective and transferable adversarial suffixes.
  • Prefilling is presented as an optimization-free method for reliably jailbreaking models such as Claude.
  • The paper introduces a manually written template and self-transfer technique as additional attack components.
  • Combining these techniques produced a 100% attack success rate on leading models including GPT-4o and Claude 3.5 Sonnet.
  • The evaluated random-search attack does not directly bypass SmoothLLM, so attacking that defense would require a new adaptive attack.The authors also note that SmoothLLM increases inference time by more than an order of magnitude and does not support streaming generation.

B EXPERIMENTAL DETAILS

The experimental procedure optimizes adversarial suffixes by random search against a target response-token logprob, using model-specific prompt templates and API settings. The appendix documents the search configuration and templates used across models.

  • Random search appends a 25-token suffix and modifies a few contiguous tokens at a randomly selected position each iteration.
  • Candidate suffixes are accepted when they increase the target token’s log-probability, typically using “Sure” as the target token.
  • The procedure uses up to 10,000 iterations and 10 random restarts, although one restart usually suffices.
  • Algorithm 1 initializes an exclamation-mark suffix, samples contiguous token substitutions, evaluates the target-token log-probability, and retains improved suffixes.
  • The experiments use model-specific system prompts, including separate prompts for GPT, Llama, R2D2, Vicuna, Phi-3, and Mistral settings.
  • The in-context learning template instead maximizes the probability of “Step,” while the desired response begins with a specified target string.
  • The GPT-4o template is split between system and user messages and was refined using manual logprob-guided optimization on one example.

C.1 EFFECT OF THE NUMBER OF TOKENS IN ADVERSARIAL SUFFIXES

The study examines how adversarial-suffix length affects random-search jailbreaking on Gemma-7B. Both target-token logprob and attack success rate follow a U-shaped trend, with 25 tokens performing optimally.

  • 25 tokens performs optimally for Gemma-7B adversarial suffixes under 1,000 random-search iterations.The evaluation tracks average logprob of “Sure” and attack success rate.
  • Adding more tokens worsens the objective value on average, indicating optimization difficulties in the complex loss landscape.
  • Very long suffixes can make the model answer an unrelated request, reducing judged attack success despite producing “Sure” first.A 60-token suffix had the lowest attack success rate in Figure 4.
  • Both average “Sure” logprob and attack success rate follow a U-shaped trend as suffix length changes.
  • Self-transfer initializes searches with a suffix found on a simpler harmful request to reduce the required iterations.
  • On Llama-3-8B, 4,000 random-search iterations take 20.9 minutes on one A100 GPU, but fewer than 10% of behaviors require that many with self-transfer.Most behaviors require fewer than 200 iterations, keeping the total experiment within a few hours.

C.3 FURTHER RESULTS ON CLAUDE MODELS

Further Claude evaluations examine transfer and prefilling attacks across request structures and judges. Transfer attacks reach nearly 100% ASR with 100 restarts, while judge disagreement and Claude 2.1 false positives qualify the results.

  • With 100 restarts, transfer attacks achieve close to 100% ASR on Claude 2.0, Claude 3 Haiku, and Claude 3 Sonnet.
  • Claude attack success is evaluated with GPT-4 and rule-based judges, with additional comparisons involving Llama-3-70B and Llama Guard 2.
  • The Claude ablations vary request structure across user, system, and assistant messages for transfer and prefilling attacks.
  • A transfer attack on Claude 3 Sonnet uses a GPT-4-generated suffix and can elicit follow-up details about the initial response.
  • Claude 2.1 produces rare false positives that receive 10/10 jailbreak scores from GPT-4 and are also accepted by the rule-based judge.
  • The rule-based judge reports at least 90% ASR for every evaluated model, while Llama Guard 2 reaches a low of 86% on Phi-3.

C.7 DIRECT COMPARISON WITH BASELINES

Against GCG, PAIR, and AIM on JailbreakBench behaviors, the adaptive method substantially outperforms plain GCG on Llama-2-Chat-7B. Stricter judging lowers reported ASRs relative to GPT-4-based stopping results.

  • Our method achieves 90% ASR on Llama-2-Chat-7B versus 3% for plain GCG.The comparison uses 100 JailbreakBench behaviors and the Llama-3 70B semantic judge.
  • Reported ASRs generally fall from 100% to the 80%-90% range when using the stricter Llama-3 70B judge instead of GPT-4.
  • The authors expect more random restarts and stopping with the target judge to raise attack success rates to 100%.
  • Against Claude 2.0, Many-Shot Jailbreaking with 128 shots reports 30-45% success, whereas the paper’s prompt and prefilling attacks achieve 100%.

C.8 ADDITIONAL EVALUATION RESULTS

Additional evaluations show that simple prompt templates and random-search suffixes break several less safety-aligned models. The method reaches 100% ASR on all four tested models, while judge and benchmark differences constrain comparisons.

  • The evaluation summary reports ASR using GPT-4 and rule-based judges where available.
  • Nemotron-4-340B reaches 100% ASR with the prompt template without random search or random restarts.
  • Vicuna-13B reaches 100% ASR using only the prompt template, matching more complex methods.
  • Mistral-7B improves from 70% ASR with the prompt alone to 100% with random-search optimization.
  • Phi-3 improves from 90% ASR with the prompt template to 100% with random search.
  • Some comparison figures use different harmful-request sets or judges, and GPT-4 exhibits false positives on a cited model.
Loading 2404.02151v4…