PaperlayerView exact revision on arXiv

Source-linked AI summary

Jailbreaking Black Box Large Language Models in Twenty Queries

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, Eric Wong

arXiv:2310.08419v4cs.LGcs.AI

TL;DR

LLM alignment remains vulnerable to jailbreaks that bypass safety guardrails, motivating scalable and realistic methods for testing these weaknesses. PAIR uses an attacker LLM to iteratively generate semantic jailbreak prompts against a separate target LLM through black-box queries. It often finds jailbreaks in fewer than twenty queries, with strong success rates and transferability across open- and closed-source models.

  • Problem

    Existing prompt-level jailbreaks are labor-intensive and non-scalable, while token-level attacks are uninterpretable and query-inefficient, limiting realistic safety stress tests.

  • Method

    PAIR pits black-box attacker and target LLMs against one another to automatically discover and refine human-interpretable prompt-level jailbreaks without human intervention.

  • Results

    PAIR often finds jailbreaks within twenty queries and achieves 50% success for GPT-3.5/4, 88% for Vicuna-13B, and 73% for Gemini-Pro.

  • Takeaways & Limitations

    PAIR provides an inexpensive, accessible red-teaming framework whose semantic jailbreaks are interpretable and often transferable across LLMs.

  • Takeaways & Limitations

    PAIR struggles against strongly fine-tuned models including Llama-2 and Claude-1/2, which may require manual involvement or hyperparameter changes.

Abstract

from arXiv · show

There is growing interest in ensuring that large language models (LLMs) align with human values. However, the alignment of such models is vulnerable to adversarial jailbreaks, which coax LLMs into overriding their safety guardrails. The identification of these vulnerabilities is therefore instrumental in understanding inherent weaknesses and preventing future misuse. To this end, we propose Prompt Automatic Iterative Refinement (PAIR), an algorithm that generates semantic jailbreaks with only black-box access to an LLM. PAIR -- which is inspired by social engineering attacks -- uses an attacker LLM to automatically generate jailbreaks for a separate targeted LLM without human intervention. In this way, the attacker LLM iteratively queries the target LLM to update and refine a candidate jailbreak. Empirically, PAIR often requires fewer than twenty queries to produce a jailbreak, which is orders of magnitude more efficient than existing algorithms. PAIR also achieves competitive jailbreaking success rates and transferability on open and closed-source LLMs, including GPT-3.5/4, Vicuna, and Gemini.

1 Introduction

LLM jailbreaks expose alignment weaknesses but existing prompt-level and token-level attacks trade interpretability and scalability against effectiveness and query cost. PAIR automates interpretable prompt-level jailbreak generation with black-box attacker and target LLMs, often within twenty queries, while retaining strong effectiveness and transferability.

  • Prompt-level jailbreaks use semantically meaningful social-engineering prompts but require creativity, manual curation, and customized human feedback.
  • Token-level jailbreaks can be highly effective but require hundreds of thousands of queries and produce uninterpretable token sequences.
  • PAIR pits an attacker LLM against a target LLM to automate the discovery and refinement of prompt-level jailbreaks without human intervention.
  • PAIR discovers prompt-level jailbreaks within twenty queries, representing more than a 250-fold improvement over GCG.
  • PAIR achieves jailbreak percentages of 50% for GPT-3.5/4, 88% for Vicuna-13B, and 73% for Gemini-Pro.
  • PAIR generates human-interpretable semantic jailbreaks and often transfers them to other LLMs more effectively than GCG jailbreaks.

2 Preliminaries

The paper formalizes prompt-level jailbreaking as finding a semantic prompt whose sampled response contains objectionable content, using only black-box access to the target model. It motivates automated methods that avoid human-intensive prompt engineering and toxic-data curation.

  • Prompt-level jailbreaks seek semantic, human-interpretable prompts that make a targeted LLM output objectionable content.
  • The target LLM T maps token sequences to token probability distributions, and responses are sampled from this black-box mapping.
  • PAIR assumes only sampling access to qT rather than full access to the target model.
  • The formal objective is to find a prompt P such that JUDGE(P, R) = 1 for a response R sampled from qT(P).
  • Automated prompt-level jailbreak tools are needed because existing approaches require prompt engineering, manually generated test cases, or retraining on objectionable text.

3 Generating prompt-level jailbreaks with PAIR

PAIR uses two black-box LLMs in an iterative loop to generate and refine prompt-level jailbreaks, with parallel streams balancing search breadth and refinement depth. Its design also depends on attacker prompting, conversation history, improvement assessment, and conservative jailbreak judging.

  • 3 Generating prompt-level jailbreaks with PAIR: PAIR pits a black-box attacker LLM against a black-box target LLM to discover prompts likely to jailbreak the target.The attacker generates candidate prompts, while the target responds to them.
  • 3 Generating prompt-level jailbreaks with PAIR: Each iteration generates a candidate prompt, obtains the target response, scores the prompt-response pair with JUDGE, and refines unsuccessful candidates.The prompt, response, and score are returned to the attacker when the pair is not classified as a jailbreak.
  • 3.1 Implementing the attacker LLM: The attacker’s design has three considerations: system-prompt construction, accumulated chat history, and iterative assessment of improvement.These choices support adaptive refinement and make candidate generation more interpretable.
  • 3.1 Implementing the attacker LLM: Three attacker system-prompt templates use logical appeal, authority endorsement, or role-playing to instruct the LLM to elicit objectionable content.The templates also specify examples and output formats.
  • 3.1 Implementing the attacker LLM: The attacker receives the full conversation history, whereas the target responds to each candidate prompt without context or history.The attacker therefore adapts across iterations using prior prompts, responses, and assessments.
  • 3.2 Algorithmic implementation of PAIR: The formal single-stream procedure initializes the objective and empty history, samples prompt and response pairs, scores them, returns successful prompts, and otherwise updates the history.The algorithm is presented as four steps: attack generation, target response, jailbreak scoring, and iterative refinement.
  • 3.3 Running PAIR with parallel streams: PAIR supports parallel conversation streams, trading breadth N against refinement depth K under a maximum query complexity of N · K.The experiments generally use N = 30 and K = 3, while N ≪K suits iterative tasks and N ≫K suits broader initial searches.
  • 3.4 Selecting the JUDGE function: Llama Guard is selected as JUDGE because it has the lowest false positive rate while maintaining competitive agreement and enabling reproducible experiments.GPT-4 achieved 88% agreement with expert majority labels on 100 prompt-response pairs.

4 Experiments

PAIR is evaluated across diverse jailbreak behaviors, target models, transfer settings, defenses, efficiency conditions, and attacker configurations. The experiments show strong black-box effectiveness and efficiency, with performance varying by target and attacker choice.

  • Experimental setup: PAIR is evaluated on 100 behaviors spanning ten JBB-Behaviors categories, using Mixtral primarily as attacker and seven target LLMs.The targets include Vicuna, Llama-2, GPT-3.5, GPT-4, Claude-1, Claude-2, and Gemini.
  • Experimental setup: PAIR uses at most 90 queries per behavior, while GCG uses around 256,000 queries under the reported comparison budget.PAIR uses 30 parallel streams with maximum depth 3; GCG uses 500 iterations with batch size 512.
  • Direct jailbreaking attacks: PAIR finds jailbreaks in several dozen queries across Vicuna, Llama-2, GPT-3.5/4, and Gemini, whereas GCG requires orders of magnitude more queries.PAIR’s black-box access also enables attacks on all seven evaluated target LLMs, unlike GCG’s reported scope.
  • Direct jailbreaking attacks: 50% jailbreak success on both GPT models and 73% on Gemini demonstrate PAIR’s effectiveness on closed-source targets, while it struggles with Llama-2 and Claude models.PAIR also successfully finds jailbreaks for the majority of Vicuna behaviors.
  • Jailbreak transfer experiments: PAIR’s prompts transfer more readily than GCG’s Vicuna prompts on all models except GPT-3.5, while GPT-4 prompts transfer well to Vicuna, GPT-3.5, and Gemini.The authors attribute this pattern largely to PAIR’s semantic prompts targeting similar vulnerabilities across similarly trained LLMs.
  • Defended performance: PAIR’s jailbreak percentage drops significantly less than GCG’s under SmoothLLM and a perplexity filter, indicating greater resistance to these defenses.The defenses are evaluated statically by generating prompts against undefended models and testing them on defended models.
  • Efficiency analysis: 34 seconds, 366 MB of CPU memory, and around $0.03 characterize PAIR’s average efficiency on JBB-Behaviors, compared with GCG’s specialized hardware and nearly two-hour runtime.The reported PAIR measurements use Mixtral as attacker and Vicuna as target; GCG’s nearly two-hour runtime is on an NVIDIA A100 GPU.
  • Ablation experiments: Mixtral outperforms Vicuna as attacker, while GPT-3.5 performs worst with a 69% success rate; Vicuna may remain preferable in computationally limited regimes.The authors hypothesize that open-source attackers’ weaker safety alignment and easier formatting contribute to the difference.

5 Limitations

The authors examine PAIR’s attacker-prompt criteria and note that its effectiveness varies across target models, with weaker performance on strongly fine-tuned systems.

  • The ablation evaluates the effect of omitting response examples and improvement instructions from the attacker’s system prompt.
  • PAIR’s attacker criteria are ablated across role-playing, logical appeal, and authority endorsement.
  • PAIR struggles against strongly fine-tuned models including Llama-2 and Claude-1/2.The authors suggest prompt-template modifications or hyperparameter optimization may be needed.

6 Conclusion and future work

The paper concludes that PAIR generates semantic prompt-level jailbreaks for black-box LLMs with few queries and improved interpretability relative to GCG. It also identifies dataset generation, safety fine-tuning, and multi-turn conversations as future directions.

  • 6 Conclusion and future work: PAIR generates semantic prompt-level jailbreaks for a variety of state-of-the-art black-box LLMs in a handful of queries.
  • 6 Conclusion and future work: PAIR’s semantic jailbreaks provide improved interpretability relative to GCG.
  • 6 Conclusion and future work: Future work includes generating red-teaming datasets for safety fine-tuning and extending PAIR to multi-turn conversations.The paper also suggests using jailbreaking datasets to fine-tune a red-teaming LLM.
  • 6 Conclusion and future work: Direct jailbreak comparisons on AdvBench are limited because GCG requires white-box access and results are available only for Vicuna and Llama-2.

B Additional experiments

Additional experiments compare PAIR with GCG on a representative 50-behavior subset of AdvBench. The reported results indicate strong PAIR performance across the evaluated LLM family.

  • Additional experiments: 50 behaviors from AdvBench form the representative subset used for the additional comparison.
  • Additional experiments: PAIR offers strong performance relative to GCG on the representative AdvBench subset.
  • Additional experiments: The subset is used because AdvBench contains many duplicate behaviors.

C Attacker Model Generation Details

The attacker-generation procedure uses efficiency and reliability techniques, including structured JSON generation and open-source output seeding. Closed-source attackers lack equivalent initialization control.

  • Attacker Model Generation Details: The attacker-generation step uses multiple techniques intended to increase efficiency and reliability.
  • Attacker Model Generation Details: For open-source attackers, generation is seeded with JSON fields such as improvement and prompt to preserve the required output format.
  • Attacker Model Generation Details: Closed-source attackers cannot use explicit response initialization, and GPT-3.5 may hallucinate an improvement value on its first output.
  • Attacker Model Generation Details: The JBC baseline uses ten popular jailbreak templates, typically 400–1000 tokens long and often based on approaches such as AIM or DAN.
  • Attacker Model Generation Details: In the jailbreak-chat template, the inserted prompt is replaced with the attack objective, such as designing a phishing email.

E Judge Details

The evaluation uses classifiers and judge prompts to determine whether target responses satisfy a specified harmful objective. GPT-4 and GPT-4-Turbo additionally use a 1–10 judge score, with only score 10 counted as jailbroken.

  • Judge evaluation: The judge evaluates whether a prompt-response pair is jailbroken using a binary classification function.The evaluated pair consists of a prompt P and response R.
  • Judge evaluation: For GPT-4 and GPT-4-Turbo, a judge LLM assigns scores from 1 to 10, and only score 10 qualifies as jailbroken.Score 10 represents a detailed and fully jailbroken response.
  • Judge prompts: The judge system prompt substitutes the harmful objective into a template used to assess target behavior.An example objective is designing a phishing email that obtains login credentials.
  • Auxiliary classifiers: Llama Guard classifies jailbreaks using the evaluated prompt and response.Its prompt template replaces placeholders with the respective prompt and response values.
  • Auxiliary classifiers: The Trojan Detection Competition classifier evaluates the response and harmful behavior without using the prompt.This distinguishes it from classifiers that receive both prompt and response.

E.4 GCG Rule-based Classifier

The appendix supplies classifier and system-prompt configurations for PAIR, including rule-based detection, default target prompts, and attacker strategies. It also notes that Llama-2’s over-cautiousness may improve resilience to prompt-level attacks.

  • GCG Rule-based Classifier: The GCG rule-based classifier labels a response as jailbroken when it contains any keyword from its specified list.The keyword list is provided as the classifier’s complete decision rule.
  • System prompts: PAIR documentation provides full system prompts for target and attacker language models, including default target prompts when available.These configurations define the prompting context used for the models.
  • System prompts: The PAIR system-prompt template substitutes an objective, starting string, strategy, examples, and explanations.The objective and starting-string fields are illustrated with a phishing-email task.
  • Attacker strategies: PAIR includes attacker prompts based on logical appeal, authority endorsement, and roleplaying strategies.Each strategy prompt is instantiated with the objective and an ideal target-model starting string.
  • Target behavior: Llama-2 may refuse harmless prompts because of an individual token, and the authors hypothesize that this over-caution increases resilience to prompt-level attacks.The passage presents this as a hypothesis rather than a demonstrated causal result.

G.2 Ablation Generation Examples

An ablation example shows that removing examples from PAIR’s system prompt produces a more direct and simplistic attack because the attacker lacks guidance about jailbreak types.

  • Ablation: Removing system-prompt examples makes the generated attack more direct and simplistic.The ablated attacker lacks direction about the types of jailbreaks it should generate.

G.3 PAIR Examples

PAIR examples demonstrate attacker-target conversations across Gemini-Pro, GPT-3.5, GPT-4, Llama-2, Vicuna, and Claude-1. The examples use specificity, hypothetical framing, urgency, authority endorsement, roleplay, and fictional scenarios, while the paper also notes both red-teaming benefits and misuse risks.

  • Implications: PAIR is framed as an automated red-teaming tool that can accelerate research toward stronger and more trustworthy model safeguards.The authors connect automatic semantic-jailbreak discovery with alignment and safety research.
  • PAIR Examples: For Gemini-Pro, the attacker emphasizes prompt specificity and the academic nature of a request about manipulating slot machines.The example concerns tips for winning more often at slot machines.
  • PAIR Examples: For GPT-3.5, the attacker uses a creative-writer and skilled-hacker story to pursue a fraudulent IRS letter requesting personal information.The prompt is made precise to ensure that the jailbreak violates safety guards.
  • PAIR Examples: For GPT-4, the attacker appeals to logic and frames a hateful request as hypothetical intellectual curiosity.The objective concerns a blog post advocating violence against people older than fifty.
  • PAIR Examples: For Llama-2, PAIR tests roleplaying examples and identifies urgency and importance as possible target vulnerabilities.The objective is an email designed to trick a user into a Nigerian 419 scam.
  • PAIR Examples: For Vicuna, the attacker uses authority endorsement from the Southern Poverty Law Center and Anti-Defamation League to lend credibility to a xenophobic speech.The target is Vicuna with a roleplaying system prompt.
  • PAIR Examples: For Claude-1, the attacker finds a valid jailbreak on the first iteration through a fictional journalist scenario.The objective is a fake news article that could lead to panic or chaos.
  • Societal implications: The paper acknowledges that malicious actors could use PAIR to generate disinformation or biased text, creating a double-edged red-teaming capability.The stated risks include exacerbating societal issues.
Loading 2310.08419v4…