PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, Tong Sun

arXiv:2310.15140v2cs.CRcs.AIcs.CLcs.LG

TL;DR

Readable, human-interpretable jailbreaks and automatic adversarial prompts each have limitations: manual attacks rely on human creativity, while unreadable prompts can be targeted by perplexity filters. AutoDAN uses gradient-based, left-to-right token generation to balance jailbreaking and readability. Its interpretable prompts bypass perplexity filtering and outperform unreadable attacks on generalization and transfer in the reported comparisons.

  • Problem

    Manual jailbreaks depend on human creativity, while unreadable automatic adversarial prompts can be detected with perplexity filters, motivating an automatic attack that is also interpretable.

    Source: §1 · p. 2

  • Method

    AutoDAN uses gradient-based, left-to-right token generation with per-token optimization that balances jailbreaking and readability objectives.

    Source: §3.2 · p. 5

  • Results

    AutoDAN prompts bypass perplexity filters, generalize better to unseen harmful behaviors, and transfer better to black-box LLMs than unreadable attacks.

    Source: §1 · p. 2, §4.1 · p. 8

  • Takeaways & Limitations

    The authors present AutoDAN as a red-teaming method and a lens for understanding jailbreak mechanisms, with potential for tasks such as prompt leaking.

    Source: §5 · p. 10

Abstract

from arXiv · show

Safety alignment of Large Language Models (LLMs) can be compromised with manual jailbreak attacks and (automatic) adversarial attacks. Recent studies suggest that defending against these attacks is possible: adversarial attacks generate unlimited but unreadable gibberish prompts, detectable by perplexity-based filters; manual jailbreak attacks craft readable prompts, but their limited number due to the necessity of human creativity allows for easy blocking. In this paper, we show that these solutions may be too optimistic. We introduce AutoDAN, an interpretable, gradient-based adversarial attack that merges the strengths of both attack types. Guided by the dual goals of jailbreak and readability, AutoDAN optimizes and generates tokens one by one from left to right, resulting in readable prompts that bypass perplexity filters while maintaining high attack success rates. Notably, these prompts, generated from scratch using gradients, are interpretable and diverse, with emerging strategies commonly seen in manual jailbreak attacks. They also generalize to unforeseen harmful behaviors and transfer to black-box LLMs better than their unreadable counterparts when using limited training data or a single proxy model. Furthermore, we show the versatility of AutoDAN by automatically leaking system prompts using a customized objective. Our work offers a new way to red-team LLMs and understand jailbreak mechanisms via interpretability.

1 INTRODUCTION

Manual jailbreaks are readable but rely on human creativity, while automatic attacks can be unreadable and vulnerable to perplexity filtering. AutoDAN combines readability and jailbreak objectives to generate interpretable prompts, with reported gains in filtering resistance, generalization, and transfer.

1 INTRODUCTION

  • The paper motivates the attack by noting that LLM jailbreaks can elicit misaligned content, while automatic gradient-based attacks offer nearly unlimited prompt generation.Such vulnerabilities open LLMs to misuse, including misinformation, malware development, and data leakage.Source: §1 · p. 2
  • AutoDAN is an interpretable gradient-based attack that generates tokens left-to-right, optimizing each token for both jailbreaking and readability.Its two-step token optimization combines preliminary and fine selection.Source: §1 · p. 2
  • AutoDAN’s generated universal prompts jailbreak common LLMs while having lower perplexity than typical benign user prompts.The paper presents this as a challenge to defenses that rely on perplexity filtering.Source: §1 · p. 2
  • AutoDAN generates interpretable prompts from scratch that generalize to unseen harmful behaviors and transfer better to black-box LLMs than unreadable attacks.The prompts exhibit strategies commonly used in manual jailbreaks despite no prior knowledge of those strategies.Source: §1 · p. 2
  • AutoDAN can be extended to system-prompt leakage using a customized objective, a task the paper says adversarial attack literature had not addressed.The authors present this as an example of the method’s minimal prior-knowledge requirement.Source: §1 · p. 2

2 RELATED WORK

Prior work includes human-crafted jailbreaks and gradient-based attacks, while perplexity filters target the unreadability of adversarial prompts. AutoDAN is presented as combining interpretable, readable prompts with automatic generation.

2 RELATED WORK

  • Manual jailbreak research reviews and categorizes strategies, while some studies use jailbreaks to steal system prompts and others attribute vulnerabilities to competing objectives and mismatched generalization.The authors say AutoDAN prompts appear to exploit these two weaknesses despite being generated automatically from scratch.Source: §2 · p. 3
  • Prior jailbreak-ready adversarial methods optimize directly in token space; AutoDAN instead generates tokens left-to-right and considers readability during preliminary selection.Earlier approaches include two-step selection and, in one method, perplexity regularization for readability.Source: §2 · p. 3
  • Perplexity filters detect unreadable adversarial prompts by checking whether a prompt is readable or in-distribution.The cited work describes LLM-based perplexity filtering as apparently robust against evasion.Source: §2 · p. 3
  • Existing adversarial attacks generate unreadable prompts or readable prompts of up to three tokens, whereas AutoDAN generates interpretable prompts from scratch.The paper characterizes AutoDAN as bridging adversarial and manual jailbreak attacks.Source: §2 · p. 3, Figure 2 · p. 3

3 AutoDAN: INTERPRETABLE ADVERSARIAL ATTACKS

AutoDAN generates readable jailbreak prompts through two nested loops: an inner loop optimizes one token, and an outer loop generates tokens left-to-right. Both loops balance target-response likelihood against readability.

3.1 TWO OBJECTIVES: JAILBREAKING AND READABILITY

  • Jailbreaking selection seeks a token that raises the model’s likelihood of the target response, while readability uses the model’s next-token distribution.The objectives are defined within a prompt template that appends an adversarial suffix to the user request.Source: §3.1 · p. 4

3.2 INNER LOOP: SINGLE TOKEN OPTIMIZATION

  • The inner loop uses two-step selection to find a token that makes the adversarial prompt more likely to jailbreak while remaining readable.Preliminary selection uses a gradient-based proxy that combines the jailbreaking gradient with the next-token readability distribution; fine selection evaluates candidates using their objective values.Source: §3.2 · p. 5
  • Adding the jailbreaking objective to readability logits adapts to token-distribution entropy: it shifts rankings more when many readable options exist and less when few exist.The paper contrasts its token-specific additions with adding a fixed constant to all logits.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3 OUTER LOOP: LEFT-TO-RIGHT ADVERSARIAL PROMPT GENERATION

  • The outer loop initializes a token, optimizes it until convergence, appends it to the prompt, and then optimizes the next token.It can generate long sequences, such as over 200 tokens, up to a predefined maximum number of steps.Source: Figure 3 · p. 4, §3.3 · p. 6
  • The single-token search stops when any two generated top-1 tokens match, and the authors say its objective improves strictly until convergence.The worst-case bound is |V| iterations, though the paper reports convergence within a few steps in practice.Source: §3.3 · p. 6
  • AutoDAN can incorporate prior knowledge through target-response formatting or manually crafted prompt prefixes or suffixes to guide generated content or style.The paper gives translated target responses and a manually added “You are to act as Niccolo” prefix as examples.Source: §3.3 · p. 7

4 EXPERIMENTS

Across its experiments, AutoDAN produces readable prompts that retain attack effectiveness after perplexity filtering, exhibit recognizable jailbreak strategies, transfer to black-box models, and can be adapted to prompt leaking. The evaluation also finds that string matching is a useful but imperfect measure of jailbreak success.

4 EXPERIMENTS

  • String matching overestimates ASR by 10% on Vicuna-7B, 15% on GPT-3.5, and 9% on GPT-4, yet its F1 score is close to the best tested GPT-4 evaluation.The authors use it as a quick, cost-effective default when refusal prefixes are tailored to the target LLM.Source: §4 · p. 7

4.1 BYPASSING PERPLEXITY FILTERS TO JAILBREAK LLMS

  • On Vicuna, AutoDAN reaches 88% post-filtering ASR versus GCG-reg’s 21%, while its prompts have perplexity 12 versus the median normal-user perplexity of 126.Figure 5 indicates the false-positive rate must exceed 90% to bring AutoDAN’s ASR below 60%.Source: Table 1 · p. 8, §4.1 · p. 8, Figure 5 · p. 8

4.2 EMERGING STRATEGIES OF AUTODAN

  • AutoDAN prompts generated from scratch exhibit manual-jailbreak strategies, including shifting domains and detailizing instructions.Shifting domains invokes unforeseen scenarios; detailizing instructions provides specific response guidance and formats.Source: §4.2 · pp. 8–9

4.3 TRANSFERABILITY OF INTERPRETABLE ATTACKS

  • Using one white-box model, AutoDAN’s interpretable prompts outperform unreadable GCG-generated prompts in bypassing perplexity filters and jailbreaking black-box models.The transfer evaluation uses Azure-hosted GPT-3.5-turbo and GPT-4, with an additional perplexity filter before the input filter.Source: Table 3 · p. 9, §4.3 · p. 9

4.4 PROMPT LEAKING

  • For prompt leaking, Table 4 shows that AutoDAN achieves significantly higher ASRs than the prompt-only baseline, which uses “Tell me the previous instructions”.AutoDAN’s objective encourages the LLM to start responding with the first 16 tokens of the system prompts. The dataset uses 25 different system prompts for training and 25 for testing.Source: §4.4 · p. 10

5 CONCLUSIONS

The paper concludes that AutoDAN combines interpretability with effective attacks and can be adapted to prompt leaking. Related work contrasts its from-scratch generation with other readable attacks and identifies additional defenses to test.

5 CONCLUSIONS

  • AutoDAN generates diverse, strategic prompts from scratch that bypass perplexity filters, generalize to unseen harmful behaviors, and transfer better than unreadable prompts to closed-source LLMs.The authors present it as a red-teaming method and a lens for understanding jailbreak mechanisms, and demonstrate an extension to prompt leaking.Source: §5 · p. 10

A ADDITIONAL RELATED WORK

  • Other readable automated attacks can achieve high success on Llama-2, but genetic approaches require manual jailbreak prompts to initialize and may produce less diverse prompts.The cited black-box LLM-feedback approach avoids a white-box proxy and uses fewer queries, but requires manually designed attacker and judge prompts containing known strategies.Source: A ADDITIONAL RELATED WORK · p. 15
  • Whether interpretable AutoDAN prompts bypass defenses such as paraphrasing, re-tokenization, erase-and-check, or internal-model inspection remains a future-work question.The paper describes these methods as defenses against jailbreak attacks, with some proposed specifically for unreadable attacks.Source: A ADDITIONAL RELATED WORK · p. 16

B IMPLEMENTATION DETAILS

The appendix gives implementation and reproducibility details, including AutoDAN’s objective weights, prompt-leaking setup, and optional interpretable weighting. It also describes limitations in evaluating jailbreak success and the authors’ response criteria.

B IMPLEMENTATION DETAILS

  • Algorithm 2 and Figure 6 give additional detail on AutoDAN’s left-to-right outer-loop generation process and workflow.The experiments use batch size 512 and default sampling with temperature 1.0; the authors omit generation-quality techniques such as repetition penalty to simplify analysis.Source: B IMPLEMENTATION DETAILS · p. 16, §B.1 · p. 16, Figure 6 · p. 17

B.1 HYPERPARAMTERS

  • The authors set w1 = 3 and w2 = 100 to balance readability and jailbreak objectives in preliminary and fine selection, respectively.The weights are log-scaled; larger weights can increase attack success while potentially hurting readability, and the same settings are used across experiments.Source: §B.1 · p. 17

B.2 INTERPRETABLE WEIGHT FOR PRELIMINARY SELECTION

  • An optional top-p-style rule chooses w1 so the B preliminary candidates’ predicted probabilities sum near a desired p∗.A larger p∗ favors more readable tokens for fine selection; the authors report no significant ASR or readability differences with this method.Source: §B.2 · p. 17

B.3 PROMPT-LEAKING DATASET

  • The dataset contains 140 user-customized prompts designed for various tasks, collected from Awesome ChatGPT Prompts2. Specific instructions were prepended to simulate attempts by app providers to protect system prompts from leaking.Source: §B.3 · p. 17
  • For prompt leaking, AutoDAN is trained to make the model output match the first k system-prompt tokens, and a universal prompt is tested on held-out examples.The authors test k = 16 and k = 64 and report no significant difference; this objective requires no manual design.Source: §B.3 · p. 18, Figure 7 · p. 19

B.4 PSEUDOCODE

  • The appendix shares PyTorch-style pseudocode for key steps and says full code will follow internal legal review.Source: §B.4 · p. 18

C IMPROVED EVALUATION OF JAILBREAK SUCCESS

  • A successful jailbreak is defined by an engaging, on-topic response containing harmful content, but the authors note that edge cases make rigorous evaluation challenging.The appendix discusses human annotation as ground truth and illustrates difficult cases in Table 6.Source: C IMPROVED EVALUATION OF JAILBREAK… · p. 19

C.1 CHALLENGES IN EVALUATION

Rigorously judging jailbreak success is difficult because harmfulness, user intent, and the required level of actionable detail can be ambiguous. The section identifies edge cases that complicate human labeling and suggests ways to clarify some requests and responses.

C.1 CHALLENGES IN EVALUATION

  • Ambiguous harmfulness makes judgments audience-dependent, as a fake-news response could seem ludicrous to some readers but believable and harmful to others.The authors also note that verifying some responses requires specific domain knowledge.Source: §C.1 · p. 19
  • A response to an underspecified request may not match the user’s intent, yet labeling it a jailbreak failure can also be inappropriate when the requested harmful act is unclear.The example contrasts an unspecified bomb request with a baking-soda-and-vinegar bomb response.Source: §C.1 · pp. 19–20
  • Vague responses can signal harmful intent without providing specific, feasible guidance, leaving the level of detail required to qualify as harmful unresolved.The authors describe this threshold as a non-trivial issue.Source: §C.1 · p. 20
  • More detailed requests may clarify intent, while multi-turn follow-up can prompt a vague response to elaborate; the authors report that models tend to continue answering malicious questions after entering a “jailbroken mode.”The authors suggest adding details to brief AdvBench requests and prompting the LLM to clarify its response.Source: §C.1 · p. 20

C.2 EVALUATION METHODS

The section compares string matching, human annotation, GPT-4-based judging, and a pretrained classifier as ways to assess jailbreak success. These methods use refusal prefixes, human labels, prompted evaluation, or harmful-content classification, respectively.

C.2 EVALUATION METHODS

  • String matching classifies a response as a failure when it begins with a refusal prefix from a fixed list, and as a success otherwise.The authors note that refusal phrasing varies across LLMs and use one refusal set across model families.Source: §C.2 · pp. 20–21
  • Two authors manually label responses as successful, failed, or ambiguous, using 0.5 when they disagree or either selects the ambiguous label.The labels are 1 for success, 0 for failure, and 0.5 for ambiguous cases.Source: §C.2 · p. 21
  • GPT-4 evaluates jailbreak success using either a single harmfulness instruction or a more complex prompt that includes an in-context example.The simple template is shown in Figure 8.Source: §C.2 · p. 21, Figure 8 · p. 21
  • A pretrained BERT classifier trained on HH-RLHF detects harmful content as a faster, cheaper alternative to GPT-4 evaluation.The authors use a detector from Huang et al. (2023a).Source: §C.2 · p. 21

C.3 META-EVALUATION RESULTS

Across three LLMs, complex-prompt GPT-4 evaluation aligns best with human labels, while string matching provides a cheaper alternative and classifier-based evaluation performs poorly. The comparison also exposes edge cases and greater evaluation difficulty for GPT-4-generated responses.

C.3 META-EVALUATION RESULTS

  • Around 10% of responses (48 of 600) were hard-to-label edge cases, revealing difficulty in determining harmfulness even for human labelers.The meta-evaluation dataset combined ten suffixes with 20 unseen behaviors across three LLMs.Source: §C.3 · p. 21
  • 90%, 85%, and 51% F1 on Vicuna-7B, GPT-3.5, and GPT-4, respectively, made string matching the second-highest evaluator, though it overestimated ASR by 10%, 15%, and 9%.The authors say it still reflects underlying true ASR and can be a fast, cheap alternative when refusal-prefix variants are covered.Source: §C.3 · pp. 22–23, Table 7 · p. 23
  • Approximately 92% F1 on Vicuna-7B and GPT-3.5, but 53% on GPT-4; complex-prompt GPT-4 evaluation measures ASR with less than 4% error, while the simple template performs worse than string matching.The simple-template F1 score on GPT-4 is 11%.Source: §C.3 · p. 22, Table 7 · p. 23
  • Classifier-based evaluation achieved only around 10% F1 across the three LLMs, possibly because AutoDAN responses differ from the HH-RLHF training distribution.The authors leave training a classifier specifically to identify refusal messages for future work.Source: §C.3 · p. 23
  • GPT-4-generated responses were harder to evaluate across all methods, in part because capable models may follow a request before switching to a harmless response.Less capable models more often reject outright or comply using their own world knowledge.Source: §C.3 · p. 23
  • String matching was the default evaluator, while GPT-4-based evaluation was used for selected additional jailbreak, transfer, Llama2, and hyperparameter results.The authors report that GPT-4-based evaluation approaches human-level performance but requires API calls.Source: §C.3 · p. 23

D.1 JAILBREAKING LLAMA2-CHAT

On Llama-2-7B-chat, AutoDAN with w2 = 150 trades slightly lower training ASR for much higher test ASR than GCG. AutoDAN suffixes have lower perplexity than GCG’s, though Llama-2 prompts remain less readable than prompts for other tested models.

D.1 JAILBREAKING LLAMA2-CHAT

  • 35.0% vs 11.7% test ASR, despite 30.8% vs 33.3% training ASR, favors AutoDAN with w2 = 150 over GCG on unseen behaviors.Results average ten runs; each suffix was tested on 25 previously unseen harmful behaviors after training on one of ten AdvBench behaviors.Source: §D.1 · p. 23, Table 8 · p. 24
  • AutoDAN’s adversarial suffixes have much lower perplexity than GCG’s, while direct perplexity regularization does not let GCG match both ASR and perplexity.The passage reports perplexities of 3e5 vs 7e2 for AutoDAN and GCG, respectively.Source: §D.1 · p. 24
  • AutoDAN suffixes on Llama-2 have much higher perplexity than those on Vicuna, Guanaco, and Pythia, indicating some readability is sacrificed to jailbreak the heavily censored model.The reported perplexities are 7e2 vs < 1e2.Source: §D.1 · p. 24
  • For Llama-2, AutoDAN’s test ASR with suffixes generated from multiple harmful behaviors is worse than with an individual behavior, possibly due to implementation issues.The authors state they aim to address the issue in a future version.Source: §D.1 · p. 24

D.2 MORE TRANSFERABILITY RESULTS

With a perplexity filter placed before the target GPTs, AutoDAN’s transferred attacks remain effective, while the filter blocks GCG attacks and partially mitigates GCG-reg attacks.

D.2 MORE TRANSFERABILITY RESULTS

  • The perplexity filter blocks all GCG attacks and partially mitigates GCG-reg attacks, but is ineffective against AutoDAN.The results concern transfer attacks against perplexity-filter-protected GPTs.Source: §D.2 · p. 24, Table 9 · p. 24
  • Figure 10 visualizes Table 9’s transferability results.Table 9 reports transfer attack success rates on perplexity-filter-protected GPTs.Source: Figure 10 · p. 24, Table 9 · p. 24

D.3 COMPLEXITY ANALYSIS

AutoDAN’s token update and GCG’s suffix update have the same computational complexity, but AutoDAN’s per-step time depends on prompt length. AutoDAN typically reaches peak ASR within 50 tokens, while its sequential generation makes it slower than GCG for multiple behaviors.

D.3 COMPLEXITY ANALYSIS

  • AutoDAN’s token update has the same computational complexity as GCG’s suffix update, modulo varying prompt lengths.AutoDAN’s preliminary selection uses batch-one forward and backward propagation to the new token, followed by batch-B forward propagation; GCG has similar operations.Source: §D.3 · p. 25
  • AutoDAN’s per-step time rises affinely with sequence length, taking less time than GCG below 20 tokens and more above 20.At lengths of 100 to 150 tokens, its per-step time doubles GCG’s; at 20 tokens, it is slightly faster.Source: §D.3 · p. 25
  • AutoDAN takes around four iteration steps to update and finalize each new token, nearly constant across sequence lengths, weight settings, and tested LLMs.Figure 12 reports convergence speed for generating new tokens.Source: §D.3 · p. 25
  • On Vicuna-7B, AutoDAN reaches peak ASR within 50 tokens in most cases, with total time to peak similar to GCG for the same number of steps.The reported peak is approximately 200 steps.Source: §D.3 · p. 26, Figure 13 · p. 26
  • For multiple behaviors, AutoDAN takes longer than GCG because it does not use GCG’s technique of gradually adding behaviors during training.Both methods’ per-iteration time increases linearly with the number of behaviors.Source: §D.3 · p. 26

D.4 HYPERPARAMETER ANALYSIS

AutoDAN’s jailbreak objectives in both selection stages are necessary for high ASR, while readability selection in the preliminary stage is necessary for readable prompts. The appendix also provides prompt examples and states the study’s security-oriented intent.

D.4 HYPERPARAMETER ANALYSIS

  • Removing the jailbreak objective from either selection stage makes ASR almost zero, while removing preliminary readability selection raises prompt perplexity above 100.The analysis finds p1 relatively insensitive within 0.1 to 0.9 when paired with a suitable w2; w2 has a sweet spot from 50 to 100 and affects readability, ASR, and prompt style.Source: §D.4 · p. 26, Figure 14 · p. 27, Figure 15 · p. 27

D.5 QUALITATIVE EXAMPLES

  • The qualitative examples include AutoDAN prompts categorized by strategy and prompts from GCG, GCG-reg, AutoDAN-prefix, and Semi-AutoDAN.Semi-AutoDAN adds manual prefixes and suffixes during optimization to alter generated prompt style or content.Source: §D.5 · p. 27, Table 10 · p. 28, Table 11 · p. 29

E ETHICAL STATEMENT

  • The authors frame the study as security-oriented red-teaming intended to expose LLM vulnerabilities and support the development of defenses.They state that the intent is not malicious.Source: E ETHICAL STATEMENT · p. 28
Loading 2310.15140v2…