PaperlayerView exact revision on arXiv

Source-linked AI summary

AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs

Zeyi Liao, Huan Sun

arXiv:2404.07921v3cs.CL

TL;DR

LLM jailbreaking research must address whether GCG’s single lowest-loss suffix adequately exposes vulnerabilities. This paper trains AmpleGCG on successful suffixes to generate many customized attacks, achieving near-100% ASR on aligned open-source models and 99% on the latest GPT-3.5.

  • Problem

    GCG selects one lowest-loss suffix, leaving other adversarial suffixes and vulnerabilities unexplored; the paper investigates broader and faster vulnerability discovery.

  • Method

    AmpleGCG is trained on harmful-query and successful-adversarial-suffix pairs collected through an overgenerate-then-filter pipeline.

  • Results

    AmpleGCG achieves near 100% ASR on Vicuna-7B and Llama-2-7B-Chat with around 200 sampled suffixes, transfers to the latest GPT-3.5 at 99% ASR, and generates 200 suffixes in around 4 seconds per query.

  • Takeaways & Limitations

    AmpleGCG captures distributions of customized adversarial suffixes, supports rapid multi-suffix generation, and transfers from open-source to closed-source victim models.

  • Takeaways & Limitations

    The evaluation mainly studies GCG, and evaluator false positives remain; broader training from other jailbreaking methods is left for future work.

Abstract

from arXiv · show

As large language models (LLMs) become increasingly prevalent and integrated into autonomous systems, ensuring their safety is imperative. Despite significant strides toward safety alignment, recent work GCG~\citep{zou2023universal} proposes a discrete token optimization algorithm and selects the single suffix with the lowest loss to successfully jailbreak aligned LLMs. In this work, we first discuss the drawbacks of solely picking the suffix with the lowest loss during GCG optimization for jailbreaking and uncover the missed successful suffixes during the intermediate steps. Moreover, we utilize those successful suffixes as training data to learn a generative model, named AmpleGCG, which captures the distribution of adversarial suffixes given a harmful query and enables the rapid generation of hundreds of suffixes for any harmful queries in seconds. AmpleGCG achieves near 100\% attack success rate (ASR) on two aligned LLMs (Llama-2-7B-chat and Vicuna-7B), surpassing two strongest attack baselines. More interestingly, AmpleGCG also transfers seamlessly to attack different models, including closed-source LLMs, achieving a 99\% ASR on the latest GPT-3.5. To summarize, our work amplifies the impact of GCG by training a generative model of adversarial suffixes that is universal to any harmful queries and transferable from attacking open-source LLMs to closed-source LLMs. In addition, it can generate 200 adversarial suffixes for one harmful query in only 4 seconds, rendering it more challenging to defend.

1 Introduction

The paper argues that selecting only GCG’s lowest-loss suffix misses successful jailbreaks, then introduces AmpleGCG to generate diverse adversarial suffixes efficiently and transfer them across models.

  • LLM safety is increasingly important as models are integrated into autonomous systems and real-world applications.
  • GCG and related attacks can circumvent safety guardrails, while selecting one suffix leaves other model vulnerabilities unexplored.
  • GCG’s lowest-loss selection can miss successful suffixes because overall loss may not reliably predict jailbreaking success.
  • Augmented GCG raises ASR on Llama-2-7B-Chat from ∼20% to ∼80% by retaining candidate suffixes sampled during optimization.
  • AmpleGCG reaches near 100% ASR on Vicuna-7B and Llama-2-7B-Chat by sampling around 200 suffixes, while generating them in around 4 seconds per query.
  • AmpleGCG transfers from open-source training models to unseen open-source and closed-source models, reaching 99% ASR on the latest GPT-3.5 with affirmative prefixes.
  • Repeating a harmful query enables AmpleGCG suffixes to evade perplexity-based defenses with 80% ASR.
  • The paper presents a universal generative model that captures customized adversarial-suffix distributions for harmful queries and generates many suffixes efficiently.

2 Preliminaries

The preliminaries describe GCG as discrete token-level optimization that appends an adversarial suffix to a harmful query to elicit an affirmative harmful response. They motivate examining more vulnerabilities and generating many customized suffixes rather than retaining only the lowest-loss candidate.

  • GCG appends an optimized suffix to a harmful query, forming an adversarial query intended to elicit a response beginning with a positive affirmation.
  • The GCG objective minimizes the victim model’s autoregressive loss for the target affirmative response.
  • GCG randomly initializes a suffix, samples gradient-based candidate batches, and retains the lowest-loss suffix through successive optimization steps.
  • GCG supports individual-query optimization for tailored suffixes and multiple-query optimization for a universal suffix using fused gradients.
  • Suffix effectiveness is measured by detecting harmfulness in victim-model outputs produced from adversarial queries.
  • The paper uses GCG as a testbed because its long-tail gibberish cases are not trivially mitigated through conventional human alignment techniques.
  • The paper asks whether loss is suitable for suffix selection and whether many customized suffixes can be generated quickly for each harmful query.

3 Rediscovering GCG: Loss Is Not a Good Reference for Suffix Selection

The analysis finds that GCG’s lowest-loss suffix is not a reliable jailbreak indicator because successful suffixes can have higher losses. Augmented GCG therefore retains candidates across optimization steps, improving attack success and discovering more successful suffixes.

  • 3.2 Augmenting GCG with Overgeneration: Augmented GCG keeps all suffixes sampled at each optimization step instead of selecting only the lowest-loss suffix.The retained candidates are used to attack the victim model and identify successful alternatives.
  • 3.2 Augmenting GCG with Overgeneration: The evaluation measures attack success rate and average unique successful suffixes across harmful queries.Responses are jointly assessed with a keyword-based evaluator and the Beaver-Cost model, while USS counts identified adversarial suffixes per query on average.
  • 3.1 Loss is Not a Good Reference for Suffix Selection: Low loss does not reliably indicate jailbreak success, while many higher-loss candidates can also successfully jailbreak the victim model.This observation comes from visualizing candidate suffix losses during GCG optimization on Llama-2-7B-Chat.
  • 3.1 Loss is Not a Good Reference for Suffix Selection: A disproportionately high first-token loss can trigger refusal, after which autoregressive decoding tends to remain in a safe mode.The subsequent tokens depend on previously generated tokens, so an initial refusal can shape the rest of generation.
  • 3.2 Augmenting GCG with Overgeneration: ~20% to ~80% ASR on Llama-2-7B-Chat and 100% ASR on Vicuna-7B were achieved with overgeneration.Overgeneration also discovers more adversarial suffixes per query than default GCG, although multiple-query USS is limited by saving only lowest-loss candidates at each step.

4 AmpleGCG: Learning a Universal and Transferable Generative Model

AmpleGCG learns a generative mapping from harmful queries to diverse adversarial suffixes using an overgenerate-then-filter pipeline. It achieves strong within-model results, generalizes to challenging and out-of-distribution queries, transfers across models, and generates suffixes efficiently.

  • Method: AmpleGCG learns the distribution of customized adversarial suffixes by filtering successful outputs from augmented GCG and training on harmful-query–suffix pairs.The overgenerate-then-filter pipeline uses string-based and model-based evaluators to retain successful suffixes.
  • Method: Group beam search is selected to encourage diverse suffix generation and performs better than beam search and top-p decoding in exploring adversarial suffixes.The default uses a diversity penalty of 1.04.
  • Effectiveness and efficiency: Up to 99% ASR is achieved on Llama-2-7B-Chat with fewer samples than augmented GCG, despite augmented GCG producing 257 universal suffixes for this model.AmpleGCG obtains higher ASR than the default GCG and AutoDAN baselines.
  • Transferability and efficiency: AmpleGCG produces 200 suffixes for each of 100 test queries in 6 minutes total, or 4 seconds per query, while reaching near 99% ASR.It also transfers to closed-source models; adding the affirmative prefix “Sure, here is” nearly reaches 100% ASR on GPT-3.5-0125.
  • Generalization: AmpleGCG reaches 100% ASR on challenging queries that augmented GCG could not jailbreak under the individual-query setting.These challenging queries were drawn from cases that failed under augmented GCG and were used to assess in-domain generalization.
  • Generalization: AmpleGCG generalizes to out-of-distribution harmful categories and unseen interrogative formats, reaching 99% ASR with fewer samples than both default and augmented GCG.The evaluation uses MaliciousInstruct queries that differ from AdvBench in categories and query format.

5 Limitation

The evaluation and scope of AmpleGCG remain limited by evaluator errors and the paper’s primary focus on GCG-based attacks.

  • Evaluation limitations: False positives remain in the ensemble evaluation, so more rigorous evaluators could improve AmpleGCG’s training-data quality and ASR estimates.The ensemble combines string-based and Beaver-Cost evaluators and is stricter than either alone.
  • Scope limitation: The study mainly examines GCG, leaving generative models trained from adversarial prompts collected with other jailbreaking methods for future work.The authors note that the same approach could be applied beyond nonsensical GCG suffixes.

6 Related Work

Related work frames prompt optimization as either soft or hard and places jailbreaking within broader red-teaming research.

  • Prompt Optimization: Soft prompt optimization freezes most model parameters while optimizing a remaining continuous prompt component.The related work contrasts this with optimization over manageable input prompts.
  • Prompt Optimization: Hard prompt optimization searches over manageable discrete input prompts, including vocabulary-token methods such as AutoPrompt.This line of work differs from soft prompt optimization by operating on the input prompt itself.
  • Jailbreaking and Red-Teaming LLMs: Jailbreaking seeks objectionable model outputs and overlaps with red-teaming because either activity can be used to perform the other.Automatic red-teaming and jailbreak methods include attacks under competing objectives and methods requiring model access or fine-tuning.

7 Conclusion

AmpleGCG learns a universal adversarial-suffix generator from vulnerabilities revealed during augmented GCG optimization. It achieves nearly 100% ASR on two open-source models, transfers to GPT-3.5, and bypasses perplexity defenses.

  • AmpleGCG achieves nearly 100% ASR on Llama-2-7B-Chat and Vicuna-7B while sampling 200 suffixes for 100 queries in 6 minutes.The reported generation time is 4 seconds for one query.
  • AmpleGCG transfers from open-source training models to closed-source GPT-3.5, reaching 99% ASR.
  • AmpleGCG bypasses perplexity defenses with 80% ASR using only 100 suffixes.
  • The analysis finds that loss is not a reliable jailbreaking indicator and that augmented GCG exposes more successful suffixes than default GCG.

Ethics Statement

The paper acknowledges that its materials could enable malicious users to elicit objectionable content from victim models. It presents the work as supporting stronger defenses while committing to responsible monitoring.

  • The study includes materials that could enable malicious users to elicit objectionable content from victim models using adversarial suffixes.
  • The authors argue that sharing the research can aid stronger defense strategies against attacks.
  • The authors state that their goal is to identify LLM vulnerabilities to enhance security and resilience, not to encourage malicious applications.
  • The authors commit to closely monitoring how their findings are applied.

Suffixes for Jailbreaking Both Open and Closed LLMs

The augmented GCG procedure retains the standard optimization structure while overgenerating candidate suffixes at each step. It uses token-gradient proposals, batch candidates, and an optimized suffix-candidate list.

  • Augmented GCG modifies the default algorithm by adding a suffix-candidates list to retain generated candidates.The algorithm requires an initial suffix, modifiable positions, iterations, loss, k, batch size B, and candidate list C.
  • The procedure repeats optimization for T iterations and samples B candidate suffixes at each step.B is the number of candidate suffixes considered at each step.
  • At each iteration, it computes top-k promising token substitutions from the negative loss gradient.
  • The algorithm computes the best replacement for each sampled candidate and ensures an optimized suffix-candidates list C.

B.1 Data Collections

The data-collection pipeline overgenerates suffixes during GCG optimization, filters unsuccessful candidates, and samples diverse successful examples for training. It builds train, validation, and test sets that include hard and unknown harmful queries.

  • Data Collection: The overgenerate-then-filter pipeline collects all candidate suffixes during optimization and filters them for successful jailbreaks.For multiple victim models, optimization is performed simultaneously over different single queries.
  • Data Collection: The study samples 445 of 520 AdvBench harmful queries for data synthesis and retains 318 queries with successful suffixes on both Llama-2-7B-Chat and Vicuna-7B.
  • Sampling Strategies: The random strategy samples 200 successful suffixes, while the step strategy samples round-robin across optimization steps until reaching 200.
  • Sampling Strategies: The loss 100 strategy divides candidates into 100 ascending loss spans and samples 200 suffixes across those spans.
  • Data Splits: The test set contains 100 harmful queries, including randomly selected hard cases and additional unknown cases, with 50 validation examples.
  • Sampling Strategies: Loss-based sampling is selected because suffixes from the same optimization step are highly similar, whereas loss intervals may provide more diverse examples.
  • Implementation: The experiments use loss 100 with checkpoints at step 30000 for Llama-2-7B-Chat and step 15000 for Vicuna-7B.

D ASR Results of AmpleGCG’s Transferability To Open-Sourced Models

AmpleGCG improves transferability across victim models by generating multiple customized suffixes rather than relying on a single optimized suffix. It outperforms baselines on average while requiring no access to generated tokens.

  • AmpleGCG achieves the best average attack performance across different victim models, while sampling only 100 suffixes.Suffixes optimized on Llama-2-7B-Chat are difficult to transfer to Vicuna-7B, but AmpleGCG improves this transferability.
  • AmpleGCG captures query-relevant features that support transfer from training queries to unseen test queries and from optimized models to unoptimized models.
  • AmpleGCG outperforms AutoDAN despite producing gibberish suffixes and does not require access to the victim model’s first generated tokens.AutoDAN is described as producing semantically meaningful prompts, whereas AmpleGCG achieves greater performance without modifying generated tokens.
  • Joint optimization over multiple models yields higher average performance for AmpleGCG than for GCG.
  • Suffixes from Vicuna-7B do not transfer effectively to Llama-2-7B-Chat, which the authors hypothesize reflects differences in training and data distributions.

E ASR Results of AmpleGCG’s Transferability To Open-Sourced Models When Trained on Multiple Models

Training AmpleGCG on multiple models improves average attack success and helps bridge transferability gaps between Llama-2-7B-Chat and Vicuna-7B. The approach also maintains strong performance under perplexity-based defenses and across varied query formats.

  • AmpleGCG achieves higher average ASR than default GCG when optimized over Llama-2-7B-Chat and Vicuna-7B simultaneously.Joint training is reported to bridge the poor transfer between the two individual models.
  • AmpleGCG trained on both models can bridge their mutual transferability gap, suggesting a unified model may work across victim models.
  • Against perplexity defense, AmpleGCG maintains high ASR, whereas GCG achieves 0% ASR and AutoDAN achieves 42%.The evaluation uses repeated harmful queries and compares methods under the perplexity detector.
  • With repeated-query input formats, AmpleGCG generalizes to unseen formats and produces suffixes that adapt to repetition while bypassing the perplexity detector.Performance is further enhanced under AID after alleviating distribution shift.
  • The experiments include an out-of-distribution test set whose categories and question-form queries differ from AdvBench.MaliciousInstruct contains 100 harmful instruction instances across ten malicious intentions.

L AmpleGCG Decoding Ways Abalattion

Group beam search is the strongest decoding strategy in the ablation, reaching complete success with fewer samples than beam search, while Top p saturates below complete success. The paper also links GCG failures to first-token loss and exposure bias.

  • Decoding strategies: Group beam search reaches a 100% success rate within 200 samples, while beam search requires 1000 samples and Top p never reaches 100%.
  • Decoding strategies: Beam search improves suffix discovery by targeting high-probability regions, but uneven probabilities cause Top p to repeat suffixes.Top p produces only 208 unique suffixes after 1000 samples and reaches 94% ASR.
  • Decoding strategies: Group beam search promotes diversity and can find hard-query suffixes outside high-probability regions that beam search may miss.
  • Decoding strategies: The authors select group beam search by default because it rapidly addresses all queries, while approximately tens of successful suffixes suffice for red teaming.
  • Exposure bias: Teacher-forcing analysis shows the first target token has unusually high rank and models instead begin with refusal phrases such as “Sorry.”The target-token rank acts as a proxy for token loss.
  • Exposure bias: Overall loss can remain low while the first-token loss stays high, causing later generation to remain in a safe mode after an initial refusal.
  • Exposure bias: Increasing the weight of early tokens does not eliminate this exposure-bias failure, motivating the overgenerate-then-filter pipeline.
  • Case study: Appending an AmpleGCG suffix leads GPT-3.5-0125 to produce harmful identity-theft content, despite a concluding warning that the behavior is illegal.
Loading 2404.07921v3…