PaperlayerView exact revision on arXiv

Source-linked AI summary

AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, Yuandong Tian

arXiv:2404.16873v2cs.CRcs.AIcs.CLcs.LG

TL;DR

LLMs remain vulnerable to jailbreaking, while manual red-teaming is time-consuming and automated attacks can be unreadable or computationally expensive. AdvPrompter trains an LLM to rapidly generate human-readable adversarial suffixes through alternating optimization, achieving competitive attack performance and enabling adversarial fine-tuning that improves robustness.

  • Problem

    LLMs remain vulnerable to jailbreaking attacks, while manual red-teaming is time-consuming and automated prompt generation can produce unreadable attacks or require expensive discrete optimization.

  • Method

    AdvPrompterTrain alternates efficient adversarial suffix generation with fine-tuning an AdvPrompter LLM to autoregressively produce human-readable prompts for a TargetLLM.

  • Results

    AdvPrompter achieves highly competitive attack success rates with low-perplexity suffixes across open- and closed-source TargetLLMs, while generated adversarial data improves TargetLLM robustness through fine-tuning.

  • Takeaways & Limitations

    Rapid human-readable prompt generation supports efficient multi-shot attacks and provides a dataset for improving alignment robustness against jailbreaking.

  • Takeaways & Limitations

    The paper identifies further alternating training between AdvPrompter and the TargetLLM beyond one cycle as an open direction for robust automated safety fine-tuning.

Abstract

from arXiv · show

Large Language Models (LLMs) are vulnerable to jailbreaking attacks that lead to generation of inappropriate or harmful content. Manual red-teaming requires a time-consuming search for adversarial prompts, whereas automatic adversarial prompt generation often leads to semantically meaningless attacks that do not scale well. In this paper, we present a novel method that uses another LLM, called AdvPrompter, to generate human-readable adversarial prompts in seconds. AdvPrompter, which is trained using an alternating optimization algorithm, generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response. Experimental results on popular open source TargetLLMs show highly competitive results on the AdvBench and HarmBench datasets, that also transfer to closed-source black-box LLMs. We also show that training on adversarial suffixes generated by AdvPrompter is a promising strategy for improving the robustness of LLMs to jailbreaking attacks.

1. Introduction

LLMs remain vulnerable to jailbreaking despite safety alignment, while manual and existing automated red-teaming approaches face scalability or readability limitations. AdvPrompter addresses these issues by rapidly generating readable, adaptive suffixes and supports efficient attacks and adversarial safety fine-tuning.

  • Safety-aligned LLMs remain vulnerable to jailbreaking attacks that bypass safeguards and elicit inappropriate, offensive, or harmful content.
  • Manual red-teaming is time-consuming and prone to blind spots, while automated methods can produce unreadable prompts or require costly discrete optimization.
  • AdvPrompterTrain alternates between generating adversarial suffix targets with AdvPrompterOpt and fine-tuning AdvPrompter using those targets.
  • AdvPrompter generates coherent, instruction-conditioned suffixes that blend naturally with context without requiring human guidance.
  • After training, AdvPrompter uses next-token prediction rather than a new optimization problem for each suffix, enabling efficient multi-shot attacks with improved ASR over one-shot attacks.
  • The method uses only TargetLLM log probabilities rather than back-propagated gradients, making TargetLLM calls faster and more memory efficient.
  • Across white- and black-box settings, AdvPrompter achieves competitive ASR with low suffix perplexity and supports adversarial fine-tuning that improves robustness while maintaining high utility.

2. Preliminaries

The paper formulates jailbreaking as finding suffixes that increase the TargetLLM’s likelihood of a desired positive response while preserving suffix coherence. It distinguishes white-box, black-box, universal, and conditional prompt-optimization settings, motivating instruction-conditioned attacks.

  • 2.1. Problem Setting: Jailbreaking Attacks: A harmful instruction normally elicits a refusal, while appending an adversarial suffix can induce the TargetLLM to produce the desired positive response.
  • 2.1. Problem Setting: Jailbreaking Attacks: Individual prompt optimization minimizes a regularized adversarial loss over suffixes for a given instruction-response pair.
  • 2.1. Problem Setting: Jailbreaking Attacks: The adversarial loss measures desired-response likelihood under the TargetLLM, while the regularizer promotes coherent suffixes under a BaseLLM.
  • 2.1. Problem Setting: Jailbreaking Attacks: The optimal suffix has no closed-form solution, so finding it requires expensive search over the discrete token space.
  • Model-transfer: White-box attacks can use gradients through the TargetLLM, whereas black-box attacks lack direct access to gradients or output log probabilities.
  • Data-transfer: Universal prompt optimization seeks one suffix that jailbreaks multiple harmful instruction-response pairs simultaneously.
  • Data-transfer: Universal suffixes cannot adapt to individual instructions, motivating conditional models that predict suffixes from instructions for more natural and successful attacks.

3. Methodology

AdvPrompter extends adversarial suffix optimization from universal prompts to instruction-conditioned generation, then trains the generator through alternating target construction and regression. Its optimization uses stochastic beam search without TargetLLM gradients, while amortizing future suffix generation across instructions.

  • Conditional adversarial prompting: AdvPrompter qθ maps instructions to adaptive adversarial suffixes, enabling rapid generation for unseen instructions without solving a new optimization problem each time.This is framed as amortized optimization: training cost is paid upfront so later problems from the same distribution become cheaper.
  • Implementation: The method initializes AdvPrompter from BaseLLM and trains a low-rank adapter to reduce GPU memory, while generated suffixes are sampled autoregressively with non-zero temperature.These implementation choices qualify the practical training and generation procedure.
  • AdvPrompterTrain: The alternating scheme addresses difficulties that make end-to-end optimization challenging, including discrete suffixes and autoregressive generation; reinforcement-learning alternatives had limited success.The paper evaluates the alternative approach and reports limited success for the reinforcement-learning setup.
  • AdvPrompterTrain: AdvPrompterTrain alternates between finding target suffixes with AdvPrompterOpt and regressing AdvPrompter onto those targets.The q-step and θ-step form an iterative self-improvement cycle in which improved predictions help produce better future targets.
  • AdvPrompterOpt: AdvPrompterOpt uses stochastic beam search to generate human-readable target suffixes by approximately minimizing a regularized adversarial loss.It maintains b beams and k candidate beams, samples next-token candidates, evaluates their losses, and returns the lowest-loss completed beam.
  • AdvPrompterOpt: AdvPrompterOpt avoids back-propagated gradients through TargetLLM and improves speed relative to AutoDAN while achieving comparable or better attack success when combined with AdvPrompter.Its main computational bottleneck is the k TargetLLM calls required to evaluate candidate losses at each iteration, motivating k ≪ N.

4. Experiments

Experiments evaluate AdvPrompter across whitebox and blackbox attacks on AdvBench and HarmBench, then test adversarial fine-tuning for robustness. The method achieves strong attack success with low-perplexity suffixes, transfers across models and datasets, and improves safety robustness while preserving general capabilities.

  • Experimental setup: AdvPrompter is evaluated on AdvBench and HarmBench across whitebox, model-transfer, and data+model-transfer settings.AdvBench contains 520 harmful instructions, while HarmBench contains 400 textual behaviors with predefined validation and test splits.
  • Whitebox attacks: AdvPrompter achieves high overall ASR with low perplexity, outperforming baselines on ASR@10 in challenging whitebox settings.On AdvBench, its ASR@10 outperforms baselines particularly on Llama2-7b-chat; on HarmBench data transfer, it outperforms attacks directly computed on the test set.
  • Efficiency: Training AdvPrompter takes around 10 hours, but this cost is constant in the number of prompts generated at inference.This scaling advantage supports generating large adversarial datasets for safety fine-tuning, and existing AdvPrompters can be fine-tuned for other models or datasets.
  • Blackbox attacks: In blackbox AdvBench attacks, AdvPrompter is on par with universal model-transfer methods at ASR@1 and outperforms them at ASR@10.The authors attribute this pattern to the adaptability and diversity of generated adversarial prompts.
  • Blackbox attacks: On blackbox HarmBench attacks, AdvPrompter matches the strongest GPT-4 baselines at ASR@10 and significantly outperforms them on GPT-3.5.At ASR@1, it performs on par with TAP-T on GPT-3.5 but falls short on GPT-4.
  • Robustness improvement: Adversarial fine-tuning significantly improves robustness against AdvPrompter prompts while preserving high MMLU and MT-bench scores.Robustness also transfers to GCG and AutoDAN attacks, although the drop is less pronounced for GCG because its high-perplexity suffixes were not targeted during fine-tuning.

5. Discussion and Conclusion

AdvPrompter automates red-teaming with fast, instruction-conditioned, human-readable adversarial prompts and reports stronger attack performance across open- and closed-source TargetLLMs. The generated prompts also support adversarial fine-tuning, while evaluation remains limited to selected defenses.

  • Contributions: AdvPrompter generates prompts faster than previous methods, enabling multi-shot attacks that significantly increase attack success rate over one-shot attacks.Its instruction-conditioned suffixes can adapt to unseen instructions in transfer attacks and remain coherent and human-readable.
  • Results: AdvPrompter achieves higher ASR and faster prompt generation than previous approaches across various open- and closed-source TargetLLMs.The method also reports low perplexity for generated adversarial suffixes.
  • Defensive use: AdvPrompter-generated adversarial prompts can be used to fine-tune TargetLLMs, improving robustness against jailbreaking attacks.The authors present this as a contribution to existing defense mechanisms.
  • Scope and future work: The evaluation covers selected defenses such as safety-enhanced system messages but does not consider advanced defense mechanisms.The authors identify investigation beyond one alternating training cycle as a future direction.

Impact Statement

The paper frames AdvPrompter as both a safety risk and a defensive research tool: it exposes production vulnerabilities while enabling adversarial fine-tuning. Its comparisons emphasize differing attack objectives, model roles, and query budgets across related methods.

  • Impact: AdvPrompter’s strong transfer-attack performance highlights safety concerns for practically deployed LLMs, although the method could also be used by bad actors.The authors describe responsible disclosure before official release.
  • Related approaches: Unlike Perez et al., AdvPrompter enforces a specific harmful instruction rather than generating any prompt that elicits an offensive response.Both approaches use another LLM for automated red-teaming.
  • Related approaches: Unlike LoFT, AdvPrompter fine-tunes an LLM to generate prompts rather than fine-tuning a proxy for the TargetLLM.
  • Related approaches: AdvPrompter includes a learnable component and amortization, unlike BEAST’s beam-search attack.The learnable component is described as crucial to the method.
  • Evaluation regime: The paper focuses on ASR@1 through ASR@10, whereas AmpleGCG reports ASR@50 through ASR@1000 for a different attack regime.The authors associate the smaller query range with practical black-box API scenarios.
  • Alignment robustness: Prior work shows that minor fine-tuning can disrupt safety alignment, and the paper explores adversarial-prompt fine-tuning as a related direction.

B. Method

Differentiating through autoregressive generations creates unstable training because parameter updates alter later-token conditioning. Teacher forcing stabilizes regression to target prompts by conditioning probabilities on the target sequence instead.

  • Autoregressive gradients: Gradient updates through autoregressively generated sequences affect the conditioning of all subsequent tokens, producing unstable training dynamics.
  • Teacher forcing: Teacher-forced loss computes token probabilities by conditioning on the target sequence q rather than on the model-generated sequence.
  • Teacher forcing: Teacher forcing makes training more stable because updates to θ do not alter the conditioning through qθ(x).This makes the teacher-forced objective more effective for regressing AdvPrompter onto target prompts.
  • Intuition: The paper relates the instability to sequentially coupled joints in a robotics analogy, while teacher forcing regresses each token toward its target independently of preceding updates.

B.2. AdvPrompterTrain: Mathematical Derivation

The mathematical derivation motivates AdvPrompterTrain as an alternating optimization scheme: generate candidate suffixes with AdvPrompterOpt, then supervise AdvPrompter using selected targets while avoiding TargetLLM gradients. The implementation emphasizes graybox access and efficient candidate search, with quality improving during training.

  • Derivation: AdvPrompterTrain can be derived from the paper’s objective by rewriting it, relaxing a constraint with a penalty, and reducing the outer problem to supervised AdvPrompter training.
  • Derivation: The inner optimization samples candidate tokens from AdvPrompter’s predicted distribution and then minimizes the adversarial loss over those candidates.This bypasses the additional penalty term by initially minimizing AdvPrompter’s negative log-likelihood.
  • Interpretation: The resulting alternating scheme is motivated as an optimization of the original objective and can also be interpreted through reinforcement learning.AdvPrompter acts as a learned policy, tokens are actions, and the q-step collects new trajectories.
  • Graybox optimization: AdvPrompterOpt uses TargetLLM output probabilities without backpropagated token gradients, making the attack applicable under graybox access.The method argues that token gradients are not crucial for finding semantically meaningful suffixes.
  • Efficiency: AdvPrompterOpt is faster than AutoDAN because it evaluates 48 rather than 512 candidates and applies its two steps once per new token.This speed is important because AdvPrompterOpt serves as an inner loop in AdvPrompterTrain.
  • Training behavior: As AdvPrompter trains, it predicts more promising candidates, and suffix quality eventually matches or surpasses that of expensively generated suffixes.
  • Implementation: With the specified hyperparameters, training averages 16 hours 12 minutes for 7B TargetLLMs and 20 hours 4 minutes for 13B TargetLLMs on two NVIDIA A100 GPUs.These timings correspond to 10 training epochs.

C.2. Evaluation

The evaluation measures attack success, prompt readability, and generation cost across TargetLLMs, with additional model-transfer experiments against GPT-3.5-0613.

  • Evaluation metrics: ASR@k records whether at least one of k attacks succeeds, while perplexity serves as an indicator of adversarial-prompt human-readability.
  • Evaluation metrics: Keyword matching is the default evaluator, searching responses for strings suggesting that the TargetLLM declined the harmful prompt.
  • Evaluation metrics: GPT4-based evaluation uses the StrongREJECT evaluator to score attack success from the harmful instruction and TargetLLM response.
  • Evaluation setup: Figure 3 reports train/test ASR@k, perplexity, and average time to generate one adversarial prompt across open-source TargetLLMs.
  • Transfer evaluation: The model-transfer experiment trains AdvPrompter against different whitebox models and tests transfer attacks on GPT-3.5-0613 using AdvBench.

C.5. Alternative LLM-based evaluations due to false positives

The section examines evaluator errors and supplements keyword matching with StrongREJECT, whose conservative scoring lowers reported metrics while preserving method ordering.

  • Evaluator limitations: Keyword matching can produce false negatives by labeling a harmful response unsuccessful when it begins with “I’m sorry.”
  • Evaluator limitations: Keyword matching can produce false positives when an adversarial suffix changes the instruction’s meaning and the response is unrelated to the original harmful request.
  • Alternative evaluation: StrongREJECT uses GPT4 to assess the harmful instruction and TargetLLM response with a conservative soft score from 0 to 1.Responses formatted as stories or jokes receive scores around 0.5.
  • Alternative evaluation: StrongREJECT computes ASR@k as the maximum score across k runs, whereas the reported table compares ASR@10 and ASR@1.
  • Comparison: Figure 3 averages each reported value over three independent training runs and includes generation time across TargetLLMs.
  • Comparison: StrongREJECT metrics are lower because of conservative scoring, but the relative ordering of attack methods remains unchanged.

C.6. Improving Robustness of Whitebox TargetLLM

AdvPrompter-generated adversarial data is used to improve TargetLLM robustness, while additional experiments examine scaling, decoding, and an amortized AutoDAN variant.

  • Robustness fine-tuning: Adversarial fine-tuning uses synthetic prompts and negative responses to improve robustness while preserving general capabilities.
  • Robustness fine-tuning: ASR decreases from 95.2/67.6 to 2.1/0.6 after adversarial fine-tuning, while MMLU remains high.
  • Robustness fine-tuning: Attacks against the fine-tuned TargetLLM are roughly 3x less effective in ASR than attacks against the original model.
  • Scope: Fully automated safety fine-tuning by alternating AdvPrompter and TargetLLM training is proposed as a future possibility rather than analyzed in depth here.
  • Additional evaluations: AdvPrompter performs competitively in ASR while producing lower-perplexity prompts faster than BEAST under limited time budgets.
  • Trial scaling: Evaluating more sampled prompts improves ASR@k, with performance stabilizing around k = 10 while lower values such as 5 can remain satisfactory.
  • Decoding: Greedy decoding is deterministic but can perform satisfactorily, whereas multinomial sampling provides diversity whose performance depends on temperature.
  • Amortized AutoDAN: Amortized AutoDAN replaces gradient-based candidate selection with AdvPrompter sampling and achieves similar or better ASR at an order-of-magnitude lower runtime.It typically uses 1–2 inner iterations per new token and fewer top-k candidates.

D. Training AdvPrompter with PPO

The section compares PPO with AdvPrompter’s training approach for adversarial suffix generation and finds that PPO does not generalize effectively to the generic attack setup.

  • PPO setup: PPO treats suffix generation as a bandit problem in which policy qθ generates an adversarial suffix and receives a reward based on the adversarial objective.
  • Single-instruction test: PPO is validated by training qθ on a single harmful instruction, where the observed behavior suggests the procedure operates as anticipated.
  • Generic attack setup: In the generic setup, PPO reduces its objective less than AdvPrompterOpt and does not improve attack success rate.
  • Failure analysis: The large action space, including about 32K possible tokens per step for Llama-2, likely contributes to PPO’s difficulty.
  • Failure analysis: Sparse rewards also hinder PPO because only a small subset of suffixes produce successful attacks, unlike typical PPO applications after supervised fine-tuning.
  • Related runtime comparison: AdvPrompter-based candidate selection gives AutoDAN a significant runtime boost in the reported Vicuna-7b setting.

E. Adversarial Suffix Examples

The section presents examples of harmful instructions paired with adversarial suffixes generated by AdvPrompter across multiple target models. The examples span phishing, physical harm, cyber abuse, sabotage, and other harmful requests, including cases where target responses are either harmful or refusals.

  • Transfer-attack examples: Tables 13–16 show harmful instructions paired with AdvPrompter-generated adversarial suffixes in transfer attacks on GPT-3.5, GPT-4, Vicuna-7b, and Mistral-7b.The GPT-3.5 table notes that maximum sequence length can make suffixes appear unfinished while they still function as jailbreaks.
  • Additional examples: The appendix examples also include requests involving reputational harm, data theft, IoT compromise, lock manipulation, and transportation-system disruption.These examples appear in the supplied instruction fragments and generated responses, including false accusations, sensitive-data theft, IoT exploitation, lock manipulation, and sabotage planning.
  • Target responses: Some adversarially prompted outputs provide harmful assistance, while another example refuses to provide bomb-making instructions.The contrast illustrates that the examples include both successful harmful responses and a refusal response.
Loading 2404.16873v2…