PaperlayerView exact revision on arXiv

Source-linked AI summary

Attacking Large Language Models with Projected Gradient Descent

Simon Geisler, Tom Wollschläger, M. H. I. Abdalla, Johannes Gasteiger, Stephan Günnemann

arXiv:2402.09154v2cs.LG

TL;DR

Earlier ordinary gradient attacks had negligible success on aligned-LLM jailbreaks compared with discrete optimization. This paper applies PGD to continuously relaxed token sequences and controls relaxation error, achieving GCG-level attack strength at up to an order of magnitude lower cost.

  • Problem

    Earlier ordinary gradient-based attacks had negligible jailbreak success compared with discrete optimization, while attacking LLM prompts is a combinatorial optimization problem.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The method applies PGD to continuously relaxed token sequences and uses entropy projection to counteract error introduced by the relaxation.

    Source: §3 · p. 2

  • Results

    PGD achieves the same attack strength as GCG up to one order of magnitude faster.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The authors identify lower computational effort as valuable for large-scale evaluation and adversarial training.

    Source: §1 · p. 1

    With advanced ASR measures such as a HarmBench judge, GCG is slightly superior in some cases.

    Source: §6 · p. 5

Abstract

from arXiv · show

Current LLM alignment methods are readily broken through specifically crafted adversarial prompts. While crafting adversarial prompts using discrete optimization is highly effective, such attacks typically use more than 100,000 LLM calls. This high computational cost makes them unsuitable for, e.g., quantitative analyses and adversarial training. To remedy this, we revisit Projected Gradient Descent (PGD) on the continuously relaxed input prompt. Although previous attempts with ordinary gradient-based attacks largely failed, we show that carefully controlling the error introduced by the continuous relaxation tremendously boosts their efficacy. Our PGD for LLMs is up to one order of magnitude faster than state-of-the-art discrete optimization to achieve the same devastating attack results.

1. Introduction

Earlier LLM jailbreak attacks relied on discrete optimization or other LLMs, while ordinary gradient attacks had negligible success against aligned models. The paper presents continuously relaxed PGD as an effective, more efficient alternative to discrete search.

  • The authors revisit gradient-based optimization because earlier ordinary gradient attacks had negligible jailbreak success compared with discrete optimization.Source: §1 · p. 1
  • PGD matches GCG’s attack effectiveness at up to one order of magnitude lower time cost, while outperforming GBDA.The paper highlights this efficiency for large-scale evaluation and adversarial training.Source: §1 · p. 1, Figure 1 · p. 1
  • The contributions include continuous relaxation of token addition and removal and analysis of cost-effectiveness trade-offs in automatic red teaming.Source: §1 · p. 1

2. Background

Attacking an LLM is framed as a combinatorial optimization problem, often approached with gradient-guided search over a continuous extension of one-hot token inputs. For jailbreaks, the attack manipulates an adversarial suffix to increase the likelihood of a harmful response.

  • LLM attacks are formulated as combinatorial optimization over an objective and a set of permissible prompt perturbations.Source: §2 · p. 2
  • Gradient-guided search uses derivatives with respect to one-hot token vectors, implicitly extending the input representation into a continuous domain.Source: §2 · p. 2
  • A jailbreak attack manipulates an adversarial suffix between a fixed prompt or request and the target response to make a harmful response likely.The objective can use cross-entropy over target-response logits and may include auxiliary terms such as a low-perplexity reward.Source: §2 · p. 2
  • GBDA uses Gumbel-Softmax to parameterize relaxed tokens, but this alone does not encourage its limiting categorical distribution to have low entropy.Source: §2 · p. 2

3. Method

The method optimizes a continuous relaxation of token one-hot encodings, then discretizes the result to obtain adversarial tokens. Simplex and entropy projections constrain the relaxed search, while a token mask permits flexible sequence length.

  • The relaxation replaces discrete tokens with a sequence of token-wise simplices, while the attack ultimately seeks a discrete adversarial sequence.Source: §3 · p. 2
  • After each gradient update, simplex projection keeps token weights on the probabilistic simplex and naturally yields sparse solutions.The stated projection runtime is O(|T| log |T|) per token.Source: §3 · p. 2
  • Entropy projection using the Tsallis entropy with q = 2, also called the Gini index, is used to counteract error from continuous relaxation.Repeated entropy projection followed by simplex projection is efficient, but does not guarantee the requested entropy; the paper says that enforcing it did not improve results.Source: §3 · p. 2
  • Each PGD iteration updates relaxed token weights, projects onto the simplex and entropy constraint, discretizes by taking the maximum-weight token, and evaluates the resulting prompt.The procedure retains the best intermediate solution and can stop early.Source: §3 · p. 3
  • A mask m allows tokens to be smoothly inserted or removed by modulating their participation in attention, with mask values clipped to [0, 1] after updates.Source: §3 · pp. 2–3
  • The implementation uses Adam, patience-based reinitialization from the best solution, strong adversarial-string initialization, and scheduled learning-rate and entropy-projection changes.The entropy projection is scaled by the flexible-length mask so removed tokens do not affect it.Source: §3 · p. 3

4. Experimental Results

On behavior jailbreaking, PGD achieves effective attacks at up to an order of magnitude lower computational cost than GCG; on the obedience task, it nearly perfectly succeeds within 100 seconds. The experiments also show PGD narrowing the token search space substantially.

4. Experimental Results

  • Across the benchmark, the study evaluates Vicuna 1.3 7B, Falcon 7B, Falcon 7B instruct, Llama3, and Gemma 2B and 7B against GBDA and GCG.The benchmark uses 100 randomly selected prompts; hyperparameter tuning is performed on 50 Vicuna prompts.Source: §4 · p. 3
  • The experiments report cross-entropy and target probability, which measures the probability of generating the exact target response.Exact target generation corresponds to attack success under this metric.Source: §4 · p. 4
  • PGD achieves almost perfect obedience-task success within 100 seconds and matches GBDA's performance 26 times faster.The attack aims to start the response with the word prohibited by the system prompt.Source: §4 · p. 4, Figure 4 · p. 4
  • PGD achieves meaningful target-response probability where GBDA barely does, matching GCG's attack results at up to one order of magnitude lower computational cost.The comparison covers the behavior-jailbreaking task.Source: §4 · p. 4, Figure 3 · p. 4
  • After projection, PGD narrows the search space from about 65,000 to 10 possibilities per token in an example Falcon-7B-instruct prompt.Figure 5 plots the number of non-zero tokens after projection, aggregated across the adversarial suffix.Source: §4 · p. 5, Figure 5 · p. 5

5. Related Work

The section situates automatic red teaming among LLM-based, discrete-optimization, and gradient-based approaches. It distinguishes this work by applying continuous one-hot relaxations to encoder-only LLMs and using entropy projection to oppose relaxation error.

5. Related Work

  • Automatic red teaming includes LLM-based generation, discrete optimization, and gradient-based optimization, with PGD and GBDA retaining relaxed tokens while PEZ discretizes before probing.It can also be framed as conditional generation of an adversarial suffix given a system prompt and goal.Source: §5 · p. 5
  • PGD extends continuous-domain adversarial optimization, widely used for images, to encoder-only LLMs through relaxed one-hot token encodings.Related work had studied similar relaxations for encoder-decoder architectures.Source: §5 · p. 5
  • Entropy projection is presented as a novel strategy to oppose error from continuous relaxation, making relaxed one-hot optimization practical for encoder-only LLMs.PGD draws on continuous relaxations also used in graph attacks and combinatorial optimization.Source: §5 · p. 5

6. Discussion

The authors report that PGD matches GCG’s attack strength at up to one order of magnitude faster, contrasting with GBDA’s near inability to fool aligned LLMs. With judge-based ASR measures, however, GCG is slightly superior in some cases.

6. Discussion

  • Up to one order of magnitude faster, PGD achieves the same attack strength as GCG, unlike GBDA, which is virtually unable to fool aligned LLMs.The authors characterize PGD as both effective and efficient for LLM attacks.Source: §6 · p. 5
  • For LLMs, PGD is effective and efficient, achieving the same attack strength as GCG up to one order of magnitude faster, unlike GBDA, which is virtually unable to fool aligned LLMs.Source: §6 · p. 5
  • With advanced ASR measures using a judge, GCG is slightly superior in some cases.The authors suggest differing implicit optimization biases may contribute, and describe a follow-up reinforcement-learning attack objective.Source: §6 · p. 5

7. Ethics Statement

The authors weigh the risks of efficient jailbreak attacks against their possible use in adversarial training and large-scale studies. Their assessment rests on a white-box assumption and excludes experiments on publicly deployed assistants.

7. Ethics Statement

  • Efficient jailbreak attacks may pose real-world risks, but the authors estimate benefits outweigh them under a white-box assumption of model access.They cite potential use in adversarial training and large-scale studies as benefits.Source: §7 · pp. 5–6
  • The authors did not test public assistants such as ChatGPT, Claude, or Gemini, and say the attack is not directly applicable to them because it assumes white-box access.Source: §7 · p. 6
  • Researchers may use efficient attacks for adversarial training and large-scale studies, potentially contributing to more robust and reliable real-world models.Source: §7 · p. 6

A. Implementation Details

The implementation details specify safeguards for optimization, tokenization, and learning-rate and entropy-projection schedules. They also describe a patience-based recovery mechanism for stalled optimization.

A. Implementation Details

  • The implementation clips each token’s gradient L2 norm to 20 to prevent exploding gradients from destabilizing Adam’s momentum terms.Source: A. Implementation Details · p. 9
  • Discretization applies argmax, then decodes and re-encodes with the attacked model’s tokenizer to mitigate encode-decode inconsistencies.Source: A. Implementation Details · p. 9
  • After 100 unimproved iterations, the patience mechanism reverts to the best state or, with 50% probability, adopts a promising batch prompt, then reinitializes from the best discretized prompt.Other batch prompts are sampled from Cat(softmax(-ℓ̃_t/0.25)).Source: A. Implementation Details · p. 9
  • Learning rate and entropy regularization ramp up for 100 iterations, followed by cosine annealing with 60-step restarts and terminal learning rate 0.325.Entropy projection strength tracks the learning rate and is adjusted according to the difference between relaxed and discrete losses; the authors conjecture it may be unnecessary when relaxed loss is close to or better than discrete loss.Source: A. Implementation Details · p. 9

B. Obedience

The obedience test asks the assistant not to use a specified word and checks compliance across a dataset of 100 words.

B. Obedience

  • The system prompt instructs the assistant not to use a specified word while otherwise giving helpful, detailed, and polite answers.Source: B. Obedience · p. 9
  • The authors generate a dataset of 100 words to check whether the assistant obeys that restriction.Source: B. Obedience · p. 9
  • The prompt varies the forbidden word across the dataset while keeping the instruction to avoid it.Source: B. Obedience · p. 9

C. Jailbreaking

The jailbreaking section reports behavior-jailbreaking results and cross-entropy scores, alongside example adversarial prompts and model responses.

C. Jailbreaking

  • Figure 6 reports results on the behavior-jailbreaking task and includes cross-entropy scores.Source: Figure 6 · p. 10, C. Jailbreaking · p. 10
  • Table 3 contains 10 example Gemma 7B prompts with adversarial suffixes initialized with 20 exclamation marks, plus model responses.The suffixes are highlighted in bold and blue.Source: Table 3 · p. 11
  • The examples include prompts concerning harmful acts, with some generated responses presenting refusals or safer alternatives.Source: C. Jailbreaking · p. 11

Related papers

Loading 2402.09154v2…