PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Language Model Attacks with Perplexity

Gabriel Alon, Michael Kamfonas

arXiv:2308.14132v3cs.CLcs.AIcs.CRcs.LG

TL;DR

Adversarial suffixes can jailbreak aligned language models, creating a need for detection methods. This paper evaluates GPT-2 perplexity and a LightGBM classifier using perplexity and token length. The classifier improves on plain perplexity filtering, but human-crafted jailbreaks remain a key limitation.

  • Problem

    Adversarial suffix jailbreaks can elicit perilous responses from aligned LLMs, while detecting them without misclassifying benign prompts is challenging.

  • Method

    The paper compares GPT-2 perplexity distributions and trains a LightGBM classifier using perplexity and token sequence length.

  • Results

    The classifier improves on plain perplexity filtering and correctly detects most tested adversarial suffix attacks.

  • Takeaways & Limitations

    Perplexity with GPT-2 is an effective initial tool for identifying machine-generated adversarial suffix attacks, especially when combined with sequence length.

  • Takeaways & Limitations

    The approach does not detect human-crafted jailbreaks, and its empirical distribution could shift with more attack-algorithm training time.

Abstract

from arXiv · show

A novel hack involving Large Language Models (LLMs) has emerged, exploiting adversarial suffixes to deceive models into generating perilous responses. Such jailbreaks can trick LLMs into providing intricate instructions to a malicious user for creating explosives, orchestrating a bank heist, or facilitating the creation of offensive content. By evaluating the perplexity of queries with adversarial suffixes using an open-source LLM (GPT-2), we found that they have exceedingly high perplexity values. As we explored a broad range of regular (non-adversarial) prompt varieties, we concluded that false positives are a significant challenge for plain perplexity filtering. A Light-GBM trained on perplexity and token length resolved the false positives and correctly detected most adversarial attacks in the test set.

1 INTRODUCTION

LLM jailbreaks use adversarial suffixes to bypass alignment safeguards, motivating perplexity-based detection. The paper finds that a classifier using perplexity and token length improves detection, though human-crafted jailbreaks remain difficult.

  • Adversarial suffixes append strange strings to prompts, enabling some jailbreaks to elicit responses that aligned models would otherwise refuse.Zou et al.'s method can transfer from an offline Vicuna-7B model to attack several deployed LLMs.
  • The study evaluates perplexity as a detector because ordinary text is optimized for low perplexity.
  • A classifier using perplexity and token sequence length can substantially improve upon plain perplexity filtering.
  • The approach detects machine-generated adversarial suffix attacks but does not succeed with human-crafted jailbreaks.

2 RELATED WORK

Related work frames LLM jailbreak detection within broader research on adversarial robustness, alignment, perplexity-based defenses, and computationally constrained attacks.

  • GCG expands AutoPrompt by searching all possible replacement tokens at each step rather than only one.
  • Adversarial suffix methods create bizarre character sequences, while related black-box attacks rely only on querying models for outputs.
  • Zou et al.'s attacks use long sequences of perceptibly modified tokens, extending adversarial robustness beyond small ℓp perturbations.
  • Alignment procedures can remain vulnerable to adversarial prompting when they do not remove all undesirable behavior.
  • Prior work used GPT-2 perplexity minimization to generate fluent adversarial examples for structured prediction tasks.
  • A windowed perplexity filter blocked 80% of white-box adaptive attacks when the attack algorithm knew the defense was perplexity-based.
  • The study examines perplexity under a more modest computational budget than the 513,000 evaluations used in related work.
  • Jain et al. argue that defenses failing against white-box ℓp-bounded attacks can still be valuable in the language domain.

3 METHODS

The study compares adversarial and regular prompts using GPT-2 perplexity, then trains and thresholds a LightGBM classifier on perplexity and sequence length.

  • The study characterizes prompts by attack intention rather than whether they penetrate a specific LLM's defenses.
  • Both machine-generated and human-crafted attacks are compared with diverse regular non-attack prompts using perplexity distributions.
  • GPT-2 is used to calculate each prompt's perplexity.
  • A single perplexity threshold is inadequate for realistic prompt mixtures, while perplexity plus token sequence length is more effective.
  • Fβ is used for imbalanced detection data, with β controlling whether recall or precision receives greater weight.
  • The datasets are pooled and visualized by perplexity versus sequence length, with adversarial prompts comprising less than 1% of the mixture.
  • LightGBM is trained after splitting adversarial data 50:25:25 and non-adversarial data 70:15:15 across train, validation, and test sets.Validation thresholds are tuned to maximize F2 before testing.
  • Test predictions are displayed as true positives, false positives, and false negatives for further analysis.

4 DATA

The data combine machine-generated and human-designed adversarial prompts with diverse benign datasets. The analysis examines their perplexity distributions and classifier outcomes.

  • 4 DATA: The machine-generated dataset contains 1407 adversarial prompts produced with Vicuna-7B-1.5 and the individual GCG method.
  • 4 DATA: Generating the machine-produced prompts required 50 hours on an A100 GPU with 40GiB of memory.
  • 4.1 ADVERSARIAL PROMPT PREPARATION: Figure 1 shows generated adversarial attacks with notably high perplexity, including a low-perplexity cluster of repeat-exclamation suffixes.
  • 4.2 ADVERSARIAL PROMPT CLUSTERS: Perplexity values could be lower under a reconfigured attack-generation process or with additional training time.
  • 4 DATA: The human-designed dataset contains 79 GPT-4 jailbreak prompts with low perplexity similar to regular text.
  • 4.4 NON-ADVERSARIAL PROMPTS: Benign comparisons include prompts from human-GPT-4 interactions, DocRED, SuperGLUE, SQuAD-v2, Platypus, Tapir, and instructional code search.
  • 4.4 NON-ADVERSARIAL PROMPTS: Figure 2 labels adversarial attacks red, regular prompts green, and repeat-exclamation machine-generated prompts magenta.

5 ANALYSIS

Perplexity separates many machine-generated adversarial suffix attacks from regular prompts, but ordinary prompts with unusual content create substantial overlap. A two-feature GBM classifier improves detection, while human-crafted jailbreaks remain difficult to classify.

  • Machine-generated adversarial prompts occupy a visually separable region from regular prompts, whereas human-crafted jailbreaks overlap ordinary prompts.The human-crafted dataset is small, and its points are disguised among non-adversarial prompts.
  • Regular prompts can produce high perplexity through non-English tokens, mathematical expressions, programming code, misspellings, or irregular symbols.These cases create false positives for plain perplexity filtering.
  • The GBM classifier achieves an F2 score of 95.6% on validation and 94.2% on the test set.F2 weights recall more heavily than precision.
  • After excluding human-crafted observations, the GBM classifier’s F2 score rises from 94.2% to 99.1%.The failed examples originate from human-crafted jailbreaks.
  • The classifier labels Zou-style attacks effectively but is not effective for human-crafted GPT-4 jailbreak prompts.Repeat-exclamation prompts are treated as disqualified rather than real attacks and count as true negatives.
  • A one-dimensional perplexity threshold scores substantially lower than the GBM classifier using perplexity and token sequence length.The tested thresholds were 400 and 1000, with an optimal threshold near 997.

6 LIMITATIONS

The study’s limitations concern data coverage, attack-generation assumptions, and reliance on GPT-2 for perplexity measurement.

  • More data reflecting the full distribution of real LLM interactions would be ideal.
  • The attack algorithm is used in its default version and treated as a black box because its code is openly available.The empirical distribution could shift with more training time.
  • Perplexity is measured strictly with GPT-2.

7 CONCLUSION

GPT-2 perplexity effectively identifies machine-generated adversarial suffix attacks, but plain perplexity risks false positives; adding sequence length substantially mitigates that risk. The classifier nevertheless fails on human-crafted jailbreaks.

  • Nearly 90 percent of more than 1400 adversarial strings had perplexity above 1000, while all relevant examples exceeded 200.These strings were generated using the GCG algorithm.
  • Plain perplexity filtering risks a high false-positive rate because some regular prompts exceed adversarial prompts’ perplexity by an order of magnitude.
  • A classifier combining perplexity with sequence length significantly mitigates the false-positive risk.
  • Sequence length distinguishes many attacks from benign high-perplexity prompts because adversarial suffix attacks are lengthy whereas many regular outliers are short.
  • The classifier could not detect human-crafted jailbreaks, although it may apply to attacks resembling machine-generated adversarial suffixes.

ETHICS STATEMENT

The authors censored components of example attack strings to reduce potential harm because these strings could transfer to other LLMs.

  • Attack-string components were blacked out in the appendix to reduce potential harm.The authors note that the strings could transfer to other LLM models.

A ADVERSARIAL DATASETS

The adversarial dataset comprises generated attack prompts analyzed through perplexity, token-length distributions, and scatter plots. The figures distinguish low-perplexity starter prompts from generated prompts that may break LLM defenses.

  • A.1 GENERATED ATTACK PROMPTS: 1407 attack prompts were generated with code from Zou et al. and analyzed using perplexity and token length.
  • A.1 GENERATED ATTACK PROMPTS: The perplexity and sequence-length distributions are visualized for generated attack prompts.
  • A.1 GENERATED ATTACK PROMPTS: Scatter plots compare perplexity with sequence length for the generated attack prompts.
  • A.1 GENERATED ATTACK PROMPTS: Low-perplexity points represent initial prompts that did not succeed in testing, while the larger right-hand cluster represents generated prompts that may break LLM defenses.
  • A.1 GENERATED ATTACK PROMPTS: Examples of successful adversarial suffix attacks were censored, and their outputs were tested on Vicuna-7B.
  • A.1 GENERATED ATTACK PROMPTS: The adversarial-prompt section also includes additional figures associated with the generated attack analysis.
  • A.2 GPT4 JAILBREAK PROMPTS: The manually constructed GPT-4 jailbreak dataset contains 79 examples, with perplexity and token-length visualizations.

B NON-ADVERSARIAL DATASETS

The non-adversarial datasets cover long-document extraction, yes/no question answering, span-based question answering, logical reasoning, and short prompts, with perplexity and token-length analyses.

  • B.1 DOCRED: DocRED validation contains 998 multi-sentence passages for entity and relation extraction from long documents.
  • B. NON-ADVERSARIAL DATASETS: The section includes perplexity, token-length, and scatter-plot visualizations for several non-adversarial datasets.
  • B.2 BOOLQ: BoolQ validation contains 3270 passages, converted into prompts with a fixed instruction, question, and passage.
  • B.3 SQUAD-V2: SQuAD v2 validation contains 11873 examples, formatted with title, context, and question fields for span-based answering.
  • B.4 OPEN PLATYPUS: Open-Platypus contributes 24926 training prompts from scientific, reasoning, and question-answering datasets without added prefixes or suffixes.
  • B. NON-ADVERSARIAL DATASETS: The shortest prompt is “Hello, AI!.”

B.5 PUFFIN

The Puffin analysis examines perplexity and token-length patterns in human prompts from GPT-4 conversations, including both complete human prompt sequences and conversation-initial prompts. It also identifies short standalone utterances that produce NaN perplexities but can be meaningful in conversational context.

  • Puffin dataset construction: Puffin contains 3000 GPT-4 conversations, yielding 6994 human prompts and a separate set of 3000 initial utterances.The initial-utterance sample was constructed because it more closely matches the structure of the observed attacks.
  • Puffin analysis: The first Puffin analysis plots perplexity, token-length frequency distributions, and perplexity-versus-sequence-length scatter plots for human prompts.
  • Puffin results: Some short, single-word Puffin utterances produce NaN perplexities.The collected examples include brief responses such as “Yes,” “Continue,” and “Hi.”
  • Puffin results: These apparently meaningless standalone utterances can be meaningful when interpreted using prior conversational interactions.
  • Initial Puffin prompts: The second Puffin analysis focuses on the initial human prompt, Puffin[0], using the same perplexity, token-length, and scatter-plot views.
  • Additional prompt datasets: The broader prompt analysis also includes 116862 instruction-following examples and 10000 Python instructional code-search prompts.The instruction-following prompts concatenate instruction and input fields, while the code-search subset is capped at its first 10000 examples.
Loading 2308.14132v3…