Detecting Language Model Attacks with Perplexity

• Adversarial suffixes append strange strings to prompts, enabling some jailbreaks to elicit responses that aligned models would otherwise refuse.Zou et al.'s method can transfer from an offline Vicuna-7B model to attack several deployed LLMs.Source: §1 · p. 1
• The study evaluates perplexity as a detector because ordinary text is optimized for low perplexity.Source: §1 · p. 1
• A classifier using perplexity and token sequence length can substantially improve upon plain perplexity filtering.Source: §1 · p. 1
• The approach detects machine-generated adversarial suffix attacks but does not succeed with human-crafted jailbreaks.Source: §1 · p. 1

• GCG expands AutoPrompt by searching all possible replacement tokens at each step rather than only one.Source: §2 · p. 2
• Adversarial suffix methods create bizarre character sequences, while related black-box attacks rely only on querying models for outputs.Source: §2 · p. 2
• Zou et al.'s attacks use long sequences of perceptibly modified tokens, extending adversarial robustness beyond small ℓp perturbations.Source: §2 · p. 2
• Alignment procedures can remain vulnerable to adversarial prompting when they do not remove all undesirable behavior.Source: §2 · p. 2
• Prior work used GPT-2 perplexity minimization to generate fluent adversarial examples for structured prediction tasks.Source: §2 · p. 2
• A windowed perplexity filter blocked 80% of white-box adaptive attacks when the attack algorithm knew the defense was perplexity-based.Source: §2 · p. 2
• The study examines perplexity under a more modest computational budget than the 513,000 evaluations used in related work.Source: §2 · p. 2
• Jain et al. argue that defenses failing against white-box ℓp-bounded attacks can still be valuable in the language domain.Source: §2 · p. 3

• The study characterizes prompts by attack intention rather than whether they penetrate a specific LLM's defenses.Source: §3 · p. 3
• Both machine-generated and human-crafted attacks are compared with diverse regular non-attack prompts using perplexity distributions.Source: §3 · p. 3
• GPT-2 is used to calculate each prompt's perplexity.Source: §3 · p. 3
• A single perplexity threshold is inadequate for realistic prompt mixtures, while perplexity plus token sequence length is more effective.Source: §3 · p. 3
• Fβ is used for imbalanced detection data, with β controlling whether recall or precision receives greater weight.Source: §3 · p. 3
• The datasets are pooled and visualized by perplexity versus sequence length, with adversarial prompts comprising less than 1% of the mixture.Source: §3 · p. 3
• LightGBM is trained after splitting adversarial data 50:25:25 and non-adversarial data 70:15:15 across train, validation, and test sets.Validation thresholds are tuned to maximize F2 before testing.Source: §3 · p. 4
• Test predictions are displayed as true positives, false positives, and false negatives for further analysis.Source: §3 · p. 4

4 DATA

• The machine-generated dataset contains 1407 adversarial prompts produced with Vicuna-7B-1.5 and the individual GCG method.Source: §4.1 · p. 4
• Generating the machine-produced prompts required 50 hours on an A100 GPU with 40GiB of memory.Source: §4.1 · p. 4

4.1 ADVERSARIAL PROMPT PREPARATION

• Figure 1 shows generated adversarial attacks with notably high perplexity, including a low-perplexity cluster of repeat-exclamation suffixes.Source: §4.2 · p. 4, Figure 1 · p. 4

4.2 ADVERSARIAL PROMPT CLUSTERS

• Perplexity values could be lower under a reconfigured attack-generation process or with additional training time.Source: §4.2 · p. 5

4 DATA

• The human-designed dataset contains 79 GPT-4 jailbreak prompts with low perplexity similar to regular text.Source: §4.3 · p. 5

4.4 NON-ADVERSARIAL PROMPTS

• Benign comparisons include prompts from human-GPT-4 interactions, DocRED, SuperGLUE, SQuAD-v2, Platypus, Tapir, and instructional code search.Source: §4.4 · p. 5
• Figure 2 labels adversarial attacks red, regular prompts green, and repeat-exclamation machine-generated prompts magenta.Source: Figure 2 · p. 5

• Machine-generated adversarial prompts occupy a visually separable region from regular prompts, whereas human-crafted jailbreaks overlap ordinary prompts.The human-crafted dataset is small, and its points are disguised among non-adversarial prompts.Source: §5 · p. 6
• Regular prompts can produce high perplexity through non-English tokens, mathematical expressions, programming code, misspellings, or irregular symbols.These cases create false positives for plain perplexity filtering.Source: §5 · p. 6
• The GBM classifier achieves an F2 score of 95.6% on validation and 94.2% on the test set.F2 weights recall more heavily than precision.Source: §5 · p. 6
• After excluding human-crafted observations, the GBM classifier’s F2 score rises from 94.2% to 99.1%.The failed examples originate from human-crafted jailbreaks.Source: §5 · p. 6
• The classifier labels Zou-style attacks effectively but is not effective for human-crafted GPT-4 jailbreak prompts.Repeat-exclamation prompts are treated as disqualified rather than real attacks and count as true negatives.Source: Table 1 · p. 7
• A one-dimensional perplexity threshold scores substantially lower than the GBM classifier using perplexity and token sequence length.The tested thresholds were 400 and 1000, with an optimal threshold near 997.Source: §5 · p. 6, Table 2 · p. 7

• More data reflecting the full distribution of real LLM interactions would be ideal.Source: §6 · p. 7
• The attack algorithm is used in its default version and treated as a black box because its code is openly available.The empirical distribution could shift with more training time.Source: §6 · p. 7
• Perplexity is measured strictly with GPT-2.Source: §6 · p. 7

• Nearly 90 percent of more than 1400 adversarial strings had perplexity above 1000, while all relevant examples exceeded 200.These strings were generated using the GCG algorithm.Source: §7 · p. 7
• Plain perplexity filtering risks a high false-positive rate because some regular prompts exceed adversarial prompts’ perplexity by an order of magnitude.Source: §7 · p. 7
• A classifier combining perplexity with sequence length significantly mitigates the false-positive risk.Source: §7 · p. 7
• Sequence length distinguishes many attacks from benign high-perplexity prompts because adversarial suffix attacks are lengthy whereas many regular outliers are short.Source: §7 · p. 7
• The classifier could not detect human-crafted jailbreaks, although it may apply to attacks resembling machine-generated adversarial suffixes.Source: §7 · p. 7

• Attack-string components were blacked out in the appendix to reduce potential harm.The authors note that the strings could transfer to other LLM models.Source: ETHICS STATEMENT · p. 8

A.1 GENERATED ATTACK PROMPTS

• 1407 attack prompts were generated with code from Zou et al. and analyzed using perplexity and token length.Source: §A.1 · p. 11
• The perplexity and sequence-length distributions are visualized for generated attack prompts.Source: Figure 4 · p. 11
• Scatter plots compare perplexity with sequence length for the generated attack prompts.Source: Figure 5 · p. 11
• Low-perplexity points represent initial prompts that did not succeed in testing, while the larger right-hand cluster represents generated prompts that may break LLM defenses.Source: §A.1 · p. 11
• Examples of successful adversarial suffix attacks were censored, and their outputs were tested on Vicuna-7B.Source: §A.1 · p. 12
• The adversarial-prompt section also includes additional figures associated with the generated attack analysis.Source: Figure 6 · p. 12, Figure 7 · p. 12, Figure 8 · p. 13

A.2 GPT4 JAILBREAK PROMPTS

• The manually constructed GPT-4 jailbreak dataset contains 79 examples, with perplexity and token-length visualizations.Source: §A.2 · p. 14

B.1 DOCRED

• DocRED validation contains 998 multi-sentence passages for entity and relation extraction from long documents.Source: §B.1 · p. 15

B. NON-ADVERSARIAL DATASETS

• The section includes perplexity, token-length, and scatter-plot visualizations for several non-adversarial datasets.Source: Figure 12 · p. 15, Figure 13 · p. 15, Figure 14 · p. 16

B.2 BOOLQ

• BoolQ validation contains 3270 passages, converted into prompts with a fixed instruction, question, and passage.Source: §B.2 · p. 16

B.3 SQUAD-V2

• SQuAD v2 validation contains 11873 examples, formatted with title, context, and question fields for span-based answering.Source: §B.3 · p. 17

B.4 OPEN PLATYPUS

• Open-Platypus contributes 24926 training prompts from scientific, reasoning, and question-answering datasets without added prefixes or suffixes.Source: §B.4 · p. 18

B. NON-ADVERSARIAL DATASETS

• The shortest prompt is “Hello, AI!.”Source: §B.4 · p. 18

Puffin dataset construction

• Puffin contains 3000 GPT-4 conversations, yielding 6994 human prompts and a separate set of 3000 initial utterances.The initial-utterance sample was constructed because it more closely matches the structure of the observed attacks.Source: §B.5 · p. 19

Puffin analysis

• The first Puffin analysis plots perplexity, token-length frequency distributions, and perplexity-versus-sequence-length scatter plots for human prompts.Source: §B.5 · p. 19, Figure 20 · p. 19, Figure 21 · p. 19

Puffin results

• Some short, single-word Puffin utterances produce NaN perplexities.The collected examples include brief responses such as “Yes,” “Continue,” and “Hi.”Source: §B.5 · p. 19
• These apparently meaningless standalone utterances can be meaningful when interpreted using prior conversational interactions.Source: §B.5 · p. 20

Initial Puffin prompts

• The second Puffin analysis focuses on the initial human prompt, Puffin[0], using the same perplexity, token-length, and scatter-plot views.Source: §B.5 · p. 20, Figure 22 · p. 20, Figure 23 · p. 20

Additional prompt datasets

• The broader prompt analysis also includes 116862 instruction-following examples and 10000 Python instructional code-search prompts.The instruction-following prompts concatenate instruction and input fields, while the code-search subset is capped at its first 10000 examples.Source: §B.6 · p. 21, §B.7 · p. 22