PaperlayerView exact revision on arXiv

Source-linked AI summary

Certifying LLM Safety against Adversarial Prompting

Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, Himabindu Lakkaraju

arXiv:2309.02705v4cs.CLcs.AIcs.CRcs.LG

TL;DR

Adversarial prompt modifications can bypass LLM safety measures and induce harmful outputs. The paper introduces erase-and-check, which erases tokens and checks subsequences to provide certified defenses, while also proposing faster empirical variants that perform well in practice.

  • Problem

    Adversarial attacks can append or insert malicious tokens into prompts to bypass LLM safety measures and cause harmful outputs.

  • Method

    Erase-and-check erases tokens from clean or adversarial prompts and uses a safety filter to inspect the resulting subsequences, with variants for suffix, insertion, and infusion attacks.

  • Results

    Erase-and-check achieved 92% certified accuracy with Llama 2 and 100% with DistilBERT on harmful AdvBench prompts, while reaching 97% empirical accuracy on clean safe prompts in suffix mode.

  • Takeaways & Limitations

    The framework provides verifiable safety guarantees against harmful prompts modified by adversarial sequences up to a defined length, alongside three more efficient empirical defenses.

  • Takeaways & Limitations

    Erase-and-check can become infeasible for long adversarial sequences, safe-prompt accuracy can decrease with larger erase lengths, and the safety classifier is not perfectly accurate.

Abstract

from arXiv · show

Large language models (LLMs) are vulnerable to adversarial attacks that add malicious tokens to an input prompt to bypass the safety guardrails of an LLM and cause it to produce harmful content. In this work, we introduce erase-and-check, the first framework for defending against adversarial prompts with certifiable safety guarantees. Given a prompt, our procedure erases tokens individually and inspects the resulting subsequences using a safety filter. Our safety certificate guarantees that harmful prompts are not mislabeled as safe due to an adversarial attack up to a certain size. We implement the safety filter in two ways, using Llama 2 and DistilBERT, and compare the performance of erase-and-check for the two cases. We defend against three attack modes: i) adversarial suffix, where an adversarial sequence is appended at the end of a harmful prompt; ii) adversarial insertion, where the adversarial sequence is inserted anywhere in the middle of the prompt; and iii) adversarial infusion, where adversarial tokens are inserted at arbitrary positions in the prompt, not necessarily as a contiguous block. Our experimental results demonstrate that this procedure can obtain strong certified safety guarantees on harmful prompts while maintaining good empirical performance on safe prompts. Additionally, we propose three efficient empirical defenses: i) RandEC, a randomized subsampling version of erase-and-check; ii) GreedyEC, which greedily erases tokens that maximize the softmax score of the harmful class; and iii) GradEC, which uses gradient information to optimize tokens to erase. We demonstrate their effectiveness against adversarial prompts generated by the Greedy Coordinate Gradient (GCG) attack algorithm. The code for our experiments is available at https://github.com/aounon/certified-llm-safety.

1 Introduction

Adversarial prompting can bypass LLM safety guardrails, motivating erase-and-check, which certifies detection of harmful prompts under bounded modifications while retaining empirical performance on safe prompts.

  • Motivation: Adversarial attacks can bypass LLM safety measures, including automated GCG-generated suffixes achieving attack success rates up to 84% on black-box GPT-3.5.Existing empirical defenses can be broken by stronger attacks, motivating certified guarantees.
  • Erase-and-check: Erase-and-check erases tokens individually up to a maximum d and applies a safety filter to the resulting subsequences, labeling the prompt harmful if any check is harmful.The procedure labels a prompt safe only when every checked sequence is classified safe.
  • Threat models: The framework covers suffix, insertion, and infusion attacks, progressing from appended adversarial sequences to tokens inserted at arbitrary, noncontiguous positions.Infusion subsumes the earlier modes and addresses a threat model for which no existing attack was known in the authors’ discussion.
  • Safety certificate: If the filter detects a harmful prompt, erase-and-check labels that prompt and all adversarial modifications up to the certified length as harmful.The certificate extends to probabilistic filters through a lower bound based on the original prompt’s detection probability.
  • Results: 92% accuracy with Llama 2 and 100% with DistilBERT on AdvBench harmful prompts equals erase-and-check’s certified accuracy, independent of adversarial sequence length, algorithm, or mode.The certified accuracy can be computed without generating adversarial prompts.
  • Results: 97% clean-safe-prompt accuracy with Llama 2 and 98% with DistilBERT was achieved in suffix mode at maximum erase length 20, with similar results for insertion and infusion.The procedure is not certified against safe prompts becoming harmful, so these results are empirical.
  • Empirical defenses: RandEC, GreedyEC, and GradEC provide faster empirical alternatives that retain good detection accuracy against GCG-generated adversarial prompts.RandEC subsamples checks, GreedyEC selects erasures using harmful-class scores, and GradEC uses classifier gradients.

2 Related Work

Prior defenses against adversarial prompting are largely heuristic and can be bypassed by stronger attacks, while certified robustness methods provide a distinct route to provable guarantees.

  • Empirical defenses: Existing NLP defenses include perplexity filtering, paraphrasing, and adversarial training targeted at particular weaknesses of adversarial sequences.These approaches have been studied specifically for attacks generated by Zou et al.
  • Limitations of empirical defenses: Empirical defenses can be broken by stronger attacks, including natural-looking AutoDAN sequences that bypass perplexity filters.This motivates defenses whose performance guarantees remain valid against unseen attacks.
  • Certified defenses: Certified defenses in computer vision use interval-bound propagation, curvature bounds, and randomized smoothing to establish provable robustness guarantees.Certified defenses have also been studied for NLP, including word-substitution robustness over predefined synonyms.
  • Evaluation context: Average runtime is reported per prompt on a single NVIDIA A100 GPU.The supplied passage provides the measurement context but no runtime value.
  • Scope of prior work: Prior certified NLP and malware defenses often restrict perturbations to synonyms, minor text changes, or small code modifications, unlike the non-imperceptible adversarial sequences considered here.Such restrictions make them inapplicable to the attacks of Zou et al.

3 Notations

The paper represents prompts and adversarial sequences as token sequences, with notation for concatenation, subsequences, deletion, unions, and bounded erasures.

  • Sequences: An input prompt P is represented as tokens ρ1, ρ2, . . . , ρn, where n = |P| is its sequence length.Adversarial-sequence tokens are denoted α1, α2, . . . , αl.
  • Operations: The symbol + denotes sequence concatenation, so an adversarial suffix is written P + α.The notation distinguishes prompt and adversarial-sequence composition.
  • Subsequences: P[s, t] denotes the subsequence from token ρs through ρt, while P − P[s, t] denotes deletion of that subsequence.These operations describe the subsequences checked by erase-and-check.
  • Erasure bound: d is the maximum number of tokens erased, distinct from l, the adversarial-sequence length; the certified guarantee holds when l ≤ d.In insertion mode, d denotes the maximum length of an erased sequence.

4 Adversarial Suffix

The suffix threat model appends a bounded adversarial sequence, and erase-and-check detects harmful prompts by checking progressively truncated subsequences. Its certified harmful-prompt accuracy equals the safety filter’s accuracy on clean prompts, while safe-prompt evaluation compares accuracy and runtime across classifiers.

  • Threat model: The suffix threat model appends up to l adversarial tokens α to a prompt P, forming P + α.
  • Threat model: Enumerating all suffix attacks grows exponentially as O(|T|^l), but erase-and-check certifies the entire threat set without enumeration.
  • Erase-and-check: Erase-and-check checks P and d progressively truncated subsequences, labeling P harmful if any checked sequence is harmful.
  • Safety certificate: For suffix attacks with |α| ≤ d, a harmful clean prompt guarantees that P + α is labeled harmful; the clean filter accuracy lower-bounds certified accuracy.
  • Certified evaluation: 92% accuracy for Llama 2 and 100% for DistilBERT on harmful AdvBench prompts determine erase-and-check’s certified accuracy.
  • Safe prompts: Safe-prompt experiments compare empirical accuracy and running time across erase lengths and the Llama 2 and DistilBERT safety filters.

5 Adversarial Insertion

The insertion threat model places a contiguous adversarial sequence anywhere inside a prompt, requiring erase-and-check to inspect many contiguous deletions. DistilBERT maintains strong accuracy and substantial speed advantages over Llama 2 as the erase length increases.

  • Threat model: The insertion threat model forms P1 + α + P2 by splicing up to l contiguous adversarial tokens into any prompt position.
  • Threat model: Insertion subsumes suffix attacks and has threat-set size O(|P||T|^l), making it harder to defend against.
  • Erase-and-check: Erase-and-check removes every contiguous token sequence of length up to d, then checks the resulting subsequences and the original prompt.
  • Accuracy: Llama 2 accuracy drops faster in insertion mode, whereas DistilBERT maintains good performance at higher erase lengths.
  • Runtime and scale: 40X speed-up for DistilBERT over Llama 2 at larger erase lengths supports testing up to 30 erased tokens, with accuracy above 98% and runtime at most 0.3 seconds.
  • Extensions: The method is generalized to multiple adversarial insertions in Appendix F.

6 Adversarial Infusion

Adversarial infusion allows multiple tokens to be inserted at arbitrary positions, subsuming suffix and insertion attacks. Erase-and-check certifies detection when the adversarial sequence length is within the erase budget, while DistilBERT outperforms Llama 2 empirically on safe prompts.

  • Threat model: Infusion inserts up to l adversarial tokens at arbitrary positions, with suffix and insertion as contiguous-sequence special cases.Its threat model is represented as P1 + τ1 + P2 + τ2 + ··· + τm + Pm+1.
  • Certified defense: When l ≤ d, erasing subsets of at most d tokens must remove the adversarial tokens and recover the original harmful prompt.This recovered subsequence establishes the safety guarantee for infusion attacks.
  • Empirical comparison: DistilBERT outperforms Llama 2 in detection accuracy on safe prompts for infusion-mode erase-and-check.The comparison also reports empirical accuracy and running time for both safety-filter implementations.

7 Efficient Empirical Defenses

The paper proposes RandEC, GreedyEC, and GradEC as faster empirical alternatives to exhaustive erase-and-check. They trade certified guarantees for efficiency while detecting GCG-generated adversarial prompts with reported accuracies above 90%, above 94%, and 76%, respectively.

  • RandEC: RandEC randomly samples erased subsequences, reducing computation but providing no certified safety guarantee.Its sampling ratio is the fraction of erased subsequences selected for checking.
  • RandEC: Checking 30% of RandEC's erased subsequences achieves accuracy above 90% on adversarial prompts of different lengths.The reported average running time is less than 0.03 seconds per prompt on one NVIDIA A100 GPU.
  • GreedyEC: GreedyEC repeatedly erases the token whose removal maximizes the DistilBERT harmful-class softmax score.It is evaluated for a finite number of iterations and does not depend on the attack mode.
  • GreedyEC: More than 94% detection is achieved by GreedyEC as iterations increase against GCG adversarial suffixes up to 20 tokens, with runtime below 0.06 seconds per prompt.
  • Trade-offs: RandEC approaches certified performance as its sampling ratio approaches one, whereas GreedyEC is more suitable when the attack mode is unknown.GreedyEC runtime grows as O(κn), significantly better than erase-and-check for insertion and infusion modes.
  • GradEC: GradEC relaxes binary erase masks to continuous values and uses gradients to optimize which token embeddings to erase.The method requires a differentiable safety filter, satisfied here by the DistilBERT classifier.

8 Limitations

Erase-and-check's main limitation is computational cost, especially for general attack modes and long adversarial sequences. Its safe-prompt accuracy can also decrease with larger erase lengths, and the learned classifier is not perfectly accurate.

  • Computational limits: Erase-and-check becomes infeasible for long adversarial sequences because the number of erased subsequences grows rapidly in infusion-like attack modes.
  • Accuracy limits: Safe-prompt accuracy decreases for larger erase lengths, especially with Llama 2, because checking more subsequences increases misclassification likelihood.
  • Classifier limits: The trained text classifier does not achieve perfect accuracy, so erase-and-check may sometimes label prompts incorrectly.

9 Conclusion

The paper presents a framework for certifying LLM safety against adversarial prompting and positions preliminary results as a foundation for future work on robust, verifiable safeguards.

  • The framework provides certified safety guarantees for detecting harmful prompts modified by adversarial sequences up to a defined length.
  • The authors identify broader threat models, more efficient safety-filter evaluation, and extensions to privacy and fairness as future directions.
  • The work is presented as a first step toward deeper exploration of robustness requirements for responsible language-model deployment.

10 Impact Statement

The paper frames certifiable defenses as important for preventing harmful LLM outputs in public-facing applications, while emphasizing practical error rates and dependence on the safety classifier.

  • Erase-and-Check, RandEC, GreedyEC, and GradEC are proposed to defend LLMs against adversarial prompts that bypass existing safety measures.
  • Harmful LLM outputs can have serious real-world consequences, particularly when disseminated to malicious entities.
  • 92% accuracy is reported for erase-and-check with Llama 2 as the safety filter, implying ineffectiveness for the remaining 8%.
  • RandEC and GreedyEC are more efficient approximations but have slightly lower detection rates than erase-and-check.
  • The methods’ efficacy depends on the safety classifier, so this dependency must be considered in practice.
  • The certificate requires only safety-filter evaluation on clean harmful prompts and is independent of the algorithm generating adversarial prompts.
  • The safety certificates also apply to probabilistic filters and can certify expected accuracy over a distribution without certifying every sample.

D Training Details of the Safety Classifier

The DistilBERT safety classifier is fine-tuned on harmful and safe prompts, including erased safe subsequences, with class balancing to address the resulting imbalance.

  • DistilBERT is fine-tuned on a safety dataset split into 400 training and 120 test examples per class.
  • The training data include erased subsequences of safe prompts corresponding to the relevant attack mode.
  • The classifier is trained for ten epochs with AdamW, while class weights and repeated harmful examples address imbalance from the added safe subsequences.

E Comparison with Smoothing-Based Certificate

The paper compares erase-and-check with smoothing-based certification, explains its multi-insertion generalization, and highlights the computational cost of checking more erased subsequences.

  • Comparison with Smoothing-Based Certificate: Existing robustness certificates generally target prediction stability, whereas erase-and-check is designed to certify LLM safety against adversarial prompting.
  • Comparison with Smoothing-Based Certificate: The smoothing-based procedure requires a majority of checked sequences to be harmful, restricting the adversarial suffix lengths it can certify.
  • Comparison with Smoothing-Based Certificate: Theorem 2 upper-bounds the largest certifiable adversarial suffix length using the maximum erase length and the number of harmful subsequences.
  • Comparison with Smoothing-Based Certificate: Figure 9 compares erase-and-check certified accuracy with the best possible smoothing-based certified accuracy across maximum erase lengths.
  • Multiple Insertions: For k adversarial insertions, erase-and-check erases k contiguous token blocks, with the number of prompts growing exponentially in k.
  • Multiple Insertions: Figures 10a and 10b compare accuracy and runtime for one versus two insertions on 30 safe prompts with maximum erase length 6.
  • Multiple Insertions: For two insertions, maximum erase length 6 per adversarial sequence permits erasing up to 12 tokens overall.

G Proof of Theorem 1

The proof shows that erase-and-check detects adversarial suffixes by checking erased subsequences that recover the original harmful prompt. The section also illustrates this construction and describes standard-error calculations for accuracy and runtime measurements.

  • Safety certificate: For an adversarial suffix of size at most d, one erased subsequence equals the original harmful prompt, so detection is at least as likely as the safety filter labeling that prompt harmful.This establishes the safety certificate for suffix attacks.
  • Suffix-mode illustration: The suffix-mode example appends an adversarial sequence to a harmful prompt and checks subsequences formed by erasing tokens.With a maximum erase length of 10, the procedure generates progressively shortened subsequences, including the original harmful prompt.
  • Suffix-mode illustration: The illustrated checked subsequences include the original harmful prompt and versions with progressively more of the appended adversarial sequence removed.The sixth checked subsequence is identified as the harmful prompt itself.
  • Safety certificate: Erase-and-check declares a prompt harmful if any checked erased subsequence is labeled harmful by the safety filter.Therefore, correctly identifying the recovered original prompt as harmful guarantees detection of the adversarial prompt.
  • Standard-error calculations: Accuracy measurements are modeled as averages of N i.i.d. Bernoulli variables representing correct or incorrect prompt classifications.The standard deviation of the mean and accuracy uses the corrected sample standard deviation, with Bessel’s correction using N −1 in the denominator.
  • Standard-error calculations: The standard error for average runtime is calculated from the corrected sample standard deviation of the procedure’s running times across prompt samples.The section states that standard deviation of the mean is used as the standard error for accuracy and average-time measurements.
Loading 2309.02705v4…