SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks

Alexander Robey; Eric Wong; Hamed Hassani; George J. Pappas

Source-linked AI summary

SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks

Alexander Robey, Eric Wong, Hamed Hassani, George J. Pappas

arXiv:2310.03684v4Oct 5, 2023cs.LGcs.AIstat.ML

TL;DR

Jailbreaking attacks can bypass LLM alignment by modifying prompts to elicit objectionable content, and prior defenses had not effectively resolved these vulnerabilities. SmoothLLM perturbs multiple copies of an input prompt and aggregates their predictions; experiments show state-of-the-art defense against four jailbreak families, with a robustness–nominal-performance trade-off and broad LLM compatibility.

Problem
Jailbreaks bypass LLM safety guardrails by modifying prompts, while no prior defense had been shown to effectively resolve these vulnerabilities.
Method
SmoothLLM randomly perturbs multiple copies of an input prompt and aggregates the corresponding outputs to detect adversarial inputs.
Results
SmoothLLM sets the state-of-the-art in defending against GCG, PAIR, RANDOMSEARCH, and AMPLEGCG attacks.
Takeaways & Limitations
SmoothLLM provides a general defense that avoids retraining and is compatible with any LLM, including black- and white-box models.
Takeaways & Limitations
Nominal performance tends to degrade for large perturbation rates, although the trade-off is manageable for q ≤5.

Abstract

from arXiv · show

Despite efforts to align large language models (LLMs) with human intentions, widely-used LLMs such as GPT, Llama, and Claude are susceptible to jailbreaking attacks, wherein an adversary fools a targeted LLM into generating objectionable content. To address this vulnerability, we propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks. Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs. Across a range of popular LLMs, SmoothLLM sets the state-of-the-art for robustness against the GCG, PAIR, RandomSearch, and AmpleGCG jailbreaks. SmoothLLM is also resistant against adaptive GCG attacks, exhibits a small, though non-negligible trade-off between robustness and nominal performance, and is compatible with any LLM. Our code is publicly available at \url{https://github.com/arobey1/smooth-llm}.

1 Introduction

Jailbreaks can bypass LLM alignment by modifying prompts to elicit objectionable content, while existing defenses had not effectively resolved these vulnerabilities. SmoothLLM addresses this gap by perturbing multiple prompt copies and aggregating their outputs, achieving strong attack mitigation with compatibility and efficiency properties.

Motivation: Adversarial prompting jailbreaks modify input prompts to fool LLMs into generating objectionable content, with recent attacks achieving 100% attack success rates.These vulnerabilities are difficult to detect or mitigate and matter for LLMs used in educational, medical, and business settings.
Motivation: No prior defense had been shown to effectively resolve the identified adversarial-prompting vulnerabilities.
SmoothLLM: SmoothLLM duplicates and randomly perturbs input prompts, then aggregates the outputs from the perturbed copies to detect adversarial inputs.The design is motivated by the instability of adversarial prompts under character-level perturbations.
Results: SmoothLLM sets the state-of-the-art in reducing ASRs for GCG, PAIR, RANDOMSEARCH, and AMPLEGCG jailbreaks relative to undefended LLMs.RANDOMSEARCH and AMPLEGCG are reduced to near-zero ASRs.
Results: Across four NLP benchmarks, SmoothLLM incurs a modest yet non-negligible trade-off between robustness and nominal performance.The trade-off can be mitigated by selecting appropriate hyperparameters.
Properties: SmoothLLM avoids retraining, improves robustness by up to 20× with one additional query, and is compatible with black- and white-box LLMs.

2 The need for defenses against jailbreaking attacks

Jailbreaking attacks can bypass LLM alignment by modifying prompts to elicit objectionable content, creating deployment challenges. Effective defenses must mitigate attacks while preserving useful generation, query efficiency, and broad model compatibility.

Adversarial prompting modifies input prompts to make targeted LLMs generate objectionable content.
Query-only defenses are needed for closed-source LLMs, while retraining-based approaches can be computationally infeasible.
A desiderata for LLM defenses against jailbreaking: Defenses should mitigate attacks, remain non-conservative, avoid retraining, and work across diverse LLM architectures and access settings.
A desiderata for LLM defenses against jailbreaking: Maintaining realistic text generation is essential because changing every prompt character could nullify attacks only by producing nonsensical inputs.

3 SMOOTHLLM: A randomized defense for LLMs

SMOOTHLLM exploits the character-level fragility of adversarial suffixes by perturbing multiple prompt copies and aggregating their LLM responses. Its analysis gives defense-success guarantees in terms of sample count, perturbation size, and suffix instability.

3.1 Adversarial suffixes are fragile to perturbations: Adversarial suffixes are fragile: perturbing 10% of characters with insertion or patch operations can reduce attack success rates below 1%.
3.2 From perturbation instability to adversarial defense: SMOOTHLLM randomly perturbs N copies of the entire input prompt, without requiring knowledge of whether or where an adversarial suffix appears.
3.2 From perturbation instability to adversarial defense: Insert, swap, and patch perturbations control character changes through the percentage q, with larger q producing larger perturbations.
3.2 From perturbation instability to adversarial defense: The defense aggregates responses by estimating the fraction that jailbreak and returning a response consistent with the majority vote.
3.3 Choosing hyperparameters for SMOOTHLLM: Under a k-unstability assumption, the defense-success probability has a closed-form expression depending on N, q, and k.
3.3 Choosing hyperparameters for SMOOTHLLM: The defense-success probability increases as the number of samples N and perturbation percentage q increase.

4 Experimental results

Experiments evaluate SMOOTHLLM across attack mitigation, nominal performance, efficiency, and adaptive robustness. The defense substantially lowers attack success rates, including for adaptive attacks, while introducing a modest robustness–utility trade-off and query overhead.

Compatibility: SMOOTHLLM is compatible with any LLM because it operates as a black-box defense.The evaluation covers attack mitigation, non-conservatism, and efficiency in addition to compatibility.
Attack mitigation: SMOOTHLLM reduces GCG ASR below one percentage point and achieves state-of-the-art results against GCG, PAIR, RANDOMSEARCH, and AMPLEGCG.It also reduces PAIR ASRs by factors of two on Vicuna and GPT-4 and by a factor of 29 on GPT-3.5.
Adaptive attacks: Adaptive GCG attacks are no stronger against SMOOTHLLM than non-adaptive attacks under the reported evaluation.The adaptive attack targets a differentiable token-space surrogate before transferring the resulting attack to SMOOTHLLM.
Attack mitigation: For swap perturbations and N > 6, SMOOTHLLM reduces GCG ASR below 1% on both Vicuna and Llama2.The corresponding reductions are roughly 50× for Llama2 and 100× for Vicuna.
Nominal performance: Across four NLP benchmarks, SMOOTHLLM incurs a modest yet non-negligible trade-off between robustness and nominal performance.An empirical majority-vote variant can improve nominal performance without trading off robustness, while larger perturbation percentages degrade nominal performance.
Efficiency: SMOOTHLLM uses N times more queries than an undefended LLM, while N = 2 requires only one extra query.Compared with GCG, it is generally five to six orders of magnitude more query-efficient.

5 Discussion, limitations, and directions for future work

The discussion identifies parameter-sensitive failure modes, computational asymmetry, and a robustness–nominal-performance trade-off. It also outlines efficiency comparisons and future directions for reducing utility degradation.

The interplay between q and the ASR: For N ≤ 4, larger q can produce larger ASRs because heavily perturbed prompts may be misclassified as jailbreaks.The authors therefore argue that q should remain small enough for the prompt to retain semantic content.
The computational burden of jailbreaking: SMOOTHLLM generally shifts computational burden toward the attacker because randomized black-box defense requires far fewer queries than GCG.The authors suggest future research should seek attacks that cannot be cheaply defended by randomized algorithms.
Addressing the nominal performance trade-off: Nominal performance degrades for large q, although the trade-off is manageable for q ≤ 5.Proposed future directions include denoising generative models and semantic transformations such as paraphrasing.

6 Conclusion

The paper presents SMOOTHLLM as a defense against LLM jailbreaking and evaluates it against four attacks. The experiments report state-of-the-art defense performance across GCG, PAIR, RANDOMSEARCH, and AMPLEGCG.

SMOOTHLLM is evaluated against GCG, PAIR, RANDOMSEARCH, and AMPLEGCG attacks and sets the state-of-the-art in defending against them.

A Robustness guarantees: Proofs and additional results

The appendix develops robustness guarantees by modeling perturbed prompts and non-jailbroken responses probabilistically. It specializes the analysis to swap and patch perturbations under an instability assumption.

The formal guarantee assumes the suffix is k-unstable for k ≤ min(M, mS), with M = ⌊qm⌋ and u = min(M, mS).The analysis treats prompts as concatenated goal and suffix strings over an alphabet of size v.
The appendix derives probabilities that SMOOTHLLM is not jailbroken for RANDOMSWAPPER and RANDOMPATCH perturbations.
The defense success probability is computed from N independent perturbed prompts and the probability α that one perturbed prompt does not yield a jailbreak.The resulting expression has the form of a binomial tail requiring at least ⌈N/2⌉ non-jailbroken responses.
Swap perturbations: For swap perturbations, the proof selects M locations uniformly and replaces each selected character with a uniformly sampled alphabet character.The analysis calculates the probability that at least k suffix locations are changed relative to the original prompt.
Patch perturbations: For patch perturbations, the proof enumerates contiguous patches and divides the calculation into four cases based on patch width, suffix length, and goal length.There are n − M + 1 possible patches before restricting to valid overlaps with the suffix.

B Further experimental details

The appendix details evaluation procedures, jailbreak judging criteria, and efficiency comparisons for SmoothLLM, alongside supplementary robustness and parameter-sweep experiments.

Evaluation methodology: Jailbreak occurrence was determined using prior implementations’ criteria, including keyword-based rules for GCG and LLM-based or classifier-based judges for other attacks.GCG used refusal-keyword criteria, RANDOMSEARCH and AMPLEGCG used GPT-4 as a judge, and PAIR used Llama Guard.
Efficiency: SmoothLLM takes 3.5–4.5 seconds per prompt, whereas generating one GCG suffix takes about 90 minutes on an A100 and two hours on an A6000.The resulting comparison makes SmoothLLM several thousand times faster than GCG.
Hyperparameter selection: SmoothLLM’s query and time efficiency makes sweeping the sample count N and perturbation percentage q a viable strategy for selecting effective hyperparameters.The authors also suggest ensembling SmoothLLM instances with different hyperparameters as future work.
Supplementary analyses: Figure 11 reproduces the DSP analysis for Vicuna, where DSP increases as the number of samples N and perturbation percentage q increase.The figure complements the corresponding Llama2 analysis.
Supplementary analyses: Figure 4 averages ASRs over five independent trials for N ∈ {2, 4, 6, 8, 10} and q ∈ {5, 10, 15, 20}.Black lines indicate the corresponding standard deviations.

B.8 Robustness guarantees in a simplified setting

Supplementary experiments examine DSP and query-efficiency trends, robustness trade-offs, and nominal performance across models and benchmarks.

Robustness guarantees: For Vicuna, DSP decreases as GCG step count increases but increases as SmoothLLM’s sample count N and perturbation percentage q increase.The corresponding average prompt and suffix lengths are m = 179 and mS = 106.
Query efficiency: Figure 12 shows that GCG’s ASR tends to increase with more iterations, while increasing N and q enables SmoothLLM to mitigate attacks on Llama2.This complements the query-efficiency analysis previously shown for Vicuna.
Query efficiency: The query count for a 500-step GCG suffix with batch size 512 is estimated as 256,000, excluding one additional loss-computation query.The estimate is calculated as 500 × 512.
Robustness trade-offs: The nominal-performance evaluation is complicated because the behaviors dataset contains only prompts requesting objectionable content, preventing direct clean-performance measurement there.The paper therefore uses standard question-answering benchmarks to assess clean-text trade-offs.
Robustness trade-offs: SmoothLLM’s defended and undefended performance are particularly close on OpenBookQA and ToxiGen.The experiments evaluate PIQA, OpenBookQA, and ToxiGen across Llama2 and Vicuna.

B.11 Defending closed-source LLMs with SmoothLLM

This appendix evaluates SmoothLLM on closed-source models, compares defenses, and discusses transferability, reproduction limits, and adaptive attack formulations.

Reproduction caveats: The reproduced GPT-4 and, to some extent, PaLM-2 results differ from prior reports, plausibly because provider patches reduced suffix effectiveness.The reproduction was also constrained by GPT usage limits.
Closed-source robustness: Across open- and closed-source LLMs, SmoothLLM reduces the ASR of transferred adversarial suffix attacks below one percentage point.Figure 14 transfers suffixes generated for Llama2 to GPT-3.5, GPT-4, Claude-1, Claude-2, and PaLM-2.
Transferability: Attacks did not transfer between Llama2 and Vicuna, so the experiments do not transfer Vicuna attacks to Llama2 or Llama2 attacks to Vicuna.This boundary applies to the transfer comparisons in Figures 1 and 14.
Defense comparison: SmoothLLM matches or surpasses the state of the art for PAIR and GCG and achieves the lowest average ASR across four models by a significant margin.The comparison uses the standardized JBB-Behaviors dataset.

C.1.2 On the differentiability of SmoothLLM

The appendix argues that SmoothLLM is non-differentiable and therefore resists direct adaptive GCG attacks, while character-level smoothing complicates differentiable surrogate attacks.

Direct adaptive attacks: SmoothLLM cannot be adaptively attacked by GCG because GCG requires gradients and SmoothLLM is non-differentiable with respect to its input.The non-differentiability arises from sampling perturbed prompts and using an indicator function.
Surrogate attacks: A natural surrogate averages model responses over perturbed prompts, but character-level perturbations can make their tokenizations have different dimensions.Aggregating the resulting gradients therefore produces a dimension mismatch after several GCG iterations.
Surrogate attacks: Token-level perturbations avoid this dimensionality problem because all perturbed tokenizations retain the same construction space.The proposed token-space surrogate enables gradients to be taken with respect to perturbed tokenizations.
Supplementary illustration: The appendix illustrates the incoherency threshold as an additional supplementary figure related to SmoothLLM’s behavior.Figure 15 is presented without further quantitative interpretation in the supplied passage.

D The incoherency threshold

High perturbation rates can make prompts incoherent, causing the safety filter to misclassify non-jailbreak responses and inflate ASR. Accordingly, q should not be chosen particularly large.

The incoherency threshold: Higher q values can produce larger ASRs at lower N because substantial perturbation confuses the LLM into falsely identifying a jailbreak.The perturbed prompt may become difficult to understand without actually eliciting objectionable content.
The incoherency threshold: The keyword-based detector can label an incoherent response as a jailbreak when it lacks the listed refusal keywords.Such responses are false positives with respect to the JB function used in [20].
The incoherency threshold: q should not be particularly large because passing the incoherency threshold reduces the semantic content of the prompt.This phenomenon is termed “passing the incoherency threshold.”

E.1 Adversarial examples, robustness, and certification

Adversarial examples expose deep-learning models to failures from seemingly innocuous input changes, motivating empirical and certified robustness defenses. SmoothLLM adapts randomized-smoothing intuition to language-generation jailbreaks, but its discrete setting and threat model differ from classifier certification.

Adversarial examples and defenses: Seemingly innocuous input changes can cause deep neural networks to fail, limiting their applicability in safety-critical areas.Adversarial examples are a particularly studied case involving imperceptible, adversarially chosen perturbations.
Adversarial examples and defenses: Empirical defenses improve performance against attacks, whereas certified defenses provide invariance guarantees for perturbations within a specified magnitude.Adversarial training dominates empirical defenses, while randomized smoothing is prevalent among certified defenses.
Randomized smoothing: Randomized smoothing creates a smoothed classifier by perturbing inputs before classification and can prove robustness to perturbations of a particular magnitude.The smoothed classifier predicts the label with the highest probability under the perturbed-input distribution.
SmoothLLM distinctions: SmoothLLM applies randomized perturbations to language-generation prompts, targeting jailbreaks rather than classification errors.Its problem setting involves variable-length generated sequences without necessarily one correct answer.
SmoothLLM distinctions: SmoothLLM samples discrete character-level perturbations, so it shares randomized-smoothing intuition but is not directly comparable to standard smoothing algorithms.The paper distinguishes the continuous-versus-discrete defense distributions and the differing task settings.

F.3 New datasets for robust evaluation

The evaluation dataset contains substantial topic repetition, motivating larger and more diverse prompt collections. The appendix also identifies additional perturbations and response-ensemble strategies as directions for stronger robustness.

New evaluation datasets: 24 of the 520 prompts in the behaviors dataset concern almost exactly the same topic: building a bomb.This repetition limits the dataset’s topical diversity for evaluating jailbreak attacks.
New evaluation datasets: Future evaluations should use larger, more diverse, and less repetitive datasets of prompts requesting objectionable content.This direction follows directly from the concentration of prompts on repeated bomb-building topics.
Perturbation directions: Future versions could add deletions, synonym replacements, and capitalization changes, or ensemble responses across different perturbation functions.The latter strategy is proposed as a possible way to strengthen empirical robustness.
Perturbation directions: The paper’s perturbation algorithms include random swaps, patches, and insertions applied to sampled prompt indices.Each function is formally defined in Algorithm 2.
Perturbation directions: Each perturbation samples M = ⌊qm⌋ new characters, involving q% of the original m prompt characters.Indices determine where perturbations are applied, and replacement characters are sampled uniformly from the alphabet A.