PaperlayerView exact revision on arXiv

Source-linked AI summary

Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM

Bochuan Cao, Yuanpu Cao, Lu Lin, Jinghui Chen

arXiv:2309.14348v3cs.CLcs.AIcs.CRcs.LG

TL;DR

LLM alignment can be bypassed by adversarially optimized or handcrafted jailbreak prompts, motivating defenses against alignment-breaking attacks. The paper proposes RA-LLM, which adds robust alignment checking to an existing aligned LLM without retraining, and reports substantial reductions in attack success rates. Its limitations include some benign-response degradation and untested extreme prompt lengths.

  • Problem

    Existing LLM alignments remain vulnerable to adversarially optimized and handcrafted jailbreak prompts, while dedicated defenses are few and external detectors have practical drawbacks.

  • Method

    RA-LLM constructs a robust alignment checking function on an existing aligned LLM by evaluating requests after random dropping, without expensive retraining or fine-tuning.

  • Results

    RA-LLM reduces attack success rates from nearly 100% to around 10% or less against state-of-the-art adversarial and popular handcrafted jailbreak prompts.

  • Takeaways & Limitations

    RA-LLM provides a defense that substantially mitigates adversarial attacks while relying on alignment capability already inside the LLM.

  • Takeaways & Limitations

    The method slightly reduces benign answering rates in some models and has not been evaluated on particularly long or short adversarial prompts.

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have made significant advancements and are now widely used across various domains. Unfortunately, there has been a rising concern that LLMs can be misused to generate harmful or malicious content. Though a line of research has focused on aligning LLMs with human values and preventing them from producing inappropriate content, such alignments are usually vulnerable and can be bypassed by alignment-breaking attacks via adversarially optimized or handcrafted jailbreaking prompts. In this work, we introduce a Robustly Aligned LLM (RA-LLM) to defend against potential alignment-breaking attacks. RA-LLM can be directly constructed upon an existing aligned LLM with a robust alignment checking function, without requiring any expensive retraining or fine-tuning process of the original LLM. Furthermore, we also provide a theoretical analysis for RA-LLM to verify its effectiveness in defending against alignment-breaking attacks. Through real-world experiments on open-source large language models, we demonstrate that RA-LLM can successfully defend against both state-of-the-art adversarial prompts and popular handcrafted jailbreaking prompts by reducing their attack success rates from nearly 100% to around 10% or less.

1 INTRODUCTION

LLMs can generate harmful content, while existing alignment remains vulnerable to adversarial jailbreaks. RA-LLM robustifies an aligned LLM through random request dropping and alignment checking, without external harmful detectors, and substantially reduces attack success rates.

  • LLMs can be misused to generate harmful responses because undesirable material exists in their training data.
  • Existing alignment methods use human or AI feedback to align LLMs with human values, but jailbreak attacks can still elicit harmful responses.
  • RA-LLM builds on an aligned LLM and treats a request as benign only when random dropping leaves its benign classification intact in most cases.The dropping operation is intended to invalidate adversarial prompts that are sensitive to small perturbations.
  • RA-LLM relies on the LLM’s existing alignment capability rather than external harmful detectors and is intended to robustify different existing alignments.
  • RA-LLM is supported by theoretical analysis and experiments on open-source LLMs against adversarial and handcrafted jailbreak prompts.The introduction reports attack success rates falling from nearly 100% to around 10% or less.

2 RELATED WORKS

Related work studies alignment methods, alignment-breaking attacks, and defenses. Prior defenses include safety filtering, LLM self-detection, perplexity-based prompt detection, and traditional adversarial-training or augmentation methods.

  • Aligning LLMs with Human Preferences: Alignment research steers LLM outputs toward human values using approaches including reinforcement learning from human feedback and AI feedback.
  • Alignment-breaking Attacks and defenses in LLMs: Jailbreak attacks bypass alignment safeguards and can elicit harmful or toxic responses, threatening practical LLM deployment.
  • Alignment-breaking Attacks and defenses in LLMs: Existing defenses include enumerating partially erased inputs, using LLMs to detect harmful responses, and checking prompt perplexity.
  • Traditional Text Adversarial Attack and Defenses: Traditional text adversarial attacks perturb inputs to increase target-model prediction error, using white-box gradients or black-box prediction information.
  • Traditional Text Adversarial Attack and Defenses: Traditional defenses improve robustness through adversarial training or synonym-based adversarial data augmentation.

3 Our Proposed Method

The method constructs a Robustly Aligned LLM by replacing ordinary alignment checks with a robustness test based on randomly dropping request tokens. It requires the original and perturbed requests to pass alignment checks, uses practical approximations, and is theoretically shown to reject certain alignment-breaking inputs.

  • Threat Model: The threat model inserts an adversarial prompt before, after, or inside a malicious question to bypass an aligned LLM’s security checks.The malicious question and adversarial prompt form the combined input xadv = x ⊕ padv.
  • Robust Alignment Check: RA-LLM replaces a vanilla alignment check with a robust check that tests whether randomly dropped versions of a request remain non-refusal outputs in most cases.The method treats a request as benign only when its original response and responses after random dropping satisfy the alignment criterion.
  • Construction: RA-LLM can be built from an existing aligned LLM and its alignment capability without external harmful-content detectors or retraining.The construction uses an alignment check function based on refusal-like output prefixes and replaces it with the robust version.
  • Practical Designs: Monte Carlo sampling approximates the robust check by generating n randomly masked requests, querying the LLM, and counting alignment-check failures.Exact probability computation would require enumerating all random dropping cases and is computationally intractable.
  • Practical Designs: A small positive threshold t avoids rejecting benign requests when random dropping removes information and causes accidental refusal-like responses.The paper gives “Do you like apples?” becoming “Do you apples?” as an example of information loss after dropping.
  • Theoretical Analysis: Under Theorem 3.1’s token-length and threshold conditions, sufficiently many random-drop trials cause RA-LLM to reject an adversarially prompted malicious request.The theorem covers adversarial prompts of length M inserted at any position in an input of length N.

4 Experiments

The experiments evaluate RA-LLM against optimized and handcrafted jailbreak attacks while measuring both attack success and benign answering. Across these settings, the defense substantially reduces attack success, and ablations examine trade-offs among its hyperparameters.

  • Experimental Setup: The evaluation measures attack success rate (ASR) and benign answering rate (BAR) to assess defense effectiveness without over-rejecting benign questions.ASR counts successful alignment circumventions, while BAR measures responses to benign requests.
  • Adversarial Prompt Attacks: Individual Attack ASR fell from 98.7% and 96.0% to 10.7% and 6.7% on Vicuna-7B-v1.3-HF and Guanaco-7B-HF, respectively.Individual Attack directly optimizes adversarial prompts for specific models and malicious requests.
  • Adversarial Prompt Attacks: Transfer Attack ASR fell from 83.3% and 78.7% to 11.3% and 8.7% on the two evaluated models.Transfer Attack optimizes generic adversarial prompts across multiple models and malicious requests.
  • Handcrafted Jailbreak Prompts: Handcrafted jailbreak ASR fell from 98.4%, 94.7%, and 82.0% to 12%, 9.3%, and 8.0% on three safety-aligned LLMs.The evaluation used five highly rated prompts and 150 samples; RA-LLM had no significant BAR impact, especially on GPT-3.5-turbo-0613.
  • Ablation Study: Increasing the random dropping ratio p lowers ASR but can reduce BAR, whereas smaller p preserves benign accuracy but weakens robust alignment checking.The dropping ratio therefore controls a trade-off between stricter rejection and benign-response preservation.
  • Ablation Study: Threshold t also trades off benign accuracy and attack reduction: too-small t can reject confusing benign variants, while too-large t limits rejection.The default settings are n = 20, p = 0.3, and t = 0.2.
  • Ablation Study: With 15 or 10 Monte Carlo trials, RA-LLM maintains BAR close to 100% and relatively low ASR, suggesting lower computational cost is feasible.The method retains stable defensive performance across various trial counts.

5 Computational Cost

RA-LLM uses limited generation and early termination to reduce the computational cost of robust alignment checking. Experiments report that its additional inference time remains below 20% of normal inference.

  • RA-LLM uses limited output generation and an early-exit mechanism to make Monte Carlo-based alignment checking more time-efficient.Generation can stop after a small maximum length, and processing can terminate when detected failures exceed a threshold.
  • Less than 20% additional time was required by RA-LLM compared with normal inference on the tested LLMs.The section reports average overhead across experiments and provides detailed timing in Table 3.

6 Adaptive Attack

The section evaluates adaptive attacks that repeat prompts, replace the target model, or optimize an additional loss term. Across these settings, the reported defense remains difficult to bypass, although some adaptive objectives impose substantial attacker overhead.

  • Repeating adversarial prompts: Around 5% ASR was maintained across repetition counts when adversarial prompts were repeated against the robustly aligned LLM.The experiments compared repeated-prompt attacks on the original LLM and RA-LLM.
  • Replacing the Target Model with RA-LLM: 0% success rate was obtained after TAP and AutoDAN repeatedly attacked RA-LLM as the substituted target model.Gradient-based GCG could not be applied directly because RA-LLM does not expose directly derivable gradients.
  • Incorporating an Additional Loss Term: Optimizing the probability of RAC returning Pass would impose approximately 20 times more overhead per attack iteration.This overhead is significant because gradient-based attacks such as GCG already require substantial computational resources.
  • Adaptive optimization: Random dropping combined with EoT caused the loss to oscillate near its initial value and fail to converge in the tested Vicuna-7B setting.The experiment dropped 30% of tokens and optimized the remaining 70% during each optimization step.
  • Adaptive optimization: Adaptive optimization is difficult because robust adversarial samples are harder to find, while random dropping makes the loss of new inputs uncertain.The described attack methods also rely on greedy updates that retain the lowest-loss jailbreak prompt.

7 Conclusion

The paper proposes robustly aligned LLMs that strengthen existing alignment checks against adversarial prompts without expensive retraining or fine-tuning. Theory and experiments support defense against automatically generated and handcrafted jailbreak prompts.

  • RA-LLM builds a robust alignment checking function on an existing aligned LLM to defend against alignment-breaking attacks.The method is designed to make existing alignments less prone to circumvention by adversarial prompts.
  • The method does not require expensive retraining or fine-tuning of the original LLM for defense.
  • The paper provides theoretical analysis and experiments supporting defense against automatically generated adversarial prompts and handcrafted jailbreak prompts.

8 Limitations

The paper identifies two limitations: random dropping can reduce benign answering rates, and extreme adversarial-prompt lengths were not evaluated.

  • Benign answering decreased from 95.3% to 92.0% for Guanaco-7B-chat-HF after applying random dropping.The authors describe this as a minor effect that future dropping methods should reduce.
  • The method was not assessed against particularly long or short adversarial prompts because current jailbreak techniques provide limited coverage of such extreme cases.The authors identify evaluation of these cases as future work.

A Proof of Theorem 3.1

The section proves Theorem 3.1 for adversarial examples formed by inserting adversarial prompts into malicious questions. It also illustrates how random dropping changes model responses under harmful-behavior and handcrafted jailbreak attacks.

  • Proof of Theorem 3.1: The proof considers adversarial examples x_adv formed by inserting M adversarial tokens at position j into an input x.The inserted adversarial prompt is denoted p_j^adv.
  • Proof of Theorem 3.1: For each insertion position, the proof applies the law of total probability to derive the probability bound used by Theorem 3.1.
  • Proof of Theorem 3.1: The proof concludes that the robust alignment checker classifies the adversarial example as Fail.
  • Concrete Examples: Concrete examples compare the original LLM’s responses before and after random dropping under harmful-behavior attacks.
  • Concrete Examples: A separate example compares responses before and after random dropping under a handcrafted jailbreak attack without explicitly labeling discarded content.

C Defensive Efficacy Against Harmful Strings Attack

The experiments evaluate RA-LLM against Harmful Strings, AutoDAN, and TAP attacks. On Vicuna-7B-v1.3, RA-LLM reduces Harmful Strings attack success from 84% to 0%.

  • Harmful Strings Attack: The Harmful Strings experiments use adversarial inputs designed to induce the LLM to generate a specific harmful string.
  • Harmful Strings Attack: 84% to 0%: RA-LLM reduces Harmful Strings attack success on Vicuna-7B-v1.3.The authors state that this demonstrates effective mitigation of Harmful String Attack.
  • Reported Metrics: Tables 5 and 6 report benign answering rate and attack success rate for the original and robustly aligned models across these attacks.
  • AutoDAN and TAP: The evaluation also tests RA-LLM against AutoDAN and Tree of Attacks, two methods that generate or refine semantic jailbreak prompts.AutoDAN uses a genetic algorithm, whereas TAP iteratively refines prompts using tree-of-thought reasoning.

E Details of Experiment

The experiments use AdvBench harmful-content data and fixed attack and RA-LLM settings, then compare RA-LLM with self-defense and perplexity-based defenses. The comparisons expose differences in detection accuracy, benign answering, and jailbreak coverage.

  • Experimental Data: The evaluation uses 500 Harmful Strings and 500 Harmful Behaviors samples from AdvBench.The datasets cover toxic content, threats, discriminatory remarks, crime methods, dangerous suggestions, and harmful-output prompts.
  • Experimental Settings: Unless stated otherwise, RA-LLM uses p = 0.3, t = 0.2, tmax = 10, and n = 20.
  • Self-Defense Comparison: Self-defense evaluations compare attacked Vicuna and Guanaco models with self-defense using themselves or GPT-3.5.The attacked models struggle to detect harmful content and produce unexpected outputs beyond the prescribed yes/no responses.
  • Self-Defense Comparison: Self-defense through GPT-3.5 achieves comparable defending effectiveness to RA-LLM but has lower benign answering rates.The authors attribute this to overcautious harmful-content detection.
  • Comparison Metrics: Tables 7 and 8 compare benign answering rate and attack success rate across self-defense, GPT-3.5 self-defense, perplexity defense, and RA-LLM.
  • Perplexity Defense Comparison: Perplexity-based defense can reduce individual GCG attack success but completely fails to detect handcrafted jailbreak prompts.

H.1 Time Cost

RA-LLM adds computation through Monte Carlo response checks, with early exit and fewer trials offering cost reductions. Experiments report moderate time overhead, while API-based cost estimates exceed the undefended model’s cost.

  • Early Exit: Early exit stops Monte Carlo checking once enough aligned or failure cases determine the classification.With n = 20 and t = 0.2, four aligned responses can trigger an early malicious classification.
  • Inference Time: RA-LLM requires less than 20% extra inference time in the reported Vicuna and Guanaco experiments.The experiments measure normal inference, RA-LLM, and full Monte Carlo simulation on 150 attack samples.
  • Method Comparison: For Vicuna, RA-LLM adds 3.93 seconds per datum, compared with 1.45 seconds for perplexity defense and 49.0 seconds for self-defense.
  • API Cost: The extra-cost ratio is approximately 1.250 under GPT-4 pricing and 1.496 under GPT-3.5 Turbo pricing.The estimate uses measured input and output token counts with the default Monte Carlo settings.
  • Cost–Performance Trade-off: Random dropping can reduce benign answering rate because dropped information may cause unable-to-answer responses.The paper describes tuning settings to prioritize either benign user experience or security.
Loading 2309.14348v3…