PaperlayerView exact revision on arXiv

Source-linked AI summary

COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability

Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, Bin Hu

arXiv:2402.08679v2cs.LGcs.AIcs.CL

TL;DR

The paper asks how LLM jailbreaks can be generated with controllable features beyond fluency for broader safety assessment. It introduces COLD-Attack by adapting constrained text-generation methods, and reports versatile, effective attacks across three settings while acknowledging misuse risks.

  • Problem

    Automatic white-box jailbreak methods have limited controllability, while comprehensive LLM safety assessment requires attacks with diverse features beyond fluency.

  • Method

    COLD-Attack connects controllable attack generation to controllable text generation and uses energy functions with Langevin dynamics to automate attacks under multiple requirements.

  • Results

    Across continuation, paraphrasing, and position constraints, COLD-Attack generates attacks that satisfy fluency, stealthiness, sentiment, style, lexical, or format requirements with broad applicability and high success rates.

  • Takeaways & Limitations

    Controllable attack generation provides a unified way to diversify adversarial prompts and evaluate LLM robustness under varied requirements.

  • Takeaways & Limitations

    COLD-Attack may be misused to spread misinformation, bypass content moderation, or generate biased or harmful outputs, creating societal risks.

Abstract

from arXiv · show

Jailbreaks on large language models (LLMs) have recently received increasing attention. For a comprehensive assessment of LLM safety, it is essential to consider jailbreaks with diverse attributes, such as contextual coherence and sentiment/stylistic variations, and hence it is beneficial to study controllable jailbreaking, i.e. how to enforce control on LLM attacks. In this paper, we formally formulate the controllable attack generation problem, and build a novel connection between this problem and controllable text generation, a well-explored topic of natural language processing. Based on this connection, we adapt the Energy-based Constrained Decoding with Langevin Dynamics (COLD), a state-of-the-art, highly efficient algorithm in controllable text generation, and introduce the COLD-Attack framework which unifies and automates the search of adversarial LLM attacks under a variety of control requirements such as fluency, stealthiness, sentiment, and left-right-coherence. The controllability enabled by COLD-Attack leads to diverse new jailbreak scenarios which not only cover the standard setting of generating fluent (suffix) attack with continuation constraint, but also allow us to address new controllable attack settings such as revising a user query adversarially with paraphrasing constraint, and inserting stealthy attacks in context with position constraint. Our extensive experiments on various LLMs (Llama-2, Mistral, Vicuna, Guanaco, GPT-3.5, and GPT-4) show COLD-Attack's broad applicability, strong controllability, high success rate, and attack transferability. Our code is available at https://github.com/Yu-Fangxu/COLD-Attack.

1. Introduction

The paper addresses controllability in automatic white-box LLM attacks, arguing that diverse requirements beyond fluency are needed for thorough safety assessment. It introduces COLD-Attack and evaluates its controllable attack generation across multiple settings.

  • Automatic white-box attacks require controllability beyond fluency to support adversarial testing across diverse attack features.Existing methods can produce semantically weak attacks or focus primarily on fluency for bypassing perplexity filters.
  • COLD-Attack connects controllable attack generation with controllable text generation and unifies automated control over fluency, stealthiness, sentiment, and left-right-coherence.It adapts COLD using Langevin dynamics in continuous logit space and guided decoding into discrete text.
  • COLD-Attack supports fluent suffix attacks, paraphrase attacks with or without sentiment control, and position-constrained attacks with output requirements.The numerical study covers continuation, paraphrasing, and position constraints.
  • COLD-Attack attacks can remain fluent while adhering to predefined user requirements, yielding a more versatile and controllable attack strategy.The authors present the framework as complementary to existing approaches rather than a replacement.

2. Related Work

Related work includes black-box and white-box jailbreak methods alongside controllable text-generation techniques that steer outputs using keywords, gradients, discriminators, reinforcement learning, or guided sampling.

  • Black-box jailbreaks commonly draw on manually crafted malicious prompts and may use personas, persuasion, or LLMs to construct attacks.
  • The paper distinguishes three attack settings: continuation constraint, paraphrasing constraint, and position constraint.
  • The related controllable text-generation literature includes keyword steering, future-token gradients, auxiliary discriminators, reinforcement learning, and guided sampling.

3. Controllability and Stealthiness for Attacks

The paper formulates controllable attacks as sequences that both jailbreak a target LLM and satisfy explicit constraints. This formulation supports stealthier and more diverse attacks through continuation, paraphrasing, and position-constrained settings.

  • 3.1. General Problem: Controllable Attack Generation: Controllable attack generation seeks a sequence y that successfully attacks an LLM while satisfying additional constraints on features such as semantics, style, keywords, or position.
  • 3.1. General Problem: Controllable Attack Generation: The formulation represents requirements with indicator functions, including successful attack and fluency, while extra constraints add further control over y.
  • 3.2. Relevance to Stealthy LLM Attacks: Flexible constraints can make attacks stealthy by enforcing fluency, semantic, sentiment, or position properties directly on the attack.
  • Attack with Continuation Constraint: Continuation attacks append y to query x so x ⊕y is fluent and can trigger harmful content.
  • Attack with Paraphrasing Constraint: Paraphrasing attacks replace query x with semantically equivalent y that is fluent and triggers harmful content, preventing simple suffix removal defenses.
  • Attack with Position Constraint: Position-constrained attacks hide y between query x and prompt p, producing fluent x ⊕y ⊕p while satisfying required harmful responses and output controls.
  • Connection to Controllable Text Generation: The paper links its attack formulation to controllable text generation because removing the successful-attack constraint leaves the standard constrained-generation problem.

4. COLD-Attack

COLD-Attack adapts energy-based controllable text generation to automate adversarial attack search under fluency, semantic, lexical, and positional requirements. It combines weighted energy functions, Langevin sampling, and a final decoding step across three attack settings.

  • Framework: COLD-Attack adapts COLD to solve controllable attack generation automatically by integrating attack-specific energy functions.The framework uses COLD's energy-based formulation and incorporates the adversarial cost used in GCG as an additional energy function.
  • Decoding: The final decoding method is necessary for producing fluent discrete attacks from the continuous logits generated by Langevin sampling.Without COLD's decoding method, the resulting text is typically not fluent.
  • Framework: The framework has three steps: formulate compositional energies, sample continuous logits with Langevin dynamics, and decode them into discrete text attacks.Langevin dynamics runs recursively for N steps before an LLM-guided decoding process converts the final logits into text.
  • Energy Functions: Each requirement is represented by an energy function, with lower values indicating better satisfaction, and weighted energies combined as E(˜y) = ∑i λiEi(˜y).The functions can encode attack success, fluency, sentiment, position, style, and keyword constraints.
  • Energy Functions: COLD-Attack uses attack-success, fluency, semantic-similarity, and lexical energies to control jailbreak behavior and output characteristics.The semantic energy encourages paraphrases of the original query, while lexical energies control desired keywords or suppress refusal phrases.
  • Attack Settings: The method supports continuation, paraphrasing, and position-constrained attacks by composing energies appropriate to each setting.Continuation attacks optimize fluency and success over x ⊕ y; paraphrase attacks add semantic similarity and optional sentiment keywords; position attacks insert y between x and p while maintaining whole-sentence fluency.

5. Experimental Evaluations

Experiments across multiple white-box LLMs and three attack settings evaluate COLD-Attack’s success, fluency, diversity, efficiency, and controllability. The method performs strongly across continuation, paraphrasing, and position-constrained attacks while satisfying varied output requirements.

  • Evaluation Setup: Evaluations use Vicuna, Llama2, Guanaco, and Mistral, with 50 harmful instructions from AdvBench and metrics including ASR, ASR-G, PPL, Succ, BERTScore, BLEU, and ROUGE.ASR measures harmful-output elicitation, while ASR-G uses GPT-4 to assess whether responses fulfill malicious instructions.
  • Continuation Constraint: COLD-Attack achieves the best or second-best ASRs across all evaluated LLMs and the highest ASR-G on Guanaco and Mistral.It ranks second on Vicuna and Llama2 with comparable success rates.
  • Continuation Constraint: COLD-Attack generates the lowest-PPL adversarial prompts across the evaluated LLMs, outperforming AutoDAN-Zhu in fluency and stealthiness.AutoDAN-Liu partially relies on manually crafted prompts, so the authors describe it as complementary rather than a directly comparable baseline.
  • Continuation Constraint: COLD-Attack consistently outperforms AutoDAN-Liu and AutoDAN-Zhu on DNS, ADN, and Self-BLEU diversity metrics.These metrics quantify the diversity of generated adversarial prompts.

6. Conclusion

The paper introduces COLD-Attack for automatically generating stealthy, controllable adversarial prompts. Evaluations across three attack settings support its versatility and effectiveness, while the authors position controllable-text-generation methods as a direction for diversifying LLM attacks.

  • Conclusion: COLD-Attack automatically embeds varied controlled features within adversarial prompts across three distinct attack settings.The conclusion characterizes the framework as versatile and effective.
  • Conclusion: The paper connects LLM jailbreaking with controllable text generation to motivate further research on diversifying LLM attacks.The authors present this unified perspective as an inspiration for future work.

Impact Statement

The paper presents COLD-Attack as a controllable adversarial-prompt method and acknowledges risks from generating fluent, stealthy, contextually coherent attacks. It recommends safeguards and further defense research to mitigate misuse.

  • Impact Statement: The method can generate fluent, stealthy, and contextually coherent adversarial prompts that pose new challenges for LLM defense.The paper identifies potential misuse including misinformation, bypassing content moderation, and harmful or biased outputs.
  • Impact Statement: The paper suggests using safeguards such as Llama Guard and using generated attacks for safety fine-tuning and safeguard design.It also calls for research on detecting stealthy prompts, adversarial training, collaborative defenses, and ethical research practices.

C.3. COLD-Attack Experimental Setup

COLD-Attack samples multiple candidate prompts from an energy-induced distribution, then uses a designed GPT-4 prompt to select the final output. Experiments use fixed iteration, step-size, noise-schedule, length, and instruction controls.

  • C.3. COLD-Attack Experimental Setup: COLD-Attack generates 8 samples of length 20 from the optimized energy-induced distribution and selects one using a designed GPT-4 prompt.The selection prompt is shown in Figure 4.
  • C.3. COLD-Attack Experimental Setup: The method runs for 2000 iterations with step size η = 0.1 and a decreasing noise schedule σ = {1,0.5,0.1,0.05,0.01}.The listed noise values are applied at iterations n = {0,50,200,500,1500}, respectively.
  • C.3. COLD-Attack Experimental Setup: For constrained attack scenarios, the experiments use four handcrafted instructions to control language-model outputs.These instructions are listed in Table 10, while Table 11 reports hyperparameter settings for each jailbreak type.

C.4. Evaluation Metrics

The evaluation combines attack success, fluency, controllability, quality, diversity, and transferability measures. It supplements substring-based ASR with GPT-4-based assessment because substring matching can misclassify responses.

  • C.4. Evaluation Metrics: Attack Success Rate (ASR) measures the percentage of instructions that elicit corresponding harmful outputs.The paper computes ASR using an augmented list of rejection phrases, while noting that substring matching can misclassify responses.
  • C.4. Evaluation Metrics: GPT-4-based ASR (ASR-G) assesses whether responses accurately fulfill malicious instructions and shows higher correlation with human annotations.The assessment uses a prompt template shown in Figure 5.
  • C.4. Evaluation Metrics: Perplexity (PPL) evaluates prompt fluency, while Succ measures the percentage of samples that satisfy specified controls such as sentiment and vocabulary.BERTScore, BLEU, and ROUGE are also used to evaluate generated controllable attacks.
  • C.4. Evaluation Metrics: Diversity is evaluated with Distinct N-grams Score, averaged distinct N-grams, and Self-BLEU.DNS and ADN increase with diversity, whereas higher Self-BLEU indicates less diversity; ADN uses n from 1 to 5.
  • C.4. Evaluation Metrics: COLD-Attack samples selected outputs using a template adapted from prior work, and transferability is evaluated on GPT-3.5 turbo and GPT-4.The transferability results are reported in Tables 13 and 14.

D.2. Against Defense

COLD-Attack is evaluated against multiple defenses, including perplexity filtering, preprocessing, moderation, SmoothLLM, RAIN, and Llama Guard. The reported results indicate robustness across defenses, with performance varying by defense and model.

  • D.2. Against Defense: COLD-Attack’s ASR increases as the perplexity-filtering threshold rises from 20 to 60, and most sampled prompts bypass the filter at threshold 60.These results are shown in the left plot of Figure 6.
  • D.2. Against Defense: After GPT-4 rewrites adversarial prompts, COLD-Attack maintains a certain degree of robustness against preprocessing defense.The before-and-after comparison appears in the right plot of Figure 6.
  • D.2. Against Defense: Llama Guard is the strongest defense among those studied, yet COLD-Attack performs best against Vicuna, Guanaco, and Mistral in the comparison.For Llama2, AutoDAN-Zhu and AutoDAN-Liu outperform COLD-Attack under Llama Guard.
  • D.2. Against Defense: COLD-Attack outperforms both AutoDAN-Zhu and AutoDAN-Liu on 13b models in both ASR and ASR-G.The larger-model evaluation covers Vicuna, Guanaco, and Llama-2 13b systems.

D.4. Ablation Study

The ablations examine how sentiment and loss-weight choices affect attack success, while testing whether success is robust to the attack-success weight. Positive sentiment can improve ASR, whereas increasing fluent-loss weight reduces it.

  • Hyper-parameter effects: Increasing the fluent-loss weight λste from 0.1 to 100 noticeably decreases ASR on Llama2 suffix attacks.The study attributes this decrease to reduced focus on minimizing jailbreak loss.
  • Sentiment steering: Positive sentiment may produce higher ASRs than negative sentiment for Llama2 under position-constrained attacks.The observation also aligns with sentiment-controlled paraphrase attacks.
  • Ablation setup: Figure 7 compares ASR (%) across different fluent-loss weights.The figure is the ablation study’s visualization of the fluent-loss weight effect.
  • Hyper-parameter robustness: Success rate remains stable when the attack-success weight λatt changes on Mistral-7b.The authors interpret this stability as evidence of robustness to this hyper-parameter.

D.5. Full Result on 520 Samples

The full evaluation reports attack success and perplexity across four evaluated LLMs, and separately assesses semantic coherence between original prompts and generated suffixes.

  • Full-dataset evaluation: Table 20 reports ASRs and PPL on all 520 malicious requests in AdvBench across four evaluated LLMs.This expands evaluation beyond selected examples to the full dataset.
  • Coherence evaluation: The coherence measure uses BERT’s next-sentence prediction objective to classify whether suffix y follows prompt x.The evaluation operationalizes semantic coherence through NSP classification.
  • Coherence evaluation: Generated suffixes are coherent with instructions according to high BERTS scores.BERTS is the percentage of suffixes predicted by BERT as the next sentence of the associated prompt.
  • Coherence evaluation: Table 21 evaluates coherence between prompts and continuations.It presents the reported prompt-continuation coherence results.

D.7. Comparison with Black-Box Methods

The paper compares COLD-Attack with black-box methods and discusses how system prompts affect attack behavior, while illustrating attacks under several control constraints.

  • Black-box comparison: COLD-Attack is compared with DeepInception, TAP, and PAIR using attack success rates.The detailed comparison is reported in Table 22.
  • System-prompt effects: System prompts are designed to make LLMs follow user queries safely and helpfully, which can improve model safety.The authors study their effects because system prompts were omitted from the main attack optimization and evaluation setup.
  • System-prompt effects: With system prompts present, COLD-Attack may fail to reduce attack loss sufficiently, especially because its compositional loss includes penalties beyond attack loss.The discussion contrasts this with GCG’s direct optimization of attack loss.
  • Optimization trajectories: Figures 8 and 9 show attack-loss, fluency-loss, and lexical-constraint trajectories for COLD-Attack and GCG.The displayed trajectories support the discussion of optimization behavior under different attack settings.
  • System-prompt effects: Very small attack-loss values may be important for maintaining attack effectiveness when system prompts are present.The authors identify improving this ability as an area for further research.
  • Controlled attack examples: Examples cover continuation, paraphrasing, and position constraints, with formatting distinguishing malicious requests, generated attacks, and control prompts.The examples also mark sentiment keywords and controlled output features where applicable.
Loading 2402.08679v2…