COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability

• Fluency alone does not guarantee stealthiness, and imposing features such as sentiment or contextual coherence on automatic white-box attacks remains an open challenge.The introduction contrasts this gap with prior efforts focused on fluency to bypass perplexity filters.Source: §1 · p. 1
• COLD-Attack formulates controllable attack generation as controllable text generation, using energy functions, Langevin sampling in continuous logit space, and guided decoding into discrete text.The framework adapts Energy-based Constrained Decoding with Langevin Dynamics (COLD) to unify automated attack search under controls including fluency, stealthiness, sentiment, and left-right coherence.Source: §1 · p. 2
• COLD-Attack supports paraphrasing a given question into an attack, optionally with sentiment constraints, and inserting an attack between two sentences while maintaining left-right coherence.The authors present it as a complement, not a replacement, to methods such as GCG and AutoDAN.Source: §1 · p. 2
• Across three settings, COLD-Attack efficiently generates fluent continuation-constrained suffix attacks that outperform AutoDAN-Zhu (Zhu et al., 2023), sentiment-controlled paraphrase attacks, and position-constrained prompts meeting diverse output requirements.The position-constrained prompts can satisfy sentiment, lexical, format, and style requirements; attacks across settings remain fluent and adhere to predefined user requirements.Source: §1 · pp. 1–2

Black-box vs. White-box Attacks

• Black-box jailbreaks adapt manual prompts, use failure modes to guide attack design, and apply persona modulation or persuasion to elicit harmful responses.The cited examples include Wei et al. (2023a), Shah et al. (2023), and Zeng et al. (2024).Source: §2 · p. 2
• White-box search diversifies attacks: GCG produces detectable gibberish suffixes, while AutoDAN-Zhu generates fluent prompts but limits control.GCG uses token-level optimization; AutoDAN-Zhu uses double-loop optimization with autoregressive token-by-token generation (Zou et al., 2023; Jain et al., 2023; Zhu et al., 2023).Source: §2 · p. 2
• AutoDAN-Liu combines automatic genetic search with manually crafted jailbreak prompts, but how to control its generated attacks remains unclear.(Liu et al., 2023d).Source: §2 · p. 2

Controllable Text Generation

• Controllable text-generation methods address lexical constraints, use discriminators or other guidance, and include COLD, which adopts energy models.Neurologic steers beam search with keyword scores (Lu et al., 2020; 2021); DeLorean uses future-token gradients (Qin et al., 2020), while PPLM and Fudge use auxiliary discriminators (Dathathri et al., 2019; Yang & Klein, 2021). Later methods include RL (Lu et al., 2022; Mudgal et al., 2023) and guided sampling (Liu et al., 2023c; Forristal et al., 2023; Mireshghallah et al., 2022); COLD is identified as state of the art (Qin et al., 2022).Source: §2 · p. 3

3.1. General Problem: Controllable Attack Generation

• A controllable attack is a sequence y that successfully attacks the target LLM while satisfying fluency and optional constraints such as sentiment, position, style, or keywords.The formulation represents each requirement with an indicator function that is 1 when y satisfies the constraint.Source: §3.1 · p. 3

3.2. Relevance to Stealthy LLM Attacks

• For a continuation attack, appending y to query x must yield fluent text that triggers harmful content; this can make the suffix more stealthy than GCG against perplexity filtering.The setting had been considered previously (Zhu et al., 2023).Source: §3.2 · pp. 3–4
• A paraphrase attack replaces x with a fluent, semantically similar y that triggers harmful content, optionally steering y toward a specified sentiment and hiding the attack position.Because the attack is in the rewritten query, removing a suffix alone cannot defend against it.Source: §3.2 · p. 4
• A position-constrained attack inserts y between query x and prompt p, requiring x ⊕ y ⊕ p to be fluent and trigger harmful responses.Unlike the earlier continuation setting, fluency applies to the text on both sides of the inserted attack; the paraphrase and position settings are new.Source: §3.2 · p. 4

3.3. Connections with Controllable Text Generation

• Controllable attack generation differs from controllable text generation only by the successful-attack constraint, which can be induced by maximizing the probability of desired responses conditioned on y.This connection supports adapting existing controllable text generation algorithms, including COLD, while retaining the other constraints.Source: §3.3 · p. 4

4. COLD-Attack

• COLD-Attack adapts COLD (Qin et al., 2022) by combining weighted control energies, Langevin sampling over continuous logits, and LLM-guided decoding into discrete attacks.It initializes sampling with large noise for broad exploration; it samples the discrete attack once at the end rather than optimizing discrete tokens at every step.Source: §4 · pp. 4–5

4.1. Energy Functions for Controllable Attacks

• Its energies encode attack success via adversarial cost (Zou et al., 2023), fluency via autoregressive predictions, semantic similarity via embeddings, and lexical inclusion or refusal suppression via n-gram matching.The differential n-gram matching function approximates BLEU-n (Liu et al., 2021b; Papineni et al., 2002).Source: §4.1 · pp. 5–6

4.2. Final Energy-based Models for Attacks

• The final models optimize successful, fluent continuation (x⊕y), paraphrase (y similar to x), or insertion (x⊕y⊕p), with sentiment keywords available for paraphrases.The insertion setting also constrains fluency and coherence of the full x⊕y⊕p sequence.Source: §4.2 · p. 6

5.1. Results: Attack with Continuation Constraint

• COLD-Attack achieves best or second-best ASRs across evaluated LLMs, leads ASR-G on Guanaco and Mistral, and produces the lowest PPL among evaluated methods.GCG and GCG-reg reach 100% ASR on Vicuna, Guanaco, and Mistral after early termination when outputs lack predefined rejection phrases; COLD-Attack also has lower PPL than AutoDAN-Zhu across models, while AutoDAN-Liu uses manually crafted prompts and has lower PPL on Vicuna and Guanaco.Source: Table 2 · p. 7, §5.1 · p. 7

(iii) COLD-Attack can generate diverse adversarial

• COLD-Attack generates more diverse adversarial prompts than AutoDAN-Liu and AutoDAN-Zhu and runs an average of 10× faster than GCG and GCG-reg.On a single NVIDIA V100 GPU, one request takes about 20 minutes for COLD-Attack (2,000 steps, batch 8), versus approximately 3.23 hours for GCG and GCG-reg (500 steps, batch 512).Source: (iii) COLD-Attack can generate diverse… · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8

5.2. Results: Attack with Paraphrasing Constraint

• Under paraphrasing constraints, COLD-Attack outperforms PRISM, PAWS, and GPT-4 paraphrasing in ASR, while sentiment steering yields model-dependent ASR-G gains.GPT-4 paraphrases score BLEU 0.37, ROUGE 0.38, and BERTScore 0.75; COLD-Attack’s BERTScore exceeds 0.7 on all LLMs. Negative sentiment raises ASR-G by 30% on Mistral and 14% on Guanaco, while positive sentiment raises it by 18% on Llama2 relative to negative sentiment.Source: Table 5 · p. 8, §5.2 · p. 8, §5.3 · p. 8

5.3. Results: Attack with Position Constraint

• For position-constrained attacks, ASR and ASR-G exceed 80%, constraint success remains above 68%, and COLD-Attack records the lowest PPL across constraints.PPL peaks at 99.03 for Guanaco under lexical constraint; against baselines, COLD-Attack’s PPL averages 2× lower than AutoDAN-Zhu and approximately 40× lower than GCG. The evaluation omits the system prompt.Source: Table 6 · p. 9, §5.3 · p. 9, Table 8 · p. 9

6. Conclusion

• Evaluations across three distinct attack settings show COLD-Attack embeds diverse controlled features and is versatile and effective, connecting LLM jailbreaking with controllable text generation.The authors hope this unified perspective will inspire further research on diversifying LLM attacks.Source: §6 · p. 9

Impact Statement

• COLD-Attack can generate fluent, stealthy, contextually coherent adversarial prompts that may enable misinformation, moderation bypass, or harmful outputs, despite the study’s stated safety aims.The authors recommend safeguards such as Llama Guard and note that the method can also generate diverse attacks for safety fine-tuning and safeguard design.Source: Impact Statement · p. 10

A.1. Safety Aligned LLMs

• Safety alignment uses data filtering, supervised safety fine-tuning, and RLHF to promote helpful and harmless LLM responses (Brown et al., 2020; Touvron et al., 2023; Chung et al., 2022; Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022; Leike et al., 2018; Glaese et al., 2022).The passage describes filtering harmful or confidential training content, teaching safety guidelines, and using human-preference reward models.Source: §A.1 · p. 15

A.2. Jailbreak LLMs; A.3. Controllable Text Generation

• Prior work covers jailbreak prompts, model-training and multimodal or in-context attacks, while controllable generation uses constrained decoding, contrastive methods, search, instructions, and diffusion controls.Examples include AutoPrompt (Shin et al., 2020), GBRT (Wichers et al., 2024), multimodal attacks (Gong et al., 2023; Tu et al., 2023; Qi et al., 2023; Carlini et al., 2023), and constrained decoding (Post & Vilar, 2018), contrastive methods (Krause et al., 2020; Liu et al., 2021a), and diffusion controls (Li et al., 2022; Liu et al., 2023a).Source: §A.2 · p. 15, §A.3 · p. 15

B.1. More Details on the Energy Functions for Controllable Attacks

• Attack-success energy targets an affirmative response, fluency energy aligns logits with the underlying LLM’s predictions, and lexical energy steers keywords or suppresses rejection phrases.The lexical constraint uses differential n-gram matching, an approximation of BLEU-n.Source: §B.1 · p. 16

B.1. More Details on the Energy Functions for Controllable Attacks; B.2. LLM-Guided Decoding Process

• For paraphrase attacks, semantic-similarity energy keeps the rewritten query close in meaning to the original; decoding then uses the underlying LLM’s top-k next-token candidates to produce fluent text.The method first samples a continuous logit sequence, then selects tokens from LLM-guided candidates rather than simply taking the most likely token at each position.Source: §B.1 · p. 16, §B.2 · p. 16

C. Experimental Details; C.1. Large Language Models

• The experimental appendix evaluates COLD-Attack on seven popular open-source LLMs and provides links to the evaluated models.The section introduces the detailed experimental setups; Table 9 is identified as the source of model links.Source: C. Experimental Details · p. 17, §C.1 · p. 17, Table 9 · p. 17

C.2. Baselines Setup

• Suffix baselines include GCG with 20 tokens and 500 steps, GCG-reg with an added readability loss, and AutoDAN, evaluated using one of eight generated suffixes per sample.GCG uses batch size 512 and top-k 256; AutoDAN responses are used to compute ASR and ASR-G for each LLM.Source: §C.2 · p. 17
• Other 20-token, 500-step suffix baselines are PEZ, run with step size 5 × 10−2, GBDA, run with step size 0.2, and UAT.PEZ optimizes in embedding space before nearest-neighbor projection, GBDA uses Gumbel-softmax, and UAT applies HotFlip with first-order Taylor loss estimates.Source: §C.2 · p. 17

C.3. COLD-Attack Experimental Setup

• COLD-Attack generates 8 samples of length 20 and uses a designed GPT-4 prompt to select one as the final output.Figure 4 shows the prompt for selecting the harmful response from the generated samples.Source: §C.3 · p. 17, Figure 4 · p. 18
• The experiments use 2000 iterations with step size η = 0.1, a decreasing noise schedule, and 4 handcrafted instructions for constrained attacks.The noise schedule is σ = {1,0.5,0.1,0.05,0.01} at iterations n = {0,50,200,500,1500}, respectively; hyperparameters and instructions are listed in Tables 11 and 10.Source: §C.3 · p. 18, Table 11 · p. 18, Table 10 · p. 18

C.4. Evaluation Metrics

• ASR counts instructions eliciting harmful outputs through rejection-phrase matching, while ASR-G uses GPT-4 to judge whether responses fulfill malicious instructions.The authors say substring matching can misclassify responses and report that ASR-G has higher correlation with human annotations, making it a more reliable measure of attack effectiveness.Source: §C.4 · p. 18, Table 12 · p. 19, Figure 5 · p. 19
• Prompt fluency is measured with perplexity calculated using Vicuna-7b, while Succ measures the percentage of samples meeting specified sentiment and vocabulary requirements.These controls assess whether generated adversarial prompts adhere to the set requirements.Source: §C.4 · p. 19
• BERTScore, BLEU, and ROUGE evaluate the quality of generated controllable attacks.The passage refers readers to further discussion for details of these metrics.Source: §C.4 · p. 20
• Distinct N-grams Score (DNS) (Li et al., 2015) measures unique n-grams relative to all n-grams, with scores nearer 1 indicating higher diversity.Scores closer to 0 indicate lower diversity.Source: §C.4 · p. 20
• Averaged distinct N-grams (ADN) (Tevet & Berant, 2020) averages distinct n-gram scores for n = 1 to 5, while Self-BLEU (Zhu et al., 2018) measures sentence-level similarity, with higher scores indicating less diversity.The authors use these metrics, alongside DNS, to quantify generated-prompt diversity.Source: §C.4 · p. 20

D.1. Transferability

• On GPT-3.5 turbo, COLD-Attack exceeds AutoDAN-Zhu on ASR-G across Guanaco, Mistral, and Llama2 among automated methods, while AutoDAN-Liu also performs effectively.The authors note that AutoDAN methods often generate off-topic content, which contributes to larger gaps between ASR and ASR-G than for COLD-Attack; they also describe COLD-Attack as generating more diverse attacks that transfer from open-source LLMs.Source: Table 13 · p. 20, §D.1 · p. 20
• On GPT-4, COLD-Attack and AutoDAN-Liu have comparable ASR-G scores and outperform or match AutoDAN-Zhu across all models; GPT-4 is less conservative than GPT-3.5.The authors suggest GPT-4’s greater willingness to fulfill user requests may increase the risk of creating inappropriate content.Source: §D.1 · p. 20

D.2. Against Defense

• The suffix-attack evaluation reports ASR after OpenAI moderation, RAIN, and Llama Guard filtering, as well as against perplexity and preprocessing defenses.The broader evaluation also includes SmoothLLM and covers the studied LLMs.Source: §D.2 · pp. 20–21, Table 15 · p. 22
• At a PPL threshold of 60, considered relatively restrictive, the majority of COLD-Attack prompts bypass the filter; ASR increases as thresholds rise from 20 to 60.The passage compares the threshold of 60 with a maximum PPL above 100 for malicious AdvBench queries.Source: §D.2 · p. 21, Figure 6 · p. 21
• COLD-Attack maintains a degree of robustness after an LLM-based preprocessing defense rewrites its adversarial prompts.GPT-4 performs the rewrite in the experiment.Source: §D.2 · p. 21, Figure 6 · p. 21
• COLD-Attack remains effective under SmoothLLM, which perturbs multiple prompt copies and uses majority voting to detect adversarial inputs.The authors conjecture that this robustness may stem from COLD-Attack producing semantically meaningful prompts compared with GCG.Source: §D.2 · p. 21, Table 15 · p. 22
• Under Llama Guard, COLD-Attack outperforms GCG and both AutoDAN variants on Vicuna, Guanaco, and Mistral, while both AutoDAN variants outperform it on Llama2.The paper notes that AutoDAN-Zhu and AutoDAN-Liu have higher PPL than COLD-Attack, and conjectures that Llama Guard may be easier to bypass with less natural sentences; higher-PPL prompts may be more detectable by simpler defenses such as PPL filtering.Source: §D.2 · p. 22, Table 16 · p. 22

D.3. Attack on Larger LLMs

• On 13b Vicuna, Guanaco, and Llama-2 models, COLD-Attack outperforms AutoDAN-Zhu and AutoDAN-Liu in ASR and ASR-G.The evaluation covers vicuna-13b-v1.5, guanaco-13B-HF, and Llama-2-13b-chat-hf.Source: §D.3 · p. 22, Table 17 · p. 22

Hyper-parameter

• On Llama2 suffix attacks, increasing fluent-loss weight λste across {0.1, 1, 10, 100} noticeably decreases ASRs.The paper attributes this decrease to greater focus on fluent loss and less focus on minimizing jailbreak loss.Source: §D.4 · p. 22, Figure 7 · p. 23

Sentiment Steering

• Under position-constrained attacks on Llama2, positive sentiment may yield higher ASRs than negative sentiment, consistent with paraphrase-attack observations.The sentiment control prompt was changed from positive to negative.Source: §D.4 · pp. 22–23, Table 18 · p. 23

Robustness of Hyperparameters

• With Mistral-7b, success rate remains stable as attack-success weight λatt changes, indicating robustness to this hyperparameter.The results are reported in Table 19.Source: §D.4 · p. 23, Table 19 · p. 23

D.5. Full Result on 520 Samples

• Table 20 reports attack success rates and PPL on the full AdvBench dataset of 520 malicious requests across four evaluated LLMs.The supplied passage does not provide the table's individual values.Source: §D.5 · p. 23, Table 20 · p. 23

D.6. Coherence of Prompt and Continuation

• The coherence assessment uses BERT, pretrained with a next-sentence prediction objective, to classify whether suffix y is the next sentence of prompt x.The passage attributes the BERT model to Devlin et al., 2018.Source: §D.6 · p. 23
• Generated suffixes are coherent with instructions, with high BERTS scores in the dataset.BERTS is the percentage of suffixes predicted as the next sentence of their associated prompt.Source: §D.6 · p. 24, Table 21 · p. 24

D.7. Comparison with Black-Box Methods

• Table 22 reports attack success rates for COLD-Attack and DeepInception, TAP, and PAIR, but the supplied passages do not establish which method performs best.The cited passages provide no table values, so no numerical comparison can be reported.Source: Table 22 · p. 24, §D.7 · p. 24

D.8. More Discussions on the Impact of System Prompts of LLMs

• System prompts instruct models to follow user queries safely and helpfully, and the authors observed that they can significantly improve LLM safety.The appendix identifies system prompts for Vicuna-7b-v1.5, Mistral-7b-Instruct, and Llama2-7b-chat-hf.Source: §D.8 · p. 24, Table 23 · p. 24
• With system prompts included, COLD-Attack performance decreases on Mistral and Llama2 but not Vicuna; Guanaco was omitted because its default prompt was unavailable.The main-paper setup excluded system prompts during attack optimization and evaluation.Source: §D.8 · p. 24, Table 24 · p. 25
• COLD-Attack’s compositional losses constrain how far its attack loss falls, while lexical constraints suppress rejection words and fluency loss supports fluent attacks.The authors report that lexical constraint loss converges quickly and that system prompts reduce the number of cases succeeding despite attack loss remaining above very small values.Source: §D.8 · p. 25
• COLD-Attack optimizes soft prompts, whose low attack loss need not imply low hard-prompt loss; the authors suggest GCG-like decoding or combining COLD-Attack with GCG.They propose further reducing hard-prompt attack loss to improve performance with system prompts and note that more research is needed.Source: §D.8 · p. 26

D.9. More Selected Examples

• Additional selected examples cover attacks with continuation, paraphrasing, and position constraints, including sentiment control in paraphrases and user-specified control prompts.Tables 25–27 show rephrased prompts or sampled text alongside the original malicious requests; Table 27 also highlights outputs associated with the control prompt.Source: §D.9 · p. 26, Table 25 · p. 27, Table 26 · p. 28