Coercing LLMs to do and reveal (almost) anything

1 INTRODUCTION

• The paper broadens adversarial-attack objectives beyond bypassing alignment, which prior work focused on in chat models tuned through RLHF to be harmless and helpful.The introduction frames the work as an overview of what objectives to optimize, complementary to work on improved optimizers.Source: §1 · pp. 2–3
• Examples include nonsense input that reliably elicits a target string and non-alphabetic input that leaks an unknown system prompt verbatim despite instructions not to do so.The examples are shown in Figure 1.Source: §1 · p. 2
• Attack categories include extracting context-window information, misdirecting users with URLs or malicious instructions, exhausting GPU resources, and manipulating model control flow.The paper also considers more complex attacks on alignment in Appendix C.1.Source: §1 · p. 2
• Optimized attacks use varied strategies, including (re)programming, language switching, role hacking, glitch tokens, calls to action, and appeals to authority.The examples in Table 1 are for the LLaMA-2 7b chat model.Source: §1 · pp. 2–3, Table 1 · p. 3

2 WHY ARE ADVERSARIAL ATTACKS RELEVANT?

• For text-only applications, the stated worst case is that a user receives information they were not supposed to obtain, such as repurposing a travel agent to write homework exercises.The paper characterizes current-generation text-only applications as text simulators.Source: §2 · p. 3
• Applications that interface with other systems or mediate a physical robot’s actions could face harmful attacks if an LLM can be coerced into arbitrary behavior.The paper contrasts these executive-capacity systems with applications that merely return text.Source: §2 · p. 3
• An email assistant could be coerced by a malicious email to copy inbox messages to the sender and delete evidence of that behavior.The paper cites this as an example of a harmful outcome for an assistant that reads and answers email.Source: §2 · p. 3

3 HOW ARE ADVERSARIAL ATTACKS AGAINST LLMS FOUND?

• Manual and semi-automated red-teaming identifies exploitable weaknesses, including refusal suppression, generalization mismatches, style injections, practical jailbreak tricks, and persuasion strategies.The cited work includes findings from competitions and attacks observed in the wild.Source: §3 · p. 4
• Earlier optimization attacks against modern LLMs initially succeeded mainly when auxiliary input was available, including in vision-language and audio-language settings.Adversarial optimization itself had already been studied in NLP.Source: §3 · p. 4
• Successful LLM attack optimizers include gradient-based, zeroth-order, and model-guided strategies; gradient attacks need white-box access but can transfer to black-box models.Gradient methods alternate continuous embedding-gradient evaluations with discrete selection of similar candidate tokens.Source: §3 · p. 4
• Under assumptions, a theoretical result states that a sufficiently long prompt can coerce any behavior with non-zero probability, despite measures such as RLHF.The formalization is attributed to Wolf et al. (2023).Source: §3 · p. 4

4.2 SETUP AND IMPLEMENTATION

• Experiments use white-box open-source models, usually LLaMA-2-7b-chat, and measure attack success as average substring overlap across 50 sampled completions.The study occasionally tests larger LLaMA-2 chat or related models.Source: §4 · p. 4, §4.2 · p. 6

4.1 BASICS

• The attack framework optimizes discrete adversarial tokens inserted between context segments to produce a target completion, using GCG in most experiments.The objective can also minimize KL divergence between source- and target-model probabilities.Source: §4 · p. 4, §4.1 · p. 5

4.3 SUSCEPTIBILITY TO BASIC ATTACKS

• Optimized prompts can elicit profanity despite chat models’ benign-response training, rediscovering strategies such as style injection, language switching, and role hacking.Programming instructions are a prominent style-switching strategy; non-Latin-only attacks can succeed mostly through role hacking.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 EXTRACTION ATTACKS

• Universal system-repeater attacks reproduce unseen test prompts, with success remaining high for non-alphabetic or non-Latin constraints.Removing code tokens or restricting optimization to English words makes the attacks less effective.Source: §4.4 · p. 8
• Extraction attacks also induce refusals across sampled contexts, exposing patterns in LLaMA-2 RLHF training data focused on concrete harms and calls to action.The table notes that interpretability decreases as attack length grows.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 MISDIRECTION ATTACKS

• Misdirection examples use Chinese-character prompts to elicit URLs, or coerce a customer-service chatbot into issuing a fake refund despite its instructions.The authors caution that the fake-refund examples show why current models cannot be used for autonomous decisions.Source: §4.5 · p. 10

4.6 DENIAL-OF-SERVICE ATTACKS

• A 64-token sponge attack raises LLaMA-2 chat’s average response length from 128 tokens to 6019 tokens, with greedy decoding reaching 14613.The authors connect this result to the need for a stricter maximum-token limit per response.Source: §4.6 · p. 11

4.7 CONTROL ATTACKS

• Universal 16-token attacks force EOS regardless of preceding context, potentially interrupting LLM-agent workflows if the behavior persists in future systems.The authors distinguish this potential relevance to agents from its limited relevance to chatbots.Source: §4.7 · pp. 11–12

5.1 WHAT STRATEGIES ARE BEING EXPLOITED?

• The attacks exploit several strategies: code-like reprogramming, language switching, role hacking, and task-directed calls to action.Reprogramming uses the model’s learned code domain and its different ruleset; language switching often uses Chinese, German, or Russian; role hacking exploits confusion among system, user, and model instructions; calls to action include requests such as “write repeatedly” and “respond Yes.”Source: §5.1 · pp. 12–13
• Appeals to imagined authorities appear in fake-refund attacks, which the authors associate with model receptivity to persuasion.Examples include invocations of imagined people in the attack string.Source: §5.1 · p. 13, Table 5 · p. 13
• The described strategies are not necessary for attack success: disabling code tokens makes system-repeater attacks harder, but does not make them impossible.The authors also report that optimization rediscovers strategies previously found through manual red-teaming and includes new approaches.Source: §5.1 · p. 13

5.2 TOKEN FREQUENCIES

• Attack-frequency analysis finds punctuation and brackets, INST, and cdnjs among recurring LLaMA-2-7b tokens, linked respectively to reprogramming or role hacking and request-following.Figure 9 presents relative token frequencies by model; the authors describe cdnjs as appearing especially at request endings.Source: §5.2 · p. 14, Figure 9 · p. 14
• LLaMA-2 attacks include glitch tokens such as Mediabestanden and oreferrer, which the authors describe as tokenizer artifacts underrepresented in training data that can induce behavior bypassing intended behavior.The authors identify several such tokens and recommend auditing tokenizer construction or using higher-quality construction data to prevent these artifacts.Source: §5.2 · p. 14

5.3 CONVERSION RATES OF CURRENT ATTACKS

• Figure 10 broadly supports a proportional relationship between target-string length and numbers-test difficulty, though the relationship may be nonlinear and optimizer quality adds noise.The authors note that longer targets may require attack strings to grow faster, which would affect the maximum text length coercible within a context window.Source: §5.3 · p. 14, Figure 10 · p. 15

6 WHAT ARE THE CURRENT LIMITS?

• The section presents unsuccessful attacks and open problems as remaining limits of current attacks.Its examples span objective misspecification, constrained token sets, floating-point overflow, collisions, and post-attack recovery.Source: §6 · p. 15, §6.1 · p. 15, §6.2 · p. 15

6.1 OBJECTIVE MISSPECIFICATION

• Refusal-suppression objectives can favor Shakespearean answers that avoid predefined refusal phrases without achieving the intended harmful-response goal.The authors stress that capable optimizers require well-specified objectives.Source: §6.1 · p. 15

6.2 CONSTRAINT SET SIZES

• Current optimizers succeed with restricted token sets, including non-alphabetic or Chinese tokens, but struggle with byte-token constraints and emoji-only targets.The section notes that such constraints could yield invisible or zalgo-character attacks; Table 6 describes constraint-set cardinalities and examples for the LLaMA-2 tokenizer.Source: §6.2 · p. 15, Table 6 · p. 16

6.3 DENIAL OF SERVICE ATTACKS THROUGH FLOATING POINT OVERFLOW

• Current attacks cannot raise intermediate activations beyond float16 precision, limiting the proposed overflow denial-of-service attack.The attack seeks to maximize activations beyond the permitted range; a successful overflow could invalidate responses across a large inference batch, especially if requests are requeued.Source: §6.3 · pp. 15–16

6.4 COLLISION ATTACKS

• Collision attacks are moderately hard to optimize and showed no greater success than directly targeting specific behaviors.They optimize KL divergence between target-sequence and attack-sequence probabilities while replacing selected target words with attack tokens.Source: §6.4 · p. 16

6.5 BREAKING THE SPELL

• Conversational models may recover after an adversarial response, sometimes because role hacking obscures that the model itself produced the harmful text.Models can also contradict themselves and comment on earlier text, which they cannot remove.Source: §6.5 · p. 16

7 CONCLUSIONS

• Misdirection, denial-of-service, and extraction attacks show capabilities that could cause harm in applications using current models, beyond jailbreaks.The work complements research improving optimizers and strategies for existing jailbreak objectives by surveying other possible objectives.Source: §7 · p. 17
• Optimized attacks recover manual red-teaming strategies and reveal behaviors such as role hacking and glitch-token use.The broadest mechanism identified is coercing models to simulate code and complete programming instructions or function calls rather than follow trained conversational behavior.Source: §7 · p. 17
• The authors conclude that arbitrary free-text inputs allow almost any outcome, so user-input-generated model outputs must be assumed insecure from a security perspective.They add that there might be no fix.Source: §7 · p. 17

BROADER IMPACT

• The paper presents new adversarial attack objectives that could technically be deployed against production systems, while acknowledging that its attacks were not optimized against real-world systems.The authors did not use model ensembles that could improve transfer to black-box systems, and state that this step is small.Source: BROADER IMPACT · p. 17

A ADDITIONAL BACKGROUND MATERIAL

• Zero-th-order attacks can work against black-box models with sufficient query access, while model-guided attacks are successful but their optimality remains unclear.The background also describes randomized substitution attacks using logit information and other jailbreak strategies based on representation manipulation, fine-tuning, or sampling parameters.Source: A ADDITIONAL BACKGROUND MATERIAL · p. 23
• Potential defenses against automated attacks are noted but are not the focus of this work.The authors say they prioritize understanding the space of possible attacks over constructing defenses based on narrowly defined characteristics of current attacks.Source: A ADDITIONAL BACKGROUND MATERIAL · p. 23

B IMPLEMENTATION

• Padding sampled inputs to uniform lengths and adapting masks and position IDs accommodates batched GCG experiments.The implementation also caches constant-token keys and values and, for non-stochastic objectives, previously computed losses.Source: B IMPLEMENTATION · p. 23
• Gradient checkpointing mitigates predictable per-iteration memory spikes without slowing attacks in practice.The authors also use Flash Attention-2 where possible and a model-parallel approach across consumer GPUs.Source: B IMPLEMENTATION · p. 23
• Figure 12 depicts a hidden jailbreaking attack against LLaMA-2-70b-chat and reports attack success rates across models; smaller LLaMA models are less capable at building bombs.The figure identifies a qualitative example on the left and success rates on the right.Source: Figure 12 · p. 24

• The section gathers additional examples.Source: C ADDITIONAL EXAMPLES · p. 24
• It also includes a few quantitative results mentioned in the main body.Source: C ADDITIONAL EXAMPLES · p. 24
• The section is best read through the main-body references.Source: C ADDITIONAL EXAMPLES · p. 24

C.1 SPECIALIZED JAILBREAKING ATTACKS

• Jailbreaking directly coerces harmful information, but complex variants are not the work’s focus and are relegated to the appendix.Source: §C.1 · p. 24
• An uninterpretable single-behavior jailbreak can avoid safety filters that detect malicious substrings in a prompt.Source: §C.1 · p. 24
• The implementation generates functional universal jailbreaks in the style of Zou et al. (2023), and these can be combined with alternative constraint sets.Source: §C.1 · p. 24
• 100% of the shown completion path is reached in the numbers attack example, with ASR also at 100%.The normal-behavior example assigns 0% probability to the target sequence.Source: Figure 13 · p. 25
• Table 8 supplies additional shutdown-attack examples across constraint sets; the examples also show reprogramming, role hacking, and glitch tokens, which are never necessary for attacks to succeed.Source: Table 8 · p. 25

C.2 PLAINTEXT REPRODUCTIONS

• A repeater variant repeats the entire conversation rather than only the system prompt.Source: Figure 14 · p. 26
• 100% ASR is reported for the Chinese-text misdirection example, while the abstract-generating attack reports 62.27% ASR.Source: Figure 15 · p. 26, Figure 16 · p. 27
• 42.75% ASR is reported for a misdirection attack against a LLaMA-2-70b-chat customer-service chatbot.Source: Figure 17 · p. 27
• The average-logit-maximizing attack is not useful as a denial-of-service attack in float16 or bfloat16, and its optimized text features long multi-token words and open clauses.Those features apparently raise logit values across the board.Source: Figure 18 · p. 28
• URL-fishing results are noisy: the left-hand attack is generally easier, but the shown Vicuna-7b run did not converge to a workable solution.Source: Figure 19 · p. 29
• The section includes additional offensive-profanity examples and a universal jailbreak generated in the style of Zou et al. (2023).Source: Figure 20 · p. 29, Figure 21 · p. 30
• One hidden attack produces an affirmative response that may be a cake recipe.Source: Figure 22 · p. 30
• Byte tokens are overrepresented in the relative-frequency analysis, but their role is hard to interpret without details of the glyphs constructed from them.The section also presents the most-used tokens by attack category with and without byte tokens.Source: Figure 23 · p. 31, Figure 24 · p. 32