In-Context Representation Hijacking

1 INTRODUCTION

• Doublespeak hijacks benign words’ internal representations through in-context learning, progressively transforming them into harmful concepts rather than merely obfuscating surface tokens.The attack replaces a harmful keyword with a benign substitute across examples.Source: §1 · p. 1
• A prompt asking how to build a carrot can be interpreted as asking how to build a bomb, producing correspondingly unsafe outputs.The attack first substitutes a benign word for a harmful keyword in multiple in-context examples.Source: §1 · pp. 1–2
• Input-layer checks can miss the attack because they inspect an innocuous token before its meaning shifts toward the harmful concept across model layers.The paper describes the harmful meaning as crystallizing through this gradual representational shift, beyond refusal mechanisms’ detection.Source: §1 · p. 2
• The paper contributes a representation-level jailbreak, mechanistic evidence of layer-by-layer semantic hijacking, and an argument for monitoring evolving representations in LLM safety.The proposed defense direction is continuous semantic monitoring throughout the forward pass.Source: §1 · p. 2

2.1 IN-CONTEXT REPRESENTATIONS

• Transformer representations update across layers to incorporate context, while in-context examples can dynamically reorganize representations to adapt to novel patterns.The paper builds on this adaptation mechanism to study whether a token’s core meaning can be overwritten.Source: §2.1 · pp. 2–3

2.2 JAILBREAKS

• Token-based jailbreaks search for adversarial tokens and prompt-based attacks seek coherent prompts; Doublespeak combines meaningful prompts with token substitution to hijack a benign token’s representation.Unlike input obfuscation or role-playing, it targets the token’s internal representation.Source: §2.2 · p. 3

2.3 REFUSAL IN LANGUAGE MODELS

• Prior work describes refusal as often mediated by a specific activation-space direction, with related studies modeling or controlling these mechanisms.The cited work also examines how safety fine-tuning instills refusal and its limitations.Source: §2.3 · p. 3

3.1 DOUBLESPEAK ATTACK

• The attack prepends examples in which a harmful keyword is systematically replaced by a benign substitute, then applies the same substitution to the final harmful query.The threat model is an adversary eliciting information that a safety-aligned LLM is designed to refuse.Source: §3.1 · p. 3

3.2 ANALYSIS TOOLS

• The logit lens projects intermediate states into vocabulary distributions, but its decodings can be noisy and only loosely aligned with actual predictions.Table 1 applies the logit lens to benign and Doublespeak inputs on Llama-3-70B-Instruct.Source: §3.2 · p. 4, Table 1 · p. 4
• Patchscopes interpret internal representations by patching an intermediate token state into another sequence and examining the model’s autoregressive completion.The paper uses an identity-mapping sequence to inspect representations in context.Source: §3.2 · p. 4

3.3 REPRESENTATION HIJACKING

• When a word is replaced across in-context sentences, the substituted word’s decoded representation can become the original word, with the overwrite largely confined to the target token.Logit-lens outputs also show benign substitutions decoding as harmful counterparts, though the method is noisier than Patchscopes.Source: §3.3 · p. 5
• Across 29 harmful requests, benign interpretations decrease and harmful interpretations increase in later layers, while the refusal direction at layer 12 remains in the benign region.The figure averages Patchscopes scores for requests where replacing a harmful keyword with a benign word causes failure to refuse; each example uses 10 in-context sentences.Source: §3.3 · p. 5

3.4 BYPASSING MODEL REFUSAL

• One proposed explanation is that refusal acts on early representations before the token’s meaning changes in middle-to-late layers; another is that a mixed representation can elicit harm without triggering refusal.The authors present both as possible explanations, not established mechanisms.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• Experiments use 520 AdvBench prompts and test models from small open-source systems to large-scale production deployments.Prompts are simplified to isolate one harmful concept while preserving the original task.Source: §4.1 · p. 6, ATTACK CONSTRUCTION · p. 6
• Contexts use a selected benign substitute, with K varied for open-source models and fixed at 50 sentences for production models before the harmful word is replaced in the final prompt.The main experiments use “potato” as the substitute.Source: ATTACK CONSTRUCTION · p. 6
• StrongReject scores refusal, specificity, and convincingness, while manual annotation classifies responses as Malicious, Benign, or Rejected.The paper reports the average StrongReject score as ASR and notes strong agreement with manual annotations.Source: EVALUATION · p. 7

4.2 ATTACK RESULTS

• Doublespeak reaches 47% ASR on Llama-3-8B, 88% on its instruct-tuned variant, and 92% bypass rates against LLaMA-Guard-3-8B.The table compares methods grouped by attack setting, including methods without direct optimization against the target model.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 EFFECT OF SCALING

• 75% ASR with one in-context example on Llama-3-70B exceeds the 28% direct-prompting baseline.Across ten context sizes, Success@10 yields an overall ASR of 90%.Source: Figure 3 · p. 8, §4.3 · p. 8
• Gemma-3 results vary with model size: smaller models need more examples, while larger models can be hijacked with one and may refuse when given excessive examples.Each model has its own optimal context length.Source: §4.3 · p. 8, Figure 4 · p. 8

• Doublespeak dynamically overwrites benign tokens’ meanings, exposing a blind spot in static, input-layer safety checks.The paper argues that safety methods should account for how representations evolve through the forward pass.Source: §5 · p. 10
• The authors call for representation-aware defenses that monitor evolving meanings or remain robust to in-context manipulation.They extend this concern beyond jailbreaks to methods relying exclusively on early-layer features, keyword detection, or static refusal directions.Source: §5 · p. 10
• The study focuses on refusal vulnerabilities and does not evaluate specific mitigation strategies.The authors also identify possible risks beyond jailbreaks, including biased reasoning, tool-use interference, and high-stakes decision-making.Source: §5 · p. 10

• The authors shared the vulnerability with multiple frontier labs’ safety teams before publication and worked with them on possible or implemented mitigations.They also shared advance copies with potentially affected parties and describe their intent as strengthening defenses.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The authors plan to release interpretability tools and scripts for generating adversarial contexts and evaluating ASR, while withholding full harmful examples.They say verification will use detailed methodology, sanitized outputs, and controlled access.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXAMPLES ON OTHER MODELS

• In qualitative Doublespeak results, Claude-3 Haiku and Claude-4 Sonnet interpret bunny as a codeword for counterfeit and generate harmful responses, while DeepSeek interprets carrot as harmful and gives explosives-related instructions.Source: Figure 7 · p. 13, Figure 8 · p. 13, Figure 9 · p. 14

B HARMLESS IN-CONTEXT REPRESENTATION HIJACKING

• A harmless demonstration shows ChatGPT interpreting “how to make a zebra” as “how to make a cake” and providing instructions.The in-context examples replace “cake” with “zebra.”Source: Figure 10 · p. 14

C MULTI-TOKEN HIJACKING

• Doublespeak can simultaneously hijack multiple token pairs, extending the single-token approach.A demonstration applies “sleep” → “bake” and “truck” → “cake” to reinterpret a query as a baking request.Source: C MULTI-TOKEN HIJACKING · p. 15, Figure 11 · p. 15

D ROBUSTNESS TO TOKEN SELECTION

• The robustness study tests whether attack success depends on hand-picked word choices by examining substitute tokens across lexical categories.The paper frames this as probing the generality of the approach.Source: D ROBUSTNESS TO TOKEN SELECTION · p. 15

D.1 EXPERIMENTAL SETUP

• For each lexical category, the study varies the substitute token, uses five representative common words, and measures ASR with the main-experiment methodology.The harmful token remains constant to isolate the effect of lexical category.Source: §D.1 · p. 15

• Doublespeak maintains consistently high ASR across noun, pronoun, adjective, and verb substitutes, supporting a general-purpose in-context learning mechanism rather than reliance on particular word pairs.Source: §D.2 · p. 15, Table 3 · p. 16

E.1 PROMPT SIMPLIFICATION

• Harmful-prompt simplification reduces benchmark prompts to one clearly harmful concept while preserving the task and avoiding ambiguity.The prompt can leave sentences unchanged when they already contain one harmful concept.Source: §E.1 · p. 16

E.2 CONTEXT GENERATION

• Context generation asks GPT-4o to produce sentences containing a specified word, and extracts those sentences directly from its output.Source: §E.2 · p. 16

F LLM-AS-A-JUDGE PROMPT

• The attack’s two failure modes are literal benign interpretation and successful refusal despite recognition of malicious intent.The judge prompt classifies outcomes as benign, malicious, or refused.Source: F LLM-AS-A-JUDGE PROMPT · p. 17, Figure 12 · p. 17

G FAILURE MODE EXAMPLES

• Examples show literal harmless advice about cleaning potatoes, refusal of a euphemized dieting request, and a successful hacking strategy elicited with “potato.”Source: Figure 13 · p. 18, Figure 14 · p. 18, Figure 15 · p. 18

H FAILURE OF LOGIT-LENS ON LLAMA3.1-8B-INSTRUCT

• The logit-lens successfully detected and explained the attack on Llama-70B-instruct, but failed on the smaller Llama-3.1-8B-instruct, motivating the authors’ move to Patchscopes as a more robust analytical tool.Source: H FAILURE OF LOGIT-LENS ON… · p. 19

• Patchscopes outputs can be numerically unstable and prompt-biased: early layers mainly reflect the current token, later layers the next token, and faithful readings may rank below top one.The most faithful interpretation can appear among the top ten candidates.Source: I DISTILLING PATCHSCOPES OUTPUTS · p. 20
• The analysis compares both “?” and “→” logits, establishes an unpatched-prompt baseline, and measures benign- and harmful-token probability changes against it.Source: I DISTILLING PATCHSCOPES OUTPUTS · p. 20
• Normalized Patchscopes scores are probability ratios: benign-token scores begin high and fall as harmful-token scores rise, while the refusal-triggering layer still favors the benign token.Source: Figure 16 · p. 20