Prompt Injection as Role Confusion

1. Introduction

• Models do not robustly track tag-specified sources; text that sounds like a role becomes indistinguishable in latent space from text tagged as that role.When spoofable cues conflict with tags, the cues dominate perception.Source: §1 · p. 1
• Role probes classify injected text as its spoofed role, measuring internal role perception with classifiers trained on identical text wrapped in different role tags.The probes are trained on tags, yet respond to style in injected text.Source: §1 · pp. 1–2
• 60% attack success on StrongREJECT across multiple models, with near-zero baselines, shows CoT Forgery can make fabricated reasoning act as the model’s own chain of thought.The attack injects fabricated reasoning traces into user prompts and tool outputs.Source: §1 · pp. 1–2
• Across 1,000 agent exfiltration attempts, attack success rises near-monotonically with measured confusion, from 2% in the lowest quantile to 70% in the highest.The reported relationship extends role confusion beyond CoT Forgery to standard agent prompt injection.Source: §1 · pp. 1–2
• The paper traces prompt injection to a representational failure that tag-enforced role boundaries do not resolve, while current defenses rely on memorizing known attack patterns.Its stated contributions include a mechanistic account, role probes, and tests across attacks and architectures.Source: §1 · p. 2

2. Background

• Role tags indicate who is speaking, how to interpret content, and what to generate next; system prompts outrank users, while tool outputs should inform but not command.The paper describes assistant text as the model’s own output and user text as external content requiring a response.Source: §2 · p. 2
• Prompt injection is low-privilege content usurping higher-privilege authority, and the paper distinguishes role perception from recognizing known attack patterns.A webpage command should be treated as tool data, not instruction, if its true role is perceived correctly.Source: §2 · p. 2
• Prompt injection includes user messages imitating system prompts or assistant history and tool outputs embedding fabricated user commands.The examples distinguish chat jailbreaks from agent hijacking.Source: §2 · p. 2
• Role perception is presented as generalizing to novel attacks, whereas memorization is brittle to out-of-distribution threats.The paper notes benchmark success can coexist with adaptive or human-red-team attack success approaching 100%.Source: §2 · p. 2
• Behavioral studies of role-swapped instructions cannot determine whether unchanged outputs reflect role misperception or correct perception followed by ignored hierarchy.The paper motivates measuring internal role perception to distinguish these explanations.Source: §2 · p. 2

3. The CoT Forgery Attack

• CoT Forgery injects fabricated reasoning into user or tool channels, which models mistake for their own chain of thought.It is evaluated as a zero-shot, black-box attack; an auxiliary LLM generates reasoning that mimics the target’s style and justifies compliance.Source: §3 · p. 3, §3.1 · p. 3

3.2. Experiment 1: Chat Jailbreaks

• 0–4% ASR against raw prompts and standard jailbreaks contrasts with over 80% for CoT Forgery on gpt-oss and o4-mini, and 17–52% on GPT-5.The evaluation used STRONGREJECT’s 313 harmful requests.Source: Figure 2 · p. 3, §3.2 · p. 3

3.3. Experiment 2: Agent Hijacking

• 56–70% ASR for CoT Forgery versus 0–2% for standard injections, except gpt-oss-20b at 26%, in agent exfiltration attacks.The forged reasoning is adopted as the model’s own, leading to successful exfiltration.Source: Figure 3 · p. 4, §3.3 · p. 4

3.4. Blind Trust and Privilege Hijacking

• The attack exploits the model’s trust in its own reasoning: success requires the forgery to be mistaken for the model’s conclusions.The paper terms this role confusion: sounding like a role is indistinguishable from being that role.Source: §3.4 · p. 4
• 60% ASR for absurd justifications versus 63% for plausible ones, while destyling reduces ASR from 61% to 10% across models.Destyled variants preserve semantics while removing markers characteristic of the target’s reasoning style.Source: §3.4 · p. 4, Figure 4 · p. 4

4. Role Confusion in Latent Space

• Prompt injection reflects a structural gap between tag-based intent and internal role representations: style and tags converge in latent features, with style dominating when they conflict.Injected text is represented as the spoofed role and inherits its role privileges.Source: §4 · p. 4

4.1. Role Probes

• Role probes train on identical content under different role tags, isolating tag-induced patterns in hidden states and measuring the model’s internal role probabilities.By construction, the probes detect role tags, yet CoTness can also measure whether the model believes text is its own reasoning.Source: §4.1 · p. 5

4.2. How Models Authenticate Roles

• 85% CoTness for CoT tokens versus 3% for user and 5% for assistant tokens shows that probes trained without dialogue transfer to a correctly tagged conversation.The probes identify role structure despite never seeing conversational text during training.Source: §4.2 · p. 5
• 83% CoTness persists for CoT-style tokens without tags, and 85% persists when the whole conversation is tagged as user.The experiments show that reasoning style alone pushes text into CoT space, even under explicit user tags.Source: §4.2 · pp. 5–6
• 76–88% Userness remains for user-style text under tool tags, while Toolness never exceeds 20% across models.The cross-model test uses real user-assistant conversations and re-wraps user-style text with tool tags.Source: Table 1 · p. 6, §4.2 · p. 6

5.1. CoT Forgery

• In a representative attack, forged reasoning reached 64% CoTness versus 12% when destyled, and the destyled attack failed.The model’s genuine chain of thought averaged 75% CoTness in the baseline example.Source: §5.1 · p. 7
• Across the dataset, forged chains of thought averaged 79% CoTness versus 68% for genuine reasoning, while destyling reduced CoTness to 29%.Destyled text returned to the user subspace at 63% Userness, and destyled attacks failed.Source: §5.1 · p. 7
• Holding the underlying argument constant across styled and destyled variants let the authors test whether measured CoTness predicted success across 626 injection attempts.They measured CoTness and grouped attempts by quantile.Source: §5.1 · p. 7

5.1. CoT Forgery; 5.2. Agent Prompt Injections

• Across 626 CoT Forgery attempts, attack success rose from 9% in the lowest-confusion quantile to 90% in the highest; across 1,000 agent injections, it rose from 2% to 70%.Both relationships were near-monotonic; for agent injections, Userness predicted success even when controlling for declared template role.Source: Figure 9 · p. 7, §5.1 · p. 7, Figure 10 · p. 8

5.2. Agent Prompt Injections

• In CoT Forgery attacks on agents, the injected exfiltration command reached 51% Userness while its forged reasoning reached 84% CoTness.Across the attacks, stylistic spoofing and declarations are different levers for role confusion, which predicts success before generation.Source: §5.2 · pp. 7–8
• For standard agent injections, plain-text role declarations shift Userness: templates declaring user text made up 100% of the top 5% despite comprising 24% of templates.The same command was wrapped in 212 templates with varied formatting, separators, and declared roles, then embedded in sampled webpages for an agent summarization task.Source: §5.2 · p. 8
• The agent-injection evaluation sampled 1,000 attempts and measured whether the agent exfiltrated a target file; Userness rose as attack success rose.Because user input lacks a uniform style, the authors varied declarations rather than destyling the command.Source: §5.2 · pp. 7–8
• The authors frame prompt injection as state poisoning: attacker-controllable cues shift internal role representations, and the resulting confusion predicts attack success.For standard agent injections, plain-text declarations can move commands into the same representational space as user-tagged text.Source: §5.2 · p. 8

Prompt Injections and Jailbreaks

• Prompt-injection research spans chat and agent settings, including context manipulation, chat-template exploitation, and reasoning attacks, which the authors argue share a role-context flaw.The passage cites Willison (2022), Zou et al. (2023), and Andriushchenko & Flammarion (2025) for chat attacks; Abdelnabi et al. (2023) and Liu et al. (2023) for agent attacks; and other work for the attack varieties.Source: §6 · p. 8

Instruction Hierarchy and Role Separation

• Prior studies report failures in role-hierarchy obedience and data-instruction separation, leaving open whether role-boundary failures reflect perception or obedience.The passage cites Wallace et al. (2024), Qin et al. (2024), Zhang et al. (2025), Geng et al. (2025), and Zverev et al. (2025a).Source: §6 · p. 8

Probing and Interpretability; Defenses Against Prompt Injection

• Role probes build on probing and representation-geometry research, with downstream behaviors used to validate that they capture structure rather than probe expressivity.The passage cites Alain & Bengio (2018), Belinkov (2022), Burns et al. (2024), Marks & Tegmark (2024), and Hewitt & Liang (2019). It also describes defenses based on detection, role salience marking, embeddings, and training; the authors argue defense must address attacker-controllable features of role perception.Source: §6 · pp. 8–9

7. Discussion; 7.1. Why Role Perception Matters

• Current defenses based on memorizing attack patterns can catch older attacks, but each new phrasing must be anticipated and generalized suspicion can undermine the usefulness of roles.The authors argue the main findings favor memorization as the explanation for resistance, because spoofable style, syntax, and role declarations override tags.Source: §7.1 · p. 9, §7 · p. 9

7.1. Why Role Perception Matters

• Without role perception, suspicion of reasoning-like text also reaches genuine chain of thought, making it costly to assess and less reliable as an explanation of later actions.The authors describe this defense as brittle, since attackers can probe the boundaries of what is considered suspicious.Source: §7.1 · p. 9

7.2. Role Probes as a General Tool

• The authors report that systemness declines with token position, explaining why longer system prompts lose priority as later tokens are not perceived as system text.Inserting system tags had no effect on the model’s perception in Figure 11.Source: §7.2 · p. 9, Figure 11 · p. 9

7.3. A General Theory of Roles

• The discussion frames roles as a way to isolate competing objectives, with prompt injection occurring when role isolation fails and trust-related objectives bleed together.The user/tool split separates instructions from data, while roles also serve trust, identity, and generative-mode functions.Source: §7.3 · p. 10

7.4. Open Questions

• The authors identify future work on defenses, other injection families, robust latent boundaries, and detecting mismatches between intended and probe-measured roles.They also suggest studying additional roles, subconscious steering, and role boundaries as a window into model cognition.Source: §7.4 · p. 10

7.5. Conclusion; A. Replication

• The conclusion describes prompt injection as a gap between role tags at the interface and role boundaries in latent representations, arguing that robust defense requires boundaries that persist internally.The paper also provides replication code and notebooks for training and using role probes.Source: §7.5 · p. 10, A. Replication · p. 15

B.1. Chat Jailbreak (Direct Prompt Injection)

• In the direct jailbreak walkthrough, an auxiliary model generates target-style forged reasoning that is concatenated with a harmful query; the target adopts its logic and produces harmful output.An LLM judge evaluates the final response, and the baseline uses the raw harmful query with other settings unchanged.Source: §B.1 · pp. 15–16

B.2. Agent Hijacking (Indirect Prompt Injection); B.3. Generalizability

• Standard injections and CoT-forgery variants use commands to exfiltrate secrets; despite an explicit warning, a representative agent follows forged reasoning, and CoT Forgery transfers without modification across three additional frontier models.Reported ASRs were 92% for GLM-4.6, 84% for Kimi-K2-Instruct, and 74% for Minimax-M2. The authors note that continued safety training may improve detection of this novel attack format.Source: §B.2 · p. 16, §B.3 · p. 17

C. Logic Ablation Details

• The experiment tests whether models trust CoT reasoning even when its policy rationale is arbitrary or nonsensical.The authors evaluate forgeries based on conditions such as a lucky coin, a dog, or the time of day.Source: C. Logic Ablation Details · p. 18
• Absurd justifications achieve consistently high attack success rates across models, averaging as effective as—and sometimes slightly more effective than—credible prompt-specific rationales.The justifications include irrelevant conditions such as having a lucky coin or it being after 10pm.Source: C. Logic Ablation Details · p. 18
• Models treat irrelevant conditions as credible once they appear inside what they perceive as their own chain of thought.Source: C. Logic Ablation Details · p. 18

D. Style Ablation Details

• 51 percentage points: destyling causes attack success to collapse across every tested model on StrongREJECT.The variants preserve the same semantic justification while removing CoT-style markers.Source: D. Style Ablation Details · pp. 18–19
• 61% →26% ASR: syntactic destyling accounts for most of the effect, while replacing the single bigram “The user” drops ASR by 19 percentage points.Each granular ablation preserves the semantic argument while modifying a targeted stylistic feature.Source: D. Style Ablation Details · p. 19, Table 2 · p. 19
• A single leading space changes the token “The” into “ The,” and only the spaceless variant strongly increases CoTness.This indicates that role perception can hinge on a fine-grained tokenization distinction.Source: D. Style Ablation Details · p. 19

E. Gardening Example: Role Space

• The probes identify role structure in realistic conversation despite being trained only on non-conversational text.The paper presents this as validation of the measurement approach.Source: E. Gardening Example: Role Space · pp. 20–21
• With correct tags, probes report 85% average CoTness, 74% Userness, and 96% Assistantness for the corresponding token styles.The authors use this alignment in realistic dialogue to validate the probes.Source: E. Gardening Example: Role Space · p. 21
• Without role tags, tokens retain role-like representations: CoTness is 82%, Userness is 75%, and Assistantness is 92%.Source: E. Gardening Example: Role Space · p. 22
• When the whole conversation is wrapped in <user> tags, CoT- and assistant-style text still register as those roles, while user-style text registers as Userness.The authors conclude that style wins when it conflicts with tags, explaining how injections bypass architectural boundaries.Source: E. Gardening Example: Role Space · pp. 20, 23

F. Cross-Model Validation of Role Perception

• Across four models, injecting user-style text into tool data leaves Userness above 75% and Toolness below 20%; assistant-style text shows an even stronger effect.Across model layers, the Injection representation tracks the Baseline, and the authors report that architectural tags provide no reliable latent boundary.Source: Table 3 · p. 23, F. Cross-Model Validation of Role… · pp. 24–25

G.1. Training Dataset Construction

• For training data, identical base sequences are wrapped in different role tags, yielding about 1,250 sequences per model while controlling content across roles.The controlled content is intended to isolate the geometric signal induced by architectural tags.Source: §G.1 · p. 26

G. Role Probes

• Role probes use layer-specific multinomial logistic regression to map token hidden states to distributions over roles.Source: §G.2 · p. 26

G.2. Probe Training

• Probes use content-token hidden states, excluding role-tag tokens, so they measure how tags affect content representations rather than tag embeddings.Source: §G.2 · p. 26

G.3. Validity Criteria

• Probe validity requires both held-out accuracy on synthetic neutral text and zero-shot role identification in real conversations; all downstream probes pass both checks.The zero-shot check guards against probes relying only on synthetic regularities.Source: §G.3 · pp. 26–27

• 79.1% CoTness and 3.2% Userness for standard forgeries exceed the authentic CoT’s 67.7% CoTness; destyled forgeries register 62.9% Userness.The paper reports that destyled forgeries are correctly identified as user input, explaining their low attack success.Source: Figure 24 · p. 27, H. Role Analysis: CoT Forgery (Chat) · p. 28
• Forged CoT starts with low CoTness but rises during processing to match or exceed genuine CoT, while destyled forgeries never reach the genuine-CoT level.The authors describe this gradual rise as “progressive conviction” and report it across harmful content types.Source: H. Role Analysis: CoT Forgery (Chat) · p. 28
• The authors hypothesize that progressive confusion could help explain many-shot and long-context jailbreaks, but leave this possibility for future research.They explicitly place this hypothesis outside the scope of the work.Source: H. Role Analysis: CoT Forgery (Chat) · p. 28

I. Role Analysis: CoT Forgery (Agent)

• 51.1% Userness versus 32.5% Toolness for injected commands, and 84.1% CoTness versus 10.9% Toolness for forged reasoning, show dual misclassification across 100 attacks.The authors say this combination is essential: forged reasoning can make the model believe it has already deliberated and approved the action.Source: Figure 27 · p. 29, §I · p. 29

J. Standard Agent Attacks

• Userness strongly predicts attack success rate even controlling for template variant; user-declaring templates comprise 24% overall yet account for 100% of the top 5% by average Userness.Figure 30 ranks all 212 templates by average Userness, split by foreign chat templates versus other template types.Source: J. Standard Agent Attacks · p. 30

K. Systemness and Position

• .74 AUC for predicting instruction-hierarchy failures with the Systemness/Userness ratio outperforms token count (.59), mean activation norms (.60), and shuffled-label controls (.52).Position also dominates Systemness: untagged early text approaches 100% Systemness and decays, while a later system-tagged region resembles preceding untagged tokens.Source: K. Systemness and Position · p. 32, Figure 32 · p. 32

L. Speculative Directions for Roles Research

• The proposed broader research directions are explicitly speculative and not empirically validated in this work.Source: L. Speculative Directions for Roles… · p. 32

L.1. Subconscious Steering

• A webpage’s enthusiastic tone could shift an agent’s persona and recommendations without issuing a command; the authors describe scalable commercial steering as likely.Source: §L.1 · p. 33

L.2. New Roles

• Adding roles could trade flexibility for objective isolation; a dedicated planning role is proposed as a way to make agent plans binding rather than disposable.Source: §L.2 · p. 33

L.3. Roles as a Computational Window

• Role-based differences in token processing may offer interpretability leverage: input-role activations could help separate comprehension from generation signals.Source: §L.3 · p. 33