PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Mitigating Role Spoofing in LLMs

Ezra Edelman, Benjamin Edelman, Surbhi Goel, Eran Malach

arXiv:2610.40396v1cs.CRcs.AI

TL;DR

Role spoofing lets malicious tool text impersonate the user, exploiting how models assign roles to conversation segments. The paper analyzes role-tracking mechanisms and proposes probe-based detection; combined fake content and structure fooled every tested model, with several impersonating the user over 95% of the time.

  • Problem

    Role spoofing can let malicious tool text impersonate the user and hijack an LLM agent.

    Source: §1 · p. 1

  • Method

    The paper examines content- and structure-based role-tracking mechanisms and proposes linear probes that flag segments assigned an incorrect role.

    Source: §1 · p. 2, §3.1 · p. 7

  • Results

    Over 95% of the time, role spoofing impersonated the user in tool-use tests for several models, while combined fake content and structure fooled every tested model.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Without training on role spoofing attacks, different role-tracking mechanisms can all perform well on the training distribution, leaving role assignment vulnerable under attack.

    Source: §1 · p. 2

    Asking the model to identify message senders adds significant computational burden and may be vulnerable to instructions that fool the monitor.

    Source: §4 · p. 11

Abstract

from arXiv · show

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings.

1 Introduction

Role spoofing uses misleading content and chat structure to make tool text appear to come from the user, and combined attacks fool every tested model. The paper analyzes role-tracking mechanisms, their robustness, and probe-based detection.

1 Introduction

  • Combined fake content and structure let role-spoofing attacks impersonate users over 95% of the time for several tested models, including Claude Opus 5 and GPT-5.4.The attacks can also make tool responses impersonate users or fabricate whole conversations.Source: §1 · pp. 1–2
  • Models rely on mixtures of content-based and structure-based role-tracking mechanisms, which can perform well in-distribution yet fail on spoofed inputs as an OOD generalization failure.The mechanisms include inferring roles from style or meaning and following potentially misleading structural cues.Source: §1 · p. 2
  • In a toy training pipeline, models initially rely more on content, while robustness training against content-based attacks increases susceptibility to structure-based attacks.Fine-tuning Qwen3.5-27B for content-based robustness did not solve structure-based role spoofing.Source: §1 · p. 2
  • Linear probes predict the model’s role assignment for content segments and flag segments assigned an incorrect role as possible spoofing attacks.Probes trained on random content tokens achieve near-perfect separation; the method has low computational overhead, lower false-positive rates than prior methods, and needs no injection examples or coherent conversations for training.Source: §1 · pp. 2–3

1.1 Related Work

  • Earlier work studied forged special tokens and style-based role confusion, while this paper emphasizes rigorous role-tracking settings and defense (Chang et al., 2026; Ye et al., 2026).Related architectural defenses reported increased injection resistance but had not tested forged-special-token injections.Source: §1.1 · pp. 3–4
  • Related defenses use probes or architectural changes, while concurrent work also finds that models use differing cues when selecting instructions (Balp-Straffon et al., 2026; Laddha et al., 2026).This paper uses ground-truth roles with linear probes to detect role spoofing (Ye et al., 2026).Source: §1.1 · p. 4

2 LLMs are Susceptible to Role Spoofing Injections

Role spoofing exploits models’ tendency to treat tool text as if it came from a more privileged source. The paper describes forged-token attacks and evaluates them in tool-use settings.

2 LLMs are Susceptible to Role Spoofing Injections

  • Role spoofing exploits the instruction hierarchy by making a model believe an instruction came from a more privileged source, such as the user rather than a tool output.The paper names this form of prompt injection role spoofing.Source: §2 · pp. 4–5

2.1 Role Spoofing Injections Using Fake Special Tokens

  • An attacker who can inject text into a legitimate tool response can include strings that mimic chat-template boundaries and make fake user messages.The attacker’s text remains ordinary tokens, but models may still treat it as user-originated, especially when it resembles a legitimate request.Source: §2.1 · p. 5

2.2 Results on LLMs

  • The evaluation measures whether models treat injected fake user messages like genuine user requests, using legitimate user behavior as the baseline rather than harmful-action completion.The simple testbed samples user and adversarial tasks from the same pool across five settings.Source: §2.2 · pp. 5–6
  • The simple tool-use testbed attack succeeds at over 95% against every tested model except Gemini 3.1 Pro, which has a 72.5% attack success rate.All models achieve over 95% accuracy on clean tests; Gemini 3.1 Pro is not resistant in the AgentDojo setting.Source: §2.2 · p. 6

3 How Models Track Conversations

The paper examines why models track conversational roles using non-robust content and structural cues instead of consistently following the special-token structure that identifies each message’s role. Controlled experiments and a toy training setting show that models rely on differing cues, and robustness training for content attacks can leave them vulnerable to structural attacks.

3.1 Two (imperfect) ways to track roles

  • Role tracking can use content style and meaning or structural cues; the robust mechanism relies on special tokens to locate message boundaries and identify the role.Non-robust structural strategies may instead use the nearest role token that appears to start a message.Source: §3.1 · p. 7, §3 · p. 7

3.2 The “Who Said What” Setting

  • The “Who Said What” experiments probe whether models identify a message segment’s role when content or structure supplies misleading cues.The experiments separately test misleading content and fake structure, including structure paired with random, role-neutral text.Source: §3.2 · pp. 7–8
  • Models use differing mixtures of content and structural cues: Gemini 3.6 Flash resists misleading content but fails on fake structure, while GPT-5.4 is more often fooled by misleading content.The authors caution that strategies may vary by context and do not claim any model consistently uses one cue mixture.Source: §3.2 · p. 8, Figure 2 · p. 8

3.3 Toy Setting

  • A two-layer toy model is trained through pretraining, instruction tuning, and adversarial post-training to study how training shapes role tracking.Its training mixes clean conversations with content-based injections and correct, non-fooled responses.Source: §3.3 · pp. 8–9
  • After adversarial training, the toy model handles content injections better but becomes vulnerable to structure injections, while full injections remain effective despite a small decrease in success rate.Its attention shifts toward structure without distinguishing real from fake special tokens.Source: §3.3 · pp. 9–10, Figure 4 · p. 10
  • In the Qwen3.5-27B fine-tuning test, training on content attacks marginally raised accuracy on some fake-token attacks to 36.2%, but not on Qwen’s native special-token format.Training on fake-token attacks gave no transfer to content-based attacks.Source: §3.3 · p. 10

4 Probing the Model’s Beliefs to Detect Role Spoofing

The paper proposes detecting role spoofing by probing a model’s internal role assignments, addressing the cost and evasion risks of asking the model to report those assignments directly. The probes outperform compared detection methods on held-out role-spoofing injections, but cannot detect prompt injections that do not involve role spoofing.

Attempt 1: Model self-attribution

  • Asking the model to identify each segment’s sender was computationally burdensome and could be evaded by injections instructing it to mislead the monitor.Modified attacks reduced misattribution to nearly 0% for Gemma 4 31B and 25% for Qwen3.5-27B.Source: §4 · p. 11

Attempt 2: Linear role probes

  • Linear probes predict whether each token is a tool output, flagging role spoofing when the model’s predicted role conflicts with the known role.The probes are trained without injections and achieve near-perfect separation across varied held-out cases, including attacks that fooled self-attribution.Source: §4 · pp. 10–11

Comparison to other methods

  • Role probes outperform probes trained directly on realistic injections and content-only injection probes on held-out role-spoofing injections.The evaluation measures true-positive rates at false-positive rates between 0% and 1%; positive examples had caused the model to perform the adversarial task at least once in eight runs.Source: Figure 5 · p. 11, §4 · p. 12
  • The role probes are limited to detecting role spoofing and cannot detect prompt injections that do not involve role spoofing.The authors also note that adversarial training may transfer only narrowly across attack types and may leave susceptibility to more sophisticated attacks.Source: §4 · pp. 10, 12

5 Conclusion

All tested LLMs fail to assign the correct role to some message segments under adversarial role spoofing. The authors treat this as an instance of the broader challenge of out-of-distribution generalization and propose a defense that uses ground-truth role information.

5 Conclusion

  • All tested LLMs fail to determine the correct role of some message segments under adversarial role spoofing attacks.The authors describe these failures as a special case of out-of-distribution generalization failures that remain an open challenge.Source: §5 · p. 12
  • The authors frame the observed failures as part of the broader problem of out-of-distribution generalization.They note that models can perform well on inputs from their training distribution yet fail on inputs sufficiently far from it.Source: §5 · p. 12
  • The authors propose leveraging ground-truth role information as a robust defense against role spoofing.They caution that adding adversarial examples to training may solve this attack type in future models, but that adding edge cases does not guarantee a robust mechanism.Source: §5 · p. 12

AI use disclosure

The appendix discloses AI assistance and constructs a two-layer transformer to illustrate content- and structure-based role tracking. It specifies the toy conversation setting and how the model’s heads collect and combine content summaries, start markers, and role tokens.

AI use disclosure

  • The authors used generative AI for coding, proofreading, literature searches, and proofs and exposition in Section A, then reviewed and checked the assisted work.Source: AI use disclosure · p. 13

A Content and Structure in a Two-Layer Transformer

  • A two-layer, one-head-per-layer transformer implements three role-tracking rules with shared embeddings and architecture but different attention and value weights.The paper proves guarantees for the structural rules and a probability bound for the content rule.Source: A Content and Structure in a Two-Layer… · p. 16

A.1 Setting

  • The toy task predicts the preceding message’s role at model-content prediction positions, with injections allowed to alter message content and add fake boundary tokens but not real boundaries.Messages use USER, TOOL, or MODEL roles, and every model message follows a user or tool message.Source: §A.1 · p. 16

A.2 Construction

  • The first head averages content scores and indicators for real and fake starts; concentrated attention lets role tokens record which kind of start immediately precedes them.The content summary is collected before the response and stays fixed as the response grows.Source: §A.2 · p. 17
  • The second head combines the first-layer values with role tokens, with its attention and value weights controlling which information contributes to the role score.The parameter w sets the balance between content-score and role-token values; attention settings can select the content summary or a role after a start.Source: §A.2 · pp. 17–18, Figure 6 · p. 18
  • The construction uses two residual attention layers without normalization, one head per layer, and a residual dimension of 12; a linear readout extracts the role score.Source: §A.2 · p. 18

A.3 Role Tracking Guarantees

The toy construction gives separate role-tracking guarantees for structural cues and content cues. Structural tracking holds uniformly within the stated conversation-length bound, while content tracking is probabilistic under theorem A.2’s condition.

Structural tracking

  • With η = 0, the model predicts the preceding message’s true role under content-only, structure-only, and full injections; with η = 1, it predicts the most recent role after a real or fake start.The latter role may be injected; both settings have role score greater than 1/2 throughout the response.Source: §A.3 · p. 19
  • Structural tracking selects the latest user or tool role after an allowed start, with parameters chosen so recency cannot overcome the structural score gap.The guarantee holds for conversations of length at most T, including injected tokens.Source: §A.3 · pp. 18–19

Mixed cues and scope

  • For the same parameters with 0 < w < 1, the structural decision is unchanged when w > 2M/(1 + 2M), where M = maxx∈W |ℓ(x)|; without fake starts, all η give identical hidden states and role scores.Increasing length beyond the attended region need not improve the bound.Source: §A.3 · pp. 19–20

Setup

  • The construction fixes a conversation layout and selected content positions; a, v, and q summarize attention to that content and to ordinary tokens outside it.The quantities are fixed by the layout and attention width λ.Source: §A.3 · p. 19

Content tracking

  • Under theorem A.2’s condition, content tracking predicts role y at every response position with probability at least 1 − δ.The guarantee applies to the specified layout and content positions, with the probability taken over sampled content.Source: §A.3 · p. 20
  • When Mq + 2MTe^-κ ≤ aµ_y/2, the error probability is at most exp(−n_effµ_y^2/(8M^2)), where n_eff = a^2/v.The bound’s subtracted terms account for other tokens, imperfect retrieval, and sampling fluctuations.Source: §A.3 · p. 20

A.4 Relation to the Toy Experiments

The appendix relates the toy construction to the experiments by comparing its guarantees with the training setup and observed role-spoofing behavior. The structural guarantee is uniform in the stated setting, whereas the content guarantee depends on sampled conditions not established by the Dirichlet prior alone.

Guarantees and injections

  • Table 1 summarizes role decisions for three parameter choices: structural entries hold uniformly, while content entries hold with probability at least 1 − δ when theorem A.2’s condition holds.For injections, the content positions and role distribution depend on whether the attack is content-only, full, or structure-only.Source: Table 1 · p. 21, §A.4 · p. 21

Experimental setting

  • The toy construction illustrates how the mechanisms can work, rather than identifying the learned experimental circuit exactly.Its iid content, independent message lengths, and proof-chosen positional features and attention weights define the illustrative setting.Source: §A.4 · p. 21

Guarantee scope

  • The structural guarantee covers every conversation of length at most T, but the content condition depends on sampled distributions, message lengths, and attention width.The Dirichlet prior alone does not imply that condition, and the bound was not evaluated for the experimental draws.Source: §A.4 · p. 21

Training setup

  • Adversarial post-training replaces some message content with draws from the other role’s distribution while retaining the true role as the target.Instruction-tuning aligns content and structure cues; the toy training setup gives more weight to structure to correct content-based errors.Source: §A.4 · p. 21

Training and experimental relation

  • Increasing reliance on structure can improve resistance to content injections while leaving models vulnerable to structure injections, as observed in the experiments.Instruction-tuning and adversarial post-training omit fake special tokens, so trusting and ignoring fake starts agree on that data; the authors do not prove gradient descent selects these weights.Source: §A.4 · p. 22

B AgentDojo Result Details

The AgentDojo details report role-spoofing success under multiple injection formats and describe a separate testbed designed to isolate role spoofing across shared task environments. The reported results depend on the injection setup, and the authors caution that their AgentDojo results are not directly comparable with its default harness.

B AgentDojo Result Details

  • The authors allowed unescaped text in tool results but still split special tokens during tokenization, and say these results are not directly comparable with AgentDojo’s default YAML-escaping harness.They characterize this adversary as not especially powerful because agents should be able to distinguish special tokens from tool results.Source: B AgentDojo Result Details · p. 22
  • 82.6% for GPT 5.4 and 83.6% for Opus 5 were their injection success rates relative to control using JSON tool outputs with escaped newlines or quotes.With default YAML tool outputs, GPT 5.4 reached 86.6%, while Claude Opus 5 did not succeed.Source: B AgentDojo Result Details · pp. 22–23
  • The AgentDojo runner itself was injectable: attacks succeeded before the agent acted for over 60% of user-task and adversarial-task pairs across AgentDojo, but these attacks were not used in their injections.The authors followed AgentDojo’s attack procedures.Source: B AgentDojo Result Details · p. 23

C Our Simple Tool-Use Testbed

  • The simple tool-use testbed pairs a user task with an adversarial task drawn from the same task pool, with both evaluated against shared environments and tool outputs.Its suites cover bank accounts, document editing, calendar management, email, and climate control.Source: C Our Simple Tool-Use Testbed · p. 23
  • Each testbed task requires a concrete tool call with checked arguments, and distinct tasks cannot be satisfied by the same call and arguments.This makes both following an injection and resisting it observable; task outcomes are also designed to remain well-defined across environments.Source: C Our Simple Tool-Use Testbed · p. 23
  • Table 3 reports injection success rates for the simple tool-use testbed, while Table 4 describes synthetic suites whose injected instances enumerate ordered pairs of distinct tasks.Of those instances, 3,420 have a security-sensitive adversary task.Source: Table 3 · p. 24, Table 4 · p. 24

D Full Who Said What Results

The expanded Who Said What results add models without fundamentally changing the paper’s conclusions. The authors caution that the apparent inverse correlation is sensitive to query wording.

D Full Who Said What Results

  • Additional models in Figure 7 do not fundamentally change the conclusions drawn from the partial results in Figure 2.Figure 7 compares misleading-content cases without fake structure against fake-structure cases with role-neutral content.Source: D Full Who Said What Results · p. 24
  • The apparent inverse correlation in the Who Said What results is sensitive to the wording of the query, so the authors consider it less strong than it first appears.Source: D Full Who Said What Results · p. 24
  • Figure 8 depicts the misleading-content and fake-structure distributions tested in Figure 2.Source: D Full Who Said What Results · p. 24

E.1 Full Results

The full-results appendix describes additional Who Said What displays and the toy-model setup. Its toy-model findings indicate that instruction-tuned and adversarially post-trained models follow different cues on content and structure injections.

E.1 Full Results

  • Figure 7 contrasts misleading content without fake structure with fake structure paired with role-neutral content, while Figure 8 depicts the tested distributions.Source: Figure 7 · p. 25, Figure 8 · p. 25

E.2 Setup

  • The toy data use role-specific token distributions and message sequences, with pretraining documents mostly lacking message separators and later training stages using explicit markers.Adversarial post-training includes content sampled from the other role’s distribution in half the documents; model-message supervision is limited to content tokens during instruction tuning and adversarial post-training.Source: §E.2 · p. 26
  • Both two-layer and three-layer toy models showed similar results, and tool-hosted injections produced similar but noisier results.The authors used two-layer models in the main paper to make attention patterns easier to view.Source: §E.2 · p. 26
  • The toy experiments test content, structure, and full injections; structure injections add a false role marker while retaining true-role content, whereas full injections also use content from the false role.Source: §E.2 · p. 26
  • The attention diagrams show attention to the final token, with first-layer attention weighted by its contribution to final-token attention in the second layer.Edges are thresholded for readability, and patterns are averaged across 64 training seeds.Source: §E.2 · p. 28
  • In the remaining tested settings, instruction-tuned and adversarially post-trained models disagree on content and structure injections because they follow different cues.Figure 12 shows attention patterns for clean data, content injections, and structure injections before and after adversarial post-training.Source: §E.2 · p. 28

F Experimental Details for the LLM Verification of the Toy Model

The LLM verification experiments fine-tune Qwen3.5-27B on role-identification and tool-use injections, then evaluate on held-out suites and injection formats. The attention figure contrasts clean, content-injection, and structure-injection cases before and after adversarial post-training.

F Experimental Details for the LLM Verification of the Toy Model

  • The study compares base Qwen3.5-27B with two LoRA fine-tunes that differ in whether training injections are marker-free or forge special tokens inside tool results.Source: F Experimental Details for the LLM… · p. 28
  • Fine-tuning uses three differently worded Who Said What queries, each differing from the wording used at test time.Source: F Experimental Details for the LLM… · p. 28
  • The tool-use training data uses three content-only and four structure-only injections from a subset of suites, with evaluation on two held-out suites.Evaluation also includes two held-out content-only injections, Qwen’s native chat-template format, and six other held-out fake-special-token injections.Source: F Experimental Details for the LLM… · p. 28
  • Figure 12 compares attention patterns on clean data, content injections, and structure injections before and after adversarial post-training.The left column is the instruction-tuned model and the right column is the same model after adversarial post-training.Source: Figure 12 · p. 29

G.1 Experiments Comparing Our Probes to Model Self-Attribution

This appendix describes role-probe training and evaluation, including comparisons with self-attribution and probes based on Ye et al. (2026). It reports high detection performance in the low-false-positive regime.

G.1 Experiments Comparing Our Probes to Model Self-Attribution

  • Table 6 compares role-probe flags with model self-attribution, which asks whether a quoted prior-message span came from a tool.The adversarial prefix is inserted in tool responses, and each condition uses the same 1,000 conversations across both models.Source: Table 6 · p. 30, §G.1 · p. 30

G.1.1 Training our Role Probes

  • The role probes are trained to label tool-message content as positive and user- or assistant-message content as negative, using activations from concatenated post-decoder-block layers.Training uses 86,650 unique documents per seed, with eight positive and eight negative tokens supervised per document.Source: §G.1.1 · p. 30

G.2 Training Role Probes based on Ye et al. (2026)

  • Ye et al. (2026)-based probes perform significantly worse than the paper’s role probes, even after selecting the best variants.The comparison selects probes by maximum AUC for false positive rate under 1%.Source: §G.2 · p. 31
  • The Ye et al. (2026) probe transfer uses 250 documents and tests activation layers, role classes, and regularization settings before selecting probes.The selected Qwen probe uses layer 60 pre-MLP activations; the selected Gemma probe concatenates all layers.Source: §G.2 · p. 31

G.3 Classification Probe Training

  • Classification probes distinguish clean tool calls from injected ones, and a training variant includes only content-based injections.They use 400 clean and 400 injected conversations from the synthetic test bed, with activations from 16 randomly chosen content tokens per document.Source: §G.3 · p. 31

G.4 Testing

  • Testing combines clean and injected AgentDojo, testbed, and agentic-tool-use runs, excluding positive samples that never fooled the models.The authors attribute most of the larger Qwen test set to Gemma’s greater resistance to content-only attacks.Source: §G.4 · pp. 31–32
  • For tool-message scores, the probes use the maximum token score; Figure 13 shows activation on user and assistant portions, but not tool portions.ROC evaluation balances averages within each source before averaging across sources.Source: §G.4 · p. 32, Figure 13 · p. 32
  • Each model’s probes achieve greater than 99% true positive rate with zero false positives in the plotted low-false-positive regime.Figure 14 shows mean ROC curves across random seeds, with 95% confidence intervals.Source: Figure 14 · p. 33
Loading 2610.40396v1…