PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents

Hwan Chang, Yonghyun Jun, Hwanhee Lee

arXiv:2509.22830v3cs.CL

TL;DR

Indirect prompt injection exploits external data processed by LLM agents, but prior approaches largely overlook structured chat templates and persuasive multi-turn context. ChatInject formats malicious payloads as native chat-template messages and simulated dialogues, achieving higher attack success, cross-model transfer, and resilience against existing defenses. Its reported limitation is that current defenses impose substantial utility trade-offs and remain insufficient against template-based attacks.

  • Problem

    Indirect prompt injection lets malicious instructions in external tool data manipulate agents, while prior methods primarily rely on plain-text attacks and overlook chat-template and multi-turn vulnerabilities.

  • Method

    ChatInject formats malicious instructions or persuasive multi-turn dialogues with model-specific chat templates to forge role hierarchies and manipulate agent behavior.

  • Results

    ChatInject consistently outperforms plain-text injection, transfers across models including closed-source LLMs, and remains effective against existing defenses and template perturbations.

  • Takeaways & Limitations

    Chat-template dependence and virtual multi-turn structure expose important vulnerabilities in current LLM-agent instruction hierarchies.

  • Takeaways & Limitations

    Existing defenses incur utility trade-offs and remain insufficient against ChatInject variants, especially Multi-turn attacks.

Abstract

from arXiv · show

The growing deployment of large language model (LLM) based agents that interact with external environments has created new attack surfaces for adversarial manipulation. One major threat is indirect prompt injection, where attackers embed malicious instructions in external environment output, causing agents to interpret and execute them as if they were legitimate prompts. While previous research has focused primarily on plain-text injection attacks, we find a significant yet underexplored vulnerability: LLMs' dependence on structured chat templates and their susceptibility to contextual manipulation through persuasive multi-turn dialogues. To this end, we introduce ChatInject, an attack that formats malicious payloads to mimic native chat templates, thereby exploiting the model's inherent instruction-following tendencies. Building on this foundation, we develop a persuasion-driven Multi-turn variant that primes the agent across conversational turns to accept and execute otherwise suspicious actions. Through comprehensive experiments across frontier LLMs, we demonstrate three critical findings: (1) ChatInject achieves significantly higher average attack success rates than traditional prompt injection methods, improving from 5.18% to 32.05% on AgentDojo and from 15.13% to 45.90% on InjecAgent, with multi-turn dialogues showing particularly strong performance at average 52.33% success rate on InjecAgent, (2) chat-template-based payloads demonstrate strong transferability across models and remain effective even against closed-source LLMs, despite their unknown template structures, and (3) existing prompt-based defenses are largely ineffective against this attack approach, especially against Multi-turn variants. These findings highlight vulnerabilities in current agent systems.

1 INTRODUCTION

ChatInject targets indirect prompt injection by forging chat-template role structure and constructing persuasive virtual dialogues inside tool outputs. Experiments show higher attack success, transferability across models, and weak existing defenses.

  • ChatInject exploits role-based hierarchies by inserting forged special tokens into low-priority tool outputs, causing malicious content to appear higher-priority.This bypasses the intended system > user > assistant > tool output hierarchy.
  • The Multi-turn variant embeds forged role tags to create a virtual persuasive dialogue that gradually frames a malicious action as necessary and user-authorized.The attacker uses simulated user and assistant turns despite having only one-shot access to the tool response.
  • ChatInject and its variants consistently achieve higher Attack Success Rates than standard plain-text injection across InjecAgent and AgentDojo.The evaluation spans frontier LLMs and reports template-based attacks as effective across models, including closed-source systems.
  • ChatInject payloads transfer across models, including closed-source LLMs with unknown template structures, and mixture-of-templates attacks work without target-model knowledge.These findings indicate that the attack does not require exact knowledge of the target agent’s proprietary template.
  • Existing prompt-based defenses are largely ineffective, while ChatInject remains robust under template perturbations that defeat rule-based parsing.The reported findings identify template dependence and multi-turn susceptibility as underexplored attack surfaces.

2 RELATED WORK

Prior indirect prompt-injection work emphasizes handcrafted or automated plain-text manipulation, while related instruction-hierarchy and multi-turn research exposes complementary weaknesses. ChatInject combines these directions by embedding simulated persuasion into structured tool outputs.

  • Existing indirect prompt-injection attacks have evolved from manual prompt engineering to automated optimization, but largely remain focused on plain-text manipulation.The related work frames template structure and conversational context as comparatively underexplored attack surfaces.
  • Instruction hierarchies use chat-template tokens to distinguish role priorities, creating a dependency that can be abused when attackers forge those tokens.The hierarchy aims to prevent lower-priority content from overriding higher-priority instructions.
  • Multi-turn jailbreaks exploit gradual persuasion, but indirect injection normally prevents interactive turn-by-turn engagement because the attacker controls only passive tool output.ChatInject addresses this constraint by embedding a simulated multi-turn conversation in one tool response.

3 CHATINJECT

ChatInject formulates indirect injection as malicious instructions embedded in tool responses, then formats either single instructions or persuasive dialogues with plain-text or model-specific templates. The study evaluates these variants across agent benchmarks, metrics, and frontier models.

  • 3.1 PROBLEM FORMULATION: INDIRECT PROMPT INJECTION: An attack succeeds when an agent executes malicious instruction Ia embedded in tool response RTu, although the attacker cannot access the user instruction or internal prompts.The attacker can manipulate only RTu and may aim to induce another tool call Ta.
  • 3.2 PAYLOAD GENERATION WITH TEMPLATE FORMATTING: The template function Ttype formats either a malicious instruction Ia or attacker-crafted conversation Ca, producing four payload variants.The variants combine plain text versus model-specific chat formatting with single-instruction versus multi-turn content.
  • 3.2 PAYLOAD GENERATION WITH TEMPLATE FORMATTING: The four variants are Default InjecPrompt, InjecPrompt + ChatInject, Default Multi-turn, and Multi-turn + ChatInject.ChatInject wraps content in model-specific role tags, whereas plain variants concatenate text or role-labeled turns.
  • 3.3 EXPERIMENTAL SETUP: The evaluation uses AgentDojo and InjecAgent, measuring Attack Success Rate and Utility under Attack across six open-source and three closed-source frontier models.AgentDojo covers Slack, travel booking, and banking, while InjecAgent includes direct-harm and data-stealing scenarios.
  • 3.3 EXPERIMENTAL SETUP: The study also evaluates reasoning and tool-calling hooks, but Table 1 restricts these variants to models that explicitly provide the corresponding template tokens.The table reports results on InjecAgent and AgentDojo for six LLM agents.

4 EVALUATING THE EFFICACY OF CHATINJECT

ChatInject substantially increases attack success across benchmarks, with persuasive multi-turn formatting producing especially strong gains. Agentic reasoning and tool-use wrappers amplify attacks further, while higher attack success often reduces legitimate-task utility.

  • 4.1 CHATINJECT DISRUPTS AGENT BEHAVIOR: 45.6% average ASR is achieved by Multi-turn + ChatInject across most models, versus 13.8% for Default Multi-turn.ChatInject consistently raises ASR over both Default InjecPrompt and Default Multi-turn on both benchmarks.
  • 4.1 CHATINJECT DISRUPTS AGENT BEHAVIOR: ChatInject’s effectiveness varies with model template structure, with concise explicit role delimiters associated with larger ASR increases than lightweight separators.Grok-2 shows only minor gains, whereas Qwen-3 and GLM-4.5 show larger increases.
  • 4.1 CHATINJECT DISRUPTS AGENT BEHAVIOR: On AgentDojo, higher ASR is accompanied by a systematic Utility decline, indicating that attacker payloads divert agents from original user instructions.Utility tends to decline even in Multi-turn settings where the system prompt permits the original instruction to coexist.
  • 4.2 EXPLOITING AGENTIC REASONING AND TOOL-USE TEMPLATES: Reasoning hooks append affirmative cues within <think> tokens, while tool-calling hooks append agent-style tool-call scaffolds within <tool> tokens.These wrappers target the agent’s reasoning and tool-execution phases.
  • 4.2 EXPLOITING AGENTIC REASONING AND TOOL-USE TEMPLATES: The agentic variant further increases ASR and reduces Utility relative to InjecPrompt + ChatInject.Tool-calling hooks produce particularly large swings, even when tools are unnamed on AgentDojo.

5 CROSS-MODEL TRANSFERABILITY OF CHATINJECT

Cross-model ChatInject transfer strengthens when foreign chat templates resemble the target’s native template, extends to closed-source models, and can be stabilized by mixing templates when the backbone is unknown.

  • 5.1 TEMPLATE SIMILARITY AS A PREDICTOR FOR ATTACK TRANSFER: Template similarity increases cross-model ASR while degrading Utility, with Qwen-3 gaining over 20% ASR from the most versus least similar tested template.GPT-oss remains comparatively robust across foreign templates, but the same trend remains visible.
  • 5.2 EMPIRICAL ANALYSIS OF CROSS-MODEL CHATINJECT TRANSFERABILITY: Foreign templates generally underperform native templates on open-source models, but frequently exceed Default InjecPrompt on AgentDojo’s more complex environments.This establishes foreign templates as a credible threat in realistic agent pipelines.
  • 5.2 EMPIRICAL ANALYSIS OF CROSS-MODEL CHATINJECT TRANSFERABILITY: Qwen-3 and Kimi-K2 show strong mutual transferability, whereas Grok-2 is robust to foreign templates and transfers poorly across models.Average foreign-template ASR is 21.4% on InjecAgent and 16.8% on AgentDojo for Qwen-3, versus 9.2% and 5.4% for Grok-2.
  • 5.2 EMPIRICAL ANALYSIS OF CROSS-MODEL CHATINJECT TRANSFERABILITY: Open-source templates also transfer effectively to closed-source targets, often exceeding Default InjecPrompt despite unknown proprietary templates.Qwen-3 averages 29.6% ASR on InjecAgent and 23.5% on AgentDojo against closed-source models.
  • 5.3 CHATINJECT AGAINST UNKNOWN AGENTS VIA TEMPLATE MIXING: Mixture-of-templates consistently exceeds Default InjecPrompt across three backbones and has lower variance than arbitrary single-template attacks.Concatenating candidate templates increases the chance of including the target’s native wrapper, improving stability when the backbone is unknown.

6 DEFENDING AGAINST CHATINJECT: EVALUATION AND BYPASS

Standard defenses do not reliably mitigate ChatInject: prompt-based methods can leave attack success above no-defense baselines, while detector defenses reduce attacks but impose coarse utility costs. Template perturbations preserve ChatInject’s advantage over default attacks, with robustness depending on the edit type and attack variant.

  • 6.1 EVALUATING STANDARD INDIRECT INJECTION DEFENSES: Higher ASR against ChatInject and Multi-turn methods than baseline no-defense conditions shows prompt-based defenses fail to reliably distinguish malicious from user intent.Repeated instructions and preemptive guidance can be overridden by structural and contextual manipulations.
  • 6.1 EVALUATING STANDARD INDIRECT INJECTION DEFENSES: Detector defenses reduce ASR across variants but retain relatively higher ASR for Default Multi-turn, indicating persuasive dialogue can evade detection.Multi-turn + ChatInject has lower ASR than Default Multi-turn, suggesting detectors react partly to special tokens.
  • 6.1 EVALUATING STANDARD INDIRECT INJECTION DEFENSES: Detector-based defenses can stall the agentic pipeline because one malicious-content flag removes the entire tool output, causing utility degradation beyond false positives.The failure mode is coarse-grained: detection removes potentially useful output rather than isolating the malicious instruction.
  • 6.2 BYPASSING TEMPLATE-STRIPPING WITH ADVERSARIAL PERTURBATIONS: Format stripping offers a natural countermeasure by removing role tags and delimiters, potentially degrading ChatInject into a vanilla injection.This motivates testing adversarial template perturbations against rule-based parsing.
  • 6.2 BYPASSING TEMPLATE-STRIPPING WITH ADVERSARIAL PERTURBATIONS: All 10%-perturbed variants outperform Default InjecPrompt and Default Multi-turn across three models, with insertion strongest for single-template attacks and removal often strongest for MoT.Insertion minimally distorts salient role delimiters, whereas MoT’s template redundancy makes it robust to dropped characters.

7 CONCLUSION

ChatInject exploits LLM chat templates to perform indirect prompt injection through model-specific formatting and multi-turn dialogues. It consistently outperforms plain-text methods, transfers across models including closed-source systems, bypasses current defenses, and remains robust to template perturbations.

  • 7 CONCLUSION: ChatInject exploits LLM chat templates and multi-turn dialogues to bypass instruction hierarchies and hijack agent behavior.The method uses model-specific formatting to embed malicious instructions in simulated conversational structures.

ETHICS STATEMENT

The study evaluates a potentially harmful prompt-injection attack under controlled research conditions. Its design emphasizes responsible disclosure and avoids exposure of real systems or users.

  • ETHICS STATEMENT: Experiments use controlled environments, publicly available datasets, and simulated scenarios rather than real systems or user data.The methodology follows stated ethical considerations and responsible disclosure principles.

REPRODUCIBILITY STATEMENT

The paper documents its datasets, models, prompts, benchmarks, metrics, and configurations to support reproducibility. It also states that an LLM assistant was used only for manuscript refinement, not research ideation or content generation.

  • REPRODUCIBILITY STATEMENT: Reproducibility materials describe multi-turn conversation construction, model and prompt choices, benchmarks, evaluation metrics, and experimental configurations.Appendix D additionally documents the methodology for the reported experiments.
  • REPRODUCIBILITY STATEMENT: The authors drafted the manuscript themselves and used ChatGPT-5 only for style, clarity, and grammar refinement.The assistant was not used for research ideation or content generation.

B LIMITATIONS AND FUTURE WORK

The study identifies limitations in synthetic dialogue generation, internal mechanistic analysis, and existing defenses against ChatInject variants.

  • Synthetic Multi-turn Generation: Synthetic multi-turn dialogues generated with GPT-4.1 may not capture the diversity of real-world persuasive conversations.The authors propose validation with naturally occurring or human-crafted persuasive conversations.
  • Limited Internal Analysis: Resource constraints prevented detailed attention analysis of how chat templates influence model behavior at the representational level.Future work could examine attention patterns and internal representations during template-based attacks using interpretability techniques.
  • Defense Limitations: Existing defenses provide only partial mitigation while increasing prompt length, runtime processing, and false positives that degrade Utility.ChatInject variants still outperform the Default InjecPrompt baseline when defenses are deployed, motivating defenses tailored to template-based and persuasive multi-turn attacks.

C.1 ANALYSIS OF MULTI-TURN CONTEXT EFFECTS

The analysis shows that multi-turn attack design involves a trade-off between attack success and utility, while template similarity and ordering strengthen transfer attacks.

  • Effect of Number of Turns: ASR remains relatively stable as dialogue turns increase, whereas Utility steadily decreases with longer multi-turn attacks.The authors attribute this pattern to repeated exposure and increasing context that shift focus toward injected instructions.
  • Analysis by Persuasion Taxonomy: Time pressure and social punishment generally produce higher ASR, while negotiation lags; authority and negotiation retain higher Utility.These results show that persuasion strategy affects both attack success and task preservation.
  • Comparison with Real-World Corpora: Persuasion-based synthetic multi-turn attacks achieve superior ASR compared with adapted real attack corpora.The comparison supports the effectiveness of the synthetic generation methodology for prompt-injection attacks.
  • Attention Analysis: Template tags shift attention toward attacker instructions across models, reallocating computational resources to template-wrapped malicious content.The attention analysis measures the relative weighting assigned to user versus attacker instructions.
  • Template Similarity: Homoglyph-encoded templates yield very low ASR, consistent with their low similarity to the models’ original templates.This supports a relationship between template similarity and attack effectiveness across tokenization schemes.
  • Mixture-of-Template Analysis: Descending Mixture-of-Templates ordering produces higher and more stable ASR than ascending ordering, with limited sensitivity to added templates.The result suggests placing high-similarity templates early is effective for unknown-backbone attacks.

E RESULTS WITH CONFIDENCE INTERVAL

The evaluation reports benchmark results with confidence intervals and documents the payload variants, model-specific templates, defenses, and task examples used in the experiments.

  • Evaluation Setup: The evaluation compares Default InjecPrompt, ChatInject, Default Multi-turn, and Multi-turn + ChatInject payload variants across InjecAgent and AgentDojo.The benchmark tables report results for six LLM agents and identify changes relative to Default InjecPrompt.
  • Confidence Intervals: 95% Wilson confidence intervals account for benchmark sampling uncertainty, with wider intervals for AgentDojo because it contains fewer samples than InjecAgent.The reported interval widths do not alter the study’s conclusions.
  • Payload Construction: Model-specific role tags and structured prompts define the ChatInject payloads and synthetically generated multi-turn dialogues.The appendix materials provide configuration tags, generation prompts, and representative payload examples.
  • Attack and Defense Examples: The experiments include defensive prompts and concrete attacker directives embedded in tool responses to evaluate execution under realistic agent interactions.Examples include sending a link and adding a directive alongside the existing user instruction.
Loading 2509.22830v3…