PaperlayerView exact revision on arXiv

Source-linked AI summary

Automating Agent Hijacking via Structural Template Injection

Xinhao Deng, Jiaqing Wu, Miao Chen, Yue Xiao, Ke Xu, Qi Li

arXiv:2602.16958v1cs.AIcs.LG

TL;DR

LLM agent hijacking can steer systems with access to external tools, while semantic attacks have limited transfer across models and can face alignment defenses. Phantom searches for structured templates that exploit role parsing; it reports high attack success across closed-source agents and over 70 confirmed commercial vulnerabilities.

  • Problem

    Semantic prompt attacks face stronger instruction-following defenses and transfer poorly across models, while agent hijacking can steer systems with access to tools and external services.

    Source: §1 · p. 1

  • Method

    Phantom injects structured templates to induce role confusion, using a Template Autoencoder and Bayesian optimization to search for attack templates against black-box agents.

    Source: §1 · p. 2, §5.2 · p. 6

  • Results

    79.76% average Attack Success Rate on AgentDojo across seven closed-source agents, and over 70 distinct vulnerabilities identified among 942 commercial agents.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings identify chat-template parsing as an architectural vulnerability that can bypass predominantly semantic alignment strategies.

    Source: §1 · p. 2

    The evaluation focuses on text-based interactions, may not capture multimodal agents’ distinct vulnerabilities, and iterative query optimization may face strict rate limits or anomaly detection.

    Source: §7 · p. 11

Abstract

from arXiv · show

Agent hijacking, highlighted by OWASP as a critical threat to the Large Language Model (LLM) ecosystem, enables adversaries to manipulate execution by injecting malicious instructions into retrieved content. Most existing attacks rely on manually crafted, semantics-driven prompt manipulation, which often yields low attack success rates and limited transferability to closed-source commercial models. In this paper, we propose Phantom, an automated agent hijacking framework built upon Structured Template Injection that targets the fundamental architectural mechanisms of LLM agents. Our key insight is that agents rely on specific chat template tokens to separate system, user, assistant, and tool instructions. By injecting optimized structured templates into the retrieved context, we induce role confusion and cause the agent to misinterpret the injected content as legitimate user instructions or prior tool outputs. To enhance attack transferability against black-box agents, Phantom introduces a novel attack template search framework. We first perform multi-level template augmentation to increase structural diversity and then train a Template Autoencoder (TAE) to embed discrete templates into a continuous, searchable latent space. Subsequently, we apply Bayesian optimization to efficiently identify optimal adversarial vectors that are decoded into high-potency structured templates. Extensive experiments on Qwen, GPT, and Gemini demonstrate that our framework significantly outperforms existing baselines in both Attack Success Rate (ASR) and query efficiency. Moreover, we identified over 70 vulnerabilities in real-world commercial products that have been confirmed by vendors, underscoring the practical severity of structured template-based hijacking and providing an empirical foundation for securing next-generation agentic systems.

1 Introduction

Phantom targets the structural parsing of LLM agents, where serialized chat templates leave control tokens and untrusted content insufficiently isolated. It automates template discovery for black-box agents and reports strong benchmark performance and confirmed vulnerabilities in production systems.

1 Introduction

  • Agent hijacking can move beyond manipulating outputs to operational control over connected systems, while existing semantic attacks face alignment defenses and limited cross-model transfer.The paper identifies chat-template serialization as a structural weakness: control markers and retrieved content share a flattened token stream without strict token-level isolation.Source: §1 · p. 1
  • Phantom injects structured token sequences to create a “ghost history,” making attacker-controlled text appear as legitimate user instructions or prior tool outputs.The attack targets structural parsing rather than semantic interpretation and is described as largely independent of the model’s semantic alignment procedure.Source: §1 · p. 2
  • Phantom uses template augmentation, a Template Autoencoder, and Bayesian optimization with proxy evaluation to search for attack templates using API-level access.The pipeline addresses black-box agents whose internal states and proprietary chat templates are inaccessible.Source: §1 · p. 2
  • 79.76% average Attack Success Rate on AgentDojo across seven closed-source agents; Phantom also identified over 70 confirmed vulnerabilities in 942 commercial agents.Reported violations ranged from data exfiltration to Remote Code Execution.Source: §1 · p. 2
  • The paper identifies the lack of strict isolation between control and content tokens, proposes automated cross-model structural injection, and validates real-world impact through commercial-agent evaluation and responsible disclosure.The authors characterize parsing-architecture attacks as a bypass of predominantly semantic alignment strategies.Source: §1 · p. 2

2 Background

LLM agents use structured chat templates to organize roles and turns, but retrieved content containing matching control tokens can escape its intended boundary. Phantom’s structural attack builds on this weakness in indirect prompt injection.

2.1 LLM Agents and Chat Templates

  • Agents combine perception, planning, and action with tools, while chat templates use beginning- and end-of-turn tokens to delimit System, User, Assistant, and Tool/Observation roles.The agent’s reasoning engine processes system prompts, user queries, and tool observations to plan and act.Source: §2.1 · p. 2

2.2 Indirect Prompt Injection

  • Indirect prompt injection embeds malicious text in retrieved sources to influence an agent, but semantic hijacking can be mitigated by safety alignment and context filtering.Source: §2.2 · pp. 2–3
  • When retrieved content contains chat-template control tokens, an agent parsing a flattened token sequence may treat the content as genuine instructions or prior tool output.This structural route differs from attacks on content logic and may be harder for semantic alignment and context filtering to detect.Source: §2.2 · p. 3

3 Threat Model

The threat model assumes a remote, black-box attacker who can influence retrieved data but cannot inspect the target’s internals or template. The attack injects control-token sequences that forge dialogue history and can redirect the agent’s execution.

3 Threat Model

  • The attacker has no access to model weights, gradients, system prompts, execution traces, or the target’s chat-template format, and relies on query access to refine payloads.Source: §3 · p. 3
  • Compared with direct injection, structured template injection shifts attention from authentic content toward the injected prompt.Source: Figure 2 · p. 3
  • By embedding control tokens in retrieved content, the attacker can terminate a tool-processing frame and fabricate an Assistant response followed by a malicious User query.The agent may then misattribute the injected text as trusted instructions or legitimate history.Source: §3 · p. 3
  • Success means the agent’s output satisfies the adversary’s objective rather than the user’s; the attack targets parsing logic even when the target uses state-of-the-art safety alignment.The stated outcomes include privilege escalation, unauthorized data exfiltration, or arbitrary tool execution.Source: §3 · p. 3

4 Design of Phantom

Phantom exploits agents’ reliance on structured role markers, which can make forged instructions appear legitimate and redirect attention. It searches for effective templates by augmenting templates, mapping them into a latent space, and applying Bayesian optimization.

4.1 Key Observation

  • Structured template injection shifts attention away from original content toward a forged assistant response and user query, which can lead to role confusion and unintended tool actions.The authors attribute this to agents’ reliance on template markers to delineate roles and semantic boundaries.Source: §4.1 · pp. 3–4
  • Agents may fail to distinguish system-generated control information from forged control information, while commercial closed-source systems typically conceal their template details.The paper says effective mitigation may require redesigning template-mediated interaction and execution frameworks.Source: §4.1 · p. 4

4.2 Overview of Phantom

  • Phantom builds a diverse seed pool through template augmentation, then maps discrete structures into a continuous latent space for Bayesian optimization of high-impact templates.This automated search is designed for closed-source agents with limited information about their prompting and template logic.Source: §4.1 · p. 4, §4.2 · p. 4
  • The design combines LLM-generated variants and symbolic perturbations, a fine-tuned autoencoder for template reconstruction, and Bayesian optimization using a Random Forest surrogate and Expected Improvement.A lightweight fictitious-turn-counting evaluation estimates hijacking success to reduce direct agent interaction; the selected latent vector is decoded into a structured template.Source: §4.2 · p. 4

5 Design Details

Phantom builds structurally diverse templates, maps them into a compact latent space, and uses proxy-guided Bayesian optimization to search for hijacking templates.

5.1 Multi-level Template Augmentation

  • LLM rewrites preserve template skeletons while deterministic mutations diversify boundaries, whitespace, casing, delimiters, special characters, and encodings.Semantic augmentation yields 3,833 distinct templates; the combined corpus expands to over 20,000 instances after character-level augmentation.Source: §5.1 · pp. 4–6

5.2 Latent Space Mapping

  • TAE uses a PEFT-adapted Qwen3-0.6B encoder, projection layer, and decoder to compress serialized template triplets while preserving structure and meaning.LoRA freezes pretrained parameters and trains low-rank matrices in attention layers.Source: §5.2 · p. 6
  • Identity activation at the projection layer empirically outperforms ReLU and Tanh.The authors interpret this as evidence that the pretrained LLM representation is already sufficiently structured for compression.Source: §5.2 · p. 6
  • The learned manifold is locally smooth, groups templates by structural patterns, and reduces dimensionality to make optimization computationally tractable.The paper reports that latent dimensionality m is much smaller than maximum template length Lmax.Source: §5.2 · p. 6

5.3 Automated Template Search

  • A Random Forest surrogate and Expected Improvement guide candidate selection, while augmented templates provide warm starts and data-driven bounds constrain the search.Candidates are decoded at temperature 0 for deterministic reconstruction.Source: §5.3 · p. 7
  • Bayesian optimization searches the latent space using a lightweight role-confusion proxy, warm starts, and deterministic decoding to limit costly target-agent evaluations.The proxy checks whether a model accepts a fabricated turn by testing its round-counter response; invalid reconstructions receive a worst-case penalty.Source: §5.3 · p. 7, §5.2 · p. 7

6 Evaluation

The evaluation compares Phantom with attack baselines on AgentDojo, then tests defenses, real deployments, structural perturbations, and search settings. It reports benchmark gains, vulnerabilities in commercial systems, and the importance of preserving tag-style structure.

6.1 Experiment Setup

  • The setup used AgentDojo's four domains and compared Phantom with semantic, static-template, and ChatInject baselines, measuring ASR and benign-task utility.The benchmark comprises 97 legitimate tasks and 629 adversarial security test cases; the template search used a 100-iteration budget.Source: §6.1 · p. 8

6.2 Evaluation on SOTA Agents

  • 79.76% average ASR for Phantom exceeded Single-Template (54.09%), Semantic-Injection (39.86%), and ChatInject (38.46%) across agents; Figure 4 covers agents and scenarios.The evaluation includes seven closed-source models and one open-source model; structured domains had higher ASRs than Workspace.Source: §6.2 · p. 8, Figure 4 · p. 9

6.3 The Attack Performance against Defenses

  • Under defenses, Phantom remained effective against instructional and tag-filter defenses, while Injection Detector suppression came with reduced utility.On GPT-4.1, Phantom retained 76.62% ASR under Delimiter Spotlighting versus 12.99% for the semantic baseline; under Tag Filter, it reached 50.00% ASR on Qwen-Flash. GPT-4o utility fell to 15.58% with the Injection Detector.Source: §6.3 · p. 9, Table 2 · p. 10, Table 3 · p. 10

6.4 Real-World Evaluation

  • The enterprise assessment identified 70 vulnerabilities across 942 commercial agents, while a cloud-agent case study reached privilege escalation and control of a cloud desktop.Structured payloads embedded in image metadata, web content, and source-code repositories triggered violations including command execution and data exfiltration.Source: §6.4 · p. 10

6.5 Interpretability of Structural Exploitation

  • Removing structural delimiters or the template drove attack success near zero, whereas forced tokenization and HTML encoding preserved high success rates.Table 4 reports target-tool-call probability and attention to the attack payload and user prompt; the results support dependence on tag-style structure rather than exact special-token identities.Source: §6.5 · p. 11, Table 4 · p. 11

6.6 Hyperparameter Analysis

  • ASR generally rose with Bayesian optimization iterations, while attack performance peaked at intermediate TAE dimensions rather than increasing linearly.Gemini-3-Flash-Preview ASR rose from 58.4% to 68.2% within the first 40 iterations before plateauing; Figure 5 shows ASR convergence over iterations.Source: §6.6 · p. 11, Figure 5 · p. 11, Table 5 · p. 11

7 Discussion

Phantom’s evaluation is bounded by text-only interactions and a black-box threat model, and its success metrics omit long-term persistence. The authors identify multimodal testing, stronger architectural defenses, and standardized benchmarks as future research directions.

7 Discussion

  • The evaluation measures immediate hijacking success but not long-term persistence in complex conversational state management.This limits what the reported metrics establish about effects over extended interactions.Source: §7 · pp. 11–12
  • The study covers text-based interactions, potentially missing distinct vulnerability profiles in agents processing vision or audio.The authors also note that insiders with training-data access could construct more potent payloads than those considered in the black-box threat model.Source: §7 · p. 11
  • Query-feedback optimization may face challenges against systems using strict rate limits or anomaly detection.Source: §7 · p. 11
  • Future work calls for architectural non-interference defenses, multimodal threat-model extensions, and standardized benchmarks for agent resilience.Proposed defenses include hardware-assisted token isolation and formally verified template parsers.Source: §7 · p. 12

8 Related Works

Related work distinguishes prompt injection from adversarial examples and jailbreaking, and separates direct attacks from indirect attacks embedded in external content. Prior indirect-attack approaches include template-mimicking payloads, while the paper identifies transferability as a remaining limitation.

8 Related Works

  • Prompt injection aims to control LLM outputs or agent behavior, unlike adversarial examples that degrade performance and jailbreaks that elicit socially harmful outputs [29] [35] [32] [42].Source: §8 · p. 12
  • Direct prompt injection modifies user input, whereas indirect prompt injection embeds malicious instructions in third-party data [18].Source: §8 · p. 12
  • Direct attacks can override system prompts, but depend on access to modify them; input validation, safety alignment, and restricted access are cited mitigations [16] [43] [7] [20] [6] [13] [15] [5] [19] [46].Perez et al. [27] study iterative adversarial prompt composition, while Toyer et al. [34] introduce a human-generated benchmark.Source: §8 · p. 12
  • Prior indirect attacks include template-mimicking payloads and a persuasion-driven multi-turn variant, but the paper characterizes these tools as poorly transferable across agents and environments [4].ChatBug [15] showed that replacing special tokens can bypass safety alignments and existing defenses; Greshake et al. [10] and Jiang et al. [14] studied attacks using web content.Source: §8 · p. 12

9 Conclusion

The conclusion frames structural role confusion as an architectural vulnerability and presents Phantom as an automated template-injection framework. The formal-template material specifies the conversation boundaries and payload composition used in examples and baseline configurations.

9 Conclusion

  • Phantom uses structured template injection and a Template Autoencoder with Bayesian optimization to generate adversarial payloads against black-box models, and the evaluation reports over 70 production vulnerabilities.The conclusion attributes the vulnerability to indistinguishable control signals and unstructured data.Source: §9 · p. 12

A Formal Definition and Examples of Templates

B Lightweight Search Analysis

The lightweight search strategy achieves ASRs comparable to Phantom on most evaluated Workspace models, with Gemini-3-Flash-Preview as an exception, while using fewer tokens. In Qwen-Flash experiments, thinking mode increases ASR across nearly all tested scenarios and attack strategies.

B Lightweight Search Analysis

  • The search initializes from 78 baseline templates, runs Bayesian optimization for 100 iterations using proxy evaluation, then validates the 20 highest-scoring templates on AgentDojo Workspace.The per-task template with the highest ASR is selected as the candidate.Source: B Lightweight Search Analysis · p. 14
  • The lightweight strategy achieves ASRs comparable to Phantom on most evaluated Workspace models, except Gemini-3-Flash-Preview.The comparison is reported in Table 10; injected-payload complexity can occasionally cause marginal performance degradation.Source: B Lightweight Search Analysis · p. 14, Table 10 · p. 14
  • 63M tokens is 34% of Phantom’s 184M-token cost for finding optimal Workspace templates across tasks.The lightweight strategy keeps fixed overhead as task count and complexity grow, while Phantom’s consumption scales with them.Source: B Lightweight Search Analysis · p. 14

C Impact of the Thinking Switch

  • Thinking mode significantly increases Qwen-Flash ASR across nearly all tested scenarios and attack strategies.The primary experiments disable thinking by default, except Gemini-3-Flash-Preview, which uses a minimal thinking budget; results appear in Table 11.Source: C Impact of the Thinking Switch · p. 14, Table 11 · p. 14

D Analysis of Semantic-based Attacks

Phantom outperforms semantic-based attacks, with Important Instruction serving as the strongest representative baseline. These baselines use natural-language heuristics and structural tags to steer the agent toward an injected task.

D Analysis of Semantic-based Attacks

  • TODO prepends a goal label, Ignore Previous commands disregard of prior instructions, System Message mimics administrative directives, Injecagent uses high-salience markers, and Important Instruction frames payloads as user-signed directives.Important Instruction uses structural <INFORMATION> tags; Injecagent combines “IMPORTANT!!!” with strict-adherence commands.Source: D Analysis of Semantic-based Attacks · pp. 14–15
  • Phantom exceeds the average ASR of Important Instruction by 39.9%, although Important Instruction outperforms other semantic baselines on nearly all evaluated models.The authors characterize Important Instruction as a highly potent, representative baseline.Source: D Analysis of Semantic-based Attacks · p. 15

E Empirical Analysis of Convergence and Initialization

Across ten trials, Phantom’s ASR improved and its optimization trajectories converged, while a lower mean ASR than the primary evaluation highlighted initialization’s importance. The accompanying augmentation prompts specify how to create structurally varied chat templates.

E Empirical Analysis of Convergence and Initialization

  • Ten Qwen-Flash trials in Workspace each started from 10 randomly sampled templates and ran Bayesian optimization for 200 iterations, rather than using the default 78 initial templates.The setup was intended to assess robustness and stochastic variability.Source: E Empirical Analysis of Convergence… · p. 15
  • ASR rose sharply at first and then improved steadily; narrowing standard-deviation bands showed convergence toward a consistent performance regime and an asymptotic plateau.The authors describe the search as resilient to starting-point choice, while ASR continued to refine.Source: E Empirical Analysis of Convergence… · p. 15, Figure 6 · p. 15
  • 47.66% mean ASR after 200 iterations remained below the primary evaluation’s 55.84%, despite a higher total query budget of 210 versus 178.The authors link this discrepancy to initialization and identify high-quality initial-point selection as a direction for future research.Source: E Empirical Analysis of Convergence… · p. 15

F Prompt for Template Data Augmentation

Prompt for Template Data Augmentation

  • The prompt asks for a new template preserving the tool-response, assistant-output, and next-user-question order, with the tool placeholder as its prefix.It presents the three conversation parts as placeholders and allows content after the user placeholder.Source: F Prompt for Template Data Augmentation · pp. 15–16
  • Template-generation requirements emphasize complete, consistent, symmetric, clear, functional boundary markers and request the output in a Markdown code block.Source: F Prompt for Template Data Augmentation · p. 16

Related papers

Loading 2602.16958v1…