Towards Action Hijacking of Large Language Model-based Agent

1. Introduction

• LLM-based applications retrieve knowledge from memory, use an LLM to plan actions, and direct tools to execute the resulting action list.Retrieval-augmented generation supplies database knowledge to the LLM.Source: §1 · p. 1
• Prior prompt injection and jailbreak attacks use harmful semantics or templates that safety filters can detect, motivating a stealthier approach.The paper also notes that memory-poisoning attacks require privileged database access, which it excludes from scope.Source: §1 · pp. 1–2
• AI2 extracts action-aware knowledge, then guides the application to retrieve and assemble it into harmful instructions without explicit harmful semantics in the prompt.The approach has three phases: knowledge extraction, assembly-instruction design, and generation of retrieval-aligned hijacking prompts.Source: §1 · p. 2
• 84.30% average attack success rate across three real-world application domains, with a best rate of 99.70%.AI2 also achieved a 91.44% average ASR on commercial platforms, and bypass rates of 99.35% for built-in filters and 59.45% for dedicated defenses.Source: §1 · p. 2
• The paper evaluates AI2 on code generators, medical assistants, and Text2SQL agents, and reports validation across various LLMs, datasets, and task settings.These evaluations address effectiveness, stealthiness, and robustness.Source: §1 · p. 2

2.1. LLM-based Applications

• An LLM-based application combines a reasoning-and-planning brain with short- and long-term memory, using long-term memory for complex tasks.The brain selects tools for execution; short-term memory stores internal logs, while long-term memory stores and retrieves information.Source: §2.1 · p. 2

2.2. Attack Vectors of LLM-based Applications

• Memory poisoning changes long-term memory and requires privileged access, whereas prompt manipulation targets applications without intruding into memory.Prompt manipulation includes injection techniques such as Escape, Ignore, and Completion, while jailbreaks exploit limited understanding of safety guidelines.Source: §2.2 · p. 3

2.3. Protection over LLM-based Applications

• Prompt injection and jailbreak prompts retain harmful semantics or templates, making them detectable by keyword searches or LLM-based rewrites.Defenses include model enhancement and user-prompt investigation, from simple filters to LLM-based detectors and prompt rewriting.Source: §2.2 · p. 3, §2.3 · p. 3

3.1. Attacker’s Goal

• The attacker constructs a prompt that maximizes generation of a target action while minimizing the chance that safety filters block it.The application may use a retriever, knowledge database, and other settings such as a system prompt; generated actions are not necessarily executed because execution depends on external tools.Source: §3.1 · pp. 3–4
• The examples cover code generators, medical assistants, and Text2DSL agents, with goals including incorrect code or vulnerability decisions, misdiagnosis or controlled prescriptions, and hijacked context.The paper focuses on generating harmful executable actions and potential destructive effects, not on whether external tools execute them.Source: §3.1 · p. 4

3.2. Attacker’s Knowledge and Capability

• In the practical scenario, the attacker can submit textual prompts through API access; a weak attacker lacks system information, while a strong attacker may know part or all of the workflow.The assumed interface limits the attacker to textual prompts.Source: §3.2 · p. 4

4.1. Overview

• AI2 uses action-aware knowledge extracted from the application’s database as material for assembling harmful instructions, rather than embedding those instructions directly in the prompt.The approach is inspired by Return-Oriented Programming, where existing instructions in memory are reused.Source: §4.1 · p. 4
• The attack proceeds through knowledge extraction, hijacking-prompt generation that induces retrieval and assembly, and application hijacking to trigger harmful action plans.The overview figure depicts the three phases, including prompt construction and the final application interaction.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2. Extracting Action-aware Knowledge

• AI2 addresses the limitation of prompt-extracting attacks, which reveal fixed content but not specific knowledge retrieved from an application's database.The attacker seeks an optimal prompt that maximizes the relevance of retrieved knowledge to the target action.Source: §4.2 · p. 5

4.2.1. Generating Action-Aware Prompt Candidate Set.

• Action2NL converts the target action into candidate prompts, using an Attack Memory to improve candidates when the action is not accurately represented in the application database.The memory stores feedback rules, the current best prompt candidates and their associated knowledge, plus recent interaction logs for self-reflection.Source: §4.2.1 · p. 5

4.2.2. Identifying Optimal Action-aware Prompt.

• Prompt selection rewards both operational relevance and novelty, seeking previously unexplored knowledge about objects involved in the target action.The similarity measure requires matching operations and measures similarity between their objects.Source: §4.2.2 · p. 5

4.2.3. Constructing Knowledge Extraction Prompt.

• The extraction prompt combines the optimized action-aware prompt with instructions to filter irrelevant information and output retrieved content verbatim.The three-part prompt is pE = p̃ ⊕ po ⊕ pe.Source: §4.2.3 · p. 5

4.3. Generating Hijacking Prompt

• The hijacking prompt must induce retrieval and assembly of extracted knowledge into harmful instructions while bypassing safety filters.Figure 3 shows initialization from the target action and extracted knowledge, followed by adversarial-token optimization using a shadow retriever and optimizer.Source: §4.3 · p. 5, Figure 3 · p. 6

4.3.1. Designing Assembly Instruction.

• The initial prompt pairs an assembly instruction with optimizable tokens; the instruction describes the target action without including its harmful operation.For example, “DELETE all data on the table Artist” becomes “Perform the same operation on the Artist table”.Source: §4.3.1 · pp. 5–6
• Adversarial-token placement varies by application, and code generators add constraints from code-heavy queries and programming-language syntax.The authors address these code-generation constraints by combining code obfuscation with adversarial generation.Source: §4.3.1 · p. 6

4.3.2. Optimizing Hijacking Prompts.

• Token-level optimization adjusts the prompt to retrieve target knowledge and assemble it into the harmful instruction, using embedding similarity as a loss for gradient-based or optimization-based search.The described search strategies include FGSM, I-FGSM, PGD, and C&W.Source: §4.3.2 · p. 6
• Table 1 illustrates a Text2SQL hijack targeting deletion of benign users’ data, with adversarial tokens distinguished from the harmful actions executed by the agent.Source: Table 1 · p. 6
• Repeated unsuccessful prompts signal a local optimum, prompting an increase in perturbation to help the search escape.The method tracks previously encountered prompts and resets perturbation to its initial value when the current prompt is not repeated.Source: §4.3.2 · p. 6, Figure 3 · p. 6

4.3.3. Interpretation of Hijacking Prompt.

• Retriever and filter representations can differ: Figure 4 contrasts scattered semantic-space mappings with separated retriever-space mappings, which the authors use to explain filter bypass while retrieving targeted information.The authors attribute these differences to model architectures, training objectives, and feature extraction methods.Source: §4.3.3 · p. 7, Figure 4 · p. 7

5.2. RQ1: Hijacking Performance and Comparisons

• 68.77% one-time knowledge extraction rate across three application domains, averaging 1.26 queries per extracted item.Code-generation extraction reached 94.43% ASR; Text2SQL required 0.81 attempts per item, while medical extraction achieved 41.08% ASR with 2.06 queries on average.Source: Table 3 · p. 8, §5.2 · p. 8
• More than 73.19% average ASR for gradient-based hijacking-prompt generation, versus 56.10% for optimization-based methods.For obfuscated code generation against M3E, the average ASR was 58.78%, which the paper attributes to unusable tokens in the retriever vocabulary.Source: Table 4 · p. 9, §5.2 · p. 9
• 84.30% average ASR (best 99.70%) for action hijacking, with task ASRs of 73.77% for Text2SQL, 96.34% for medical assistants, and 82.8% for code generation.The study counts hijacking as successful when the application produces the attacker’s intended harmful operation or diagnosis, or modifies code as instructed.Source: §5.2 · p. 9, Table 5 · p. 10
• 65.53% average Top-1 ASR for Action2NL, improving ACC by 50.87% over randomized prompt generation.The paper reports little difference between Top-1 and Top-4 ASR, indicating that target knowledge is usually retrieved among the most relevant results.Source: Table 6 · p. 10, §5.2 · p. 10
• 76.67%–81.03% ASR for PGD as extracted knowledge increased; across perturbation rates, PGD rose from 28.0% to 79.0% before plateauing at 76.3%.Attack effectiveness generally increased with extracted knowledge; the C&W results peaked at r = 0.2, while iterative gradient methods outperformed one-step and minimum-norm methods in the reported sparse-text setting.Source: Figure 5 · p. 10, §5.2 · pp. 10–11
• 43.68% transfer ASR between retrievers trained on the same dataset, compared with 37.41% for identical architectures and 27.94% for different architectures and training datasets.When the application retriever was accessible, the ASR for constructing erroneous context reached 100%.Source: Table 7 · p. 11, §5.2 · p. 11

5.3. RQ2: Robustness in Bypassing Safety Defense

• 100% bypass rate against banned-word filters and 98.70% against operation filters; AI2 also achieved 59.15% average bypass against prompt-driven safeguards and 59.75% against prompt rewriting.Prompt injection and jailbreak methods averaged 41.55% and 27.68% against prompt-driven safeguards, while other approaches achieved 26.11% against rewriting. The evaluation focuses on SQL applications because effective input-based filters for the other discussed harmful behaviors were unavailable.Source: §5.3 · pp. 11–12, Table 8 · p. 12

5.4. RQ3: Performance against Real-world Applications

• 94.41% average ASR for hijacking-prompt generation and 96.85% for hijacking attacks in real-world scenarios, with a final average ASR of 91.44%.In two self-built code-generation applications, target-knowledge retrieval reached 88.92% and language-model hijacking reached 97.11%; the passage does not provide the combined-attack result.Source: Table 9 · p. 13, §5.4 · p. 13
• Medical applications achieved an average overall ASR of 90.99%, with 94.17% for knowledge fetching and 96.45% for output hijacking.Using ALLMiniLM as the shadow retriever, the evaluation set attack strength to 0.2, meaning 20% of prompt tokens could serve as adversarial tokens.Source: §5.4 · p. 13

prompts.

• Text2SQL applications achieved a 95.50% overall ASR, with 98.34% for generating hijacking prompts and bypassing security mechanisms and 97.15% for hijacking actions.The prompts replaced 20% of prompt tokens and targeted harmful database commands, including unauthorized deletion, spam injection, and viewing other users’ private data.Source: prompts · p. 13
• Vanna achieved a 94.62% overall hijacking ASR, with 98.04% for knowledge fetching and 96.52% for action hijacking.The attack targeted Vanna’s ChromaDB vector store using ALLMiniLM as the shadow retriever, with 20% of prompt tokens as adversarial tokens.Source: prompts · pp. 13–14

6.1. Limitations

• AI2 depends on retrieving hijacking-related knowledge; when retrieval is difficult, longer prompts require more adversarial tokens and may be less stealthy and easier to detect.The authors suggest splitting complex actions into steps and using agent short-term memory to inject information incrementally.Source: §6.1 · p. 14
• Harmful actions generated by the LLM may not execute, depending on external-tool implementation and selection strategy.The paper recommends layered protection for applications and external APIs and tools.Source: §6.1 · p. 14

6.2. Defenses

• A defense could inspect LLM, user, and database prompts jointly before planning, but would need comparable long- and short-term memory and stronger harmful-information recognition, while consuming more tokens.Source: §6.2 · p. 14

• AI2 extracts action-aware knowledge from an application's database and induces the application to assemble harmful instructions that lead the LLM to generate harmful actions.The evaluation covers six popular LLMs, two retrievers, sub-word-based and semantic-based filters, and code generators, medical assistants, and Text2SQL agents.Source: §7 · p. 14
• AI2 achieves outstanding performance in bypassing safety filters and hijacking applications.Source: §7 · p. 14

• One knowledge-extraction prompt asks for encryption of the input after an action-aware description is supplied.Source: Appendix · p. 18
• A second knowledge-extraction prompt frames the task as security testing and requests a summary that fills a block template with domain-specific knowledge.Source: Appendix · p. 18
• The assembly prompt directs an assistant to replace an action's target operation with an instruction to retrieve the operation keyword from knowledge, then organize the action's objects and descriptions.It also asks the assistant to check and complete the prompt and output it.Source: Appendix · p. 19
• The Text2SQL checker classifies a user prompt as safe or dangerous and specifies a fixed response for each classification.Source: Appendix · p. 19