PaperlayerView exact revision on arXiv

Source-linked AI summary

The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis

Peiran Wang, Xinfeng Li, Chong Xiang, Jinghuai Zhang, Ying Li, Lixia Zhang, Xiaofeng Wang, Yuan Tian

arXiv:2602.10453v1cs.CRcs.CL

TL;DR

Prompt-injection research lacks realistic evaluation of context-dependent LLM-agent tasks, where runtime observations legitimately guide actions. This SoK systematizes attacks and defenses, introduces AgentPI, and finds that existing defenses cannot jointly optimize trustworthiness, utility, and latency while often overstating security by suppressing contextual inputs.

  • Problem

    Existing defenses and benchmarks largely overlook context-dependent tasks, despite agents relying on runtime observations for legitimate planning and action.

  • Method

    The SoK reviews and systematizes attacks by payload generation and defenses by intervention stage, then evaluates 8 defenses with the AgentPI benchmark and multidimensional metrics.

  • Results

    No tested defense simultaneously achieves high trustworthiness, high utility, and low latency, while many defenses appear effective by suppressing contextual inputs.

  • Takeaways & Limitations

    Secure agent design must account for context-dependent reasoning rather than relying on defenses that isolate or reject environmental inputs.

  • Takeaways & Limitations

    Existing benchmarks remain predominantly static and can produce a false sense of security against context-aware agent attacks.

Abstract

from arXiv · show

The evolution of Large Language Models (LLMs) has resulted in a paradigm shift towards autonomous agents, necessitating robust security against Prompt Injection (PI) vulnerabilities where untrusted inputs hijack agent behaviors. This SoK presents a comprehensive overview of the PI landscape, covering attacks, defenses, and their evaluation practices. Through a systematic literature review and quantitative analysis, we establish taxonomies that categorize PI attacks by payload generation strategies (heuristic vs. optimization) and defenses by intervention stages (text, model, and execution levels). Our analysis reveals a key limitation shared by many existing defenses and benchmarks: they largely overlook context-dependent tasks, in which agents are authorized to rely on runtime environmental observations to determine actions. To address this gap, we introduce AgentPI, a new benchmark designed to systematically evaluate agent behavior under context-dependent interaction settings. Using AgentPI, we empirically evaluate representative defenses and show that no single approach can simultaneously achieve high trustworthiness, high utility, and low latency. Moreover, we show that many defenses appear effective under existing benchmarks by suppressing contextual inputs, yet fail to generalize to realistic agent settings where context-dependent reasoning is essential. This SoK distills key takeaways and open research problems, offering structured guidance for future research and practical deployment of secure LLM agents.

1 Introduction

This SoK systematizes prompt-injection attacks and defenses, then introduces AgentPI to address the neglect of context-dependent agent tasks. Its evaluation finds a persistent trade-off among trustworthiness, utility, and latency.

  • LLM agents expand capabilities through external tools and environments but remain vulnerable to prompt injections that hijack behavior and leak sensitive data.
  • The SoK categorizes attacks by payload generation and defenses by intervention stage, covering 78 papers and the field’s movement toward practical black-box agent attacks.
  • Existing defenses share a key limitation: they largely overlook context-dependent tasks in which runtime observations affect agent actions.
  • AgentPI systematizes five context-dependent tasks and corresponding attacks to evaluate defenses under realistic environment-dependent interaction.
  • No tested defense simultaneously achieves high trustworthiness, high utility, and low latency, revealing a fundamental defense trilemma.

2 Preliminary and Problem Setup

The paper models agents as iterative execution loops that combine trusted prompts, tool observations, and supply-chain data in one context memory. Prompt injection arises because these inputs lack effective privilege isolation.

  • LLM agents execute through iterative steps involving prompts, optional retrieval and reasoning, and subsequent tool interaction.
  • Different input types coexist in a single context memory, where separators distinguish them but do not create separate interfaces.
  • Agent inputs include trusted system and user prompts, untrusted tool observations, and partially trusted retrieval or training data.
  • Prompt injection exploits insufficient privilege isolation because the model processes differently trusted inputs as one semantic sequence.

3 Taxonomy of Attacks

The attack taxonomy divides payload generation into heuristic and optimization approaches, while the broader analysis tracks surfaces, victims, capabilities, visibility, and goals. The field is shifting toward stealthier, environment-driven integrity attacks against agents.

  • Payload generation: 37 attack papers divide into 23 heuristic-based and 14 optimization-based works according to payload generation methodology.
  • Heuristic attacks: Heuristic attacks include manual templates, LLM-generated prompts, and structural encodings that exploit instruction-following or non-semantic formats.
  • Optimization attacks: Optimization attacks use gradient-based white-box methods or genetic and sampling-based black-box methods to search for adversarial payloads.
  • Evaluation gap: Existing evaluations primarily test heuristic templates even though optimization-based attacks constitute 14 of 37 works, creating an evaluation gap.
  • Attack paradigms: The attack surface has expanded from direct prompt injection to indirect and supply-chain vectors embedded in external resources, RAG, or training data.
  • Attack paradigms: 20 works study indirect prompt injection versus 17 direct-prompt-injection works, while integrity compromise appears in 30 of 37 studies.
  • Attacker capabilities: Most studies operate under black-box constraints, while agentic environmental side channels expose text, scores, or execution effects for state inference.
  • Evolution: The field has moved from visible overrides and theoretical violations toward stealthy environment-driven attacks targeting action or goal hijacking.

4 Taxonomy of Defenses

The SoK organizes prompt-injection defenses by intervention stage—text, model, and execution—and analyzes their protected attributes, granularity, explainability, compatibility, and costs. Across these dimensions, existing defenses leave availability unprotected, often trade utility for security, and struggle with context-dependent tasks.

  • Taxonomy: 41 defense papers are categorized by intervention stage into text-level, model-level, and execution-level defenses.The taxonomy also records protected security attributes, explainability reliance, compatibility cost, and intervention granularity.
  • Intervention stages: Text-level defenses modify input or output strings, model-level defenses intervene in parameters or intermediate representations, and execution-level defenses constrain runtime tool behavior.Execution-level methods operate on the environment side and generally avoid modifying model weights or prompt structure.
  • Protected attributes: 15 of 16 execution-level defenses provide confidentiality protection by tracking tool-call effects on the environment.Across all collected defenses, integrity is protected, while confidentiality is addressed by 17 of 41 works.
  • Open problems: No selected defense protects availability, despite attacks that create endless loops or denial-of-service conditions.The paper attributes this gap to defenses lacking design for cost status.
  • Granularity: Only 6 of 41 defenses operate at segment-level intervention, while coarse-grained refusal can nullify benign task components and create a utility-security trade-off.The paper recommends isolating malicious segments while preserving legitimate content.
  • Defense cost: 33 of 41 defenses incur additional computation, while roughly half require medium or high compatibility cost or operate outside the agent loop.Additional models, probes, and other components can add time, monetary cost, or structural incompatibility.
  • Context-dependent utility: 34 of 41 defenses cannot solve context-dependent tasks because they treat the user prompt as authoritative while ignoring authorized environmental observations.Isolation can remove the attack surface but disrupt tasks whose later actions depend on retrieved context.

5 Benchmarks & Metrics

Existing prompt-injection benchmarks have expanded from static text datasets to tool-using agents, but their tasks, attacks, and metrics remain insufficiently context-aware. Most benchmarks use static tasks and binary outcome checks, which can misrepresent both attack success and defense utility.

  • Benchmark evolution: Benchmarking progressed from static text datasets to agent environments evaluating function-call integrity and dynamic multi-step interactions.Examples include OpenPI and BIPIA for static text, and InjecAgent, AgentDojo, and ASB for agent settings.
  • Task design: Existing benchmark tasks are predominantly static, so subsequent actions depend on the initial prompt rather than runtime observations.They typically do not require using retrieved data, such as file contents, to determine the next action.
  • Context awareness: All 5 benchmarks provide direct tasks and template attacks, while 7 of 41 defense works consider context-dependent scenes without a benchmark covering them.This mismatch leaves context-aware defenses without corresponding evaluation settings.
  • Metrics: Execution-trajectory analysis is needed to capture semantic attack success and cases where a defense neutralizes an attack but leaves the agent non-functional.The paper contrasts trajectory-based evaluation with checks of only static final states.
  • Metrics: 4 of 5 benchmarks mainly use deterministic binary criteria, such as string matching or environment-state and function-call flags, to judge attack success.These criteria can produce high false negatives and falsely low attack-success rates.

6 Proposed AGENTPI Benchmark

AGENTPI introduces dynamic, context-dependent tasks in which an agent’s action depends on an environmental observation, together with context-aware attacks tied to environmental feedback. It identifies five attack types spanning control, logic, and authority flows for evaluating agent behavior.

  • Benchmark design: AGENTPI defines a dynamic task so the action at step t depends functionally on the observation o_t−1 retrieved from the environment.This contrasts with static tasks whose execution plans are fully determined by the user prompt.
  • Benchmark design: The benchmark includes context-aware attacks whose malicious commands are coupled with environmental feedback, such as file content.The design targets settings where runtime context is necessary for subsequent actions.
  • Attack coverage: AGENTPI identifies 5 context-dependent task types and 5 corresponding attacks covering control flow, logic flow, and authority flow.The attacks affect both control and data panels.
  • Control flow: Action switching targets tool-selection integrity, while parameter manipulation targets data extraction and filling.These categories address distinct control-flow and data-flow failures.
  • Logic flow: Branch divergence manipulates conditional execution, and reasoning corruption targets operations requiring cognitive calculation.Both attacks use manipulated observations or reasoning to redirect execution.
  • Authority flow: Delegation exploitation abuses user-authorized external context to execute commands beyond the implicit safety boundary of the original intent.The benchmark distinguishes benign instruction-following from malicious exploitation of the user’s trust chain.

7 Evaluation

The evaluation measures security, utility, and computational cost across text-level and execution-level defenses. Execution-level defenses can block explicit action violations, but context-aware attacks expose reasoning-integrity failures and a sharp security–utility–latency trade-off.

  • Evaluation setup: The evaluation covers nine defenses across text-level and execution-level categories, using GPT-4o-mini and metrics for attack success, utility, time, and tokens.The reported evaluation presentation uses ASR across five attack vectors, utility in no-attack cases, and computational costs summarized by time and token measures.
  • Explicit action attacks: Execution-level defenses such as Progent and Melon reduce action-switching and delegation-exploitation ASR to near zero.These attacks explicitly trigger unauthorized tools or override authority grants, making them amenable to strict policy checks and tool-call whitelisting.
  • Context-aware attacks: Context-aware attacks remain difficult: parameter manipulation, branch divergence, and reasoning corruption retain high ASR even against Progent and Melon.Policy filters validate whether an action is allowed but generally do not verify whether the reasoning that produced it remains intact.
  • Security–utility trade-off: Lower ASR can sharply reduce utility, as Progent reaches below 0.3 utility while text-level defenses preserve baseline-like utility but provide negligible security gains.The result is attributed to coarse-grained refusal strategies that block legitimate context-dependent instructions resembling adversarial patterns.
  • Computational cost: Some defenses show low latency through early refusal rather than efficiency, whereas heavyweight methods incur substantial overhead, including 310.78% peak latency for Task Shield.Task Shield and Melon rely on auxiliary LLM supervision and serial inference, while Progent and Tool Filter can reduce time overhead by preemptively blocking suspicious requests.
  • Computational cost: The cost analysis reports absolute baseline values and defense costs relative to baseline for time, input tokens, and output tokens.This comparison is presented in Table 5.
  • Computational cost: Text-level defenses impose roughly 100% overhead but limited security efficacy, while robust defenses often exchange security for latency or aggressive service denial.Early termination correlates with low utility scores, showing why lower measured latency may reflect refusal rather than efficient execution.

8 Conclusion

The SoK systematizes prompt-injection attacks and defenses, identifies context-dependent reasoning as a critical gap, and introduces AGENTPI to evaluate execution integrity under context-aware attacks. Its evaluation finds that current defenses often fail to preserve reasoning integrity and struggle to jointly achieve high trustworthiness, utility, and low latency.

  • Conclusion: The SoK categorizes attacks by payload generation and defenses by intervention stage across text, model, and execution levels.It presents a unified systematization of the prompt-injection landscape.
  • Conclusion: Existing approaches largely overlook context-dependent tasks requiring agent actions to adapt dynamically to environmental observations.The paper identifies this as a critical gap in current paradigms.
  • Conclusion: AGENTPI is introduced as a benchmark for assessing agent execution integrity under context-aware attacks.The benchmark addresses the identified gap in evaluating realistic context-dependent behavior.
  • Conclusion: Current defenses often fail to preserve reasoning integrity and cannot reliably distinguish legitimate context-driven behavior from malicious logic manipulation.The conclusion frames this as a central empirical finding of the evaluation.
  • Conclusion: Existing defenses struggle to achieve high trustworthiness, high utility, and low latency simultaneously.The paper connects this trade-off to the need for more robust architectural solutions.

A Discussion

The discussion identifies unresolved tensions in prompt-injection defense, including the trade-off among security, utility, and latency, insecure user prompts, fragmented multi-agent intent, and uncertain semantic boundaries.

  • D1: No current approach simultaneously provides high security, high utility, and low latency.Human authorization improves security and explainability but introduces decision latency; isolation and related defenses can reduce utility for context-dependent tasks.
  • D2: User-written mis-authorization and other insecure prompts can weaken defense effectiveness by granting untrusted resources excessive authority.
  • D3: Multi-agent systems lack reliable decision boundaries for attacks whose intent unfolds across multiple agents and split contexts.Current defenses largely emphasize static, single-turn detection rather than tracking intent evolution across fragmented inputs.
  • D4: Prompt-injection classification remains ambiguous when malicious influence resembles biased but seemingly benign input.The discussion specifically questions how to distinguish implicit semantic manipulation from general data influence.
  • D5: Context-dependent attacks expose limits in defining prompt injection solely as data-to-control privilege escalation.An attacker may corrupt context-based decisions without explicitly hijacking control flow or escalating privileges.

B Future Direction

The proposed future directions target architectural separation, resource-aware monitoring, dynamic trust boundaries, reasoning verification, and risk-sensitive human involvement.

  • FD1: Fine-grained attention masking could enforce a firewall between trusted instructions and untrusted observations.The proposal targets active prevention of privilege escalation by selectively masking attention heads.
  • FD2: Resource-aware execution monitors should detect availability attacks through consumption rates and cyclical tool invocation patterns.The proposed monitors operate independently of semantic analysis to terminate cascading infinite loops.
  • FD3: Dynamic trust boundaries should permit authorized observations to fill parameters while blocking their influence on branching logic.This direction addresses utility loss from strict isolation in context-dependent tasks.
  • FD4: Lightweight auxiliary verifiers should audit whether environmental observations logically entail the agent’s reasoning.The proposal shifts execution-level validation beyond final actions toward reasoning integrity.
  • FD5: Risk-quantified hybrid arbitration could reserve human authorization for high-stakes, low-confidence transitions.Uncertainty estimation or anomaly detection is proposed to reduce human workload while preserving rigorous oversight.

C Benchmark Details

AGENTPI is presented as a benchmark for evaluating context-dependent agent security across diverse attack cases and domains.

  • Benchmark scope: AGENTPI organizes its benchmark details around five context-aware attack cases: action switching, parameter manipulation, branch divergence, reasoning corruption, and delegation exploitation.
  • Benchmark scope: The benchmark is documented with statistics and descriptions for four evaluation domains.

C.1 Benchmark Data Statistics

AGENTPI evaluates context-dependent reasoning through a diverse benchmark of structured environmental observations, tools, domains, and five attack vectors.

  • Benchmark composition: 200 evaluation samples cover 5 attack vectors across 4 domains, with 10 unique samples per combination.
  • Benchmark composition: The benchmark spans action switching, parameter manipulation, branch divergence, reasoning corruption, and delegation exploitation across Banking, Travel, Workspace, and Slack.
  • Domain heterogeneity: AGENTPI simulates 66 unique tools across four representative domains to approximate heterogeneous real-world agent ecosystems.
  • Benchmark design: AGENTPI is designed for context-dependent tasks in which agents must parse environmental feedback to determine control flow.
  • Context complexity: Tool observations average approximately 280 tokens, requiring attention over substantial structured data such as JSON or logs.
  • Action switching: Action switching directly overrides the user’s intent by redirecting tool selection from a benign transfer to an unauthorized payment function.
  • Parameter manipulation: Parameter manipulation preserves the intended tool while falsifying transaction context to induce a malicious amount parameter.
  • Branch divergence: Branch divergence exploits conflicting observations so that a legitimate conditional transfer appears justified despite fabricated context.
Loading 2602.10453v1…