Source-linked AI summary
A Framework for Formalizing LLM Agent Security
Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, Neil Gong, Chenguang Wang, Dawn Song
TL;DR
LLM-agent security is contextual, but existing definitions and defenses often miss the authorization context behind inputs and actions. The paper formalizes this context with four security properties and oracle functions, then uses them to redefine attacks and clarify defense requirements. Its framework makes contextual authorization requirements explicit while leaving practical oracle implementation and scope boundaries as important limitations.
Problem
Existing LLM-agent security definitions often ignore instruction sources, authorized objectives, action purpose, and permitted information flows, creating a utility-security trade-off for defenses.
Method
The paper formalizes execution context with four security properties and oracle functions that specify the information needed to verify those properties during agent execution.
Results
The framework reformulates attacks as contextual violations and makes implicit requirements of existing defenses explicit, enabling precise vulnerability classification.
Takeaways & Limitations
Agent security requires verification of authorization context beyond content filtering, supported by improved approximations of the framework’s oracle functions.
Takeaways & Limitations
Practical oracle implementations remain challenging because attribution, information-flow tracking, and reliable semantic judgments require tractable approximations.
Abstract
from arXiv · showhide
Security in LLM agents is inherently contextual. For example, the same action taken by an agent may represent legitimate behavior or a security violation depending on whose instruction led to the action, what objective is being pursued, and whether the action serves that objective. However, existing definitions of security attacks against LLM agents often fail to capture this contextual nature. As a result, defenses face a fundamental utility-security tradeoff: applying defenses uniformly across all contexts can lead to significant utility loss, while applying defenses in insufficient or inappropriate contexts can result in security vulnerabilities. In this work, we present a framework that systematizes existing attacks and defenses from the perspective of contextual security. To this end, we propose four security properties that capture contextual security for LLM agents: task alignment (pursuing authorized objectives), action alignment (individual actions serving those objectives), source authorization (executing commands from authenticated sources), and data isolation (ensuring information flows respect privilege boundaries). We further introduce a set of oracle functions that enable verification of whether these security properties are violated as an agent executes a user task. Using this framework, we reformalize existing attacks, such as indirect prompt injection, direct prompt injection, jailbreak, task drift, and memory poisoning, as violations of one or more security properties, thereby providing precise and contextual definitions of these attacks. Similarly, we reformalize defenses as mechanisms that strengthen oracle functions or perform security property checks. Finally, we discuss several important future research directions enabled by our framework.
1 Introduction
The paper argues that LLM-agent security depends on execution context, which existing attack definitions and defenses often neglect. It introduces contextual security properties and oracle functions to formalize attacks and defenses more precisely.
- Existing attack definitions often overlook who issued an instruction, which objective is authorized, whether actions advance it, and which information flows are permitted.
- Context-agnostic defenses create a utility-security trade-off: broad blocking causes false positives, while insufficient filtering leaves agents vulnerable.
- The framework defines task alignment, action alignment, source authorization, and data isolation as contextual security properties.
- Oracle functions specify the information theoretically required to verify security-property violations during task execution.
- The framework reformulates attacks including indirect and direct prompt injection as violations of specific security properties, producing more precise contextual definitions.
- Defenses are characterized as mechanisms that strengthen oracle functions or directly enforce security-property checks, while current approaches remain limited by context blindness and single-property coverage.
2 Background on LLM Agents
LLM agents use language models and tools to observe environments, reason, and act across multi-turn tasks. Their execution state includes prompts, trajectories, memory, environment updates, and source permissions.
- Agent execution: LLM agents combine language models with tool use to act in external environments and complete complex multi-turn tasks.At each step, the agent observes the environment, reasons about the next action, and executes a tool call.
- Running example: A cooking assistant searches recipes, asks the user to choose one, checks pantry inventory, and orders missing ingredients.
- Execution model: The user prompt defines the high-level objective that drives the agent’s planning and action selection.
- Execution model: A trajectory records action-observation pairs over time, including tool calls, messages, and corresponding environment feedback.The trajectory extends from Tr_t−1 to Tr_t after action a_t produces observation obs_t.
- Execution state: The agent’s context aggregates inputs from the user, tools, and other sources while persistent memory and the environment evolve during execution.
- Source permissions: A source permission graph specifies which sources may access or influence one another, including user-specific resource permissions.
3 Threat Model
The threat model covers tool-using agents that receive information from sources with different trust levels and operate across tools with varying privileges. Security risks include confidentiality and integrity violations, persistent memory compromise, and unauthorized actions or objectives.
- Threat surface: Agents interact with external systems through tools while receiving inputs from users, external content, and system sources.
- Security violations: Confidentiality violations include exfiltrating sensitive information or leaking data across privilege boundaries.
- Security violations: Integrity violations include unauthorized commands, out-of-scope objectives, and action sequences inconsistent with legitimate workflows.
- Threats and persistence: Agents may be attacked into deleting production data, redirecting funds, or retaining a persistent compromise through memory.
- System assumptions: The model assumes synchronous turn-based execution, tools with different privilege levels, and memory that persists across interactions.
4 Framework Formalization
The framework models security as contextual authorization: identical actions can be legitimate or violating depending on execution context. It operationalizes this view with four security properties and oracle functions that evaluate them during execution.
- 4.1 Contextual Security: Execution context Ct captures the prompt, prior trajectory, memory, environment state, authenticated sources, and permission graph relevant to authorization.
- 4.1 Contextual Security: Identical actions can have different security status when their execution contexts differ, including differences in authenticated sources or authorized objectives.
- 4.1 Contextual Security: Contextual security requires task alignment, action alignment, source authorization, and data isolation to hold simultaneously.
- 4.2 Oracle Functions: The framework uses five oracle functions to identify action-causing inputs, trace their sources, and evaluate prompt, trajectory, and action objectives.The objective functions Hp, HTr, and Ha assess the authorized objective, trajectory alignment, and action alignment, respectively.
- 4.3.1 Task Alignment: Task alignment requires a safe initial objective within O and a trajectory that remains consistent with that objective.HTr can be approximated by formal verification or an LLM judge, but accurate approximations remain an open problem.
- 4.3.1 Task Alignment: Task refinement remains aligned when subtasks serve the original objective, whereas unrelated activity can constitute task drift.The cooking example distinguishes meal-preparation subtasks from browsing cookware and adding an unrelated skillet.
- 4.3.2 Action Alignment: Action alignment checks whether each individual action contributes to the authorized objective, using Ha rather than the trajectory-level HTr.For example, recipe search serves a recipe objective, whereas querying medical records does not.
- 4.3.3 Source Authorization: Source authorization and integrated security distinguish legitimate delegated instructions from violations by considering provenance, objective alignment, and the necessity of each property.An unauthenticated source need not violate security when the resulting action independently satisfies action alignment; each property remains individually necessary.
5 Systematizing Agent Security Violations
The taxonomy defines agent security violations contextually rather than by action content alone. It distinguishes attacks by which authorization property fails, allowing legitimate and malicious uses of similar instructions or actions to be separated.
- Instruction following is not inherently secure because the same action may be legitimate or malicious depending on source, objective, action alignment, and information flow.
- Action-content definitions can misclassify authorized operations as attacks because they omit execution context and permission boundaries.
- 5.1 Indirect Prompt Injection: An external prompt is necessary but insufficient for indirect prompt injection; the attack requires unauthenticated instructions to cause execution of an unauthorized task.
- 5.1 Indirect Prompt Injection: Externally sourced instructions can remain legitimate when the resulting action serves the authorized objective, so content-only detection creates a utility-security trade-off.
- 5.2 Direct Prompt Injection: Direct prompt injection involves authenticated user instructions that conflict with higher-priority system objectives, violating task alignment even when source authorization holds.
- 5.3 Jailbreak: Jailbreaking differs from direct prompt injection because it requests objectives outside the allowed space O, whereas direct injection conflicts with higher-priority instructions within permissible space.
- 5.4 Confused Deputy: Confused deputy attacks occur when the agent uses its own elevated permissions to access resources that the authenticated user cannot access.Formally, the user-resource permission edge is absent while the agent-resource edge exists, violating source authorization.
5.5 Task Drift and Agentic Misalignment
The framework separates several temporal and authorization failures that can produce similar harmful behavior. Task drift concerns trajectory-level objective deviation, while agentic misalignment adds self-generated goals; other attacks target action use, information flow, tools, or memory.
- 5.5 Task Drift and Agentic Misalignment: Task drift occurs when an agent autonomously diverges from its initial objective without authenticated authorization.
- 5.5 Task Drift and Agentic Misalignment: Agentic misalignment is more severe because the agent intentionally pursues self-generated objectives lacking any authenticated source, violating source authorization and task alignment.
- 5.6 Capability Misuse: Capability misuse violates action alignment when an authorized agent performs an unnecessary action despite the overall task remaining aligned.Querying medical records during recipe recommendation illustrates action-level misuse without trajectory-level drift.
- 5.7 Cross-Context Information Leakage: Cross-context leakage violates data isolation when information flows between sources and destinations lacking a permission edge.The same disclosure can be legitimate within one user session but impermissible across users or privilege contexts.
- 5.7 Cross-Context Information Leakage: Figure 3 shows that cross-session credential reuse can violate memory constraints while remaining invisible to benchmarks that reset context between tasks.Detecting these violations requires tracking information flow across the full trajectory.
- 5.7 Cross-Context Information Leakage: A coding agent can violate data isolation by carrying an API key from test-code review into production deployment or emailing prior conversations externally.
- 5.8 Malicious Tool Exploitation: Malicious tool exploitation corrupts observations or causes unauthorized side effects, ultimately producing task-alignment violations through a divergent trajectory.It differs from indirect prompt injection because the attacker controls tool execution rather than injecting instructions into observations.
- 5.9 Memory Poisoning: Memory poisoning inserts malicious information at one time and later corrupts authorization decisions, such as treating an absent user-resource permission as present.
6 Analysis of Existing Defenses
Existing defenses can be organized by whether they approximate oracle functions or directly check security properties, but systematic gaps leave contextual authorization unresolved. Empirical evidence suggests action-alignment defenses can substantially reduce attack success, often with utility costs.
- Defenses are organized into prevention mechanisms that strengthen oracle approximations and detection mechanisms that check security properties.Existing defenses implicitly approximate oracle functions but lack a systematic account of contextual authorization.
- Prevention defenses: Prompt engineering improves task-boundary and source-prioritization representations but remains vulnerable because it does not solve causal instruction attribution.
- Prevention defenses: Training-based defenses improve approximations of instruction and source attribution, but defensive fine-tuning can degrade capabilities without providing robustness.
- Prevention defenses: Architectural privilege separation restricts access through isolation or sandboxing, yet overly restrictive separation reduces utility and insufficient separation fails to contain attacks.
- Detection defenses: Input filtering detects instruction-like patterns but lacks causal attribution, while task-drift and tool-use defenses approximate task and action alignment.
- Detection defenses: 86% and 88% relative reductions in targeted ASR came from the PI detector and tool filter, whereas delimiting and repeat-prompt defenses reduced ASR by at most 28%.The PI detector’s benign utility dropped from 69.0% to 41.5%.
- Multi-property defenses: Sandboxing provides coarse-grained access guarantees but cannot prevent violations within permitted capabilities, while runtime monitoring evaluates behavior across multiple properties.
7 Open Problems and Future Directions
Future work centers on implementing tractable oracle approximations and evaluating security across compositional, dynamic, temporal, and realistic authorization contexts. These challenges reflect gaps in current property checks and benchmarks.
- Oracle function implementation: Practical oracle implementations must balance accuracy, efficiency, and deployability across instruction attribution, source attribution, and objective-evaluation functions.Current approaches provide coarse attribution or inconsistent semantic judgments, especially under adversarial inputs.
- Oracle function implementation: Reasoning traces and hybrid learned-symbolic objective representations are proposed as directions for improving instruction attribution and goal evaluation.
- Compositional security: Compositional violations arise when individually benign actions combine unsafely, requiring reasoning about environment-state changes across full action sequences.Current action-alignment checks evaluate each action in trajectory context but do not model accumulated state changes sufficiently.
- Dynamic environments: Dynamic environments require accounting for state changes made concurrently by other agents, users, or processes, which current approaches do not handle.
- Temporal security: Temporal security introduces race conditions in which properties hold during verification but fail during execution, including TOCTOU attacks and cross-session memory poisoning.Long-horizon trajectories make it increasingly difficult to track which information influenced decisions.
- Realistic benchmarks with authorization context: Current benchmarks isolate individual interactions and omit authenticated sources, authorized objectives, and permitted information flows needed for property evaluation.
- Realistic benchmarks with authorization context: Future benchmarks should evaluate task sequences, adaptive attacks, and utility-security tradeoffs within explicit authorization contexts.
8 Discussion and Limitations
The framework makes contextual authorization requirements explicit through oracle functions and security properties, while acknowledging important scope and completeness limitations. Its practical guarantees depend on imperfect approximations and settings that the current formulation does not fully cover.
- Formalizing security through oracle functions and properties clarifies requirements for improving approximations and identifying unaddressed challenges.
- Operational definition of instructions: The operational definition of an instruction treats an input as instructional when it caused an action, but requires causal attribution and overlooks continuous influence.
- Scope and applicability: The framework targets synchronous deployed agents and confidentiality or integrity violations, excluding training-time attacks and treating availability failures as distinct.
- Oracle approximation quality: Practical security depends on imperfect, context-sensitive oracle approximations whose accuracy may degrade under adversarial inputs.The required approximation quality for practical guarantees remains an open question.
- Multi-agent and delegated authority: The framework focuses on one agent acting for one authenticated user and does not fully specify multi-agent coordination, delegation, or multiple principals.
- Compositional safety and framework completeness: The four properties do not fully formalize compositional safety or claim formal completeness; they systematize documented attack classes instead.
9 Conclusion
The framework formalizes agent security through four authorization dimensions and oracle functions, exposing limitations in existing defenses and motivating contextual verification research.
- The framework decomposes agent security into task alignment, action alignment, source authorization, and data isolation.
- Oracle functions define the information theoretically required for contextual authorization and support precise vulnerability classification across superficially different attacks.
- Pattern-based defenses cannot distinguish context-dependent authorization, while static evaluation misses temporal violations across sessions.
- Future work targets oracle-function implementations, delegation mechanisms for multi-agent coordination, and benchmarks covering extended interactions.
Ethical Considerations
The paper presents a defensive systematization of agent-security research while acknowledging dual-use concerns and emphasizing its non-operational scope.
- The systematization synthesizes existing research on agent-security risks, threat models, and defense strategies.
- The work analyzes documented attack classes at an abstract level using synthetic, public, or high-level examples.
- Clearer taxonomies and threat models could benefit attackers as well as defenders, but the paper omits proof-of-concept code, attack playbooks, and novel exploitation techniques.
- The framework is intended to improve defenses by identifying verification gaps, clarifying pattern-based limitations, and providing vocabulary for contextual authorization.
Open Science
The paper is an openly assessable conceptual systematization based on public sources, without experiments, collected data, or software artifacts.
- The work synthesizes and analyzes existing research on agent security as a systematization of knowledge.
- The study conducted no experiments, collected no data, and developed no software artifacts.
- Its analysis draws on publicly available academic literature, industry documentation, and security advisories.
- Readers can evaluate the classifications and defense analysis using the cited public sources and the paper’s text and diagrams.