Source-linked AI summary
Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework
Zimo Ji, Daoyuan Wu, Wenyuan Jiang, Pingchuan Ma, Zongjie Li, Yudong Gao, Shuai Wang, Yingjiu Li
TL;DR
LLM-based agents remain vulnerable to natural-language attacks that induce actions beyond the least privilege required for a user’s task. This paper formalizes privilege escalation, identifies attacks including multi-agent confused deputy scenarios, and introduces SEAgent, an ABAC-based MAC framework that enforces policies over execution flows. SEAgent blocks benchmarked attacks with 0% ASR while retaining low false-positive rates, minimal overhead, and strong utility in multi-agent settings.
Problem
Natural-language attacks can exploit over-privileged tool use, while existing defenses can introduce attack surfaces and be bypassed; a unified security definition is needed for complex agent architectures.
Method
SEAgent uses ABAC-based mandatory access control to enforce fine-grained policies over agent-system information-flow paths, with context tracing for multi-round and multi-agent interactions.
Results
0% attack success rate (ASR) was achieved against all benchmarked attack types, while SEAgent matched the naive agent’s task success rate with low false-positive rates and minimal degradation.
Takeaways & Limitations
SEAgent provides strong protection against known and newly demonstrated privilege escalation while maintaining low overhead and supporting multi-agent systems.
Takeaways & Limitations
SEAgent’s hybrid policy and label generation still uses LLM automation with human verification, and static subject attributes may require more automation as systems evolve.
Abstract
from arXiv · showhide
Large Language Model (LLM)-based agent systems are increasingly deployed for complex real-world tasks but remain vulnerable to natural language-based attacks that exploit over-privileged tool use. This paper aims to understand and mitigate such attacks through the lens of privilege escalation, defined as agent actions exceeding the least privilege required for a user's intended task. Based on a formal model of LLM agent systems, we identify novel privilege escalation scenarios, particularly in multi-agent systems, including a variant akin to the classic confused deputy problem. To defend against both known and newly demonstrated privilege escalation, we propose SEAgent, a mandatory access control (MAC) framework built upon attribute-based access control (ABAC). SEAgent monitors agent-tool interactions via an information flow graph and enforces customizable security policies based on entity attributes. Our evaluations show that SEAgent effectively blocks various privilege escalation while maintaining a low false positive rate and negligible system overhead. This demonstrates its robustness and adaptability in securing LLM-based agent systems.
1 Introduction
LLM-based agents expand real-world capabilities through tool use and multi-agent interaction, but natural-language attacks can exploit these systems to cause over-privileged actions. The paper unifies these threats as privilege escalation and proposes SEAgent, a policy-driven MAC defense evaluated across protection and utility benchmarks.
- Natural-language attacks can hijack tool execution or corrupt agent memory, potentially causing privacy violations or physical harm.
- Existing detection-, model-, and system-level defenses often rely on probabilistic components that introduce attack surfaces and can be bypassed.
- Privilege escalation is defined as agent action beyond the minimal actions required to fulfill the user’s query.
- The threat analysis identifies five attack vectors: direct and indirect prompt injection, RAG poisoning, untrusted agents, and confused deputy attacks in multi-agent systems.
- SEAgent applies ABAC-based MAC policies to information-flow paths, labels agents, tools, and RAG databases, and traces context across multi-round and multi-agent execution.
- 0% attack success rate (ASR) was achieved across benchmarked attack types, while utility evaluation covered single-agent single-round and multi-agent multi-round scenarios.
2 Background
The background situates agent privilege escalation within established computing-system vulnerabilities and describes typical LLM-agent architectures. It connects traditional confused-deputy and collusion patterns with modern agents that combine planning, tools, retrieval, and memory.
- Traditional privilege escalation includes horizontal access to another user’s resources and vertical access to root privileges.
- In Android, confused deputy attacks use a higher-privilege app to perform unauthorized actions, while colluding apps combine permissions neither possesses alone.
- Traditional mitigations include Unix-like isolation mechanisms and policy-based access control models for Android.
- LLM agents are deployed in real-world applications, including systems that coordinate multiple agents or control graphical user interfaces.
- A typical LLM agent contains understand, plan, and act modules, with RAG retrieving contextual information from a database and memory functioning as a specialized RAG database.
3 Preliminaries
The formal model represents both single-agent and multi-agent systems through sets of agents, tools, users, databases, response actions, and invocation actions. Execution proceeds through state transitions in which agents generate tool or inter-agent invocations until no new invocation remains.
- The unified framework covers single-agent and multi-agent designs using core sets for agents, tools, users, databases, response actions, and invocation actions.
- Response actions include user queries, tool results, and retrieved database content, while invocation actions include agent-tool calls and, in MAS, inter-agent messages.
- Each agent maps received response actions to a set of invocation actions and a natural-language response.
- The system state tracks every agent’s current context, current invocation actions, and natural-language responses.
- A state transition updates context from response actions and computes new agent responses and invocations from the current state.
- One execution round is a sequence of state transitions that typically starts with a user query and ends when no new invocation is generated.
- This model reflects ReAct-style agents that autonomously invoke tools in response to intermediate outputs until reaching a final answer.
4 Privilege Escalation in Agent
Privilege escalation occurs when an agent performs an invocation outside the minimal actions required by the user's query, including tool calls and inter-agent messages. Case studies show that existing defenses remain vulnerable, while multi-agent communication enables confused deputy attacks in which untrusted agents induce trusted agents to perform privileged actions.
- Definition and assumptions: Privilege escalation is any invocation action outside the minimal set required to fulfill the user's query.The action set includes tool invocations, inter-agent messages, and their arguments.
- Definition and assumptions: The threat model treats single-agent systems as trusted but permits untrusted system prompts in third-party multi-agent components.The adversary may know the architecture and toolset and manipulate system inputs without directly modifying runtime execution.
- Attacking single-agent systems: Existing defenses remain vulnerable to privilege escalation across prompt injection and RAG poisoning scenarios.The study reports attacks against SecAlign and IsolateGPT, including malicious instructions and poisoned retrieved content.
- Attacking multi-agent systems: In AIOS-AutoGen, an untrusted search agent broadcasts a crafted unlock request, causing the trusted smart lock agent to invoke UnlockDoor.The search agent lacks direct access to UnlockDoor but manipulates the trusted agent through broadcast communication.
- Attacking multi-agent systems: The resulting confused deputy attack lets an untrusted peer exploit a benign agent to perform privilege escalation, and indirect prompt injection can trigger the same pattern.Proof-of-concept attacks reproduced this susceptibility in standard AutoGen and AIOS-MetaGPT.
5 SEAgent
SEAgent models agent-system execution as information flows and enforces attribute-based mandatory access policies over those flows. It combines per-round monitoring, policy checks, subject labeling, and memory controls to block privilege escalation while managing flexibility and false positives.
- Policy Enforcement: The Decision Engine matches System View subgraphs against Policy DB rules and applies actions such as blocking calls or raising warnings.The framework monitors execution in real time and evaluates newly detected tool invocations against stored policies.
- System View: SEAgent maintains a directed System View graph of users, agents, tools, and RAG databases, with edges representing interactions or information transfers.The graph is initialized each round and updated as tool calls, tool results, database retrievals, and inter-agent messages occur.
- Policy Enforcement: Figure 4’s SMS example blocks a path from reading an SMS through agents to the high-impact Uninstall_App tool.The suspicious path is identified as Read_SMS→SMS Agent→Setting Agent→Uninstall_App and is denied by policy settings.
- SEMemory: Per-round System View resets improve efficiency and reduce false positives, but clearing the graph while retaining agent context can cause false negatives.A malicious instruction retained in context may influence a later tool call or query without appearing in the reset graph.
- Subject Labeling: ABAC labels tools, agents, and RAG databases with security-relevant attributes to balance fine-grained control against policy rigidity and false positives.Tool policies use operation and result attributes, while agent labels distinguish verified and unverified origins.
- SEMemory: SEMemory standardizes context and memory management, while immutable extractive retrieval keeps prior events under System View surveillance.This design treats retrieved context as reintroduced prior events and is intended to prevent SEMemory from introducing new attack surfaces.
6 Implementation
SEAgent’s implementation configures policies for four attack vectors and addresses incomplete subject metadata through hybrid labeling. Its policy database uses information-flow rules to block risky actions while supporting scenario-specific predicates.
- Policy Coverage: SEAgent focuses policy implementation on indirect prompt injection, RAG poisoning, confused deputy, and untrusted-agent attacks.
- Policy Implementation: Its indirect-prompt-injection policies block risky write or execution actions triggered by unfiltered external information and prevent sensitive personal-data exfiltration through email.The policies deny flows involving moderate- or high-sensitivity operations and personal data sent through the email tool.
- Policy Implementation: The RAG-poisoning policy blocks invocations of non-low-sensitivity tools triggered by untrusted retrieved content.
- Policy Implementation: The confused-deputy and untrusted-agent policy denies untrusted-agent paths to tools above LOW sensitivity, including indirect paths matched by wildcards.
- Subject Labeling: Hybrid labeling combines initial LLM-based labeling with human verification and correction to handle benchmarks lacking detailed subject metadata.Experiments on InjecAgent indicate that this approach reduces human workload while maintaining high accuracy.
7 Evaluation
The evaluation tests SEAgent’s security, task performance, false positives, runtime, and token use across single-agent and multi-agent settings. It reports complete attack blocking, preserved or improved utility, and low overhead, while noting benchmark-specific trade-offs.
- RQ1: Security Protection Analysis: SEAgent achieves 0% attack success rate across all attacks in InjecAgent and AgentDojo.The protection analysis covers indirect prompt injection, RAG poisoning, untrusted agents, and confused deputy attacks.
- RQ1: Security Protection Analysis: SEAgent blocks the confused-deputy scenario by denying UnlockDoor after detecting an untrusted-agent information-flow path.The Decision Engine triggers the policy and issues a targeted warning to the user.
- RQ2: Utility and False Positives: SEAgent achieves comparable or higher correctness than the naive agent on one-tool and three-tool-or-more reconstructed API-Bank tasks, while IsolateGPT drops over 20% in multi-tool tasks.IsolateGPT’s false-positive rate reaches 18.31% in two-tool scenarios.
- RQ2: Utility and False Positives: SEAgent records only two false positives in reconstructed API-Bank, both involving the Wikipedia tool labeled UNFILTERED.Labeling Wikipedia as TRUSTED in deployment would prevent these cases.
- RQ3: Runtime and Token Consumption: SEAgent’s execution time is close to or faster than the naive agent in selected single-agent task categories, whereas IsolateGPT more than doubles execution time.With SEMemory enabled, short-context API-Bank evaluation incurs higher token and execution-time overhead, while longer contexts reduce this burden.
- RQ3: Runtime and Token Consumption: In AWS multi-agent scenarios, SEAgent achieves better or comparable goal success rates, reduces token usage, and keeps execution time on par with P2PEnv.Clearing context requires a user-query increase within 20%, while SEMemory compresses relevant history across rounds.
- RQ3: Runtime and Token Consumption: Policy checks add negligible delay, averaging 0.00586s in travel and 0.00307s in mortgage.SEMemory accounts for most internal token usage but contributes minimally to execution time.
8 Discussion
SEAgent’s current labeling approach combines LLM automation with human verification, while its static attributes provide deterministic and auditable enforcement. The discussion identifies dynamic attribute inference as a future improvement for adaptability and precision.
- Policy and Label Generation: SEAgent combines LLM-based automation with human verification for scalable and accurate subject labeling.The approach covers diverse agent behaviors, interactions, execution contexts, and attack vectors.
- Static Subject Attributes: Static security attributes cannot be updated dynamically at runtime, limiting adaptability as agent systems and attack vectors evolve.The paper proposes runtime inference based on execution history as a future direction.
- Static Subject Attributes: Static attributes support deterministic and auditable policy enforcement for high-assurance and safety-critical applications.The authors contrast this predictability with the adaptability that dynamic inference could provide.
9 Related Work
Related work includes black-box and white-box attacks on LLM-based agents, alongside defenses that target detection, model behavior, or system security. Base LLM jailbreak defenses provide an additional line of related research.
- Attacking LLM-based agent systems: Agent Security Bench, RAG-Thief, and ChatInject study black-box natural-language attacks, while AgentPoison, Breaking Agents, Imprompter, and Zhang et al. study white-box attacks.These attack methods typically achieve high success rates, motivating effective defense mechanisms.
- Securing LLM-based agent systems: Jailbreak defenses such as SelfDefend, RAIN, Eraser, CAT, and LED aim to prevent LLMs from generating harmful content.These systems provide insights for defending LLM-based agents.
10 Conclusion
The paper introduces privilege escalation attacks in agent systems and proposes SEAgent to mitigate them. Its decision engine evaluates information-flow paths against policies and returns enforcement actions, defaulting to Allow when no rule triggers.
- 10 Conclusion: The paper defines privilege escalation attacks in agent systems and demonstrates their prevalence and severity through case studies.SEAgent is evaluated across diverse scenarios for detecting and preventing these attacks.
- Security Attributes: SEAgent’s action vocabulary distinguishes READ, WRITE, and EXECUTE operations, while sensitivity ranges from LOW to HIGH and privacy distinguishes GENERAL from PERSONAL data.These attributes support policy decisions over tool operations and outputs.
- Security Attributes: Integrity classification includes TRUSTED outputs that have been filtered or verified and UNFILTERED outputs that may contain prompt injections or phishing content.The policy language also distinguishes LOCAL, EXTERNAL, and PHYSICAL tool operations.
- B Syntax of Policy Language: SEAgent’s policy language uses a context-free grammar to formalize security-policy specifications.The grammar defines paths through agent, tool, and database nodes with variable or wildcard specifications.
- Decision Engine: The Decision Engine parses and prioritizes policies by specificity, matches candidate graph paths, evaluates Boolean rules, and returns the corresponding enforcement goal.More specific policies are applied before general policies, implementing a first-match principle.
- Decision Engine: The engine outputs an enforcement action from Allow, Deny, or Ask based on the current System View and Policy DB.If no policy rule evaluates to true, the default action is Allow.
D Robustness of SEMemory
SEMemory maintains agent context through an entity dictionary and Memory LLM, and its analyzed attack surfaces reduce to established privilege-escalation vectors. Per-user separation of context, System View, and memory provides non-interference across users.
- SEMemory Components: SEMemory consists of an entity dictionary and a Memory LLM that selects relevant dictionary entries for each query.The selected keys help initialize agent context and reconstruct the System View.
- SEMemory Attack Surfaces: Attackers may poison the entity dictionary or manipulate the Memory LLM’s output, creating two principal SEMemory attack surfaces.The threat model includes control over trusted-agent responses and untrusted-agent invocations.
- Attack Reduction: Entity dictionary poisoning reduces to direct prompt injection, indirect prompt injection, or RAG poisoning depending on the source of the malicious entry.Entries originating from users, tools, or RAG databases map to the corresponding established attack vector.
- Attack Reduction: All SEMemory-related privilege-escalation attacks can be mapped to existing attack vectors, introducing no new attack surfaces under the stated threat model.The paper identifies direct and indirect prompt injection, RAG poisoning, untrusted agents, and confused deputy attacks as the relevant vectors.
- E User-Level Isolation: SEAgent maintains independent context, invocation actions, tool responses, System Views, and SEMemory for each user.Context, System View, and memory are compartmentalized throughout the full execution lifecycle.
- E User-Level Isolation: Per-user isolation ensures that one user’s actions or data cannot affect another user’s execution state, supporting non-interference in multi-user deployments.The design is intended to eliminate privilege escalation risks from cross-user context contamination or prompt injection.
F Hybrid Labeling Method Evaluation
SEAgent uses LLM-generated tool labels followed by human review to balance labeling accuracy and workload. On 80 InjecAgent tools, LLM and human annotations showed high overall agreement, but critical sensitivity and privacy discrepancies remained.
- LLM-based automatic labeling followed by human review balances labeling accuracy and human workload.The paper adopts this hybrid strategy after finding that most LLM-generated labels can be used directly with minor corrections.
- 80 tools were independently labeled by an LLM and human annotators across SEAgent’s five core attributes.Annotators received each tool’s functional description, input parameters, and output specifications.
- Cohen’s κ = 0.9456 indicates high overall agreement between LLM and human labels.
- Sensitivity and Privacy produced notable disagreements, including a phishing-capable browser tool rated MODERATE by humans but LOW by the LLM.The paper states that such misclassification may cause false negatives in policy enforcement and compromise user safety.
- The labeling prompt assigns object, action, sensitivity, privacy, and integrality attributes from structured tool descriptions.Examples distinguish READ, WRITE, and EXECUTE actions and illustrate labels for note-reading and camera-control tools.
H Details of the AWS Benchmark
The AWS evaluation uses travel-planning and mortgage-financing multi-agent scenarios, adapting the benchmark to allow arbitrary peer-to-peer communication. Because toolkit descriptions omit target, data-source, and implementation details, the evaluation excludes specific tool labeling.
- The AWS benchmark covers travel planning, mortgage financing, and software development, with thirty test instances per scenario.Each instance includes a scenario description, an initial user query, and multiple assertions.
- The benchmark is generalized from supervisor-mediated tree communication to a fully connected P2P network.This removes the centralized supervisor bottleneck and evaluates security under arbitrary communication flows.
- The evaluation prioritizes travel and mortgage scenarios because software development uses hard-coded communication constraints incompatible with the open-ended P2P protocol.These two retained domains provide diversity in tool complexity and agent coordination patterns.
- The final dataset contains 9 travel-planning agents and 5 mortgage-financing agents, but excludes specific tool details because descriptions lack essential implementation information.
- LLMs simulate tool responses and users, while an assertion-judge LLM evaluates whether scenario assertions are satisfied from interaction histories and tool invocations.The user simulator can communicate freely with any agent in the adapted evaluation.
- The evaluated toolkit includes high-sensitivity actions such as granting guest access, unlocking doors, paying bills, transferring funds, and placing Binance orders.
J Tool Labels of AgentDojo Benchmark
The AgentDojo benchmark provides the tool inventory and corresponding security labels used to evaluate SEAgent’s protection capabilities. The listed tools span external read, write, and execute operations with varying sensitivity, integrity, and privacy attributes.
- AgentDojo tools and their corresponding labels are listed in Table 10 for SEAgent security evaluation.
- The inventory includes external tools such as email deletion, webpage retrieval, email search, channel messaging, and user addition.
- AgentDojo labels distinguish LOW, MODERATE, and HIGH sensitivity alongside READ, WRITE, and EXECUTE actions.The listed labels also encode object type, integrality, and privacy.