Source-linked AI summary
Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents
Juhee Kim, Woohyuk Choi, Byoungyoung Lee
TL;DR
LLM agents’ runtime interpretation of user and tool data creates privilege-escalation risks because untrusted inputs can influence privileged operations. PFI combines agent isolation, secure untrusted-data processing, and deterministic guardrails, achieving higher secure utility and zero attacked tasks in the reported benchmarks.
Problem
Untrusted tool data can influence LLM-agent behavior and privileged operations, while probabilistic data flow makes prompt and data separation difficult to enforce.
Method
PFI isolates trusted and untrusted agents, represents untrusted data with data IDs, and uses deterministic data-flow and control-flow guardrails.
Results
PFI improved Secure Utility Rate from 27.84% to 55.67% on AgentDojo and from 2.63% to 67.79% on AgentBench OS, while Attacked Task Rate fell to zero on both.
Takeaways & Limitations
PFI provides robust protection against attacks while preserving LLM-agent utility, improving SUR by 28-63%p compared with ReAct.
Takeaways & Limitations
Prompt transformation introduces a security trade-off by merging the separated responsibilities of the trusted and untrusted agents, so PFI permits it only under specified conditions.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are combined with tools to create powerful LLM agents that provide a wide range of services. Unlike traditional software, LLM agent's behavior is determined at runtime by natural language prompts from either user or tool's data. This flexibility enables a new computing paradigm with unlimited capabilities and programmability, but also introduces new security risks, vulnerable to privilege escalation attacks. Moreover, user prompts are prone to be interpreted in an insecure way by LLM agents, creating non-deterministic behaviors that can be exploited by attackers. To address these security risks, we propose Prompt Flow Integrity (PFI), a system security-oriented solution to prevent privilege escalation in LLM agents. Analyzing the architectural characteristics of LLM agents, PFI features three mitigation techniques -- i.e., agent isolation, secure untrusted data processing, and privilege escalation guardrails. Our evaluation result shows that PFI effectively mitigates privilege escalation attacks while successfully preserving the utility of LLM agents.
1 Introduction
LLM agents combine broad tool-enabled capabilities with substantial privilege-escalation risks because untrusted data can influence runtime decisions. PFI addresses these risks through isolation, secure untrusted-data processing, and guardrails, improving secure utility while preserving performance.
- Motivation: LLM agents connect language models to tools that provide services such as web search, email, calendars, cloud storage, and file access.These tools may access sensitive data or perform critical operations on users’ behalf.
- Motivation: Untrusted tool data can be interpreted as instructions or influence privileged tool usage, enabling attackers to access private data or perform unauthorized operations.The agent’s probabilistic processing makes least-privilege enforcement difficult because the full context can influence the next action.
- PFI: PFI proposes agent isolation, secure untrusted-data processing, and privilege-escalation guardrails as its three core mitigation techniques.The design separates trusted and untrusted processing, represents untrusted data through data IDs, and applies a fine-grained policy framework.
- Evaluation: 27.84% to 55.67% on AgentDojo and 2.63% to 67.79% on AgentBench OS, PFI’s Secure Utility Rate increased relative to ReAct across diverse models.Secure Utility Rate measures agent performance while remaining robust against attacks.
- Evaluation: PFI reduced Attacked Task Rate to zero on both AgentDojo and AgentBench OS.The paper attributes the improvement to PFI’s strong and deterministic security guarantee.
2 LLM Agents
LLM agents combine an LLM, tools, and an agent context to iteratively select tool calls or return answers. Each inference uses the accumulated context, including prompts, tool calls, and tool results.
- Architecture: Tools provide services such as web search, email access, and host-system access, allowing agents to interact with external systems.These services support information retrieval, personal-data management, and system actions.
- Architecture: An LLM agent consists of an LLM, a set of tools, and an agent context containing relevant interaction information.The context includes system and user prompts, tool calls, and tool results.
- Workflow: At each step, the LLM receives the current agent context and decides whether to call a tool or return the final answer.After a tool call, its result is appended to the context and the process repeats until the agent returns an answer.
3 Motivation
LLM agents often operate as a single principal with access to privileged tools while processing attacker-controlled data, creating privilege-escalation risks. The motivation for PFI is to control these risks without eliminating useful interactions with untrusted data.
- Threat Model and Security Risk: Attackers can poison external systems so malicious data enters the agent through tools and influences privileged tool calls or final answers.The attacker’s goal is unauthorized access to private data or execution of actions requiring the user’s permission.
- Threat Model and Security Risk: LLM agents may access privileged tools containing private user data while operating without compartmentalization or least-privilege policies.Current agents can call any tool at any time despite the different privileges associated with those tools.
- Privilege Escalation Attacks: Prompt injection attacks use attacker-provided data as instructions for specific tools or tasks, while data injection attacks exploit the agent’s best-effort behavior without explicit instructions.The paper identifies unsafe data flow and unsafe control flow as two forms of data injection.
- Privilege Escalation Attacks: The root cause of prompt injection is the lack of separation between prompts and data, while probabilistic data flow makes influence difficult to quantify or enforce deterministically.Every input token can contribute to the model’s output, and internal model inspection is often infeasible.
- Previous Defenses: Previous defenses may leave trusted agents exposed to malicious results and rely on trusted planning for unsafe data flows, while some restrict useful untrusted-data influence.PFI instead isolates untrusted data and permits controlled use according to user intent, including trusted endorsement and user approval.
4 Prompt Flow Integrity
PFI secures LLM agents by separating trusted and untrusted processing, representing untrusted data with data IDs, and enforcing deterministic guardrails around privilege escalation. These mechanisms preserve access to necessary computation while restricting untrusted data from privileged operations.
- 4 Prompt Flow Integrity: PFI consists of trusted and untrusted agents plus prompt-flow policies governing data trust and access-token privilege.The trusted agent processes user requests with privileged access, while the untrusted agent handles potentially attacker-controlled data.
- 4.1 Agent Isolation: The trusted agent AT processes only trusted data under privileged token TP, while the untrusted agent AU processes untrusted data under restricted token TU.PFI isolates each agent’s context and limits AU to the minimum tools and resources needed for untrusted-data processing.
- 4.1 Agent Isolation: PFI isolates AU from sensitive trusted data while allowing AT to spawn AU for unrestricted processing of untrusted data.AU receives only the query and necessary trusted formatting information, preventing untrusted processing from exposing AT’s private context.
- 4.1 Agent Isolation: Access tokens enforce least privilege by granting TP full user-authorized access and TU limited access to nonsensitive operations.This token-based design extends access-control mechanisms used by tools and external APIs to the two-agent architecture.
- 4.2 Secure Untrusted Data Processing: Data IDs encode untrusted data into trusted identifiers, allowing AT to reference or decode the data without directly exposing raw content to AT.The trusted Enc and Dec functions connect AT and AU while keeping raw untrusted data within the protected processing flow.
- 4.2 Secure Untrusted Data Processing: Computation offloading lets AU analyze raw untrusted data and return encoded results, combining restricted processing with the LLM’s analytical capabilities.In the example, AU extracts conference date and location from webpage content and returns them as new data IDs for AT.
- 4.2 Secure Untrusted Data Processing: PFI permits prompt transformation only when necessary according to AT’s context and explicitly approved by the user.The transformation improves utility but creates a security trade-off by merging responsibilities otherwise separated between AT and AU.
- 4.3 Privilege Escalation Guardrails: DataGuard and CtrlGuard use deterministic data IDs and prompt queries to detect unsafe data and control flows before privileged operations proceed.Alerts identify the source, sink, and flow type, enabling users to review and deny operations such as sending attacker-controlled content by email.
5 Evaluation
PFI was evaluated across security, utility, usability, and cost on AgentDojo and AgentBench OS. It prevented attacks and improved secure utility while introducing computational overhead and some utility failures from data-ID processing.
- Secure Utility Rate: 61.86% and 68.42% SUR made PFI the highest-performing agent on AgentDojo and AgentBench OS, respectively.Baseline achieved 12.37% and 0% SUR despite higher total utility success rates of 81.44% and 89.47%.
- Secure Utility Rate: Across all models, PFI achieved the highest SUR on both benchmarks and offered a deterministic security guarantee independent of environment and tools.
- Security: PFI completely prevented prompt injection and data injection attacks, achieving 0.00% ATR across all models and benchmarks.
- Utility Analysis: PFI’s main utility failures involved improper data-ID usage in 75.93% of analyzed tasks, including invalid IDs at 54.63% and improper query generation at 21.30%.DT-processing failures were rare at 5.56%, indicating difficulty using DU through data IDs rather than insecure data handling.
- Usability-Security Trade-off: PFI reduced alerts per task by 63.91% on AgentDojo and 42.86% on AgentBench OS versus Full-Alert while maintaining 0% ATR-Any.PFI raised 1.49 and 1.05 alerts per task, compared with Full-Alert’s 4.11 and 1.84.
- Cost Evaluation: PFI increased latency by 63.49% on AgentDojo and 214.60% on AgentBench OS, while token usage rose 90.56% and 253.90%, respectively.The overhead came from additional LLM invocations and tokens required to process untrusted data in a separate AU.
6 Discussion
The discussion identifies improving utility and defining security policies as future directions for PFI, while retaining its security-oriented design.
- Improving Utility: PFI’s primary utility limitation is that LLMs cannot effectively process untrusted data in the trusted-agent component.The paper proposes fine-tuning or better in-context prompts to improve utility while preserving PFI’s security properties.
- Improving Utility: A hybrid approach could combine PFI’s deterministic security guarantee with probabilistic model alignment for utility improvement.
- Policy Definition: Future work must define security policies for LLM agents as a new class of security principal.The discussion points to app-store ecosystems as one possible model for policy definition.
7 Conclusion
PFI is a secure LLM-agent framework that rethinks system-security principles to address privilege-escalation risks while preserving agent utility.
- PFI isolates LLM agents into trusted and untrusted components, processes untrusted data securely, and applies privilege-escalation guardrails.
- 28-63%p improvement in Secure Utility Rate compared with the baseline ReAct agent demonstrates PFI’s reported evaluation gain.
A Policies
This appendix section describes the policies used to evaluate PFI.
- The evaluation policies define how PFI’s security behavior is assessed.
A.1 Access Token Privilege
The policy analysis classifies benchmark tools by their data attributes and assigns privileged or unprivileged access tokens accordingly.
- The authors inspected all 75 tools in AgentDojo and AgentBench to define the evaluation policies.
- Tool-specific attributes are determined from returned data sources, origins, or sharing levels such as public and private.
- Privileged tokens access every tool without data restrictions, whereas unprivileged tokens access public-data or data-free tools.
A.2 Data Trust Policy
The benchmark’s data trust policy classifies security attributes into categories and identifies which attributes are trusted. Trust assignments distinguish user-controlled or private data from external or shared data sources.
- The policy classifies security attributes into seven categories and lists trusted attributes with reasons for their trustworthiness.
- Trusted data includes system, user, and tool outputs, user or company emails, private cloud and shell files, and the user’s own banking transactions.
- Public shared-folder files and transactions from other users are treated as untrusted because they may contain malicious content.
- Table 4 documents the benchmark’s security attributes and access-token privileges used for policy definition.
B.1 Benchmark Suites
PFI was evaluated on extended AgentDojo and AgentBench OS suites containing utility tasks and security cases involving prompt and data injection attacks. Additional tasks and environment data were generated or crafted for these evaluations.
- PFI was evaluated on AgentDojo and AgentBench, with benchmark tasks and environment data extended to include trusted and untrusted data flows.
- AgentDojo was modified with 10 utility tasks involving untrusted data and 97 additional security tasks involving data injection attacks.
- The AgentDojo security cases injected malicious data, including phishing links and false information, into the agent’s context.
- AgentBench OS evaluation used 19 utility tasks, seven prompt injection attacks, and 19 data injection attacks, including researcher-crafted cases.
B.2 Full Performance Evaluation Results
The full evaluation reports utility and security outcomes for PFI across AgentDojo and AgentBench. The evaluation materials also specify the system prompts and task-generation requirements used for the agent experiments.
- Table 6 reports Secure Utility Rate, Successful Task Rate, Attacked Task Rate, and attack success rates for prompt and data injection attacks.
- PFI configures agent system prompts for AT and AU, shown in Figures 9 and 10.
- Generated AgentBench OS tasks use bash tools and require reading files from the /external directory.
- The task-generation prompt requires populating a query with tool-assisted values and returning the result in JSON format.