PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents

Juhee Kim, Woohyuk Choi, Byoungyoung Lee

arXiv:2503.15547v2cs.CRcs.AIcs.MA

TL;DR

LLM agents must process untrusted external data, which can let attacker-controlled context influence tool use and enable privilege escalation. PFI combines isolation, secure data processing, and guardrails; it improves Secure Utility Rate over ReAct on AgentDojo and AgentBench OS.

  • Problem

    LLM agents must process attacker-controlled external data, while poisoned context can influence tool use and enable privilege escalation.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    PFI combines trusted and untrusted agent isolation, secure untrusted-data processing, and privilege-escalation guardrails.

    Source: §4.2 · p. 7, §4.3 · p. 8, §7 · p. 14

  • Results

    PFI raised Secure Utility Rate (SUR) versus ReAct [59] from 27.84% to 55.67% on AgentDojo and from 2.63% to 67.79% on AgentBench OS, across diverse models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PFI achieved zero Attacked Task Rate on AgentDojo and AgentBench OS while improving SUR over ReAct in these evaluations.

    Source: §1 · p. 2

    PFI retains a limited risk that AT may leak sensitive information to AU through response-format keys, treated as necessary declassification for computation offloading.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) are combined with tools to create powerful LLM agents that provide a wide range of services. Unlike traditional software, LLM agent's behavior is determined at runtime by natural language prompts from either user or tool's data. This flexibility enables a new computing paradigm with unlimited capabilities and programmability, but also introduces new security risks, vulnerable to privilege escalation attacks. Moreover, user prompts are prone to be interpreted in an insecure way by LLM agents, creating non-deterministic behaviors that can be exploited by attackers. To address these security risks, we propose Prompt Flow Integrity (PFI), a system security-oriented solution to prevent privilege escalation in LLM agents. Analyzing the architectural characteristics of LLM agents, PFI features three mitigation techniques -- i.e., agent isolation, secure untrusted data processing, and privilege escalation guardrails. Our evaluation result shows that PFI effectively mitigates privilege escalation attacks while successfully preserving the utility of LLM agents.

1 Introduction

Tool-using LLM agents can let attacker-controlled inputs influence privileged operations, while their probabilistic behavior makes least-privilege enforcement difficult. PFI proposes a system security-oriented defense using agent isolation, secure untrusted-data processing, and privilege-escalation guardrails.

1 Introduction

  • Agents process attacker-controlled external data through tools that may access sensitive information or perform critical operations, so malicious inputs can trigger unauthorized privileged actions.Because agents probabilistically process the full context, untrusted data can influence privileged operations, making enforcement of least privilege challenging.Source: §1 · p. 1
  • The paper identifies direct prompt injection [26], which instructs tool use, and passive malicious data, which alters action selection and makes data flow difficult to determine.For example, a fake software README may smuggle shell commands into an installation task, while users cannot anticipate every unsafe outcome through prompting alone.Source: §1 · p. 2
  • PFI isolates trusted and untrusted processing into separate agents, restricting the untrusted agent’s privileges to contain attacker-controlled data and limit its impact on sensitive information.This enforces least privilege by keeping untrusted data within the restricted untrusted agent.Source: §1 · p. 2
  • PFI replaces untrusted data with data IDs and enables referencing and computation without exposing the data itself, preventing prompt injection.Its privilege-escalation guardrails constrain misuse of untrusted data in privileged operations, while a fine-grained policy framework enforces agent-specific security principles.Source: §1 · p. 2
  • PFI raised Secure Utility Rate (SUR) over ReAct [59] from 27.84% to 55.67% on AgentDojo and from 2.63% to 67.79% on AgentBench OS.PFI also outperformed IsolateGPT and f-secure LLM [57] in SUR and reduced Attacked Task Rate (ATR) to zero on both benchmarks.Source: §1 · p. 2

2 LLM Agents

LLM agents combine a language model, tools, and contextual information to act on user prompts. They iteratively use context to choose tool calls or produce a final answer.

2 LLM Agents

  • Tools let LLM agents interact with external systems, including web search, email, cloud storage, and host-system tools such as bash.These tools support retrieving current information, managing personal data, and performing actions on the user’s system.Source: §2 · p. 3
  • An LLM agent combines a pre-trained language model, tools, and context containing prompts, tool calls, and results to choose actions or answer users.The system prompt defines the assistant’s behavior and available tools, while the user prompt specifies the task.Source: §2 · p. 3
  • An agent repeatedly runs inference on its context, calls a tool and appends its result when selected, and stops when the model returns a final answer.The workflow begins with the user prompt; each inference uses the accumulated context to select a tool call or final response.Source: §2 · p. 3

3 Motivation

Section 3 motivates PFI by explaining how untrusted tool data can manipulate agents with broad privileges, why existing defenses fall short, and how PFI aims to control that risk while preserving intended use of untrusted data.

3.1 Threat Model

  • Users delegate private-data access and actions to agents, while attackers can poison external systems so tool-returned data manipulates privileged tool calls.Tools may perform unprivileged tasks such as web search or privileged tasks such as reading email or cloud storage; users may not assess each tool call’s safety.Source: §3.1 · p. 3

3.2 Privilege Escalation Attacks

  • LLM agents violate least privilege by acting as one principal with access to all tools, allowing poisoned context to influence actions and grant attackers privileged control.The agent supplies its entire context to a probabilistic LLM, so untrusted data can influence its next action.Source: §3.2 · pp. 3–4

3.2.1 Prompt Injection Attack

  • Prompt injection uses malicious tool data as instructions for privileged actions because agents do not separate prompts from data in the context they use to choose actions.A malicious website retrieved during a news search could instruct the agent to delete files through a shell tool.Source: §3.2.1 · pp. 4–5

3.2.2 Data Injection Attack

  • Data injection exploits benign user requests: untrusted data can alter tool arguments or answers, or be treated as instructions that redirect the agent’s actions.Preventing these attacks is difficult because LLM data flow is probabilistic, and quantifying input influence typically requires impractical internal model inspection.Source: §3.2.2 · p. 5

3.3 Previous Defenses

  • Earlier defenses use probabilistic model alignment or trusted/untrusted agent designs, but incomplete mediation and unsafe data-flow planning leave attack paths open.PFI isolates untrusted data from the trusted agent, tracks it for safe use, and permits its use in planning when the trusted planner endorses it and the user approves.Source: §3.3 · pp. 5–6

4 Prompt Flow Integrity

PFI aims to prevent privilege escalation by separating trusted and untrusted processing, mediating untrusted data through data IDs and least-privilege agents, and checking risky flows with guardrails.

4 Prompt Flow Integrity

  • PFI combines isolated trusted and untrusted agents, data IDs for untrusted results, and guardrails that track data and alert on unsafe use.The trusted agent handles user-facing work with a privileged token; the untrusted agent processes raw tool data with restricted privileges.Source: §4 · p. 6

4.1 Agent Isolation

  • The trusted agent processes only trusted data, while a fresh untrusted-agent context handles raw untrusted data with minimal trusted context and restricted access.The trusted agent receives a privileged token; the untrusted agent receives an unprivileged token and cannot access data not explicitly passed to it.Source: §4.1 · pp. 6–7

4.2 Secure Untrusted Data Processing

  • Data IDs mask untrusted data from the trusted agent, which can reference it in tool calls or answers and offload its analysis to the untrusted agent.Offloading keeps raw untrusted data out of the trusted agent while enabling it to obtain information extracted by the untrusted agent.Source: §4.2 · pp. 7–8
  • PFI permits transforming untrusted data into a prompt only when the trusted agent requests it as necessary and the user explicitly approves.This trades separation of agent responsibilities for more flexible use of external data; the response-format keys may also reveal minimal sensitive information to the untrusted agent.Source: §4.2 · p. 8

4.3 Privilege Escalation Guardrails

  • DataGuard flags untrusted data used in privileged operations, while CtrlGuard flags untrusted data returned in prompt form; both use deterministic indicators and alert the user.Alerts identify the data source and attributes, the sink where the data is used, and whether the flow is data or control.Source: §4.3 · pp. 8–9
  • Guardrail alerts let users review and deny unsafe email data flows or prompt transformations, preventing the email or malicious instructions from proceeding.PFI presents the unsafe operation, data attributes and content for review before the user decides whether to proceed.Source: §4.3 · p. 9

4.4 Prompt Flow Policy

  • PFI defaults tool-result attributes to untrusted but lets tool developers and users configure trust, including trusted, untrusted, transparent, and saved user choices.User choices include Trust Once, Trust Always, and Trust Never, which PFI stores for later use with matching attributes.Source: §4.4 · p. 10
  • PFI defaults to full access for the trusted token and no access for the untrusted token, while allowing developers to set tool sensitivity to improve usability.The untrusted agent can access only data and response formats passed to it by the trusted agent; more restrictive defaults increase guardrail alerts.Source: §4.4 · p. 10

5 Evaluation

PFI achieved higher secure task completion and zero measured prompt- and data-injection attack success across the tested benchmarks, while introducing alert and execution-cost overheads.

5.1 Evaluation Setup

  • The evaluation extended AgentDojo and AgentBench OS with trusted and untrusted data flows, including an AgentBench OS scenario where an unprivileged attacker targets a privileged agent through shared files.AgentDojo measures utility and security on realistic tool tasks; the evaluation defines SUR as successful, secure user tasks and ATR as attacked tasks regardless of completion.Source: §5.1 · pp. 10–11

5.2 Evaluation Results

  • PFI had the highest SUR across models and benchmarks: 61.86% on AgentDojo and 68.42% on AgentBench OS, versus Baseline’s 12.37% and 0%, respectively.PFI also outperformed IsolateGPT, which had around 10% SUR on both benchmarks, and f-secure, which averaged 40.21% on AgentDojo and 10.53% on AgentBench OS.Source: Figure 7 · p. 12, §5.2 · p. 12
  • Baseline ATR-Prompt ranged from 12.37% to 78.95% and ATR-Data from 34.02% to 89.47%; PFI recorded 0.00% for both across models and benchmarks.ATR measures the share of user tasks attacked, regardless of whether the user task succeeds.Source: Table 1 · p. 12, §5.2 · p. 12
  • Among 108 baseline-success/PFI-failure tasks per model, 75.93% stemmed from improper DU use in AT (54.63% invalid data IDs; 21.30% improper queries), versus 5.56% from DT processing.The authors attribute utility loss to LLMs’ difficulty processing DU through data IDs as intended.Source: §5.2 · pp. 12–13
  • PFI raised 1.49 alerts per task on AgentDojo and 1.05 on AgentBench OS, reducing alerts versus Full-Alert by 63.91% and 42.86%, respectively, while maintaining 0% ATR-Any.IsolateGPT raised fewer alerts, but its ATR-Any showed attacks succeeded on more than 70% of tasks.Source: Table 2 · p. 13, §5.2 · p. 13
  • PFI recorded no guardrail false positives or false negatives: alerts target untrusted data in privileged operations or prompts, and log review found no unapproved raw DU use.By design, guardrails track DU with data IDs or alert when DU is transformed into a prompt.Source: §5.2 · p. 13
  • PFI increased latency by 63.49% on AgentDojo and 214.60% on AgentBench OS, and token usage by 90.56% and 253.90%, respectively.Separate AU invocations add overhead; on AgentBench OS, AT continued issuing shell commands after AU failures, lengthening execution traces.Source: Table 3 · p. 13, §5.2 · pp. 13–14

6 Discussion

The discussion proposes future work to improve PFI’s utility and define security policies for LLM-agent tools. It also calls for agent-specific mechanisms to address the limits of user inspection and authorization.

Improving Utility

  • PFI could combine fine-tuning for utility with its deterministic security guarantees, since prior security-oriented fine-tuning offers probabilistic guarantees vulnerable to malicious prompts [29] [64].The paper identifies difficulty processing untrusted data as the primary cause of PFI’s utility drop.Source: §6 · p. 14

Policy Definition

  • Tool developers could define security policies for a centralized app store, where experts review them and users consult policies and reputations when choosing tools.The app store could provide a reputation system to help users select trustworthy tools.Source: §6 · p. 14

Manual User Inspection

  • Because users may lack expertise or approve PFI guardrail alerts for convenience [52], future work should develop effective security mechanisms specific to LLM agents [56].PFI relies on users inspecting alerts and authorizing or blocking privilege escalation; the alerts provide detailed information.Source: §6 · p. 14

7 Conclusion

PFI is a secure LLM agent framework that rethinks system security principles to address agent security challenges. It combines agent isolation, secure untrusted data processing, and privilege escalation guardrails.

7 Conclusion

  • PFI improves Secure Utility Rate (SUR) by 28-63%p compared to the baseline ReAct [59] agent.The conclusion describes this as robust protection against attacks.Source: §7 · p. 14
  • PFI isolates LLM agents into trusted and untrusted components.This is one of PFI’s suggested design principles.Source: §7 · p. 14
  • PFI’s design principles also include secure untrusted data processing and privilege escalation guardrails.Source: §7 · p. 14

A Policies

The section describes the policies used in PFI’s evaluation (§5).

A Policies

  • The section presents the policies used to evaluate PFI.The evaluation is described in §5.Source: A Policies · p. 17

A.1 Access Token Privilege

The section defines tool security attributes from tool results and data sources, and distinguishes privileged from unprivileged token access.

A.1 Access Token Privilege

  • The policy analysis inspected all 75 AgentDojo [13] and AgentBench [34] benchmark tools and reports their results, security attributes, and privileges in Table 4.The tool-specific security attributes are abbreviated Attr.Source: §A.1 · p. 17
  • Tool-specific Attr is based on returned messages, data origin, or sharing level, depending on what the tool returns.Returned tool-specific result messages use a tool: prefix; data from sources such as email or web origin is attributed by origin, while public or private data is attributed by sharing level.Source: §A.1 · p. 17
  • Privileged tokens (TP) can access every tool without data-access restrictions, while unprivileged tokens (TU) can access tools that use public data or no data.Source: §A.1 · p. 17

A.2 Data Trust Policy

The benchmark defines trust by data attributes across seven categories, using trusted and untrusted inputs to balance data flows from DT and DU. Trust assignments vary by source, with specific assumptions for Slack and web data.

A.2 Data Trust Policy

  • The benchmark classifies Attr into seven format-based categories and lists trusted Attr with reasons for trustworthiness.Hypothetical user-configured lists and trusted Slack members from the AgentDojo environment balance benchmark data flows from DT and DU.Source: §A.2 · p. 17
  • Trusted data varies by source: private cloud and OS files, the user’s banking transactions, selected Slack messages, and reliable or user-configured web origins are trusted.Other shared-folder data and other users’ transactions are untrusted; Slack trust includes listed group members, a trusted bot, and private channels.Source: §A.2 · p. 17
  • Private Slack channel messages are trusted under the assumption that only trusted users can write there and the channel is managed to prevent malicious data injection.Trusted web origins include government and educational sites and selected news sites.Source: §A.2 · p. 17

B Evaluation Details · B.1 Benchmark Suites

PFI was evaluated on AgentDojo [13] and AgentBench [34], with benchmark tasks and environment data extended to test trusted and untrusted data flows. The extensions added utility and security cases, including data-injection attacks.

B.1 Benchmark Suites

  • The evaluation used AgentDojo [13] and AgentBench [34], extending tasks and environment data to cover data flows involving trusted and untrusted data.Additional test cases and environment data were generated using ChatGPT (GPT-4o model).Source: §B.1 · p. 19
  • For AgentDojo, 10 of 97 utility tasks were modified to involve untrusted data.Source: §B.1 · p. 19
  • AgentDojo’s 609 prompt-injection security tasks were supplemented with 97 data-injection tasks involving malicious context data such as phishing links and false information.Source: §B.1 · p. 19
  • For AgentBench OS [34], evaluation selectively covered attack-vulnerable tasks that read file names or contents.Source: §B.1 · p. 19
  • The AgentBench OS [34] utility evaluation used 19 tasks: 5 from the original suite and 14 crafted by the authors.Source: §B.1 · p. 19
  • AgentBench OS [34] security evaluation included 7 prompt-injection and 19 data-injection test cases.ChatGPT assisted in generating candidate user tasks; the generation prompt appears in Figure 8.Source: §B.1 · p. 19

B.2 Full Performance Evaluation Results · C PFI System Prompts

The section describes PFI’s benchmark evaluation measures and identifies the system prompts and task-generation prompt shown in its figures.

B.2 Full Performance Evaluation Results

  • ASR is the percentage of successful attacks among all attack attempts in AgentDojo and AgentBench security evaluation.Source: §B.2 · p. 19
  • ATR counts attacked user tasks, whereas ASR counts successful attacks, whose goals vary across attacks.Source: §B.2 · p. 19

C PFI System Prompts

  • PFI configures system prompts for its agents, with the AT and AU prompts shown in Figures 9 and 10, respectively.Source: C PFI System Prompts · p. 19

B.2 Full Performance Evaluation Results

  • Table 6 reports percentage evaluation results for PFI and the baseline on AgentDojo and AgentBench, using SUR, STR, ATR, and ASR for prompt and data injection attacks.The supplied passages identify the table’s scope and metrics but do not provide its cell values.Source: §B.2 · p. 19, Table 6 · p. 20

C PFI System Prompts

Related papers

Loading 2503.15547v2…