PaperlayerView exact revision on arXiv

Source-linked AI summary

Progent: Securing AI Agents with Privilege Control

Tianneng Shi, Jingxuan He, Zhun Wang, Hongwei Li, Linyu Wu, Wenbo Guo, Dawn Song

arXiv:2504.11703v3cs.CRcs.AI

TL;DR

AI agents’ tool access enables indirect prompt injection and creates security challenges as behavior and required privileges evolve. Progent combines symbolic least-privilege policies, LLM-based adaptation, and deterministic SMT checks to control updates. On AgentDojo and ASB, it substantially reduces attack success rates while maintaining utility and integrates with real-world agent frameworks.

  • Problem

    Indirect prompt injection can redirect agents’ tool calls to unauthorized actions, while security requirements vary with task and evolving execution context.

  • Method

    Progent uses symbolic tool-call policies with deterministic enforcement, and an SMT check classifies LLM-proposed updates as automatic narrowings or approval-required expansions.

  • Results

    39.9% to 1.0% on AgentDojo and 70.3% to 3.9% on ASB: Progent reduces attack success rates while maintaining utility.

  • Takeaways & Limitations

    Progent provides a defense-in-depth framework for protecting single- and multi-agent systems through deterministic privilege control and practical framework integration.

  • Takeaways & Limitations

    Progent cannot eliminate risks from incorrect approval of widening updates or ambiguous initial tasks, and it does not address text-only attacks.

Abstract

from arXiv · show

AI agents interact with external environments through tool calls, exposing them to attacks like indirect prompt injection that can trigger unauthorized actions. Securing these agents is challenging: they behave autonomously and probabilistically, security requirements evolve depending on the user's task and execution state, and there is an inherent tradeofff between security and utility. In this work, we introduce Progent, a novel framework that secures AI agents via privilege control. Progent represents privilege as a security policy consisting of symbolic rules over tool names and arguments. These rules specify which tool calls are allowed for task completion and which unnecessary ones are blocked for security. Every tool call is checked against such a policy through a deterministic procedure, enforcing the principle of least privilege. To handle diverse user tasks and evolving execution contexts, an LLM automatically generates the initial policy from the user's task and updates it during execution as new information arrives. Each proposed update is determined by an SMT solver to be either a narrowing (applied automatically) or an expansion (requiring explicit approval), ensuring that the agent's effective action space can only shrink without approval (monotonic confinement). This deterministic update mechanism preserves utility and prevents silent privilege escalation, even when adversarial inputs are present. Our evaluation on popular benchmarks (i.e., AgentDojo and ASB) shows that Progent significantly reduces attack success rates while maintaining high utility. We further validate Progent's practicality by showcasing its effectiveness in real-world agent frameworks such as LangChain and OpenAI Agents SDK.

1 Introduction

Progent secures AI agents by deterministically controlling tool-call privileges with symbolic policies that enforce least privilege. An LLM adapts policies to task and execution context, while SMT-checked updates prevent silent privilege escalation and evaluation shows strong attack reduction with maintained utility.

  • AI agents’ external tool access creates attack surfaces where indirect prompt injections can trigger unauthorized transactions or data exfiltration.
  • Security is difficult because agent behavior is probabilistic, requirements vary with task and runtime context, and legitimate and adversarial privilege expansions share a channel.
  • Progent enforces least privilege through symbolic rules over tool names and arguments, deterministically allowing essential calls and blocking unnecessary ones.
  • An LLM generates and updates policies from the user’s task and execution context, reducing manual security decisions while introducing nondeterminism into policy construction.
  • SMT-checked updates classify narrowings as automatic and expansions as approval-gated, ensuring effective permissions only shrink without approval and preventing silent escalation.
  • 39.9% to 1.0% on AgentDojo and 70.3% to 3.9% on ASB: Progent reduces attack success rates while maintaining utility.
  • Progent combines symbolic, deterministically enforced tool-call policies with LLM-based context adaptation and SMT-based monotonic-confinement checks.

2 Overview

The overview illustrates how indirect prompt injection exploits an agent’s evolving execution context and how Progent mediates tool calls with least-privilege policies. Policies adapt as task information arrives, but deterministic checks automatically apply narrowings and require approval for expansions.

  • 2.1 Challenges for Agent Security: The running example shows an agent completing a benign task while an injected instruction causes an inbox-exfiltrating tool call, leaving the malicious action potentially unnoticed.
  • 2.1 Challenges for Agent Security: Agent behavior is nondeterministic because natural-language inputs and tool returns can vary, while adversarial content in tool returns can redirect subsequent actions.
  • 2.1 Challenges for Agent Security: Different tasks authorize different tools and arguments, so security decisions must automatically track task context and evolving execution context.
  • 2.1 Challenges for Agent Security: Legitimate task updates may require new privileges, whereas an injected send_email request could enable exfiltration through the same tool-return channel.
  • 2.2 Progent: Defense via Privilege Control: Progent enforces structured tool calls rather than unstructured reasoning, deterministically allowing or blocking calls through symbolic rules over tool names and typed arguments.
  • 2.2 Progent: Defense via Privilege Control: Policies evolve per step from LLM-generated context updates, adding required tools, narrowing arguments when information becomes known, and leaving permissions unchanged after an injection.
  • 2.2 Progent: Defense via Privilege Control: SMT-based relations classify updates as expansions or narrowings; narrowings apply automatically, while expansions require configurable approval before taking effect.
  • 2.2 Progent: Defense via Privilege Control: Monotonic confinement ensures the effective action space can only narrow without explicit approval, preventing silent privilege escalation.

3 Problem Statement and Threat Model

The paper models agents as black-box systems that iteratively transform observations into tool calls or completion signals while interacting with an environment. Its threat model assumes attackers manipulate external data to induce unauthorized tool calls, and its defense scope excludes attacks within least privilege and text-only attacks.

  • The agent setup includes a user, black-box agent, tool set, and environment, with each step producing either a tool call or completion signal from the previous observation.
  • Tools are typed functions that return string observations and can range from whole applications to APIs in generated code.
  • The attacker seeks unauthorized actions that benefit them, expressed as malicious tool calls because external effects flow through tool execution.
  • The threat model allows attackers to embed commands in external data retrieved by the agent, while assuming the user and initial query are benign.
  • Progent does not address attacks operating within least privilege, such as preference manipulation, or attacks targeting text outputs rather than tool calls.

4 Progent’s Privilege Control Policy

Progent represents privilege as symbolic rules over tool names and arguments, then deterministically transforms each tool call according to the applicable policy. It compares policies by the sets of tool calls they allow, reducing narrowing checks to SMT solving.

  • Policy Definition: A Progent policy is a list of rules targeting tools and allowing or forbidding calls based on their arguments.Rules may specify fallback operations for blocked calls.
  • Policy Definition: Tool calls consist of a tool identifier and typed argument values, while policy conditions are Boolean expressions over tool arguments.Supported argument types include numbers, strings, booleans, and arrays.
  • Runtime Enforcement: Runtime enforcement filters rules for the target tool, applies forbid rules before allow rules, and evaluates conditions after substituting observed argument values.This deterministic procedure determines whether each rule takes effect.
  • Runtime Enforcement: Blocked calls are replaced by fallback functions, allowed calls remain unchanged, and calls matching no rule are blocked by default.Fallbacks can terminate execution, request inspection, or return a message.
  • Policy Comparison via SMT Solving: A policy permits exactly the tool calls that its enforcement procedure returns unchanged, defining the allowed-call set A(P).Policy comparison uses this set to characterize expansions and narrowings.
  • Policy Comparison via SMT Solving: A narrowing permits a subset of the previous calls, whereas an expansion permits at least one call previously disallowed; narrowing is checked by SMT solving.The check is expressed as whether every call allowed by the new policy is also allowed by the old policy.

5 Securing Agent Execution with Progent

Progent secures agent execution by initializing a task-specific policy, enforcing every tool call, and updating the policy as execution context changes. LLM-generated updates are checked deterministically so narrowings apply automatically while expansions require approval.

  • Agent Execution Loop with Progent: Progent initializes a policy from the user’s task and applies its policy-governed version instead of executing each unprotected tool call.The initial policy is expected to restrict tools and arguments according to least privilege when the user query is benign.
  • Agent Execution Loop with Progent: Policy updates incorporate newly acquired execution context because least-privilege requirements depend on information gathered during execution.Runtime environment information may be untrusted and manipulated by attackers.
  • Agent Execution Loop with Progent: The LLM generates candidate policies, and an SMT check accepts narrowings automatically while sending expansions for approval.This separates automated privilege reduction from privilege increases.
  • Initial Policy Generation: The initial LLM-generated policy reduces AgentDojo attack success rate from 39.9% to 2.5% while maintaining utility.This result is reported without later policy updates.
  • Policy Update: Candidate generation first decides whether an update is needed without the tool result, then uses the full context to generate a policy that undergoes the SMT comparison.This avoids exposing the update-necessity decision to potentially malicious tool output.

6 Progent’s Security Guarantee

Progent guarantees monotonic confinement by enforcing that automatically applied policy updates cannot increase the set of allowed tool calls. Any expansion therefore requires explicit approval.

  • Security Guarantee: The SMT expansion check enforces the security condition for every tool at every policy-update step.Automatically applied updates must preserve the relevant allowed-call containment relation.
  • Security Guarantee: Monotonic Confinement means the agent’s action space cannot increase without explicit approval.Between approved expansions, permissions can only decrease.
  • Security Guarantee: In the running example, the approved expansion from P1 to P2 is followed by a sequence whose allowed calls only narrow or remain unchanged.The stated relation is A(P2) ⊇ A(P3) = A(P4) = A(P5).

7 Implementation

Progent is implemented as a modular policy layer that can validate tool calls and combine task-specific policies with persistent stakeholder constraints. Its approval module exposes configurable controls for the security–utility tradeoff.

  • Implementation: Progent uses JSON Schema to implement privilege policies and validate tool calls in JSON-based agent interfaces.The format supports LLM-generated policies without fine-tuning.
  • Modular Integration: The policy check can be inserted between an agent and its tools without modifying the agent’s internal architecture.Progent offers library and proxy integration modes.
  • Generic Policies: Generic policies encode persistent organizational, developer, or user requirements that remain fixed across tasks.They are composed with task-specific policies sequentially.
  • Generic Policies: Multiple generic policies use explicit priority ordering, preventing lower-priority or task-specific policies from weakening stronger guarantees.Higher-priority policies are enforced first, while lower-priority policies may further restrict the action space.
  • Approver Module: The approver module can automatically approve or deny all expansions or apply finer-grained per-tool approval rules.These settings control the desired security–utility tradeoff.

8 Experimental Evaluation

Progent is evaluated on benchmark, ablation, and real-world agent settings, where it substantially reduces attack success while preserving utility. The experiments also examine configuration choices, model robustness, and deployment across single- and multi-agent systems.

  • Benchmark effectiveness: Progent reduces ASR from 39.9% to 1.0% on AgentDojo while maintaining utility in no-attack and under-attack scenarios.The comparison includes a no-defense baseline and prior defense mechanisms.
  • Benchmark effectiveness: Progent reduces ASR from 70.3% to 3.9% on ASB while keeping utility comparable to the no-defense setting.The LLM-generated policies identify and permit tools needed for benign user tasks.
  • Configuration ablations: Manual Approval achieves 0% ASR, while Auto-Deny provides the strongest security among fully automated configurations.Auto-Approve improves utility and security over Disable Update but has slightly higher ASR than Auto-Deny because widened policies may permit malicious actions.
  • Configuration ablations: Only 6% of policy updates are expansions requiring approval; the remaining updates are narrowings handled automatically.This keeps approval overhead low while preserving strong security.
  • Model ablations: Across policy LLMs, Progent reduces ASR below 5%, reaching 1% with the best-performing LLM.These experiments fix the underlying agent LLM to gpt-4o.
  • Model ablations: Across agent LLMs, Progent maintains or marginally reduces no-attack utility and significantly reduces attack ASR, including to 0.5% and 0.3% for two strong safety models.The results indicate that Progent complements existing defenses as a defense-in-depth layer.
  • Real-world integration: On real-world agents, Progent reduces ASR to around 1% while maintaining strong utility, and in multi-agent systems it defends against more than 95% of successful attacks.The evaluation covers real-world agents and multi-agent systems, with marginal variations attributed to differences in system prompts and retry mechanisms.

9 Limitation and Discussion

Progent’s guarantees are bounded by user decisions, interface coverage, and modality: mistaken approvals or ambiguous tasks can exceed minimal privilege, proxy mode misses bypassing built-in tools, and text-only policies do not cover multimodal actions.

  • User Mistakes: Incorrect approvals of widening policy updates or ambiguous initial tasks may permit policies exceeding minimal privilege.Invariant constraints and explicit approval mitigate but cannot eliminate risks from human misjudgment.
  • Defense Scope: Progent does not address attacks affecting only textual outputs without triggering dangerous tool calls.Such text-to-text attacks are treated as model-level risks outside Progent’s focus.
  • Defense Scope: Proxy mode cannot protect built-in tools that bypass external MCP interfaces.Library mode secures built-in tools with a few lines of code changes, creating a trade-off between deployability and protection depth.
  • Extension to Multimodal Agents: The current method cannot be applied to agent tool calls involving multimodal elements such as browser clicks or screen icons.Future policies could constrain actions to selected applications or screen regions.

10 Related Work

Related defenses use authorization, isolation, information-flow, data-access, or prompt-injection detection mechanisms. Progent distinguishes itself through deterministic privilege enforcement and modular integration at the tool-call level.

  • Security Policy Languages: Security-policy research spans Datalog-style authorization, hardware checks, domain-specific authorization languages, and cloud-provider policy languages.These examples demonstrate programming-based approaches to enforcing security principles.
  • System-Level Defenses for Agents: Prior system-level defenses include execution isolation, trust-label propagation, private-data access control, flow tracking, taint tracking, and LLM-generated policies.These approaches differ in whether they require architectural changes, upfront labeling, or deterministic verification.
  • System-Level Defenses for Agents: Progent’s expansion check prevents silent privilege escalation, providing monotonic confinement that Conseca and DRIFT do not provide.Its modular design also enables integration with existing agent implementations using minimal changes.
  • Model-Level Prompt Injection Defenses: Progent empirically outperforms cited guardrail methods and provides deterministic security enforcement that guardrail models cannot.The comparison concerns model-level prompt-injection defenses evaluated alongside Progent.

11 Conclusion

Progent secures agents through deterministic, SMT-checked privilege control while using an LLM to adapt policies to tasks and execution context. Evaluations show reduced attack success with high utility, and released artifacts support reproducibility and practical adoption.

  • Conclusion: Progent enforces symbolic tool-call policies deterministically and classifies LLM-proposed updates as automatic narrowings or approval-required expansions.This provides monotonic confinement and prevents silent privilege escalation.
  • Conclusion: Evaluations on AgentDojo and ASB significantly reduce attack success rates while preserving high utility.Integration with LangChain and OpenAI Agents SDK further supports Progent as a non-intrusive defense-in-depth layer.
  • Artifacts: The implementation, experiment-reproduction code, and artifact instructions are publicly available.The authors report no policies or licensing restrictions preventing artifact release.
  • Ethics and Evaluation: The reported experiments use local sandboxed environments and publicly available datasets without private or sensitive data.The ethics statement describes the work as a defense mechanism intended to improve control over agent tool permissions.

D Experiment Details

The experiments generally use gpt-4o, include samples of generated policies, and manually inspect a benchmark case whose trace-based success criterion conflicts with blocked tool calls.

  • Models: Most experiments consistently use gpt-4o unless a comparison specifies different models.Listed checkpoints include gpt-4o-2024-08-06, gpt-4.1, claude-sonnet-4, gemini-2.5-flash, and several defense models.
  • Generated Policies: Figure 9 presents sample generated policies.The figure is referenced as an illustration of policy outputs.
  • Benchmark Handling: One Slack injection task is excluded because its implementation cannot distinguish the relevant attack outcome.For another Slack task, blocked calls remain in traces and require manual result checks to avoid misclassification.

E Adaptive Attacks

Progent is evaluated against adaptive attacks designed to exploit its policy-update process, including scenarios with automatically approved updates. The results show that these attacks only marginally increase attack success rates, supporting robustness under the considered attacks.

  • Adaptive Attack Setup: The evaluation uses three adaptive attacks intended to circumvent Progent’s defense through malicious instructions and policy-update manipulation.The attack strategy includes prompting the update process to incorporate tool calls required for the attack task.
  • Adaptive Attack Setup: Progent’s adaptive-attack evaluation models a risky user by automatically approving every policy-update request.This tests whether attackers can exploit the policy-update process when approvals are consistently granted.
  • Results: Adaptive attacks only marginally increase Progent’s attack success rate, demonstrating robustness under the considered attacks.The results are plotted in Figure 10.
  • Practical Integration: In self-built agents, developers wrap tools, initialize policies, and pass tool-call results to an updater followed by an invariant checker.The interfaces automate tool-information retrieval and the policy-update and invariant-checking steps after each tool call.
  • Practical Integration: Progent supports existing frameworks through LangChain middleware and an OpenAI Agents SDK wrapper.These integrations reduce the work required to insert Progent when frameworks internally wrap agent logic.
  • Practical Integration: Proxy Mode lets users protect existing agent products, including potentially closed-source agents, without modifying their implementations.The proxy mediates API and MCP endpoints, while confirmations and generic policy configuration are handled on the proxy side.
Loading 2504.11703v3…