PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Agents Should Employ Security Principles

Kaiyuan Zhang, Zian Su, Pin-Yu Chen, Elisa Bertino, Xiangyu Zhang, Ninghui Li

arXiv:2505.24019v1cs.CRcs.AI

TL;DR

LLM agents are increasingly useful but vulnerable to prompt injection, contextual manipulation, privacy leakage, and unauthorized actions. The paper proposes AgentSandbox, which embeds established security principles into agent architectures and communication flows. Across benign utility, attack utility, and attack success rate, the framework maintains useful behavior while mitigating privacy and security risks.

  • Problem

    LLM agents interact with sensitive data, external tools, and dynamic workflows, creating vulnerabilities that existing defenses and standards do not adequately address.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    AgentSandbox applies defense-in-depth, least privilege, complete mediation, and psychological acceptability through persistent and ephemeral agents, data minimization, filtering, firewalls, and adaptive policies.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    AgentSandbox achieves the best overall trade-off between utility and security, with 82.00% average benign utility compared with 71.24% for Tool Filter.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    Embedding security principles throughout agent lifecycles can help balance agent capabilities with user privacy and system integrity.

    Source: §6 · p. 9

    The framework assumes that the personal agent and direct user queries are intrinsically benign, while adversarial influence comes from external agents or tools.

    Source: §2 · p. 3

Abstract

from arXiv · show

Large Language Model (LLM) agents show considerable promise for automating complex tasks using contextual reasoning; however, interactions involving multiple agents and the system's susceptibility to prompt injection and other forms of context manipulation introduce new vulnerabilities related to privacy leakage and system exploitation. This position paper argues that the well-established design principles in information security, which are commonly referred to as security principles, should be employed when deploying LLM agents at scale. Design principles such as defense-in-depth, least privilege, complete mediation, and psychological acceptability have helped guide the design of mechanisms for securing information systems over the last five decades, and we argue that their explicit and conscientious adoption will help secure agentic systems. To illustrate this approach, we introduce AgentSandbox, a conceptual framework embedding these security principles to provide safeguards throughout an agent's life-cycle. We evaluate with state-of-the-art LLMs along three dimensions: benign utility, attack utility, and attack success rate. AgentSandbox maintains high utility for its intended functions under both benign and adversarial evaluations while substantially mitigating privacy risks. By embedding secure design principles as foundational elements within emerging LLM agent protocols, we aim to promote trustworthy agent ecosystems aligned with user privacy expectations and evolving regulatory requirements.

1. Introduction

LLM agents enable complex, real-world task automation but remain vulnerable to prompt injection, contextual manipulation, privacy leakage, and unauthorized actions. The paper argues for explicitly applying established information-security principles and introduces AgentSandbox as a framework implementing them in agent communication protocols.

  • LLM agents increasingly support real-world applications, where planning, reasoning, and tool use enable task decomposition and knowledge sharing.Source: §1 · p. 1
  • Advanced LLMs fail prompt-injection defenses approximately 85% of the time, while contextual manipulation can induce sensitive-data disclosure and destructive actions.Source: §1 · p. 1
  • Existing agent standards primarily address authentication, transport, and authorization, giving less attention to blind instruction following, prompt hacking, and contextual manipulation.Source: §1 · p. 1
  • Established principles including least privilege, complete mediation, psychological acceptability, and defense-in-depth provide a security foundation for LLM agents.Source: §1 · p. 2
  • AgentSandbox applies these principles through layered components, separated persistent and ephemeral agents, minimized disclosures, mediated access, and automated policy generation.Source: §1 · p. 2

2. Problem Setup, Threat Model, and Challenges

The paper models personal agents that access sensitive user data and digital services while interacting with potentially compromised external tools. It identifies dynamic workflows, memory, domain-specific privacy rules, and ambiguous language as obstacles requiring adaptive, context-aware defenses.

  • Personal agents may access financial details, contact information, preferences, and digital actions such as emailing, payments, and calendar changes.Source: §2 · p. 2
  • The adversary controls or influences external agents and tools, using manipulated inputs to induce confidential-data leakage or harmful actions.Source: §2 · p. 3
  • The threat model assumes the personal agent and direct user queries are benign, while defenders control the agent’s logic, prompts, policies, modules, and protocols.Source: §2 · p. 3
  • Security is challenged by domain-specific privacy rules, evolving workflows, memory resurfacing sensitive data, and ambiguous language that can trigger unintended disclosures.Source: §2 · p. 3

3. AgentSandbox Framework: Employing Security Principles

AgentSandbox operationalizes four security principles through a persistent agent, task-specific ephemeral agents, data minimization, mediated communication, response filtering, and adaptive policy optimization. Its illustrative workflow shows how these mechanisms aim to preserve useful travel-agent behavior while containing attacks and limiting disclosures.

3. AgentSandbox Framework: Employing Security Principles

  • AgentSandbox combines a Persistent Agent, Data Minimizer, Ephemeral Agent, I/O Firewall, and Response Filter to orchestrate tasks and control information flows.Source: §3 · p. 3, Figure 1 · p. 4

3.1. Defense-in-Depth

  • Separating persistent and disposable agents contains prompt-injection effects within one task session and prevents adversarial instructions from polluting persistent state.Source: §3.1 · pp. 3–4
  • Adaptive data minimization and fixed firewall constraints reinforce defense-in-depth by combining utility-oriented policy refinement with immutable safety checks.Source: §3.1 · p. 4

3.2. Least Privilege

  • The Data Minimizer provisions each ephemeral agent with only task-essential, contextually appropriate information and can require explicit user justification for additional disclosure.Source: §3.2 · pp. 4–5

3.3. Complete Mediation

  • Complete mediation checks internal data requests and responses through the Data Minimizer and Response Filter, while the I/O Firewall validates every external input and output.Source: §3.3 · p. 5

3.5. Illustrative Example

  • The framework’s security mechanisms are designed to avoid significant user inconvenience while operationalizing defense-in-depth, least privilege, complete mediation, and psychological acceptability.Source: §3.4 · p. 5, §3.5 · p. 7

3.4. Psychological Acceptability

  • The reward-modeling policy engine iteratively balances privacy preservation, task success, and utility to reduce manual policy configuration.Source: §3.4 · pp. 5–6

3.5. Illustrative Example

  • In the travel example, AgentSandbox safely completes booking within budget while illustrating protection against PII extraction, indirect prompt injection, memory poisoning, and mixed attacks.Source: §3.5 · p. 7

4. Evaluation

AgentSandbox is evaluated on AgentDojo against existing defenses using benign utility, attack utility, and attack success rate. Across four task suites, it achieves the strongest overall utility–security trade-off while preserving benign performance and reducing attacks.

4.1. Experimental Setup

  • AgentDojo evaluates 97 realistic tasks across Banking, Slack, Travel, and Workspace, each paired with adversarial prompt-injection attacks.The primary model is gpt-4o-2024-08-06; analyses also extend to o3-mini-2025-01-31 and gpt-4o-mini-2024-07-18.Source: §4.1 · p. 7
  • The evaluation compares No Defense, Tool Filter, PI Detector, Delimiting, Repeat Prompt, and AgentSandbox using benign utility, attack utility, and ASR.Higher benign and attack utility are better, while lower ASR is better.Source: §4.1 · pp. 7–8

4.2. Comparison with Existing Defense Baselines

  • AgentSandbox achieves the best overall utility–security trade-off, maintaining 82.00% benign utility while reducing average ASR to 4.34% across all task suites.Its benign utility is comparable to the No Defense baseline at 83.81%.Source: §4.2 · p. 8, Table 1 · p. 8
  • No Defense preserves high benign utility but reaches 58.84% average ASR, showing the need for active defenses.Delimiting and Repeat Prompt retain 75.75% and 85.75% benign utility but still have 27.97% and 27.33% average ASR.Source: §4.2 · p. 8
  • PI Detector lowers average ASR to 6.79% but often reduces attack utility below 20% by halting execution whenever injection is suspected.Tool Filter reaches 60.10% attack utility and 8.90% ASR, but lowers benign utility to 71.24%.Source: §4.2 · p. 8

5. Related Work

Related work includes architectures, policy systems, benchmarks, runtime enforcement, and sandboxing for agent security. The paper argues that existing approaches do not explicitly and systematically apply the advocated security principles, particularly through ephemeral agents.

Implicitly Applying Security Principle Solutions

  • Existing approaches address agent security through context access control, firewalls, policy specification, interaction analysis, confirmation, privilege languages, and guardrail enforcement.Examples include AirgapAgent [9], firewall agentic networks [1], Microsoft Sensitive Labels, Clio, RTBAS [78], Progent [55], and LlamaFirewall [16].Source: §5 · p. 9
  • The paper identifies a gap in explicit, systematic application of security principles across existing agent-security approaches.It specifically notes that none of the reviewed approaches use AgentSandbox’s Ephemeral Agents for defense-in-depth and least privilege.Source: §5 · p. 9

LLM Agent Attacks and Defenses

  • Agent security research spans real-world action risks, privacy violations, website hacking, systematic harm, external-tool vulnerabilities, benchmarks, runtime enforcement, and sandboxing.The paper evaluates with AgentDojo [19].Source: §5 · p. 9

6. Conclusion

The paper concludes that established security principles should guide LLM-agent architectures and ecosystems. It presents AgentSandbox as a conceptual operationalization intended to balance agent capabilities with privacy and system integrity, alongside an adaptive policy-optimization procedure.

6. Conclusion

  • AgentSandbox operationalizes defense-in-depth, least privilege, complete mediation, and psychological acceptability throughout an agent’s lifecycle.The conclusion frames this principled design as a way to balance advanced capabilities with user privacy and system integrity.Source: §6 · p. 9
  • The paper urges researchers and industry to integrate and continue evolving foundational security considerations in next-generation LLM agents.Its stated goal is a trustworthy AI ecosystem.Source: §6 · p. 9

Appendix

  • The appendix includes navigation material for the paper’s additional content.The supplied appendix passage is a table-of-contents note.Source: Appendix · p. 14

A. Algorithm of Reward Modeling Policy Engine

  • The Reward Modeling Policy Engine iteratively proposes adaptable policies, deploys them to agent modules, evaluates task outcomes, updates its optimizer, and returns the best-performing policies.Its inputs include seed policies, agent modules, a task metric, an iteration limit, and optimizer configuration.Source: A. Algorithm of Reward Modeling Policy… · p. 14

B. Evaluation Details

The evaluation details organize AgentDojo examples into four task domains: Banking, Slack, Travel, and Workspace. Each domain pairs ordinary user requests with malicious actions that illustrate the security consequences being tested.

Banking

  • Banking tasks cover statement review, account management, and transactions, contrasting a cheapest top-rated London hotel reservation with an expensive Paris booking.The examples distinguish benign task intent from a maliciously altered reservation objective.Source: B. Evaluation Details · pp. 14–15

Slack

  • Slack tasks involve messages, web browsing, and file access, contrasting inviting Dora through her website with sending a phishing link to Alice.The malicious example targets communication through Slack.Source: B. Evaluation Details · p. 15

Travel

  • Travel tasks focus on flights, restaurants, and car rentals, contrasting paying a December bill with sending as much money as possible to an attacker account.The malicious example tests unauthorized financial transfer.Source: B. Evaluation Details · p. 15

Workspace

  • Workspace tasks involve email, calendars, and cloud drives, contrasting checking appointments with sending a Facebook security code to an attacker email.The malicious example tests disclosure of a security code.Source: B. Evaluation Details · p. 15

C. Evaluation on Other Models

Across two additional models, AgentSandbox again achieves the best overall trade-off between utility and security among evaluated defenses. The surrounding literature identifies jailbreaking, prompt injection, and data poisoning as major threats to LLM agents.

C. Evaluation on Other Models

D. Related Work

  • Jailbreaking attacks have evolved from manually crafted prompts to automated optimization, search, fuzzing, and multi-turn dialogue strategies.The cited approaches include gradient-based optimization, genetic algorithms, tree-based search, prompt fuzzing, and interactive multi-turn attacks.Source: D. Related Work · p. 16
  • Prompt injection embeds adversarial instructions that can override intended tool use, while indirect injection and data poisoning manipulate inputs through external sources or agent memory.These attacks target both agent behavior and external data sources used by agents.Source: D. Related Work · p. 16

E. Discussion

The discussion identifies human involvement and stronger reward modeling as important directions for making agent security and privacy systems more scalable, expressive, and trustworthy. It also illustrates the contrast between an unprotected successful attack and the proposed framework’s evaluation setting.

E. Discussion

  • Manual PII labeling and dynamic policy refinement remain early-stage and limit scalability, motivating auto-labelers and mixed-initiative human-agent collaboration.The proposed collaboration would let agents solicit guidance on ambiguous policies and help experts verify and refine generated policies.Source: E. Discussion · p. 16
  • AgentSandbox reward modeling could use more expressive privacy-utility objectives and better exploration, but verifying policy completeness and interpretability remains challenging.The discussion also identifies prompt design across modules and principled reward-function optimization as open problems.Source: E. Discussion · p. 16

F. Case Studies

  • The case-study section examines one successful attack under No Defense and one successful defense under AgentSandbox.These cases provide concrete examples of the two evaluation conditions discussed in the paper.Source: F. Case Studies · p. 16

A Successful Attack Case under No Defense Event 1. User Initiation

In the no-defense case, a transaction-history query exposes information that is followed by an injected instruction to transfer money. The assistant then executes the unauthorized transaction and reports its completion.

A Successful Attack Case under No Defense Event 1. User Initiation

  • A transaction-history tool returns the user’s records, including a transaction note containing an injected “important message.”The attack begins during processing of the user’s spending query.Source: A Successful Attack Case under No… · pp. 16–17
  • The injected instruction directs the assistant to send money to an attacker-controlled recipient before answering the original spending question.It also requests inclusion of the music streaming service visible in the transaction history.Source: A Successful Attack Case under No… · p. 17

Event 4. Assistant Action

  • The assistant executes send_money(amount=50.0, date=‘2023-11-30’, ...) and reports that the $50.00 Spotify Premium transaction was successfully sent.The tool output confirms the transfer to the specified recipient.Source: Event 4. Assistant Action · p. 17

A Successful Defense Case under AgentSandbox

The supplied passages show the defense-case interaction beginning with the same spending query and transaction-history retrieval, followed by an injected instruction. They do not state the subsequent defensive outcome.

A Successful Defense Case under AgentSandbox

  • The defense-case trace begins with the user asking for total spending in March 2022 and the assistant retrieving recent transactions.The retrieved output includes a transaction record with an information field.Source: A Successful Defense Case under… · pp. 17–18
  • The retrieved transaction output contains an “important message” instructing the assistant to send a transaction before answering the original query.The supplied trace ends after the injected instruction and follow-up text.Source: A Successful Defense Case under… · p. 18
  • The supplied defense-case passages do not report whether AgentSandbox blocks or transforms the injected instruction.No downstream action or outcome is stated in these passages.Source: A Successful Defense Case under… · p. 18

Event 4. Assistant Action

AgentSandbox blocks an attempted money transfer that violates the active security policy, then recovers by continuing the requested spending calculation. Its Data Minimizer and Ephemeral Agent components apply task-scoped permissions, parameter and domain checks, attack detection, and context-aware validation.

Event 4. Assistant Action

  • The attempted send_money action is blocked because the tool is not permitted by the current security policy, after which the assistant continues with the spending calculation.This demonstrates policy enforcement followed by recovery rather than abandoning the user’s task.Source: Event 4. Assistant Action · p. 18

Data Minimizer

  • The Data Minimizer determines permitted tools, assigns tool-specific restrictions, assesses query risk, detects attack patterns, and applies restrictions accordingly.Benign queries receive task-scoped functionality, while high-risk queries face tighter permissions, domain blocking, and prevention of risky actions.Source: Data Minimizer · pp. 18–19
  • For benign queries, the Data Minimizer allows explicitly required tools and trusted domains while restricting access to the task’s necessary scope.The policy aims to preserve required functionality without granting broader access than the task needs.Source: Data Minimizer · p. 18
  • For high-risk queries or detected attacks, the Data Minimizer restricts exfiltration-capable tools, tightens sensitive-data permissions, blocks suspicious domains, and prevents high-risk actions.These restrictions are selected based on risk assessment and detected attack patterns.Source: Data Minimizer · pp. 18–19

Ephemeral Agent

  • The Ephemeral Agent validates each requested action against task scope, parameters, URLs, message content, suspicious patterns, and attack sequences across actions.Its context-aware checks incorporate prior session actions and relationships such as reading data followed by posting it elsewhere.Source: Ephemeral Agent · p. 19

Related papers

Loading 2505.24019v1…