PaperlayerView exact revision on arXiv

Source-linked AI summary

Design Patterns for Securing LLM Agents against Prompt Injections

Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, Ezinwanne Ozoani, Andrew Paverd, Florian Tramèr, Václav Volhejn

arXiv:2506.08837v3cs.LGcs.CR

TL;DR

LLM agents create security risks because prompt injections can manipulate agents that process natural-language input, especially when tools or sensitive information are involved. The paper proposes six simple, principled design patterns that constrain application-specific agents, analyzes their security–utility trade-offs, and demonstrates them across ten case studies. It concludes that application-specific agents can be secured through principled system design, while robust security generally requires combining patterns.

  • Problem

    LLM agents face prompt injection threats that can manipulate behavior and cause unauthorized actions, creating a critical security challenge as agents gain tool access and handle sensitive information.

  • Method

    The paper proposes six simple design patterns that constrain LLM agents, analyzes their security–utility trade-offs, and applies them across ten case studies.

  • Results

    The patterns are demonstrated across ten diverse case studies, providing meaningful resistance to prompt injections without overly restricting agent utility.

  • Takeaways & Limitations

    Developers should prioritize application-specific agents with clear trust boundaries and combine design patterns because no single pattern is likely to suffice across all threat models or use cases.

  • Takeaways & Limitations

    General-purpose agents remain out of reach for secure deployment with current capabilities, and allowlists can falsely block benign requests dissimilar from predefined entries.

Abstract

from arXiv · show

As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge. Among the most pressing threats are prompt injection attacks, which exploit the agent's resilience on natural language inputs -- an especially dangerous threat when agents are granted tool access or handle sensitive information. In this work, we propose a set of principled design patterns for building AI agents with provable resistance to prompt injection. We systematically analyze these patterns, discuss their trade-offs in terms of utility and security, and illustrate their real-world applicability through a series of case studies.

1 Introduction

LLM agents expand natural-language automation but introduce security vulnerabilities, especially prompt injections that can cause severe unauthorized actions. The paper proposes constrained design patterns and ten case studies to balance security with utility.

  • LLM agents interpret instructions, plan tasks, and execute actions through external tools and APIs, exposing vulnerabilities that traditional application security frameworks are ill-equipped to address.
  • Prompt injections embed malicious data in processed content to manipulate agent behavior, with consequences including data exfiltration, privilege escalation, and remote code execution.
  • Existing defenses range from heuristic detection and adversarial training to principled system-level isolation, but proposed measures are primarily generic for general-purpose agents.
  • The paper introduces design patterns that constrain agents from solving arbitrary tasks, offering a trade-off between agent utility and security.
  • Ten case studies, from OS function assistants to software engineering agents, illustrate design decisions that provide meaningful prompt-injection resistance without overly restricting utility.
  • The authors intend these design patterns and case studies to guide designers, developers, and decision makers toward building secure LLM agents.

2 Background

Prompt injections manipulate LLM agents through untrusted instructions or data, enabling unauthorized actions and other harms. Background defenses operate at LLM, user, and system levels, but heuristic methods lack guarantees while human confirmation can reduce usability.

  • LLM agents convert natural-language input into action plans and execute them through tools or Agent Computer Interfaces.
  • Prompt injections introduce adversarial instructions into content processed by an LLM, causing deviation from intended behavior.
  • Attackers may seek unauthorized tool execution, protected-data exfiltration, reasoning or output manipulation, or denial of service through resource exhaustion.
  • The paper treats direct injections from end-users and indirect injections through third-party data as prompt injections without distinguishing between them.
  • Existing defenses are categorized as LLM-level, user-level, and system-level approaches.
  • Prompt engineering and adversarial training provide some protection but do not provide guarantees, while confirmation mechanisms can reduce automation and usability.
  • System-level defenses integrate external verification and control mechanisms and are described as a promising path toward robust safety guarantees.
  • Input/output detection systems raise the bar for attackers but remain fundamentally heuristic and cannot guarantee prevention of all attacks.

3 Design Patterns for Securing LLM Agents Against Prompt Injections

The paper proposes constrained agent designs that isolate untrusted data from consequential actions, seeking meaningful prompt-injection resistance while preserving useful functionality. Six patterns progressively restrict how untrusted inputs can influence control flow, tools, outputs, or other agents.

  • 3 Design Patterns for Securing LLM Agents Against Prompt Injections: General-purpose agents combine powerful tools with untrusted third-party data, while heuristic defenses are unlikely to provide reliable safety guarantees.The paper therefore asks what useful agents can be built today with principled resistance to prompt injection.
  • 3 Design Patterns for Securing LLM Agents Against Prompt Injections: The proposed patterns intentionally constrain agents so untrusted input cannot trigger consequential actions or compromise system integrity and confidentiality.The designs are presented as concrete, composable strategies balancing utility and security.
  • 3 Design Patterns for Securing LLM Agents Against Prompt Injections: Six patterns enforce varying degrees of isolation between untrusted data and agent control flow, alongside separate best practices such as sandboxing and user confirmation.The patterns are distinguished from general security practices that should be incorporated where possible.
  • 3.1 Design Patterns for Securing LLM Agents: The action-selector pattern translates natural-language requests into predefined tool calls and prevents action feedback from returning to the agent.It can select fixed or templated actions, including actions over untrusted data, but cannot adapt action choice based on resulting feedback.
  • 3.1 Design Patterns for Securing LLM Agents: The plan-then-execute pattern fixes the allowed tool-call sequence before untrusted data is processed, preventing injections from adding new tools while permitting feedback to affect tool inputs.For example, an email-and-calendar assistant can formulate a plan before reading calendar data.
  • 3.1 Design Patterns for Securing LLM Agents: Plan-then-execute does not prevent untrusted data from altering outputs, and adversarial feedback can still manipulate inputs to the fixed tool calls.Calendar data could alter the body of an email even though it cannot inject a new instruction or tool.
  • 3.1 Design Patterns for Securing LLM Agents: The map-reduce pattern processes each untrusted document with an isolated agent, then aggregates results using a tamper-robust operation or constrained LLM output.A malicious document cannot directly affect another document’s processing, though it may still influence its own mapped result.
  • 3.1 Design Patterns for Securing LLM Agents: The dual LLM pattern separates a tool-using privileged LLM from a tool-less quarantined LLM that processes untrusted text and returns reference-based results.The code-then-execute pattern extends this separation by having the agent write a formal program that calls tools and unprivileged LLMs over untrusted data.

4 Case Studies

The case studies apply the proposed design patterns across diverse LLM-agent applications, comparing naïve designs with constrained alternatives. The operating-system assistant illustrates how isolation, action restriction, and user confirmation trade utility against security.

  • Case-study scope: The ten case studies span everyday productivity tools and sensitive applications, covering diverse domains, security requirements, and threat models.Each case study compares naïve vulnerable implementations with more robust designs using the proposed patterns and standard security practices.
  • Operating-system assistant: The operating-system assistant uses tool access to search and manipulate files, while attackers may control file contents and seek insecure commands or data exfiltration.The naïve shell design exposes a broad attack surface because files can contain instructions that trigger arbitrary code execution.
  • Operating-system assistant: Hardcoded action selection prevents prompt injections because the LLM never directly processes data, but it sacrifices much of the agent’s fuzzy-search utility.Most functionality is shifted into predefined commands, reducing flexibility.
  • Operating-system assistant: Plan-then-execute restricts commands to those committed for the task, but innocuous commands can still be combined to perform unsafe actions.The pattern may also be difficult when command choices depend on earlier results.
  • Operating-system assistant: The dual-LLM/map-reduce design dispatches isolated models with constrained outputs, limiting a malicious file’s effect to that file when decomposition is possible.Its main limitations are that strict decomposition may be impossible and a malicious file may still be moved or copied incorrectly.

4.2 SQL Agent

The SQL Agent combines database querying with Python-based analysis and visualization, providing broad utility but exposing risks from user or database prompt injections. Plan-then-execute and action sandboxing reduce these risks while introducing specific utility and residual-security trade-offs.

  • Architecture: The SQL Agent answers database questions using SQL queries and Python code for analysis, verification, and visualization, with iterative refinement of both.It can be used directly by people or called by other agents.
  • Threat model: The agent’s capabilities support accessible data insights but also enable unauthorized extraction or modification, resource waste, denial of service, and Python remote code execution.Attackers may control the input query or database content.
  • Naïve design: The naïve design grants complete database access and arbitrary Python execution, requiring heuristic guardrails that remain vulnerable to adversarial attacks.Reconnaissance and information extraction remain possible in this design.
  • Plan-then-execute: Plan-then-execute prevents database contents from influencing the LLM that generates analysis code, avoiding prompt injections embedded in those contents.Utility decreases because the agent loses feedback about whether query results are sufficient, although basic query analysis remains.
  • Action sandboxing: Sandboxing Python execution addresses many traditional and AI-specific threats without reducing utility, while SQL requires less sandboxing because specialized syntax limits execution threats.Residual attacks can still target answer-sufficiency reasoning or user-provided inputs, motivating additional patterns.
  • Guardrailing: User approval and task-specific guardrails can improve safety, but false positives, added compute, adversarial examples, and incomplete lists may reduce utility.User approval is not practical at scale for every transfer step.

4.3 Email and Calendar Assistant

The email and calendar assistant can perform consequential actions with the user’s privileges, making third-party content a significant prompt-injection threat. User confirmation, execution planning, and quarantined LLMs offer complementary protections but leave usability and output-manipulation limitations.

  • Capabilities: The assistant searches email and calendar data, synthesizes information, and performs actions such as sending messages, invitations, and replies.It may be accessed through desktop text chat or mobile voice interfaces.
  • Threat model: Third-party attackers can inject content through emails, invitations, images, or attachments, seeking data exfiltration, unauthorized messages, deletion, or covert modification of replies.Successful injections may cause unrequested actions or alter requested actions using hidden encodings.
  • User confirmation: User confirmation before consequential actions can prevent many attacks when users are vigilant, but covert exfiltration may evade review and voice interfaces increase review burden.The pattern does not significantly reduce autonomy when users already expect the assistant to act.
  • Plan-then-execute: Plan-then-execute preserves utility when action choice should not depend on untrusted data and prevents that data from changing control flow.Action parameters such as email recipients or bodies can still be influenced by untrusted content.
  • Dual LLM: The Dual LLM pattern quarantines untrusted-data processing, but the quarantined model can still access sensitive inputs and produce attacker-controlled outputs.Symbolic processing of its response therefore does not eliminate output tampering risks.

4.4 Customer service chatbot

The customer-service chatbot combines retrieval and tool use for information and user-authorized actions, but prompt injections threaten data confidentiality and company reputation. Topic classifiers and allowlists constrain requests, while least privilege and context minimization address specific attack paths with trade-offs.

  • Capabilities: The chatbot answers product and policy questions through retrieval or tools and performs actions such as returns, installations, and cancellations.Its goal is to provide customer support for a furniture retailer.
  • Access control: Least-privilege access limits a malicious user to their own calendar data, while authentication or session restrictions prevent actions on other users’ orders.These controls preserve usefulness while narrowing the consequences of compromised instructions.
  • Threat model: Prompt injections can exfiltrate customer data or create reputational harm through off-topic or disparaging outputs, making topic restriction the central application problem.The prompt may reach the system directly from the user or through manipulated interaction.
  • Topic classifier: A topic classifier generally preserves usefulness but can falsely refuse requests and allows mixed prompts to pass when any part appears relevant.Its binary decision can therefore be exploited by combining related and unrelated questions.
  • Action selector: An allowlist of permitted requests strengthens restriction, but dissimilar benign requests may be falsely blocked and embeddings remain manipulable.Requests sufficiently similar to allowlisted examples are executed and returned to the user.
  • Context minimization: Context minimization removes the customer’s prompt after sanitizing the request and response, preventing it from injecting instructions into post-processing.This supports safer summarization or formatting of query results.

4.6 Product Recommender

The product recommender analyzes online-store reviews to summarize products, but direct processing leaves it vulnerable to review-based and user-prompt injections. Processing reviews individually into fixed categories limits each review’s influence to its own product, trading flexibility for stronger isolation.

  • Description: The application uses an LLM to summarize the benefits and downsides of products by analyzing retrieved online-store reviews.
  • Threat Model: Review injections can boost a product’s ranking or discredit competing products, while user-prompt injections can make the agent behave inappropriately.
  • Possible Designs: Directly feeding raw reviews to the LLM fully automates analysis but allows malicious reviews to manipulate recommendations.An injection could instruct the model to recommend another product instead.
  • Possible Designs: Data attribution asks the LLM to cite review snippets, but users must validate the attributions and manipulation may remain undetected.
  • Possible Designs: Map-reduce processing sanitizes each review into fixed categories before aggregation, preventing one review’s injection from influencing other reviews or products.Users may need to supply relevant categories, although the LLM can propose them.

4.7 Resume Screening Assistant

The resume-screening assistant ranks and summarizes candidates but must handle malicious instructions embedded in resumes. Designs range from flexible but vulnerable raw processing to isolated retrieval and dual-LLM formatting, with security gains accompanied by utility or coverage trade-offs.

  • Description: The assistant ranks resumes, answers questions about candidates, and compares candidates using untrusted resume data.
  • Possible Designs: Direct processing supports arbitrary ranking criteria but allows malicious resumes to boost their own rankings.
  • Possible Designs: Action sandboxing with RAG avoids processing raw resumes, making the base design robust to injections while retaining similarity-based ranking.Optional summarization improves utility but exposes the top K resumes to prompt injections.
  • Possible Designs: Map-reduce retrieval uses isolated LLMs to sanitize each resume into a predetermined format, enabling flexible ranking while isolating resume-level processing.
  • Possible Designs: Dual LLM summarization protects the privileged template-generating LLM from top-resume injections, but unprivileged LLM outputs can still be manipulated.Its utility depends on template quality, and raw processing accuracy may be constrained by context length.

4.8 Medication Leaflet Chatbot

The medication chatbot must answer only from trusted leaflets while avoiding illegal or incorrect responses. Data attribution is transparent but vulnerable, whereas context minimization ensures returned text comes from the leaflet while potentially omitting relevant content or remaining vulnerable during retrieval.

  • Threat Model: The chatbot answers medication questions from leaflets assumed to be trusted and must avoid competitor statements, unsupported medical advice, and incorrect answers.
  • Possible Designs: Data attribution makes source selection transparent and fully automates the functionality, but does not guarantee correctness or prevent illegal responses.
  • Possible Designs: Context minimization retrieves relevant leaflet passages and generates a summary without consulting the user question.This automates relevant-information search but is somewhat inflexible.
  • Possible Designs: The context-minimization design ensures returned text comes from the leaflet, but adversarial users may manipulate retrieval and hide relevant contraindications.
  • Possible Designs: In the diagnosis variant, removing the patient prompt prevents injection through the diagnosis summary, but RAG responses and doctor-visible text remain potential attack surfaces.
  • Possible Designs: The strong context-minimization pattern also removes the symptoms summary, preventing prompt manipulation of the diagnosis summary while reducing responsiveness to the patient.
  • Possible Designs: Structured symptom formatting should preserve most utility when symptoms can be represented rigidly.

4.10 Software Engineering Agent

The software-engineering agent can read documentation, install packages, and modify repositories, creating risks from malicious third-party content. User confirmation, sandboxing, and strict dual-LLM formatting progressively constrain those risks while reducing utility or relying on sandbox design.

  • Description: The coding assistant has tool access to online documentation, software installation, commits, and pushes.
  • Threat Model: Imported documentation or third-party code could trigger malicious package installation or sensitive-data exfiltration through commits and web requests.
  • Possible Designs: User confirmation for sensitive actions is burdensome and may miss stealthy attacks that introduce malicious behavior indirectly.
  • Possible Designs: Action sandboxing restricts sensitive actions to trusted documentation or code sources, but security depends on the sandbox and selected tools.
  • Possible Designs: Dual-LLM strict data formatting exposes the agent only to formal API descriptions instead of arbitrary third-party code or documentation.Strict formatting makes it unlikely that prompt injections survive conversion, but removes natural-language descriptions and examples.

5 Conclusions & Recommendations

General-purpose agents remain unsecured with current capabilities, but application-specific agents can be secured through principled system design. The paper proposes six simple patterns, demonstrates them across ten case studies, and recommends combining patterns while clearly defining trust boundaries.

  • Current capabilities do not yet secure general-purpose agents, whereas application-specific agents can be secured through principled system design.
  • The paper proposes six intentionally simple design patterns for making AI agents resilient to prompt injection attacks.Their simplicity supports analysis and reasoning in high-stakes or safety-critical settings.
  • Ten case studies spanning diverse domains demonstrate the practical applicability of the proposed patterns.
  • Developers should prioritize application-specific agents that follow secure design patterns and clearly define trust boundaries.
  • Robust security requires combining design patterns because no single pattern is likely to suffice across all threat models or use cases.

A Best Practices for LLM Agent Security

The paper recommends conservative system practices for model privileges, user permissions, confirmations, and data attribution. These practices reduce avoidable risk but require careful attention to least privilege, usability, and the burden placed on users.

  • General best practices address model privileges, user permissions, user confirmations, and data attribution alongside system-level design patterns.
  • Action sandboxing: Action sandboxing defines minimal permissions and action granularity, including sanitized tools with fewer privileges and side effects.The paper contrasts a sanitized file-search tool with unrestricted shell commands such as find.
  • Strict data formatting: Strict data formatting constrains LLM outputs to well-specified formats such as JSON, enforced algorithmically or through validation and retry.
  • User permissions: Agent permissions should not exceed the authenticated user’s access rights and should be reduced further to limit avoidable damage.
  • User confirmation: Human confirmation can increase security for non-time-critical tasks, but fatigue, annoyance, and over-reliance can undermine its usefulness.Usability is crucial because users may ignore feedback or rely too heavily on algorithmic suggestions.
  • Data and action attribution: Agent outcomes should, when possible, explain reasoning or identify supporting data, while recognizing that attribution itself is difficult to make robust.
Loading 2506.08837v3…