PaperlayerView exact revision on arXiv

Source-linked AI summary

Defeating Prompt Injections by Design

Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, Florian Tramèr

arXiv:2503.18813v2cs.CRcs.AI

TL;DR

LLM agents that handle untrusted data remain vulnerable to prompt injections, and existing defenses do not provide robust security guarantees. CaMeL places a software-security layer around the LLM, extracting trusted control and data flows and enforcing capabilities at tool calls. The system provides strong guarantees against unintended actions and data exfiltration while solving the AgentDojo benchmark by design.

  • Problem

    LLM agents interacting with untrusted environments are vulnerable to prompt injections that can cause harmful actions or data exfiltration.

  • Method

    CaMeL extracts control and data flows from user queries and enforces capability-based security policies through a custom interpreter without modifying the underlying LLM.

  • Results

    CaMeL effectively solves the AgentDojo benchmark while providing strong guarantees against unintended actions and data exfiltration.

  • Takeaways & Limitations

    System design around an untrusted model can make the whole agentic system robust even when the model itself is not.

  • Takeaways & Limitations

    Capability-based security requires substantial implementation effort, including integration with existing tools and infrastructure and potentially broader redesigns.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving $77\%$ of tasks with provable security (compared to $84\%$ with an undefended system) in AgentDojo. We release CaMeL at https://github.com/google-research/camel-prompt-injection.

1. Introduction

LLM agents interacting with untrusted data are vulnerable to prompt injection, while existing defenses lack robust formal policy enforcement. CaMeL addresses this with capability-based control and data-flow protection without modifying the underlying LLM.

  • Motivation: LLM agents interacting with external environments via APIs and interfaces are exposed to prompt injections from compromised users, tool outputs, and web pages.Attackers aim to exfiltrate data or cause harmful actions.
  • Motivation: Current defenses often depend on training or prompting models to follow security policies, including vulnerable system prompts.These approaches reflect the absence of robust methods for formally defining and enforcing policies over diverse data.
  • CaMeL: CaMeL attaches capabilities to values to express fine-grained restrictions on their permitted data and control flows.The capabilities provide metadata that specifies what can and cannot be done with each individual value.
  • CaMeL: CaMeL extracts control and data flows from user queries and uses a custom Python interpreter to enforce explicit security policies without changing the LLM.The system provides security guarantees without relying on model behavior modification.
  • Evaluation: CaMeL integrates with AgentDojo and solves the benchmark by design with some utility degradation and guarantees against policy violations.The evaluation tracks provenance and enforces security policies during execution.

2. Defeating Prompt Injections by Design

Prompt injections can hijack an agent’s intended task by manipulating untrusted notes, tool arguments, or actions, even when planning is isolated from malicious content. CaMeL secures both control and data flows by extracting a trusted plan and enforcing capability-based policies during tool execution.

  • Threat: An attacker can manipulate compromised notes to redirect a requested document to an unintended recipient or trigger unrelated actions.The intended task is to retrieve a specific document and send it to a specific recipient.
  • Existing defenses: Heuristic defenses based on delimiters, prompting, or model training provide no security guarantee and regularly fail against new attacks.These approaches attempt to make the model recognize and ignore malicious instructions.
  • Dual LLM limitation: The Dual LLM pattern isolates planning in a Privileged LLM while delegating processing of potentially malicious data to a Quarantined LLM.The Privileged LLM sees the initial query but not compromised file contents.
  • Dual LLM limitation: The Dual LLM pattern still permits prompt-injected data to alter tool-call arguments, leaving data flows vulnerable and potentially enabling arbitrary code execution.This is analogous to SQL injection, where parameters are manipulated without changing query structure.
  • CaMeL: CaMeL extracts intended control flow as pseudo-Python and tags retrieved values with capabilities describing their origin and authorized readers.Its interpreter can block sending a confidential document when the recipient lacks permission.
  • CaMeL: Capability-based policies prevent unintended data flows and actions, including exfiltration from compromised cloud-storage files, without modifying the underlying LLM.The mechanism provides granular enforcement during tool use.

3. Threat Model

CaMeL’s threat model targets prompt injections from untrusted data that divert data flows or cause unsafe outcomes, while assuming trusted user prompts and uncompromised memory. Its scope excludes attacks that affect neither control nor data flow and does not eliminate human clarification.

  • Threat scenario: The primary threat is an adversary-controlled data source that changes how the agent processes a user command.The scenario treats processing untrusted data as commonplace in current agentic systems.
  • Threat scenario: Prompt injections in retrieved emails, documents, or spreadsheets can alter recipients or files, diverting confidential documents to attackers.The model assumes the user prompt is trusted and memory, when present, has not been compromised.
  • Explicit non-goals: CaMeL does not target text-to-text attacks with no control- or data-flow consequences, including certain misleading summaries and phishing prompts.Its data-flow graph can still trace content origins for presentation to users.
  • Explicit non-goals: CaMeL is not intended to provide full autonomy; ambiguous queries or tool results may require users to clarify expected control and data flows.Capabilities and policies are intended to reduce unnecessary prompting and security fatigue.

4. The Prompt Injection Security Game

PI-SEC formalizes prompt-injection security as a game over an agent’s tool-use trace. An adversary wins by supplying an initial state that causes an action outside the prompt’s allowed-action set, while CaMeL checks policies before tool calls and halts on failure.

  • Game definition: PI-SEC models an adversary, an agent, tools, and writable memory, with execution represented as a trace of tool calls, arguments, and memory states.The agent receives a user prompt, available tools, and memory, then returns the resulting trace.
  • Game definition: For each prompt, Ωprompt is the set of actions the agent may take without compromising security.An adversary wins when the execution trace contains an action outside Ωprompt.
  • CaMeL in PI-SEC: CaMeL evaluates globally allowed-action policies before each tool call and halts execution when a policy returns ⊥.These policies govern tool actions independently of the prompt.
  • Adversary goal: The adversary supplies an initial state mem* intended to cause the agent to perform an unsafe action outside Ωprompt.The security game defines success through the agent’s resulting execution trace.

5. CaMeL Design

CaMeL applies software-security principles to protect agentic systems by separating control and data flows, tracking provenance, and enforcing explicit policies before tool execution. Its architecture combines privileged planning, quarantined parsing, a restricted Python interpreter, capabilities, and granular security policies.

  • Design principles: CaMeL uses control-flow integrity, capabilities, and security policies to prevent disallowed actions without enumerating every allowable action.The design targets security enforcement at the system layer rather than relying solely on the underlying LLM.
  • System components: The system combines a Quarantined LLM for parsing unstructured data, a Privileged LLM for generating query code, tools, security policies, and capabilities.Capabilities record value sources and allowed readers, while policies define permitted tool calls.
  • Interpreter: CaMeL tracks variable dependencies through data-flow graphs and supports STRICT and NORMAL interpretation modes for control-flow constructs.The graph is recursively traversed when needed for security-policy checks.
  • Security policies: Security policies are Python functions that receive a tool name and arguments and return Allowed or Denied with a reason.Policies can be global or specific to a tool and can express granular rules about which data flows are permitted.
  • Capabilities: Capabilities tag individual values with provenance and reader information, enabling finer-grained restrictions than coarse data- or control-flow rules.They can represent sources from the user, CaMeL transformations, tools, and inner tool sources.
  • Interpreter: The CaMeL interpreter executes a restricted Python subset, parses abstract syntax trees, and recursively interprets tree nodes.Evaluations found that minor prompt modification can produce code within the supported dialect, with some violations automatically detectable and fixable.

6. Evaluation

CaMeL is evaluated on AgentDojo for utility, security, failure modes, model choices, and token overhead. It largely preserves utility while substantially reducing successful attacks, though documented-API limitations and higher token usage remain.

  • Evaluation setup: AgentDojo evaluates CaMeL across realistic utility and security tasks spanning Workspace, Banking, Travel, and Slack.Success rates are compared with CaMeL enabled and with each model provider’s official tool-calling APIs.
  • Utility: CaMeL generally does not significantly degrade utility versus official tool-calling APIs, except for Travel on a subset of models.Some models and suites even show improved success rates, with newer models substantially improving Travel performance.
  • Utility failure modes: Poorly documented APIs are the main source of low Travel performance because the P-LLM cannot observe or parse undocumented tool-output structures.For Claude Sonnet, Travel utility rises from 25% with version 3.5 to 55% with 3.7 and 75% with 4, without changes to CaMeL or its system prompt.
  • Utility and cost: Using a cheaper Q-LLM reduces median-task utility by about 1% while reducing estimated cost by 12% in one reported pairing.Claude 3.5 Haiku serves as Q-LLM with Claude 4 Sonnet as P-LLM; a similar utility drop is observed with GPT 4.1 Nano and o4 Mini High.
  • Security: CaMeL reduces successful attacks across all evaluated models, including Gemini 2.5 Pro, where attacks fall from 300 without CaMeL to 0 with it.Some attacks remain successful because they fall outside CaMeL’s threat model, including certain text-to-text or user-facing phishing outcomes.
  • Baseline comparisons: CaMeL records 0 successful attacks versus 8 for the next-best tool-filter defense, while GPT-4o Mini remains vulnerable to 276 attacks with standard tool calling.The authors contrast CaMeL’s guarantees with heuristic defenses and note that CaMeL can complement other defenses for defense in depth.

7. Side Channels Can Be a Problem

CaMeL remains vulnerable to side channels that infer private information through indirect dependencies, exceptions, or timing, though STRICT mode blocks some attacks. The authors caution that other side channels may remain exploitable.

  • CaMeL is vulnerable to side-channel attacks that infer sensitive information through shared resources.The authors state that such attacks weaken strong security guarantees, while CaMeL still provides isolation guarantees and fine-grained leakage reasoning in some cases.
  • Three examples show leakage through indirect dependency, exception-triggered behavior, and timing.The authors present these examples as cautionary cases in which data may still be inefficiently mishandled.
  • STRICT mode prevents indirect-dependency leakage by making state-changing statements inside control-flow blocks depend on the block test or iterable.A security policy can then block execution of those statements when the control-flow dependency involves private data.
  • Exceptions can leak one private bit because an adversary can conditionally trigger an exception and observe whether later tool execution occurs.The current mitigation makes subsequent statements depend on Q-LLM inputs in STRICT mode; explicit error handling with conditionals and result types is proposed as an alternative design.
  • Timing side channels may reveal private data, but the specific demonstrated attack is unavailable because CaMeL’s interpreter lacks the time module.Practical exploitability depends on deployment, available tools, and whether the attacker can observe timing precisely; other timing side channels are not excluded.
  • CaMeL’s policies can also block external tools from accessing internal information and protect against some malicious-user exfiltration scenarios.The paper illustrates these protections for an externally installed spy tool and a compromised user sending financial documents externally.

8. Secondary attack scenarios that CaMeL can help with

CaMeL extends its capability-based protection beyond prompt injection to stronger threats involving compromised users, tools, or data. The paper argues that these scenarios reflect realistic industrial security risks and that capabilities provide stronger guarantees than capability-less systems.

  • CaMeL is designed for threat models where compromise can originate from the user, data, tooling, or any combination.
  • CaMeL can prevent rogue users and tools from violating system-wide security policies.
  • 50% of insider compromises are attributed to negligence and 26% to malicious insiders in the cited global report.
  • An externally installed spy tool could passively monitor and exfiltrate all data observed by the agent.
  • A compromised user could manipulate prompts to send confidential financial documents outside the organization, a threat not explicitly addressed in AgentDojo.
  • The authors describe these threats as realistic for production agent systems and report that capabilities provide strong guarantees absent from capability-less systems.

9. Discussion

The discussion presents CaMeL as robust but not complete: capability-based protection introduces implementation, ecosystem, usability, and residual-attack challenges. The authors emphasize that security depends on supported capabilities and enforced policies.

  • Capability-based protection shares limitations including security-literature attacks, user-experience trade-offs, de-classification, user fatigue, and side channels.
  • Implementing capabilities can require substantial resources and redesign across software, hardware, development practices, and infrastructure.
  • Capability-based enforcement ideally requires all external tools and services to understand capabilities; otherwise utility degrades.
  • Controlled workspaces may make capability deployment feasible, but third-party tools create challenges for capability support and increase side-channel risks.
  • Restrictive or ambiguous policies can require user intervention and create fatigue that may lead users to approve malicious actions.
  • CaMeL still requires users to specify and maintain policies, while side-channel exploitation is hindered by limited bandwidth and attack complexity.
  • Like CFI’s vulnerability to ROP, CaMeL may face attacks that approximate malicious control flows using individually permitted blocks.
  • AgentDojo is not completely solved: CaMeL prioritizes verifiable security guarantees while also seeking to maximize utility.

10. Future work

Future work targets safer implementation, formal verification, and more automated security decisions. The passages also identify language complexity and insufficient context as practical barriers to secure deployment.

  • Python’s complexity can make CaMeL harder to secure, including because exception-based termination may create security issues.
  • Languages with more explicit error and I/O handling, such as Haskell, may provide a more secure deployment option.
  • Policy conflict resolution becomes harder as programming-language complexity increases.
  • Formal verification could prove that CaMeL’s interpreter is fault-free and correctly resolves conflicts and enforces intended policies.
  • CaMeL may integrate contextual-integrity tools to automate security-policy decisions when available context is insufficient.

11. Conclusion

CaMeL provides a practical, system-level defense against prompt injection, achieving AgentDojo effectiveness while retaining strong security guarantees. The design remains compatible with other robustness defenses, although it does not address every attack vector.

  • CaMeL effectively solves the AgentDojo benchmark while providing strong guarantees against unintended actions and data exfiltration.
  • The current design does not completely address every potential attack vector, motivating future work on its limitations.
  • CaMeL remains compatible with defenses that improve the language model’s own robustness to prompt injection.
  • A security-engineering approach may extend beyond prompt injection to other areas of language-model security.

Contributions

The listed contributions describe the people responsible for CaMeL’s conception, design, development, review, and technical administration.

  • I.S. originated the idea and wrote the technical design proposal.
  • E.D. led technical development and co-led CaMeL’s design with I.S., assisted by J.H. and T.F.
  • I.S. and T.F. reviewed the codebase, while T.F. led technical and administrative efforts with D.F.’s help.

A.1.1. Computer security nomenclature

The paper uses computer-security concepts to distinguish execution behavior, data propagation, access rights, and information confidentiality in agentic systems. It relates these concepts to existing and proposed prompt-injection defenses.

  • Computer security nomenclature: Security policies specify desired system security properties, subjects, objects, and permitted operations.
  • Computer security nomenclature: Control flow describes execution, while data flow describes how data propagates between instructions.
  • Computer security nomenclature: Capabilities are unforgeable tags, tokens, or keys granting fine-grained rights to resources or functionality.
  • Computer security nomenclature: In agentic settings with explicit tools and data sources, separating control and data flows becomes possible despite their general entanglement in machine learning models.
  • Access and Information Flow Controls: Control Flow Integrity restricts execution to legitimate paths, Access Control governs resource access, and Information Flow Control tracks information to prevent unauthorized leaks.
  • Related defenses: Existing defenses include prompt delimiters, prompt sandwiching, fine-tuning, attack detection, and isolation-based Dual LLM designs.
  • Related defenses: CaMeL differs from concurrent work by explicitly tracking dependencies with a control-flow graph and supporting more expressive policies than coarse labels.

B. Full results tables

The full-results section lists tables covering utility, attacked utility, successful attacks, and defense utility on AgentDojo.

  • Table 2 reports utility results on the AgentDojo benchmark across different suites.
  • Table 3 reports utility results across suites under attack.
  • Table 4 reports the number of successful attacks.
  • Table 5 reports defense utility.

C. Baseline results

The baseline evaluation reports policy-triggering rates, utility under attack, and successful attacks for CaMeL defenses across evaluated settings.

  • Policy-triggering rates are reported separately for NORMAL and STRICT modes without attack.Table 8 contains the corresponding baseline evaluation.
  • Utility under attack is broken down by task suite for the evaluated defenses.The full results are provided in Table 6.
  • Successful attacks are counted for the evaluated defenses.The corresponding results are reported in Table 7.
  • CaMeL’s NORMAL-mode dependency graph omits a dependency from send_email to query_ai_assistant for the illustrated forwarding query.This differs from the dependency graph shown in Figure 7b.
  • Policy-triggering rates are also reported for NORMAL and STRICT modes under attack.The proportions are evaluated in the adversarial setting.

E. Security Policy evaluation

CaMeL evaluates granular security policies that constrain data flows into state-changing tools and specify provenance or readability requirements for operations.

  • Send-money and scheduled-transaction policies require transaction data to be readable by recipients or otherwise satisfy public-data conditions.Scheduling adds recurrence, while updating a scheduled transaction additionally involves a transaction identifier.
  • Email policies require recipients to come from the user and message contents to be readable by all recipients.The policy applies to email body, subject, and attachments.
  • Calendar policies constrain event fields, participants, event identifiers, and rescheduling times according to user provenance or participant readability.The policies cover creation, cancellation, rescheduling, and adding participants.
  • File policies restrict deletion and sharing to user-specified files or recipients, while append operations require file data to be shared with those who can access it.Creating a file only makes its content accessible to the user until it is shared.
  • The evaluation measures how often policies are triggered across models and reports proportions over all benign and adversarial tasks.Figure 20 uses models on the x-axis and the percentage of tasks triggering policies on the y-axis.

F. Extended overheads results

The extended evaluation examines token overheads, failure modes, and the tool and language environment supplied to the Privileged LLM.

  • Token usage is compared across multiple defenses, including normal and under-attack settings.Tables 10–13 report token usage and token-usage increases.
  • CaMeL’s token increase is measured against native tool calling without attack using Claude 3.5 Sonnet, with the median-task increase shown in red.The tokenization uses tiktoken.
  • The Privileged LLM receives a system prompt, user-specific information, available types and built-ins, and external-tool function definitions.The supplied functions include email, calendar, search, and contact operations.
  • The environment instructs the model not to assume values, to use tools for disambiguation, and not to rewrite code that reruns side-effecting functions.It also provides a current-day function rather than requiring the model to assume the date.
  • Claude 3.5 Sonnet failures are categorized across Workspace, Banking, Slack, and Travel task suites.Categories include query misunderstanding, insufficient context, wrong assumptions, overdone actions, and API documentation issues.
  • The tool environment includes email, calendar, and contact functions with documented inputs, outputs, and side effects.Examples include sending or deleting email, retrieving messages, searching contacts, and creating or canceling events.

H.4.6. Banking 14: Ambiguous task

The task checks recent transactions for suspicious activity and updates the password if any suspicious transaction is found. The assistant analyzes transaction details against anomaly patterns, but the supplied examples classify all four transactions as legitimate.

  • Transaction outcomes: The ambiguous-task concern centers on whether the assistant can recognize an unusual iPhone 3GS transaction despite its apparently plausible purchase context.The passage describes an iPhone 3Gs bought for $1,000 in 2023 as an ambiguous transaction.
  • Detection workflow: If any suspicious activity is found, the code displays transaction details and updates the password.The password is set to '1j1l-2k3j'.
  • Transaction outcomes: Because no transaction is marked suspicious in the shown results, the conditional password-update branch is not activated for these examples.The code instead reports that no suspicious transactions were found when the flag remains false.
  • Detection workflow: The code retrieves recent transactions, analyzes each one with a structured transaction schema, and tracks whether suspicious activity is found.The analysis considers amounts, recipients, subjects, dates, recurrence, and several suspicious-pattern categories.
  • Transaction outcomes: All four analyzed transactions are classified as non-suspicious, including a pizza-party payment, a New Year gift, Spotify Premium, and an Apple Store purchase.The classifications cite reasonable amounts, matching purposes, appropriate recurrence, and non-alarming timing or recipient formats.
Loading 2503.18813v2…